![\"z=\\sum_iw_ix_i+b\"](\"assets/equations/23c659c8111d9781.svg\")
The supplied timetable schedules Deep Learning MT2 for Tuesday, 13 October 2026, 3:00–4:00 pm, for 10 marks.
","Faculty update received on 12 October: prepare Unit 3 (optimization and regularization) and Unit 4 (CNNs) for theory and numericals. The message highlights possible Unit 3 numericals and two Unit 4 numericals. The 2-mark questions can combine theory and a numerical or be theory only.
","Three 3-mark choices, as clarified: either Numerical + Numerical + Theory, or one mixed Theory-and-Numerical question + one Numerical + one Theory question. Their exact order and numerical topics are not specified.
","The common M2 layout already supplied is Q1(a,b), 2 marks each, compulsory; answer any two of Q2–Q4, 3 marks each. Our mocks use this layout. Their exact topic allocation is practice design, not an official prediction.
","Maths, losses, gradients and the neuron model remain prerequisite explanations. RNN is not named in this faculty update and is now kept in the TEE route, outside this focused MT2 reading path.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c1-s0","search":"Examination schedule and scope The supplied timetable schedules Deep Learning MT2 for **Tuesday, 13 October 2026, 3:00–4:00 pm**, for **10 marks**.\n\n**Faculty update received on 12 October:** prepare **Unit 3 (optimization and regularization) and Unit 4 (CNNs)** for theory and numericals. The message highlights possible Unit 3 numericals and two Unit 4 numericals. The 2-mark questions can combine theory and a numerical or be theory only.\n\n**Three 3-mark choices, as clarified:** either **Numerical + Numerical + Theory**, or **one mixed Theory-and-Numerical question + one Numerical + one Theory question**. Their exact order and numerical topics are not specified.\n\nThe common M2 layout already supplied is **Q1(a,b), 2 marks each, compulsory; answer any two of Q2–Q4, 3 marks each**. Our mocks use this layout. Their exact topic allocation is practice design, not an official prediction.\n\nMaths, losses, gradients and the neuron model remain prerequisite explanations. **RNN is not named in this faculty update and is now kept in the TEE route, outside this focused MT2 reading path.**\n"},{"title":"Recommended reading order","blocks":["Study Optimization algorithms → Regularization and generalization → CNN calculations → CNN architectures. If a symbol or step is unfamiliar, use the Mathematics revision and Neural-network foundations chapters as short prerequisite references, then return to the Unit 3–4 lesson.
","For every numerical, reproduce: Given → To find → formula and why → numbered working → final answer. For theory, learn the technical definition, mechanism, labelled architecture or relevant formula, advantages/limitations and a concrete example where appropriate.
","Hinglish cue: Unit 3 mein gradient aur update ka order; Unit 4 mein window, shape aur parameter count alag rakho. Theory dono units ki karni hai.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c1-s1","search":"Recommended reading order Study **Optimization algorithms → Regularization and generalization → CNN calculations → CNN architectures**. If a symbol or step is unfamiliar, use the Mathematics revision and Neural-network foundations chapters as short prerequisite references, then return to the Unit 3–4 lesson.\n\nFor every numerical, reproduce: **Given → To find → formula and why → numbered working → final answer**. For theory, learn the technical definition, mechanism, labelled architecture or relevant formula, advantages/limitations and a concrete example where appropriate.\n\n**Hinglish cue:** Unit 3 mein gradient aur update ka order; Unit 4 mein window, shape aur parameter count alag rakho. Theory dono units ki karni hai.\n"},{"title":"Preparation sessions before 13 October","blocks":["| Session | What to cover | What to write on paper |
|---|---|---|
| 1 | Essential prerequisite revision only | Derivative of the stated loss; meaning of gradient and learning rate; neuron/CNN notation |
| 2 | GD, batch/SGD/mini-batch, momentum and Nesterov | Two iterations, update counts, old versus new state, lookahead gradient |
| 3 | AdaGrad, RMSProp and Adam | Accumulators, bias correction, sign changes and complete two-step working |
| 4 | Regularization and generalization | L1/L2, dropout, early stopping, underfitting/overfitting, training/validation/test; theory and numerical examples |
| 5 | Convolution, stride, padding and pooling | Every window product/sum, multi-channel bias, floor and SAME-padding cases, max/min/average pooling |
| 6 | CNN shapes, parameters and architectures | Full layer table, flatten/dense count, bottlenecks, residual/concatenation, architecture and transfer-learning theory |
| 7 | Mock A | Two Unit 4 numerical choices and Unit 3 theory; full answers directly below each question |
| 8 | Mock B and final revision | Mixed Unit 3/4 numerical choices; review both units' theory and redo incorrect calculations |
These are short study blocks, not eight full days. The exam is on 13 October; spend the remaining time on the Unit 3–4 blocks you have not completed. Do not spend this MT2 preparation period on the extra RNN material retained in TEE.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c1-s2","search":"Preparation sessions before 13 October | Session | What to cover | What to write on paper |\n| 1 | Essential prerequisite revision only | Derivative of the stated loss; meaning of gradient and learning rate; neuron/CNN notation |\n| 2 | GD, batch/SGD/mini-batch, momentum and Nesterov | Two iterations, update counts, old versus new state, lookahead gradient |\n| 3 | AdaGrad, RMSProp and Adam | Accumulators, bias correction, sign changes and complete two-step working |\n| 4 | Regularization and generalization | L1/L2, dropout, early stopping, underfitting/overfitting, training/validation/test; theory and numerical examples |\n| 5 | Convolution, stride, padding and pooling | Every window product/sum, multi-channel bias, floor and SAME-padding cases, max/min/average pooling |\n| 6 | CNN shapes, parameters and architectures | Full layer table, flatten/dense count, bottlenecks, residual/concatenation, architecture and transfer-learning theory |\n| 7 | Mock A | Two Unit 4 numerical choices and Unit 3 theory; full answers directly below each question |\n| 8 | Mock B and final revision | Mixed Unit 3/4 numerical choices; review both units' theory and redo incorrect calculations |\n\nThese are short study blocks, not eight full days. The exam is on 13 October; spend the remaining time on the Unit 3–4 blocks you have not completed. Do not spend this MT2 preparation period on the extra RNN material retained in TEE.\n"},{"title":"Mock-paper conventions","blocks":["Both practice papers have 60 minutes and 10 attempted marks. Q1(a,b) contributes 4 compulsory marks; any two 3-mark questions contribute 6 marks. All five answers are provided so every alternative can be studied.
","Mock A practises theory-only Q1 plus Numerical + Numerical + Theory among Q2–Q4, with both long numericals from Unit 4. Mock B practises a theory/numerical Q1 plus Theory-and-Numerical mixed + Numerical + Theory among Q2–Q4. Its mixed question is from Unit 3; the other two questions are from Unit 4. These match the two clarified combinations; exact topic assignment is practice design.
","Use the Unit 3 and Unit 4 historical questions under Solved Questions for longer numerical variations and theory preparation. Historical 5/10-mark questions retain their original marks; they are not predictions of MT2 marks. The new faculty message takes priority over the older provisional course-policy scope inference.
","For Unit 3, be ready to differentiate the supplied objective, initialize optimizer state, compute the requested iterations and explain the mechanism. For Unit 4, be ready to compute feature maps, pool them, track tensor dimensions, count parameters, compare computational cost and explain the named architectures. Prepare theory from both units even when practising numericals.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c1-s3","search":"Mock-paper conventions Both practice papers have **60 minutes and 10 attempted marks**. Q1(a,b) contributes 4 compulsory marks; any two 3-mark questions contribute 6 marks. All five answers are provided so every alternative can be studied.\n\nMock A practises theory-only Q1 plus **Numerical + Numerical + Theory** among Q2–Q4, with both long numericals from Unit 4. Mock B practises a theory/numerical Q1 plus **Theory-and-Numerical mixed + Numerical + Theory** among Q2–Q4. Its mixed question is from Unit 3; the other two questions are from Unit 4. These match the two clarified combinations; exact topic assignment is practice design.\n\n### How the supplied papers ask questions\nUse the Unit 3 and Unit 4 historical questions under **Solved Questions** for longer numerical variations and theory preparation. Historical 5/10-mark questions retain their original marks; they are not predictions of MT2 marks. The new faculty message takes priority over the older provisional course-policy scope inference.\n\nFor Unit 3, be ready to differentiate the supplied objective, initialize optimizer state, compute the requested iterations and explain the mechanism. For Unit 4, be ready to compute feature maps, pool them, track tensor dimensions, count parameters, compare computational cost and explain the named architectures. Prepare theory from both units even when practising numericals.\n"},{"title":"Writing technical answers","blocks":["For theory, start with the definition, state the mathematical model or labelled architecture, explain its working, and add the relevant advantage, limitation or comparison. For a numerical, show given values, formula, intermediate states and final result. In iterative algorithms, use one row or clearly labelled block per iteration.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c1-s4","search":"Writing technical answers For theory, start with the definition, state the mathematical model or labelled architecture, explain its working, and add the relevant advantage, limitation or comparison. For a numerical, show given values, formula, intermediate states and final result. In iterative algorithms, use one row or clearly labelled block per iteration.\n"}]},{"n":2,"title":"Mathematics revision","sections":[{"title":"Before maths: what are we trying to learn?","blocks":["Imagine a tiny machine that estimates a bill. You tell it how many notebooks you bought. This information going INTO the machine is called the INPUT. Abhi bas naam samjho: input = andar diya hua number.
","Calculation: Input = 3 notebooks
","Suppose the machine says the bill is 30 rupees. This answer coming OUT is called the OUTPUT. Input aur output alag jobs hain: 3 was the quantity; 30 is the predicted bill.
","Calculation: Input: 3 notebooks → Output: 30 rupees
","If each notebook costs 10 rupees, three notebooks cost 10 + 10 + 10 = 30. Multiplication is just the shorter calculation: 3 × 10. In this simple model the multiplying number is called a WEIGHT.
","Calculation: 3 × 10 = 30
","Keep the input 3. Change only the weight from 10 to 8. The output becomes 3 × 8 = 24. A weight controls how strongly an input affects the answer. Neural-network weights can also be negative or fractions.
","Calculation: Old: 3 × 10 = 30. New: 3 × 8 = 24.
","The real bill is 30, but the machine predicted 24. The correct answer supplied for a training example is its TARGET. Our prediction is 6 too small because 30 − 24 = 6.
","Calculation: Target = 30. Prediction = 24. Difference = 6.
","For this tiny example, changing the weight from 8 to 10 changes the prediction from 24 to the correct 30. Learning means using examples to adjust weights and reduce prediction error. Later, gradient descent will calculate HOW to adjust them.
","Calculation: 3 × 8 = 24 → adjust weight → 3 × 10 = 30
","Now suppose there is a fixed packing fee of 2 rupees. First calculate the notebook cost: 3 × 10 = 30. Then add 2. That extra adjustable number is called a BIAS. In a neural network, it lets the result shift independently of the input.
","Calculation: 3 × 10 + 2 = 30 + 2 = 32
","Instead of writing input, weight and bias every time, books use short names. x means input; w means weight; b means bias. So x × w + b says exactly the calculation you already did. A letter is a named box holding a number.
","Calculation: x = 3, w = 10, b = 2 → x × w + b = 32
","A neural network connects many small calculating units called neurons. They multiply inputs by weights, add a bias and apply an activation rule. Several layers can learn more complex relationships. First we will learn the small calculation; phir layers jodenge.
","Calculation: Input → calculations in layers → prediction
","Question. A toy model has input 4, weight 5 and bias 2. Find its output using input × weight + bias.
","A toy model has input 4, weight 5 and bias 2.
","Find the preactivation and the requested neuron output.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","![\"a=f(z)\"](\"assets/equations/66df67478858e480.svg\")
Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Step 1: multiply input by weight: 4 × 5 = 20.
","Step 2: add the bias: 20 + 2 = 22. Final output = 22. The bias is added once, after multiplication.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c2-s0","search":"Before maths: what are we trying to learn? \n### Machine ko ek number do.\nImagine a tiny machine that estimates a bill. You tell it how many notebooks you bought. This information going INTO the machine is called the INPUT. Abhi bas naam samjho: input = andar diya hua number.\n\n\n\n**Calculation:** Input = 3 notebooks\n\n### Machine se ek answer niklega.\nSuppose the machine says the bill is 30 rupees. This answer coming OUT is called the OUTPUT. Input aur output alag jobs hain: 3 was the quantity; 30 is the predicted bill.\n\n\n\n**Calculation:** Input: 3 notebooks → Output: 30 rupees\n\n### Multiply kyun karte hain?\nIf each notebook costs 10 rupees, three notebooks cost 10 + 10 + 10 = 30. Multiplication is just the shorter calculation: 3 × 10. In this simple model the multiplying number is called a WEIGHT.\n\n\n\n**Calculation:** 3 × 10 = 30\n\n### Weight badla, answer badla.\nKeep the input 3. Change only the weight from 10 to 8. The output becomes 3 × 8 = 24. A weight controls how strongly an input affects the answer. Neural-network weights can also be negative or fractions.\n\n\n\n**Calculation:** Old: 3 × 10 = 30. New: 3 × 8 = 24.\n\n### Correct answer ko target kehte hain.\nThe real bill is 30, but the machine predicted 24. The correct answer supplied for a training example is its TARGET. Our prediction is 6 too small because 30 − 24 = 6.\n\n\n\n**Calculation:** Target = 30. Prediction = 24. Difference = 6.\n\n### Learning ka matlab weight improve karna.\nFor this tiny example, changing the weight from 8 to 10 changes the prediction from 24 to the correct 30. Learning means using examples to adjust weights and reduce prediction error. Later, gradient descent will calculate HOW to adjust them.\n\n\n\n**Calculation:** 3 × 8 = 24 → adjust weight → 3 × 10 = 30\n\n### Bias ek extra add hone wala number hai.\nNow suppose there is a fixed packing fee of 2 rupees. First calculate the notebook cost: 3 × 10 = 30. Then add 2. That extra adjustable number is called a BIAS. In a neural network, it lets the result shift independently of the input.\n\n\n\n**Calculation:** 3 × 10 + 2 = 30 + 2 = 32\n\n### Letters numbers ke naam hain.\nInstead of writing input, weight and bias every time, books use short names. x means input; w means weight; b means bias. So x × w + b says exactly the calculation you already did. A letter is a named box holding a number.\n\n\n\n**Calculation:** x = 3, w = 10, b = 2 → x × w + b = 32\n\n### Deep learning mein aage kya hoga?\nA neural network connects many small calculating units called neurons. They multiply inputs by weights, add a bias and apply an activation rule. Several layers can learn more complex relationships. First we will learn the small calculation; phir layers jodenge.\n\n\n\n**Calculation:** Input → calculations in layers → prediction\n\n**Question.** A toy model has input 4, weight 5 and bias 2. Find its output using input × weight + bias.\n\n\n\n### Given\n\nA toy model has input 4, weight 5 and bias 2.\n\n### To find\n\nFind the preactivation and the requested neuron output.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n### Solution — step by step\n\n\n\n\n**Step 1:** multiply input by weight: 4 × 5 = 20.\n\n*"},{"title":"Negative numbers are direction","blocks":["Zero is the reference point. Positive numbers such as 1, 2 and 3 lie to its right on a number line. Negative numbers such as −1, −2 and −3 lie to its left. The number −3 is three units below zero; it is not an instruction to subtract until another number is supplied.
","In 5 − 2, the minus sign is an operation: subtract 2 from 5. In 5 + (−2), the sign inside the brackets belongs to the number negative 2. Both expressions finish at 3. Brackets keep the signed number together.
","Start at 5. Adding positive 2 moves two places right, to 7. Adding negative 2 moves two places left, to 3.
","![\"\\begin{aligned}5+2&=7\\\\5+(-2)&=3\\end{aligned}\"](\"assets/equations/d25cc2f4afc0b12d.svg\")
If both numbers are negative, keep moving left. Start at −3 and add −2: the answer is −5.
","![\"(-3)+(-2)=-5\"](\"assets/equations/e6f43c9051ebbeef.svg\")
Subtracting positive 2 moves left. Subtracting negative 2 reverses a leftward movement, so it moves right. A money example is removing a debt of 2: your balance increases by 2.
","![\"5-(-2)=5+2=7\"](\"assets/equations/5db46d4affd9d09b.svg\")
The shortcut “two negatives make a positive” is not a rule for every calculation. Negative 3 plus negative 2 is still −5. First identify the operation: addition, subtraction, multiplication or division.
","For multiplication and division, equal signs give a positive result and different signs give a negative result. Calculate the ordinary size, then attach the correct sign.
","| Signs being multiplied | Complete calculation | Result sign |
|---|---|---|
| Positive × positive | 3 × 2 = 6 | Positive |
| Negative × positive | (−3) × 2 = −6 | Negative |
| Positive × negative | 3 × (−2) = −6 | Negative |
| Negative × negative | (−3) × (−2) = 6 | Positive |
Division uses the same sign pattern: (−6) ÷ 2 = −3 and (−6) ÷ (−2) = 3. The divisor must be nonzero.
","Question. Calculate −3 + 2, −3 − 2 and −3 − (−2).
","Answer — step by step.
","Expressions: −3+2; −3−2; −3−(−2).
","Evaluate the three signed-number expressions.
","Order of operations. Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.
","A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.
","![\"a-(-b)=a+b\"](\"assets/equations/204904c78b9dfc36.svg\")
Yaad rakhne ka cue: Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.
","Step 1: evaluate the first expression. start at −3 and move two places right. The answer is −1.
","![\"-3+2=-1\"](\"assets/equations/e35736ce1cfc9727.svg\")
Step 2: evaluate the second expression. start at −3 and move two places left. The answer is −5.
","![\"-3-2=-5\"](\"assets/equations/53bef08286131079.svg\")
Step 3: evaluate the third expression. replace subtract-negative-2 with add-positive-2. The answer is −1.
","![\"\\begin{aligned}-3-(-2)&=-3+2\\\\&=-1\\end{aligned}\"](\"assets/equations/2cc0c9e475076c06.svg\")
Later, a weight update subtracts a correction that can itself be negative. The third expression explains exactly why that update can increase a weight.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c2-s1","search":"Negative numbers are direction ### 1. The minus sign can describe a number\nZero is the reference point. Positive numbers such as 1, 2 and 3 lie to its right on a number line. Negative numbers such as −1, −2 and −3 lie to its left. The number −3 is three units below zero; it is not an instruction to subtract until another number is supplied.\n\nIn 5 − 2, the minus sign is an operation: subtract 2 from 5. In 5 + (−2), the sign inside the brackets belongs to the number negative 2. Both expressions finish at 3. Brackets keep the signed number together.\n### 2. Adding a negative moves left\nStart at 5. Adding positive 2 moves two places right, to 7. Adding negative 2 moves two places left, to 3.\n\nIf both numbers are negative, keep moving left. Start at −3 and add −2: the answer is −5.\n\n### 3. Subtracting reverses that movement\nSubtracting positive 2 moves left. Subtracting negative 2 reverses a leftward movement, so it moves right. A money example is removing a debt of 2: your balance increases by 2.\n\nThe shortcut “two negatives make a positive” is not a rule for every calculation. Negative 3 plus negative 2 is still −5. First identify the operation: addition, subtraction, multiplication or division.\n### 4. Multiplication has its own sign rule\nFor multiplication and division, equal signs give a positive result and different signs give a negative result. Calculate the ordinary size, then attach the correct sign.\n| Signs being multiplied | Complete calculation | Result sign |\n| Positive × positive | 3 × 2 = 6 | Positive |\n| Negative × positive | (−3) × 2 = −6 | Negative |\n| Positive × negative | 3 × (−2) = −6 | Negative |\n| Negative × negative | (−3) × (−2) = 6 | Positive |\nDivision uses the same sign pattern: (−6) ÷ 2 = −3 and (−6) ÷ (−2) = 3. The divisor must be nonzero.\n### 5. Fully worked arithmetic\n**Question.** Calculate −3 + 2, −3 − 2 and −3 − (−2).\n**Answer — step by step.**\n\n\n\n### Given\n\nExpressions: −3+2; −3−2; −3−(−2).\n\n### To find\n\nEvaluate the three signed-number expressions.\n\n### Formula — which one and why\n\n**Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.\n\nA minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.\n\n\n\n**Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.\n\n### Solution — step by step\n\n\n\n\n**Step 1: evaluate the first expression.** start at −3 and move two places right. The answer is −1.\n\n**Step 2: evaluate the second expression.** start at −3 and move two places left. The answer is −5.\n\n**Step 3: evaluate the third expression.** replace subtract-negative-2 with add-positive-2. The answer is −1.\n\nLater, a weight update subtracts a correction that can itself be negative. The third expression explains exactly why that update can increase a weight.\n"},{"title":"Fractions decimals and percentages","blocks":["The upper number is the numerator. The lower number is the denominator. In one-quarter, split one whole into four equal parts and take one part. The fraction bar means division.
","![\"\\frac14=1\\div4=0.25\"](\"assets/equations/13b13c0346dc6e67.svg\")
A decimal and a fraction can name the same amount. One-half is 0.5, one-tenth is 0.1, and one-hundredth is 0.01. The denominator cannot be zero: division by zero is undefined.
","![\"\\begin{aligned}\\frac12&=0.5\\\\\\frac1{10}&=0.1\\\\\\frac1{100}&=0.01\\end{aligned}\"](\"assets/equations/61c8b62ceed00f18.svg\")
Multiplying by 0.1 means taking one-tenth. Multiplying by 0.2 means taking two-tenths. For 0.2 × 0.3, express both numbers as fractions. Multiply the numerators and multiply the denominators.
","![\"\\begin{aligned}0.2\\times0.3&=\\frac2{10}\\times\\frac3{10}\\\\&=\\frac6{100}\\\\&=0.06\\end{aligned}\"](\"assets/equations/ac4c9d847d5a49fc.svg\")
This also explains a calculation used later: 0.1 × (−0.6) is one-tenth of negative 0.6, which is −0.06. The negative sign is kept because the factors have different signs.
","Two halves fit inside 1. Four halves fit inside 2. Therefore dividing 2 by one-half gives 4, even though division by a number greater than 1 would make 2 smaller.
","![\"\\begin{aligned}2\\div0.5&=2\\div\\frac12\\\\&=2\\times2\\\\&=4\\end{aligned}\"](\"assets/equations/e8f8b190ccce77f7.svg\")
The reciprocal flips a nonzero fraction: the reciprocal of 1/2 is 2/1. Dividing by a fraction means multiplying by its reciprocal. For decimals, multiplying both the numerator and denominator by the same nonzero number preserves their ratio.
","![\"\\begin{aligned}\\frac{0.8}{0.5}&=\\frac{0.8\\times10}{0.5\\times10}\\\\&=\\frac85\\\\&=1.6\\end{aligned}\"](\"assets/equations/e15debe515ad70bf.svg\")
To turn a percentage into a decimal, divide by 100. To turn a decimal into a percentage, multiply by 100.
","![\"\\begin{aligned}20\\%&=\\frac{20}{100}=0.2\\\\50\\%&=\\frac{50}{100}=0.5\\end{aligned}\"](\"assets/equations/c062ee6009f1f875.svg\")
If 20% is removed, 80% remains. As decimals: 1 − 0.2 = 0.8. The number 1 represents the whole amount, or 100%. This same arithmetic will be used for dropout later.
","For 2, 4 and 9, first add all three numbers: 2 + 4 + 9 = 15. Divide by the number of values, which is 3. The mean is 5. Do not divide by 9 just because it is the largest value.
","![\"\\text{mean}=\\frac{\\text{sum}}{\\text{count}}\"](\"assets/equations/4d51e471133d25a2.svg\")
Here the sum is 15 and the count is 3. Apply those two quantities to the formula.
","![\"\\begin{aligned}\\text{mean}&=\\frac{2+4+9}{3}\\\\&=\\frac{15}{3}\\\\&=5\\end{aligned}\"](\"assets/equations/503bca869d213251.svg\")
A weighted mean gives different shares of importance to the values. With shares 0.9 and 0.1, the first value receives 90% of the importance and the second receives 10%. The shares add to 1.
","Question. Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively.
","Answer — step by step.
","Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively.
","Find the weighted mean.
","Weighted sum. Multiply each value by its own coefficient and then add. For a weighted mean, divide by the sum of the weights unless that sum is already 1.
","xᵢ = value i; aᵢ = its weight. Each input keeps its own matching coefficient.
","![\"\\bar](\"assets/equations/305c0c88eaed50a3.svg\")
Yaad rakhne ka cue: Har number apne weight ke saath multiply; phir sab add.
","Step 1: multiply the first value by its weight: 0.9 × 10 = 9.
","Step 2: multiply the second value by its weight: 0.1 × 20 = 2.
","Step 3: add the contributions: 9 + 2 = 11. The weighted mean is 11. It is closer to 10 because 10 received most of the importance.
","![\"\\begin{aligned}0.9(10)+0.1(20)&=9+2\\\\&=11\\end{aligned}\"](\"assets/equations/eef093497c3c04cc.svg\")
If importance weights do not sum to 1, divide their weighted sum by the sum of the weights to obtain the weighted mean. A neuron's general weighted sum does not require weights to sum to 1 and is not automatically a mean.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c2-s2","search":"Fractions decimals and percentages ### 1. A fraction tells us to divide\nThe upper number is the **numerator**. The lower number is the **denominator**. In one-quarter, split one whole into four equal parts and take one part. The fraction bar means division.\n\n\nA decimal and a fraction can name the same amount. One-half is 0.5, one-tenth is 0.1, and one-hundredth is 0.01. The denominator cannot be zero: division by zero is undefined.\n\n### 2. Why small decimals become smaller when multiplied\nMultiplying by 0.1 means taking one-tenth. Multiplying by 0.2 means taking two-tenths. For 0.2 × 0.3, express both numbers as fractions. Multiply the numerators and multiply the denominators.\n\nThis also explains a calculation used later: 0.1 × (−0.6) is one-tenth of negative 0.6, which is −0.06. The negative sign is kept because the factors have different signs.\n### 3. Dividing by one-half asks how many halves fit\nTwo halves fit inside 1. Four halves fit inside 2. Therefore dividing 2 by one-half gives 4, even though division by a number greater than 1 would make 2 smaller.\n\nThe **reciprocal** flips a nonzero fraction: the reciprocal of 1/2 is 2/1. Dividing by a fraction means multiplying by its reciprocal. For decimals, multiplying both the numerator and denominator by the same nonzero number preserves their ratio.\n\n### 4. Percent means out of one hundred\nTo turn a percentage into a decimal, divide by 100. To turn a decimal into a percentage, multiply by 100.\n\nIf 20% is removed, 80% remains. As decimals: 1 − 0.2 = 0.8. The number 1 represents the whole amount, or 100%. This same arithmetic will be used for dropout later.\n### 5. A mean is a fair-share amount\nFor 2, 4 and 9, first add all three numbers: 2 + 4 + 9 = 15. Divide by the number of values, which is 3. The mean is 5. Do not divide by 9 just because it is the largest value.\n\nHere the sum is 15 and the count is 3. Apply those two quantities to the formula.\n\nA **weighted mean** gives different shares of importance to the values. With shares 0.9 and 0.1, the first value receives 90% of the importance and the second receives 10%. The shares add to 1.\n**Question.** Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively.\n**Answer — step by step.**\n\n\n\n### Given\n\nFind the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively.\n\n### To find\n\nFind the weighted mean.\n\n### Formula — which one and why\n\n**Weighted sum.** Multiply each value by its own coefficient and then add. For a weighted mean, divide by the sum of the weights unless that sum is already 1.\n\nxᵢ = value i; aᵢ = its weight. Each input keeps its own matching coefficient.\n\n\n\n**Yaad rakhne ka cue:** Har number apne weight ke saath multiply; phir sab add.\n\n### Solution — step by step\n\n\n\n\n**Step 1:** multiply the first value by its weight: 0.9 × 10 = 9.\n**Step 2:** multiply the second value by its weight: 0.1 × 20 = 2.\n**Step 3:** add the contributions: 9 + 2 = 11. The weighted mean is **11**. It is closer to 10 because 10 received most of the importance.\n\nIf importance weights do not sum to 1, divide their weighted sum by the sum of the weights to obtain the weighted mean. A neuron's general weighted sum does not require weights to sum to 1 and is not automatically a mean.\n"},{"title":"Brackets before everything else","blocks":["An expression is a written calculation. Brackets group part of that calculation. In 1 ÷ (1 + 3), the whole quantity 1 + 3 is the divisor, so calculate it before dividing.
","![\"\\frac1{1+3}=\\frac14=0.25\"](\"assets/equations/425b33a05c6633ee.svg\")
Removing the brackets changes the calculation. In 1 ÷ 1 + 3, division comes before addition: 1 ÷ 1 = 1, then 1 + 3 = 4.
","![\"1\\div1+3=1+3=4\"](\"assets/equations/1dbb10cc9ab1aacc.svg\")
First calculate inside brackets. Then evaluate powers and roots. Then do multiplication and division, reading left to right. Finally do addition and subtraction, again reading left to right. Multiplication does not have priority over division; they share a level.
","For 8 ÷ 2 × 3, calculate 8 ÷ 2 = 4 and then 4 × 3 = 12. Do not change it into 8 ÷ (2 × 3).
","Question. Evaluate (2 + 3)² ÷ 5.
","Answer — step by step.
","Evaluate (2 + 3)² ÷ 5.
","Evaluate each expression and state its value.
","Order of operations. Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.
","A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.
","Yaad rakhne ka cue: Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.
","Step 1: finish the brackets: 2 + 3 = 5. The expression becomes 5² ÷ 5.
","Step 2: square the bracket result: 5² = 25. The expression becomes 25 ÷ 5.
","Step 3: divide: 25 ÷ 5 = 5.
","![\"\\begin{aligned}\\frac{(2+3)^2}{5}&=\\frac{5^2}{5}\\\\&=\\frac{25}{5}\\\\&=5\\end{aligned}\"](\"assets/equations/113124579583cb24.svg\")
Question. Evaluate 2 + 3² × 4.
","Answer — step by step.
","Evaluate 2 + 3² × 4.
","Evaluate each expression and state its value.
","Order of operations. Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.
","A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.
","Yaad rakhne ka cue: Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.
","Step 1: calculate the power: 3² = 9. Keep the other numbers where they are: 2 + 9 × 4.
","Step 2: multiply: 9 × 4 = 36. The expression becomes 2 + 36.
","Step 3: add: 2 + 36 = 38.
","![\"\\begin{aligned}2+3^2\\times4&=2+9\\times4\\\\&=2+36\\\\&=38\\end{aligned}\"](\"assets/equations/ef5577a23a205b11.svg\")
In a printed fraction, everything below the bar belongs to the denominator. When entering that fraction in a single calculator line, put brackets around the complete denominator. A printed fraction with 1 above and 1 + 3 below must be entered as 1 ÷ (1 + 3).
"],"question":[],"answer":[],"kind":"read","id":"mt2-c2-s3","search":"Brackets before everything else ### 1. Brackets tell us what must stay together\nAn expression is a written calculation. Brackets group part of that calculation. In 1 ÷ (1 + 3), the whole quantity 1 + 3 is the divisor, so calculate it before dividing.\n\n\nRemoving the brackets changes the calculation. In 1 ÷ 1 + 3, division comes before addition: 1 ÷ 1 = 1, then 1 + 3 = 4.\n\n### 2. Use this order for the expressions in this course\nFirst calculate inside brackets. Then evaluate powers and roots. Then do multiplication and division, reading left to right. Finally do addition and subtraction, again reading left to right. Multiplication does not have priority over division; they share a level.\nFor 8 ÷ 2 × 3, calculate 8 ÷ 2 = 4 and then 4 × 3 = 12. Do not change it into 8 ÷ (2 × 3).\n### 3. A full example with brackets and a power\n**Question.** Evaluate (2 + 3)² ÷ 5.\n**Answer — step by step.**\n\n\n\n### Given\n\nEvaluate (2 + 3)² ÷ 5.\n\n### To find\n\nEvaluate each expression and state its value.\n\n### Formula — which one and why\n\n**Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.\n\nA minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.\n\n\n\n**Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.\n\n### Solution — step by step\n\n\n\n\n**Step 1:** finish the brackets: 2 + 3 = 5. The expression becomes 5² ÷ 5.\n**Step 2:** square the bracket result: 5² = 25. The expression becomes 25 ÷ 5.\n**Step 3:** divide: 25 ÷ 5 = **5**.\n\n### 4. A full example without grouping the addition\n**Question.** Evaluate 2 + 3² × 4.\n**Answer — step by step.**\n\n\n\n### Given\n\nEvaluate 2 + 3² × 4.\n\n### To find\n\nEvaluate each expression and state its value.\n\n### Formula — which one and why\n\n**Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.\n\nA minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.\n\n\n\n**Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.\n\n### Solution — step by step\n\n\n\n\n**Step 1:** calculate the power: 3² = 9. Keep the other numbers where they are: 2 + 9 × 4.\n**Step 2:** multiply: 9 × 4 = 36. The expression becomes 2 + 36.\n**Step 3:** add: 2 + 36 = **38**.\n\n### 5. A fraction bar groups the whole denominator\nIn a printed fraction, everything below the bar belongs to the denominator. When entering that fraction in a single calculator line, put brackets around the complete denominator. A printed fraction with 1 above and 1 + 3 below must be entered as 1 ÷ (1 + 3).\n"},{"title":"Powers roots and very small numbers","blocks":["In 2³, 2 is the base and 3 is the exponent. For a positive whole-number exponent, multiply that many copies of the base. It does not mean 2 × 3.
","![\"2^3=2\\times2\\times2=8\"](\"assets/equations/11e21b993b310c0e.svg\")
The exponent 2 is called a square: 3² = 3 × 3 = 9. Three rows of three cells make nine cells.
","Squaring negative 2 means multiplying negative 2 by negative 2. The equal signs of the factors give a positive result.
","![\"(-2)^2=(-2)(-2)=4\"](\"assets/equations/756026084f28a30c.svg\")
Without brackets, −2² means first calculate 2², then apply the minus sign: −(4) = −4. Enter brackets when the entire negative number is the base.
","For decimals, use the same rule. Negative 0.3 squared is positive 0.09 because 3/10 multiplied by 3/10 is 9/100.
","![\"\\begin{aligned}(-0.3)^2&=(-0.3)(-0.3)\\\\&=0.09\\end{aligned}\"](\"assets/equations/24f2cf37fb12267b.svg\")
Because 3 × 3 = 9, the principal square root of 9 is 3. Although (−3)² also equals 9, the symbol √9 by itself means the nonnegative root, 3.
","![\"\\begin{aligned}\\sqrt9&=3\\\\\\sqrt{0.09}&=0.3\\end{aligned}\"](\"assets/equations/2838a7934ce26d9d.svg\")
The absolute value of a number is its distance from zero: |−4| = 4 and |4| = 4. Squaring and taking a square root returns this nonnegative size.
","![\"\\sqrt{(-4)^2}=\\sqrt{16}=4\"](\"assets/equations/bed469d925782c9b.svg\")
Start with powers of 10. Each time the exponent decreases by 1, divide the previous value by 10: 1000 → 100 → 10 → 1 → 0.1 → 0.01. This explains both the zero power and the negative powers.
","| Power | Calculation | Value |
|---|---|---|
| 10³ | 10 × 10 × 10 | 1000 |
| 10² | 1000 ÷ 10 | 100 |
| 10¹ | 100 ÷ 10 | 10 |
| 10⁰ | 10 ÷ 10 | 1 |
| 10⁻¹ | 1 ÷ 10 | 0.1 |
| 10⁻² | 0.1 ÷ 10 | 0.01 |
| 10⁻³ | 0.01 ÷ 10 | 0.001 |
For a nonzero base, power zero is 1. A negative exponent means the reciprocal of the corresponding positive power. It does not mean a negative answer.
","![\"\\begin{aligned}10^{-3}&=\\frac1{10^3}\\\\&=\\frac1{1000}\\\\&=0.001\\end{aligned}\"](\"assets/equations/f45c43d6b073f1bb.svg\")
The notation 1 × 10⁻⁸ equals 0.00000001. Calculators may display it as 1E−8. In optimizer formulas, ε, pronounced “epsilon”, often names a small positive constant added to a denominator. Read its supplied value before calculating.
","Question. Calculate 1 − 0.9².
","Expression: 1−0.9².
","Evaluate the power first, then the subtraction.
","A square before subtraction. The exponent applies to 0.9. Squaring is repeated multiplication; perform it before subtracting the result from 1.
","q denotes the number being squared. Parentheses would be required to square the whole difference.
","![\"q^2=q\\times](\"assets/equations/12754a3d0dc07c58.svg\")
![\"1-q^2=1-(q\\times](\"assets/equations/5166132378f964d6.svg\")
Yaad rakhne ka cue: Jis number pe power hai, sirf uska square; subtraction baad mein.
","Step 1: square only the 0.9.
","![\"0.9^2=0.9(0.9)=0.81\"](\"assets/equations/3338c806844083ae.svg\")
Step 2: subtract the result from 1.
","![\"1-0.81=0.19\"](\"assets/equations/41335385ba7e04c7.svg\")
Final answer: 0.19. The square acts on 0.9, not on the entire difference.
","![\"\\begin{aligned}1-0.9^2&=1-0.81\\\\&=0.19\\end{aligned}\"](\"assets/equations/df575ab27bdf9309.svg\")
These notes assume familiarity with algebra and basic calculus. The revision here covers the notation used in neural-network equations and the conventions that affect numerical answers.
","| Notation | Technical meaning |
|---|---|
| x, y, ŷ | Input feature vector, target, model prediction |
| W, b, θ | Weight matrix, bias vector, complete trainable parameter vector |
| z, a | Preactivation (affine input) and activation (layer output) |
| L, J | Per-example loss and aggregate training objective |
| ∂L/∂w | Partial derivative with respect to one parameter |
| ∇θJ | Gradient: vector of partial derivatives with respect to θ |
| η, λ, ε | Learning rate, regularization coefficient, numerical stabilizer |
| Wᵀ, u⊙v | Transpose and elementwise multiplication |
| θₜ, W⁽ˡ⁾ | Update index t and layer index l; neither denotes a power |
Vectors are columns unless a question specifies otherwise. A weight matrix has one row per receiving neuron and one column per incoming feature. Write matrix dimensions before multiplying. A bias is added once per receiving neuron or output feature map.
","Keep at least six significant digits during intermediate calculations and round the final result as requested. The loss reduction matters: a sum, mean and half-sum produce different gradient factors. The location of ε also matters: a denominator √s + ε differs from √(s + ε).
","The sign of a gradient belongs to the derivative, not to the loss value. For example, w = 0.4, η = 0.1 and gradient −2 give the update below.
","![\"w_{\\mathrm{new}}=0.4-0.1(-2)=0.6\"](\"assets/equations/aab9aaca5264ba06.svg\")
In later calculations, state the optimizer convention, initial state and iteration number before substituting values. This prevents mixing momentum velocity with Adam's second-moment accumulator.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c2-s5","search":"Variables functions and symbols These notes assume familiarity with algebra and basic calculus. The revision here covers the notation used in neural-network equations and the conventions that affect numerical answers.\n\n| Notation | Technical meaning |\n| x, y, ŷ | Input feature vector, target, model prediction |\n| W, b, θ | Weight matrix, bias vector, complete trainable parameter vector |\n| z, a | Preactivation (affine input) and activation (layer output) |\n| L, J | Per-example loss and aggregate training objective |\n| ∂L/∂w | Partial derivative with respect to one parameter |\n| ∇θJ | Gradient: vector of partial derivatives with respect to θ |\n| η, λ, ε | Learning rate, regularization coefficient, numerical stabilizer |\n| Wᵀ, u⊙v | Transpose and elementwise multiplication |\n| θₜ, W⁽ˡ⁾ | Update index t and layer index l; neither denotes a power |\n\n### Conventions used in numerical solutions\nVectors are columns unless a question specifies otherwise. A weight matrix has one row per receiving neuron and one column per incoming feature. Write matrix dimensions before multiplying. A bias is added once per receiving neuron or output feature map.\n\nKeep at least six significant digits during intermediate calculations and round the final result as requested. The loss reduction matters: a sum, mean and half-sum produce different gradient factors. The location of ε also matters: a denominator √s + ε differs from √(s + ε).\n\nThe sign of a gradient belongs to the derivative, not to the loss value. For example, w = 0.4, η = 0.1 and gradient −2 give the update below.\n\nIn later calculations, state the optimizer convention, initial state and iteration number before substituting values. This prevents mixing momentum velocity with Adam's second-moment accumulator.\n"},{"title":"Exponentials and natural logarithms","blocks":["Definitions. The exponential function exp(x) = eˣ uses Euler's constant e. The natural logarithm ln is its inverse on positive real values. These functions occur in sigmoid, tanh, softmax and cross-entropy.
","![\"e^{-x}=\\frac1{e^x},\\qquad\\ln(e^x)=x\"](\"assets/equations/8726504f24b7b8cc.svg\")
![\"e^{\\ln](\"assets/equations/0f89b5131dee6b2d.svg\")
For positive constants and valid domains, exponent and log rules simplify expressions before a calculator is needed. For example, e^(−ln 2) = 1/2. Use natural ln for the loss formulas, not the calculator's base-10 log.
","![\"\\frac{d}{dz}e^{-z}=-e^{-z},\\qquad\\frac{d}{dp}[-\\ln](\"assets/equations/b8f3afdc22e0eaaf.svg\")
The minus sign in the first expression comes from differentiating the inner function −z. Applying the chain rule correctly here gives the sigmoid derivative derived in the activation section.
","Integration reverses differentiation up to an additive constant. The basic relevant identities are:
","![\"\\int](\"assets/equations/19176aa5b80f6365.svg\")
![\"\\int\\frac1x\\,dx=\\ln|x|+C,\\qquad\\int](\"assets/equations/b17aa54fb4500d95.svg\")
For angles in radians, d(sin x)/dx = cos x and d(cos x)/dx = −sin x. The identity sin²x + cos²x = 1 concerns ordinary trigonometric functions. Hyperbolic tanh is a different function; using tan instead of tanh gives an incorrect recurrent-network answer.
","![\"\\tanh](\"assets/equations/419a6fb08fcd40a6.svg\")
The supplied MT2 calculation families mainly require derivatives, matrix products, logarithms and elementary statistics. Extended integration or trigonometric exercises are not needed for those families.
","At z = 2, sigmoid = 1/(1 + e⁻²) ≈ 0.880797. At z = 1, tanh ≈ 0.761594. Keep unrounded values when a later gradient uses these activations.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c2-s6","search":"Exponentials and natural logarithms **Definitions.** The exponential function exp(x) = eˣ uses Euler's constant e. The natural logarithm ln is its inverse on positive real values. These functions occur in sigmoid, tanh, softmax and cross-entropy.\n\n\nFor positive constants and valid domains, exponent and log rules simplify expressions before a calculator is needed. For example, e^(−ln 2) = 1/2. Use natural ln for the loss formulas, not the calculator's base-10 log.\n\n### Derivatives used in neural networks\n\nThe minus sign in the first expression comes from differentiating the inner function −z. Applying the chain rule correctly here gives the sigmoid derivative derived in the activation section.\n\n### Brief integration and trigonometry reminder\nIntegration reverses differentiation up to an additive constant. The basic relevant identities are:\n\n\nFor angles in radians, d(sin x)/dx = cos x and d(cos x)/dx = −sin x. The identity sin²x + cos²x = 1 concerns ordinary trigonometric functions. Hyperbolic tanh is a different function; using tan instead of tanh gives an incorrect recurrent-network answer.\n\nThe supplied MT2 calculation families mainly require derivatives, matrix products, logarithms and elementary statistics. Extended integration or trigonometric exercises are not needed for those families.\n\n### Calculator values for activation calculations\nAt z = 2, sigmoid = 1/(1 + e⁻²) ≈ 0.880797. At z = 1, tanh ≈ 0.761594. Keep unrounded values when a later gradient uses these activations.\n"},{"title":"Calculator routine","blocks":["You reported a Casio ES-series calculator. The exact model string is not confirmed. The functions below are mathematical entries; use the keys labelled on your model. On fx-991 ES/fx-991 ES PLUS models, eˣ is normally the SHIFT function of ln. Do not confuse eˣ with the × 10ˣ scientific-notation key. Use the function labels on your own calculator for these entries.
","1. Use ordinary computation mode. Enter complete brackets around a denominator or negative exponent.
","2. For sigmoid at z = 2, enter 1 ÷ (1 + e^(−2)). Result: 0.880797078. Keep the full calculator value; write about six decimals during intermediate work.
","3. For tanh at z = 1, use the hyperbolic tanh function if available. A universal alternative is (e² − 1) ÷ (e² + 1), giving 0.761594156. Ordinary tan is a different function; do not use it.
","4. For cross-entropy with true probability 0.8, enter −ln(0.8), giving 0.223143551.
","5. For softmax [2.3, 4.5, 3.2], subtract 4.5 first, then calculate e⁻²·², 1, e⁻¹·³. Add them, then divide each exponential by the same total.
","6. If a result displays as a fraction, use your model’s fraction/decimal conversion key, often S⇔D. Do not reset the calculator unnecessarily.
","Five-minute warm-up: compute sigmoid(0) = 0.5; sigmoid(−2) ≈ 0.119203; tanh(0) = 0; −ln(0.5) ≈ 0.693147; √0.25 = 0.5. A different result signals an entry mistake, not a reason to memorize a wrong value.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c2-s7","search":"Calculator routine You reported a Casio ES-series calculator. The exact model string is not confirmed. The functions below are mathematical entries; use the keys labelled on your model. On fx-991 ES/fx-991 ES PLUS models, eˣ is normally the SHIFT function of ln. Do not confuse eˣ with the × 10ˣ scientific-notation key. Use the function labels on your own calculator for these entries.\n\n\n\n1. Use ordinary computation mode. Enter complete brackets around a denominator or negative exponent.\n2. For sigmoid at z = 2, enter 1 ÷ (1 + e^(−2)). Result: 0.880797078. Keep the full calculator value; write about six decimals during intermediate work.\n3. For tanh at z = 1, use the hyperbolic tanh function if available. A universal alternative is (e² − 1) ÷ (e² + 1), giving 0.761594156. Ordinary tan is a different function; do not use it.\n4. For cross-entropy with true probability 0.8, enter −ln(0.8), giving 0.223143551.\n5. For softmax [2.3, 4.5, 3.2], subtract 4.5 first, then calculate e⁻²·², 1, e⁻¹·³. Add them, then divide each exponential by the same total.\n6. If a result displays as a fraction, use your model’s fraction/decimal conversion key, often S⇔D. Do not reset the calculator unnecessarily.\n**Five-minute warm-up: ** compute sigmoid(0) = 0.5; sigmoid(−2) ≈ 0.119203; tanh(0) = 0; −ln(0.5) ≈ 0.693147; √0.25 = 0.5. A different result signals an entry mistake, not a reason to memorize a wrong value."},{"title":"A complete arithmetic calculation","blocks":["Evaluate 0.4 − 0.1 × (−2). The multiplication has priority over the subtraction.
","Step 1: multiply 0.1 by −2, giving −0.2.
","Step 2: substitute that entire result, including its sign: 0.4 − (−0.2).
","Step 3: subtracting a negative means adding its positive value: 0.4 + 0.2 = 0.6.
","The answer is 0.6. This calculation will later appear inside a weight update; the arithmetic is exactly the same.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c2-s8","search":"A complete arithmetic calculation Evaluate 0.4 − 0.1 × (−2). The multiplication has priority over the subtraction.\n\n\n\n**Step 1:** multiply 0.1 by −2, giving −0.2.\n**Step 2:** substitute that entire result, including its sign: 0.4 − (−0.2).\n**Step 3:** subtracting a negative means adding its positive value: 0.4 + 0.2 = 0.6.\nThe answer is **0.6**. This calculation will later appear inside a weight update; the arithmetic is exactly the same.\n"},{"title":"Loss: how one wrong prediction becomes a number","blocks":["Definition. A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative.
","![\"e_i=\\hat](\"assets/equations/ae0cc9bf3a93ca00.svg\")
For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective.
","When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual:
","![\"\\frac{\\partial\\mathrm{SSE}}{\\partial\\hat](\"assets/equations/3476c309d4e1d90b.svg\")
The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight.
","Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.
","Predictions = [0.2,0.7,0.4].
","Targets = [0,1,1]; n = 3 scalar predictions.
","Find SSE, half SSE and MSE, then all three prediction-gradient lists.
","Squared-error objectives. Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.
","ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.
","![\"e_i=\\hat](\"assets/equations/4b31e4fcfd52ba58.svg\")
![\"\\mathrm{SSE}=\\sum_i](\"assets/equations/a8dd5fa4e080919c.svg\")
![\"L_{\\mathrm{half}}=\\frac12\\sum_i](\"assets/equations/ef8982ea86700e61.svg\")
![\"\\mathrm{MSE}=\\frac1n\\sum_i](\"assets/equations/d9bd32ee9a4320cb.svg\")
Yaad rakhne ka cue: Pehle error square; phir question ke hisaab se sum, half ya mean.
","Step 1: calculate the three signed prediction errors.
","![\"e=[0.2-0,\\](\"assets/equations/23245cd97ea478bc.svg\")
Step 2: square every error.
","![\"e\\odot](\"assets/equations/1fddc4677249bcbb.svg\")
Step 3: add the three squared errors.
","![\"\\mathrm{SSE}=0.04+0.09+0.36=0.49\"](\"assets/equations/4890329b3b657194.svg\")
Step 4: apply the requested loss divisors.
","![\"L_{\\mathrm{half}}=\\frac{0.49}{2}=0.245,\\qquad](\"assets/equations/98422634bd86c12f.svg\")
Step 5: multiply every error by 2 for the SSE gradient.
","![\"\\nabla_{\\hat](\"assets/equations/fec22e50bd202c79.svg\")
Step 6: use each error itself for the half-SSE gradient.
","![\"\\nabla_{\\hat](\"assets/equations/dfa71d7fa81c823c.svg\")
Step 7: multiply every error by 2/3 for the MSE gradient.
","![\"\\nabla_{\\hat](\"assets/equations/a584bba4763cda9f.svg\")
The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c2-s9","search":"Loss: how one wrong prediction becomes a number \n**Definition.** A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative.\n\nFor this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective.\n\n### Derivatives and their scaling\nWhen differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual:\n\nThe factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight.\n\n**Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.**\n\n\n\n### Given\n\nPredictions = [0.2,0.7,0.4].\n\nTargets = [0,1,1]; n = 3 scalar predictions.\n\n### To find\n\nFind SSE, half SSE and MSE, then all three prediction-gradient lists.\n\n### Formula — which one and why\n\n**Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.\n\nŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: calculate the three signed prediction errors.**\n\n\n**Step 2: square every error.**\n\n\n**Step 3: add the three squared errors.**\n\n\n**Step 4: apply the requested loss divisors.**\n\n\n**Step 5: multiply every error by 2 for the SSE gradient.**\n\n\n**Step 6: use each error itself for the half-SSE gradient.**\n\n\n**Step 7: multiply every error by 2/3 for the MSE gradient.**\n\n\nThe residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate.\n\n"}]},{"n":3,"title":"The maths behind a network","sections":[{"title":"A vector is a list and a matrix is a table","blocks":["The input [2, 3] is a vector containing two features. A matrix is a rectangle of numbers. “2 by 3” means two rows and three columns. A column vector with two entries has shape 2 by 1. Always write rows before columns.
","In this course, unless a question’s diagram explicitly uses another convention, each row of a weight matrix belongs to one receiving neuron. Thus W has shape number of output neurons by number of input features, and the input is a column vector.
","![\"W=\\begin{bmatrix}0.11&0.21\\\\0.12&0.08\\end{bmatrix},\\quad](\"assets/equations/5226c7a9750fc29a.svg\")
Do it slowly: row 1 says 0.11 × 2 = 0.22 and 0.21 × 3 = 0.63, so its answer is 0.85. Row 2 says 0.12 × 2 = 0.24 and 0.08 × 3 = 0.24, so its answer is 0.48. Matrix multiplication is several weighted sums written together.
","Shape check: (2 by 2) × (2 by 1) gives (2 by 1). The inside numbers must match; the outside numbers are the answer shape. This check catches many transposed-weight errors before arithmetic begins.
","Transpose swaps rows and columns. Elementwise multiplication does not do row-by-column sums: [2, 3] ⊙ [4, 5] = [8, 15]. Dot product does add: [2, 3] · [4, 5] = 8 + 15 = 23. A training dropout mask uses elementwise multiplication, not a dot product.
","Practice. Multiply W = ![\"\\begin{bmatrix}1](\"assets/equations/af9cca7044f770c7.svg\")
by x = [2, 1] as a column. Answer: first row 1 × 2 + 2 × 1 = 4; second row 3 × 2 + 4 × 1 = 10. With b = [1, −1], Wx + b = [5, 9].
A row runs horizontally, from left to right. A column runs vertically, from top to bottom. A matrix is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows.
","A vector is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors.
","The superscript T means transpose, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m.
","![\"[A^T]_{ij}=A_{ji}\"](\"assets/equations/07cc0861475dad67.svg\")
The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1.
","Yaad rakhne ka cue: transpose = row ko column bana; number aur sign ko haath mat laga.
","Question. Find the transpose of the following matrix A and state the size of both matrices.
","![\"A=\\begin{bmatrix}2&1&3\\end{bmatrix}\"](\"assets/equations/7f5189f9c2294a43.svg\")
Aᵀ and both matrix sizes.
","A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.
","A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.
","Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.
","Step 1: write the original size.
","A has 1 row(s) and 3 column(s), so its size is 1 × 3.
","Step 2: move original row 1 into new column 1.
","Copy [2, 1, 3] down column 1, in the same order.
","Step 3: write the transposed matrix.
","![\"A^T=\\begin{bmatrix}2\\\\1\\\\3\\end{bmatrix}\"](\"assets/equations/dafe6e5e80b48abd.svg\")
Step 4: state its new size.
","The result has 3 row(s) and 1 column(s): 3 × 1.
","Final answer: Aᵀ has size 3 × 1. Transposing it again gives the original A.
","| 2 | 1 | 3 |
Read each colored row left to right.
| 2 |
| 1 |
| 3 |
Each original row is now the matching colored column.
The entry at row i, column j moves to row j, column i. Its value and sign do not change.
Question. Find the transpose of the following matrix A and state the size of both matrices.
","![\"A=\\begin{bmatrix}1&2&3\\\\4&5&6\\end{bmatrix}\"](\"assets/equations/19f2e9948f5cc39f.svg\")
Aᵀ and both matrix sizes.
","A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.
","A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.
","Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.
","Step 1: write the original size.
","A has 2 row(s) and 3 column(s), so its size is 2 × 3.
","Step 2: move original row 1 into new column 1.
","Copy [1, 2, 3] down column 1, in the same order.
","Step 3: move original row 2 into new column 2.
","Copy [4, 5, 6] down column 2, in the same order.
","Step 4: write the transposed matrix.
","![\"A^T=\\begin{bmatrix}1&4\\\\2&5\\\\3&6\\end{bmatrix}\"](\"assets/equations/78bde861f7b64a93.svg\")
Step 5: state its new size.
","The result has 3 row(s) and 2 column(s): 3 × 2.
","Final answer: Aᵀ has size 3 × 2. Transposing it again gives the original A.
","| 1 | 2 | 3 |
| 4 | 5 | 6 |
Read each colored row left to right.
| 1 | 4 |
| 2 | 5 |
| 3 | 6 |
Each original row is now the matching colored column.
The entry at row i, column j moves to row j, column i. Its value and sign do not change.
Question. Find the transpose of the following matrix A and state the size of both matrices.
","![\"A=\\begin{bmatrix}1&-2\\\\0&3\\end{bmatrix}\"](\"assets/equations/ffb51554b200bfc3.svg\")
Aᵀ and both matrix sizes.
","A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.
","A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.
","Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.
","Step 1: write the original size.
","A has 2 row(s) and 2 column(s), so its size is 2 × 2.
","Step 2: move original row 1 into new column 1.
","Copy [1, -2] down column 1, in the same order.
","Step 3: move original row 2 into new column 2.
","Copy [0, 3] down column 2, in the same order.
","Step 4: write the transposed matrix.
","![\"A^T=\\begin{bmatrix}1&0\\\\-2&3\\end{bmatrix}\"](\"assets/equations/3f6121853d314156.svg\")
Step 5: state its new size.
","The result has 2 row(s) and 2 column(s): 2 × 2.
","Final answer: Aᵀ has size 2 × 2. Transposing it again gives the original A.
","| 1 | −2 |
| 0 | 3 |
Read each colored row left to right.
| 1 | 0 |
| −2 | 3 |
Each original row is now the matching colored column.
The entry at row i, column j moves to row j, column i. Its value and sign do not change.
Question. Find the transpose of the following matrix A and state the size of both matrices.
","![\"A=\\begin{bmatrix}-1\\\\4\\\\2\\end{bmatrix}\"](\"assets/equations/b21656f3402af964.svg\")
Aᵀ and both matrix sizes.
","A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.
","A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.
","Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.
","Step 1: write the original size.
","A has 3 row(s) and 1 column(s), so its size is 3 × 1.
","Step 2: move original row 1 into new column 1.
","Copy [-1] down column 1, in the same order.
","Step 3: move original row 2 into new column 2.
","Copy [4] down column 2, in the same order.
","Step 4: move original row 3 into new column 3.
","Copy [2] down column 3, in the same order.
","Step 5: write the transposed matrix.
","![\"A^T=\\begin{bmatrix}-1&4&2\\end{bmatrix}\"](\"assets/equations/5b69eb45dcd67397.svg\")
Step 6: state its new size.
","The result has 1 row(s) and 3 column(s): 1 × 3.
","Final answer: Aᵀ has size 1 × 3. Transposing it again gives the original A.
","| −1 |
| 4 |
| 2 |
Read each colored row left to right.
| −1 | 4 | 2 |
Each original row is now the matching colored column.
The entry at row i, column j moves to row j, column i. Its value and sign do not change.
A dot product turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed.
","![\"w^Tx=w_1x_1+w_2x_2+w_3x_3\"](\"assets/equations/97799e8f51bd1031.svg\")
This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products.
","Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.
","Weights w = [2, 3].
","Matching inputs x = [4, 1].
","Both lists contain 2 numbers.
","One scalar value wᵀx.
","A dot product is a sum of matching products. There is no bias in this question.
","Σ means add; the index i visits each matching pair.
","![\"w^Tx=\\sum_i](\"assets/equations/ce95abc5189aaf50.svg\")
Yaad rakhne ka cue: Same position multiply; sab products add.
","Step 1: multiply matching pair 1.
","![\"(2)(4)=8\"](\"assets/equations/fb39ee363580ed9c.svg\")
Step 2: multiply matching pair 2.
","![\"(3)(1)=3\"](\"assets/equations/9934bcd789a00e7d.svg\")
Step 3: add the products.
","![\"w^Tx=(8)+(3)=11\"](\"assets/equations/4b366d83f24fd5dd.svg\")
Final answer: wᵀx = 11.
","Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.
","Weights w = [1, -2].
","Matching inputs x = [2, 3].
","Both lists contain 2 numbers.
","One scalar value wᵀx.
","A dot product is a sum of matching products. There is no bias in this question.
","Σ means add; the index i visits each matching pair.
","Yaad rakhne ka cue: Same position multiply; sab products add.
","Step 1: multiply matching pair 1.
","![\"(1)(2)=2\"](\"assets/equations/7259499711e389d2.svg\")
Step 2: multiply matching pair 2.
","![\"(-2)(3)=-6\"](\"assets/equations/02560299473dc26e.svg\")
Step 3: add the products.
","![\"w^Tx=(2)+(-6)=-4\"](\"assets/equations/368a39aeec6c928d.svg\")
Final answer: wᵀx = -4.
","Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.
","Weights w = [0.5, 1, -1].
","Matching inputs x = [4, 2, 3].
","Both lists contain 3 numbers.
","One scalar value wᵀx.
","A dot product is a sum of matching products. There is no bias in this question.
","Σ means add; the index i visits each matching pair.
","Yaad rakhne ka cue: Same position multiply; sab products add.
","Step 1: multiply matching pair 1.
","![\"(0.5)(4)=2\"](\"assets/equations/e3430445c8f6a121.svg\")
Step 2: multiply matching pair 2.
","Step 3: multiply matching pair 3.
","![\"(-1)(3)=-3\"](\"assets/equations/c5d478557eb834e1.svg\")
Step 4: add the products.
","![\"w^Tx=(2)+(2)+(-3)=1\"](\"assets/equations/bf7dc79763479e35.svg\")
Final answer: wᵀx = 1.
","To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B.
","![\"C_{ij}=\\sum_{k=1}^{n}A_{ik}B_{kj}\"](\"assets/equations/d566b516b84686ee.svg\")
For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once.
","Yaad rakhne ka cue: multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column.
","Definition. A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation.
","![\"z=Wx+b\"](\"assets/equations/28b5d89094e7d00d.svg\")
For d input features and m receiving neurons, the dimensions are:
","![\"W\\in\\mathbb](\"assets/equations/9fb325b4580ff43a.svg\")
The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product.
","Question. Given W = ![\"\\begin{bmatrix}1](\"assets/equations/aad1d6a46cb2c7f1.svg\")
, x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b.
Weights W = .
Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ.
","Find both components of z = Wx+b.
","Row–column multiplication. Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.
","Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.
","![\"z_j=\\sum_i](\"assets/equations/dc40cdac8ad4c81c.svg\")
Yaad rakhne ka cue: Ek row × poora input column = ek receiving neuron ka answer.
","Step 1: verify compatible dimensions.
","W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column.
","Step 2: calculate the first receiving row products.
","![\"1(2)=2\"](\"assets/equations/07ce1f7a3ced3979.svg\")
![\"(-2)(1)=-2\"](\"assets/equations/28ad4ad477ec80b7.svg\")
Step 3: sum row 1 and add its bias.
","![\"z_1=2+(-2)+0.1=0.1\"](\"assets/equations/9f6eacd9d3a609cc.svg\")
Step 4: calculate the second receiving row products.
","![\"0.5(2)=1\"](\"assets/equations/b4b945ff5abf26e2.svg\")
![\"3(1)=3\"](\"assets/equations/464c4958d256b6c8.svg\")
Step 5: sum row 2 and add its bias.
","![\"z_2=1+3-0.2=3.8\"](\"assets/equations/b340e43d0e100bc4.svg\")
![\"z_1=1(2)+(-2)(1)+0.1=0.1\"](\"assets/equations/133f56723bb2a1d8.svg\")
| 1 | −2 |
| 0.5 | 3 |
| 2 |
| 1 |
| 2 | −2 |
First × first; second × second.
One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component.
![\"z_2=0.5(2)+3(1)-0.2=3.8\"](\"assets/equations/dbd8ceb08ae05e33.svg\")
| 1 | −2 |
| 0.5 | 3 |
| 2 |
| 1 |
| 1 | 3 |
First × first; second × second.
One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component.
![\"z=\\begin{bmatrix}0.1\\\\3.8\\end{bmatrix}\"](\"assets/equations/0bcbcaa175e8ffb8.svg\")
An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W.
","Question. Multiply A = by B = ![\"\\begin{bmatrix}2](\"assets/equations/969e894914a74419.svg\")
. Show all four output entries.
![\"A=\\begin{bmatrix}1&2\\\\3&4\\end{bmatrix}\"](\"assets/equations/f32200eba37c0ea5.svg\")
![\"B=\\begin{bmatrix}2&0\\\\1&3\\end{bmatrix}\"](\"assets/equations/d45fb6a21c1dc67e.svg\")
The 2 × 2 matrix C = AB.
","Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication.
","Cᵢⱼ is the entry in output row i, column j.
","![\"C_{ij}=\\sum_k](\"assets/equations/46672bbce5b57e87.svg\")
Yaad rakhne ka cue: Output ki position dekho: wahi row, wahi column.
","Step 1: check the dimensions.
","(2 × 2)(2 × 2) is valid; output size = 2 × 2.
","Step 2: row 1 with column 1.
","![\"C_{11}=1(2)+2(1)=2+2=4\"](\"assets/equations/646952b8a744b648.svg\")
Step 3: row 1 with column 2.
","![\"C_{12}=1(0)+2(3)=0+6=6\"](\"assets/equations/2d6dd50233caecfb.svg\")
Step 4: row 2 with column 1.
","![\"C_{21}=3(2)+4(1)=6+4=10\"](\"assets/equations/c49b369448ab5419.svg\")
Step 5: row 2 with column 2.
","![\"C_{22}=3(0)+4(3)=0+12=12\"](\"assets/equations/b9ab897087e15f9b.svg\")
Step 6: assemble the output by position.
","![\"C=\\begin{bmatrix}4&6\\\\10&12\\end{bmatrix}\"](\"assets/equations/65427879b2654199.svg\")
Final answer: C = ![\"\\begin{bmatrix}4](\"assets/equations/6f769fba9b2febe4.svg\")
, with size 2 × 2.
Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.
","a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5.
","The 2 × 1 entry-by-entry product and the 2 × 2 outer product.
","⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons.
","⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row.
","![\"(a\\odot](\"assets/equations/696e2da1033ab6fa.svg\")
![\"(ab^T)_{ij}=a_i](\"assets/equations/11b839db8ece52c1.svg\")
Yaad rakhne ka cue: ⊙ mein same jagah; outer product mein har pair.
","Step 1: multiply matching entries.
","![\"a\\odot](\"assets/equations/c69bdb17e45bd1d3.svg\")
Step 2: transpose b.
","![\"b^T=\\begin{bmatrix}4&5\\end{bmatrix}\"](\"assets/equations/a491055be3cc3700.svg\")
Step 3: form row 1 of the outer product.
","![\"2(4)=8,\\qquad2(5)=10\"](\"assets/equations/3e7e9d7e4e1bc080.svg\")
Step 4: form row 2 of the outer product.
","![\"3(4)=12,\\qquad3(5)=15\"](\"assets/equations/399713a01b02cfa8.svg\")
Step 5: write both rows.
","![\"ab^T=\\begin{bmatrix}8&10\\\\12&15\\end{bmatrix}\"](\"assets/equations/89b24f1f7b777962.svg\")
Final answer: Entry-by-entry result = [8,15]ᵀ. Outer product = ![\"\\begin{bmatrix}8](\"assets/equations/db73fdbe57c2c901.svg\")
.
A graph shows how one value changes with another. Put the weight w on the horizontal axis and the loss L on the vertical axis. A slope is change in height divided by change in horizontal position. If L rises by 6 when w rises by 2, the average slope is 3.
","A derivative is the slope for an extremely small change at one location. A positive derivative says increasing the weight initially increases loss. A negative derivative says increasing the weight initially decreases loss. Gradient descent moves against that local slope. Gradient batata hai loss kis taraf badhega; hum ulte taraf chhota step lete hain.
","![\"L(w)=w^2,\\qquad](\"assets/equations/aab6f621e9833b07.svg\")
At w = 3, slope = 6. With learning rate 0.1, the update is 3 − 0.1 × 6 = 2.4. Loss falls from 9 to 5.76. At w = −3, slope = −6, so the update is −3 − 0.1 × (−6) = −2.4. Both moves approach zero.
","Definition. A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector.
","![\"\\nabla_\\theta](\"assets/equations/ef6399f58ca20acf.svg\")
The rules needed for the supplied neural-network numericals are:
","| Function or operation | Derivative |
|---|---|
| Constant c | 0 |
| xⁿ | nxⁿ⁻¹ |
| c f(x) | c f′(x) |
| f(x) + g(x) | f′(x) + g′(x) |
| f(x)g(x) | f′(x)g(x) + f(x)g′(x) |
| eˣ | eˣ |
| ln x, x > 0 | 1/x |
| f(g(x)) | f′(g(x))g′(x) |
For z = wx + b, the variable named in the denominator determines which derivative is required.
","![\"\\frac{\\partial](\"assets/equations/8e90f73571d42f2e.svg\")
At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities.
","For a small parameter displacement Δθ, first-order Taylor expansion gives:
","![\"J(\\theta+\\Delta\\theta)\\approx](\"assets/equations/9060e5490f15d100.svg\")
The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large.
","Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.
","J(w) = 3w^2+5; w = 2.
","The function value and its derivative value.
","The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.
","c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.
","![\"\\frac{d(cw^n+k)}{dw}=cnw^{n-1}\"](\"assets/equations/706ea9b2290fe843.svg\")
Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.
","Step 1: evaluate the function itself.
","![\"J(2)=3(2)^2+5=17\"](\"assets/equations/b64892fad56f04e4.svg\")
Step 2: differentiate symbolically.
","![\"J\\prime(w)=3(2)w^{2-1}+0\"](\"assets/equations/f98b439990365e5d.svg\")
Step 3: put the given w into the derivative.
","![\"J\\prime(2)=3(2)(2)^{1}=12\"](\"assets/equations/d4fbd4bd0e97e8fb.svg\")
Final answer: J = 17; derivative = 12.
","Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.
","J(w) = 0.7w^3+0; w = 1.
","The function value and its derivative value.
","The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.
","c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.
","Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.
","Step 1: evaluate the function itself.
","![\"J(1)=0.7(1)^3+0=0.7\"](\"assets/equations/5c598d8efe6ecd14.svg\")
Step 2: differentiate symbolically.
","![\"J\\prime(w)=0.7(3)w^{3-1}+0\"](\"assets/equations/6f7d282bf1129194.svg\")
Step 3: put the given w into the derivative.
","![\"J\\prime(1)=0.7(3)(1)^{2}=2.1\"](\"assets/equations/38040daaa10cb32e.svg\")
Final answer: J = 0.7; derivative = 2.1.
","Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.
","J(w) = 2w^1+4; w = -1.
","The function value and its derivative value.
","The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.
","c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.
","Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.
","Step 1: evaluate the function itself.
","![\"J(-1)=2(-1)^1+4=2\"](\"assets/equations/625a082643eb2a43.svg\")
Step 2: differentiate symbolically.
","![\"J\\prime(w)=2(1)w^{1-1}+0\"](\"assets/equations/091a2e97f7d74170.svg\")
Step 3: put the given w into the derivative.
","![\"J\\prime(-1)=2(1)(-1)^{0}=2\"](\"assets/equations/d63d0415964167b7.svg\")
Final answer: J = 2; derivative = 2.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s3","search":"Derivatives without a calculus course **Definition.** A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector.\n\nThe rules needed for the supplied neural-network numericals are:\n\n| Function or operation | Derivative |\n| Constant c | 0 |\n| xⁿ | nxⁿ⁻¹ |\n| c f(x) | c f′(x) |\n| f(x) + g(x) | f′(x) + g′(x) |\n| f(x)g(x) | f′(x)g(x) + f(x)g′(x) |\n| eˣ | eˣ |\n| ln x, x > 0 | 1/x |\n| f(g(x)) | f′(g(x))g′(x) |\n\n### A parameter derivative is not an input derivative\nFor z = wx + b, the variable named in the denominator determines which derivative is required.\n\nAt x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities.\n\n### Gradient direction\nFor a small parameter displacement Δθ, first-order Taylor expansion gives:\n\nThe gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large.\n\n\n\n### Differentiate before substituting: 3w^2+5\n\n**Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.**\n\n\n\n### Given\n\nJ(w) = 3w^2+5; w = 2.\n\n### To find\n\nThe function value and its derivative value.\n\n### Formula — which one and why\n\nThe power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.\n\nc is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.\n\n\n\n**Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero.\n\n### Solution — step by step\n\n\n\n**Step 1: evaluate the function itself.**\n\n\n\n\n\n\n\n**Step 2: differentiate symbolically.**\n\n\n\n\n\n\n\n**Step 3: put the given w into the derivative.**\n\n\n\n\n\n\n\n**Final answer:** J = 17; derivative = 12.\n\n\n\n### Differentiate before substituting: 0.7w^3+0\n\n**Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.**\n\n\n\n### Given\n\nJ(w) = 0.7w^3+0; w = 1.\n\n### To find\n\nThe function value and its derivative value.\n\n### Formula — which one and why\n\nThe power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.\n\nc is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.\n\n\n\n**Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero.\n\n### Solution — step by step\n\n\n\n**Step 1: evaluate the function itself.**\n\n\n\n\n\n\n\n**Step 2: differentiate symbolically.**\n\n\n\n\n\n\n\n**Step 3: put the given w into the derivative.**\n\n\n\n\n\n\n\n**Final answer:** J = 0.7; derivative = 2.1.\n\n\n\n### Differentiate before substituting: 2w^1+4\n\n**Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.**\n\n\n\n### Given\n\nJ(w) = 2w^1+4; w = -1.\n\n### To find\n\nThe function value and its derivative value.\n\n### Formula — which one and why\n\nThe power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.\n\nc is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.\n\n\n\n**Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero.\n\n### Solution — step by step\n\n\n\n**Step 1: evaluate the function itself.**\n\n\n\n\n\n\n\n**Step 2: differentiate symbolically.**\n\n\n\n\n\n\n\n**S"},{"title":"Derivatives: from an ordinary change to a local slope","blocks":["Let L(w) = w². At w = 3, loss is 9. If we increase w to 4, loss becomes 16. The loss change is 16 − 9 = 7 and the weight change is 4 − 3 = 1. Their ratio is 7. This is an average slope over a whole interval.
","![\"\\text{average](\"assets/equations/ddbf51217582c207.svg\")
For the same starting weight 3, use an increase of 0.1. The new weight is 3.1 and its square is 9.61. The slope estimate is (9.61 − 9)/0.1 = 6.1. With an increase of 0.01, it is (9.0601 − 9)/0.01 = 6.01. Smaller intervals approach 6.
","| Increase in w | New w | New loss w² | Change in loss | Change ratio |
|---|---|---|---|---|
| 1 | 4 | 16 | 7 | 7 |
| 0.1 | 3.1 | 9.61 | 0.61 | 6.1 |
| 0.01 | 3.01 | 9.0601 | 0.0601 | 6.01 |
The derivative is the limiting local slope as that interval approaches zero. We do not substitute a literal zero into a division by the interval.
","Call a small nonzero change Δw, read “delta w”. Expanding the new square gives:
","![\"(w+\\Delta](\"assets/equations/4e8dd2d8556610b9.svg\")
Subtract the old square w² and divide by the change Δw:
","![\"\\frac{(w+\\Delta](\"assets/equations/1e547bc2dd023797.svg\")
As Δw approaches zero, this approaches 2w. That is the origin of the rule d(w²)/dw = 2w. At w = 3 it gives 6, matching the table's trend.
","In dL/dw, the top names the quantity whose response we study, and the bottom names what we vary. Read it as “the derivative of L with respect to w”. The notation is not asking you to divide the current loss 9 by the current weight 3; that would give 3 and answer a different question.
","For a function of several variables, ∂L/∂w is a partial derivative: change w while holding the other independent variables fixed. The gradient groups those partial derivatives for all parameters.
","If the derivative is positive, increasing w locally increases loss, so we try a small decrease. If it is negative, increasing w locally decreases loss, so we try a small increase. The subtraction in gradient descent automatically handles both signs.
","![\"w_{\\mathrm{new}}=w-\\eta](\"assets/equations/06b81f0d4da782b2.svg\")
This is local information. A very large step can cross a minimum and make loss worse. The learning rate η keeps control of the step; the derivative does not promise that any step size is safe.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s4","search":"Derivatives: from an ordinary change to a local slope ### 1. Function value and change are different numbers\nLet L(w) = w². At w = 3, loss is 9. If we increase w to 4, loss becomes 16. The loss change is 16 − 9 = 7 and the weight change is 4 − 3 = 1. Their ratio is 7. This is an average slope over a whole interval.\n\n\n\n\n### 2. Shrink the interval\nFor the same starting weight 3, use an increase of 0.1. The new weight is 3.1 and its square is 9.61. The slope estimate is (9.61 − 9)/0.1 = 6.1. With an increase of 0.01, it is (9.0601 − 9)/0.01 = 6.01. Smaller intervals approach 6.\n| Increase in w | New w | New loss w² | Change in loss | Change ratio |\n| 1 | 4 | 16 | 7 | 7 |\n| 0.1 | 3.1 | 9.61 | 0.61 | 6.1 |\n| 0.01 | 3.01 | 9.0601 | 0.0601 | 6.01 |\nThe derivative is the limiting local slope as that interval approaches zero. We do not substitute a literal zero into a division by the interval.\n\n### 3. Why the square has derivative 2w\nCall a small nonzero change Δw, read “delta w”. Expanding the new square gives:\n\nSubtract the old square w² and divide by the change Δw:\n\nAs Δw approaches zero, this approaches 2w. That is the origin of the rule d(w²)/dw = 2w. At w = 3 it gives 6, matching the table's trend.\n\n### 4. Reading derivative notation\nIn dL/dw, the top names the quantity whose response we study, and the bottom names what we vary. Read it as “the derivative of L with respect to w”. The notation is not asking you to divide the current loss 9 by the current weight 3; that would give 3 and answer a different question.\nFor a function of several variables, ∂L/∂w is a partial derivative: change w while holding the other independent variables fixed. The gradient groups those partial derivatives for all parameters.\n\n### 5. Why subtract the gradient?\nIf the derivative is positive, increasing w locally increases loss, so we try a small decrease. If it is negative, increasing w locally decreases loss, so we try a small increase. The subtraction in gradient descent automatically handles both signs.\n\nThis is local information. A very large step can cross a minimum and make loss worse. The learning rate η keeps control of the step; the derivative does not promise that any step size is safe.\n"},{"title":"Partial derivatives hold other things still","blocks":["For z = wx + b, the derivative with respect to w is x, because x is held fixed. The derivative with respect to b is 1. The derivative with respect to x is w. This is the entire local maths of a neuron’s weighted sum.
","![\"\\frac{\\partial](\"assets/equations/6437deea893fce68.svg\")
Question: z = 3w + b, w = 2, b = 1. Find z, ∂z/∂w and ∂z/∂b.
","z = 3w+b; w = 2; b = 1.
","Find z, ∂z/∂w and ∂z/∂b.
","Partial derivatives. Differentiate only with respect to the named variable; all other independent inputs and parameters remain fixed.
","∂z/∂w reads “rate of change of z when w changes”. For z = wx+b, x is the coefficient of w, while b has coefficient 1.
","![\"\\frac{\\partial](\"assets/equations/c448c09e6087f1d5.svg\")
![\"\\frac{\\partial](\"assets/equations/d2edf1641e728713.svg\")
Yaad rakhne ka cue: Neeche jis letter ka naam hai, sirf usko change samjho.
","Step 1: calculate the function value.
","![\"z=3(2)+1=6+1=7\"](\"assets/equations/c86ffb173c8d136d.svg\")
Step 2: differentiate with respect to w while b is fixed.
","![\"\\frac{\\partial](\"assets/equations/3ab15e4d1d60b4eb.svg\")
Step 3: differentiate with respect to b while w is fixed.
","![\"\\frac{\\partial](\"assets/equations/edaff0abc794c10d.svg\")
Final answer: z = 7; weight derivative = 3; bias derivative = 1. A function value and its rate of change are different quantities.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s5","search":"Partial derivatives hold other things still For z = wx + b, the derivative with respect to w is x, because x is held fixed. The derivative with respect to b is 1. The derivative with respect to x is w. This is the entire local maths of a neuron’s weighted sum.\n\n\n\n\n**Question: ** z = 3w + b, w = 2, b = 1. Find z, ∂z/∂w and ∂z/∂b.\n\n\n\n### Given\n\nz = 3w+b; w = 2; b = 1.\n\n### To find\n\nFind z, ∂z/∂w and ∂z/∂b.\n\n### Formula — which one and why\n\n**Partial derivatives.** Differentiate only with respect to the named variable; all other independent inputs and parameters remain fixed.\n\n∂z/∂w reads “rate of change of z when w changes”. For z = wx+b, x is the coefficient of w, while b has coefficient 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Neeche jis letter ka naam hai, sirf usko change samjho.\n\n### Solution — step by step\n\n\n\n\n**Step 1: calculate the function value.**\n\n\n\n\n\n\n\n**Step 2: differentiate with respect to w while b is fixed.**\n\n\n\n\n\n\n\n**Step 3: differentiate with respect to b while w is fixed.**\n\n\n\n\n\n\n\n**Final answer:** z = 7; weight derivative = 3; bias derivative = 1. A function value and its rate of change are different quantities."},{"title":"The chain rule is multiplying linked effects","blocks":["Definition. The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions.
","Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency.
","The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w.
","Yaad rakhne ka cue: ek raaste par slopes multiply; alag raaston ke full answers add.
","Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.
","w = 1; u = 3w+1; v = 2u.
","The two function values and the rate dv/dw.
","v depends on w through u. Both links must be included, so multiply the two local derivatives.
","du/dw is the slope of 3w+1. dv/du is the slope of 2u.
","![\"\\frac{dv}{dw}=\\frac{dv}{du}\\frac{du}{dw}\"](\"assets/equations/17086b5a41fb1edf.svg\")
Yaad rakhne ka cue: Andar ka slope × bahar ka slope.
","Step 1: calculate the intermediate value.
","![\"u=3(1)+1=4\"](\"assets/equations/b14c00d5dab41947.svg\")
Step 2: calculate the final value.
","![\"v=2(4)=8\"](\"assets/equations/396fa5f1dfbfd157.svg\")
Step 3: differentiate the first link.
","![\"\\frac{du}{dw}=3\"](\"assets/equations/1379a65c2b33abc7.svg\")
Step 4: differentiate the second link.
","![\"\\frac{dv}{du}=2\"](\"assets/equations/0af363fd8a85c3ab.svg\")
Step 5: multiply along the path.
","![\"\\frac{dv}{dw}=2(3)=6\"](\"assets/equations/746afadc90e4f6c9.svg\")
Final answer: u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different.
","Question. A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1.
","Input x = 2; target y = 4.
","Old weight w = 1; old bias b = 0.
","Linear prediction z = wx+b; loss L = ½(z−y)².
","Learning rate η = 0.1; update both parameters once.
","Find the loss gradient for w and for b, then w_new and b_new. Finally evaluate the changed prediction and loss.
","Chain rule for a linear neuron. The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.
","z = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.
","![\"z=wx+b\"](\"assets/equations/00f49c1127dda70e.svg\")
![\"L=\\frac12(z-y)^2\"](\"assets/equations/f50d7e1cde6462d5.svg\")
![\"g_w=\\frac{\\partial](\"assets/equations/955c63d58ce8cb9b.svg\")
![\"g_b=\\frac{\\partial](\"assets/equations/1a5a6192e644e19b.svg\")
![\"w_{\\mathrm{new}}=w-\\eta](\"assets/equations/33df8d5a70c57c14.svg\")
![\"b_{\\mathrm{new}}=b-\\eta](\"assets/equations/c2843cb0410cd77b.svg\")
Yaad rakhne ka cue: Ek path mein multiply; alag paths mil rahe hon toh add.
","Step 1 — evaluate the original model. Use the given parameters before applying any update.
","![\"z=wx+b=1(2)+0=2\"](\"assets/equations/73aa46b2ed8f2b55.svg\")
![\"e=z-y=2-4=-2\"](\"assets/equations/ef2733b3de84bfed.svg\")
![\"L=\\frac12e^2=\\frac12(-2)^2=2\"](\"assets/equations/7adb597b6b18fec3.svg\")
The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide.
","Step 2 — differentiate the loss, keeping the target constant. First differentiate the square, then the error inside it.
","![\"\\frac{\\partial](\"assets/equations/ba371707e98b2bc9.svg\")
![\"\\frac{\\partial](\"assets/equations/66db443def7ab596.svg\")
![\"\\delta_z=\\frac{\\partial](\"assets/equations/e9802e836c671214.svg\")
The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection.
","Step 3 — differentiate the neuron's affine expression. When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1.
","![\"\\frac{\\partial](\"assets/equations/6badd88b171bea97.svg\")
Step 4 — multiply along each parameter's path.
","![\"g_w=\\frac{\\partial](\"assets/equations/b51cdfbc8d705d9a.svg\")
![\"g_b=\\frac{\\partial](\"assets/equations/6a2c0a4ed56c043f.svg\")
Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate.
","Step 5 — update both parameters simultaneously. Gradient descent subtracts the learning rate times the corresponding gradient.
","![\"w_{\\mathrm{new}}=w-\\eta](\"assets/equations/8330219a13690888.svg\")
![\"b_{\\mathrm{new}}=b-\\eta](\"assets/equations/d5507ad8ee01624d.svg\")
Both updates use gradients from the same original forward pass. Re-evaluate the model afterward.
Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure.
","Step 6 — perform a new forward pass to evaluate the update.
","![\"z_{\\mathrm{new}}=1.4(2)+0.2=3\"](\"assets/equations/f27e43a3756292cb.svg\")
![\"L_{\\mathrm{new}}=\\frac12(3-4)^2=0.5\"](\"assets/equations/1762c7935645936f.svg\")
Final answer: weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3.
","A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward.
","Question. A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed.
","h = wx+b; outputs o₁ = 2h and o₂ = −h.
","x = 2; w = 0.5; b = 0.1; targets = [1,0].
","Loss = half the sum of the two squared output errors.
","Output multipliers 2 and −1 stay fixed.
","Find total loss and its derivatives with respect to h, w and b.
","Chain rule with more than one path. The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b.
","oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path.
","![\"L=\\frac12\\sum_j(o_j-y_j)^2\"](\"assets/equations/737ddfb3a05bc433.svg\")
![\"\\frac{\\partial](\"assets/equations/2e472e733a927b24.svg\")
![\"\\frac{\\partial](\"assets/equations/19ebe8d0b832533d.svg\")
![\"\\frac{\\partial](\"assets/equations/44f3dd28940be744.svg\")
Yaad rakhne ka cue: Har raaste ka effect pehle multiply, junction par sab add.
","Step 1 — forward values.
","![\"h=0.5(2)+0.1=1.1\"](\"assets/equations/c54d1626a0bf10fc.svg\")
![\"o_1=2(1.1)=2.2,\\qquad](\"assets/equations/efc23fa7e963abec.svg\")
![\"L_1=\\frac12(2.2-1)^2=0.72\"](\"assets/equations/9cca9af1e961da75.svg\")
![\"L_2=\\frac12(-1.1-0)^2=0.605\"](\"assets/equations/31043c1e17e75c9f.svg\")
![\"L=L_1+L_2=0.72+0.605=1.325\"](\"assets/equations/115053bc42452eb3.svg\")
Step 2 — output derivatives. Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output.
","![\"\\frac{\\partial](\"assets/equations/0cee4ce79f255618.svg\")
![\"\\frac{\\partial](\"assets/equations/1551245d19941245.svg\")
Step 3 — calculate the contribution of each complete path. The derivative of 2h with respect to h is 2; the derivative of −h is −1.
","![\"\\frac{\\partial](\"assets/equations/b03138eedef68e56.svg\")
![\"\\frac{\\partial](\"assets/equations/74549258b69e068b.svg\")
Step 4 — add the two contributions at the shared hidden value.
","![\"\\frac{\\partial](\"assets/equations/a4fb627a77ae8284.svg\")
![\"\\frac{\\partial](\"assets/equations/61ff2149ac92a571.svg\")
Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path.
","Step 5 — continue to the hidden weight and bias. Since h = wx + b, its derivatives with respect to w and b are x and 1.
","![\"\\frac{\\partial](\"assets/equations/54071389e64f7570.svg\")
![\"\\frac{\\partial](\"assets/equations/327ca2cf6e5be3c3.svg\")
Final answer: L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation.
","Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.
","w = 1; z = 3w+1; L = z².
","z, L and the derivative of L with respect to w.
","The square acts on z, not directly on w. Differentiate the square first and include the change of z with w.
","dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression.
","![\"\\frac{dL}{dw}=\\frac{dL}{dz}\\frac{dz}{dw}\"](\"assets/equations/434e48fc84f871b4.svg\")
![\"\\frac{d(z^2)}{dz}=2z\"](\"assets/equations/33825b3d129b23da.svg\")
Yaad rakhne ka cue: Square ka 2z; andar ka 3; dono multiply.
","Step 1: evaluate the inner expression.
","![\"z=3(1)+1=4\"](\"assets/equations/8a300d26d67e84f4.svg\")
Step 2: evaluate the loss expression.
","![\"L=4^2=16\"](\"assets/equations/e7a8c4341fdaf1f8.svg\")
Step 3: differentiate the square and use z = 4.
","![\"\\frac{dL}{dz}=2z=2(4)=8\"](\"assets/equations/3a7684d15448f2e4.svg\")
Step 4: differentiate the inner expression.
","![\"\\frac{dz}{dw}=3\"](\"assets/equations/dbe8fc0fd6788a9a.svg\")
Step 5: multiply, because the changes occur in sequence.
","![\"\\frac{dL}{dw}=8(3)=24\"](\"assets/equations/90d0d059a0fe1d53.svg\")
Final answer: z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s6","search":"The chain rule is multiplying linked effects **Definition.** The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions.\n\n### First understand why the rates multiply\nSuppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency.\n\nThe notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w.\n\n**Yaad rakhne ka cue:** ek raaste par slopes multiply; alag raaston ke full answers add.\n\n\n### Chain-rule example 1: two linear links\n\n**Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.**\n\n\n\n### Given\n\nw = 1; u = 3w+1; v = 2u.\n\n### To find\n\nThe two function values and the rate dv/dw.\n\n### Formula — which one and why\n\nv depends on w through u. Both links must be included, so multiply the two local derivatives.\n\ndu/dw is the slope of 3w+1. dv/du is the slope of 2u.\n\n\n\n**Yaad rakhne ka cue:** Andar ka slope × bahar ka slope.\n\n### Solution — step by step\n\n\n\n**Step 1: calculate the intermediate value.**\n\n\n\n\n\n\n\n**Step 2: calculate the final value.**\n\n\n\n\n\n\n\n**Step 3: differentiate the first link.**\n\n\n\n\n\n\n\n**Step 4: differentiate the second link.**\n\n\n\n\n\n\n\n**Step 5: multiply along the path.**\n\n\n\n\n\n\n\n**Final answer:** u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different.\n\n### 2. One output: calculate both parameter gradients\n**Question.** A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1.\n\n\n\n### Given\n\nInput x = 2; target y = 4.\n\nOld weight w = 1; old bias b = 0.\n\nLinear prediction z = wx+b; loss L = ½(z−y)².\n\nLearning rate η = 0.1; update both parameters once.\n\n### To find\n\nFind the loss gradient for w and for b, then w_new and b_new. Finally evaluate the changed prediction and loss.\n\n### Formula — which one and why\n\n**Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.\n\nz = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.\n\n\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add.\n\n### Solution — step by step\n\n\n\n\n\n\n\n**Step 1 — evaluate the original model.** Use the given parameters before applying any update.\n\n\n\nThe prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide.\n\n**Step 2 — differentiate the loss, keeping the target constant.** First differentiate the square, then the error inside it.\n\n\n\nThe symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection.\n\n**Step 3 — differentiate the neuron's "},{"title":"The derivative of prediction error","blocks":["For the half squared loss, use prediction minus target. Let target be y and prediction be ŷ.
","![\"L=\\frac12(\\hat](\"assets/equations/3bd0a5a4ecdf16a4.svg\")
The 2 from differentiating a square cancels the half. Writing the loss as ½(y − ŷ)² gives the same derivative: the derivative of the inside y − ŷ adds a minus sign. This is why “target minus prediction” without the extra minus gives the wrong gradient.
","If the question uses plain squared error, its derivative is 2(ŷ − y). If it averages errors over n values, also divide by n. Read the stated loss before using a memorized gradient.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s7","search":"The derivative of prediction error For the half squared loss, use prediction minus target. Let target be y and prediction be ŷ.\n\n\n\n\nThe 2 from differentiating a square cancels the half. Writing the loss as ½(y − ŷ)² gives the same derivative: the derivative of the inside y − ŷ adds a minus sign. This is why “target minus prediction” without the extra minus gives the wrong gradient.\nIf the question uses plain squared error, its derivative is 2(ŷ − y). If it averages errors over n values, also divide by n. Read the stated loss before using a memorized gradient."},{"title":"Loss scaling: derive the gradient instead of guessing a factor","blocks":["Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square:
","![\"\\frac{\\partial](\"assets/equations/5f66834fe766593d.svg\")
Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below.
","| Stated loss over n outputs | Derivative with respect to one prediction aⱼ |
|---|---|
| SSE: sum of all squared errors | 2(aⱼ − yⱼ) |
| Half SSE: half of that sum | aⱼ − yⱼ |
| MSE: that sum divided by n | 2(aⱼ − yⱼ)/n |
| Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n |
Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor.
","Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages.
","If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore:
","![\"\\delta=\\frac{\\partial](\"assets/equations/d269ffed64d3f80c.svg\")
For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question.
","For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042.
","The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s8","search":"Loss scaling: derive the gradient instead of guessing a factor ### 1. One error, three possible objectives\nLet a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square:\n\n\n\n\nMultiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below.\n\n| Stated loss over n outputs | Derivative with respect to one prediction aⱼ |\n| SSE: sum of all squared errors | 2(aⱼ − yⱼ) |\n| Half SSE: half of that sum | aⱼ − yⱼ |\n| MSE: that sum divided by n | 2(aⱼ − yⱼ)/n |\n| Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n |\n\n### 2. Why only the matching output appears\nSuppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor.\nLater, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages.\n\n### 3. The activation contributes another factor\nIf a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore:\n\nFor a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question.\n\n### 4. A numerical comparison\nFor a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042.\nThe forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used.\n"},{"title":"Probability mean and variance in plain language","blocks":["Definitions. A probability assigns a value between 0 and 1 to an event. A discrete probability distribution assigns probabilities to mutually exclusive outcomes whose probabilities sum to 1. Expectation is a probability-weighted average. Variance measures expected squared deviation from the mean.
","![\"\\sum_i](\"assets/equations/a6a90dd79f192150.svg\")
![\"\\operatorname{Var}(X)=\\mathbb](\"assets/equations/2228ef692c83a513.svg\")
P(A|B) is the probability of A conditional on B, defined when P(B) > 0. Bayes' rule relates it to the reverse conditional probability.
","![\"P(A\\mid](\"assets/equations/f464138921aade18.svg\")
For classification, the network estimates P(class | input). A softmax output is normalized across mutually exclusive classes. Independent sigmoid outputs are used for multiple labels that can be present together.
","For N observed values, use the stated variance convention. These notes use division by N for the population-style variance in the numerical; an unbiased sample-variance estimate uses N − 1.
","![\"\\mu=\\frac1N\\sum_i](\"assets/equations/554920a13a606ed9.svg\")
![\"\\sigma=\\sqrt{\\sigma^2},\\qquad](\"assets/equations/2f7fcc7ba3415fbf.svg\")
For values [3, 3, 7, 7], μ = 5, squared deviations are [4, 4, 4, 4], σ² = 16/4 = 4 and σ = 2. Thus a value 7 standardizes to (7 − 5)/2 = 1: one standard deviation above the mean. For preprocessing, fit μ and σ on training data and reuse them for validation/test data; σ must be nonzero.
","If the correct class has predicted probability p, its negative-log loss is −ln p. For p = 0.8, this is approximately 0.223144. For p = 0.1, it is approximately 2.302585. Increasing correct-class probability lowers this loss. Over independent examples, maximizing the product of correct-class probabilities is equivalent to minimizing the sum of negative log probabilities.
","Question. For the population [1, 2, 3], find mean, population variance and standard deviation.
","Values = [1, 2, 3]; N = 3. Use population division by N.
","Mean μ, variance σ² and standard deviation σ.
","The question specifies a population, so use N rather than the N−1 sample-variance divisor.
","μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.
","![\"\\mu=\\frac1N\\sum_i](\"assets/equations/d96a5eb55c670639.svg\")
![\"\\sigma^2=\\frac1N\\sum_i(x_i-\\mu)^2\"](\"assets/equations/d7a9d7e061fca1d7.svg\")
![\"\\sigma=\\sqrt{\\sigma^2}\"](\"assets/equations/7b5afbd2f5b393f2.svg\")
Yaad rakhne ka cue: Mean, distance, square, mean again, then root.
","Step 1: add all values.
","![\"1+2+3=6\"](\"assets/equations/23a1d761557fba99.svg\")
Step 2: divide by the number of values.
","![\"\\mu=\\frac{6}{3}=2\"](\"assets/equations/6f48ae6fcd2c250e.svg\")
Step 3: find and square deviation 1.
","![\"(1-2)=-1\"](\"assets/equations/cde1ff63a64aa7ce.svg\")
![\"(-1)^2=1\"](\"assets/equations/46782fbf983b8482.svg\")
Step 4: find and square deviation 2.
","![\"(2-2)=0\"](\"assets/equations/21c8488d16b7e891.svg\")
![\"(0)^2=0\"](\"assets/equations/a8b9442558db09d6.svg\")
Step 5: find and square deviation 3.
","![\"(3-2)=1\"](\"assets/equations/ebe985168f035d36.svg\")
![\"(1)^2=1\"](\"assets/equations/a4b8daac35aec1bc.svg\")
Step 6: average the squared deviations.
","![\"\\sigma^2=\\frac{1+0+1}{3}\\approx0.666666667\"](\"assets/equations/e913fcfda4146fb6.svg\")
Step 7: take the nonnegative square root.
","![\"\\sigma=\\sqrt{0.666666667}\\approx0.816496581\"](\"assets/equations/0c7d94a64e886a59.svg\")
Final answer: μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581.
","Question. For the population [2, 2, 2], find mean, population variance and standard deviation.
","Values = [2, 2, 2]; N = 3. Use population division by N.
","Mean μ, variance σ² and standard deviation σ.
","The question specifies a population, so use N rather than the N−1 sample-variance divisor.
","μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.
","Yaad rakhne ka cue: Mean, distance, square, mean again, then root.
","Step 1: add all values.
","![\"2+2+2=6\"](\"assets/equations/3fc38a48a0a6427f.svg\")
Step 2: divide by the number of values.
","Step 3: find and square deviation 1.
","Step 4: find and square deviation 2.
","Step 5: find and square deviation 3.
","Step 6: average the squared deviations.
","![\"\\sigma^2=\\frac{0+0+0}{3}\\approx0.000000000\"](\"assets/equations/b0fc9edc8362e2a5.svg\")
Step 7: take the nonnegative square root.
","![\"\\sigma=\\sqrt{0.000000000}\\approx0.000000000\"](\"assets/equations/9ee45ab162936835.svg\")
Final answer: μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s9","search":"Probability mean and variance in plain language **Definitions.** A probability assigns a value between 0 and 1 to an event. A discrete probability distribution assigns probabilities to mutually exclusive outcomes whose probabilities sum to 1. Expectation is a probability-weighted average. Variance measures expected squared deviation from the mean.\n\n\n\n### Conditional probability and classification\nP(A|B) is the probability of A conditional on B, defined when P(B) > 0. Bayes' rule relates it to the reverse conditional probability.\n\nFor classification, the network estimates P(class | input). A softmax output is normalized across mutually exclusive classes. Independent sigmoid outputs are used for multiple labels that can be present together.\n\n### Empirical mean, variance and standardization\nFor N observed values, use the stated variance convention. These notes use division by N for the population-style variance in the numerical; an unbiased sample-variance estimate uses N − 1.\n\n\nFor values [3, 3, 7, 7], μ = 5, squared deviations are [4, 4, 4, 4], σ² = 16/4 = 4 and σ = 2. Thus a value 7 standardizes to (7 − 5)/2 = 1: one standard deviation above the mean. For preprocessing, fit μ and σ on training data and reuse them for validation/test data; σ must be nonzero.\n\n### Likelihood and cross-entropy\nIf the correct class has predicted probability p, its negative-log loss is −ln p. For p = 0.8, this is approximately 0.223144. For p = 0.1, it is approximately 2.302585. Increasing correct-class probability lowers this loss. Over independent examples, maximizing the product of correct-class probabilities is equivalent to minimizing the sum of negative log probabilities.\n\n### Population mean and spread: [1, 2, 3]\n\n**Question. For the population [1, 2, 3], find mean, population variance and standard deviation.**\n\n\n\n### Given\n\nValues = [1, 2, 3]; N = 3. Use population division by N.\n\n### To find\n\nMean μ, variance σ² and standard deviation σ.\n\n### Formula — which one and why\n\nThe question specifies a population, so use N rather than the N−1 sample-variance divisor.\n\nμ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Mean, distance, square, mean again, then root.\n\n### Solution — step by step\n\n\n\n**Step 1: add all values.**\n\n\n\n\n\n\n\n**Step 2: divide by the number of values.**\n\n\n\n\n\n\n\n**Step 3: find and square deviation 1.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 4: find and square deviation 2.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 5: find and square deviation 3.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 6: average the squared deviations.**\n\n\n\n\n\n\n\n**Step 7: take the nonnegative square root.**\n\n\n\n\n\n\n\n**Final answer:** μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581.\n\n\n\n\n\n### Population mean and spread: [2, 2, 2]\n\n**Question. For the population [2, 2, 2], find mean, population variance and standard deviation.**\n\n\n\n### Given\n\nValues = [2, 2, 2]; N = 3. Use population division by N.\n\n### To find\n\nMean μ, variance σ² and standard deviation σ.\n\n### Formula — which one and why\n\nThe question specifies a population, so use N rather than the N−1 sample-variance divisor.\n\nμ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Mean, distance, square, mean again, then root.\n\n### Solution — step by step\n\n\n\n**Step 1: add all values.**\n\n\n\n\n\n\n\n**Step 2: divide by the number of values.**\n\n\n\n\n\n\n\n**Step 3: find and square deviation 1.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 4: find and square deviation 2.**\n\n\n"},{"title":"Mean, variance and standard deviation from ordinary arithmetic","blocks":["Definition. The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units.
","![\"\\mu=\\frac{1}{N}\\sum_{i=1}^{N}x_i,\\qquad](\"assets/equations/8b9cec002314f2de.svg\")
These are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable.
","Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].
","Observations = [3,3,7,7]; N = 4.
","Use population variance (divide by N).
","Find the population mean μ, variance σ² and standard deviation σ.
","Mean and spread. The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.
","N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.
","![\"\\mu=\\frac{1}{N}\\sum_{i=1}^{N}x_i\"](\"assets/equations/aa827d67a05a9234.svg\")
![\"\\sigma^2=\\frac{1}{N}\\sum_{i=1}^{N}(x_i-\\mu)^2\"](\"assets/equations/a1a03dea0d7cf33e.svg\")
Yaad rakhne ka cue: Mean = sabka total ÷ count; variance = squared distances ka mean.
","Step 1: add all four observations.
","![\"3+3+7+7=20\"](\"assets/equations/b29c1c44847515f7.svg\")
Step 2: divide by the count to obtain the mean.
","![\"\\mu=\\frac{20}{4}=5\"](\"assets/equations/6eafa89b652a5378.svg\")
Step 3: subtract the mean from each observation.
","![\"[3-5,3-5,7-5,7-5]=[-2,-2,2,2]\"](\"assets/equations/f5609e02cf210acd.svg\")
Step 4: square each deviation.
","![\"[(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4]\"](\"assets/equations/8423c755b10f2fe0.svg\")
Step 5: add the squared deviations.
","![\"4+4+4+4=16\"](\"assets/equations/8b51d1e9b84e8106.svg\")
Step 6: divide by four for population variance.
","![\"\\sigma^2=\\frac{16}{4}=4\"](\"assets/equations/47e0cbf4557b9859.svg\")
Step 7: take the nonnegative square root.
","![\"\\sigma=\\sqrt4=2\"](\"assets/equations/5a8d993f2bb3889d.svg\")
Answer: mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s10","search":"Mean, variance and standard deviation from ordinary arithmetic \n**Definition.** The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units.\n\nThese are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable.\n\n**Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].**\n\n\n\n### Given\n\nObservations = [3,3,7,7]; N = 4.\n\nUse population variance (divide by N).\n\n### To find\n\nFind the population mean μ, variance σ² and standard deviation σ.\n\n### Formula — which one and why\n\n**Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.\n\nN = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: add all four observations.**\n\n\n\n\n\n\n\n**Step 2: divide by the count to obtain the mean.**\n\n\n\n\n\n\n\n**Step 3: subtract the mean from each observation.**\n\n\n\n\n\n\n\n**Step 4: square each deviation.**\n\n\n\n\n\n\n\n**Step 5: add the squared deviations.**\n\n\n\n\n\n\n\n**Step 6: divide by four for population variance.**\n\n\n\n\n\n\n\n**Step 7: take the nonnegative square root.**\n\n\n\n\n\n\n\n**Answer:** mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3.\n\n"},{"title":"Standardization: what the answer 1 actually means","blocks":["Definition. Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units.
","![\"z=\\frac{x-\\mu_{\\mathrm{train}}}{\\sigma_{\\mathrm{train}}},\\qquad](\"assets/equations/45ffeb13784592e1.svg\")
Subtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1].
","Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.
","Training observations = [3,3,7,7].
","New observation = 9.
","Estimate the population standard deviation from the training values.
","Find the training mean and standard deviation; standardize all four training values and the new observation.
","Mean and spread. The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.
","N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.
","Yaad rakhne ka cue: Mean = sabka total ÷ count; variance = squared distances ka mean.
","Standardization. Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units.
","x = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value.
","![\"z=\\frac{x-\\mu}{\\sigma}\"](\"assets/equations/05f73c2bd4e79c0a.svg\")
Yaad rakhne ka cue: Pehle centre hatao, phir spread se divide karo.
","Step 1: add all four observations.
","Step 2: divide by the count to obtain the mean.
","Step 3: subtract the mean from each observation.
","Step 4: square each deviation.
","Step 5: add the squared deviations.
","Step 6: divide by four for population variance.
","Step 7: take the nonnegative square root.
","Step 8: standardize each original training observation.
","![\"\\frac{3-5}{2}=-1\"](\"assets/equations/fb029ad3b6f0a91a.svg\")
![\"\\frac{7-5}{2}=1\"](\"assets/equations/c0a2851f2b573cc2.svg\")
The repeated observations give [-1, −1, 1, 1].
","Step 9: subtract the training mean from the new observation.
","![\"9-5=4\"](\"assets/equations/116784e6eda9bc25.svg\")
Step 10: divide by the training standard deviation.
","![\"z_{\\mathrm{new}}=\\frac42=2\"](\"assets/equations/a563909e15c191b2.svg\")
Answer: the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy.
","Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.
","x = 3; training mean μ = 5; training standard deviation σ = 2.
","Standardized value z and its meaning.
","Subtract the training centre, then measure that distance in units of the training standard deviation.
","A negative z means below the training mean; positive means above it; zero means equal to it.
","Yaad rakhne ka cue: Centre hatao, spread se divide; answer batata hai kitne σ door.
","Step 1: find distance from the mean.
","![\"x-\\mu=3-5=-2\"](\"assets/equations/29b71ab0c556c6dc.svg\")
Step 2: divide by one standard-deviation unit.
","![\"z=\\frac{-2}{2}=-1\"](\"assets/equations/679dccde63fa3152.svg\")
Step 3: interpret sign and size.
","The value is 1 standard-deviation unit(s) below the training mean.
","Final answer: z = -1. Use the same training statistics when transforming a new test input.
","Question. Training mean is 5 and training standard deviation is 2. Standardize x = 5 and interpret the result.
","x = 5; training mean μ = 5; training standard deviation σ = 2.
","Standardized value z and its meaning.
","Subtract the training centre, then measure that distance in units of the training standard deviation.
","A negative z means below the training mean; positive means above it; zero means equal to it.
","Yaad rakhne ka cue: Centre hatao, spread se divide; answer batata hai kitne σ door.
","Step 1: find distance from the mean.
","![\"x-\\mu=5-5=0\"](\"assets/equations/46cf71d500f243a5.svg\")
Step 2: divide by one standard-deviation unit.
","![\"z=\\frac{0}{2}=0\"](\"assets/equations/d8d3a4b4364ff836.svg\")
Step 3: interpret sign and size.
","The value is 0 standard-deviation unit(s) away from the training mean.
","Final answer: z = 0. Use the same training statistics when transforming a new test input.
","Question. Training mean is 5 and training standard deviation is 2. Standardize x = 9 and interpret the result.
","x = 9; training mean μ = 5; training standard deviation σ = 2.
","Standardized value z and its meaning.
","Subtract the training centre, then measure that distance in units of the training standard deviation.
","A negative z means below the training mean; positive means above it; zero means equal to it.
","Yaad rakhne ka cue: Centre hatao, spread se divide; answer batata hai kitne σ door.
","Step 1: find distance from the mean.
","![\"x-\\mu=9-5=4\"](\"assets/equations/640d8d05be10b7d3.svg\")
Step 2: divide by one standard-deviation unit.
","![\"z=\\frac{4}{2}=2\"](\"assets/equations/b913089938e80c1f.svg\")
Step 3: interpret sign and size.
","The value is 2 standard-deviation unit(s) above the training mean.
","Final answer: z = 2. Use the same training statistics when transforming a new test input.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s11","search":"Standardization: what the answer 1 actually means \n**Definition.** Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units.\n\nSubtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1].\n\n**Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.**\n\n\n\n### Given\n\nTraining observations = [3,3,7,7].\n\nNew observation = 9.\n\nEstimate the population standard deviation from the training values.\n\n### To find\n\nFind the training mean and standard deviation; standardize all four training values and the new observation.\n\n### Formula — which one and why\n\n**Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.\n\nN = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean.\n\n**Standardization.** Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units.\n\nx = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value.\n\n\n\n**Yaad rakhne ka cue:** Pehle centre hatao, phir spread se divide karo.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: add all four observations.**\n\n\n\n\n\n\n\n**Step 2: divide by the count to obtain the mean.**\n\n\n\n\n\n\n\n**Step 3: subtract the mean from each observation.**\n\n\n\n\n\n\n\n**Step 4: square each deviation.**\n\n\n\n\n\n\n\n**Step 5: add the squared deviations.**\n\n\n\n\n\n\n\n**Step 6: divide by four for population variance.**\n\n\n\n\n\n\n\n**Step 7: take the nonnegative square root.**\n\n\n\n\n\n\n\n**Step 8: standardize each original training observation.**\n\n\n\n\n\n\n\n\n\nThe repeated observations give [-1, −1, 1, 1].\n\n**Step 9: subtract the training mean from the new observation.**\n\n\n\n\n\n\n\n**Step 10: divide by the training standard deviation.**\n\n\n\n\n\n\n\n**Answer:** the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy.\n\n### Standardization with x = 3\n\n**Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.**\n\n\n\n### Given\n\nx = 3; training mean μ = 5; training standard deviation σ = 2.\n\n### To find\n\nStandardized value z and its meaning.\n\n### Formula — which one and why\n\nSubtract the training centre, then measure that distance in units of the training standard deviation.\n\nA negative z means below the training mean; positive means above it; zero means equal to it.\n\n\n\n**Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door.\n\n### Solution — step by step\n\n\n\n**Step 1: find distance from the mean.**\n\n\n\n\n\n\n\n**Step 2: divide by one standard-deviation unit.**\n\n\n\n\n\n\n\n"},{"title":"Chain rule: the complete calculation","blocks":["When one value depends on another, a change passes through both calculations. The total derivative multiplies the two local derivatives. It does not add them. Here w is the number being changed, z is an intermediate value, and L is the final value.
","Let z = 3w + 1 and L = z². We will calculate the derivative of L with respect to w at w = 1.
","Step 1: find the intermediate value. Replace w by 1: z = 3 × 1 + 1 = 4.
","Step 2: differentiate the last calculation. For L = z², the power rule gives dL/dz = 2z. At z = 4, this is 2 × 4 = 8. This measures how L responds to z.
","Step 3: differentiate the first calculation. For z = 3w + 1, the derivative of 3w is 3 and the derivative of the constant 1 is 0. Therefore dz/dw = 3. This measures how z responds to w.
","Step 4: connect the two effects. Multiply 8 by 3. The final derivative is 24.
","![\"\\frac{dL}{dw}=\\frac{dL}{dz}\\frac{dz}{dw}=(2z)(3)=8\\times3=24\"](\"assets/equations/f43476deed69b900.svg\")
The function value is L = 4² = 16. The derivative is 24. These have different meanings: 16 is the current value of L; 24 is its local rate of change per unit change in w.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c3-s12","search":"Chain rule: the complete calculation When one value depends on another, a change passes through both calculations. The total derivative multiplies the two local derivatives. It does not add them. Here w is the number being changed, z is an intermediate value, and L is the final value.\n\n\n\nLet z = 3w + 1 and L = z². We will calculate the derivative of L with respect to w at w = 1.\n**Step 1: find the intermediate value.** Replace w by 1: z = 3 × 1 + 1 = 4.\n**Step 2: differentiate the last calculation.** For L = z², the power rule gives dL/dz = 2z. At z = 4, this is 2 × 4 = 8. This measures how L responds to z.\n**Step 3: differentiate the first calculation.** For z = 3w + 1, the derivative of 3w is 3 and the derivative of the constant 1 is 0. Therefore dz/dw = 3. This measures how z responds to w.\n**Step 4: connect the two effects.** Multiply 8 by 3. The final derivative is 24.\n\nThe function value is L = 4² = 16. The derivative is 24. These have different meanings: 16 is the current value of L; 24 is its local rate of change per unit change in w.\n"}]},{"n":4,"title":"Neural-network foundations","sections":[{"title":"What does learning from data mean","blocks":["Definition. Machine learning develops models from data so that they can perform a task on new examples. A model is a calculation rule with adjustable numbers. Deep learning uses neural networks with multiple trainable layers; each layer transforms the previous layer's outputs.
","A sample means one example in the dataset. Suppose one example says: an input number is 2 and the required answer is 4. We write the input as x = 2 and the correct answer, or target, as y = 4. These letters are labels for the two numbers.
","An input feature is one recorded property used to make a prediction. One house might have area, number of rooms and age as three features. The three values together describe one sample. An input vector is just an ordered list of those feature values. Order matters: the first model weight must always multiply the first feature.
","The model's calculated answer is the prediction, written ŷ, read “y hat”. The hat helps us distinguish the calculated answer ŷ from the required answer y. A prediction is a probability only when the task and output function give it that interpretation.
","In the following model, x is the input, w is a weight and b is a bias. A weight multiplies the input. A bias is added once after multiplication. Both w and b are adjustable model numbers, called parameters.
","![\"\\hat](\"assets/equations/b896c82c08881db8.svg\")
The model first multiplies w and x; then it adds b. For several inputs it uses one matching weight per input, adds their products, and then adds the bias.
","Yaad rakhne ka cue: x diya hua input; y sahi answer; ŷ model ka answer; w aur b training mein badlenge.
","Question. A linear model uses ŷ = wx+b. For x = 2, w = 1 and b = 0, calculate its prediction.
","Input x = 2.
","Weight w = 1; bias b = 0.
","The output is linear: no extra activation is specified.
","The predicted value ŷ.
","Use the stated model. The multiplication gives the input contribution; the bias shifts the result.
","x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.
","Yaad rakhne ka cue: Pehle input × weight, phir bias add.
","Step 1: multiply the input by its weight.
","![\"wx=(1)(2)=2\"](\"assets/equations/0690bd581db62ce5.svg\")
Step 2: add the bias once.
","![\"\\hat](\"assets/equations/31e3d42796c5dd84.svg\")
Final answer: ŷ = 2. This is a prediction, not a loss or a learned weight.
","Question. A linear model uses ŷ = wx+b. For x = 2, w = 1.5 and b = 1, calculate its prediction.
","Input x = 2.
","Weight w = 1.5; bias b = 1.
","The output is linear: no extra activation is specified.
","The predicted value ŷ.
","Use the stated model. The multiplication gives the input contribution; the bias shifts the result.
","x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.
","Yaad rakhne ka cue: Pehle input × weight, phir bias add.
","Step 1: multiply the input by its weight.
","![\"wx=(1.5)(2)=3\"](\"assets/equations/3cafd271941bad15.svg\")
Step 2: add the bias once.
","![\"\\hat](\"assets/equations/3a2f92dc89e2b2d2.svg\")
Final answer: ŷ = 4. This is a prediction, not a loss or a learned weight.
","Question. A linear model uses ŷ = wx+b. For x = 3, w = 1.5 and b = 1, calculate its prediction.
","Input x = 3.
","Weight w = 1.5; bias b = 1.
","The output is linear: no extra activation is specified.
","The predicted value ŷ.
","Use the stated model. The multiplication gives the input contribution; the bias shifts the result.
","x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.
","Yaad rakhne ka cue: Pehle input × weight, phir bias add.
","Step 1: multiply the input by its weight.
","![\"wx=(1.5)(3)=4.5\"](\"assets/equations/4b2ec1addfdbb9fd.svg\")
Step 2: add the bias once.
","![\"\\hat](\"assets/equations/b28b202a19e96ace.svg\")
Final answer: ŷ = 5.5. This is a prediction, not a loss or a learned weight.
","Question. A linear model uses ŷ = wx+b. For x = 2, w = -0.5 and b = 3, calculate its prediction.
","Input x = 2.
","Weight w = -0.5; bias b = 3.
","The output is linear: no extra activation is specified.
","The predicted value ŷ.
","Use the stated model. The multiplication gives the input contribution; the bias shifts the result.
","x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.
","Yaad rakhne ka cue: Pehle input × weight, phir bias add.
","Step 1: multiply the input by its weight.
","![\"wx=(-0.5)(2)=-1\"](\"assets/equations/0ee6e60703add42d.svg\")
Step 2: add the bias once.
","![\"\\hat](\"assets/equations/f2369bf69098b26c.svg\")
Final answer: ŷ = 2. This is a prediction, not a loss or a learned weight.
","An initial model can predict 2 when the target is 4. A loss function measures the discrepancy. Training calculates how changing each parameter changes the loss; that rate is a gradient. An optimizer uses the gradient to adjust the parameters. The learning rate controls the size of this adjustment. The loss, derivative and complete update are explained with full examples in the next lessons.
","The learning rate is a hyperparameter: a setting chosen for training. A learned weight is a parameter: an adjustable number inside the model. During ordinary inference, the trained weights stay fixed and the model calculates an output for a new input.
","For many training samples, the objective normally averages the individual losses. θ (theta) means the collection of adjustable parameters; N is the number of training examples; Lᵢ is the loss for example i.
","![\"J(\\theta)=\\frac1N\\sum_{i=1}^{N}L_i(\\theta)\"](\"assets/equations/48f2afe9071bd1c9.svg\")
Σ is an instruction to add the listed terms. For three losses it means L₁ + L₂ + L₃, followed by division by 3. Regularization may add a penalty; the regularization chapter explains that separate term.
","A layer contains several neurons that calculate outputs from the same incoming values. In an image model, earlier layers can respond to simple patterns such as edges; later layers combine earlier responses into more complex features. This hierarchy is learned from data. A deep network is not guaranteed to learn useful features merely because it has many layers; training data, architecture and optimization still matter.
","Generalization means performing well on new examples. Training data adjusts weights; validation data helps choose settings and stopping time; held-out test data evaluates the chosen model. Small training loss alone does not establish good generalization.
","Yaad rakhne ka cue: training = answer nikalo → loss nikalo → gradients nikalo → parameters update karo; inference = trained model se answer nikalo.
","Question. Three training examples have losses 2, 0.5 and 0.5. Calculate their mean training loss.
","L₁ = 2; L₂ = 0.5; L₃ = 0.5; N = 3.
","The mean loss J.
","The question asks for the mean, so add all sample losses and divide by the number of samples.
","Lᵢ means loss of example i. N counts examples.
","![\"J=\\frac{L_1+L_2+L_3}{N}\"](\"assets/equations/3763c395ed6a5c3c.svg\")
Yaad rakhne ka cue: Mean = total loss ÷ examples.
","Step 1: add the losses.
","![\"2+0.5+0.5=3\"](\"assets/equations/a5dc1556ae91ba13.svg\")
Step 2: divide by the number of examples.
","![\"J=\\frac33=1\"](\"assets/equations/ddda757808debff5.svg\")
Final answer: Mean training loss = 1.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s0","search":"What does learning from data mean **Definition.** Machine learning develops models from data so that they can perform a task on new examples. A model is a calculation rule with adjustable numbers. Deep learning uses neural networks with multiple trainable layers; each layer transforms the previous layer's outputs.\n\n### Start with one complete training example\nA **sample** means one example in the dataset. Suppose one example says: an input number is 2 and the required answer is 4. We write the input as x = 2 and the correct answer, or **target**, as y = 4. These letters are labels for the two numbers.\n\nAn **input feature** is one recorded property used to make a prediction. One house might have area, number of rooms and age as three features. The three values together describe one sample. An **input vector** is just an ordered list of those feature values. Order matters: the first model weight must always multiply the first feature.\n\nThe model's calculated answer is the **prediction**, written ŷ, read “y hat”. The hat helps us distinguish the calculated answer ŷ from the required answer y. A prediction is a probability only when the task and output function give it that interpretation.\n\n### The simplest model: multiply, then add\nIn the following model, x is the input, w is a weight and b is a bias. A **weight** multiplies the input. A **bias** is added once after multiplication. Both w and b are adjustable model numbers, called **parameters**.\n\nThe model first multiplies w and x; then it adds b. For several inputs it uses one matching weight per input, adds their products, and then adds the bias.\n\n**Yaad rakhne ka cue:** x diya hua input; y sahi answer; ŷ model ka answer; w aur b training mein badlenge.\n\n\n\n\n### Model example 1: calculate a prediction\n\n**Question. A linear model uses ŷ = wx+b. For x = 2, w = 1 and b = 0, calculate its prediction.**\n\n\n\n### Given\n\nInput x = 2.\n\nWeight w = 1; bias b = 0.\n\nThe output is linear: no extra activation is specified.\n\n### To find\n\nThe predicted value ŷ.\n\n### Formula — which one and why\n\nUse the stated model. The multiplication gives the input contribution; the bias shifts the result.\n\nx is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.\n\n\n\n**Yaad rakhne ka cue:** Pehle input × weight, phir bias add.\n\n### Solution — step by step\n\n\n\n**Step 1: multiply the input by its weight.**\n\n\n\n\n\n\n\n**Step 2: add the bias once.**\n\n\n\n\n\n\n\n**Final answer:** ŷ = 2. This is a prediction, not a loss or a learned weight.\n\n\n\n### Model example 2: calculate a prediction\n\n**Question. A linear model uses ŷ = wx+b. For x = 2, w = 1.5 and b = 1, calculate its prediction.**\n\n\n\n### Given\n\nInput x = 2.\n\nWeight w = 1.5; bias b = 1.\n\nThe output is linear: no extra activation is specified.\n\n### To find\n\nThe predicted value ŷ.\n\n### Formula — which one and why\n\nUse the stated model. The multiplication gives the input contribution; the bias shifts the result.\n\nx is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.\n\n\n\n**Yaad rakhne ka cue:** Pehle input × weight, phir bias add.\n\n### Solution — step by step\n\n\n\n**Step 1: multiply the input by its weight.**\n\n\n\n\n\n\n\n**Step 2: add the bias once.**\n\n\n\n\n\n\n\n**Final answer:** ŷ = 4. This is a prediction, not a loss or a learned weight.\n\n\n\n### Model example 3: calculate a prediction\n\n**Question. A linear model uses ŷ = wx+b. For x = 3, w = 1.5 and b = 1, calculate its prediction.**\n\n\n\n### Given\n\nInput x = 3.\n\nWeight w = 1.5; bias b"},{"title":"Your first complete training step","blocks":["Question. A linear model is defined by ![\"\\hat](\"assets/equations/b0668b91b6f454d0.svg\")
. For the training example x = 2, y = 4, use initial weight w₀ = 1, learning rate η = 0.1 and half squared loss. Perform two gradient-descent updates and report the prediction and loss after each update.
Model: ŷ = wx; bias is absent and is not updated.
","x = 2; y = 4; initial w₀ = 1; η = 0.1.
","Loss = half squared error; perform two updates.
","Find the weight, prediction and loss after each update.
","Chain rule with only a trainable weight. The weight changes the prediction and the prediction changes the half squared loss. Multiply both rates, then subtract the scaled gradient. The stated model has no bias parameter.
","ŷ = prediction; y = target; x = input; w = weight; η = learning rate; g = loss derivative with respect to w.
","![\"L=\\frac12(\\hat](\"assets/equations/df5fc0c0a9ad8296.svg\")
![\"g=(\\hat](\"assets/equations/f90a893376d444b0.svg\")
Yaad rakhne ka cue: Prediction minus target, phir input se multiply; naya weight = purana − ηg.
","The input and target remain fixed; only the weight is trainable. There is no bias parameter in this question.
","![\"\\hat](\"assets/equations/991254a061f484f9.svg\")
Apply the chain rule through the prediction:
","![\"g_t=\\left.\\frac{\\partial](\"assets/equations/48779dd96502307c.svg\")
The factor 1/2 in the objective cancels the factor 2 from differentiating the square. The remaining factor x comes from differentiating wx with respect to w.
","Step 1: forward evaluation at the current parameter.
","![\"\\hat](\"assets/equations/8f7cd4947851e2db.svg\")
![\"L_0=\\frac12(-2)^2=2\"](\"assets/equations/6de0a03e9715872f.svg\")
Step 2: gradient at the current parameter.
","![\"g_1=(\\hat](\"assets/equations/eca6d4b6240460b4.svg\")
Step 3: parameter update.
","![\"w_1=w_0-\\eta](\"assets/equations/10fc5d59ff5f6fd3.svg\")
Step 4: evaluate the updated model.
","![\"\\hat](\"assets/equations/80c9306a0dfccc42.svg\")
Use w₁ = 1.4 and recompute the gradient; reusing the previous gradient would not be gradient descent on this objective.
","![\"g_2=(\\hat](\"assets/equations/6a7b033dfd881b35.svg\")
![\"w_2=w_1-\\eta](\"assets/equations/ef24d99282ed8b6f.svg\")
![\"\\hat](\"assets/equations/09afd5ffebd67c9e.svg\")
| Model state | Weight | Prediction | Half squared loss |
|---|---|---|---|
| Initial | 1 | 2 | 2 |
| After update 1 | 1.4 | 2.8 | 0.72 |
| After update 2 | 1.64 | 3.28 | 0.2592 |
Interpretation. Both gradients are negative, so the updates increase the weight toward the optimum w = y/x = 2. The loss decreases in these iterations. Gradient descent does not guarantee a decrease for every learning rate; step size matters.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s1","search":"Your first complete training step \n**Question.** A linear model is defined by \\(\\hat y=wx\\). For the training example x = 2, y = 4, use initial weight w₀ = 1, learning rate η = 0.1 and half squared loss. Perform two gradient-descent updates and report the prediction and loss after each update.\n\n\n\n### Given\n\nModel: ŷ = wx; bias is absent and is not updated.\n\nx = 2; y = 4; initial w₀ = 1; η = 0.1.\n\nLoss = half squared error; perform two updates.\n\n### To find\n\nFind the weight, prediction and loss after each update.\n\n### Formula — which one and why\n\n**Chain rule with only a trainable weight.** The weight changes the prediction and the prediction changes the half squared loss. Multiply both rates, then subtract the scaled gradient. The stated model has no bias parameter.\n\nŷ = prediction; y = target; x = input; w = weight; η = learning rate; g = loss derivative with respect to w.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Prediction minus target, phir input se multiply; naya weight = purana − ηg.\n\n### Solution — step by step\n\n\n\n\n\n### Model, objective and gradient\nThe input and target remain fixed; only the weight is trainable. There is no bias parameter in this question.\n\nApply the chain rule through the prediction:\n\nThe factor 1/2 in the objective cancels the factor 2 from differentiating the square. The remaining factor x comes from differentiating wx with respect to w.\n\n\n\n### Iteration 1\n**Step 1: forward evaluation at the current parameter.**\n\n\n**Step 2: gradient at the current parameter.**\n\n**Step 3: parameter update.**\n\n**Step 4: evaluate the updated model.**\n\n\n### Iteration 2\nUse w₁ = 1.4 and recompute the gradient; reusing the previous gradient would not be gradient descent on this objective.\n\n\n\n| Model state | Weight | Prediction | Half squared loss |\n| Initial | 1 | 2 | 2 |\n| After update 1 | 1.4 | 2.8 | 0.72 |\n| After update 2 | 1.64 | 3.28 | 0.2592 |\n**Interpretation.** Both gradients are negative, so the updates increase the weight toward the optimum w = y/x = 2. The loss decreases in these iterations. Gradient descent does not guarantee a decrease for every learning rate; step size matters.\n\n"},{"title":"Classification regression and learning types","blocks":["Definition. A learning problem is specified by its data, desired output and objective. Classification estimates a discrete label or class distribution; regression estimates numerical responses. The supervision type describes the feedback available during training.
","Regression predicts a numerical quantity, such as temperature, price or a continuous rating. Classification predicts a category, such as defective/non-defective or cat/dog/bird. Binary classification has two classes. Multiclass classification chooses one of several classes. Multilabel classification can choose several labels together, such as “beach” and “sunset” for the same image.
","Supervised learning uses known input–target pairs. Most examination numericals in this course use it. Unsupervised learning searches for patterns without supplied target labels, for example grouping similar samples. Reinforcement learning learns actions using rewards from an environment. You only need these distinctions as vocabulary here; detailed algorithms from last semester are not prerequisites for this course.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s2","search":"Classification regression and learning types **Definition.** A learning problem is specified by its data, desired output and objective. Classification estimates a discrete label or class distribution; regression estimates numerical responses. The supervision type describes the feedback available during training.\n\n### Explanation and working\n**Regression** predicts a numerical quantity, such as temperature, price or a continuous rating. **Classification** predicts a category, such as defective/non-defective or cat/dog/bird. Binary classification has two classes. Multiclass classification chooses one of several classes. Multilabel classification can choose several labels together, such as “beach” and “sunset” for the same image.\n\n\n\n**Supervised learning** uses known input–target pairs. Most examination numericals in this course use it. **Unsupervised learning** searches for patterns without supplied target labels, for example grouping similar samples. **Reinforcement learning** learns actions using rewards from an environment. You only need these distinctions as vocabulary here; detailed algorithms from last semester are not prerequisites for this book."},{"title":"Training validation and test data","blocks":["Definition. Dataset partitioning separates parameter estimation, model selection and final performance assessment. This separation is essential to evaluate generalization without contaminating the evaluation with selection decisions.
","Training data changes the weights. Validation data helps choose settings such as layer size, learning rate and the stopping epoch. Test data is held back for the final estimate of performance on unseen samples. A possible split is 70% training, 15% validation, 15% testing, but the best split depends on the task and dataset size.
","For 1,000 independent samples, this split gives 700 training, 150 validation and 150 testing. Avoid putting duplicates or near-identical views of the same subject across splits. For time series, later observations normally belong after earlier observations rather than being mixed randomly without considering time.
","Data leakage occurs when information that would not be available at prediction time enters training or model selection. Fitting a normalization rule to the whole dataset before splitting can leak information. Estimate preprocessing quantities from training data and apply the same rule to validation and test data.
","Normalization example. To map a training range 0 to 100 into 0 to 1, divide by 100. A value 40 becomes 0.4. Standardization instead subtracts the training mean and divides by the training standard deviation. Scaling can make gradients easier to optimize; it does not create labels or guarantee accuracy.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s3","search":"Training validation and test data **Definition.** Dataset partitioning separates parameter estimation, model selection and final performance assessment. This separation is essential to evaluate generalization without contaminating the evaluation with selection decisions.\n\n### Explanation and working\nTraining data changes the weights. Validation data helps choose settings such as layer size, learning rate and the stopping epoch. Test data is held back for the final estimate of performance on unseen samples. A possible split is 70% training, 15% validation, 15% testing, but the best split depends on the task and dataset size.\n\n\n\nFor 1,000 independent samples, this split gives 700 training, 150 validation and 150 testing. Avoid putting duplicates or near-identical views of the same subject across splits. For time series, later observations normally belong after earlier observations rather than being mixed randomly without considering time.\n**Data leakage** occurs when information that would not be available at prediction time enters training or model selection. Fitting a normalization rule to the whole dataset before splitting can leak information. Estimate preprocessing quantities from training data and apply the same rule to validation and test data.\n**Normalization example.** To map a training range 0 to 100 into 0 to 1, divide by 100. A value 40 becomes 0.4. Standardization instead subtracts the training mean and divides by the training standard deviation. Scaling can make gradients easier to optimize; it does not create labels or guarantee accuracy."},{"title":"Parameters and hyperparameters","blocks":["Definition. Model parameters are quantities estimated during training, whereas hyperparameters specify the architecture or training procedure and are selected separately. Optimizer states are algorithmic accumulators associated with the training procedure.
","Weights and biases are parameters learned during training. Learning rate, batch size, number of epochs, network architecture, dropout rate and regularization strength are hyperparameters chosen or tuned using validation performance. Adam’s moving averages are optimizer state: they are updated by the algorithm but are not extra network connection weights.
","An epoch is one complete pass through the training dataset. A batch is the group used to calculate one update. An iteration here means one update. If there are 100 samples and batch size 10, one epoch has 10 updates. One epoch does not generally mean one update.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s4","search":"Parameters and hyperparameters **Definition.** Model parameters are quantities estimated during training, whereas hyperparameters specify the architecture or training procedure and are selected separately. Optimizer states are algorithmic accumulators associated with the training procedure.\n\n### Explanation and working\nWeights and biases are parameters learned during training. Learning rate, batch size, number of epochs, network architecture, dropout rate and regularization strength are hyperparameters chosen or tuned using validation performance. Adam’s moving averages are optimizer state: they are updated by the algorithm but are not extra network connection weights.\n\n\n\nAn **epoch** is one complete pass through the training dataset. A **batch** is the group used to calculate one update. An **iteration** here means one update. If there are 100 samples and batch size 10, one epoch has 10 updates. One epoch does not generally mean one update."},{"title":"What one artificial neuron does","blocks":["Definition. An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output.
","An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive.
","The bias is one extra adjustable number added to the total. The total before activation is called z or preactivation. The activation function is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the activation.
","Yaad rakhne ka cue: inputs × apne weights → products ka sum → ek bias → activation → output.
","![\"z=\\sum_{i=1}^{d}w_ix_i+b=w^Tx+b\"](\"assets/equations/8032854fc489fa3f.svg\")
| Symbol | Meaning and role |
|---|---|
| xᵢ | Input component i; the neuron receives d such components |
| wᵢ | Trainable weight multiplying that input component |
| b | Trainable bias added once to this neuron's total |
| z | Preactivation: the result before applying the activation function |
| f | Chosen activation function, such as sigmoid or ReLU |
| a | Activation: the neuron's output after applying f |
The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs.
","The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin.
","The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values.
","Question. A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output.
","Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5].
","Bias = 0.1; activation = sigmoid.
","Find the preactivation z, then the sigmoid output.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid output. The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.
","z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.
","![\"a=\\frac1{1+e^{-z}}\"](\"assets/equations/3188e642a44baa47.svg\")
Yaad rakhne ka cue: Minus-z ki e power, plus one, phir one divided by whole denominator.
","Step 1: multiply input 1 by weight 1.
","![\"x_1w_1=2(0.4)=0.8\"](\"assets/equations/568d8f8a50e883d1.svg\")
Step 2: multiply input 2 by weight 2.
","![\"x_2w_2=1(-0.2)=-0.2\"](\"assets/equations/2aecb1c03e00ecf3.svg\")
The minus sign belongs to the weight, so the contribution stays negative.
","Step 3: multiply input 3 by weight 3.
","![\"x_3w_3=3(0.5)=1.5\"](\"assets/equations/b26c8cde278f5783.svg\")
Step 4: add the three contributions.
","![\"s=0.8+(-0.2)+1.5=0.6+1.5=2.1\"](\"assets/equations/928ca4d8a84ae46e.svg\")
Step 5: add the bias once.
","![\"z=s+b=2.1+0.1=2.2\"](\"assets/equations/fc2fbdf24fc7fcdc.svg\")
This is the preactivation. It is not yet the requested sigmoid output.
","Step 6: negate z for the exponential.
","![\"-z=-2.2\"](\"assets/equations/6ae6042702919060.svg\")
![\"e^{-z}=e^{-2.2}\\approx0.110803158\"](\"assets/equations/f3e8120863f30069.svg\")
Step 7: calculate the complete denominator.
","![\"1+e^{-z}=1+0.110803158=1.110803158\"](\"assets/equations/417b75e5d36172d1.svg\")
Step 8: divide one by that denominator.
","![\"a=\\frac1{1+e^{-z}}=\\frac1{1.110803158}\\approx0.900249511\"](\"assets/equations/0bc1ac01a29b10b0.svg\")
Final answer: sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025.
","With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron:
","![\"z=Wx+b,\\qquad](\"assets/equations/247718acabda6724.svg\")
Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together.
","Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.
","Inputs = [2].
","Weights = [3]; bias = 1.
","Activation = ReLU.
","z before activation and a after activation.
","The weighted sum combines the inputs. The output must then use the activation explicitly requested.
","z names the sum; a names the activated result. They are not always equal.
","![\"z=\\sum_i](\"assets/equations/3497cbcbfa494dd1.svg\")
![\"a=\\max(0,z)\"](\"assets/equations/e3267d2a19366c7b.svg\")
Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.
","Step 1: multiply input 1 by its own weight.
","![\"(2)(3)=6\"](\"assets/equations/0789e90228011be2.svg\")
Step 2: add the input contributions.
","![\"s=(6)=6\"](\"assets/equations/7d688f2957123f9f.svg\")
Step 3: add the bias once.
","![\"z=6+(1)=7\"](\"assets/equations/7cd7ec47b25d8ce7.svg\")
Step 4: apply the stated activation.
","![\"a=\\max(0,7)=7\"](\"assets/equations/a730b8a58ff43cba.svg\")
Final answer: z = 7; a = 7.
","Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.
","Inputs = [2, 1].
","Weights = [-1, 0.5]; bias = 0.2.
","Activation = ReLU.
","z before activation and a after activation.
","The weighted sum combines the inputs. The output must then use the activation explicitly requested.
","z names the sum; a names the activated result. They are not always equal.
","Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.
","Step 1: multiply input 1 by its own weight.
","![\"(2)(-1)=-2\"](\"assets/equations/cc29fd0da04569d8.svg\")
Step 2: multiply input 2 by its own weight.
","![\"(1)(0.5)=0.5\"](\"assets/equations/47470f92bf2adcc4.svg\")
Step 3: add the input contributions.
","![\"s=(-2)+(0.5)=-1.5\"](\"assets/equations/f2345e5dfa56c33e.svg\")
Step 4: add the bias once.
","![\"z=-1.5+(0.2)=-1.3\"](\"assets/equations/3c99c8f72adf2141.svg\")
Step 5: apply the stated activation.
","![\"a=\\max(0,-1.3)=0\"](\"assets/equations/2503c315df3664fa.svg\")
Final answer: z = -1.3; a = 0.
","Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.
","Inputs = [1, 2].
","Weights = [0.2, -0.1]; bias = 0.
","Activation = sigmoid.
","z before activation and a after activation.
","The weighted sum combines the inputs. The output must then use the activation explicitly requested.
","z names the sum; a names the activated result. They are not always equal.
","Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.
","Step 1: multiply input 1 by its own weight.
","![\"(1)(0.2)=0.2\"](\"assets/equations/5c759a37e815fafb.svg\")
Step 2: multiply input 2 by its own weight.
","![\"(2)(-0.1)=-0.2\"](\"assets/equations/ba87aedd127c080a.svg\")
Step 3: add the input contributions.
","![\"s=(0.2)+(-0.2)=0\"](\"assets/equations/9a1dc06bff5e31d8.svg\")
Step 4: add the bias once.
","![\"z=0+(0)=0\"](\"assets/equations/c8d9f0d00314027b.svg\")
Step 5: apply the stated activation.
","![\"a=\\frac1{1+e^{0}}=\\frac1{1+1}=0.5\"](\"assets/equations/2d7874ea2511447b.svg\")
Final answer: z = 0; a = 0.5.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s5","search":"What one artificial neuron does **Definition.** An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output.\n\n### What goes into a neuron?\nAn input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive.\n\nThe **bias** is one extra adjustable number added to the total. The total before activation is called z or **preactivation**. The **activation function** is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the **activation**.\n\n**Yaad rakhne ka cue:** inputs × apne weights → products ka sum → ek bias → activation → output.\n\n### Mathematical model and notation\n\n\n| Symbol | Meaning and role |\n| xᵢ | Input component i; the neuron receives d such components |\n| wᵢ | Trainable weight multiplying that input component |\n| b | Trainable bias added once to this neuron's total |\n| z | Preactivation: the result before applying the activation function |\n| f | Chosen activation function, such as sigmoid or ReLU |\n| a | Activation: the neuron's output after applying f |\n\nThe dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs.\n\n### Why both a bias and an activation are needed\nThe bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin.\n\nThe activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values.\n\n\n\n### Worked numerical\n**Question.** A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output.\n\n\n\n### Given\n\nInputs = [2,1,3]; matching weights = [0.4,−0.2,0.5].\n\nBias = 0.1; activation = sigmoid.\n\n### To find\n\nFind the preactivation z, then the sigmoid output.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.\n\nz = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.\n\n\n\n**Yaad rakhne ka cue:** Minus-z ki e power, plus"},{"title":"Model answer: explain an artificial neuron","question":["Question. Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example.
"],"answer":["An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers.
","![\"z=w_1x_1+w_2x_2+\\cdots+w_nx_n+b,\\qquad](\"assets/equations/8477fc1ddc03ffc3.svg\")
| Part | Meaning | What it does |
|---|---|---|
| x₁, x₂, … | Input features | Supply the information for this calculation |
| w₁, w₂, … | Weights | Control the contribution of each matching input |
| b | Bias | Shifts the weighted sum independently of the inputs |
| z | Net input | The number before activation |
| f | Activation rule | Transforms z, often nonlinearly |
| a | Activated output | The number sent onward |
Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.
","Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU.
","Preactivation z and activated output a.
","Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total.
","z means before activation; a means after activation.
","![\"z=w_1x_1+w_2x_2+b\"](\"assets/equations/04168cf0cec0712f.svg\")
Yaad rakhne ka cue: Product, product, sum, bias, phir activation.
","Step 1: multiply the first pair.
","Step 2: multiply the second pair.
","![\"(-0.2)(1)=-0.2\"](\"assets/equations/44552ee4afe034fa.svg\")
Step 3: add both contributions.
","![\"1+(-0.2)=0.8\"](\"assets/equations/d74c1d06c5bc866b.svg\")
Step 4: add the bias once.
","![\"z=0.8+0.1=0.9\"](\"assets/equations/fe5a369324074a4c.svg\")
Step 5: apply ReLU.
","![\"a=\\max(0,0.9)=0.9\"](\"assets/equations/abc54eeca8cf7516.svg\")
Final answer: z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged.
","The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once.
"],"kind":"theory","id":"mt2-c4-s6","search":"Model answer: explain an artificial neuron **Question.** Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example.\n\n**Model answer.**\nAn artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers.\n\n\n\n\n### Formula and symbols\n\n| Part | Meaning | What it does |\n| x₁, x₂, … | Input features | Supply the information for this calculation |\n| w₁, w₂, … | Weights | Control the contribution of each matching input |\n| b | Bias | Shifts the weighted sum independently of the inputs |\n| z | Net input | The number before activation |\n| f | Activation rule | Transforms z, often nonlinearly |\n| a | Activated output | The number sent onward |\n\n\n\n### Worked ReLU neuron example\n\n**Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.**\n\n\n\n### Given\n\nInputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU.\n\n### To find\n\nPreactivation z and activated output a.\n\n### Formula — which one and why\n\nEach input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total.\n\nz means before activation; a means after activation.\n\n\n\n\n\n**Yaad rakhne ka cue:** Product, product, sum, bias, phir activation.\n\n### Solution — step by step\n\n\n\n**Step 1: multiply the first pair.**\n\n\n\n\n\n\n\n**Step 2: multiply the second pair.**\n\n\n\n\n\n\n\n**Step 3: add both contributions.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply ReLU.**\n\n\n\n\n\n\n\n**Final answer:** z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged.\n\n\n\nThe weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once.\n"},{"title":"Activation functions and how to calculate them","blocks":["A neuron first produces the weighted sum z. The activation function f is a rule that transforms that sum into the output a = f(z). Applying an activation does not change the stored weights; it calculates a new output from the current sum.
","If every layer only multiplies and adds, several layers still combine into a single affine transformation. A nonlinear activation lets the network represent more complicated relationships. The output activation must also suit the task: sigmoid for a binary-class probability, softmax for mutually exclusive class probabilities, and often a linear output for unrestricted regression.
","The activation value says what the neuron outputs. The activation derivative says how quickly that output changes when z changes, which is needed during backpropagation. For sigmoid at z = 0, the value is 0.5 but the derivative is 0.25. They serve different purposes.
","Yaad rakhne ka cue: z = activation se pehle ka sum; a = uske baad ka answer; f′ = answer kitni tezi se badlega.
","Definition. An activation function maps a neuron's preactivation to its output. Nonlinear activations allow a multilayer network to represent nonlinear functions; the output activation also encodes the prediction type. Without a nonlinear activation, a composition of affine layers remains affine.
","![\"\\sigma(z)=\\frac{1}{1+e^{-z}},\\quad](\"assets/equations/0b7ede87d0558048.svg\")
Sigmoid converts any finite real input into a value strictly between 0 and 1. At zero it is 0.5. It is useful for a binary-class probability or independent multilabel outputs. At very positive or negative inputs it saturates, so its derivative becomes small. Its derivative is at most 0.25.
","![\"\\tanh(z)=\\frac{e^{2z}-1}{e^{2z}+1},\\quad](\"assets/equations/151466f4a43db268.svg\")
Tanh lies between −1 and +1 and is zero-centered. It is common in recurrent hidden states. It also saturates. “Zero-centered” does not mean “no vanishing gradients”.
","![\"\\operatorname{ReLU}(z)=\\max(0,z),\\quad](\"assets/equations/8ed42aad8099504b.svg\")
ReLU returns the input if positive and zero if negative. It is cheap to compute and has derivative 1 on its positive side. Units can become inactive on all relevant inputs, known as dying ReLU. At z = 0 the mathematical derivative is undefined; a convention such as 0 is used in implementations and this course’s exercises when needed.
","![\"\\operatorname{LeakyReLU}(z)=\\begin{cases}z&z\\ge0\\\\az&z<0\\end{cases},\\qquad](\"assets/equations/ba63451671574a0f.svg\")
The small negative slope a lets some gradient through for negative inputs. At z = −2 and a = 0.01, output = −0.02. Do not take the absolute value. PReLU learns the negative slope; Leaky ReLU normally fixes it.
","![\"p_i=\\frac{e^{z_i}}{\\sum_j](\"assets/equations/6ad7e7d9830e70ac.svg\")
Softmax acts on a group of class scores, called logits. All probabilities are positive and sum to 1. Subtracting the same maximum c from all logits leaves the result unchanged and avoids very large exponentials. It is suited to mutually exclusive multiclass output.
","Full softmax example. For [−1, 0, 1], exponentials are [0.367879, 1, 2.718282]. Sum = 4.086161. Divide each by that same sum: [0.090031, 0.244728, 0.665241]. Check their sum is approximately 1. Three separate sigmoids would give [0.268941, 0.5, 0.731059], which are different and do not sum to 1.
","Write sigmoid as a negative power. The outer derivative contributes −1, and differentiating the inner exponential contributes a second negative sign:
","![\"\\sigma(z)=(1+e^{-z})^{-1}\"](\"assets/equations/e4d8f5baacfd2120.svg\")
![\"\\sigma'(z)=-(1+e^{-z})^{-2}(-e^{-z})=\\frac{e^{-z}}{(1+e^{-z})^2}\"](\"assets/equations/25b5ff83cfc93c4e.svg\")
Separate the expression into two factors:
","![\"\\sigma'(z)=\\frac{1}{1+e^{-z}}\\frac{e^{-z}}{1+e^{-z}}=\\sigma(z)[1-\\sigma(z)]\"](\"assets/equations/8342957fe889eca9.svg\")
The second factor is 1−σ(z) because subtracting 1/(1+e⁻ᶻ) from 1 leaves e⁻ᶻ/(1+e⁻ᶻ). Once a forward pass has supplied σ(z), its derivative requires no further exponential evaluation.
","Use the quotient rule on the exponential definition. Here u = e²ᶻ−1 and v = e²ᶻ+1, with u′ = v′ = 2e²ᶻ.
","![\"\\frac{d}{dz}\\tanh](\"assets/equations/1eaf1c302e07c3ae.svg\")
The identity 1−tanh²(z) gives the same expression, since the numerator becomes (e²ᶻ+1)²−(e²ᶻ−1)² = 4e²ᶻ. Therefore the backward-pass factor is 1−a² when a = tanh(z).
","For two affine layers without a nonlinear activation:
","![\"h=W_1x+b_1,\\qquad](\"assets/equations/886e2a40c2cd8e26.svg\")
The composition is still one affine transformation. A nonlinear activation between layers changes the family of functions the network can represent. This is the mathematical reason for using nonlinear hidden activations; merely adding affine layers does not provide this expressive advantage.
","Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.
","z = -2; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","![\"f\\prime(z)=a(1-a)\"](\"assets/equations/97012822a1c7e305.svg\")
Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","![\"-z=-(-2)=2\"](\"assets/equations/a9feed82e056cf24.svg\")
Step 2: calculate the exponential.
","![\"e^{2}\\approx7.389056099\"](\"assets/equations/b83d9532e6445b5a.svg\")
Step 3: add one in the denominator.
","![\"1+7.389056099\\approx8.389056099\"](\"assets/equations/4026cb0f8944f24f.svg\")
Step 4: divide one by the complete denominator.
","![\"a=\\frac1{8.389056099}\\approx0.119202922\"](\"assets/equations/f4a3cb96f1098e79.svg\")
Step 5: use the activated output in the slope.
","![\"f\\prime(-2)=(0.119202922)(1-0.119202922)\\approx0.104993585\"](\"assets/equations/154ffcd0747b53aa.svg\")
Final answer: a ≈ 0.119202922. Derivative ≈ 0.104993585.
","Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.
","z = 0; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","![\"-z=-(0)=0\"](\"assets/equations/17055d0f6bb7d8cb.svg\")
Step 2: calculate the exponential.
","![\"e^{0}\\approx1.000000000\"](\"assets/equations/6b377e92ad2fd68c.svg\")
Step 3: add one in the denominator.
","![\"1+1.000000000\\approx2.000000000\"](\"assets/equations/88f353ee1ae5e616.svg\")
Step 4: divide one by the complete denominator.
","![\"a=\\frac1{2.000000000}\\approx0.500000000\"](\"assets/equations/55f56d862c8a637f.svg\")
Step 5: use the activated output in the slope.
","![\"f\\prime(0)=(0.500000000)(1-0.500000000)\\approx0.250000000\"](\"assets/equations/5e112d2ee69475de.svg\")
Final answer: a ≈ 0.500000000. Derivative ≈ 0.250000000.
","Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.
","z = 2; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","![\"-z=-(2)=-2\"](\"assets/equations/cbbb55e23c6ba6a0.svg\")
Step 2: calculate the exponential.
","![\"e^{-2}\\approx0.135335283\"](\"assets/equations/9f6c4ea0634c6cc7.svg\")
Step 3: add one in the denominator.
","![\"1+0.135335283\\approx1.135335283\"](\"assets/equations/1264450d54002695.svg\")
Step 4: divide one by the complete denominator.
","![\"a=\\frac1{1.135335283}\\approx0.880797078\"](\"assets/equations/f08d0c34f5e2fdf7.svg\")
Step 5: use the activated output in the slope.
","![\"f\\prime(2)=(0.880797078)(1-0.880797078)\\approx0.104993585\"](\"assets/equations/278a4098b6b446b4.svg\")
Final answer: a ≈ 0.880797078. Derivative ≈ 0.104993585.
","Question. For preactivation z = -2, calculate the tanh output and state its derivative.
","z = -2; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","![\"a=\\frac{e^{2z}-1}{e^{2z}+1}\"](\"assets/equations/4f6e15283db79ff2.svg\")
![\"f\\prime(z)=1-a^2\"](\"assets/equations/7e7efe34a06943be.svg\")
Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","![\"2z=2(-2)=-4\"](\"assets/equations/0ab8a21d8aab882c.svg\")
Step 2: calculate the exponential.
","![\"e^{-4}\\approx0.018315639\"](\"assets/equations/9c3b8c152fd312b9.svg\")
Step 3: calculate numerator and denominator separately.
","![\"0.018315639-1\\approx-0.981684361\"](\"assets/equations/782210c69a20fd3b.svg\")
![\"0.018315639+1\\approx1.018315639\"](\"assets/equations/c7713b25655beb72.svg\")
Step 4: divide.
","![\"a=\\frac{-0.981684361}{1.018315639}\\approx-0.964027580\"](\"assets/equations/300fc0089c5516db.svg\")
Step 5: square the activated output for its slope.
","![\"f\\prime(-2)=1-(-0.964027580)^2\\approx0.070650825\"](\"assets/equations/bb2d4ce554c14198.svg\")
Final answer: a ≈ -0.964027580. Derivative ≈ 0.070650825.
","Question. For preactivation z = 0, calculate the tanh output and state its derivative.
","z = 0; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","![\"2z=2(0)=0\"](\"assets/equations/83fefb855224c584.svg\")
Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","![\"1.000000000-1\\approx0.000000000\"](\"assets/equations/40327f28206586d2.svg\")
![\"1.000000000+1\\approx2.000000000\"](\"assets/equations/c5e16c3256057084.svg\")
Step 4: divide.
","![\"a=\\frac{0.000000000}{2.000000000}\\approx0.000000000\"](\"assets/equations/081512ac7f7a1b51.svg\")
Step 5: square the activated output for its slope.
","![\"f\\prime(0)=1-(0.000000000)^2\\approx1.000000000\"](\"assets/equations/d3899b5f76509b8c.svg\")
Final answer: a ≈ 0.000000000. Derivative ≈ 1.000000000.
","Question. For preactivation z = 2, calculate the tanh output and state its derivative.
","z = 2; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","![\"2z=2(2)=4\"](\"assets/equations/28a6e73374f6be65.svg\")
Step 2: calculate the exponential.
","![\"e^{4}\\approx54.598150033\"](\"assets/equations/dcc7f746c4400033.svg\")
Step 3: calculate numerator and denominator separately.
","![\"54.598150033-1\\approx53.598150033\"](\"assets/equations/f63e045301bde055.svg\")
![\"54.598150033+1\\approx55.598150033\"](\"assets/equations/2c86d55c9549cedd.svg\")
Step 4: divide.
","![\"a=\\frac{53.598150033}{55.598150033}\\approx0.964027580\"](\"assets/equations/22a5642fac7a0997.svg\")
Step 5: square the activated output for its slope.
","![\"f\\prime(2)=1-(0.964027580)^2\\approx0.070650825\"](\"assets/equations/52d252bd261b679f.svg\")
Final answer: a ≈ 0.964027580. Derivative ≈ 0.070650825.
","Question. For preactivation z = -2, calculate the ReLU output and state its derivative.
","z = -2; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","-2 is negative, so use the negative branch.
","Step 2: calculate the output.
","![\"a=0\"](\"assets/equations/e93d3c89fb4a0a4b.svg\")
Step 3: state the slope on that branch.
","![\"f\\prime(-2)=0\"](\"assets/equations/fbc65c947d4ed97f.svg\")
Final answer: a ≈ 0.000000000. Derivative ≈ 0.000000000.
","Question. For preactivation z = 0, calculate the ReLU output and state its derivative.
","z = 0; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","Both branches meet at zero. The value is zero, but the left and right slopes differ.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.
","Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.
","Question. For preactivation z = 2, calculate the ReLU output and state its derivative.
","z = 2; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","2 is positive, so retain it.
","Step 2: calculate the output.
","![\"a=2\"](\"assets/equations/d3043f41a0385109.svg\")
Step 3: state the slope on that branch.
","![\"f\\prime(2)=1\"](\"assets/equations/1b767f92de440262.svg\")
Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.
","Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = -2; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","![\"a=\\begin{cases}z&z\\ge0\\\\0.1z&z<0\\end{cases}\"](\"assets/equations/ce8d71b3a5eb9dfc.svg\")
Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","-2 is negative, so use the negative branch.
","Step 2: calculate the output.
","![\"a=-0.2\"](\"assets/equations/24b81e63d794388e.svg\")
Step 3: state the slope on that branch.
","![\"f\\prime(-2)=0.1\"](\"assets/equations/eaab86572902924a.svg\")
Final answer: a ≈ -0.200000000. Derivative ≈ 0.100000000.
","Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = 0; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","Both branches meet at zero. The value is zero, but the left and right slopes differ.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.
","Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.
","Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = 2; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","2 is positive, so retain it.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.
","Question. Find stable softmax probabilities for logits [0, 0, 0].
","Logits = [0, 0, 0]; there are three mutually exclusive classes.
","All three class probabilities.
","Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.
","A logit is a raw class score. c is the largest score; pᵢ is probability for class i.
","![\"c=\\max_j](\"assets/equations/c40885bb51eddb21.svg\")
![\"p_i=\\frac{e^{z_i-c}}{\\sum_j](\"assets/equations/8c7e5aa437b15aae.svg\")
Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.
","Step 1: subtract the largest score from every entry.
","![\"z-c=\\begin{bmatrix}0&0&0\\end{bmatrix}\"](\"assets/equations/7e35e2c8f440c3bf.svg\")
Step 2: exponentiate shifted entry 1.
","Step 3: exponentiate shifted entry 2.
","Step 4: exponentiate shifted entry 3.
","Step 5: add all exponentials into one denominator.
","![\"D=1.000000000+1.000000000+1.000000000\\approx3.000000000\"](\"assets/equations/5cbcea061b92c91b.svg\")
Step 6: divide entry 1 by the same denominator.
","![\"p_1=\\frac{1.000000000}{3.000000000}\\approx0.333333333\"](\"assets/equations/abb38634253b454f.svg\")
Step 7: divide entry 2 by the same denominator.
","![\"p_2=\\frac{1.000000000}{3.000000000}\\approx0.333333333\"](\"assets/equations/6f93f83cf16d42a1.svg\")
Step 8: divide entry 3 by the same denominator.
","![\"p_3=\\frac{1.000000000}{3.000000000}\\approx0.333333333\"](\"assets/equations/dd8dca9605ca9fba.svg\")
Final answer: Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding.
","Question. Find stable softmax probabilities for logits [1, 2, 3].
","Logits = [1, 2, 3]; there are three mutually exclusive classes.
","All three class probabilities.
","Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.
","A logit is a raw class score. c is the largest score; pᵢ is probability for class i.
","Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.
","Step 1: subtract the largest score from every entry.
","![\"z-c=\\begin{bmatrix}-2&-1&0\\end{bmatrix}\"](\"assets/equations/36a969786afa504d.svg\")
Step 2: exponentiate shifted entry 1.
","Step 3: exponentiate shifted entry 2.
","![\"e^{-1}\\approx0.367879441\"](\"assets/equations/c8b081af8baa5f1b.svg\")
Step 4: exponentiate shifted entry 3.
","Step 5: add all exponentials into one denominator.
","![\"D=0.135335283+0.367879441+1.000000000\\approx1.503214724\"](\"assets/equations/ff6afad7c399e8cd.svg\")
Step 6: divide entry 1 by the same denominator.
","![\"p_1=\\frac{0.135335283}{1.503214724}\\approx0.090030573\"](\"assets/equations/60d7f2c1aed7f3ca.svg\")
Step 7: divide entry 2 by the same denominator.
","![\"p_2=\\frac{0.367879441}{1.503214724}\\approx0.244728471\"](\"assets/equations/958d44e6661ca616.svg\")
Step 8: divide entry 3 by the same denominator.
","![\"p_3=\\frac{1.000000000}{1.503214724}\\approx0.665240956\"](\"assets/equations/6153a5efe8276ec2.svg\")
Final answer: Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s7","search":"Activation functions and how to calculate them \n### What an activation function does, and why it is needed\nA neuron first produces the weighted sum z. The activation function f is a rule that transforms that sum into the output a = f(z). Applying an activation does not change the stored weights; it calculates a new output from the current sum.\n\nIf every layer only multiplies and adds, several layers still combine into a single affine transformation. A nonlinear activation lets the network represent more complicated relationships. The output activation must also suit the task: sigmoid for a binary-class probability, softmax for mutually exclusive class probabilities, and often a linear output for unrestricted regression.\n\nThe **activation value** says what the neuron outputs. The **activation derivative** says how quickly that output changes when z changes, which is needed during backpropagation. For sigmoid at z = 0, the value is 0.5 but the derivative is 0.25. They serve different purposes.\n\n**Yaad rakhne ka cue:** z = activation se pehle ka sum; a = uske baad ka answer; f′ = answer kitni tezi se badlega.\n\n**Definition.** An activation function maps a neuron's preactivation to its output. Nonlinear activations allow a multilayer network to represent nonlinear functions; the output activation also encodes the prediction type. Without a nonlinear activation, a composition of affine layers remains affine.\n\n### Explanation and working\n\nSigmoid converts any finite real input into a value strictly between 0 and 1. At zero it is 0.5. It is useful for a binary-class probability or independent multilabel outputs. At very positive or negative inputs it saturates, so its derivative becomes small. Its derivative is at most 0.25.\n\n\n\n\nTanh lies between −1 and +1 and is zero-centered. It is common in recurrent hidden states. It also saturates. “Zero-centered” does not mean “no vanishing gradients”.\n\nReLU returns the input if positive and zero if negative. It is cheap to compute and has derivative 1 on its positive side. Units can become inactive on all relevant inputs, known as dying ReLU. At z = 0 the mathematical derivative is undefined; a convention such as 0 is used in implementations and this book’s exercises when needed.\n\nThe small negative slope a lets some gradient through for negative inputs. At z = −2 and a = 0.01, output = −0.02. Do not take the absolute value. PReLU learns the negative slope; Leaky ReLU normally fixes it.\n\nSoftmax acts on a group of class scores, called logits. All probabilities are positive and sum to 1. Subtracting the same maximum c from all logits leaves the result unchanged and avoids very large exponentials. It is suited to mutually exclusive multiclass output.\n**Full softmax example.** For [−1, 0, 1], exponentials are [0.367879, 1, 2.718282]. Sum = 4.086161. Divide each by that same sum: [0.090031, 0.244728, 0.665241]. Check their sum is approximately 1. Three separate sigmoids would give [0.268941, 0.5, 0.731059], which are different and do not sum to 1.\n\n### Derivation of the sigmoid derivative\nWrite sigmoid as a negative power. The outer derivative contributes −1, and differentiating the inner exponential contributes a second negative sign:\n\n\nSeparate the expression into two factors:\n\nThe second factor is 1−σ(z) because subtracting 1/(1+e⁻ᶻ) from 1 leaves e⁻ᶻ/(1+e⁻ᶻ). Once a forward pass has supplied σ(z), its derivative requires no further exponential evaluation.\n\n### Derivation of the tanh derivative\nUse the q"},{"title":"Model answer: why a network needs nonlinear activation","question":["Question. Why are activation functions required in a multilayer neural network? Compare sigmoid, tanh and ReLU, including their limitations.
"],"answer":["An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary.
","For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second:
","![\"y=3(2x+1)+4=6x+3+4=6x+7\"](\"assets/equations/f54676c512498a38.svg\")
This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent.
","| Activation | Output range | Useful property | Main limitation |
|---|---|---|---|
| Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small |
| Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients |
| ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient |
Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score.
","Exam conclusion: select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction.
"],"kind":"theory","id":"mt2-c4-s8","search":"Model answer: why a network needs nonlinear activation **Question.** Why are activation functions required in a multilayer neural network? Compare sigmoid, tanh and ReLU, including their limitations.\n\n**Model answer.**\nAn activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary.\n\n\n\n\n### Why two linear layers are still linear in the input\nFor the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second:\n\nThis is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent.\n\n| Activation | Output range | Useful property | Main limitation |\n| Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small |\n| Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients |\n| ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient |\n\nSigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score.\n\n**Exam conclusion:** select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction.\n"},{"title":"Choosing a loss","blocks":["Definition. A loss function quantifies the discrepancy between a prediction and its target. The training objective aggregates per-example losses and may add regularization. The loss must be compatible with the task and output parameterization.
","Activation produces the prediction. Loss measures how wrong that prediction is. An optimizer uses gradients to update parameters. These are three different roles.
","![\"\\operatorname{MSE}=\\frac1n\\sum_{i=1}^n(\\hat](\"assets/equations/46872a64dc44481e.svg\")
Question. Targets are [2, 4], predictions are [1, 6]. Find MSE and MAE.
","Targets = [2,4]; predictions = [1,6]; n = 2.
","Find mean squared error and mean absolute error.
","Squared-error objectives. Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.
","ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.
","Yaad rakhne ka cue: Pehle error square; phir question ke hisaab se sum, half ya mean.
","Mean absolute error. Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes.
","n = number of predictions. |e| means the nonnegative magnitude of error e.
","![\"\\mathrm{MAE}=\\frac1n\\sum_i|\\hat](\"assets/equations/da84fa0919b7d150.svg\")
Yaad rakhne ka cue: MAE mein magnitude; MSE mein square.
","Step 1: calculate prediction minus target.
","![\"e_1=1-2=-1\"](\"assets/equations/dba84d8eacba9175.svg\")
![\"e_2=6-4=2\"](\"assets/equations/7004e5d2962bc876.svg\")
Step 2: square both errors.
","![\"e_1^2=(-1)^2=1\"](\"assets/equations/725d81fce4be12d0.svg\")
![\"e_2^2=2^2=4\"](\"assets/equations/a4282b80c2e07509.svg\")
Step 3: add squares and divide by two observations.
","![\"\\mathrm{MSE}=\\frac{1+4}{2}=2.5\"](\"assets/equations/9258cd0363de1324.svg\")
Step 4: take the absolute value of each original error.
","![\"|-1|=1,\\qquad](\"assets/equations/5a8f4cb66d218e6f.svg\")
Step 5: add absolute errors and divide by two.
","![\"\\mathrm{MAE}=\\frac{1+2}{2}=1.5\"](\"assets/equations/2d85cc8a7db473e8.svg\")
Final answer: MSE = 2.5; MAE = 1.5. Squaring gives larger errors more influence; MAE is not differentiable at zero error.
","![\"L_{\\mathrm{BCE}}=-[y\\ln](\"assets/equations/9ec44fbf24f3156c.svg\")
For binary y = 1, BCE becomes −ln(p). For y = 0, it becomes −ln(1 − p). For one-hot multiclass labels, CCE is simply minus the natural log of the predicted probability assigned to the correct class.
","Question. True class is Mild, encoded [0,1,0], and probabilities are [0.1,0.8,0.1]. Find CCE.
","True class = Mild; one-hot target = [0,1,0].
","Predicted probabilities = [0.1,0.8,0.1].
","Find the categorical cross-entropy.
","Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.
","yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.
","![\"L=-\\sum_j](\"assets/equations/3b5ae8d980bc0fd4.svg\")
![\"L=-\\ln](\"assets/equations/96bd39e07641cf89.svg\")
Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.
","Step 1: write one weighted log term for each class.
","![\"L=-[0\\ln(0.1)+1\\ln(0.8)+0\\ln(0.1)]\"](\"assets/equations/6cc766252350138b.svg\")
Step 2: remove terms with zero target weight.
","![\"L=-\\ln(0.8)\"](\"assets/equations/17a83c5bb645a7c5.svg\")
Step 3: evaluate the natural logarithm.
","![\"\\ln(0.8)\\approx-0.223143551\"](\"assets/equations/4d6b8e2a24a05d6f.svg\")
Step 4: apply the outer minus sign.
","![\"L\\approx-(-0.223143551)=0.223143551\"](\"assets/equations/b71a143e9ddc291e.svg\")
Final answer: CCE ≈ 0.223144. Only the correct-class probability contributes for this one-hot target.
","| Task | Typical output | Typical loss |
|---|---|---|
| Unbounded numerical regression | One linear output | MSE or MAE |
| Binary classification | One sigmoid output | Binary cross-entropy |
| Exactly one of K classes | K softmax outputs | Categorical cross-entropy |
| Several independent labels | K sigmoid outputs | Binary cross-entropy per label |
A continuous rating restricted to 0–10 can use a linear output with a stated range policy, or 10 × sigmoid with an appropriate regression loss. If ratings are treated as discrete ordered categories, modelling choices differ. State the interpretation instead of selecting a loss merely from the number 10.
","For n included scalar predictions, let eᵢ = ŷᵢ−yᵢ. Three frequently used objectives are:
","![\"\\mathrm{SSE}=\\sum_i](\"assets/equations/90823ff8e0467213.svg\")
Differentiating with respect to one prediction produces different factors:
","![\"\\frac{\\partial\\mathrm{SSE}}{\\partial\\hat](\"assets/equations/1bf294550797614e.svg\")
Use the exact objective in the question. The teacher's two-output backpropagation example uses half SSE. Do not automatically insert the MSE factor into that example.
","For a one-hot target, the likelihood of the observed class is the probability assigned to it. Maximizing that likelihood is equivalent to minimizing its negative logarithm. For independent training examples, multiplying probabilities becomes adding their logarithms. A low correct-class probability therefore produces a large loss.
","Let p = σ(z). First differentiate the loss with respect to p, holding y constant:
","![\"\\frac{\\partial](\"assets/equations/bc392cea6dae6dda.svg\")
Then multiply by the sigmoid derivative:
","![\"\\frac{\\partial](\"assets/equations/6d9c2d8649b6476d.svg\")
Thus the logit delta is p−y for this paired output and loss. With sigmoid and half squared error it is instead (p−y)p(1−p). Confusing these two cases changes every subsequent gradient.
","For a normalized target vector, the derivative of softmax component pᵢ with respect to logit zⱼ is pᵢ times (δᵢⱼ−pⱼ), where δᵢⱼ is 1 if i = j and 0 otherwise. Therefore:
","![\"\\frac{\\partial](\"assets/equations/add665e2f73a094d.svg\")
This result assumes the target components sum to 1. For a mean loss over a batch, include the corresponding averaging factor as well.
","Question. Predictions are [1, 3] and targets are [2, 3]. Find SSE, half SSE and MSE.
","Predictions = [1, 3].
","Targets = [2, 3].
","Number of scalar predictions n = 2.
","SSE, half SSE and MSE separately.
","All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention.
","ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers.
","Yaad rakhne ka cue: Square sab mein; half mein 2, mean mein n se divide.
","Step 1: calculate error 1.
","Step 2: square error 1.
","Step 3: calculate error 2.
","![\"e_2=3-3=0\"](\"assets/equations/168d50f2de11bd29.svg\")
Step 4: square error 2.
","![\"e_2^2=(0)^2=0\"](\"assets/equations/760944c761b54733.svg\")
Step 5: add squared errors.
","![\"\\mathrm{SSE}=1+0=1\"](\"assets/equations/9d7eadaf7257dae1.svg\")
Step 6: divide by 2 for half SSE.
","![\"L_{\\mathrm{half}}=\\frac{1}{2}=0.5\"](\"assets/equations/b6b038fb864ecaa7.svg\")
Step 7: divide by the number of predictions for MSE.
","![\"\\mathrm{MSE}=\\frac{1}{2}\\approx0.500000000\"](\"assets/equations/89fe10c4fdc1b2d9.svg\")
Final answer: SSE = 1; half SSE = 0.5; MSE ≈ 0.500000000.
","Question. Predictions are [2, 5, 1] and targets are [1, 3, 2]. Find SSE, half SSE and MSE.
","Predictions = [2, 5, 1].
","Targets = [1, 3, 2].
","Number of scalar predictions n = 3.
","SSE, half SSE and MSE separately.
","All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention.
","ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers.
","Yaad rakhne ka cue: Square sab mein; half mein 2, mean mein n se divide.
","Step 1: calculate error 1.
","![\"e_1=2-1=1\"](\"assets/equations/b0c87b502c9af9cc.svg\")
Step 2: square error 1.
","![\"e_1^2=(1)^2=1\"](\"assets/equations/527befb1696e696d.svg\")
Step 3: calculate error 2.
","![\"e_2=5-3=2\"](\"assets/equations/776a0d96be6de464.svg\")
Step 4: square error 2.
","![\"e_2^2=(2)^2=4\"](\"assets/equations/0c723f0acd2a3070.svg\")
Step 5: calculate error 3.
","![\"e_3=1-2=-1\"](\"assets/equations/b685cf877781a273.svg\")
Step 6: square error 3.
","![\"e_3^2=(-1)^2=1\"](\"assets/equations/1212dd80c80f5b96.svg\")
Step 7: add squared errors.
","![\"\\mathrm{SSE}=1+4+1=6\"](\"assets/equations/8252220326cba08a.svg\")
Step 8: divide by 2 for half SSE.
","![\"L_{\\mathrm{half}}=\\frac{6}{2}=3\"](\"assets/equations/5e41d17ab2e98c3d.svg\")
Step 9: divide by the number of predictions for MSE.
","![\"\\mathrm{MSE}=\\frac{6}{3}\\approx2.000000000\"](\"assets/equations/e4955c778fd95a4f.svg\")
Final answer: SSE = 6; half SSE = 3; MSE ≈ 2.000000000.
","Question. Predictions are [4] and targets are [1]. Find SSE, half SSE and MSE.
","Predictions = [4].
","Targets = [1].
","Number of scalar predictions n = 1.
","SSE, half SSE and MSE separately.
","All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention.
","ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers.
","Yaad rakhne ka cue: Square sab mein; half mein 2, mean mein n se divide.
","Step 1: calculate error 1.
","![\"e_1=4-1=3\"](\"assets/equations/35c4c4ba3ef807a6.svg\")
Step 2: square error 1.
","![\"e_1^2=(3)^2=9\"](\"assets/equations/1c1774a6ed0f013e.svg\")
Step 3: add squared errors.
","![\"\\mathrm{SSE}=9=9\"](\"assets/equations/3acb15eecc0c073c.svg\")
Step 4: divide by 2 for half SSE.
","![\"L_{\\mathrm{half}}=\\frac{9}{2}=4.5\"](\"assets/equations/6c789702092108e1.svg\")
Step 5: divide by the number of predictions for MSE.
","![\"\\mathrm{MSE}=\\frac{9}{1}\\approx9.000000000\"](\"assets/equations/e1fcd30496290844.svg\")
Final answer: SSE = 9; half SSE = 4.5; MSE ≈ 9.000000000.
","Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.
","Target y = 1; p = P(y=1) = 0.8; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","![\"L=-[y\\ln](\"assets/equations/e55d3d5e343976d3.svg\")
Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","![\"p=0.8\"](\"assets/equations/fcb072c6f4519069.svg\")
Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","![\"L=-(-0.223143551)=0.223143551\"](\"assets/equations/65af868a518fd08e.svg\")
Final answer: Loss ≈ 0.223143551 nats.
","Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.
","Target y = 0; p = P(y=1) = 0.8; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","![\"1-p=1-0.8=0.2\"](\"assets/equations/438398f064595bb7.svg\")
Step 2: evaluate its natural logarithm.
","![\"\\ln(0.2)\\approx-1.609437912\"](\"assets/equations/a82bb66e18d1bf7c.svg\")
Step 3: apply the minus sign outside the logarithm.
","![\"L=-(-1.609437912)=1.609437912\"](\"assets/equations/fe3e4a4ff07bc0e0.svg\")
Final answer: Loss ≈ 1.609437912 nats.
","Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.
","Target y = 1; p = P(y=1) = 0.5; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","![\"p=0.5\"](\"assets/equations/8c8d4b00d7d4788e.svg\")
Step 2: evaluate its natural logarithm.
","![\"\\ln(0.5)\\approx-0.693147181\"](\"assets/equations/89c211e38f586f86.svg\")
Step 3: apply the minus sign outside the logarithm.
","![\"L=-(-0.693147181)=0.693147181\"](\"assets/equations/e0e8fd22135c9d7b.svg\")
Final answer: Loss ≈ 0.693147181 nats.
","Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.
","p = 0.8; y = 1; single-example BCE; p = sigmoid(z).
","dL/dz, not dL/dp.
","Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.
","p is an activated probability; z is its raw score; the requested derivative is with respect to z.
","![\"\\frac{dL}{dp}=\\frac{p-y}{p(1-p)}\"](\"assets/equations/6db0182710f5e684.svg\")
![\"\\frac{dp}{dz}=p(1-p)\"](\"assets/equations/0c0df88360e53bd3.svg\")
![\"\\frac{dL}{dz}=p-y\"](\"assets/equations/2e2f1b40461ded43.svg\")
Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.
","Step 1: calculate the sigmoid slope.
","![\"p(1-p)=0.8(0.2)=0.16\"](\"assets/equations/e1d74c616a26b88e.svg\")
Step 2: differentiate BCE with respect to p.
","![\"\\frac{dL}{dp}=\\frac{0.8-1}{0.16}=-1.25\"](\"assets/equations/55554414edc61afc.svg\")
Step 3: multiply the two local derivatives.
","![\"\\frac{dL}{dz}=(-1.25)(0.16)=-0.2\"](\"assets/equations/352e50cd272ef421.svg\")
Final answer: dL/dz = -0.2. No further sigmoid derivative is needed.
","Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.
","p = 0.8; y = 0; single-example BCE; p = sigmoid(z).
","dL/dz, not dL/dp.
","Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.
","p is an activated probability; z is its raw score; the requested derivative is with respect to z.
","Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.
","Step 1: calculate the sigmoid slope.
","Step 2: differentiate BCE with respect to p.
","![\"\\frac{dL}{dp}=\\frac{0.8-0}{0.16}=5\"](\"assets/equations/7dcb217487ddffdd.svg\")
Step 3: multiply the two local derivatives.
","![\"\\frac{dL}{dz}=(5)(0.16)=0.8\"](\"assets/equations/86de3ab6d9deb754.svg\")
Final answer: dL/dz = 0.8. No further sigmoid derivative is needed.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s9","search":"Choosing a loss **Definition.** A loss function quantifies the discrepancy between a prediction and its target. The training objective aggregates per-example losses and may add regularization. The loss must be compatible with the task and output parameterization.\n\n### Explanation and working\nActivation produces the prediction. Loss measures how wrong that prediction is. An optimizer uses gradients to update parameters. These are three different roles.\n\n\n\n\n**Question.** Targets are [2, 4], predictions are [1, 6]. Find MSE and MAE.\n\n\n\n### Given\n\nTargets = [2,4]; predictions = [1,6]; n = 2.\n\n### To find\n\nFind mean squared error and mean absolute error.\n\n### Formula — which one and why\n\n**Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.\n\nŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean.\n\n**Mean absolute error.** Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes.\n\nn = number of predictions. |e| means the nonnegative magnitude of error e.\n\n\n\n**Yaad rakhne ka cue:** MAE mein magnitude; MSE mein square.\n\n### Solution — step by step\n\n\n\n\n**Step 1: calculate prediction minus target.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 2: square both errors.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 3: add squares and divide by two observations.**\n\n\n\n\n\n\n\n**Step 4: take the absolute value of each original error.**\n\n\n\n\n\n\n\n**Step 5: add absolute errors and divide by two.**\n\n\n\n\n\n\n\n**Final answer:** MSE = 2.5; MAE = 1.5. Squaring gives larger errors more influence; MAE is not differentiable at zero error.\n\nFor binary y = 1, BCE becomes −ln(p). For y = 0, it becomes −ln(1 − p). For one-hot multiclass labels, CCE is simply minus the natural log of the predicted probability assigned to the correct class.\n**Question.** True class is Mild, encoded [0,1,0], and probabilities are [0.1,0.8,0.1]. Find CCE.\n\n\n\n### Given\n\nTrue class = Mild; one-hot target = [0,1,0].\n\nPredicted probabilities = [0.1,0.8,0.1].\n\n### To find\n\nFind the categorical cross-entropy.\n\n### Formula — which one and why\n\n**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.\n\nyⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.\n\n\n\n\n\n**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.\n\n### Solution — step by step\n\n\n\n\n**Step 1: write one weighted log term for each class.**\n\n\n\n\n\n\n\n**Step 2: remove terms with zero target weight.**\n\n\n\n\n\n\n\n**Step 3: evaluate the natural logarithm.**\n\n\n\n\n\n\n\n**Step 4: apply the outer minus sign.**\n\n\n\n\n\n\n\n**Final answer:** CCE ≈ 0.223144. Only the correct-class probability contributes for this one-hot target.\n| Task | Typical output | Typical loss |\n| Unbounded numerical regression | One linear output | MSE or MAE |\n| Binary classification | One sigmoid output | Binary cross-entropy |\n| Exactly one of K classes | K softmax outputs | Categorical cross-entropy |\n| Several independent labels | K sigmoid outputs | Binary cross-entropy per label |\nA continuous rating restricted to 0–10 can us"},{"title":"Forward propagation through multiple neurons","blocks":["Definitions. A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations.
","The input layer supplies the feature vector. A hidden layer learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The output layer produces the prediction in the form needed by the task.
","In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections.
","Let a⁽⁰⁾ = x. For layer l:
","![\"z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)}\"](\"assets/equations/87472012ebbb4a9d.svg\")
![\"a^{(l)}=f^{(l)}(z^{(l)})\"](\"assets/equations/da3b4bdac78b839b.svg\")
If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j.
","![\"W^{(l)}\\in\\mathbb](\"assets/equations/bb1ec5bee5b7117c.svg\")
For one hidden layer, these computations compose to give:
","![\"h=f^{(1)}(W^{(1)}x+b^{(1)})\"](\"assets/equations/4ff7d7b3e6546d64.svg\")
![\"\\hat](\"assets/equations/fb13b3127fe27394.svg\")
The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer.
","Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture.
","Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = ![\"\\begin{bmatrix}0.2](\"assets/equations/59303f63a5ad2b9b.svg\")
and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.
A feedforward network receives x = [1, 0]ᵀ.
","Its hidden-layer weights are W¹ = and biases are b¹ = [0.1, 0.2]ᵀ.
The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1.
","Use ReLU in both layers and one row per receiving neuron.
","Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","![\"z_h=W_1x+b_1\"](\"assets/equations/3929c1e524affa01.svg\")
![\"h=f_1(z_h)\"](\"assets/equations/5a067fbd2b571e3c.svg\")
![\"z_o=W_2h+b_2\"](\"assets/equations/02e0daad7201e074.svg\")
![\"o=f_2(z_o)\"](\"assets/equations/0badfa3430a92383.svg\")
Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","![\"\\operatorname{ReLU}(z)=\\max(0,z)\"](\"assets/equations/67fab45c729a152a.svg\")
![\"\\operatorname{LeakyReLU}(z)=\\begin{cases}z&z\\ge0\\\\\\alpha](\"assets/equations/4edc237fa121ff9b.svg\")
Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Step 1: identify the matrix dimensions. W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output.
","Step 2: calculate the first hidden preactivation. Multiply the first row by x, then add the first hidden bias.
","![\"z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3\"](\"assets/equations/2f1a7c307bfbc37e.svg\")
Step 3: calculate the second hidden preactivation.
","![\"z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5\"](\"assets/equations/ee1bd9d85fa15d84.svg\")
Step 4: activate the hidden layer. Both values are positive, so ReLU retains them.
","![\"a^{(1)}_1=\\max(0,0.3)=0.3\"](\"assets/equations/0b16aa9984127202.svg\")
![\"a^{(1)}_2=\\max(0,0.5)=0.5\"](\"assets/equations/6263aed38871ad59.svg\")
![\"a^{(1)}=\\begin{bmatrix}0.3\\\\0.5\\end{bmatrix}\"](\"assets/equations/368b9c59527b8200.svg\")
Step 5: calculate the output preactivation. Use the activated hidden values as this layer's inputs.
","![\"z^{(2)}=0.5(0.3)+0.6(0.5)+0.1\"](\"assets/equations/f3fff5a6e4a523e4.svg\")
![\"z^{(2)}=0.15+0.30+0.10=0.55\"](\"assets/equations/2fda741950753a37.svg\")
Step 6: apply the output activation.
","![\"\\hat](\"assets/equations/79fc763b55540917.svg\")
Final answer: hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s10","search":"Forward propagation through multiple neurons **Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations.\n\n### Input, hidden and output layers\nThe **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task.\n\nIn a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections.\n\n\n\n### General layer equations and dimensions\nLet a⁽⁰⁾ = x. For layer l:\n\n\nIf the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j.\n\nFor one hidden layer, these computations compose to give:\n\n\nThe hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer.\n\n### Forward propagation versus training\nForward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture.\n\n### Complete two-layer numerical\n**Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.**\n\n\n\n### Given\n\nA feedforward network receives x = [1, 0]ᵀ.\n\nIts hidden-layer weights are W¹ = and biases are b¹ = [0.1, 0.2]ᵀ.\n\nThe output weights are W² = [0.5, 0.6] and output bias is b² = 0.1.\n\nUse ReLU in both layers and one row per receiving neuron.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota"},{"title":"Backpropagation with one hidden neuron","blocks":["Definition. Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters.
","Consider one input, one hidden neuron and one output neuron:
","![\"z_h=w_1x+b_1,\\qquad](\"assets/equations/3f2141c2576fb066.svg\")
![\"z_o=w_2h+b_2,\\qquad\\hat](\"assets/equations/0b90638b5849c176.svg\")
For the half squared loss used in this example:
","Define a neuron's delta as the derivative of the loss with respect to its preactivation, not with respect to its output:
","![\"\\delta_o=\\frac{\\partial](\"assets/equations/6215ea243129f507.svg\")
The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂:
","![\"\\frac{\\partial](\"assets/equations/338a3082f0298cb5.svg\")
To reach the hidden neuron, continue backwards through the old output weight and the hidden activation:
","![\"\\delta_h=\\delta_o](\"assets/equations/6409774c2652b14e.svg\")
![\"\\frac{\\partial](\"assets/equations/675e596602704d40.svg\")
The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass.
","Question. x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1.
","x = 2, y = 1.
","w₁ = 0.5, b₁ = 0.1.
","w₂ = 0.8, b₂ = 0.2.
","Both neurons use ReLU.
","Use half squared loss.
","Find the output and all four parameter gradients, then update all four parameters using η = 0.1.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","![\"\\frac{\\partial](\"assets/equations/8faacff5464dcaf1.svg\")
Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","![\"\\delta_{o_j}=\\frac{\\partial](\"assets/equations/bea7700a623139b7.svg\")
![\"\\delta_{h_i}=f_h\\prime(z_{h_i})\\sum_jW_{ji}^{(2)}\\delta_{o_j}\"](\"assets/equations/ea9470bc5f0a15d2.svg\")
![\"\\frac{\\partial](\"assets/equations/b40acc83c437110d.svg\")
![\"\\frac{\\partial](\"assets/equations/b9f5adb924b1f09e.svg\")
![\"W_{ji,\\mathrm{new}}=W_{ji}-\\eta\\frac{\\partial](\"assets/equations/2a87ae98981ac797.svg\")
Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1: forward propagation through the hidden neuron.
","![\"z_h=0.5(2)+0.1=1.1\"](\"assets/equations/5613b3868c5f8d85.svg\")
![\"h=\\operatorname{ReLU}(1.1)=1.1\"](\"assets/equations/5206a5b2ac9abfe4.svg\")
Step 2: forward propagation through the output neuron.
","![\"z_o=0.8(1.1)+0.2=1.08\"](\"assets/equations/5a7535982ea7d2cd.svg\")
![\"\\hat](\"assets/equations/8a3b4bb2871113ef.svg\")
ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here.
","Step 3: calculate the loss and output delta.
","![\"L=\\frac12(1.08-1)^2=0.0032\"](\"assets/equations/e51ef56449f25818.svg\")
Both preactivations are positive, so both ReLU derivatives are 1.
","![\"\\delta_o=(1.08-1)(1)=0.08\"](\"assets/equations/9ead62e2abd6d370.svg\")
Step 4: calculate the output-layer parameter gradients.
","![\"\\frac{\\partial](\"assets/equations/20d536b8c74dbceb.svg\")
![\"\\frac{\\partial](\"assets/equations/57f3bbde46d88ee8.svg\")
Step 5: propagate the derivative to the hidden neuron.
","Use w₂ = 0.8, its value during the forward pass:
","![\"\\delta_h=0.08(0.8)(1)=0.064\"](\"assets/equations/7cf97525165cb969.svg\")
![\"\\frac{\\partial](\"assets/equations/a31fd300551ec8e1.svg\")
![\"\\frac{\\partial](\"assets/equations/fc73ac9e8421c88b.svg\")
Step 6: update each parameter using its own gradient.
","![\"w_{1,\\mathrm{new}}=0.5-0.1(0.128)=0.4872\"](\"assets/equations/6a6ad843a78157c3.svg\")
![\"b_{1,\\mathrm{new}}=0.1-0.1(0.064)=0.0936\"](\"assets/equations/d766af9fb7ee76f0.svg\")
![\"w_{2,\\mathrm{new}}=0.8-0.1(0.088)=0.7912\"](\"assets/equations/c2a2db66205fe592.svg\")
![\"b_{2,\\mathrm{new}}=0.2-0.1(0.08)=0.192\"](\"assets/equations/4678bc376cf0d02b.svg\")
| Parameter | Gradient from the original forward pass | Updated value |
|---|---|---|
| w₁ | 0.128 | 0.4872 |
| b₁ | 0.064 | 0.0936 |
| w₂ | 0.088 | 0.7912 |
| b₂ | 0.08 | 0.192 |
Final answer: the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s11","search":"Backpropagation with one hidden neuron \n**Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters.\n\n### Network equations and the meaning of delta\nConsider one input, one hidden neuron and one output neuron:\n\n\nFor the half squared loss used in this example:\n\nDefine a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output:\n\nThe output-weight derivative contains the hidden activation because zₒ = w₂h + b₂:\n\nTo reach the hidden neuron, continue backwards through the old output weight and the hidden activation:\n\n\nThe bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass.\n\n### Complete worked numerical\n**Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1.\n\n\n\n### Given\n\nx = 2, y = 1.\n\nw₁ = 0.5, b₁ = 0.1.\n\nw₂ = 0.8, b₂ = 0.2.\n\nBoth neurons use ReLU.\n\nUse half squared loss.\n\nFind the output and all four parameter gradients, then update all four parameters using η = 0.1.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n\n\n**Step 1: forward propagation through the hidden neuron.**\n\n\n\n**Step 2: forward propagation through the output neuron.**\n\n\nReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here.\n\n**Step 3: calculate the loss and output delta.**\n\nBoth preactivations are positive, so bot"},{"title":"More outputs and vectorized backpropagation","blocks":["Definition. Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path.
","If hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives:
","![\"\\frac{\\partial](\"assets/equations/bf829204c0f41f49.svg\")
For any number of connected outputs, sum their contributions and then differentiate the hidden activation:
","![\"\\delta_{h_j}=\\left(\\sum_k](\"assets/equations/f94e1705f6027563.svg\")
Use the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss.
","Use column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes half sum of squared errors and an elementwise output activation f₂:
","![\"L=\\frac12\\sum_{j=1}^{k}(\\hat](\"assets/equations/a3370e1f71500a0e.svg\")
![\"\\delta_2=(\\hat](\"assets/equations/07eda20c3b7d69f3.svg\")
![\"\\frac{\\partial](\"assets/equations/7401adc017e3686d.svg\")
![\"\\delta_1=(W_2^T\\delta_2)\\odot](\"assets/equations/b9b8719d9f5e5e0f.svg\")
![\"\\frac{\\partial](\"assets/equations/88df80b8a0e03820.svg\")
Here ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an outer product: every receiving delta is multiplied by every sending activation.
","| Quantity | Shape | Reason |
|---|---|---|
| W₁ and its gradient | m × d | m receiving hidden neurons, d inputs |
| W₂ and its gradient | k × m | k receiving output neurons, m hidden activations |
| δ₂ | k × 1 | One derivative per output preactivation |
| W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit |
| δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ |
For MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target:
","![\"\\delta_{\\mathrm{output}}=\\hat](\"assets/equations/0d69b7fe4a0f11cd.svg\")
That simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required.
","For a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s12","search":"More outputs and vectorized backpropagation \n**Definition.** Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path.\n\n### Why the hidden derivative is a sum\nIf hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives:\n\nFor any number of connected outputs, sum their contributions and then differentiate the hidden activation:\n\nUse the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss.\n\n\n\n### Matrix equations for one training sample\nUse column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes **half sum of squared errors** and an elementwise output activation f₂:\n\n\n\n\n\nHere ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an **outer product**: every receiving delta is multiplied by every sending activation.\n\n| Quantity | Shape | Reason |\n| W₁ and its gradient | m × d | m receiving hidden neurons, d inputs |\n| W₂ and its gradient | k × m | k receiving output neurons, m hidden activations |\n| δ₂ | k × 1 | One derivative per output preactivation |\n| W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit |\n| δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ |\n\n### How the loss changes the output delta\nFor MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target:\n\nThat simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required.\n\nFor a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question.\n\n"},{"title":"Model answer: forward propagation, backpropagation and updating","question":["Question. Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations.
"],"answer":["Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs.
","| Operation | Direction or action | Result |
|---|---|---|
| Forward propagation | Inputs move through successive layers | Predictions and intermediate activations |
| Loss calculation | Compare predictions with targets | A scalar objective for the example or batch |
| Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters |
| Optimizer update | Use those gradients to change parameters | A new set of weights and biases |
At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next.
","![\"z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\\qquad](\"assets/equations/2845b5af7a9f9056.svg\")
Use the loss specified by the task. For one regression output, half squared loss is one possible choice:
","![\"L=\\frac12(a-y)^2\"](\"assets/equations/b45e01da7c114be3.svg\")
Here a is the prediction and y is the target. Other stated losses produce different derivatives.
","Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation.
","![\"\\frac{\\partial](\"assets/equations/c4e0f7ded9389700.svg\")
Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it.
","![\"w_{\\mathrm{new}}=w_{\\mathrm{old}}-\\eta\\frac{\\partial](\"assets/equations/116cb3cc8bb58606.svg\")
Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations.
"],"kind":"theory","id":"mt2-c4-s13","search":"Model answer: forward propagation, backpropagation and updating **Question.** Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations.\n\n**Model answer.**\nTraining repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs.\n\n\n\n\n| Operation | Direction or action | Result |\n| Forward propagation | Inputs move through successive layers | Predictions and intermediate activations |\n| Loss calculation | Compare predictions with targets | A scalar objective for the example or batch |\n| Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters |\n| Optimizer update | Use those gradients to change parameters | A new set of weights and biases |\n\n### 1. Forward propagation\nAt each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next.\n\n### 2. Loss\nUse the loss specified by the task. For one regression output, half squared loss is one possible choice:\n\nHere a is the prediction and y is the target. Other stated losses produce different derivatives.\n### 3. Backpropagation\nMultiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation.\n\nHere δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it.\n### 4. Update and repeat\n\nCompute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations.\n"},{"title":"Counting parameters in a dense network","blocks":["Definition. A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases.
","For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters:
","![\"P_{\\mathrm{dense}}=mn+n=(m+1)n\"](\"assets/equations/e846c0a1de0fb579.svg\")
Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram.
","Question. A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters.
","Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs.
","Every hidden/output neuron has one bias.
","Count weights and biases at each receiving layer, then the total.
","Dense-layer parameters. Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.
","n_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.
","![\"P=(n_{\\mathrm{in}}+1)n_{\\mathrm{out}}\"](\"assets/equations/494312d0893d20d2.svg\")
Yaad rakhne ka cue: Har output ke liye saare inputs plus ek bias.
","Step 1: identify the consecutive layer widths.
","![\"10\\longrightarrow5\\longrightarrow5\\longrightarrow5\\longrightarrow4\"](\"assets/equations/68fe56c651575649.svg\")
There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters.
","Step 2: count weights and biases for each transformation.
","| Transformation | Weights | Biases | Layer total |
|---|---|---|---|
| Input → hidden 1 | 10 × 5 = 50 | 5 | 55 |
| Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 |
| Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 |
| Hidden 3 → output | 5 × 4 = 20 | 4 | 24 |
Step 3: add the layer totals.
","![\"P=55+30+30+24=139\"](\"assets/equations/1c0d23dc90da3211.svg\")
Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139.
","Final answer: 139 trainable parameters. If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s14","search":"Counting parameters in a dense network \n**Definition.** A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases.\n\n### Deriving the parameter count\nFor m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters:\n\nActivation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram.\n\n\n\n### Complete numerical\n**Question.** A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters.\n\n\n\n### Given\n\nLayer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs.\n\nEvery hidden/output neuron has one bias.\n\n### To find\n\nCount weights and biases at each receiving layer, then the total.\n\n### Formula — which one and why\n\n**Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.\n\nn_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.\n\n\n\n**Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: identify the consecutive layer widths.**\n\nThere are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters.\n\n**Step 2: count weights and biases for each transformation.**\n| Transformation | Weights | Biases | Layer total |\n| Input → hidden 1 | 10 × 5 = 50 | 5 | 55 |\n| Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 |\n| Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 |\n| Hidden 3 → output | 5 × 4 = 20 | 4 | 24 |\n\n**Step 3: add the layer totals.**\n\nEquivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139.\n\n**Final answer: 139 trainable parameters.** If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network.\n\n"},{"title":"What you should understand before optimization","blocks":["You should be able to explain why a target and prediction differ, compute a weighted sum, choose an activation, measure a loss, find a one-neuron gradient and update a weight. If these steps are weak, repeat the first training example and the ReLU backprop example. They contain the entire calculation pattern that the next chapters extend.
","Question. A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update.
","x = 2; w = 0; b = 0; y = 1.
","Sigmoid output; half squared loss; η = 0.1.
","Find the weight-gradient sign, exact weight update and resulting weight.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","![\"p=\\frac{1}{1+e^{-z}}\"](\"assets/equations/5dd16ba0b4d79641.svg\")
Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 forward: net = 0(2) + 0 = 0; sigmoid output = 0.5.
","Step 2 backward: prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125.
","Step 3 gradients and update: weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s16","search":"Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update.\n\n\n\n### Given\n\nx = 2; w = 0; b = 0; y = 1.\n\nSigmoid output; half squared loss; η = 0.1.\n\n### To find\n\nFind the weight-gradient sign, exact weight update and resulting weight.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5.\n\n\n\n**Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125.\n**Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree.\n"},{"title":"Cross-entropy for probability distributions","blocks":["Definition. For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities.
","![\"H(p,q)=-\\sum_{i=1}^{K}p_i\\log](\"assets/equations/ca68a6ddd1ac17cc.svg\")
Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms.
","Natural logarithms produce nats; base-2 logarithms produce bits. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2:
","![\"\\log_2](\"assets/equations/85b0a11172f30b61.svg\")
For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities.
","Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.
","A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5].
","Calculate cross-entropy using natural logarithms and using base-2 logarithms.
","Find cross-entropy in the requested units.
","Cross-entropy between distributions. Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute.
","pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits.
","![\"H(p,q)=-\\sum_i](\"assets/equations/feb21e9c89a912d7.svg\")
![\"H_2(p,q)=\\frac{H(p,q)}{\\ln2}\"](\"assets/equations/f4684a5e966eb42a.svg\")
Yaad rakhne ka cue: Target probability batati hai kis log-loss ka kitna hissa lena hai.
","Step 1: write the two terms. Both target entries are nonzero, so neither term can be discarded.
","![\"H(p,q)=-[0.75\\ln(0.5)+0.25\\ln(0.5)]\"](\"assets/equations/fcc9ac503722bcd6.svg\")
Step 2: calculate the logarithms.
","Step 3: weight each negative logarithm by its target probability.
","![\"-0.75\\ln(0.5)\\approx0.519860385\"](\"assets/equations/ba03d4ba50c7c4e4.svg\")
![\"-0.25\\ln(0.5)\\approx0.173286795\"](\"assets/equations/7dbaf11af5ca2bb7.svg\")
Step 4: add the contributions, retaining calculator precision.
","![\"H(p,q)\\approx0.519860385+0.173286795\\approx0.693147181\\](\"assets/equations/77f2dae5c0c0431a.svg\")
Step 5: repeat with base 2. Since 2⁻¹ = 0.5, log₂(0.5) = −1.
","![\"H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\\](\"assets/equations/cf0bfa788dbc4b07.svg\")
Final answer: 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale.
","These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio.
","![\"H(p)=-\\sum_i](\"assets/equations/4ac39baa15925718.svg\")
![\"D_{\\mathrm{KL}}(p\\Vert](\"assets/equations/8933903c2570cd47.svg\")
For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus.
","Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.
","For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.
","Find entropy and KL divergence in nats.
","Entropy and relative entropy. Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p.
","p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence.
","![\"D_{\\mathrm{KL}}(p\\Vert](\"assets/equations/1dc0b474b12a2bf8.svg\")
Yaad rakhne ka cue: KL = cross-entropy minus target ki own entropy.
","Step 1: calculate the target entropy.
","![\"H(p)=-[0.75\\ln(0.75)+0.25\\ln(0.25)]\"](\"assets/equations/78338077753f61a2.svg\")
![\"\\ln(0.75)\\approx-0.287682072,\\qquad\\ln(0.25)\\approx-1.386294361\"](\"assets/equations/c8cc7ed1293c1673.svg\")
![\"H(p)\\approx0.215761554+0.346573590=0.562335145\"](\"assets/equations/b8c34bbe332a549f.svg\")
Step 2: subtract target entropy from cross-entropy.
","![\"D_{\\mathrm{KL}}(p\\Vert](\"assets/equations/6a763ffee09c5ad1.svg\")
Final answer: H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c4-s17","search":"Cross-entropy for probability distributions **Definition.** For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities.\n\nHere pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms.\n\n### Logarithm base and units\nNatural logarithms produce **nats**; base-2 logarithms produce **bits**. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2:\n\nFor a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities.\n\n### A target distribution with two nonzero entries\n**Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.**\n\n\n\n### Given\n\nA two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5].\n\nCalculate cross-entropy using natural logarithms and using base-2 logarithms.\n\n### To find\n\nFind cross-entropy in the requested units.\n\n### Formula — which one and why\n\n**Cross-entropy between distributions.** Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute.\n\npᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits.\n\n\n\n\n\n**Yaad rakhne ka cue:** Target probability batati hai kis log-loss ka kitna hissa lena hai.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: write the two terms.** Both target entries are nonzero, so neither term can be discarded.\n\n\n**Step 2: calculate the logarithms.**\n\n\n**Step 3: weight each negative logarithm by its target probability.**\n\n\n\n**Step 4: add the contributions, retaining calculator precision.**\n\n\n**Step 5: repeat with base 2.** Since 2⁻¹ = 0.5, log₂(0.5) = −1.\n\n**Final answer:** 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale.\n\n### Cross-entropy, entropy and relative entropy\nThese names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio.\n\n\nFor fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus.\n\n**Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.**\n\n\n\n### Given\n\nFor the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.\n\n### To find\n\nFind entropy and KL divergence in nats.\n\n### Formula — which one and why\n\n**Entropy and relative entropy.** Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p.\n\np = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence.\n\n\n\n\n\n**Yaad rakhne ka cue:** KL"}]},{"n":5,"title":"Optimization algorithms","sections":[{"title":"Reading an optimizer formula without getting lost","blocks":["An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated.
","At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position.
","![\"\\theta_t=\\theta_{t-1}-\\eta](\"assets/equations/6df9cee9010e92ee.svg\")
η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition.
","An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares.
","If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square.
","![\"0.9\\times4=3.6,\\qquad(1-0.9)\\times9=0.9\"](\"assets/equations/0d4d3672b448c16a.svg\")
![\"s_{\\mathrm{new}}=3.6+0.9=4.5\"](\"assets/equations/4044336c67492dab.svg\")
The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum.
","For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g².
","ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s0","search":"Reading an optimizer formula without getting lost An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated.\n\n\n\n### Old, current and new\nAt update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position.\n\nη, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition.\n### What a stored state means\nAn optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares.\n### A weighted average from ordinary arithmetic\nIf the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square.\n\n\nThe coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum.\n### Squared memory and signed direction\nFor gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g².\n### Epsilon and hats\nε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second.\n"},{"title":"Optimizer notation: old value, memory, new value","blocks":["θ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries.
","At time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values.
","If the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead.
","For θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64.
","Both 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square.
","If s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero.
","Write one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1.
","The worked examples below use this exact order, with every state shown.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s1","search":"Optimizer notation: old value, memory, new value ### 1. A parameter and a stored state are different\nθ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries.\n\n\n\nAt time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values.\n\n### 2. A supplied gradient versus a supplied objective\nIf the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead.\nFor θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64.\n\n### 3. A squared gradient keeps size and loses sign\nBoth 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square.\n\n### 4. Square root and epsilon belong to the denominator\nIf s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero.\n\n### 5. A table prevents mixing iterations\nWrite one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1.\nThe worked examples below use this exact order, with every state shown.\n"},{"title":"What an optimizer is trying to do","blocks":["Definition. Optimization is the process of selecting model parameters to minimize a scalar objective. In neural-network training, a gradient-based optimizer updates weights and biases using derivatives of the training objective.
","For empirical risk with an optional regularizer:
","![\"J(\\theta)=\\frac1N\\sum_{i=1}^{N}L_i(\\theta)+\\lambda\\Omega(\\theta)\"](\"assets/equations/e8f2cd5bb532bd92.svg\")
The loss value indicates the current objective. Its gradient indicates local sensitivity to parameters. Backpropagation computes this gradient efficiently; the optimizer determines how the parameters are updated from it. A gradient and a loss value cannot be interchanged in the update equation.
","For a small displacement Δθ, first-order Taylor expansion approximates the objective by:
","Choose Δθ = −η∇J(θ), with η > 0. Substitution gives:
","![\"J(\\theta-\\eta\\nabla](\"assets/equations/f0fe0e6b83fb6f66.svg\")
The squared norm is nonnegative. Therefore the negative-gradient direction is locally a descent direction when the gradient is nonzero. The approximation explains the sign; it does not guarantee that an arbitrarily large finite step decreases loss.
","A local minimum has objective value no larger than nearby points. A global minimum is minimal over the whole feasible domain. At a saddle point, the objective increases in some directions and decreases in others. A small gradient alone does not establish a minimum: a stationary point can also be a maximum or saddle.
","Neural-network objectives are generally nonconvex. Training can encounter flat regions, ill-conditioned valleys, noisy gradient estimates and poor step sizes. Momentum modifies the update direction using previous steps; adaptive methods rescale parameter-specific updates. These methods improve particular optimization behavior, without guaranteeing the global optimum or best test performance.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s2","search":"What an optimizer is trying to do **Definition.** Optimization is the process of selecting model parameters to minimize a scalar objective. In neural-network training, a gradient-based optimizer updates weights and biases using derivatives of the training objective.\n\n### Objective and gradient\nFor empirical risk with an optional regularizer:\n\nThe loss value indicates the current objective. Its gradient indicates local sensitivity to parameters. Backpropagation computes this gradient efficiently; the optimizer determines how the parameters are updated from it. A gradient and a loss value cannot be interchanged in the update equation.\n\n### Why gradient descent subtracts the gradient\nFor a small displacement Δθ, first-order Taylor expansion approximates the objective by:\n\nChoose Δθ = −η∇J(θ), with η > 0. Substitution gives:\n\nThe squared norm is nonnegative. Therefore the negative-gradient direction is locally a descent direction when the gradient is nonzero. The approximation explains the sign; it does not guarantee that an arbitrarily large finite step decreases loss.\n\n\n\n### Local minima, global minima and saddle points\nA local minimum has objective value no larger than nearby points. A global minimum is minimal over the whole feasible domain. At a saddle point, the objective increases in some directions and decreases in others. A small gradient alone does not establish a minimum: a stationary point can also be a maximum or saddle.\n\nNeural-network objectives are generally nonconvex. Training can encounter flat regions, ill-conditioned valleys, noisy gradient estimates and poor step sizes. Momentum modifies the update direction using previous steps; adaptive methods rescale parameter-specific updates. These methods improve particular optimization behavior, without guaranteeing the global optimum or best test performance.\n"},{"title":"The ordinary gradient descent update","blocks":["Definition. Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate.
","![\"g_t=J\\prime(\\theta_{t-1})\"](\"assets/equations/52ef83d9d4ea4381.svg\")
Here update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent.
","Read it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction.
","Question. J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations.
","J(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1.
","Differentiate J and perform two gradient-descent iterations.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","Step 1. Differentiate. g(θ) = 0.7 × 3 × θ² = 2.1 θ².
","Step 2. First update. g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79.
","Step 3. Second gradient. Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061.
","Step 4. Second update. θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = 0.658939.
","The paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum.
","Common error: using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s3","search":"The ordinary gradient descent update **Definition.** Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate.\n\n### Explanation and working\n\n\nHere update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent.\nRead it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction.\n\n\n\n**Question.** J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations.\n\n\n\n### Given\n\nJ(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1.\n\n### To find\n\nDifferentiate J and perform two gradient-descent iterations.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1. Differentiate.** g(θ) = 0.7 × 3 × θ² = 2.1 θ².\n**Step 2. First update.** g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79.\n**Step 3. Second gradient.** Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061.\n**Step 4. Second update.** θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = **0.658939**.\nThe paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum.\n**Common error: ** using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients."},{"title":"Batch stochastic and mini batch gradient descent","blocks":["Definition. Batch, stochastic and mini-batch gradient descent differ in the subset of training examples used to estimate the gradient for each parameter update. Their update equation is the same; their gradient estimates and update frequency differ.
","All three use the same update idea. They differ in how much data contributes to one gradient.
","| Method | Data for one update | Updates in one epoch with N samples | Typical behaviour |
|---|---|---|---|
| Batch GD | All N samples | 1 | Stable full-data gradient; costly updates |
| Stochastic GD | One sample | N | Frequent noisy updates |
| Mini-batch GD | B samples | Ceiling of N/B if final partial batch is retained | Efficient parallel calculation with moderate noise |
Question. N = 1,000 and B = 100. Find updates per epoch.
","N = 1,000 examples; mini-batch size B = 100; one epoch.
","Count full-batch, stochastic and mini-batch updates per epoch.
","Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.
","N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.
","![\"U_{\\mathrm{batch}}=E\"](\"assets/equations/89b7eb4a772465f4.svg\")
![\"U_{\\mathrm{SGD}}=EN\"](\"assets/equations/0eab2be6df842149.svg\")
![\"U_{\\mathrm{keep}}=E\\left\\lceil\\frac](\"assets/equations/f9d09d1e5734cccd.svg\")
![\"U_{\\mathrm{drop}}=E\\left\\lfloor\\frac](\"assets/equations/bf7e65981e0892c7.svg\")
Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.
","Step 1: count full-batch updates.
","All 1,000 samples form one group, so one epoch has 1 update.
","Step 2: count stochastic updates.
","One sample per update gives 1,000 updates per epoch.
","Step 3: count mini-batches.
","![\"\\frac{N}{B}=\\frac{1000}{100}=10\"](\"assets/equations/044096791068cfc7.svg\")
Final answer: batch = 1; SGD = 1,000; mini-batch = 10 updates per epoch.
","Question. A dataset contains 105 samples and the mini-batch size is 20. Keeping the incomplete last batch, find the updates per epoch.
","N = 105; B = 20; retain the final incomplete batch.
","The number of updates per epoch.
","Every processed group causes one update. A remaining nonempty group also counts.
","The ceiling symbol rounds upward to the next integer.
","![\"U=\\lceil](\"assets/equations/a73af57990a3f346.svg\")
Yaad rakhne ka cue: Bacha hua batch process kiya toh ek aur update.
","Step 1: count complete batches.
","![\"5(20)=100\"](\"assets/equations/165b56a31834874f.svg\")
Step 2: count remaining examples.
","![\"105-100=5\"](\"assets/equations/018ad701e47e6473.svg\")
Step 3: include the nonempty last batch.
","![\"U=5+1=6\"](\"assets/equations/298ed637938b4bf6.svg\")
Final answer: 6 updates. If a question explicitly drops the final incomplete batch, it instead gives 5.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s4","search":"Batch stochastic and mini batch gradient descent **Definition.** Batch, stochastic and mini-batch gradient descent differ in the subset of training examples used to estimate the gradient for each parameter update. Their update equation is the same; their gradient estimates and update frequency differ.\n\n### Explanation and working\nAll three use the same update idea. They differ in how much data contributes to one gradient.\n\n\n\n| Method | Data for one update | Updates in one epoch with N samples | Typical behaviour |\n| Batch GD | All N samples | 1 | Stable full-data gradient; costly updates |\n| Stochastic GD | One sample | N | Frequent noisy updates |\n| Mini-batch GD | B samples | Ceiling of N/B if final partial batch is retained | Efficient parallel calculation with moderate noise |\n**Question.** N = 1,000 and B = 100. Find updates per epoch.\n\n\n\n### Given\n\nN = 1,000 examples; mini-batch size B = 100; one epoch.\n\n### To find\n\nCount full-batch, stochastic and mini-batch updates per epoch.\n\n### Formula — which one and why\n\n**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.\n\nN = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.\n\n### Solution — step by step\n\n\n\n\n**Step 1: count full-batch updates.**\n\nAll 1,000 samples form one group, so one epoch has 1 update.\n\n**Step 2: count stochastic updates.**\n\nOne sample per update gives 1,000 updates per epoch.\n\n**Step 3: count mini-batches.**\n\n\n\n\n\n\n\n**Final answer:** batch = 1; SGD = 1,000; mini-batch = 10 updates per epoch.\n\n\n### Mini-batch calculation with a remainder\n\n**Question. A dataset contains 105 samples and the mini-batch size is 20. Keeping the incomplete last batch, find the updates per epoch.**\n\n\n\n### Given\n\nN = 105; B = 20; retain the final incomplete batch.\n\n### To find\n\nThe number of updates per epoch.\n\n### Formula — which one and why\n\nEvery processed group causes one update. A remaining nonempty group also counts.\n\nThe ceiling symbol rounds upward to the next integer.\n\n\n\n**Yaad rakhne ka cue:** Bacha hua batch process kiya toh ek aur update.\n\n### Solution — step by step\n\n\n\n**Step 1: count complete batches.**\n\n\n\n\n\n\n\n**Step 2: count remaining examples.**\n\n\n\n\n\n\n\n**Step 3: include the nonempty last batch.**\n\n\n\n\n\n\n\n**Final answer:** 6 updates. If a question explicitly drops the final incomplete batch, it instead gives 5.\n\n"},{"title":"Model answer: batch, stochastic and mini-batch training","question":["Question. Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration.
"],"answer":["All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update.
","A batch is the group of examples used together for an update. An iteration, in this context, is one optimizer update. An epoch is one complete pass through the chosen training dataset. One epoch usually contains several iterations.
","| Basis | Batch GD | Stochastic GD | Mini-batch GD |
|---|---|---|---|
| Examples in one gradient | Entire training dataset | One example | A smaller group of B examples |
| Updates per epoch | One | N | Ceiling of N/B if the remainder is retained |
| Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging |
| Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation |
With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch.
","The basic parameter update is unchanged:
","![\"\\theta_{\\mathrm{new}}=\\theta_{\\mathrm{old}}-\\eta](\"assets/equations/8a6ef020e7632784.svg\")
For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples.
"],"kind":"theory","id":"mt2-c5-s5","search":"Model answer: batch, stochastic and mini-batch training **Question.** Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration.\n\n**Model answer.**\nAll three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update.\n\n\n\n\nA **batch** is the group of examples used together for an update. An **iteration**, in this context, is one optimizer update. An **epoch** is one complete pass through the chosen training dataset. One epoch usually contains several iterations.\n\n| Basis | Batch GD | Stochastic GD | Mini-batch GD |\n| Examples in one gradient | Entire training dataset | One example | A smaller group of B examples |\n| Updates per epoch | One | N | Ceiling of N/B if the remainder is retained |\n| Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging |\n| Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation |\n\n### Example to distinguish the three counts\nWith 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch.\n\nThe basic parameter update is unchanged:\n\nFor batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples.\n"},{"title":"A tiny dataset shows why the methods differ","blocks":["Question. Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order.
","Model ŷ = wx; initial w = 1; η = 0.1.
","Samples in order: (1,2), (2,4).
","Mean half squared loss for the batch.
","Compare one batch update with a sequential SGD epoch.
","Batch mean versus sequential SGD. A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.
","B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.
","![\"g_{\\mathrm{batch}}=\\frac1B\\sum_i](\"assets/equations/657f63b5043d3907.svg\")
![\"g_i=(wx_i-y_i)x_i\"](\"assets/equations/0b4939a48933eb0a.svg\")
Yaad rakhne ka cue: Batch: same old weight par average; SGD: har sample ke baad weight change.
","Step 1: predict sample 1 with the original weight.
","![\"\\hat](\"assets/equations/d19c8d7124e25bc5.svg\")
Step 2: calculate its error.
","Step 3: multiply its error by its input.
","![\"g_1=(-1)(1)=-1\"](\"assets/equations/fdbcda3ad0050efb.svg\")
Step 4: predict sample 2 with the same original weight.
","![\"\\hat](\"assets/equations/7be604573d864f92.svg\")
Step 5: calculate its error.
","![\"e_2=2-4=-2\"](\"assets/equations/55c62638d290dff9.svg\")
Step 6: multiply its error by its input.
","![\"g_2=(-2)(2)=-4\"](\"assets/equations/6f6d46df3c042284.svg\")
Step 7: average the two gradients.
","![\"g=\\frac{-1+(-4)}2=-2.5\"](\"assets/equations/8cfa6ee1614fe8e3.svg\")
Step 8: update once.
","![\"w_{\\mathrm{new}}=1-0.1(-2.5)=1.25\"](\"assets/equations/f263219d999d6318.svg\")
Step 1: use the first sample gradient.
","![\"g_1=-1\"](\"assets/equations/7a43ae4bf3e05e38.svg\")
Step 2: update after the first sample.
","![\"w_1=1-0.1(-1)=1.1\"](\"assets/equations/f2a537796fbcbe31.svg\")
Step 3: predict sample 2 using the changed weight.
","![\"\\hat](\"assets/equations/b434bd09ecf3d6fa.svg\")
Step 4: find sample 2 error.
","![\"e_2=2.2-4=-1.8\"](\"assets/equations/cbcb5d7c79c1f6bd.svg\")
Step 5: find the new sample 2 gradient.
","![\"g_2=(-1.8)(2)=-3.6\"](\"assets/equations/352a3c332b7e819d.svg\")
Step 6: update after sample 2.
","![\"w_2=1.1-0.1(-3.6)=1.46\"](\"assets/equations/337ee1d4f3248df4.svg\")
Final answer: batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples.
","Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s6","search":"A tiny dataset shows why the methods differ **Question.** Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order.\n\n\n\n### Given\n\nModel ŷ = wx; initial w = 1; η = 0.1.\n\nSamples in order: (1,2), (2,4).\n\nMean half squared loss for the batch.\n\n### To find\n\nCompare one batch update with a sequential SGD epoch.\n\n### Formula — which one and why\n\n**Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.\n\nB = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change.\n\n### Solution — step by step\n\n\n\n\n### One batch update\n\n**Step 1: predict sample 1 with the original weight.**\n\n\n\n\n\n\n\n**Step 2: calculate its error.**\n\n\n\n\n\n\n\n**Step 3: multiply its error by its input.**\n\n\n\n\n\n\n\n**Step 4: predict sample 2 with the same original weight.**\n\n\n\n\n\n\n\n**Step 5: calculate its error.**\n\n\n\n\n\n\n\n**Step 6: multiply its error by its input.**\n\n\n\n\n\n\n\n**Step 7: average the two gradients.**\n\n\n\n\n\n\n\n**Step 8: update once.**\n\n\n\n\n\n\n\n\n\n\n\n### One sequential SGD epoch\n\n**Step 1: use the first sample gradient.**\n\n\n\n\n\n\n\n**Step 2: update after the first sample.**\n\n\n\n\n\n\n\n**Step 3: predict sample 2 using the changed weight.**\n\n\n\n\n\n\n\n**Step 4: find sample 2 error.**\n\n\n\n\n\n\n\n**Step 5: find the new sample 2 gradient.**\n\n\n\n\n\n\n\n**Step 6: update after sample 2.**\n\n\n\n\n\n\n\n**Final answer:** batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples.\nLarger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim."},{"title":"Learning rate and common training problems","blocks":["Definition. The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss.
","A very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate.
","| Observation | Possible reason | Sensible first action |
|---|---|---|
| Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation |
| Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate |
| Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum |
| Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping |
| All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights |
Convergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s7","search":"Learning rate and common training problems **Definition.** The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss.\n\n### Explanation and working\nA very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate.\n\n| Observation | Possible reason | Sensible first action |\n| Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation |\n| Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate |\n| Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum |\n| Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping |\n| All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights |\nConvergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum."},{"title":"A moving average remembers the recent past","blocks":["Definition. An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients.
","![\"m_t=\\beta](\"assets/equations/278e38d8879b5e28.svg\")
For m₀ = 0, expanding the recurrence gives:
","![\"m_t=(1-\\beta)\\sum_{k=1}^{t}\\beta^{t-k}g_k\"](\"assets/equations/3400900697b48b50.svg\")
The most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history.
","An average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form.
","Question. Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂.
","β = 0.9; initial m₀ = 0.
","Gradients in order: g₁ = 2, g₂ = −1.
","Find m₁ and m₂, preserving the sign of each gradient.
","Exponential moving average. Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state.
","β (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state.
","![\"m_t=\\beta](\"assets/equations/7999b979f742c872.svg\")
Yaad rakhne ka cue: Beta purani memory rakhta hai; 1−beta naya gradient leta hai.
","Step 1: find the recent-gradient coefficient.
","![\"1-0.9=0.1\"](\"assets/equations/1d3a9c6b5c275e6a.svg\")
Step 2: retain the old memory for step 1.
","![\"0.9(0)=0\"](\"assets/equations/4351a4e3442a023e.svg\")
Step 3: weight the first gradient.
","![\"0.1(2)=0.2\"](\"assets/equations/687125898d5f60b7.svg\")
Step 4: add to obtain the first memory.
","![\"m_1=0+0.2=0.2\"](\"assets/equations/624684fe1c8cf03d.svg\")
Step 5: retain the first memory for step 2.
","![\"0.9(0.2)=0.18\"](\"assets/equations/30b407cc303a0e19.svg\")
Step 6: weight the second gradient.
","![\"0.1(-1)=-0.1\"](\"assets/equations/995079ed548eec78.svg\")
Step 7: add to obtain the second memory.
","![\"m_2=0.18+(-0.1)=0.08\"](\"assets/equations/0e62fb9876ad9dac.svg\")
Final answer: m₁ = 0.2; m₂ = 0.08.
","The second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s8","search":"A moving average remembers the recent past **Definition.** An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients.\n\nFor m₀ = 0, expanding the recurrence gives:\n\nThe most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history.\n\n### Signed gradients versus squared gradients\nAn average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form.\n\n**Question.** Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂.\n\n\n\n### Given\n\nβ = 0.9; initial m₀ = 0.\n\nGradients in order: g₁ = 2, g₂ = −1.\n\n### To find\n\nFind m₁ and m₂, preserving the sign of each gradient.\n\n### Formula — which one and why\n\n**Exponential moving average.** Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state.\n\nβ (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state.\n\n\n\n**Yaad rakhne ka cue:** Beta purani memory rakhta hai; 1−beta naya gradient leta hai.\n\n### Solution — step by step\n\n\n\n\n**Step 1: find the recent-gradient coefficient.**\n\n\n\n\n\n\n\n**Step 2: retain the old memory for step 1.**\n\n\n\n\n\n\n\n**Step 3: weight the first gradient.**\n\n\n\n\n\n\n\n**Step 4: add to obtain the first memory.**\n\n\n\n\n\n\n\n**Step 5: retain the first memory for step 2.**\n\n\n\n\n\n\n\n**Step 6: weight the second gradient.**\n\n\n\n\n\n\n\n**Step 7: add to obtain the second memory.**\n\n\n\n\n\n\n\n**Final answer:** m₁ = 0.2; m₂ = 0.08.\nThe second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable.\n"},{"title":"Momentum remembers movement","blocks":["Definition. Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions.
","This course uses signed displacement velocity v. It already includes the learning rate, and the parameter update adds it to the old parameter:
","![\"g_t=J'(\\theta_{t-1})\"](\"assets/equations/c19372684b38c7e4.svg\")
![\"v_t=\\beta](\"assets/equations/14e98f110476e8df.svg\")
![\"\\theta_t=\\theta_{t-1}+v_t\"](\"assets/equations/14de701208151b5b.svg\")
θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters.
","Question. J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity.
","J(θ) = ½θ²; θ₀ = 1; v₀ = 0.
","η = 0.1; β = 0.9; signed-displacement velocity.
","Find the gradient, velocity and actual parameter after each of two iterations.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","Derivative used: differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration.
","Step 1: Iteration 1: gradient, velocity, parameter.
","![\"g_1=J'(1)=1\"](\"assets/equations/9e80b135dd68a0a3.svg\")
![\"v_1=0.9(0)-0.1(1)=-0.1\"](\"assets/equations/e23af655c3740539.svg\")
![\"\\theta_1=1+(-0.1)=0.9\"](\"assets/equations/e96386ed9eb17f75.svg\")
Step 2: Iteration 2: use θ₁ and v₁.
","![\"g_2=J'(0.9)=0.9\"](\"assets/equations/14dc30bcccfa9d0f.svg\")
![\"v_2=0.9(-0.1)-0.1(0.9)=-0.18\"](\"assets/equations/0143acec3cdc1b06.svg\")
![\"\\theta_2=0.9+(-0.18)=0.72\"](\"assets/equations/4502fcbdb27f5deb.svg\")
| Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter |
|---|---|---|---|---|
| 1 | 1 | 1 | −0.1 | 0.9 |
| 2 | 0.9 | 0.9 | −0.18 | 0.72 |
Final answer: θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient.
","Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective.
","Some texts store a positive gradient accumulator u instead:
","![\"u_t=\\beta](\"assets/equations/fcab484ef321b3b2.svg\")
With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s9","search":"Momentum remembers movement \n**Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions.\n\n### Update equations and convention\nThis course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter:\n\n\n\nθ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters.\n\n\n\n### Complete numerical: two iterations\n**Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity.\n\n\n\n### Given\n\nJ(θ) = ½θ²; θ₀ = 1; v₀ = 0.\n\nη = 0.1; β = 0.9; signed-displacement velocity.\n\n### To find\n\nFind the gradient, velocity and actual parameter after each of two iterations.\n\n### Formula — which one and why\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n**Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration.\n\n**Step 1: Iteration 1: gradient, velocity, parameter.**\n\n\n\n\n**Step 2: Iteration 2: use θ₁ and v₁.**\n\n\n\n\n| Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter |\n| 1 | 1 | 1 | −0.1 | 0.9 |\n| 2 | 0.9 | 0.9 | −0.18 | 0.72 |\n\n**Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient.\n\n### Properties and alternative notation\nMomentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective.\n\nSome texts store a positive gradient accumulator u instead:\n\nWith constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions.\n\n"},{"title":"Nesterov looks before correcting","blocks":["Definition. Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update.
","Ordinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured:
","![\"\\theta_{\\mathrm{look},t}=\\theta_{t-1}+\\beta](\"assets/equations/64563c533a276a7d.svg\")
![\"g_t=J'(\\theta_{\\mathrm{look},t})\"](\"assets/equations/0ec7b790dfa7e622.svg\")
The lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁.
","Question. J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity.
","J(θ) = ½θ²; θ₀ = 1; v₀ = 0.
","η = 0.1; β = 0.9; signed-displacement velocity.
","Find the gradient, velocity and actual parameter after each of two iterations.
","Nesterov lookahead. First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.
","θ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.
","![\"\\theta_{\\mathrm{look}}=\\theta_{t-1}+\\beta](\"assets/equations/1a4a295c7aac3f17.svg\")
![\"g_t=J\\prime(\\theta_{\\mathrm{look}})\"](\"assets/equations/4d83df079bf260cc.svg\")
Yaad rakhne ka cue: Pehle look ahead; gradient wahan; update original position se.
","The derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective.
","Iteration 1 — step 1: locate the lookahead point.
","![\"\\theta_{\\mathrm{look},1}=1+0.9(0)=1\"](\"assets/equations/0abe86d8e2a6e122.svg\")
Step 2: evaluate the gradient there.
","Step 3: calculate the new velocity and actual parameter.
","Iteration 2 — step 1: use the previous actual parameter and velocity.
","![\"\\theta_{\\mathrm{look},2}=0.9+0.9(-0.1)=0.81\"](\"assets/equations/25448224eab650c3.svg\")
Step 2: evaluate the gradient at 0.81, not at 0.9.
","![\"g_2=J'(0.81)=0.81\"](\"assets/equations/e8429bc60dff5c5c.svg\")
Step 3: calculate the new velocity.
","![\"v_2=0.9(-0.1)-0.1(0.81)=-0.171\"](\"assets/equations/35f06a75a728a22a.svg\")
Step 4: update the old actual parameter.
","![\"\\theta_2=0.9+(-0.171)=0.729\"](\"assets/equations/5b7b6422b337bab9.svg\")
| Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ |
|---|---|---|---|---|---|
| 1 | 1 | 1 | 1 | −0.1 | 0.9 |
| 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 |
Final answer: θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s10","search":"Nesterov looks before correcting \n**Definition.** Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update.\n\n### What changes relative to momentum\nOrdinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured:\n\n\n\n\nThe lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁.\n\n\n\n### Complete numerical: two iterations\n**Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity.\n\n\n\n### Given\n\nJ(θ) = ½θ²; θ₀ = 1; v₀ = 0.\n\nη = 0.1; β = 0.9; signed-displacement velocity.\n\n### To find\n\nFind the gradient, velocity and actual parameter after each of two iterations.\n\n### Formula — which one and why\n\n**Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.\n\nθ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se.\n\n### Solution — step by step\n\n\n\n\n\nThe derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective.\n\n**Iteration 1 — step 1: locate the lookahead point.**\n\n**Step 2: evaluate the gradient there.**\n\n**Step 3: calculate the new velocity and actual parameter.**\n\n\n\n**Iteration 2 — step 1: use the previous actual parameter and velocity.**\n\n**Step 2: evaluate the gradient at 0.81, not at 0.9.**\n\n**Step 3: calculate the new velocity.**\n\n**Step 4: update the old actual parameter.**\n\n\n| Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ |\n| 1 | 1 | 1 | 1 | −0.1 | 0.9 |\n| 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 |\n\n**Final answer:** θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice.\n\n"},{"title":"Model answer: momentum and Nesterov","question":["Question. Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ?
"],"answer":["Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions.
","This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained.
","Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter.
","![\"g_t=J'(\\theta_{\\mathrm{look}})\"](\"assets/equations/88dc28f2501b464e.svg\")
The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point.
","| Basis | Momentum | Nesterov |
|---|---|---|
| Gradient evaluated at | Current parameter | Lookahead parameter |
| Information retained | Previous velocity | Previous velocity |
| Intuition | Continue useful movement | Inspect the slope where old momentum points |
| Initial zero velocity | No movement contribution yet | First lookahead equals current parameter |
Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently.
"],"kind":"theory","id":"mt2-c5-s11","search":"Model answer: momentum and Nesterov **Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ?\n\n**Model answer.**\nMomentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions.\n\n\n\n\n### Momentum formulas\nThis answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained.\n\n\n\n### Nesterov's change\nNesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter.\n\n\nThe velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point.\n\n| Basis | Momentum | Nesterov |\n| Gradient evaluated at | Current parameter | Lookahead parameter |\n| Information retained | Previous velocity | Previous velocity |\n| Intuition | Continue useful movement | Inspect the slope where old momentum points |\n| Initial zero velocity | No movement contribution yet | First lookahead equals current parameter |\n\nBoth require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently.\n"},{"title":"AdaGrad adds every squared gradient","blocks":["Definition. AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator.
","For one parameter θ, let gₜ be its current gradient and s₀ = 0:
","![\"s_t=s_{t-1}+g_t^2\"](\"assets/equations/32f05266b341eaa2.svg\")
![\"\\eta_{\\mathrm{effective},t}=\\frac{\\eta}{\\sqrt{s_t}+\\varepsilon}\"](\"assets/equations/5cf38c4d809aeefb.svg\")
![\"\\theta_t=\\theta_{t-1}-\\eta_{\\mathrm{effective},t}g_t\"](\"assets/equations/52438222ad8d94df.svg\")
s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign.
","Question. Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero.
","θ₀ = 1; initial state(s) = 0; η = 0.1.
","Supplied gradients g₁ = 2, g₂ = 1.
","ε = 0 for this hand calculation.
","Calculate all required states and the parameter after each of two updates.
","AdaGrad. Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.
","G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.
","![\"G_t=G_{t-1}+g_t^2\"](\"assets/equations/221c2b3d2079d996.svg\")
![\"\\theta_t=\\theta_{t-1}-\\frac{\\eta](\"assets/equations/2633d5c0ed034026.svg\")
Yaad rakhne ka cue: Square karo, purane sum mein jodo, root se scale karo.
","The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter.
","Iteration 1 — step 1: update the accumulator.
","![\"s_1=0+2^2=4\"](\"assets/equations/18b84cdbfe918936.svg\")
Step 2: calculate the effective learning rate.
","![\"\\eta_{\\mathrm{effective},1}=\\frac{0.1}{\\sqrt4}=0.05\"](\"assets/equations/8acc57815fd4951a.svg\")
Step 3: apply the signed gradient.
","![\"\\theta_1=1-0.05(2)=0.9\"](\"assets/equations/8aa23a9e3a9c5875.svg\")
Iteration 2 — step 1: retain s₁ and add the new squared gradient.
","![\"s_2=4+1^2=5\"](\"assets/equations/1e29c54438050499.svg\")
Step 2: calculate the new coefficient.
","![\"\\eta_{\\mathrm{effective},2}=\\frac{0.1}{\\sqrt5}\\approx0.044721360\"](\"assets/equations/4d33a28de17065dc.svg\")
Step 3: update from θ₁.
","![\"\\theta_2=0.9-\\frac{0.1(1)}{\\sqrt5}\\approx0.855278640\"](\"assets/equations/81f6101b82b353cb.svg\")
Final answer: θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5.
","Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes.
","ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s12","search":"AdaGrad adds every squared gradient \n**Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator.\n\n### State, effective learning rate and update\nFor one parameter θ, let gₜ be its current gradient and s₀ = 0:\n\n\n\ns stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign.\n\n\n\n### Complete numerical: supplied gradients\n**Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero.\n\n\n\n### Given\n\nθ₀ = 1; initial state(s) = 0; η = 0.1.\n\nSupplied gradients g₁ = 2, g₂ = 1.\n\nε = 0 for this hand calculation.\n\n### To find\n\nCalculate all required states and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.\n\nG = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.\n\n\n\n\n\n**Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo.\n\n### Solution — step by step\n\n\n\n\n\nThe gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter.\n\n**Iteration 1 — step 1: update the accumulator.**\n\n**Step 2: calculate the effective learning rate.**\n\n**Step 3: apply the signed gradient.**\n\n\n**Iteration 2 — step 1: retain s₁ and add the new squared gradient.**\n\n**Step 2: calculate the new coefficient.**\n\n**Step 3: update from θ₁.**\n\n\n**Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5.\n\n### Advantage, limitation and epsilon convention\nCoordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes.\n\nε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical.\n\n"},{"title":"RMSProp remembers recent squared gradients","blocks":["Definition. RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator.
","Using decay β and zero initial state:
","![\"s_t=\\beta](\"assets/equations/6b2de5dc1fbe8c99.svg\")
![\"\\theta_t=\\theta_{t-1}-\\frac{\\eta](\"assets/equations/03fd4ff4e7ab2e6e.svg\")
β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update.
","If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever.
","Question. θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation.
","θ₀ = 1; initial state(s) = 0; η = 0.1.
","Supplied gradients g₁ = 2, g₂ = 1.
","ε = 0 for this hand calculation.
","β = 0.9.
","Calculate all required states and the parameter after each of two updates.
","RMSProp. Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.
","s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.
","![\"s_t=\\rho](\"assets/equations/9b03d623d9793c84.svg\")
![\"\\theta_t=\\theta_{t-1}-\\frac{\\eta](\"assets/equations/8de35fe2ee69fe63.svg\")
Yaad rakhne ka cue: Saare purane squares nahi—unki fading average.
","Iteration 1 — step 1: calculate the squared-gradient average.
","![\"s_1=0.9(0)+0.1(2^2)=0.4\"](\"assets/equations/ea96f08ec1d03a73.svg\")
Step 2: calculate the denominator and signed correction.
","![\"\\sqrt{s_1}=\\sqrt{0.4}\\approx0.632455532\"](\"assets/equations/d388f002971175a6.svg\")
![\"\\frac{\\eta](\"assets/equations/91debda43e9c7015.svg\")
Step 3: update the parameter.
","![\"\\theta_1=1-0.316227766\\approx0.683772234\"](\"assets/equations/46ce5315f0e4b144.svg\")
Iteration 2 — step 1: use s₁, not the original zero state.
","![\"s_2=0.9(0.4)+0.1(1^2)=0.46\"](\"assets/equations/a780b29bab6859eb.svg\")
Step 2: calculate the denominator and signed correction.
","![\"\\sqrt{s_2}=\\sqrt{0.46}\\approx0.678232998\"](\"assets/equations/10263e4a8476b1fa.svg\")
![\"\\frac{\\eta](\"assets/equations/ce8864a56ffdc251.svg\")
Step 3: update from θ₁, retaining calculator precision.
","![\"\\theta_2\\approx0.683772234-0.147441956\\approx0.536330278\"](\"assets/equations/dd5064d2dd1b316c.svg\")
Final answer: θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46.
","AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s13","search":"RMSProp remembers recent squared gradients \n**Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator.\n\n### Recurrence and interpretation\nUsing decay β and zero initial state:\n\n\nβ is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update.\n\nIf recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever.\n\n\n\n### Complete numerical: two iterations\n**Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation.\n\n\n\n### Given\n\nθ₀ = 1; initial state(s) = 0; η = 0.1.\n\nSupplied gradients g₁ = 2, g₂ = 1.\n\nε = 0 for this hand calculation.\n\nβ = 0.9.\n\n### To find\n\nCalculate all required states and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.\n\ns = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.\n\n\n\n\n\n**Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average.\n\n### Solution — step by step\n\n\n\n\n\n**Iteration 1 — step 1: calculate the squared-gradient average.**\n\n**Step 2: calculate the denominator and signed correction.**\n\n\n**Step 3: update the parameter.**\n\n\n**Iteration 2 — step 1: use s₁, not the original zero state.**\n\n**Step 2: calculate the denominator and signed correction.**\n\n\n**Step 3: update from θ₁, retaining calculator precision.**\n\n\n**Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46.\n\n### Comparison with AdaGrad and Adam\nAdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum.\n\n"},{"title":"Adam remembers both direction and squared size","blocks":["Definition. Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates.
","| State or coefficient | Meaning |
|---|---|
| mₜ | Moving average of signed gradients; retains directional information |
| sₜ | Moving average of squared gradients; estimates their uncentered second moment |
| β₁ | Decay coefficient for the first-moment estimate |
| β₂ | Decay coefficient for the second-moment estimate |
| t | Update number, starting at 1 for the first update |
| η and ε | Base learning rate and small denominator stabilizer |
The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity.
","![\"m_t=\\beta_1m_{t-1}+(1-\\beta_1)g_t\"](\"assets/equations/c684b3ce4733fb3b.svg\")
![\"s_t=\\beta_2s_{t-1}+(1-\\beta_2)g_t^2\"](\"assets/equations/0be5e35b2aa1dc00.svg\")
Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight:
","![\"\\hat](\"assets/equations/b6ee8d213ad96014.svg\")
![\"\\hat](\"assets/equations/39aff9d162818e1c.svg\")
The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator:
","![\"\\theta_t=\\theta_{t-1}-\\frac{\\eta\\hat](\"assets/equations/ada7a51c96cfce92.svg\")
The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ.
","Question. θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic.
","θ₀ = 1; initial state(s) = 0; η = 0.1.
","Supplied gradients g₁ = 2, g₂ = 1.
","ε = 0 for this hand calculation.
","β₁ = 0.9; β₂ = 0.999; use bias correction.
","Calculate all required states and the parameter after each of two updates.
","Adam with bias correction. Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update.
","m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence.
","![\"\\theta_t=\\theta_{t-1}-\\frac{\\eta\\hat](\"assets/equations/b1b0801ea4c5e3b1.svg\")
Yaad rakhne ka cue: Direction memory + square memory; hats lagao, phir update.
","Iteration 1 — step 1: update the two raw states.
","![\"m_1=0.9(0)+0.1(2)=0.2\"](\"assets/equations/5ac027abcdfcdb75.svg\")
![\"s_1=0.999(0)+0.001(2^2)=0.004\"](\"assets/equations/664a8100f810d926.svg\")
Step 2: correct both states using t = 1.
","![\"\\hat](\"assets/equations/daa0ad010127bd94.svg\")
![\"\\hat](\"assets/equations/e60fd90ed8957afb.svg\")
Step 3: update the parameter.
","![\"\\theta_1=1-\\frac{0.1(2)}{\\sqrt4}=0.9\"](\"assets/equations/dd2c0d46e60db2c0.svg\")
Start with the current gradient and the two states saved from the previous iteration.
Carry forward raw states: m = 0.2, s = 0.004. The corrected values are used for this update.
Iteration 2 — step 1: carry forward the raw states m₁ and s₁.
","The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4.
","![\"m_2=0.9(0.2)+0.1(1)=0.28\"](\"assets/equations/a6a9a858d7c18aeb.svg\")
![\"s_2=0.999(0.004)+0.001(1^2)=0.004996\"](\"assets/equations/528b23a17d8f719c.svg\")
Step 2: correct both states using t = 2.
","![\"1-0.9^2=0.19,\\qquad1-0.999^2=0.001999\"](\"assets/equations/0416e7f741fa3055.svg\")
![\"\\hat](\"assets/equations/3d16280e17e8b649.svg\")
![\"\\hat](\"assets/equations/a22ef3b407f486c3.svg\")
Step 3: divide the corrected first moment by the corrected root.
","![\"\\frac{\\eta\\hat](\"assets/equations/d8416409081efc8e.svg\")
Step 4: update from θ₁.
","![\"\\theta_2\\approx0.9-0.093217964\\approx0.806782036\"](\"assets/equations/6c55d4ff26b9e4b3.svg\")
Start with the current gradient and the two states saved from the previous iteration.
Carry forward raw states: m = 0.28, s = 0.004996. The corrected values are used for this update.
| State | After iteration 1 | After iteration 2 |
|---|---|---|
| Raw first moment m | 0.2 | 0.28 |
| Raw second moment s | 0.004 | 0.004996 |
| Corrected first moment | 2 | 1.473684211 |
| Corrected second moment | 4 | 2.499249625 |
| Parameter θ | 0.9 | 0.806782036 |
Final answer: θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update.
","Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s14","search":"Adam remembers both direction and squared size \n**Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates.\n\n### What the two states represent\n| State or coefficient | Meaning |\n| mₜ | Moving average of signed gradients; retains directional information |\n| sₜ | Moving average of squared gradients; estimates their uncentered second moment |\n| β₁ | Decay coefficient for the first-moment estimate |\n| β₂ | Decay coefficient for the second-moment estimate |\n| t | Update number, starting at 1 for the first update |\n| η and ε | Base learning rate and small denominator stabilizer |\n\nThe second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity.\n\n\n\n### Recurrences, bias correction and update\n\n\nZero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight:\n\n\nThe update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator:\n\nThe derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ.\n\n### Complete numerical: changing gradient magnitude\n**Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic.\n\n\n\n### Given\n\nθ₀ = 1; initial state(s) = 0; η = 0.1.\n\nSupplied gradients g₁ = 2, g₂ = 1.\n\nε = 0 for this hand calculation.\n\nβ₁ = 0.9; β₂ = 0.999; use bias correction.\n\n### To find\n\nCalculate all required states and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update.\n\nm = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update.\n\n### Solution — step by step\n\n\n\n\n\n**Iteration 1 — step 1: update the two raw states.**\n\n\n**Step 2: correct both states using t = 1.**\n\n\n**Step 3: update the parameter.**\n\n\n**Iteration 2 — step 1: carry forward the raw states m₁ and s₁.**\nThe recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4.\n\n\n**Step 2: correct both states using t = 2.**\n\n\n\n**Step 3: divide the corrected first moment by the corrected root.**\n\n**Step 4: update from θ₁.**\n\n\n| State | After iteration 1 | After iteration 2 |\n| Raw first moment m | 0.2 | 0.28 |\n| Raw second moment s | 0.004 | 0.004996 |\n| Corrected first moment | 2 | 1.473684211 |\n| Corrected second moment | 4 | 2.499249625 |\n| Parameter θ | 0.9 | 0.806782036 |\n\n**Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update.\n\n### Properties and limitations\nAdam combines directional "},{"title":"Adam bias correction: why the denominator is one minus beta to a power","blocks":["Definition. Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate.
","Adam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization.
","![\"m_1=(1-\\beta)g_1\"](\"assets/equations/311f8a66ba1558a5.svg\")
At the second step, substitute the first expression into the recurrence:
","![\"m_2=\\beta(1-\\beta)g_1+(1-\\beta)g_2\"](\"assets/equations/e5bd41b82ae93376.svg\")
After the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term:
","![\"\\beta(1-\\beta)+(1-\\beta)=(1-\\beta)(1+\\beta)=1-\\beta^2\"](\"assets/equations/4c68b61c5bdf0798.svg\")
After t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit.
","![\"\\widehat](\"assets/equations/4c24c9160cc537a1.svg\")
The squared-gradient memory uses its own decay β₂ and the same reasoning:
","![\"\\widehat](\"assets/equations/c3ebf23cb24a812d.svg\")
Let both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38.
","![\"\\widehat](\"assets/equations/95360064605e4687.svg\")
![\"\\widehat](\"assets/equations/569c2f898bf0d2e3.svg\")
Both corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient.
","At update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators.
","The parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows.
","This derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s15","search":"Adam bias correction: why the denominator is one minus beta to a power **Definition.** Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate.\n\n### Explanation and working\n### 1. What the moving average actually contains\nAdam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization.\n\n\n\n\nAt the second step, substitute the first expression into the recurrence:\n\n### 2. Add the coefficients\nAfter the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term:\n\nAfter t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit.\n\nThe squared-gradient memory uses its own decay β₂ and the same reasoning:\n\n\n### 3. A constant-gradient example makes the purpose visible\nLet both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38.\n\n\nBoth corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient.\n\n### 4. Two betas mean two separate powers\nAt update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators.\n\n### 5. Correct first, then combine\nThe parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows.\nThis derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude.\n"},{"title":"Model answer: AdaGrad, RMSProp and Adam","question":["Question. Compare AdaGrad, RMSProp and Adam. Explain what each method stores and how that memory changes a parameter update.
"],"answer":["AdaGrad, RMSProp and Adam are adaptive gradient-based optimizers. Each stores separate state for each trainable parameter and uses that state to scale its update. Let gₜ be the current gradient for parameter θ, η the base learning rate and ε a small positive stabilizer. The equations below put ε outside the square root.
","Starting at s₀ = 0, every squared gradient contributes permanently. Parameters with frequent large gradients acquire smaller effective learning rates. This can be useful for sparse features, but the growing denominator can make later updates too small. The base learning rate still has to be chosen.
","Old contributions decay by repeated multiplication by β. The state can decrease when recent squared gradients are small, avoiding AdaGrad's permanent accumulation. Basic RMSProp has one squared-gradient state and no separate signed-gradient average.
","Initialize m₀ = s₀ = 0 and count the first update as t = 1:
","![\"\\hat](\"assets/equations/a84b9734c9141df3.svg\")
The first moment stores signed directional information; the second stores squared-gradient scale. Correction compensates for their zero initialization. The corrected first moment is already the numerator's gradient estimate, so the update is not multiplied by gₜ again. Raw moments are carried into the next iteration.
","| Aspect | AdaGrad | RMSProp | Adam |
|---|---|---|---|
| Squared-gradient history | Cumulative sum | Decaying average | Decaying average |
| Separate signed-gradient average | No | No in the basic method | Yes |
| Standard initialization correction | No | No in the basic method | Corrects both moments |
| State values per parameter | One | One | Two |
| Main concern | Effective rate can become very small | Learning rate and decay need tuning | More state; hyperparameters still matter |
All three adapt updates coordinate by coordinate. None removes the need for validation or guarantees a global minimum for a general neural-network objective.
"],"kind":"theory","id":"mt2-c5-s16","search":"Model answer: AdaGrad, RMSProp and Adam \n**Question.** Compare AdaGrad, RMSProp and Adam. Explain what each method stores and how that memory changes a parameter update.\n\n**Model answer.**\nAdaGrad, RMSProp and Adam are adaptive gradient-based optimizers. Each stores separate state for each trainable parameter and uses that state to scale its update. Let gₜ be the current gradient for parameter θ, η the base learning rate and ε a small positive stabilizer. The equations below put ε outside the square root.\n\n### AdaGrad: cumulative squared-gradient history\n\n\nStarting at s₀ = 0, every squared gradient contributes permanently. Parameters with frequent large gradients acquire smaller effective learning rates. This can be useful for sparse features, but the growing denominator can make later updates too small. The base learning rate still has to be chosen.\n\n### RMSProp: a decaying squared-gradient average\n\n\nOld contributions decay by repeated multiplication by β. The state can decrease when recent squared gradients are small, avoiding AdaGrad's permanent accumulation. Basic RMSProp has one squared-gradient state and no separate signed-gradient average.\n\n### Adam: first and second moments with bias correction\nInitialize m₀ = s₀ = 0 and count the first update as t = 1:\n\n\n\n\nThe first moment stores signed directional information; the second stores squared-gradient scale. Correction compensates for their zero initialization. The corrected first moment is already the numerator's gradient estimate, so the update is not multiplied by gₜ again. Raw moments are carried into the next iteration.\n\n| Aspect | AdaGrad | RMSProp | Adam |\n| Squared-gradient history | Cumulative sum | Decaying average | Decaying average |\n| Separate signed-gradient average | No | No in the basic method | Yes |\n| Standard initialization correction | No | No in the basic method | Corrects both moments |\n| State values per parameter | One | One | Two |\n| Main concern | Effective rate can become very small | Learning rate and decay need tuning | More state; hyperparameters still matter |\n\nAll three adapt updates coordinate by coordinate. None removes the need for validation or guarantees a global minimum for a general neural-network objective.\n\n"},{"title":"One comparison to rebuild all optimizer formulas","blocks":["| Method | What is remembered | What changes the step |
|---|---|---|
| GD or SGD | No optimizer memory in the basic rule | Current gradient × η |
| Momentum | Previous movement | Old velocity plus current correction |
| Nesterov | Previous movement | Gradient evaluated at lookahead |
| AdaGrad | Sum of all squared gradients | Divide current gradient by accumulated root |
| RMSProp | Moving average of squared gradients | Divide current gradient by recent root |
| Adam | Moving averages of gradients and squares | Divide corrected mean by corrected root |
Working order. Start each method on a separate line with its states initialized. The solved examples use the same supplied gradients to show how their stored states differ. They do not have to give the same parameter value. The prescribed gradient sequence is practice data; if a question instead gives J(θ), recalculate its gradient at the appropriate current or lookahead parameter.
","The gradient has a sign, but its square is nonnegative. Both methods store squared gradients to control update size. The signed gradient is still used separately in the update numerator to preserve direction.
","For an old stored state of 4 and a new gradient of −3, first square the gradient: (−3) × (−3) = 9.
","AdaGrad: add the new square to the entire old sum: 4 + 9 = 13.
","RMSProp with decay 0.9: keep 0.9 × 4 = 3.6 of the old state; add (1 − 0.9) × 9 = 0.1 × 9 = 0.9 of the new square. The new state is 3.6 + 0.9 = 4.5.
","AdaGrad retains all earlier squares in a growing sum. RMSProp gradually discounts earlier squares. This difference explains why their effective step sizes change differently during training.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c5-s18","search":"AdaGrad and RMSProp with a negative gradient The gradient has a sign, but its square is nonnegative. Both methods store squared gradients to control update size. The signed gradient is still used separately in the update numerator to preserve direction.\n\n\n\nFor an old stored state of 4 and a new gradient of −3, first square the gradient: (−3) × (−3) = 9.\n**AdaGrad:** add the new square to the entire old sum: 4 + 9 = 13.\n**RMSProp with decay 0.9:** keep 0.9 × 4 = 3.6 of the old state; add (1 − 0.9) × 9 = 0.1 × 9 = 0.9 of the new square. The new state is 3.6 + 0.9 = 4.5.\nAdaGrad retains all earlier squares in a growing sum. RMSProp gradually discounts earlier squares. This difference explains why their effective step sizes change differently during training.\n"}]},{"n":6,"title":"Regularization and generalization","sections":[{"title":"Recognizing overfitting and underfitting","blocks":["Definitions. Generalization is a model's ability to perform on unseen examples from the relevant task distribution. Underfitting occurs when the model fails to capture the task's important relationships. Overfitting occurs when fitting the training data produces substantially poorer generalization, often by modeling noise or incidental details.
","Underfitting commonly gives high training and validation errors. Overfitting commonly gives low training error with a substantially higher validation error, or validation loss that rises while training loss continues to fall. These patterns are diagnostic clues; distribution shift, leakage and an unrepresentative split can also affect the curves.
","| Observation | Plausible explanation | Appropriate response |
|---|---|---|
| Both errors remain high | Insufficient capacity, weak features or inadequate optimization | Improve representation, train appropriately, reconsider architecture or reduce excessive regularization |
| Training error low; validation error high | Overfitting or a distribution/split problem | Inspect the split; use suitable regularization, more representative data or early stopping |
| Both losses fluctuate or diverge | Optimization instability | Examine learning rate, data scaling and gradient magnitude |
Insufficient model flexibility can create high approximation bias. A highly flexible model can be sensitive to the particular training sample, increasing variance. Regularization trades some training fit for constraints that can improve generalization. A more complex network is not automatically more accurate on unseen data.
","The validation set is used to choose hyperparameters and checkpoints. The test set should remain outside these choices. Repeatedly selecting models from test results turns the test data into part of the selection process and weakens its role as an unbiased final evaluation.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c6-s0","search":"Recognizing overfitting and underfitting **Definitions.** Generalization is a model's ability to perform on unseen examples from the relevant task distribution. Underfitting occurs when the model fails to capture the task's important relationships. Overfitting occurs when fitting the training data produces substantially poorer generalization, often by modeling noise or incidental details.\n\n### Interpreting learning curves\nUnderfitting commonly gives high training and validation errors. Overfitting commonly gives low training error with a substantially higher validation error, or validation loss that rises while training loss continues to fall. These patterns are diagnostic clues; distribution shift, leakage and an unrepresentative split can also affect the curves.\n\n\n\n| Observation | Plausible explanation | Appropriate response |\n| Both errors remain high | Insufficient capacity, weak features or inadequate optimization | Improve representation, train appropriately, reconsider architecture or reduce excessive regularization |\n| Training error low; validation error high | Overfitting or a distribution/split problem | Inspect the split; use suitable regularization, more representative data or early stopping |\n| Both losses fluctuate or diverge | Optimization instability | Examine learning rate, data scaling and gradient magnitude |\n\n### Bias–variance perspective\nInsufficient model flexibility can create high approximation bias. A highly flexible model can be sensitive to the particular training sample, increasing variance. Regularization trades some training fit for constraints that can improve generalization. A more complex network is not automatically more accurate on unseen data.\n\nThe validation set is used to choose hyperparameters and checkpoints. The test set should remain outside these choices. Repeatedly selecting models from test results turns the test data into part of the selection process and weakens its role as an unbiased final evaluation.\n"},{"title":"L1 and L2 add a cost for large weights","blocks":["Definition. Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss.
","![\"J_{L1}=J_{\\mathrm{data}}+\\lambda\\sum_i|w_i|,\\qquad](\"assets/equations/73a39a472fd87b8c.svg\")
Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit.
","Question. Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention.
","Data loss = 0.5; weights = [2,−1]; λ = 0.1.
","The L2 convention here has no factor ½.
","Find L1/L2 penalties and the two regularized objective values.
","L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.
","λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.
","![\"J=J_{\\mathrm{data}}+\\lambda\\sum_i|w_i|\"](\"assets/equations/1baed4b772bbe05f.svg\")
![\"g_i=g_{\\mathrm{data},i}+\\lambda\\operatorname{sign}(w_i)\"](\"assets/equations/ad178fc1be6044c9.svg\")
Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.
","L2 penalty without a factor of one-half. This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective.
","λ = penalty strength; J_data = original data loss; wᵢ = weight i.
","![\"J=J_{\\mathrm{data}}+\\lambda\\sum_iw_i^2\"](\"assets/equations/008650efadbf4e29.svg\")
![\"\\frac{\\partial](\"assets/equations/3465dcd568b96ffb.svg\")
Yaad rakhne ka cue: Half nahi diya toh L2 derivative mein 2 lambda × weight.
","Step 1: take absolute values for L1.
","![\"|2|=2,\\qquad|-1|=1\"](\"assets/equations/0f14566cc7516925.svg\")
Step 2: add the absolute values.
","![\"2+1=3\"](\"assets/equations/23688a205b4ece73.svg\")
Step 3: multiply by lambda.
","![\"0.1(3)=0.3\"](\"assets/equations/10346aae4a408d26.svg\")
Step 4: add the data loss.
","![\"J_{L1}=0.5+0.3=0.8\"](\"assets/equations/a8ed142793124641.svg\")
Step 5: square both weights for L2.
","![\"2^2=4,\\qquad(-1)^2=1\"](\"assets/equations/e11b2a71ed1c964f.svg\")
Step 6: add the squares.
","![\"4+1=5\"](\"assets/equations/0c79fa7cad3c69e9.svg\")
Step 7: multiply by lambda without adding a half.
","![\"0.1(5)=0.5\"](\"assets/equations/d3298085ce085f2d.svg\")
Step 8: add the data loss.
","![\"J_{L2}=0.5+0.5=1.0\"](\"assets/equations/fde60160d7428d14.svg\")
For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective.
","Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.
","w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½.
","Penalty gradient, total gradient and updated weight.
","The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent.
","λ is penalty strength; η is learning rate; g is the full objective gradient.
","![\"g_{\\mathrm{penalty}}=2\\lambda](\"assets/equations/e2245fb17ab44d83.svg\")
![\"g=g_{\\mathrm{data}}+g_{\\mathrm{penalty}}\"](\"assets/equations/9ed52defd9fef6cb.svg\")
Yaad rakhne ka cue: Penalty ka gradient data gradient mein jodo, phir update.
","Step 1: calculate the penalty gradient.
","![\"2(0.1)(2)=0.4\"](\"assets/equations/b8811e22460d0c91.svg\")
Step 2: add the data gradient.
","![\"g=0.3+0.4=0.7\"](\"assets/equations/74fdb43956c5f462.svg\")
Step 3: scale by the learning rate.
","![\"\\eta](\"assets/equations/9603f5d8320b5539.svg\")
Step 4: subtract from the old weight.
","![\"w_{\\mathrm{new}}=2-0.007=1.993\"](\"assets/equations/28a66f5f8853795a.svg\")
Final answer: Updated weight = 1.993. Bias penalties are omitted unless the objective includes them.
","Under the objective written above, the data gradient and penalty gradient add:
","![\"\\nabla_w](\"assets/equations/56481f4684193b6b.svg\")
For ordinary gradient descent, regrouping the update gives:
","![\"w_{t+1}=w_t-\\eta(\\nabla_w](\"assets/equations/745f5ba3fa313836.svg\")
This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c6-s1","search":"L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss.\n\n### Explanation and working\n\nLambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit.\n\n\n\n**Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention.\n\n\n\n### Given\n\nData loss = 0.5; weights = [2,−1]; λ = 0.1.\n\nThe L2 convention here has no factor ½.\n\n### To find\n\nFind L1/L2 penalties and the two regularized objective values.\n\n### Formula — which one and why\n\n**L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.\n\nλ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.\n\n\n\n\n\n**Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai.\n\n**L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective.\n\nλ = penalty strength; J_data = original data loss; wᵢ = weight i.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight.\n\n### Solution — step by step\n\n\n\n\n**Step 1: take absolute values for L1.**\n\n\n\n\n\n\n\n**Step 2: add the absolute values.**\n\n\n\n\n\n\n\n**Step 3: multiply by lambda.**\n\n\n\n\n\n\n\n**Step 4: add the data loss.**\n\n\n\n\n\n\n\n\n**Step 5: square both weights for L2.**\n\n\n\n\n\n\n\n**Step 6: add the squares.**\n\n\n\n\n\n\n\n**Step 7: multiply by lambda without adding a half.**\n\n\n\n\n\n\n\n**Step 8: add the data loss.**\n\n\n\n\n\n\n\n\nFor this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective.\n\n\n### L2 gradient update without a half factor\n\n**Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.**\n\n\n\n### Given\n\nw = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½.\n\n### To find\n\nPenalty gradient, total gradient and updated weight.\n\n### Formula — which one and why\n\nThe derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent.\n\nλ is penalty strength; η is learning rate; g is the full objective gradient.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update.\n\n### Solution — step by step\n\n\n\n**Step 1: calculate the penalty gradient.**\n\n\n\n\n\n\n\n**Step 2: add the data gradient.**\n\n\n\n\n\n\n\n**Step 3: scale by the learning rate.**\n\n\n\n\n\n\n\n**Step 4: subtract from the old weight.**\n\n\n\n\n\n\n\n**Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them.\n\n\n\n### Deriving the L2 update\nUnder the objective written above, the data gradient and penalty gradient add:\n\nFor ordinary gradient descent, regrouping the update gives:\n\nThis shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ."},{"title":"Model answer: regularization and the L1–L2 difference","question":["Question. What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength.
"],"answer":["Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost.
","![\"J_{L1}=J_{\\mathrm{data}}+\\lambda\\sum_i|w_i|\"](\"assets/equations/52bb091c05d16021.svg\")
![\"J_{L2}=J_{\\mathrm{data}}+\\lambda\\sum_iw_i^2\"](\"assets/equations/9438fe7d9420663e.svg\")
Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question.
","| Basis | L1 | L2 |
|---|---|---|
| Quantity penalized | Absolute weight magnitude | Squared weight magnitude |
| Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention |
| Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes |
| Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights |
The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total.
","Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective.
"],"kind":"theory","id":"mt2-c6-s2","search":"Model answer: regularization and the L1–L2 difference **Question.** What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength.\n\n**Model answer.**\nRegularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost.\n\n\n\n\n### Objectives used in this guide\n\n\nJdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question.\n\n| Basis | L1 | L2 |\n| Quantity penalized | Absolute weight magnitude | Squared weight magnitude |\n| Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention |\n| Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes |\n| Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights |\n\nThe total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total.\n\nIncreasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective.\n"},{"title":"Dropout removes activations during training","blocks":["Definition. Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training.
","Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count.
","Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions.
","![\"a_{\\mathrm{train}}=m\\odot](\"assets/equations/b6b21f6d03d2989f.svg\")
![\"a_{\\mathrm{train}}=\\frac{m\\odot](\"assets/equations/7c8360afe0eaec6b.svg\")
Question. Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs.
","Activations = [0.8,−0.4,0.6,0.2].
","Dropout rate = 50%, so keep probability q = 0.5.
","Chosen mask = [1,0,1,0].
","Find training outputs under inverted dropout.
","Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.
","aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.
","![\"a_i^{\\mathrm{train}}=\\frac{m_ia_i}{q}\"](\"assets/equations/93807008ce7b52a8.svg\")
![\"a_i^{\\mathrm{test}}=a_i\"](\"assets/equations/de8c0e93f06e9aa8.svg\")
Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.
","Step 1. q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key.
","Step 2. For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = [1.6,0,1.2,0]. Inference then uses the full network without that extra scaling. State the convention in an answer.
","Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c6-s3","search":"Dropout removes activations during training **Definition.** Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training.\n\n### Explanation and working\nDropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count.\n\n\n\nLet drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions.\n\n\n**Question.** Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs.\n\n\n\n### Given\n\nActivations = [0.8,−0.4,0.6,0.2].\n\nDropout rate = 50%, so keep probability q = 0.5.\n\nChosen mask = [1,0,1,0].\n\n### To find\n\nFind training outputs under inverted dropout.\n\n### Formula — which one and why\n\n**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.\n\naᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.\n\n\n\n\n\n**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.\n\n### Solution — step by step\n\n\n\n\n**Step 1.** q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key.\n**Step 2.** For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = **[1.6,0,1.2,0]**. Inference then uses the full network without that extra scaling. State the convention in an answer.\nRandom dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition."},{"title":"Early stopping uses the best validation point","blocks":["Definition. Early stopping is a regularization and model-selection procedure that terminates training when a monitored validation criterion no longer improves according to a specified rule, typically retaining the best checkpoint.
","Monitor validation loss after epochs. When it stops improving for a chosen patience interval, stop training and restore the best validation checkpoint. Training loss can fluctuate; it does not always decrease monotonically. Never select the stopping epoch by repeatedly consulting the test set.
","Question. Validation losses after epochs 1–5 are [0.8,0.6,0.5,0.53,0.55]. Patience is 2 with no required minimum improvement. When do you stop and what do you restore?
","Validation losses at epochs 1–5 = [0.8,0.6,0.5,0.53,0.55].
","Patience = 2; no required minimum improvement.
","Find the stop epoch and the best checkpoint to restore.
","Early stopping rule. Compare each validation loss with the best one so far. A new best resets the waiting count; a non-improvement increases it. Stop when the count reaches patience and restore the best checkpoint.
","Patience = allowed consecutive epochs without improvement. With minimum improvement zero, a strictly smaller validation loss counts as improvement.
","Yaad rakhne ka cue: Best validation save; patience khatam toh stop aur best restore.
","Step 1: process epoch 1.
","Loss 0.8 establishes the first best checkpoint; wait = 0.
","Step 2: process epoch 2.
","0.6 < 0.8: save epoch 2; wait = 0.
","Step 3: process epoch 3.
","0.5 < 0.6: save epoch 3; wait = 0.
","Step 4: process epoch 4.
","0.53 > 0.5: no improvement; wait = 1.
","Step 5: process epoch 5.
","0.55 > 0.5: no improvement; wait = 2, equal to patience. Stop.
","Step 6: restore the best checkpoint.
","The smallest observed validation loss is 0.5 at epoch 3. Restore its weights.
","Final answer: stop after epoch 5; restore epoch 3. These are different epochs.
","Question. Validation losses are [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.
","Validation losses in epoch order = [0.8, 0.7, 0.72, 0.68, 0.69, 0.7].
","Patience = 2; smaller loss is better; ties are not improvements.
","The stopping epoch and best checkpoint.
","Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs.
","Patience counts consecutive non-improvements. A new best resets the count.
","Yaad rakhne ka cue: Naya best mila toh save aur wait zero; patience khatam toh best restore.
","Step 1: process epoch 1.
","Validation loss = 0.8; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.8.
","Step 2: process epoch 2.
","Validation loss = 0.7; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 2, loss 0.7.
","Step 3: process epoch 3.
","Validation loss = 0.72; no strict improvement: wait count becomes 1. Best checkpoint is epoch 2, loss 0.7.
","Step 4: process epoch 4.
","Validation loss = 0.68; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 4, loss 0.68.
","Step 5: process epoch 5.
","Validation loss = 0.69; no strict improvement: wait count becomes 1. Best checkpoint is epoch 4, loss 0.68.
","Step 6: process epoch 6.
","Validation loss = 0.7; no strict improvement: wait count becomes 2. Best checkpoint is epoch 4, loss 0.68.
","Final answer: Stop after epoch 6; restore epoch 4, with validation loss 0.68.
","Question. Validation losses are [0.6, 0.6, 0.61, 0.62]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.
","Validation losses in epoch order = [0.6, 0.6, 0.61, 0.62].
","Patience = 2; smaller loss is better; ties are not improvements.
","The stopping epoch and best checkpoint.
","Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs.
","Patience counts consecutive non-improvements. A new best resets the count.
","Yaad rakhne ka cue: Naya best mila toh save aur wait zero; patience khatam toh best restore.
","Step 1: process epoch 1.
","Validation loss = 0.6; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.6.
","Step 2: process epoch 2.
","Validation loss = 0.6; no strict improvement: wait count becomes 1. Best checkpoint is epoch 1, loss 0.6.
","Step 3: process epoch 3.
","Validation loss = 0.61; no strict improvement: wait count becomes 2. Best checkpoint is epoch 1, loss 0.6.
","Final answer: Stop after epoch 3; restore epoch 1, with validation loss 0.6.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c6-s4","search":"Early stopping uses the best validation point **Definition.** Early stopping is a regularization and model-selection procedure that terminates training when a monitored validation criterion no longer improves according to a specified rule, typically retaining the best checkpoint.\n\n### Explanation and working\nMonitor validation loss after epochs. When it stops improving for a chosen patience interval, stop training and restore the best validation checkpoint. Training loss can fluctuate; it does not always decrease monotonically. Never select the stopping epoch by repeatedly consulting the test set.\n\n\n\n**Question.** Validation losses after epochs 1–5 are [0.8,0.6,0.5,0.53,0.55]. Patience is 2 with no required minimum improvement. When do you stop and what do you restore?\n\n\n\n### Given\n\nValidation losses at epochs 1–5 = [0.8,0.6,0.5,0.53,0.55].\n\nPatience = 2; no required minimum improvement.\n\n### To find\n\nFind the stop epoch and the best checkpoint to restore.\n\n### Formula — which one and why\n\n**Early stopping rule.** Compare each validation loss with the best one so far. A new best resets the waiting count; a non-improvement increases it. Stop when the count reaches patience and restore the best checkpoint.\n\nPatience = allowed consecutive epochs without improvement. With minimum improvement zero, a strictly smaller validation loss counts as improvement.\n\n**Yaad rakhne ka cue:** Best validation save; patience khatam toh stop aur best restore.\n\n### Solution — step by step\n\n\n\n\n**Step 1: process epoch 1.**\n\nLoss 0.8 establishes the first best checkpoint; wait = 0.\n\n**Step 2: process epoch 2.**\n\n0.6 < 0.8: save epoch 2; wait = 0.\n\n**Step 3: process epoch 3.**\n\n0.5 < 0.6: save epoch 3; wait = 0.\n\n**Step 4: process epoch 4.**\n\n0.53 > 0.5: no improvement; wait = 1.\n\n**Step 5: process epoch 5.**\n\n0.55 > 0.5: no improvement; wait = 2, equal to patience. Stop.\n\n**Step 6: restore the best checkpoint.**\n\nThe smallest observed validation loss is 0.5 at epoch 3. Restore its weights.\n\n**Final answer:** stop after epoch 5; restore epoch 3. These are different epochs.\n\n### Early stopping with [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]\n\n**Question. Validation losses are [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.**\n\n\n\n### Given\n\nValidation losses in epoch order = [0.8, 0.7, 0.72, 0.68, 0.69, 0.7].\n\nPatience = 2; smaller loss is better; ties are not improvements.\n\n### To find\n\nThe stopping epoch and best checkpoint.\n\n### Formula — which one and why\n\nSave an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs.\n\nPatience counts consecutive non-improvements. A new best resets the count.\n\n**Yaad rakhne ka cue:** Naya best mila toh save aur wait zero; patience khatam toh best restore.\n\n### Solution — step by step\n\n\n\n**Step 1: process epoch 1.**\n\nValidation loss = 0.8; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.8.\n\n**Step 2: process epoch 2.**\n\nValidation loss = 0.7; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 2, loss 0.7.\n\n**Step 3: process epoch 3.**\n\nValidation loss = 0.72; no strict improvement: wait count becomes 1. Best checkpoint is epoch 2, loss 0.7.\n\n**Step 4: process epoch 4.**\n\nValidation loss = 0.68; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is ep"},{"title":"Model answer: generalization and early stopping","question":["Question. Distinguish underfitting and overfitting. Explain how validation data and early stopping help control overfitting.
"],"answer":["Underfitting occurs when a model has not captured enough useful structure in the training data. Both training and validation errors are usually high relative to a sensible task baseline. Causes can include insufficient capacity, inadequate features, excessive regularization or incomplete optimization.
","Overfitting occurs when a model fits training-specific details or noise that do not generalize. Training error is low while validation error is substantially higher. A mismatch between training and validation data can produce a similar gap, so the split must also be examined.
","1. Train using the training set; do not learn parameters from the validation targets as training samples.
","2. After an epoch or another chosen interval, evaluate validation loss.
","3. Save the parameters whenever validation loss reaches a new best value.
","4. Count consecutive intervals with no qualifying improvement.
","5. When the count reaches the chosen patience, stop and restore the saved best parameters.
","For validation losses 0.8, 0.6, 0.5, 0.53 and 0.55, epoch 3 is best. With patience 2, epochs 4 and 5 supply two non-improvements: stop after epoch 5 and restore epoch 3. The stopping epoch and the retained model epoch are different.
","Other remedies for overfitting include appropriate L1/L2 penalties, dropout, augmentation and more representative training data. Underfitting may instead require a better model, adequate training or less regularization. Use the test set for final evaluation after those choices, not for repeatedly choosing the stopping point.
"],"kind":"theory","id":"mt2-c6-s5","search":"Model answer: generalization and early stopping **Question.** Distinguish underfitting and overfitting. Explain how validation data and early stopping help control overfitting.\n\n**Model answer.**\nUnderfitting occurs when a model has not captured enough useful structure in the training data. Both training and validation errors are usually high relative to a sensible task baseline. Causes can include insufficient capacity, inadequate features, excessive regularization or incomplete optimization.\n\n\n\n\nOverfitting occurs when a model fits training-specific details or noise that do not generalize. Training error is low while validation error is substantially higher. A mismatch between training and validation data can produce a similar gap, so the split must also be examined.\n\n### How early stopping works\n1. Train using the training set; do not learn parameters from the validation targets as training samples.\n2. After an epoch or another chosen interval, evaluate validation loss.\n3. Save the parameters whenever validation loss reaches a new best value.\n4. Count consecutive intervals with no qualifying improvement.\n5. When the count reaches the chosen patience, stop and restore the saved best parameters.\n\nFor validation losses 0.8, 0.6, 0.5, 0.53 and 0.55, epoch 3 is best. With patience 2, epochs 4 and 5 supply two non-improvements: stop after epoch 5 and restore epoch 3. The stopping epoch and the retained model epoch are different.\n\nOther remedies for overfitting include appropriate L1/L2 penalties, dropout, augmentation and more representative training data. Underfitting may instead require a better model, adequate training or less regularization. Use the test set for final evaluation after those choices, not for repeatedly choosing the stopping point.\n"},{"title":"Choosing a remedy","blocks":["For overfitting: try L1/L2, dropout, suitable data augmentation, a smaller model, more representative training data and early stopping. For underfitting: improve features or architecture, train adequately, check optimization, or reduce excessive regularization. Evaluate changes on validation data.
","Data augmentation makes label-preserving variations of training examples, such as suitable crops or brightness changes in images. It must respect the task: flipping a digit or a medical image may change meaning. Augmentation happens within the training split; do not leak augmented copies into validation.
","| Change | Usual bias tendency | Usual variance tendency |
|---|---|---|
| Stronger weight regularization | Increase | Decrease |
| More hidden units with other factors fixed | Decrease | Increase |
| Dropout | Increase | Decrease |
| More representative data from same distribution | Roughly unchanged model bias | Decrease |
These are classical tendencies for exam discussion, not exact laws for every deep-network training run. Distinguish model capacity from optimization failure.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c6-s6","search":"Choosing a remedy For overfitting: try L1/L2, dropout, suitable data augmentation, a smaller model, more representative training data and early stopping. For underfitting: improve features or architecture, train adequately, check optimization, or reduce excessive regularization. Evaluate changes on validation data.\n\n\n\nData augmentation makes label-preserving variations of training examples, such as suitable crops or brightness changes in images. It must respect the task: flipping a digit or a medical image may change meaning. Augmentation happens within the training split; do not leak augmented copies into validation.\n| Change | Usual bias tendency | Usual variance tendency |\n| Stronger weight regularization | Increase | Decrease |\n| More hidden units with other factors fixed | Decrease | Increase |\n| Dropout | Increase | Decrease |\n| More representative data from same distribution | Roughly unchanged model bias | Decrease |\nThese are classical tendencies for exam discussion, not exact laws for every deep-network training run. Distinguish model capacity from optimization failure.\n\n"},{"title":"Inverted dropout during training and inference","blocks":["Definition. Inverted dropout divides retained training activations by their keep probability so that their expectation equals the original activation. At inference, dropout is disabled and no compensating activation scaling is required.
","Inverted dropout first chooses which activations to keep, then divides each kept activation by the keep probability. This compensates for the missing activations in expectation. A dropped activation becomes zero.
","For activation 3 and keep probability q = 0.5, a kept value becomes 3 ÷ 0.5 = 6. A dropped value becomes 0. Since half the masks keep it, the average is 0.5 × 6 + 0.5 × 0 = 3, equal to the original activation.
","At inference, dropout is disabled and the ordinary activation 3 is used. Under this inverted-dropout convention, no extra multiplication by q is required at inference.
","For a fixed activation a and a Bernoulli keep mask m with P(m = 1) = q:
","![\"\\mathbb](\"assets/equations/8a9a35419aebc53b.svg\")
The equality concerns the activation's expectation over masks. It does not say every sampled layer preserves its sum, nor does it generally make a nonlinear network's complete stochastic prediction equal to its deterministic inference prediction.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c6-s7","search":"Inverted dropout during training and inference **Definition.** Inverted dropout divides retained training activations by their keep probability so that their expectation equals the original activation. At inference, dropout is disabled and no compensating activation scaling is required.\n\n### Explanation and working\nInverted dropout first chooses which activations to keep, then divides each kept activation by the keep probability. This compensates for the missing activations in expectation. A dropped activation becomes zero.\n\n\n\nFor activation 3 and keep probability q = 0.5, a kept value becomes 3 ÷ 0.5 = 6. A dropped value becomes 0. Since half the masks keep it, the average is 0.5 × 6 + 0.5 × 0 = 3, equal to the original activation.\nAt inference, dropout is disabled and the ordinary activation 3 is used. Under this inverted-dropout convention, no extra multiplication by q is required at inference.\n\n### Expectation calculation\nFor a fixed activation a and a Bernoulli keep mask m with P(m = 1) = q:\n\nThe equality concerns the activation's expectation over masks. It does not say every sampled layer preserves its sum, nor does it generally make a nonlinear network's complete stochastic prediction equal to its deterministic inference prediction.\n"},{"title":"Model answer: dropout with training and inference explained","question":["Question. Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling.
"],"answer":["Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights.
","Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop.
","![\"a'=\\frac{ma}{q}\"](\"assets/equations/fda2e84f024891bd.svg\")
A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks.
","For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is:
","![\"0.5(6)+0.5(0)=3\"](\"assets/equations/83765955bdb2c901.svg\")
Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions.
","Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data.
"],"kind":"theory","id":"mt2-c6-s8","search":"Model answer: dropout with training and inference explained **Question.** Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling.\n\n**Model answer.**\nDropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights.\n\n\n\n\nLet p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop.\n\n### Inverted dropout during training\n\nA kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks.\n\nFor a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is:\n\n### Inference\nDisable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions.\n\nDropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data.\n"}]},{"n":7,"title":"Convolutional neural networks","sections":[{"title":"Reading image shapes before doing CNN arithmetic","blocks":["Definition. An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension.
","The shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors.
","A filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels.
","A 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels.
","The spatial-size formula computes height and width. It does not compute the channel count.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\"](\"assets/equations/930fa72046a69449.svg\")
N = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position.
","Shape counts the arrangement of output values. Parameters count learned weights and biases. Computation counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights.
","In a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s0","search":"Reading image shapes before doing CNN arithmetic **Definition.** An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension.\n\n### Explanation and working\n### Height, width and channels\nThe shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors.\n\n\n\n### One position versus the whole map\nA filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels.\n### Why filter depth matches input channels\nA 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels.\n### Reading a complete shape calculation\nThe spatial-size formula computes height and width. It does not compute the channel count.\n\nN = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position.\n### Shape, parameters and computation are three different counts\n**Shape** counts the arrangement of output values. **Parameters** count learned weights and biases. **Computation** counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights.\nIn a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count.\n"},{"title":"A computer sees an image as numbers","blocks":["Definition. A digital image is a spatially arranged numerical tensor. In a CNN, convolutions transform input channels into learned feature maps using local connectivity and spatially shared kernel parameters.
","A pixel is one position in an image. A grayscale image stores one intensity per position. A colour RGB image usually stores three numbers per position: red, green and blue. Therefore 32 × 32 × 3 means height 32, width 32 and 3 input channels. It contains 32 × 32 × 3 = 3,072 numbers, not 32 neurons.
","A feature is a useful pattern, such as an edge or texture. A feature map is a grid of responses that says where a learned filter matches the input. Early CNN layers often detect local simple patterns; later layers combine them into parts and more complex patterns.
","A filter, also called a kernel in many exam questions, is a small set of learned weights. Slide it across the image, multiply corresponding numbers, and add. The same weights are reused at every spatial position. This is weight sharing.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s1","search":"A computer sees an image as numbers **Definition.** A digital image is a spatially arranged numerical tensor. In a CNN, convolutions transform input channels into learned feature maps using local connectivity and spatially shared kernel parameters.\n\n### Explanation and working\nA pixel is one position in an image. A grayscale image stores one intensity per position. A colour RGB image usually stores three numbers per position: red, green and blue. Therefore 32 × 32 × 3 means height 32, width 32 and 3 input channels. It contains 32 × 32 × 3 = 3,072 numbers, not 32 neurons.\n\n\n\nA **feature** is a useful pattern, such as an edge or texture. A **feature map** is a grid of responses that says where a learned filter matches the input. Early CNN layers often detect local simple patterns; later layers combine them into parts and more complex patterns.\nA filter, also called a kernel in many exam questions, is a small set of learned weights. Slide it across the image, multiply corresponding numbers, and add. The same weights are reused at every spatial position. This is **weight sharing**."},{"title":"The convolution calculation","blocks":["Definition. A convolutional layer calculates local weighted sums with a spatially shared kernel and optional bias. Standard deep-learning implementations ordinarily use cross-correlation, without flipping the kernel; examination calculations must follow the stated convention.
","In deep-learning software, the operation normally called convolution is cross-correlation: use the kernel as written, without flipping it. This is the convention in this course’s CNN numericals and the supplied latest answer key. Mathematical convolution would rotate the kernel; state that convention if a question explicitly requests it.
","Question. Apply a 2 × 2 kernel Matrix 1 shown below to the 3 × 3 image Matrix 2 shown below, with stride 1, no padding and zero bias.
","Input X = ![\"\\begin{bmatrix}1](\"assets/equations/0cf23d37eabd00e4.svg\")
.
Kernel K = ![\"\\begin{bmatrix}1](\"assets/equations/5c899cfe4dee6d06.svg\")
.
Stride = 1; padding = 0; bias = 0; no kernel flip.
","Find the 2 × 2 output map.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","![\"Y_{ij}=\\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b\"](\"assets/equations/6652d02289b39594.svg\")
Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","![\"X=\\begin{bmatrix}1&2&3\\\\4&5&6\\\\7&8&9\\end{bmatrix},\\quad](\"assets/equations/fbe5e0aa8714ed4e.svg\")
Step 1. Top-left window. 1(1) + 2(0) + 4(0) + 5(−1) = 1 − 5 = −4.
","Step 2. Move right by one. 2(1) + 3(0) + 5(0) + 6(−1) = 2 − 6 = −4.
","Step 3. Move down and return left. 4(1) + 5(0) + 7(0) + 8(−1) = 4 − 8 = −4.
","Step 4. Bottom-right window. 5(1) + 6(0) + 8(0) + 9(−1) = 5 − 9 = −4. Output is ![\"\\begin{bmatrix}-4](\"assets/equations/7f45e7b63383ada2.svg\")
. If ReLU follows, that next layer changes all four entries to zero; do not silently apply it when it was not requested.
Definition. Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output.
","Stride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension.
","N is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: ![\"\\left\\lfloor](\"assets/equations/a8acf65164593cb2.svg\")
= 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels.
Why plus one? Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1.
","Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.
","N = 6; K = 3; S = 2; P = 0; dilation = 1.
","The integer output side length.
","The formula counts complete kernel placements. Flooring removes an incomplete final placement.
","N is input side; P is padding on each side; K is kernel side; S is stride.
","Yaad rakhne ka cue: Padding dono side; stride se divide; floor; phir +1.
","Step 1: include padding and subtract kernel size.
","![\"6+2(0)-3=3\"](\"assets/equations/b4027b9bc8f6fe22.svg\")
Step 2: divide by the stride.
","![\"\\frac32=1.5\"](\"assets/equations/d103d3b0cd150006.svg\")
Step 3: take the floor.
","![\"\\lfloor1.5\\rfloor=1\"](\"assets/equations/2ff7c16eb83493b5.svg\")
Step 4: count the first placement.
","![\"N_{\\mathrm{out}}=1+1=2\"](\"assets/equations/6f3dfc60f0024b2c.svg\")
Final answer: Output side = 2; a square input gives a 2 × 2 map per filter.
","Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.
","N = 5; K = 3; S = 1; P = 1 on each side.
","Padded input and output dimensions.
","Padding adds P entries on each of two opposite sides; the output counts complete kernel placements.
","N = unpadded side; K = kernel side; S = stride; P = per-side padding.
","![\"N_{\\mathrm{padded}}=N+2P\"](\"assets/equations/e6bbc5ec5f7e71fe.svg\")
![\"N_{\\mathrm{out}}=\\lfloor(N+2P-K)/S\\rfloor+1\"](\"assets/equations/500258811ea3eb2f.svg\")
Yaad rakhne ka cue: Padded input aur output ka size alag cheez hain.
","Step 1: count both padding edges.
","![\"2P=2(1)=2\"](\"assets/equations/741118212eb25f73.svg\")
Step 2: calculate the padded input side.
","![\"N+2P=5+2=7\"](\"assets/equations/a0c02a700e3aef2c.svg\")
Step 3: subtract kernel width.
","![\"7-3=4\"](\"assets/equations/3b2ddd3dd39f8f0c.svg\")
Step 4: divide by stride and take the floor.
","![\"\\lfloor4/1\\rfloor=4\"](\"assets/equations/5ce7e863527f4bb8.svg\")
Step 5: include the first placement.
","![\"N_{out}=4+1=5\"](\"assets/equations/94437545b203f62e.svg\")
Final answer: Padded input = 7 × 7; output = 5 × 5 per filter.
","“Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s3","search":"Stride padding and floor **Definition.** Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output.\n\n### Explanation and working\nStride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension.\n\n\n\n\nN is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: \\(\\left\\lfloor 2.5\\right\\rfloor\\) = 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels.\n**Why plus one?** Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1.\n\n\n### Output dimension with a fractional intermediate result\n\n**Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.**\n\n\n\n### Given\n\nN = 6; K = 3; S = 2; P = 0; dilation = 1.\n\n### To find\n\nThe integer output side length.\n\n### Formula — which one and why\n\nThe formula counts complete kernel placements. Flooring removes an incomplete final placement.\n\nN is input side; P is padding on each side; K is kernel side; S is stride.\n\n\n\n**Yaad rakhne ka cue:** Padding dono side; stride se divide; floor; phir +1.\n\n### Solution — step by step\n\n\n\n**Step 1: include padding and subtract kernel size.**\n\n\n\n\n\n\n\n**Step 2: divide by the stride.**\n\n\n\n\n\n\n\n**Step 3: take the floor.**\n\n\n\n\n\n\n\n**Step 4: count the first placement.**\n\n\n\n\n\n\n\n**Final answer:** Output side = 2; a square input gives a 2 × 2 map per filter.\n\n\n\n\n### Padding on both sides\n\n**Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.**\n\n\n\n### Given\n\nN = 5; K = 3; S = 1; P = 1 on each side.\n\n### To find\n\nPadded input and output dimensions.\n\n### Formula — which one and why\n\nPadding adds P entries on each of two opposite sides; the output counts complete kernel placements.\n\nN = unpadded side; K = kernel side; S = stride; P = per-side padding.\n\n\n\n\n\n**Yaad rakhne ka cue:** Padded input aur output ka size alag cheez hain.\n\n### Solution — step by step\n\n\n\n**Step 1: count both padding edges.**\n\n\n\n\n\n\n\n**Step 2: calculate the padded input side.**\n\n\n\n\n\n\n\n**Step 3: subtract kernel width.**\n\n\n\n\n\n\n\n**Step 4: divide by stride and take the floor.**\n\n\n\n\n\n\n\n**Step 5: include the first placement.**\n\n\n\n\n\n\n\n**Final answer:** Padded input = 7 × 7; output = 5 × 5 per filter.\n\n\n“Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case."},{"title":"Convolution over several channels","blocks":["Definition. A standard convolutional filter spans every input channel. At one spatial position, channel-wise weighted sums are accumulated and one filter bias is added, producing one scalar output in that filter's feature map.
","A 3 × 3 filter receiving RGB input has depth 3, so it contains 3 × 3 × 3 weights. At one position, compute a weighted sum in each of the three channels, add those sums, and then add the filter’s one bias. One filter produces one output channel. Sixteen filters produce sixteen output channels.
","Tiny example. At one position the red, green and blue channel sums are 2, −1 and 4. With bias 0.5, the output is 2 − 1 + 4 + 0.5 = 5.5. Do not keep three separate outputs for this one filter; they have been added.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s4","search":"Convolution over several channels **Definition.** A standard convolutional filter spans every input channel. At one spatial position, channel-wise weighted sums are accumulated and one filter bias is added, producing one scalar output in that filter's feature map.\n\n### Explanation and working\nA 3 × 3 filter receiving RGB input has depth 3, so it contains 3 × 3 × 3 weights. At one position, compute a weighted sum in each of the three channels, add those sums, and then add the filter’s one bias. One filter produces one output channel. Sixteen filters produce sixteen output channels.\n\n\n\n**Tiny example.** At one position the red, green and blue channel sums are 2, −1 and 4. With bias 0.5, the output is 2 − 1 + 4 + 0.5 = 5.5. Do not keep three separate outputs for this one filter; they have been added."},{"title":"One filter over multiple channels: all contributions go into one cell","blocks":["Definition. Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels.
","Imagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3.
","![\"z=0.1(2)+(-0.2)(5)+0.5(1)+0.3\"](\"assets/equations/86f1a8b54476b2cb.svg\")
![\"z=0.2-1+0.5+0.3=0\"](\"assets/equations/8c377dccccbc387a.svg\")
The three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels.
","A 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias.
","The channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified.
","Each filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth.
","For three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters.
","A 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed.
","This is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s5","search":"One filter over multiple channels: all contributions go into one cell **Definition.** Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels.\n\n### Explanation and working\n### 1. Separate spatial size from depth\nImagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3.\n\n\n\n\n\nThe three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels.\n\n### 2. A larger filter has one spatial grid per input channel\nA 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias.\nThe channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified.\n\n### 3. What changes when there are several filters?\nEach filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth.\nFor three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters.\n\n### 4. One-by-one does not mean one weight\nA 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed.\nThis is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately.\n"},{"title":"Choosing more or fewer filters","blocks":["Definition. The number of convolutional filters determines the number of output channels. Kernel height, kernel width, stride and padding determine spatial processing; input-channel count determines each filter's depth.
","Each filter produces one output channel and can learn a different pattern. More filters increase representational capacity, output depth, parameter count and activation memory. They may improve performance but can overfit small datasets or exceed a compute budget. Choose counts using validation results and resource constraints. More filters in one layer increase its width, not the number of network layers.
","Definition. Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases.
","Max pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged.
","Question. Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding:
","The 4 × 4 input map is printed below.
","Window = 2 × 2; stride = 2; padding = 0.
","Find the pooled output by the operation(s) requested.
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","![\"y_{\\max}=\\max](\"assets/equations/e971be750f6cd770.svg\")
![\"y_{\\mathrm{avg}}=\\frac1n\\sum_{v\\in](\"assets/equations/7e5c101541d074f2.svg\")
Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","![\"A=\\begin{bmatrix}21&11&15&10\\\\12&9&10&11\\\\11&13&18&12\\\\7&4&15&12\\end{bmatrix}\"](\"assets/equations/dbeeead3eba02270.svg\")
Step 1: select the four entries.
","[21, 11, 12, 9]
","Step 2: choose the largest entry.
","![\"y_{\\max}=21\"](\"assets/equations/cf103da2c4c32945.svg\")
Step 3: add all entries.
","![\"21+11+12+9=53\"](\"assets/equations/6f9a0a96d6f9e048.svg\")
Step 4: divide the sum by four for the average.
","![\"y_{\\mathrm{avg}}=\\frac{53}4=13.25\"](\"assets/equations/a3fb67b8edc22789.svg\")
Step 1: select the four entries.
","[15, 10, 10, 11]
","Step 2: choose the largest entry.
","![\"y_{\\max}=15\"](\"assets/equations/6a17ef02b89e67e2.svg\")
Step 3: add all entries.
","![\"15+10+10+11=46\"](\"assets/equations/18d73c8a689f86bd.svg\")
Step 4: divide the sum by four for the average.
","![\"y_{\\mathrm{avg}}=\\frac{46}4=11.5\"](\"assets/equations/4bcedeb9a26a70b6.svg\")
Step 1: select the four entries.
","[11, 13, 7, 4]
","Step 2: choose the largest entry.
","![\"y_{\\max}=13\"](\"assets/equations/37fcb9070f6a3c0a.svg\")
Step 3: add all entries.
","![\"11+13+7+4=35\"](\"assets/equations/ba50b6f7e4060a25.svg\")
Step 4: divide the sum by four for the average.
","![\"y_{\\mathrm{avg}}=\\frac{35}4=8.75\"](\"assets/equations/90fc093dce5b3464.svg\")
Step 1: select the four entries.
","[18, 12, 15, 12]
","Step 2: choose the largest entry.
","![\"y_{\\max}=18\"](\"assets/equations/6e005f373e798e70.svg\")
Step 3: add all entries.
","![\"18+12+15+12=57\"](\"assets/equations/437a8b87df7ffce7.svg\")
Step 4: divide the sum by four for the average.
","![\"y_{\\mathrm{avg}}=\\frac{57}4=14.25\"](\"assets/equations/aaffd26053e275ce.svg\")
Outputs are max ![\"\\begin{bmatrix}21](\"assets/equations/89ec596cf1c6ec68.svg\")
and average ![\"\\begin{bmatrix}13.25](\"assets/equations/94682b0fab3f0b97.svg\")
.
Pooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s7","search":"Pooling needs comparisons or averages **Definition.** Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases.\n\n### Explanation and working\nMax pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged.\n**Question.** Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding:\n\n\n\n### Given\n\nThe 4 × 4 input map is printed below.\n\nWindow = 2 × 2; stride = 2; padding = 0.\n\n### To find\n\nFind the pooled output by the operation(s) requested.\n\n### Formula — which one and why\n\n**Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.\n\nV = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.\n\n\n\n\n\n**Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count.\n\n### Solution — step by step\n\n\n\n\n\n### Top-left window\n\n**Step 1: select the four entries.**\n\n[21, 11, 12, 9]\n\n**Step 2: choose the largest entry.**\n\n\n\n\n\n\n\n**Step 3: add all entries.**\n\n\n\n\n\n\n\n**Step 4: divide the sum by four for the average.**\n\n\n\n\n\n\n\n\n### Top-right window\n\n**Step 1: select the four entries.**\n\n[15, 10, 10, 11]\n\n**Step 2: choose the largest entry.**\n\n\n\n\n\n\n\n**Step 3: add all entries.**\n\n\n\n\n\n\n\n**Step 4: divide the sum by four for the average.**\n\n\n\n\n\n\n\n\n### Bottom-left window\n\n**Step 1: select the four entries.**\n\n[11, 13, 7, 4]\n\n**Step 2: choose the largest entry.**\n\n\n\n\n\n\n\n**Step 3: add all entries.**\n\n\n\n\n\n\n\n**Step 4: divide the sum by four for the average.**\n\n\n\n\n\n\n\n\n### Bottom-right window\n\n**Step 1: select the four entries.**\n\n[18, 12, 15, 12]\n\n**Step 2: choose the largest entry.**\n\n\n\n\n\n\n\n**Step 3: add all entries.**\n\n\n\n\n\n\n\n**Step 4: divide the sum by four for the average.**\n\n\n\n\n\n\n\n\nOutputs are max and average .\n\nPooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance."},{"title":"Parameters are different from activation values","blocks":["Definition. Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters.
","An activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights.
","![\"P_{\\mathrm{conv}}=(K_hK_wC_{\\mathrm{in}}+1)C_{\\mathrm{out}}\"](\"assets/equations/859b60e888192c5a.svg\")
The +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation.
","Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.
","Kernel = 3 × 3; input channels = 3; filters = 16; one bias per filter.
","Weights, biases, total trainable parameters and output channels.
","Each filter spans every input channel and owns one bias. Spatial reuse does not create extra weights.
","Kₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters.
","![\"P=(K_hK_wC_{\\mathrm{in}}+1)C_{\\mathrm{out}}\"](\"assets/equations/5a603987eb773068.svg\")
Yaad rakhne ka cue: Ek filter ka count banao, phir filters se multiply.
","Step 1: count weights in one filter.
","![\"3(3)(3)=27\"](\"assets/equations/ee13fc57d61624d3.svg\")
Step 2: include its one bias.
","![\"27+1=28\"](\"assets/equations/5f8acbeb6a1db0ac.svg\")
Step 3: count all weights.
","![\"27(16)=432\"](\"assets/equations/f7cebde5567e220a.svg\")
Step 4: count all biases.
","![\"1(16)=16\"](\"assets/equations/96319009caa148f5.svg\")
Step 5: add and state channel count.
","![\"P=432+16=448\"](\"assets/equations/17c5c3c49aac5f53.svg\")
Each filter produces one channel, so output channels = 16.
","Final answer: 432 weights + 16 biases = 448 trainable parameters; 16 output channels.
","![\"P_{\\mathrm{pool}}=0,\\quad](\"assets/equations/676bf495296f4651.svg\")
Flatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s8","search":"Parameters are different from activation values **Definition.** Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters.\n\n### Explanation and working\nAn activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights.\n\n\n\n\nThe +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation.\n\n\n### One filter, then all filters\n\n**Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.**\n\n\n\n### Given\n\nKernel = 3 × 3; input channels = 3; filters = 16; one bias per filter.\n\n### To find\n\nWeights, biases, total trainable parameters and output channels.\n\n### Formula — which one and why\n\nEach filter spans every input channel and owns one bias. Spatial reuse does not create extra weights.\n\nKₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters.\n\n\n\n**Yaad rakhne ka cue:** Ek filter ka count banao, phir filters se multiply.\n\n### Solution — step by step\n\n\n\n**Step 1: count weights in one filter.**\n\n\n\n\n\n\n\n**Step 2: include its one bias.**\n\n\n\n\n\n\n\n**Step 3: count all weights.**\n\n\n\n\n\n\n\n**Step 4: count all biases.**\n\n\n\n\n\n\n\n**Step 5: add and state channel count.**\n\n\n\n\n\nEach filter produces one channel, so output channels = 16.\n\n**Final answer:** 432 weights + 16 biases = 448 trainable parameters; 16 output channels.\n\n\n\nFlatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values."},{"title":"One complete CNN pipeline","blocks":["Question. Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron.
","Input = 128 × 128 × 3.
","Conv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2.
","Conv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2.
","Flatten → dense 240 → dense 120 → dense 5.
","One bias per learned filter/dense neuron.
","Find every output shape and layer parameter count, then the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{N+2P_{\\mathrm{pad}}-K}{S}\\right\\rfloor+1\"](\"assets/equations/cb685a42551287a1.svg\")
![\"P_{\\mathrm{conv}}=(K^2C_{\\mathrm{in}}+1)C_{\\mathrm{out}}\"](\"assets/equations/f3eea0634c9f3976.svg\")
![\"n_{\\mathrm{flat}}=HWC\"](\"assets/equations/1321cd9ddfcf5b42.svg\")
![\"P_{\\mathrm{dense}}=(n_{\\mathrm{in}}+1)n_{\\mathrm{out}}\"](\"assets/equations/c0e2ff46bbfc4867.svg\")
![\"P_{\\mathrm{pool}}=P_{\\mathrm{flatten}}=0\"](\"assets/equations/cd31b9774832f122.svg\")
Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Step 1. Conv 1 size. (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448.
","Step 2. Pool 1 size. ![\"\\left\\lfloor\\frac{126](\"assets/equations/9e62fef4c3e773c7.svg\")
= 63; output 63 × 63 × 16. Parameters = 0.
Step 3. Conv 2 size. (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640.
","Step 4. Pool 2 size. ![\"\\left\\lfloor\\frac{61](\"assets/equations/4d2232cedf251bfa.svg\")
= ![\"\\left\\lfloor](\"assets/equations/dfb80f08b9b0c4e1.svg\")
+ 1 = 30; output 30 × 30 × 32. Parameters = 0.
Step 5. Flatten. 30 × 30 × 32 = 28,800 values. Parameters = 0.
","Step 6. Dense 240. 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240.
","Step 7. Dense 120. (240 + 1) × 120 = 28,920.
","Step 8. Classifier. (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer.
","Step 9. Total. 448 + 4,640 + 6,912,240 + 28,920 + 605 = 6,946,853. Most parameters come from the first dense layer.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s9","search":"One complete CNN pipeline **Question.** Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron.\n\n\n\n### Given\n\nInput = 128 × 128 × 3.\n\nConv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2.\n\nConv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2.\n\nFlatten → dense 240 → dense 120 → dense 5.\n\nOne bias per learned filter/dense neuron.\n\n### To find\n\nFind every output shape and layer parameter count, then the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n**Step 1. Conv 1 size.** (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448.\n\n\n\n**Step 2. Pool 1 size.** \\(\\left\\lfloor\\frac{126 - 2}{2}\\right\\rfloor+1\\) = 63; output 63 × 63 × 16. Parameters = 0.\n**Step 3. Conv 2 size.** (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640.\n**Step 4. Pool 2 size.** \\(\\left\\lfloor\\frac{61 - 2}{2}\\right\\rfloor+1\\) = \\(\\left\\lfloor 29.5\\right\\rfloor\\) + 1 = 30; output 30 × 30 × 32. Parameters = 0.\n**Step 5. Flatten.** 30 × 30 × 32 = 28,800 values. Parameters = 0.\n**Step 6. Dense 240.** 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240.\n**Step 7. Dense 120.** (240 + 1) × 120 = 28,920.\n**Step 8. Classifier.** (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer.\n**Step 9. Total.** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853**. Most parameters come from the first dense layer."},{"title":"CNN versus a fully connected image network","blocks":["Definition. A convolutional neural network uses local receptive fields and spatial parameter sharing, unlike a dense layer's independent connection weights. These architectural constraints exploit spatial structure and can greatly reduce parameter counts.
","A fully connected layer can learn image features, but flattening pixels and connecting every pixel to every neuron ignores the useful local structure built into a CNN and can require many parameters. CNNs exploit local connections and shared filters. They learn hierarchical features and reuse a pattern detector at many locations.
","For a 32 × 32 × 3 input, a dense layer with 100 neurons has (3,072 + 1) × 100 = 307,300 parameters. A convolution with 100 filters of size 3 × 3 has (27 + 1) × 100 = 2,800 parameters. The outputs are different shapes, so this is a parameter-efficiency illustration, not two identical computational tasks.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s10","search":"CNN versus a fully connected image network **Definition.** A convolutional neural network uses local receptive fields and spatial parameter sharing, unlike a dense layer's independent connection weights. These architectural constraints exploit spatial structure and can greatly reduce parameter counts.\n\n### Explanation and working\nA fully connected layer can learn image features, but flattening pixels and connecting every pixel to every neuron ignores the useful local structure built into a CNN and can require many parameters. CNNs exploit local connections and shared filters. They learn hierarchical features and reuse a pattern detector at many locations.\n\n\n\nFor a 32 × 32 × 3 input, a dense layer with 100 neurons has (3,072 + 1) × 100 = 307,300 parameters. A convolution with 100 filters of size 3 × 3 has (27 + 1) × 100 = 2,800 parameters. The outputs are different shapes, so this is a parameter-efficiency illustration, not two identical computational tasks.\n"},{"title":"Model answer: CNN structure and parameter sharing","question":["Question. Explain the main layers of a convolutional neural network and why CNNs are suitable for image data. Compare them with a fully connected image network.
"],"answer":["A CNN processes spatially arranged data using filters that examine local regions. The same filter weights are reused at different positions. This combination of local connectivity and weight sharing builds an image-specific structure into the model and can greatly reduce the number of learned weights.
","Input → convolution → nonlinear activation → optional pooling → further feature extraction → classifier.
","| Layer | Operation | Learned parameters? |
|---|---|---|
| Convolution | Multiply a local window by a filter, sum and add its bias | Filter weights and usually one bias per filter |
| ReLU | Replace negative values by zero | No ordinary ReLU parameters |
| Pooling | Summarize each window by maximum or average | None for ordinary max/average pooling |
| Flatten | Rearrange H × W × C values into a vector | None |
| Dense layer | Form weighted combinations of all incoming values | Weights and receiving-neuron biases |
| Softmax | Convert class scores into probabilities | No additional weights of its own |
Early filters can learn local patterns such as edges. Later layers combine features over larger receptive fields. A filter responding at several image positions uses one shared set of weights, not a new set at every location.
","Flattening an image and using a dense layer does not enforce this local, shared structure. A dense layer has a separate connection weight for each input–output pair. CNNs can therefore be more parameter-efficient for visual patterns, although they still need suitable data and training. Pooling can provide some robustness to small shifts; it does not guarantee complete invariance to every translation.
"],"kind":"theory","id":"mt2-c7-s11","search":"Model answer: CNN structure and parameter sharing **Question.** Explain the main layers of a convolutional neural network and why CNNs are suitable for image data. Compare them with a fully connected image network.\n\n**Model answer.**\nA CNN processes spatially arranged data using filters that examine local regions. The same filter weights are reused at different positions. This combination of local connectivity and weight sharing builds an image-specific structure into the model and can greatly reduce the number of learned weights.\n\n\n\n\n### Typical layer sequence\n**Input → convolution → nonlinear activation → optional pooling → further feature extraction → classifier.**\n\n| Layer | Operation | Learned parameters? |\n| Convolution | Multiply a local window by a filter, sum and add its bias | Filter weights and usually one bias per filter |\n| ReLU | Replace negative values by zero | No ordinary ReLU parameters |\n| Pooling | Summarize each window by maximum or average | None for ordinary max/average pooling |\n| Flatten | Rearrange H × W × C values into a vector | None |\n| Dense layer | Form weighted combinations of all incoming values | Weights and receiving-neuron biases |\n| Softmax | Convert class scores into probabilities | No additional weights of its own |\n\nEarly filters can learn local patterns such as edges. Later layers combine features over larger receptive fields. A filter responding at several image positions uses one shared set of weights, not a new set at every location.\n\nFlattening an image and using a dense layer does not enforce this local, shared structure. A dense layer has a separate connection weight for each input–output pair. CNNs can therefore be more parameter-efficient for visual patterns, although they still need suitable data and training. Pooling can provide some robustness to small shifts; it does not guarantee complete invariance to every translation.\n"},{"title":"Output size and parameter count are different","blocks":["Definition. Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters.
","Question. A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias.
","Input = 7 × 7 × 3; two 3 × 3 filters.
","Stride = 2; padding = 1; include biases.
","Find the output shape and trainable parameter count.
","Convolution/pooling spatial size. Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.
","N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.
","Yaad rakhne ka cue: Input + dono sides ki padding − kernel; stride se divide; floor + 1.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","![\"N_w=K_hK_wC_{\\mathrm{in}}C_{\\mathrm{out}}\"](\"assets/equations/74209557d398075e.svg\")
![\"N_b=C_{\\mathrm{out}}\"](\"assets/equations/614d1f9d70b9928b.svg\")
![\"P=N_w+N_b\"](\"assets/equations/61924cb4898d8799.svg\")
Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","Step 1 spatial size: padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2.
","Step 2 parameters: one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s12","search":"Output size and parameter count are different **Definition.** Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters.\n\n### Explanation and working\n**Question.** A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias.\n\n\n\n### Given\n\nInput = 7 × 7 × 3; two 3 × 3 filters.\n\nStride = 2; padding = 1; include biases.\n\n### To find\n\nFind the output shape and trainable parameter count.\n\n### Formula — which one and why\n\n**Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.\n\nN = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.\n\n\n\n**Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n**Step 1 spatial size:** padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2.\n\n\n\n**Step 2 parameters:** one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters.\n\n"},{"title":"Model answer: shape, activations and parameters","question":["Question. Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them.
"],"answer":["Output shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions.
","For an ordinary convolution with dilation 1:
","N is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters.
","![\"A=H_{\\mathrm{out}}W_{\\mathrm{out}}C_{\\mathrm{out}}\"](\"assets/equations/8c9b86a3e71336c4.svg\")
This is the number of output values for one example, excluding any batch dimension.
","Each filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions.
","A 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels.
"],"kind":"theory","id":"mt2-c7-s13","search":"Model answer: shape, activations and parameters **Question.** Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them.\n\n**Model answer.**\nOutput shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions.\n\n\n\n\n### Spatial output shape\nFor an ordinary convolution with dilation 1:\n\nN is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters.\n\n### Output activations\n\nThis is the number of output values for one example, excluding any batch dimension.\n\n### Trainable parameters\n\nEach filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions.\n\n### Small example\nA 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels.\n"},{"title":"Faculty example: a complete binary-image convolution","blocks":["Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)
","Image X = ![\"\\begin{bmatrix}1](\"assets/equations/acdfbb90974d971f.svg\")
.
Filter K = ![\"\\begin{bmatrix}1](\"assets/equations/305a3f29286c4e55.svg\")
. These are the values in the Unit 4 lecture example.
The output shape and all nine feature-map values.
","The output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input.
","Here N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients.
","For this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always top-left, top-right, centre, bottom-left, bottom-right within the current window.
","Step 1: find the shape. (5 − 3)/1 + 1 = 3. The output is 3 × 3.
","Step 2: top-left output, showing all nine products. 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = 4.
","Step 3: top-middle output. Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = 3.
","Step 4: top-right output. Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = 4.
","Step 5: middle-left output. Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = 2.
","Step 6: middle-middle output. Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = 4.
","Step 7: middle-right output. Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = 3.
","Step 8: bottom-left output. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = 2.
","Step 9: bottom-middle output. Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = 3.
","Step 10: bottom-right output. Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = 4.
","Final answer: Y = ![\"\\begin{bmatrix}4](\"assets/equations/12aa767f92286deb.svg\")
. All nine outputs use the same nine filter weights. They do not introduce nine different filters.
Yaad rakho: Window badalta hai; filter ke numbers same rehte hain.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s14","search":"Faculty example: a complete binary-image convolution **Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)**\n\n### Given\nImage X = .\n\nFilter K = . These are the values in the Unit 4 lecture example.\n\n### To find\nThe output shape and all nine feature-map values.\n\n### Formula — which one and why\nThe output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input.\n\n\n\nHere N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients.\n\n\n\nFor this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always **top-left, top-right, centre, bottom-left, bottom-right** within the current window.\n\n\n\n### Solution — step by step\n**Step 1: find the shape.** (5 − 3)/1 + 1 = 3. The output is 3 × 3.\n\n**Step 2: top-left output, showing all nine products.** 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = **4**.\n\n**Step 3: top-middle output.** Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = **3**.\n\n**Step 4: top-right output.** Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = **4**.\n\n**Step 5: middle-left output.** Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**.\n\n**Step 6: middle-middle output.** Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = **4**.\n\n**Step 7: middle-right output.** Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = **3**.\n\n**Step 8: bottom-left output.** Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**.\n\n**Step 9: bottom-middle output.** Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = **3**.\n\n**Step 10: bottom-right output.** Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = **4**.\n\n**Final answer:** Y = . All nine outputs use the same nine filter weights. They do not introduce nine different filters.\n\n**Yaad rakho:** Window badalta hai; filter ke numbers same rehte hain.\n"},{"title":"Faculty example: combine three RGB channel sums","blocks":["Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)
","Red patch = ![\"\\begin{bmatrix}0](\"assets/equations/54e18a92743f2660.svg\")
; red kernel = ![\"\\begin{bmatrix}-1](\"assets/equations/05d6756a9472b44c.svg\")
.
Green patch = ![\"\\begin{bmatrix}0](\"assets/equations/ca174e19344d6ef9.svg\")
; green kernel = ![\"\\begin{bmatrix}1](\"assets/equations/7b136c068c1d807b.svg\")
.
Blue patch = ![\"\\begin{bmatrix}0](\"assets/equations/8798bafaa1889aed.svg\")
; blue kernel = ![\"\\begin{bmatrix}0](\"assets/equations/fccc1fad371104fc.svg\")
. Bias = 1.
These are three slices of one full-depth filter, not three independent output filters.
","The contribution of each input channel and the final scalar output at this position.
","Multiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions.
","![\"z=s_R+s_G+s_B+b\"](\"assets/equations/15d63177c46d68fd.svg\")
The quantities s_R, s_G and s_B are intermediate weighted sums, not separate final output channels.
","Step 1: red-channel products. The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1.
","Step 2: red-channel sum. 156 − 155 + 153 + 154 = 308.
","Step 3: green-channel products. The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1).
","Step 4: green-channel sum. −167 −166 +0 −165 = −498.
","Step 5: blue-channel products. The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1.
","Step 6: blue-channel sum. 163 +0 −160 +161 = 164.
","Step 7: add the channel contributions. 308 −498 +164 = −26.
","Step 8: add one filter bias. −26 +1 = −25.
","Final answer: The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages.
","Yaad rakho: Teen channels ka contribution jodo; ek filter ka bias ek baar jodo.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s15","search":"Faculty example: combine three RGB channel sums **Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)**\n\n### Given\nRed patch = ; red kernel = .\n\nGreen patch = ; green kernel = .\n\nBlue patch = ; blue kernel = . Bias = 1.\n\nThese are three slices of **one** full-depth filter, not three independent output filters.\n\n### To find\nThe contribution of each input channel and the final scalar output at this position.\n\n### Formula — which one and why\nMultiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions.\n\n\n\nThe quantities s_R, s_G and s_B are intermediate weighted sums, not separate final output channels.\n\n### Solution — step by step\n**Step 1: red-channel products.** The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1.\n\n**Step 2: red-channel sum.** 156 − 155 + 153 + 154 = **308**.\n\n**Step 3: green-channel products.** The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1).\n\n**Step 4: green-channel sum.** −167 −166 +0 −165 = **−498**.\n\n**Step 5: blue-channel products.** The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1.\n\n**Step 6: blue-channel sum.** 163 +0 −160 +161 = **164**.\n\n**Step 7: add the channel contributions.** 308 −498 +164 = −26.\n\n**Step 8: add one filter bias.** −26 +1 = **−25**.\n\n**Final answer:** The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages.\n\n**Yaad rakho:** Teen channels ka contribution jodo; ek filter ka bias ek baar jodo.\n"},{"title":"Max, min and average pooling on the same lecture grid","blocks":["Definition. Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases.
","Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)
","X = ![\"\\begin{bmatrix}12](\"assets/equations/33f58372f0707267.svg\")
. Window = 2 × 2; stride = 2; padding = 0.
The shared output shape and each of the four cells for each pooling method.
","Each output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap.
","![\"y_{\\max}=\\max(x_1,x_2,x_3,x_4)\"](\"assets/equations/208abbe18f11b701.svg\")
![\"y_{\\min}=\\min(x_1,x_2,x_3,x_4)\"](\"assets/equations/619bb1c028de4233.svg\")
![\"y_{\\mathrm{avg}}=\\frac{x_1+x_2+x_3+x_4}{4}\"](\"assets/equations/0dcf475f3132c0a4.svg\")
Step 1: output size. (4 −2)/2 +1 = 2. Each output has shape 2 × 2.
","Step 2: top-left window [12,20,8,12]. Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13.
","Step 3: top-right window [30,0,2,0]. Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8.
","Step 4: bottom-left window [34,70,112,100]. Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79.
","Step 5: bottom-right window [37,4,25,12]. Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = 19.5. Do not silently round this to 20.
","Step 6: put each result back in its window's position. Max = ![\"\\begin{bmatrix}20](\"assets/equations/63c5529d08234533.svg\")
. Min = ![\"\\begin{bmatrix}8](\"assets/equations/54afd252549c8a81.svg\")
. Average = ![\"\\begin{bmatrix}13](\"assets/equations/7d1100eaf9980bd3.svg\")
.
Final answer: Three 2 × 2 maps as above; zero trainable parameters for each pooling operation.
","Max retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour.
","Pooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter.
","Yaad rakho: Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s16","search":"Max, min and average pooling on the same lecture grid **Definition.** Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases.\n\n**Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)**\n\n### Given\nX = . Window = 2 × 2; stride = 2; padding = 0.\n\n### To find\nThe shared output shape and each of the four cells for each pooling method.\n\n### Formula — which one and why\nEach output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap.\n\n\n\n\n\n\n\n### Solution — step by step\n**Step 1: output size.** (4 −2)/2 +1 = 2. Each output has shape 2 × 2.\n\n**Step 2: top-left window [12,20,8,12].** Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13.\n\n**Step 3: top-right window [30,0,2,0].** Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8.\n\n**Step 4: bottom-left window [34,70,112,100].** Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79.\n\n**Step 5: bottom-right window [37,4,25,12].** Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = **19.5**. Do not silently round this to 20.\n\n**Step 6: put each result back in its window's position.** Max = . Min = . Average = .\n\n**Final answer:** Three 2 × 2 maps as above; zero trainable parameters for each pooling operation.\n\n### Why the choice matters\nMax retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour.\n\nPooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter.\n\n**Yaad rakho:** Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide.\n"},{"title":"Faculty CNN table: shapes, activations and 62958 parameters","blocks":["Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)
","Input = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question.
","Output shape and number of output values for each layer, its trainable parameters, and the total parameters.
","Use the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias.
","A counts produced activations. P_conv and P_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here.
","Step 1: input. Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0.
","Step 2: Conv1 spatial size. (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272.
","Step 3: Conv1 parameters. One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = 608.
","Step 4: Pool1. (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight.
","Step 5: Conv2 spatial size. (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600.
","Step 6: Conv2 parameters. Its input has eight channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = 3,216.
","Step 7: Pool2 and flatten. (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = 400. Neither operation adds parameters.
","Step 8: dense 120. Weights = 400×120 = 48,000. Biases = 120. Parameters = 48,120. Output contains 120 activations.
","Step 9: dense 84. Weights = 120×84 = 10,080. Biases = 84. Parameters = 10,164. Output contains 84 activations.
","Step 10: dense 10 and softmax. Weights = 84×10 = 840. Biases = 10. Dense parameters = 850. Softmax converts the ten logits into ten probabilities and adds zero parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation.
","Step 11: add only the trainable counts. 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = 62,958.
","| Layer | Output shape | Activations | Parameters |
|---|---|---|---|
| Input | 32 × 32 × 3 | 3,072 | 0 |
| Conv1 | 28 × 28 × 8 | 6,272 | 608 |
| Pool1 | 14 × 14 × 8 | 1,568 | 0 |
| Conv2 | 10 × 10 × 16 | 1,600 | 3,216 |
| Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 |
| Dense 120 | 120 | 120 | 48,120 |
| Dense 84 | 84 | 84 | 10,164 |
| Dense 10 → softmax | 10 | 10 | 850 + 0 |
Final answer: The network has 62,958 trainable parameters. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network.
","Yaad rakho: Shape aage carry karo; weights aur activations ko alag columns mein rakho.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s17","search":"Faculty CNN table: shapes, activations and 62958 parameters **Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)**\n\n### Given\nInput = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question.\n\n### To find\nOutput shape and number of output values for each layer, its trainable parameters, and the total parameters.\n\n### Formula — which one and why\nUse the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias.\n\n\n\n\n\n\nA counts produced activations. P_conv and P_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here.\n\n### Solution — step by step\n**Step 1: input.** Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0.\n\n**Step 2: Conv1 spatial size.** (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272.\n\n**Step 3: Conv1 parameters.** One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = **608**.\n\n**Step 4: Pool1.** (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight.\n\n**Step 5: Conv2 spatial size.** (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600.\n\n**Step 6: Conv2 parameters.** Its input has **eight** channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = **3,216**.\n\n**Step 7: Pool2 and flatten.** (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = **400**. Neither operation adds parameters.\n\n**Step 8: dense 120.** Weights = 400×120 = 48,000. Biases = 120. Parameters = **48,120**. Output contains 120 activations.\n\n**Step 9: dense 84.** Weights = 120×84 = 10,080. Biases = 84. Parameters = **10,164**. Output contains 84 activations.\n\n**Step 10: dense 10 and softmax.** Weights = 84×10 = 840. Biases = 10. Dense parameters = **850**. Softmax converts the ten logits into ten probabilities and adds **zero** parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation.\n\n**Step 11: add only the trainable counts.** 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = **62,958**.\n\n| Layer | Output shape | Activations | Parameters |\n|---|---|---|---|\n| Input | 32 × 32 × 3 | 3,072 | 0 |\n| Conv1 | 28 × 28 × 8 | 6,272 | 608 |\n| Pool1 | 14 × 14 × 8 | 1,568 | 0 |\n| Conv2 | 10 × 10 × 16 | 1,600 | 3,216 |\n| Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 |\n| Dense 120 | 120 | 120 | 48,120 |\n| Dense 84 | 84 | 84 | 10,164 |\n| Dense 10 → softmax | 10 | 10 | 850 + 0 |\n\n**Final answer:** The network has **62,958 trainable parameters**. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network.\n\n**Yaad rakho:** Shape aage carry karo; weights aur activati"},{"title":"Even-sized kernels: SAME padding is not always symmetric","blocks":["Definition. A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides.
","For a 2 by 2 kernel at stride 1, preserving the input size requires one padding entry in total per axis, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention.
","Hinglish cue: P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho.
","Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)
","Input height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter.
","The padded input dimensions and the convolution output dimensions.
","The two opposite borders contribute P entries each. The output formula counts complete filter placements after padding.
","N is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel.
","Step 1: count the two padding borders.
","Step 2: find the padded side length.
","![\"N_{\\mathrm{padded}}=5+2=7\"](\"assets/equations/e288268af2ced78c.svg\")
The padded input is 7 × 7 × 1.
","Step 3: subtract the kernel size.
","![\"7-2=5\"](\"assets/equations/b07e3025ba7ec8bf.svg\")
Step 4: divide by stride, take the floor and count the first placement.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{5}{1}\\right\\rfloor+1=5+1=6\"](\"assets/equations/c9e9b01d59dde654.svg\")
Step 5: write height, width and channels.
","Both spatial dimensions equal 6. There is one filter, so there is one output channel.
","Final answer: padded input = 7 × 7 × 1; output = 6 × 6 × 1. This explicitly specified padding does not implement SAME for this even kernel.
","Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)
","Input side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found.
","Padding above, below, left and right; padded-input shape; final output shape.
","For unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{N+P_{\\mathrm{before}}+P_{\\mathrm{after}}-K}{S}\\right\\rfloor+1\"](\"assets/equations/1fb9e63f94b876bf.svg\")
At stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1.
","![\"P_{\\mathrm{total}}=P_{\\mathrm{before}}+P_{\\mathrm{after}}=K-1\"](\"assets/equations/a9c76650bca9e58b.svg\")
Step 1: calculate total padding along the height.
","![\"P_{\\mathrm{total}}=2-1=1\"](\"assets/equations/ebca391be173fb93.svg\")
Only one extra row in total is needed.
","Step 2: place the row using the stated convention.
","Padding above = 0; padding below = 1.
","Step 3: apply the same calculation to width.
","Only one extra column in total is needed. Padding left = 0; padding right = 1.
","Step 4: find the padded-input shape.
","![\"H_{\\mathrm{padded}}=5+0+1=6\"](\"assets/equations/bd37f3052feb0554.svg\")
![\"W_{\\mathrm{padded}}=5+0+1=6\"](\"assets/equations/ab52e1fac6848205.svg\")
The padded input is 6 × 6 × 1.
","Step 5: count the output positions along each axis.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{5+0+1-2}{1}\\right\\rfloor+1=4+1=5\"](\"assets/equations/9a5597c2b0d54bcf.svg\")
Step 6: include the output channel.
","One filter creates one output channel.
","Final answer: add 0 above, 1 below, 0 left and 1 right; padded input = 6 × 6 × 1; output = 5 × 5 × 1.
","The two questions have the same input and kernel but different padding instructions. Read the stated boundary padding before applying the formula.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c7-s18","search":"Even-sized kernels: SAME padding is not always symmetric \n**Definition.** A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides.\n\nFor a 2 by 2 kernel at stride 1, preserving the input size requires **one padding entry in total per axis**, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention.\n\n**Hinglish cue:** P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho.\n\n### Example 1: one entry on every side\n\n**Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)**\n\n### Given\n\nInput height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter.\n\n### To find\n\nThe padded input dimensions and the convolution output dimensions.\n\n### Formula — which one and why\n\nThe two opposite borders contribute P entries each. The output formula counts complete filter placements after padding.\n\n\n\n\n\nN is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel.\n\n### Solution — step by step\n\n**Step 1: count the two padding borders.**\n\n\n\n**Step 2: find the padded side length.**\n\n\n\nThe padded input is 7 × 7 × 1.\n\n**Step 3: subtract the kernel size.**\n\n\n\n**Step 4: divide by stride, take the floor and count the first placement.**\n\n\n\n**Step 5: write height, width and channels.**\n\nBoth spatial dimensions equal 6. There is one filter, so there is one output channel.\n\n**Final answer:** padded input = **7 × 7 × 1**; output = **6 × 6 × 1**. This explicitly specified padding does not implement SAME for this even kernel.\n\n### Example 2: preserve the size with SAME padding\n\n**Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)**\n\n### Given\n\nInput side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found.\n\n### To find\n\nPadding above, below, left and right; padded-input shape; final output shape.\n\n### Formula — which one and why\n\nFor unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis.\n\n\n\nAt stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1.\n\n\n\n### Solution — step by step\n\n**Step 1: calculate total padding along the height.**\n\n\n\nOnly one extra row in total is needed.\n\n**Step 2: place the row using the stated convention.**\n\nPadding above = 0; padding below = 1.\n\n**Step 3: apply the same calculation to width.**\n\nOnly one extra column in total is needed. Padding left = 0; padding right = 1.\n\n**St"}]},{"n":8,"title":"CNN architectures and transfer learning","sections":[{"title":"AlexNet and the reason for its components","blocks":["Definition. AlexNet is a deep convolutional architecture introduced for ImageNet classification, conventionally described as five convolutional layers followed by three fully connected layers. Its historical significance is the effective combination of learned convolutional features, nonlinear activations, GPU training and regularization at large scale.
","Input image → convolution and ReLU stages → spatial pooling → fully connected representations → class scores and softmax. The original model used local response normalization in selected early stages and overlapping max pooling. Grouped computation across two GPUs was part of the original implementation. Exact sizes must follow the architecture specified in an examination question.
","ReLU supplies a nonlinear transformation and avoids sigmoid-style saturation for positive preactivations. Convolution uses local connectivity and spatial weight sharing. Pooling reduces spatial dimensions and increases tolerance to small local changes. Dropout regularizes the fully connected layers. Data augmentation supplies transformed training examples to reduce dependence on a fixed view. These components perform different jobs and should not be described as interchangeable methods of reducing overfitting.
","The dense layers contribute a large number of parameters; the model has substantial memory and computation requirements. AlexNet is much shallower than later residual architectures. Its original use of local response normalization is different from batch normalization. For a comparison answer, identify its sequential convolutional design, then contrast it with Inception's parallel branches and ResNet's residual addition.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s0","search":"AlexNet and the reason for its components **Definition.** AlexNet is a deep convolutional architecture introduced for ImageNet classification, conventionally described as five convolutional layers followed by three fully connected layers. Its historical significance is the effective combination of learned convolutional features, nonlinear activations, GPU training and regularization at large scale.\n\n### Architecture and information flow\nInput image → convolution and ReLU stages → spatial pooling → fully connected representations → class scores and softmax. The original model used local response normalization in selected early stages and overlapping max pooling. Grouped computation across two GPUs was part of the original implementation. Exact sizes must follow the architecture specified in an examination question.\n\n\n\n### Role of the components\n**ReLU** supplies a nonlinear transformation and avoids sigmoid-style saturation for positive preactivations. **Convolution** uses local connectivity and spatial weight sharing. **Pooling** reduces spatial dimensions and increases tolerance to small local changes. **Dropout** regularizes the fully connected layers. **Data augmentation** supplies transformed training examples to reduce dependence on a fixed view. These components perform different jobs and should not be described as interchangeable methods of reducing overfitting.\n\n### Limitations and exam distinction\nThe dense layers contribute a large number of parameters; the model has substantial memory and computation requirements. AlexNet is much shallower than later residual architectures. Its original use of local response normalization is different from batch normalization. For a comparison answer, identify its sequential convolutional design, then contrast it with Inception's parallel branches and ResNet's residual addition.\n"},{"title":"Inception examines several spatial scales","blocks":["Definition. An Inception module is a convolutional building block that processes the same input through parallel branches using different receptive-field operations and concatenates their output feature maps along the channel dimension.
","A conventional early Inception module includes 1 × 1, 3 × 3 and 5 × 5 convolution branches and a pooling branch. Suitable stride and padding preserve compatible spatial dimensions. The branch outputs are then concatenated. If branch channel counts are c₁, c₂, c₃ and c₄, the output depth is c₁ + c₂ + c₃ + c₄.
","Different branch receptive fields can capture patterns at different spatial scales. A 1 × 1 convolution mixes channel information at each position. Before an expensive larger kernel, it can reduce the number of input channels and therefore the parameter count and multiplication cost. The bottleneck calculation is developed in the following section.
","Concatenation increases channel depth and retains each branch's feature maps separately. Elementwise addition requires equal shapes and combines corresponding values. Inception commonly uses concatenation; a standard residual block uses addition. A question giving branch sizes tests this distinction as well as the convolution formula.
","Early Inception designs reduce reliance on a large fully connected classifier through global average pooling. The parallel design still needs compatible branch shapes and carefully chosen channel widths; increasing the number of branches or filters increases computation.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s1","search":"Inception examines several spatial scales **Definition.** An Inception module is a convolutional building block that processes the same input through parallel branches using different receptive-field operations and concatenates their output feature maps along the channel dimension.\n\n### Structure and working\nA conventional early Inception module includes 1 × 1, 3 × 3 and 5 × 5 convolution branches and a pooling branch. Suitable stride and padding preserve compatible spatial dimensions. The branch outputs are then concatenated. If branch channel counts are c₁, c₂, c₃ and c₄, the output depth is c₁ + c₂ + c₃ + c₄.\n\n\n\n### Why use several branches?\nDifferent branch receptive fields can capture patterns at different spatial scales. A 1 × 1 convolution mixes channel information at each position. Before an expensive larger kernel, it can reduce the number of input channels and therefore the parameter count and multiplication cost. The bottleneck calculation is developed in the following section.\n\n### Concatenation is not addition\nConcatenation increases channel depth and retains each branch's feature maps separately. Elementwise addition requires equal shapes and combines corresponding values. Inception commonly uses concatenation; a standard residual block uses addition. A question giving branch sizes tests this distinction as well as the convolution formula.\n\nEarly Inception designs reduce reliance on a large fully connected classifier through global average pooling. The parallel design still needs compatible branch shapes and carefully chosen channel widths; increasing the number of branches or filters increases computation.\n"},{"title":"Why one by one convolution saves work","blocks":["Definition. A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts.
","Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.
","Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size.
","Direct MACs, total two-stage bottleneck MACs and percentage saved.
","Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added.
","![\"\\mathrm{MACs}=H_{\\mathrm{out}}W_{\\mathrm{out}}C_{\\mathrm{out}}K^2C_{\\mathrm{in}}\"](\"assets/equations/6c847f45a17553cd.svg\")
One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs.
","Step 1: direct layer. Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs.
","![\"\\mathrm{MACs}_{\\mathrm{direct}}=25088(4800)=120422400\"](\"assets/equations/7f08b61aa51f8fc4.svg\")
Step 2: 1 × 1 reduction. There are 16 receiving filters, each spanning all 192 original channels.
","![\"\\mathrm{MACs}_{1\\times1}=28(28)(16)(192)=2408448\"](\"assets/equations/aaefb7a933a188a0.svg\")
Step 3: reduced-depth spatial layer. Its 32 filters now span 16 channels.
","![\"\\mathrm{MACs}_{5\\times5}=28(28)(32)(25)(16)=10035200\"](\"assets/equations/b6f81bb572b38322.svg\")
Step 4: add the two bottleneck costs.
","![\"\\mathrm{MACs}_{\\mathrm{bottleneck}}=2408448+10035200=12443648\"](\"assets/equations/3917ad1b6a4cc0eb.svg\")
Step 5: calculate the saving.
","![\"\\mathrm{saving}=120422400-12443648=107978752\"](\"assets/equations/fe529a7c8d14263e.svg\")
![\"\\mathrm{saving\\](\"assets/equations/e22e38a0d749a968.svg\")
Final answer: direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption.
","Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.
","Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1.
","Direct MACs, bottleneck MACs and saving.
","Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers.
","Step 1: direct spatial convolution.
","![\"8(8)(8)(3)(3)(16)=73728\"](\"assets/equations/dffc8608ebe7e697.svg\")
Step 2: 1 × 1 reduction.
","![\"8(8)(4)(1)(1)(16)=4096\"](\"assets/equations/cbfbed2568dc51ee.svg\")
Step 3: reduced-depth 3 × 3 convolution.
","![\"8(8)(8)(3)(3)(4)=18432\"](\"assets/equations/60dc7f05c85051f9.svg\")
Step 4: add the stages and subtract from the direct cost.
","![\"4096+18432=22528\"](\"assets/equations/e4821e959cb7a516.svg\")
![\"73728-22528=51200\"](\"assets/equations/be3a4d13f46b02ba.svg\")
Final answer: direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s2","search":"Why one by one convolution saves work **Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts.\n\n\n\n**Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.**\n\n### Given\nBoth paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size.\n\n### To find\nDirect MACs, total two-stage bottleneck MACs and percentage saved.\n\n### Formula — which one and why\nEach output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added.\n\nOne MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs.\n\n### Solution — step by step\n**Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs.\n\n\n**Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels.\n\n\n**Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels.\n\n\n**Step 4: add the two bottleneck costs.**\n\n\n**Step 5: calculate the saving.**\n\n\n\n**Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption.\n\n### A second bottleneck with different dimensions\n**Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.**\n\n### Given\nSpatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1.\n\n### To find\nDirect MACs, bottleneck MACs and saving.\n\n### Formula — which one and why\nUse the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers.\n\n### Solution — step by step\n**Step 1: direct spatial convolution.**\n\n\n**Step 2: 1 × 1 reduction.**\n\n\n**Step 3: reduced-depth 3 × 3 convolution.**\n\n\n**Step 4: add the stages and subtract from the direct cost.**\n\n\n**Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values.\n"},{"title":"ResNet adds a shortcut","blocks":["Definition. A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x.
","![\"y=F(x)+x\"](\"assets/equations/0d9d34020eaab826.svg\")
If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch.
","For the identity-shortcut formulation, the derivative includes a direct identity contribution.
","![\"\\frac{\\partial](\"assets/equations/505f1041667e53bb.svg\")
This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients.
","If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride.
","![\"y=F(x)+W_sx\"](\"assets/equations/9ffeadda67cc02f9.svg\")
An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition.
","Question. Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum.
","x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3].
","Find the three components of x+F(x).
","Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.
","x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.
","![\"y=x+F(x)\"](\"assets/equations/ec48bb473e2d7934.svg\")
Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.
","Step 1: add the first matching pair.
","![\"1+0.1=1.1\"](\"assets/equations/db025849e060d366.svg\")
Step 2: add the second matching pair.
","![\"2+(-0.2)=1.8\"](\"assets/equations/ab49a448719b780e.svg\")
Step 3: add the third matching pair.
","![\"3+0.3=3.3\"](\"assets/equations/d117d7aaf6f65395.svg\")
Step 4: assemble in the original order.
","![\"y=[1.1,1.8,3.3]\"](\"assets/equations/2f3d1f51931e7a8d.svg\")
The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s3","search":"ResNet adds a shortcut **Definition.** A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x.\n\nIf the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch.\n\n\n\n### Effect on gradient propagation\nFor the identity-shortcut formulation, the derivative includes a direct identity contribution.\n\nThis provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients.\n\n### Projection when dimensions differ\nIf channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride.\n\nAn identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition.\n\n**Question.** Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum.\n\n\n\n### Given\n\nx = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3].\n\n### To find\n\nFind the three components of x+F(x).\n\n### Formula — which one and why\n\n**Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.\n\nx = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.\n\n\n\n**Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye.\n\n### Solution — step by step\n\n\n\n\n**Step 1: add the first matching pair.**\n\n\n\n\n\n\n\n**Step 2: add the second matching pair.**\n\n\n\n\n\n\n\n**Step 3: add the third matching pair.**\n\n\n\n\n\n\n\n**Step 4: assemble in the original order.**\n\n\n\n\n\n\n\n\nThe vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum.\n"},{"title":"Transfer learning from a small dataset","blocks":["Definition. Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers.
","Transfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort.
","Full scenario. Build a face-attendance model with few images per student. Explain a training plan.
","Step 1. Define and split the data. Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization.
","Step 2. Choose a suitable pretrained backbone. For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available.
","Step 3. Replace the classifier. Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy.
","Step 4. Freeze and train. Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring.
","Step 5. Fine-tune if justified. Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint.
","Step 6. Evaluate. Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition.
","Feature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s4","search":"Transfer learning from a small dataset **Definition.** Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers.\n\n### Explanation and working\nTransfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort.\n\n\n\n**Full scenario.** Build a face-attendance model with few images per student. Explain a training plan.\n**Step 1. Define and split the data.** Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization.\n**Step 2. Choose a suitable pretrained backbone.** For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available.\n**Step 3. Replace the classifier.** Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy.\n**Step 4. Freeze and train.** Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring.\n**Step 5. Fine-tune if justified.** Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint.\n**Step 6. Evaluate.** Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition.\nFeature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario."},{"title":"Model answer: feature extraction and fine-tuning","question":["Question. Explain transfer learning for image classification with limited labeled data. Distinguish feature extraction from fine-tuning and describe the training procedure.
"],"answer":["Transfer learning reuses knowledge represented by a model trained on a source task to support a related target task. A pretrained CNN can provide useful visual representations, reducing the target data and training effort needed compared with learning all features from random initialization.
","1. Define the target classes and create separate training, validation and test splits. Prevent duplicate or closely related captures from leaking across splits.
","2. Choose a suitable pretrained backbone and apply its required input size and preprocessing.
","3. Replace the original classifier with a new head whose output count matches the target classes.
","4. Freeze the backbone and train the new head. Only the unfrozen parameters receive optimizer updates.
","5. If validation results justify it, unfreeze selected later backbone layers and continue training with a suitably smaller learning rate.
","6. Select hyperparameters and checkpoints using validation data. Evaluate the selected model on the held-out test set.
","| Feature extraction | Fine-tuning |
|---|---|
| Pretrained backbone weights remain fixed | Some or all selected pretrained weights are updated |
| New classifier learns from fixed representations | Representations can adapt to the target task |
| Usually fewer trainable parameters | Usually more trainable parameters and greater risk of overfitting on a small dataset |
The benefit depends on how useful the source features are for the target data. Transfer is not guaranteed to help every unrelated task. For a numerical, count all stored parameters separately from the trainable subset; freezing a layer does not delete its weights.
"],"kind":"theory","id":"mt2-c8-s5","search":"Model answer: feature extraction and fine-tuning **Question.** Explain transfer learning for image classification with limited labeled data. Distinguish feature extraction from fine-tuning and describe the training procedure.\n\n**Model answer.**\nTransfer learning reuses knowledge represented by a model trained on a source task to support a related target task. A pretrained CNN can provide useful visual representations, reducing the target data and training effort needed compared with learning all features from random initialization.\n\n\n\n\n### Procedure\n1. Define the target classes and create separate training, validation and test splits. Prevent duplicate or closely related captures from leaking across splits.\n2. Choose a suitable pretrained backbone and apply its required input size and preprocessing.\n3. Replace the original classifier with a new head whose output count matches the target classes.\n4. Freeze the backbone and train the new head. Only the unfrozen parameters receive optimizer updates.\n5. If validation results justify it, unfreeze selected later backbone layers and continue training with a suitably smaller learning rate.\n6. Select hyperparameters and checkpoints using validation data. Evaluate the selected model on the held-out test set.\n\n| Feature extraction | Fine-tuning |\n| Pretrained backbone weights remain fixed | Some or all selected pretrained weights are updated |\n| New classifier learns from fixed representations | Representations can adapt to the target task |\n| Usually fewer trainable parameters | Usually more trainable parameters and greater risk of overfitting on a small dataset |\n\nThe benefit depends on how useful the source features are for the target data. Transfer is not guaranteed to help every unrelated task. For a numerical, count all stored parameters separately from the trainable subset; freezing a layer does not delete its weights.\n"},{"title":"How to draw the architecture diagrams","blocks":["For AlexNet, draw a sequence of convolution/pooling stages followed by dense layers and classifier. For Inception, draw four parallel branches and a channel-concatenation join. For ResNet, draw one transformation branch and one shortcut meeting at a plus sign. For transfer learning, draw frozen pretrained base → new trainable head, then mark selected base layers optionally unfrozen for fine-tuning.
","Definition. Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths.
","Concatenation joins feature channels; elementwise addition sums matching entries. They are different operations.
","Four Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64.
","A residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s7","search":"Inception concatenation and residual addition **Definition.** Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths.\n\n### Explanation and working\nConcatenation joins feature channels; elementwise addition sums matching entries. They are different operations.\n\n\n\nFour Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64.\nA residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition.\n"},{"title":"Model answer: AlexNet, Inception and ResNet","question":["Question. Compare the main ideas of AlexNet, Inception and ResNet. Explain how to recognize their architecture diagrams.
"],"answer":["All three are CNN architectures, but their characteristic structures differ.
","| Architecture | Main idea | What to show in a diagram |
|---|---|---|
| AlexNet | Successive convolutional feature extraction followed by dense classification | Five convolutional layers and three fully connected layers, with the stated pooling/normalization stages |
| Inception v1 | Parallel branches analyze features at different spatial scales | 1 × 1, reduced 3 × 3, reduced 5 × 5 and pooling branches joined by channel concatenation |
| ResNet | Learn a residual transformation and add a shortcut | A transformation branch and input shortcut meeting at a plus sign |
AlexNet's exam-relevant features include ReLU, GPU-based training, augmentation and dropout in the fully connected part. A parameter calculation must follow the given channel connectivity; a grouped implementation and a simplified fully connected-channel diagram need not have the same count.
","Inception uses 1 × 1 convolutions to mix channels and, when a smaller number of filters is chosen, reduce the channel depth before expensive spatial filters. Branch outputs must have compatible height and width for concatenation. Concatenation adds channel counts by joining their lists.
","ResNet computes a learned transformation F(x) and combines it with the shortcut:
","![\"H(x)=F(x)+x\"](\"assets/equations/d30a7ff979462f45.svg\")
This is entrywise addition. The two shapes must match; a projection can adapt a shortcut when they do not. The shortcut supplies an additional route for information and gradients and makes identity-like mappings easy to represent.
","Key distinction: an Inception join concatenates feature maps; a residual join adds corresponding values. Their diagram symbols and output-channel calculations must therefore differ.
"],"kind":"theory","id":"mt2-c8-s8","search":"Model answer: AlexNet, Inception and ResNet **Question.** Compare the main ideas of AlexNet, Inception and ResNet. Explain how to recognize their architecture diagrams.\n\n**Model answer.**\nAll three are CNN architectures, but their characteristic structures differ.\n\n\n\n\n| Architecture | Main idea | What to show in a diagram |\n| AlexNet | Successive convolutional feature extraction followed by dense classification | Five convolutional layers and three fully connected layers, with the stated pooling/normalization stages |\n| Inception v1 | Parallel branches analyze features at different spatial scales | 1 × 1, reduced 3 × 3, reduced 5 × 5 and pooling branches joined by channel concatenation |\n| ResNet | Learn a residual transformation and add a shortcut | A transformation branch and input shortcut meeting at a plus sign |\n\nAlexNet's exam-relevant features include ReLU, GPU-based training, augmentation and dropout in the fully connected part. A parameter calculation must follow the given channel connectivity; a grouped implementation and a simplified fully connected-channel diagram need not have the same count.\n\nInception uses 1 × 1 convolutions to mix channels and, when a smaller number of filters is chosen, reduce the channel depth before expensive spatial filters. Branch outputs must have compatible height and width for concatenation. Concatenation adds channel counts by joining their lists.\n\nResNet computes a learned transformation F(x) and combines it with the shortcut:\n\nThis is entrywise addition. The two shapes must match; a projection can adapt a shortcut when they do not. The shortcut supplies an additional route for information and gradients and makes identity-like mappings easy to represent.\n\n**Key distinction:** an Inception join concatenates feature maps; a residual join adds corresponding values. Their diagram symbols and output-channel calculations must therefore differ.\n"},{"title":"Applications of convolutional neural networks","blocks":["Definition. A CNN application combines a convolutional feature extractor with an output head designed for a particular prediction task. Shared local filters learn spatial patterns; the output head determines what is predicted and the loss determines how it is trained.
","| Task | Required output | What distinguishes it |
|---|---|---|
| Image classification | One label or class distribution for the image | Answers what category the whole image belongs to |
| Object detection | Object classes and bounding-box coordinates | Also identifies where each detected object is located |
| Semantic segmentation | A class label or distribution for each pixel | Produces a spatial label map rather than one image-wide class |
| Face recognition | Identity class or a learned face representation | Compares or classifies learned visual features |
A softmax image classifier alone does not produce boxes or pixel labels. Detection and segmentation require appropriate task heads, targets and objectives. In an applications question, explain the prediction required instead of listing model names without their purpose.
","Medical imaging: classify an image or use a segmentation model to identify regions of interest. The target may be an image-level label or a pixel mask, so the architecture and loss must match that choice.
","Autonomous perception: use camera images for tasks such as detecting pedestrians and vehicles or segmenting road regions. These outputs are inputs to a larger perception and decision system.
","Manufacturing: classify product images as defective or non-defective, or localize defects when position information is needed.
","Document analysis: classify cropped handwritten characters; a complete document-reading system must also locate and order text regions.
","Remote sensing: classify land-use regions or segment different surface categories in satellite images. Spatial context and texture can help distinguish classes.
","Question. Describe how a CNN can classify product photographs into three classes: acceptable, scratched and cracked. Explain the input, network stages, output activation, loss, training and evaluation.
","Step 1: define data and labels. Each example contains an image and one of the three class labels. Use separate training, validation and test sets, keeping near-duplicate photographs of the same item out of different splits. Apply the chosen model's input size and scaling consistently.
","Step 2: extract spatial features. Convolutional layers calculate local responses using shared kernels. Nonlinear activations allow more complex combinations of those responses. Further convolution stages combine local patterns; pooling may reduce spatial dimensions.
","Step 3: produce three class scores. Convert the final representation into a feature vector, using flattening or an appropriate global pooling operation, and apply a dense layer with three outputs. These raw outputs are logits.
","Step 4: calculate class probabilities and loss. For mutually exclusive classes, use softmax and categorical cross-entropy. A one-hot label selects the probability of the correct class.
","![\"q_k=\\frac{e^{z_k}}{\\sum_{j=1}^{3}e^{z_j}},\\qquad](\"assets/equations/7f4b48b92a27eca9.svg\")
Step 5: train the parameters. Run forward propagation, evaluate the loss, backpropagate derivatives and update the trainable convolutional and dense parameters using the selected optimizer. Use validation results for hyperparameters and stopping decisions.
","Step 6: evaluate the selected model. Assess the final model on held-out photographs. Report overall classification performance and inspect which defects are confused. With limited data, a pretrained CNN can provide features and selected layers can be fine-tuned.
","Numerical extension. If the last pooled feature vector has 64 components, how many parameters does the three-class dense head have, including bias?
","Step 1: each output receives 64 weights, so three outputs require 64 × 3 = 192 weights.
","Step 2: each of the three outputs has one bias, giving 3 biases.
","![\"P_{\\mathrm{head}}=64(3)+3=(64+1)3=195\"](\"assets/equations/5453c1fee1f9ccef.svg\")
Final answer: 195 trainable parameters in this head. Softmax adds no trainable parameters. The backbone's parameters are counted separately.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s9","search":"Applications of convolutional neural networks **Definition.** A CNN application combines a convolutional feature extractor with an output head designed for a particular prediction task. Shared local filters learn spatial patterns; the output head determines what is predicted and the loss determines how it is trained.\n\n### Classification, detection and segmentation\n| Task | Required output | What distinguishes it |\n| Image classification | One label or class distribution for the image | Answers what category the whole image belongs to |\n| Object detection | Object classes and bounding-box coordinates | Also identifies where each detected object is located |\n| Semantic segmentation | A class label or distribution for each pixel | Produces a spatial label map rather than one image-wide class |\n| Face recognition | Identity class or a learned face representation | Compares or classifies learned visual features |\n\nA softmax image classifier alone does not produce boxes or pixel labels. Detection and segmentation require appropriate task heads, targets and objectives. In an applications question, explain the prediction required instead of listing model names without their purpose.\n\n### Examples across domains\n**Medical imaging:** classify an image or use a segmentation model to identify regions of interest. The target may be an image-level label or a pixel mask, so the architecture and loss must match that choice.\n\n**Autonomous perception:** use camera images for tasks such as detecting pedestrians and vehicles or segmenting road regions. These outputs are inputs to a larger perception and decision system.\n\n**Manufacturing:** classify product images as defective or non-defective, or localize defects when position information is needed.\n\n**Document analysis:** classify cropped handwritten characters; a complete document-reading system must also locate and order text regions.\n\n**Remote sensing:** classify land-use regions or segment different surface categories in satellite images. Spatial context and texture can help distinguish classes.\n\n### Complete exam-style application answer\n**Question. Describe how a CNN can classify product photographs into three classes: acceptable, scratched and cracked. Explain the input, network stages, output activation, loss, training and evaluation.**\n\n**Step 1: define data and labels.** Each example contains an image and one of the three class labels. Use separate training, validation and test sets, keeping near-duplicate photographs of the same item out of different splits. Apply the chosen model's input size and scaling consistently.\n\n**Step 2: extract spatial features.** Convolutional layers calculate local responses using shared kernels. Nonlinear activations allow more complex combinations of those responses. Further convolution stages combine local patterns; pooling may reduce spatial dimensions.\n\n**Step 3: produce three class scores.** Convert the final representation into a feature vector, using flattening or an appropriate global pooling operation, and apply a dense layer with three outputs. These raw outputs are logits.\n\n**Step 4: calculate class probabilities and loss.** For mutually exclusive classes, use softmax and categorical cross-entropy. A one-hot label selects the probability of the correct class.\n\n\n**Step 5: train the parameters.** Run forward propagation, evaluate the loss, backpropagate derivatives and update the trainable convolutional and dense parameters using the selected optimizer. Us"},{"title":"Reading LeNet and AlexNet layer diagrams","blocks":["A layer diagram describes two different things: the size of the data moving through the network and the trainable operations that transform that data. A number such as 9,216 beside a flatten operation describes data length; it does not automatically mean another dense layer with 9,216 neurons.
","The classroom LeNet layout begins with a 32 × 32 grayscale image. A valid 5 × 5 convolution with six filters produces 28 × 28 × 6. A 2 × 2 pooling operation with stride 2 produces 14 × 14 × 6. The next valid 5 × 5 convolution produces 10 × 10 × 16, and pooling reduces this to 5 × 5 × 16. A 5 × 5 convolution across those 16 channels produces 1 × 1 × 120. The final representation passes through 84 units and then ten class outputs.
","| Stage | What happens | Output shape |
|---|---|---|
| Input | One grayscale image | 32 × 32 × 1 |
| C1 | Six 5 × 5 filters, stride 1, no padding | 28 × 28 × 6 |
| S2 | 2 × 2 average pooling, stride 2 | 14 × 14 × 6 |
| C3 | Sixteen 5 × 5 filters, stride 1, no padding | 10 × 10 × 16 |
| S4 | 2 × 2 average pooling, stride 2 | 5 × 5 × 16 |
| C5 | 120 filters covering the complete 5 × 5 spatial map | 1 × 1 × 120 |
| F6 | Dense transformation | 84 values |
| Classifier | Modern classroom softmax classifier | Ten probabilities |
The original LeNet-5 used details such as partial channel connectivity, trainable subsampling and an RBF output. A modern full-channel implementation with ordinary pooling and softmax is a different parameter-count convention. For a numerical, follow the question's diagram and state the convention. The complete LeNet parameter numerical in the solved-paper section uses the modern classroom convention.
","For the faculty's 227 × 227 × 3 classroom input, the main shape sequence is 55 × 55 × 96 → 27 × 27 × 96 → 27 × 27 × 256 → 13 × 13 × 256 → 13 × 13 × 384 → 13 × 13 × 384 → 13 × 13 × 256 → 6 × 6 × 256. Flattening gives 9,216 values; the three dense outputs are 4,096, 4,096 and 1,000. There is no extra learned 9,216-unit layer between flattening and the first 4,096-unit layer.
","Question (practice, 3 marks). An AlexNet-style first layer receives a 227 × 227 RGB image and uses 96 filters of size 11 × 11, stride 4 and no padding. Find its output shape, trainable parameters including bias, and the number of output activation values.
","Input height and width = 227; input channels = 3; kernel = 11; stride = 4; padding = 0; output filters = 96. One bias is used per filter.
","The three-dimensional output shape, parameter count and activation count.
","The output-size formula counts valid window positions. Parameter count counts stored filter coefficients, reused at every position. Activation count counts the values produced for one image.
","![\"H_o=W_o=\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\"](\"assets/equations/368322f7f93c9e79.svg\")
![\"A=H_oW_oC_{\\mathrm{out}}\"](\"assets/equations/fd2a21cf67217e86.svg\")
Here N is input side length, P is padding per side, K is kernel side length, S is stride, and C denotes channel count. The symbol P in the output formula means padding; the labelled quantity P_conv means parameters.
","Step 1: calculate one spatial side.
","![\"\\frac{227-11}{4}+1=\\frac{216}{4}+1=54+1=55\"](\"assets/equations/8187570cd03b5f03.svg\")
Step 2: attach the output depth. There is one output map per filter, so the output is 55 × 55 × 96.
","Step 3: count the weights in one filter.
","![\"11(11)(3)=363\"](\"assets/equations/f8b467d436e9d44e.svg\")
Step 4: add one bias and multiply by 96 filters.
","![\"P_{\\mathrm{conv}}=(363+1)(96)=34944\"](\"assets/equations/a95b022f4ca93164.svg\")
Step 5: count activation values for one image.
","![\"A=55(55)(96)=290400\"](\"assets/equations/200bca238c47286b.svg\")
Final answer: output = 55 × 55 × 96; parameters = 34,944; activations = 290,400. The weights are reused, so parameters and activations are different counts. The full faculty-style ungrouped AlexNet table totals 62,378,344 parameters; the original grouped model uses different channel connectivity.
","Yaad rakho: shape = kitne answers nikle; parameters = kitne numbers seekhe.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s10","search":"Reading LeNet and AlexNet layer diagrams \n**A layer diagram describes two different things:** the size of the data moving through the network and the trainable operations that transform that data. A number such as 9,216 beside a flatten operation describes data length; it does not automatically mean another dense layer with 9,216 neurons.\n\n### LeNet: follow the image as it becomes a class prediction\n\nThe classroom LeNet layout begins with a 32 × 32 grayscale image. A valid 5 × 5 convolution with six filters produces 28 × 28 × 6. A 2 × 2 pooling operation with stride 2 produces 14 × 14 × 6. The next valid 5 × 5 convolution produces 10 × 10 × 16, and pooling reduces this to 5 × 5 × 16. A 5 × 5 convolution across those 16 channels produces 1 × 1 × 120. The final representation passes through 84 units and then ten class outputs.\n\n| Stage | What happens | Output shape |\n| Input | One grayscale image | 32 × 32 × 1 |\n| C1 | Six 5 × 5 filters, stride 1, no padding | 28 × 28 × 6 |\n| S2 | 2 × 2 average pooling, stride 2 | 14 × 14 × 6 |\n| C3 | Sixteen 5 × 5 filters, stride 1, no padding | 10 × 10 × 16 |\n| S4 | 2 × 2 average pooling, stride 2 | 5 × 5 × 16 |\n| C5 | 120 filters covering the complete 5 × 5 spatial map | 1 × 1 × 120 |\n| F6 | Dense transformation | 84 values |\n| Classifier | Modern classroom softmax classifier | Ten probabilities |\n\nThe original LeNet-5 used details such as partial channel connectivity, trainable subsampling and an RBF output. A modern full-channel implementation with ordinary pooling and softmax is a different parameter-count convention. For a numerical, follow the question's diagram and state the convention. The complete LeNet parameter numerical in the solved-paper section uses the modern classroom convention.\n\n### AlexNet: identify the flatten step correctly\n\n\n\nFor the faculty's 227 × 227 × 3 classroom input, the main shape sequence is 55 × 55 × 96 → 27 × 27 × 96 → 27 × 27 × 256 → 13 × 13 × 256 → 13 × 13 × 384 → 13 × 13 × 384 → 13 × 13 × 256 → 6 × 6 × 256. Flattening gives 9,216 values; the three dense outputs are 4,096, 4,096 and 1,000. There is no extra learned 9,216-unit layer between flattening and the first 4,096-unit layer.\n\n**Question (practice, 3 marks). An AlexNet-style first layer receives a 227 × 227 RGB image and uses 96 filters of size 11 × 11, stride 4 and no padding. Find its output shape, trainable parameters including bias, and the number of output activation values.**\n\n### Given\nInput height and width = 227; input channels = 3; kernel = 11; stride = 4; padding = 0; output filters = 96. One bias is used per filter.\n\n### To find\nThe three-dimensional output shape, parameter count and activation count.\n\n### Formula — which one and why\nThe output-size formula counts valid window positions. Parameter count counts stored filter coefficients, reused at every position. Activation count counts the values produced for one image.\n\n\n\nHere N is input side length, P is padding per side, K is kernel side length, S is stride, and C denotes channel count. The symbol P in the output formula means padding; the labelled quantity P_conv means parameters.\n\n### Solution — step by step\n**Step 1: calculate one spatial side.**\n\n\n**Step 2: attach the output depth.** There is one output map per filter, so the output is **55 × 55 × 96**.\n\n**Step 3: count the weights in one filter.**\n\n\n**Step 4: add one bias and multiply by 96 filters.**\n\n\n**Step 5: count activation values for one image.**\n\n\n**Final answer:** output = 55 × 55 ×"},{"title":"Top-1 and top-5 error: what an ImageNet result means","blocks":["Top-1 prediction is the class with the highest score. A top-1 mistake occurs when that class is wrong. Top-5 prediction is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models.
","For an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures.
","Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.
","Total images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes.
","The percentage of mistakes under each rule.
","Subtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage.
","![\"E_k=\\frac{N-C_k}{N}\\times100\\%\"](\"assets/equations/3fb17785121a672d.svg\")
N means total evaluated images; C_k means images whose correct class appears among the top k predictions.
","Step 1: top-1 mistakes. 100 − 78 = 22.
","Step 2: top-1 error percentage.
","![\"E_1=\\frac{22}{100}\\times100\\%=22\\%\"](\"assets/equations/e21c77c5c190c14b.svg\")
Step 3: top-5 mistakes. 100 − 92 = 8.
","Step 4: top-5 error percentage.
","![\"E_5=\\frac{8}{100}\\times100\\%=8\\%\"](\"assets/equations/cbf5783f9d42ea19.svg\")
Final answer: top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions.
","Yaad rakho: top-five mein sahi naam mil gaya toh top-five success.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s11","search":"Top-1 and top-5 error: what an ImageNet result means \n**Top-1 prediction** is the class with the highest score. A top-1 mistake occurs when that class is wrong. **Top-5 prediction** is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models.\n\nFor an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures.\n\n**Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.**\n\n### Given\nTotal images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes.\n\n### To find\nThe percentage of mistakes under each rule.\n\n### Formula — which one and why\nSubtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage.\n\nN means total evaluated images; C_k means images whose correct class appears among the top k predictions.\n\n### Solution — step by step\n**Step 1: top-1 mistakes.** 100 − 78 = 22.\n\n**Step 2: top-1 error percentage.**\n\n\n**Step 3: top-5 mistakes.** 100 − 92 = 8.\n\n**Step 4: top-5 error percentage.**\n\n\n**Final answer:** top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions.\n\n**Yaad rakho:** top-five mein sahi naam mil gaya toh top-five success.\n"},{"title":"VGG16: small filters in a regular sequence","blocks":["Definition. VGG16 is a convolutional architecture with 13 convolutional layers and three dense layers. Its standard convolution blocks contain 2, 2, 3, 3 and 3 convolutions. Pooling, activations and flattening are not counted in the name's 16 weight layers.
","Each convolution uses a 3 × 3 kernel, stride 1 and padding 1, preserving height and width. Each block ends with 2 × 2 max pooling of stride 2. The final classifier is 4,096 → 4,096 → 1,000 for the original 1,000-class task.
","| Block | Convolutions | Channels after its convolutions | Size after its pooling |
|---|---|---|---|
| 1 | Two | 64 | 112 × 112 × 64 |
| 2 | Two | 128 | 56 × 56 × 128 |
| 3 | Three | 256 | 28 × 28 × 256 |
| 4 | Three | 512 | 14 × 14 × 512 |
| 5 | Three | 512 | 7 × 7 × 512 |
This table starts with 224 × 224 × 3. The flattened length is 7 × 7 × 512 = 25,088. Flattening has zero learned parameters. Stacking small filters provides repeated nonlinear transformations while progressively combining spatial information. A limitation is the large dense classifier and its memory cost.
","Question (practice, 3 marks). VGG16's first block contains two same-padded 3 × 3 convolutions with 64 output filters each, followed by 2 × 2 max pooling with stride 2. Input is 224 × 224 × 3. Find both convolution parameter counts including bias and the pooled output shape.
","Conv1: 3 input channels, 64 output channels. Conv2: 64 input channels, 64 output channels. Both have kernel 3, stride 1 and padding 1. Pooling has kernel 2, stride 2 and no padding.
","Trainable parameters in each convolution and the output after pooling.
","Use each layer's own input depth; the second layer receives the first layer's 64 maps. Use the spatial-output formula for pooling. Max pooling introduces no trainable parameters.
","Step 1: first convolution. Each filter has 3 × 3 × 3 = 27 weights and one bias.
","![\"P_1=(27+1)(64)=1792\"](\"assets/equations/e82763edab004507.svg\")
Step 2: second convolution. Each filter now spans 64 channels, giving 3 × 3 × 64 = 576 weights.
","![\"P_2=(576+1)(64)=36928\"](\"assets/equations/ff96ebed9174c6f1.svg\")
Step 3: add parameters.
","![\"P_{\\mathrm{block}}=1792+36928=38720\"](\"assets/equations/fc9e657e6b2a7ee5.svg\")
Step 4: calculate pooled side length. The same-padded convolutions retained side length 224.
","![\"\\frac{224-2}{2}+1=111+1=112\"](\"assets/equations/71feab89feae610a.svg\")
Final answer: 1,792 and 36,928 parameters; total = 38,720; pooled shape = 112 × 112 × 64.
","Exam convention: use the architecture explicitly supplied in a calculation. For the standard VGG16 architecture, remember 2–2–3–3–3 convolutions and 2 × 2 pooling; 25,088 is the flattened feature length.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s12","search":"VGG16: small filters in a regular sequence \n\n\n**Definition.** VGG16 is a convolutional architecture with 13 convolutional layers and three dense layers. Its standard convolution blocks contain 2, 2, 3, 3 and 3 convolutions. Pooling, activations and flattening are not counted in the name's 16 weight layers.\n\nEach convolution uses a 3 × 3 kernel, stride 1 and padding 1, preserving height and width. Each block ends with 2 × 2 max pooling of stride 2. The final classifier is 4,096 → 4,096 → 1,000 for the original 1,000-class task.\n\n| Block | Convolutions | Channels after its convolutions | Size after its pooling |\n| 1 | Two | 64 | 112 × 112 × 64 |\n| 2 | Two | 128 | 56 × 56 × 128 |\n| 3 | Three | 256 | 28 × 28 × 256 |\n| 4 | Three | 512 | 14 × 14 × 512 |\n| 5 | Three | 512 | 7 × 7 × 512 |\n\nThis table starts with 224 × 224 × 3. The flattened length is 7 × 7 × 512 = 25,088. Flattening has zero learned parameters. Stacking small filters provides repeated nonlinear transformations while progressively combining spatial information. A limitation is the large dense classifier and its memory cost.\n\n**Question (practice, 3 marks). VGG16's first block contains two same-padded 3 × 3 convolutions with 64 output filters each, followed by 2 × 2 max pooling with stride 2. Input is 224 × 224 × 3. Find both convolution parameter counts including bias and the pooled output shape.**\n\n### Given\nConv1: 3 input channels, 64 output channels. Conv2: 64 input channels, 64 output channels. Both have kernel 3, stride 1 and padding 1. Pooling has kernel 2, stride 2 and no padding.\n\n### To find\nTrainable parameters in each convolution and the output after pooling.\n\n### Formula — which one and why\n\nUse each layer's own input depth; the second layer receives the first layer's 64 maps. Use the spatial-output formula for pooling. Max pooling introduces no trainable parameters.\n\n### Solution — step by step\n**Step 1: first convolution.** Each filter has 3 × 3 × 3 = 27 weights and one bias.\n\n\n**Step 2: second convolution.** Each filter now spans 64 channels, giving 3 × 3 × 64 = 576 weights.\n\n\n**Step 3: add parameters.**\n\n\n**Step 4: calculate pooled side length.** The same-padded convolutions retained side length 224.\n\n\n**Final answer:** 1,792 and 36,928 parameters; total = 38,720; pooled shape = **112 × 112 × 64**.\n\n**Exam convention:** use the architecture explicitly supplied in a calculation. For the standard VGG16 architecture, remember **2–2–3–3–3 convolutions and 2 × 2 pooling**; 25,088 is the flattened feature length.\n"},{"title":"Global average pooling: one average for each channel","blocks":["Definition. Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number.
","![\"g_c=\\frac{1}{HW}\\sum_{i=1}^{H}\\sum_{j=1}^{W}a_{i,j,c}\"](\"assets/equations/002749b4b374cf54.svg\")
The letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier.
","Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = ![\"\\begin{bmatrix}1](\"assets/equations/e857333332fbdd17.svg\")
and channel B = ![\"\\begin{bmatrix}2](\"assets/equations/c890243a56df9f91.svg\")
. Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.
H = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs.
","The GAP result and dense-head parameter count.
","Use one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias.
","Step 1: add channel A. 1 + 3 + 5 + 7 = 16.
","Step 2: average channel A. 16 ÷ 4 = 4.
","Step 3: add channel B. 2 + 4 + 6 + 8 = 20.
","Step 4: average channel B. 20 ÷ 4 = 5. Thus GAP outputs [4,5].
","Step 5: count the classifier's weights. 2 inputs × 3 outputs = 6 weights.
","Step 6: add three biases. 6 + 3 = 9 parameters. GAP itself contributes zero.
","Final answer: GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters.
","Yaad rakho: ek channel ka poora average; channels ko aapas mein mat milao.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s13","search":"Global average pooling: one average for each channel \n\n\n**Definition.** Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number.\n\n\n\nThe letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier.\n\n**Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = and channel B = . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.**\n\n### Given\nH = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs.\n\n### To find\nThe GAP result and dense-head parameter count.\n\n### Formula — which one and why\nUse one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias.\n\n\n### Solution — step by step\n**Step 1: add channel A.** 1 + 3 + 5 + 7 = 16.\n\n**Step 2: average channel A.** 16 ÷ 4 = 4.\n\n**Step 3: add channel B.** 2 + 4 + 6 + 8 = 20.\n\n**Step 4: average channel B.** 20 ÷ 4 = 5. Thus GAP outputs **[4,5]**.\n\n**Step 5: count the classifier's weights.** 2 inputs × 3 outputs = 6 weights.\n\n**Step 6: add three biases.** 6 + 3 = **9 parameters**. GAP itself contributes zero.\n\n**Final answer:** GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters.\n\n**Yaad rakho:** ek channel ka poora average; channels ko aapas mein mat milao.\n"},{"title":"The faculty's 480-channel bottleneck numerical","blocks":["Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.
","Both paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters.
","MACs for the direct path, both bottleneck stages, their total and percentage saved.
","![\"M=H_oW_oC_oK^2C_i\"](\"assets/equations/48494d663392b9d5.svg\")
M is the MAC count. H_oW_oC_o counts output values; K²C_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck.
","![\"\\mathrm{saving}\\%=100\\frac{M_{\\mathrm{direct}}-M_{\\mathrm{bottleneck}}}{M_{\\mathrm{direct}}}\"](\"assets/equations/78cf62ab96d05d20.svg\")
Step 1: direct output count. 14 × 14 × 48 = 9,408.
","Step 2: products per direct output. 5 × 5 × 480 = 12,000.
","Step 3: direct MACs.
","![\"9408(12000)=112896000\"](\"assets/equations/7c9631645fc9930b.svg\")
Step 4: reduction-layer MACs. Each of 14 × 14 × 16 outputs combines 480 values.
","![\"14(14)(16)(480)=1505280\"](\"assets/equations/8486a56714848659.svg\")
Step 5: reduced spatial-layer MACs. Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values.
","![\"9408(400)=3763200\"](\"assets/equations/d32e7e22a58c8e10.svg\")
Step 6: add both bottleneck costs.
","![\"1505280+3763200=5268480\"](\"assets/equations/7061d250d8255604.svg\")
Step 7: calculate the saving.
","![\"112896000-5268480=107627520\"](\"assets/equations/a866c1ac7a2fce39.svg\")
![\"100\\frac{107627520}{112896000}\\approx95.333333\\%\"](\"assets/equations/18ccfe0746a80b04.svg\")
Final answer: direct = 112.896 million MACs; bottleneck = 5.26848 million MACs; saving ≈ 95.33%. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s14","search":"The faculty's 480-channel bottleneck numerical \n\n\n**Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.**\n\n### Given\nBoth paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters.\n\n### To find\nMACs for the direct path, both bottleneck stages, their total and percentage saved.\n\n### Formula — which one and why\n\nM is the MAC count. H_oW_oC_o counts output values; K²C_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck.\n\n\n### Solution — step by step\n**Step 1: direct output count.** 14 × 14 × 48 = 9,408.\n\n**Step 2: products per direct output.** 5 × 5 × 480 = 12,000.\n\n**Step 3: direct MACs.**\n\n\n**Step 4: reduction-layer MACs.** Each of 14 × 14 × 16 outputs combines 480 values.\n\n\n**Step 5: reduced spatial-layer MACs.** Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values.\n\n\n**Step 6: add both bottleneck costs.**\n\n\n**Step 7: calculate the saving.**\n\n\n\n**Final answer:** direct = **112.896 million MACs**; bottleneck = **5.26848 million MACs**; saving ≈ **95.33%**. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone.\n"},{"title":"Batch normalization inside a residual network","blocks":["Definition. Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training.
","For one feature with m values in a training batch:
","![\"\\mu_B=\\frac{1}{m}\\sum_{i=1}^{m}x_i\"](\"assets/equations/a91b1c9216cbd799.svg\")
![\"\\sigma_B^2=\\frac{1}{m}\\sum_{i=1}^{m}(x_i-\\mu_B)^2\"](\"assets/equations/7dcfc8cc5fd3adc8.svg\")
![\"\\hat{x}_i=\\frac{x_i-\\mu_B}{\\sqrt{\\sigma_B^2+\\epsilon}}\"](\"assets/equations/13126b94bcd9d415.svg\")
![\"y_i=\\gamma\\hat{x}_i+\\beta\"](\"assets/equations/7fd3e426f48f08b9.svg\")
The batch mean μ_B is the centre; batch variance σ²_B measures squared spread. Epsilon ε is a small positive stability constant inside the square root. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch.
","Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.
","x₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5.
","Batch mean, variance, normalized values and scale-and-shift outputs.
","Use the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale.
","Step 1: mean.
","![\"\\mu_B=\\frac{2+4}{2}=3\"](\"assets/equations/dcbc368ca301a562.svg\")
Step 2: deviations. 2 − 3 = −1 and 4 − 3 = 1.
","Step 3: squared deviations and variance.
","![\"\\sigma_B^2=\\frac{(-1)^2+1^2}{2}=\\frac{1+1}{2}=1\"](\"assets/equations/f8c2f38c1c4da922.svg\")
Step 4: stable denominator.
","![\"d=\\sqrt{1+0.00001}\\approx1.00000499999\"](\"assets/equations/64e8bea52d19d8d8.svg\")
Step 5: normalize separately.
","![\"\\hat{x}_1=\\frac{-1}{d}\\approx-0.99999500004\"](\"assets/equations/3676422778896944.svg\")
![\"\\hat{x}_2=\\frac{1}{d}\\approx0.99999500004\"](\"assets/equations/3c9c2142072c8bc9.svg\")
Step 6: scale the first normalized value, then shift it.
","![\"y_1=2(-0.99999500004)+0.5\\approx-1.49999000008\"](\"assets/equations/4655c2949211af8b.svg\")
Step 7: scale the second normalized value, then shift it.
","![\"y_2=2(0.99999500004)+0.5\\approx2.49999000008\"](\"assets/equations/3848a4f079b59c74.svg\")
Final answer: output ≈ [−1.49999,2.49999]. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here.
","Yaad rakho: centre hatao → spread se divide → learned scale lagao → shift add karo.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s15","search":"Batch normalization inside a residual network \n**Definition.** Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training.\n\nFor one feature with m values in a training batch:\n\n\n\n\n\nThe batch mean μ_B is the centre; batch variance σ²_B measures squared spread. Epsilon ε is a small positive stability constant **inside the square root**. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch.\n\n**Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.**\n\n### Given\nx₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5.\n\n### To find\nBatch mean, variance, normalized values and scale-and-shift outputs.\n\n### Formula — which one and why\nUse the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale.\n\n### Solution — step by step\n**Step 1: mean.**\n\n\n**Step 2: deviations.** 2 − 3 = −1 and 4 − 3 = 1.\n\n**Step 3: squared deviations and variance.**\n\n\n**Step 4: stable denominator.**\n\n\n**Step 5: normalize separately.**\n\n\n\n**Step 6: scale the first normalized value, then shift it.**\n\n\n**Step 7: scale the second normalized value, then shift it.**\n\n\n**Final answer:** output ≈ **[−1.49999,2.49999]**. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here.\n\n**Yaad rakho:** centre hatao → spread se divide → learned scale lagao → shift add karo.\n"},{"title":"Residual-block order and architecture qualifications","blocks":["The original post-activation basic block has this sequence: convolution → BN → ReLU → convolution → BN → add shortcut → ReLU. Thus the second branch convolution is not followed by a separate ReLU before the addition in this particular block. A pre-activation ResNet is another design; use the stated diagram.
","An identity shortcut has no learned parameters. A projection shortcut changes the shape using learned weights. Original ImageNet ResNets include an initial max-pooling operation; a basic residual block does not need a max-pooling layer inside it. ResNet-18/34 use two-convolution basic blocks, whereas deeper standard variants use bottleneck blocks. Avoid treating every ResNet as the same two-layer diagram.
","The main advantage is easier optimization of deep networks: shortcuts provide an additional information and gradient path. This reduces the degradation problem but does not prove immunity to every vanishing/exploding-gradient case. Large networks still need compute, memory, suitable data and regularization. Reusing early features can help; it does not guarantee that every deeper model generalizes better.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s16","search":"Residual-block order and architecture qualifications \nThe original post-activation basic block has this sequence: **convolution → BN → ReLU → convolution → BN → add shortcut → ReLU**. Thus the second branch convolution is not followed by a separate ReLU before the addition in this particular block. A pre-activation ResNet is another design; use the stated diagram.\n\n\n\nAn identity shortcut has no learned parameters. A projection shortcut changes the shape using learned weights. Original ImageNet ResNets include an initial max-pooling operation; a basic residual block does not need a max-pooling layer inside it. ResNet-18/34 use two-convolution basic blocks, whereas deeper standard variants use bottleneck blocks. Avoid treating every ResNet as the same two-layer diagram.\n\nThe main advantage is easier optimization of deep networks: shortcuts provide an additional information and gradient path. This reduces the degradation problem but does not prove immunity to every vanishing/exploding-gradient case. Large networks still need compute, memory, suitable data and regularization. Reusing early features can help; it does not guarantee that every deeper model generalizes better.\n"},{"title":"Choosing frozen layers from dataset size and similarity","blocks":["Early image layers commonly learn reusable edges and textures. Later layers become more specialized to source-task categories. “Earlier” means closer to the input, regardless of whether a drawing places those layers at the top or bottom.
","| Target dataset | Sensible starting choice | Reason |
|---|---|---|
| Small and similar | Freeze the backbone; train a new head first | Useful features already exist, while few examples make overfitting likely |
| Large and similar | Train a new head, then fine-tune more layers if validation improves | More data can support more trainable parameters |
| Small and different | Try a more relevant pretrained model or fixed earlier/intermediate features; tune cautiously | Later source features may be unsuitable, but small data still makes extensive tuning risky |
| Large and different | Compare broad fine-tuning with training from scratch | Sufficient data permits stronger adaptation; the value of source features must be tested |
These are starting strategies, not universal rules. Fine-tuning an early layer means updating its pretrained weights, not automatically learning it “from scratch”. With very little data, simply unfreezing early layers because the task differs can overfit. Choose changes using validation performance and apply the required preprocessing consistently.
","Example. With 100 labelled microscopic images, an ImageNet model may provide reusable low-level patterns, but its animal/object classifier is inappropriate. Replace the head, test fixed representations and monitor validation before increasing the number of trainable layers. With a large representative image dataset, broader fine-tuning becomes a more reasonable candidate.
","Yaad rakho: data kam toh kam weights hilao; task alag toh features ki suitability pehle dekho.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s17","search":"Choosing frozen layers from dataset size and similarity \nEarly image layers commonly learn reusable edges and textures. Later layers become more specialized to source-task categories. “Earlier” means closer to the input, regardless of whether a drawing places those layers at the top or bottom.\n\n| Target dataset | Sensible starting choice | Reason |\n| Small and similar | Freeze the backbone; train a new head first | Useful features already exist, while few examples make overfitting likely |\n| Large and similar | Train a new head, then fine-tune more layers if validation improves | More data can support more trainable parameters |\n| Small and different | Try a more relevant pretrained model or fixed earlier/intermediate features; tune cautiously | Later source features may be unsuitable, but small data still makes extensive tuning risky |\n| Large and different | Compare broad fine-tuning with training from scratch | Sufficient data permits stronger adaptation; the value of source features must be tested |\n\nThese are starting strategies, not universal rules. Fine-tuning an early layer means updating its pretrained weights, not automatically learning it “from scratch”. With very little data, simply unfreezing early layers because the task differs can overfit. Choose changes using validation performance and apply the required preprocessing consistently.\n\n**Example.** With 100 labelled microscopic images, an ImageNet model may provide reusable low-level patterns, but its animal/object classifier is inappropriate. Replace the head, test fixed representations and monitor validation before increasing the number of trainable layers. With a large representative image dataset, broader fine-tuning becomes a more reasonable candidate.\n\n**Yaad rakho:** data kam toh kam weights hilao; task alag toh features ki suitability pehle dekho.\n"},{"title":"Four ways to reuse a pretrained model","blocks":["Definition. A pretrained model has already learned parameters from an earlier training task. We can reuse its complete classifier, reuse its feature extractor, or use its learned weights as the starting point for further training. The choice determines which output labels are available and which parameters change.
","First identify two parts. The backbone converts an input image into useful features. The head converts those features into the required prediction, such as class scores. A new task can reuse the backbone while replacing the head.
","Yaad rakhne ka cue: Purane model ka poora answer use karna, sirf features lena, ya uske weights ko aage train karna—yeh teen decisions alag hain.
","The faculty reference divides these choices into the following four uses. The two feature-extractor uses differ in whether features are saved separately or computed inside the new model.
","| Use | What happens to the pretrained part? | What learns for the target task? | Concrete example |
|---|---|---|---|
| Direct classifier | Keep the complete model and its existing output head. Run it on new inputs. | Nothing is updated during ordinary prediction. | A model already trained to classify cats and dogs labels new cat/dog photographs. It can do this because the required labels match its existing outputs. |
| Standalone feature extractor | Run images through the fixed backbone and save the resulting feature vectors. A feature vector is an ordered list of numbers describing the image. | Train a separate classifier using these saved vectors. | Extract one vector per product photograph, then train a classifier to distinguish acceptable, scratched and cracked products. The extractor stays unchanged. |
| Integrated feature extractor | Place the pretrained backbone inside the new network and freeze it. Add a new head. The backbone calculates features whenever the new model runs. | Train the new head while the backbone's weights remain fixed. | Use a frozen image backbone followed by a new three-output head for acceptable, scratched and cracked products. The complete image-to-class pipeline is one model. |
| Weight initialization followed by fine-tuning | Start from the pretrained weights, then allow selected pretrained layers to update. Replace or adapt the head as required. | Train the new head and the unfrozen part of the backbone. | After first training a product-defect head, unfreeze the last backbone block and train it with a smaller learning rate to adapt to the factory's photographs. |
A cat/dog head has two outputs with learned meanings: cat and dog. Giving it a photograph of a scratched product does not create a new output called “scratched.” For the product task, use an appropriate new head and train it on labeled product examples. Direct reuse of the original classifier is suitable only when its output meanings and preprocessing match the required task. When the task and data distribution are unchanged, this is ordinary inference rather than adaptation to a new task.
","Both methods can keep the backbone frozen. In the standalone method, feature vectors are calculated first and saved as a separate dataset; a later classifier reads that dataset. In the integrated method, each image passes through the backbone and head within one pipeline. Freezing means that the backbone parameters receive no training updates; it does not mean that the backbone stops performing calculations or that its stored weights disappear.
","It is useful when labeled target data are limited and a pretrained model has learned representations relevant to the target task. It can reduce training time and the amount of target data needed. Improvement is not guaranteed: unrelated source features, unsuitable preprocessing or excessive fine-tuning on a small dataset can hurt performance. Choose the frozen or fine-tuned configuration using validation data, then evaluate the chosen model on the held-out test set.
","Exam answer structure: define transfer learning → identify the backbone and head → explain the selected reuse mode → state which weights remain fixed and which learn → give a matching example and a limitation.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c8-s18","search":"Four ways to reuse a pretrained model \n**Definition.** A pretrained model has already learned parameters from an earlier training task. We can reuse its complete classifier, reuse its feature extractor, or use its learned weights as the starting point for further training. The choice determines which output labels are available and which parameters change.\n\n**First identify two parts.** The **backbone** converts an input image into useful features. The **head** converts those features into the required prediction, such as class scores. A new task can reuse the backbone while replacing the head.\n\n**Yaad rakhne ka cue:** Purane model ka poora answer use karna, sirf features lena, ya uske weights ko aage train karna—yeh teen decisions alag hain.\n\nThe faculty reference divides these choices into the following four uses. The two feature-extractor uses differ in whether features are saved separately or computed inside the new model.\n\n| Use | What happens to the pretrained part? | What learns for the target task? | Concrete example |\n| Direct classifier | Keep the complete model and its existing output head. Run it on new inputs. | Nothing is updated during ordinary prediction. | A model already trained to classify cats and dogs labels new cat/dog photographs. It can do this because the required labels match its existing outputs. |\n| Standalone feature extractor | Run images through the fixed backbone and save the resulting feature vectors. A feature vector is an ordered list of numbers describing the image. | Train a separate classifier using these saved vectors. | Extract one vector per product photograph, then train a classifier to distinguish acceptable, scratched and cracked products. The extractor stays unchanged. |\n| Integrated feature extractor | Place the pretrained backbone inside the new network and freeze it. Add a new head. The backbone calculates features whenever the new model runs. | Train the new head while the backbone's weights remain fixed. | Use a frozen image backbone followed by a new three-output head for acceptable, scratched and cracked products. The complete image-to-class pipeline is one model. |\n| Weight initialization followed by fine-tuning | Start from the pretrained weights, then allow selected pretrained layers to update. Replace or adapt the head as required. | Train the new head and the unfrozen part of the backbone. | After first training a product-defect head, unfreeze the last backbone block and train it with a smaller learning rate to adapt to the factory's photographs. |\n\n### Why the existing class labels matter\n\nA cat/dog head has two outputs with learned meanings: cat and dog. Giving it a photograph of a scratched product does not create a new output called “scratched.” For the product task, use an appropriate new head and train it on labeled product examples. Direct reuse of the original classifier is suitable only when its output meanings and preprocessing match the required task. When the task and data distribution are unchanged, this is ordinary inference rather than adaptation to a new task.\n\n### Standalone versus integrated features\n\nBoth methods can keep the backbone frozen. In the standalone method, feature vectors are calculated first and saved as a separate dataset; a later classifier reads that dataset. In the integrated method, each image passes through the backbone and head within one pipeline. Freezing means that the backbone parameters receive no training updates; it does not mean that "}]},{"n":9,"title":"Recurrent neural networks","sections":[{"title":"Start here","blocks":["This recurrent-network material is retained as an additional reference. The faculty update received on 12 October focuses MT2 on Units 3 and 4; use the TEE route for RNN study.
","Definition. A recurrent neural network (RNN) is a neural architecture that models sequential data by updating a hidden state from the current input and the previous state, using shared parameters across time steps.
","![\"h_t=\\phi(W_{xh}x_t+W_{hh}h_{t-1}+b_h)\"](\"assets/equations/c5a9455cdb766031.svg\")
The state hₜ is a learned representation of information available up to time t. It is not a guaranteed lossless record of the entire sequence. The initial state is supplied, initialized to zero or learned, according to the model or question.
","In language, audio and time series, the meaning of an observation can depend on earlier observations and their order. A basic feedforward network treating each input independently has no recurrent state. An RNN allows the output at a time step to depend on previous processing through hₜ₋₁.
","The unrolled cells use the same Wₓₕ, Wₕₕ and bₕ at every time step. A longer input sequence increases the number of evaluations, not the number of independently learned copies of these parameters. Each new hidden state must be calculated before it is used by the next step.
","The TEE route contains the state equation, time-step computations, shared parameters, sequence mappings, BPTT, vanishing/exploding gradients, LSTM and GRU. RNN is outside the current focused MT2 path based on the 12 October faculty update.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c9-s0","search":"What makes a sequence different **Definition.** A recurrent neural network (RNN) is a neural architecture that models sequential data by updating a hidden state from the current input and the previous state, using shared parameters across time steps.\n\nThe state hₜ is a learned representation of information available up to time t. It is not a guaranteed lossless record of the entire sequence. The initial state is supplied, initialized to zero or learned, according to the model or question.\n\n### Why order matters\nIn language, audio and time series, the meaning of an observation can depend on earlier observations and their order. A basic feedforward network treating each input independently has no recurrent state. An RNN allows the output at a time step to depend on previous processing through hₜ₋₁.\n\n\n\n### Parameter sharing\nThe unrolled cells use the same Wₓₕ, Wₕₕ and bₕ at every time step. A longer input sequence increases the number of evaluations, not the number of independently learned copies of these parameters. Each new hidden state must be calculated before it is used by the next step.\n\n### MT2 interpretation\nThe TEE route contains the state equation, time-step computations, shared parameters, sequence mappings, BPTT, vanishing/exploding gradients, LSTM and GRU. RNN is outside the current focused MT2 path based on the 12 October faculty update.\n"},{"title":"Model answer: why recurrent networks use a hidden state","question":["Question. What is a recurrent neural network? Explain how a hidden state allows it to process a sequence and distinguish it from a feedforward network.
"],"answer":["An RNN processes ordered inputs using a hidden state that is carried from one time step to the next. The current state combines information from the new input with information retained from earlier inputs. The same recurrent weights are reused across time steps.
","For a simple tanh RNN:
","![\"h_t=\\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h)\"](\"assets/equations/015d839835ad8254.svg\")
| Symbol | Meaning |
|---|---|
| xₜ | Input at the current time step |
| hₜ₋₁ | Hidden state passed from the previous step |
| Wxh | Weights connecting the current input to the hidden state |
| Whh | Weights connecting the previous state to the current state |
| bₕ | Hidden-state bias |
| hₜ | New state after the activation |
For words in a sentence, earlier words affect the state used when the next word arrives. Changing the order can therefore change the final representation. The state is a learned summary; it is not a guaranteed exact copy of every earlier input.
","A feedforward network maps inputs through its layers without this recurrent state transition. An RNN can accept a sequence, reuse its parameters at each step and produce either an output at each step or an output from a final state, depending on the task. Examples include sentiment classification, next-word prediction and time-series modeling.
","Unrolling an RNN draws its repeated use at t = 1, 2, 3 and so on. Those boxes share parameters; drawing three time steps does not create three independently learned copies of Whh.
"],"kind":"theory","id":"mt2-c9-s1","search":"Model answer: why recurrent networks use a hidden state **Question.** What is a recurrent neural network? Explain how a hidden state allows it to process a sequence and distinguish it from a feedforward network.\n\n**Model answer.**\nAn RNN processes ordered inputs using a hidden state that is carried from one time step to the next. The current state combines information from the new input with information retained from earlier inputs. The same recurrent weights are reused across time steps.\n\n\n\n\nFor a simple tanh RNN:\n\n| Symbol | Meaning |\n| xₜ | Input at the current time step |\n| hₜ₋₁ | Hidden state passed from the previous step |\n| Wxh | Weights connecting the current input to the hidden state |\n| Whh | Weights connecting the previous state to the current state |\n| bₕ | Hidden-state bias |\n| hₜ | New state after the activation |\n\nFor words in a sentence, earlier words affect the state used when the next word arrives. Changing the order can therefore change the final representation. The state is a learned summary; it is not a guaranteed exact copy of every earlier input.\n\nA feedforward network maps inputs through its layers without this recurrent state transition. An RNN can accept a sequence, reuse its parameters at each step and produce either an output at each step or an output from a final state, depending on the task. Examples include sentiment classification, next-word prediction and time-series modeling.\n\nUnrolling an RNN draws its repeated use at t = 1, 2, 3 and so on. Those boxes share parameters; drawing three time steps does not create three independently learned copies of Whh.\n"},{"title":"The vanilla RNN equations from scratch","blocks":["Definition. A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction.
","For a tanh recurrent layer:
","![\"a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h\"](\"assets/equations/99a78e2772b9da6e.svg\")
![\"h_t=\\tanh(a_t)\"](\"assets/equations/e57a9d6af94ad1bb.svg\")
aₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit.
","For classification at time t, an output layer can produce logits and probabilities:
","![\"z_t=W_{hy}h_t+b_y,\\qquad](\"assets/equations/f6b889b0139771ed.svg\")
A regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation.
","Question. Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation.
","h₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0.
","x₁ = 1; x₂ = 2; activation = tanh.
","Find h₁ and h₂ in order.
","Recurrent forward pass. At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.
","x_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.
","![\"a_t=W_{xh}x_t+W_{hh}h_{t-1}+b\"](\"assets/equations/cac638ba5ee98e1c.svg\")
Yaad rakhne ka cue: Naya input + pichhla activated state; phir activation.
","Step 1: Time step 1 — calculate the preactivation.
","![\"a_1=0.5(1)+0.2(0)+0=0.5\"](\"assets/equations/a11d26405f8ffbf6.svg\")
Apply the recurrent activation.
","![\"h_1=\\tanh(0.5)\\approx0.462117157\"](\"assets/equations/bc76aa14d3af5acb.svg\")
This activated value, not a₁ = 0.5, is passed to the next time step.
","Step 2: Time step 2 — combine the new input and previous state.
","![\"a_2=0.5(2)+0.2h_1+0\"](\"assets/equations/a530e8ec85bf260b.svg\")
![\"a_2\\approx1+0.2(0.462117157)\\approx1.092423431\"](\"assets/equations/67ad1cca6bc26681.svg\")
Apply tanh to the complete new preactivation.
","![\"h_2=\\tanh(a_2)\\approx0.797760954\"](\"assets/equations/47f73b96c961ac1f.svg\")
| Time step | Current input | Previous state | Preactivation | New state |
|---|---|---|---|---|
| 1 | 1 | 0 | 0.5 | 0.462117157 |
| 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 |
Final answer: h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c9-s2","search":"The vanilla RNN equations from scratch \n**Definition.** A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction.\n\n### Hidden-state and output equations\nFor a tanh recurrent layer:\n\n\naₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit.\n\nFor classification at time t, an output layer can produce logits and probabilities:\n\nA regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation.\n\n\n\n### Complete numerical: two recurrent time steps\n**Question.** Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation.\n\n\n\n### Given\n\nh₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0.\n\nx₁ = 1; x₂ = 2; activation = tanh.\n\n### To find\n\nFind h₁ and h₂ in order.\n\n### Formula — which one and why\n\n**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.\n\nx_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: Time step 1 — calculate the preactivation.**\n\n**Apply the recurrent activation.**\n\nThis activated value, not a₁ = 0.5, is passed to the next time step.\n\n**Step 2: Time step 2 — combine the new input and previous state.**\n\n\n**Apply tanh to the complete new preactivation.**\n\n\n| Time step | Current input | Previous state | Preactivation | New state |\n| 1 | 1 | 0 | 0.5 | 0.462117157 |\n| 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 |\n\n**Final answer:** h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update.\n\n"},{"title":"A complete vector RNN calculation","blocks":["Question from R26 Q5a. xₜ = [0,1], hₜ₋₁ = [0,1], no bias is supplied, Wxh is shown below, Whh is shown below. Use tanh and rows as receiving neurons.
","Input x = [0,1]ᵀ; previous h = [0,1]ᵀ.
","Wxh = ![\"\\begin{bmatrix}0.6](\"assets/equations/4f3aa9485d9c39f6.svg\")
.
Whh = ![\"\\begin{bmatrix}0.5](\"assets/equations/339cb87bfb0225e4.svg\")
; bias = 0.
Apply tanh separately to the two sums.
","Find the new two-component hidden state; output-layer parameters are not supplied.
","Recurrent forward pass. At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.
","x_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.
","Yaad rakhne ka cue: Naya input + pichhla activated state; phir activation.
","Step 1 new input: Wxh x = [0.6 × 0 + 0.4 × 1, 0.3 × 0 + 0.7 × 1] = [0.4,0.7].
","Step 2 old memory: Whh hprevious = [0.5 × 0 + 0.3 × 1, 0.2 × 0 + 0.4 × 1] = [0.3,0.4].
","Step 3 add: a = [0.4+0.3, 0.7+0.4] = [0.7,1.1].
","Step 4 activate each element: h = [tanh(0.7), tanh(1.1)] = [0.604368,0.800499]. An output-layer prediction cannot be numerically calculated without its weights and bias.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c9-s3","search":"A complete vector RNN calculation **Question from R26 Q5a.** xₜ = [0,1], hₜ₋₁ = [0,1], no bias is supplied, Wxh is shown below, Whh is shown below. Use tanh and rows as receiving neurons.\n\n\n\n### Given\n\nInput x = [0,1]ᵀ; previous h = [0,1]ᵀ.\n\nWxh = .\n\nWhh = ; bias = 0.\n\nApply tanh separately to the two sums.\n\n### To find\n\nFind the new two-component hidden state; output-layer parameters are not supplied.\n\n### Formula — which one and why\n\n**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.\n\nx_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.\n\n### Solution — step by step\n\n\n\n\n\n\n**Step 1 new input: ** Wxh x = [0.6 × 0 + 0.4 × 1, 0.3 × 0 + 0.7 × 1] = [0.4,0.7].\n**Step 2 old memory: ** Whh hprevious = [0.5 × 0 + 0.3 × 1, 0.2 × 0 + 0.4 × 1] = [0.3,0.4].\n**Step 3 add: ** a = [0.4+0.3, 0.7+0.4] = [0.7,1.1].\n**Step 4 activate each element: ** h = [tanh(0.7), tanh(1.1)] = **[0.604368,0.800499]**. An output-layer prediction cannot be numerically calculated without its weights and bias."},{"title":"Sequence mappings and applications","blocks":["Definition. A sequence-model configuration specifies how many inputs and outputs are processed and how their time indices relate. One-to-many, many-to-one and many-to-many configurations correspond to different prediction tasks.
","| Mapping | Meaning | Example and output |
|---|---|---|
| One to many | One input conditions several outputs | Image captioning: image to a word sequence |
| Many to one | Sequence summarized as one output | Review sentiment: words to positive neutral or negative |
| Many to many aligned | Output associated with each input time | Video frame labeling or per-word tagging |
| Many to many unaligned | Output sequence length may differ | Encoder-decoder translation |
Music generation can use earlier notes to predict the next note repeatedly. Next-word prediction uses the prefix to produce a probability distribution over a vocabulary. A word with the largest predicted probability is a simple choice, but the precise prediction cannot be calculated without the learned model's weights.
","Sentence explanation: In “The dogs that bark near my house are loud,” the model should retain the plural subject across the intervening words when choosing “are”. A useful hidden representation should retain subject information while reading later words. This is an interpretation of what a trained model may learn, not a claim that an individual gate is guaranteed to represent grammar.
","Sentiment model answer: tokenize the review, map tokens to numerical embeddings, process them with a vanilla RNN, use a final state or pooled sequence representation, then a dense layer and softmax over three sentiment classes. Train with categorical cross-entropy on labeled reviews; evaluate on held-out data. A bidirectional model is not needed for this answer and is not explicitly listed in the current syllabus.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c9-s4","search":"Sequence mappings and applications **Definition.** A sequence-model configuration specifies how many inputs and outputs are processed and how their time indices relate. One-to-many, many-to-one and many-to-many configurations correspond to different prediction tasks.\n\n### Explanation and working\n\n| Mapping | Meaning | Example and output |\n| One to many | One input conditions several outputs | Image captioning: image to a word sequence |\n| Many to one | Sequence summarized as one output | Review sentiment: words to positive neutral or negative |\n| Many to many aligned | Output associated with each input time | Video frame labeling or per-word tagging |\n| Many to many unaligned | Output sequence length may differ | Encoder-decoder translation |\nMusic generation can use earlier notes to predict the next note repeatedly. Next-word prediction uses the prefix to produce a probability distribution over a vocabulary. A word with the largest predicted probability is a simple choice, but the precise prediction cannot be calculated without the learned model's weights.\n**Sentence explanation: ** In “The dogs that bark near my house are loud,” the model should retain the plural subject across the intervening words when choosing “are”. A useful hidden representation should retain subject information while reading later words. This is an interpretation of what a trained model may learn, not a claim that an individual gate is guaranteed to represent grammar.\n**Sentiment model answer: ** tokenize the review, map tokens to numerical embeddings, process them with a vanilla RNN, use a final state or pooled sequence representation, then a dense layer and softmax over three sentiment classes. Train with categorical cross-entropy on labeled reviews; evaluate on held-out data. A bidirectional model is not needed for this answer and is not explicitly listed in the current syllabus.\n"},{"title":"Shared weights and RNN applications","blocks":["Definition. Temporal parameter sharing means that one recurrent transition function is reused at each sequence step. It permits variable sequence lengths without learning a separate transition matrix for each position.
","The RNN applies the same input and recurrent weights at every time step. The hidden state changes as new inputs arrive; the learned rule itself is reused. This is temporal weight sharing.
","For sentiment classification, a sequence of words leads to one sentiment label: many-to-one. For image captioning, an image representation conditions a sequence of words: one-to-many. These patterns describe how inputs and outputs are arranged, not separate update rules.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c9-s5","search":"Shared weights and RNN applications **Definition.** Temporal parameter sharing means that one recurrent transition function is reused at each sequence step. It permits variable sequence lengths without learning a separate transition matrix for each position.\n\n### Explanation and working\nThe RNN applies the same input and recurrent weights at every time step. The hidden state changes as new inputs arrive; the learned rule itself is reused. This is temporal weight sharing.\n\n\n\nFor sentiment classification, a sequence of words leads to one sentiment label: many-to-one. For image captioning, an image representation conditions a sequence of words: one-to-many. These patterns describe how inputs and outputs are arranged, not separate update rules.\n"}]},{"n":10,"title":"Prerequisite repair and teacher backprop examples","sections":[{"title":"Start here","blocks":["Sources S03, S01 and the three Teams photographs supplied later. Use the short neuron and gradient questions first. The longer teacher examples are backup practice after completing the Unit 3 and Unit 4 priorities; their presence here is not a confirmed MT2 announcement.
","Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible.
"],"kind":"read","id":"mt2-c10-intro"},{"title":"MT1A Q2","question":["Question (3 marks). A sigmoid neuron has inputs [2,1,3], weights [0.4,−0.2,0.5] and bias 0.1. Compute its output.
"],"answer":["Inputs x₁ = 2,x₂ = 1,x₃ = 3.
","Matching weights w₁ = 0.4,w₂ = −0.2,w₃ = 0.5.
","Bias b = 0.1; activation = sigmoid.
","Calculate net input z and activated output a.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid output. The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.
","z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.
","Yaad rakhne ka cue: Minus-z ki e power, plus one, phir one divided by whole denominator.
","Step 1: multiply input 1 by weight 1.
","Step 2: multiply input 2 by weight 2.
","The minus sign belongs to the weight, so the contribution stays negative.
","Step 3: multiply input 3 by weight 3.
","Step 4: add the three contributions.
","Step 5: add the bias once.
","This is the preactivation. It is not yet the requested sigmoid output.
","Step 6: negate z for the exponential.
","Step 7: calculate the complete denominator.
","Step 8: divide one by that denominator.
","Final answer: sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025.
"],"kind":"practice","id":"mt2-c10-s0","search":"MT1A Q2 **Question (3 marks).** A sigmoid neuron has inputs [2,1,3], weights [0.4,−0.2,0.5] and bias 0.1. Compute its output.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInputs x₁ = 2,x₂ = 1,x₃ = 3.\n\nMatching weights w₁ = 0.4,w₂ = −0.2,w₃ = 0.5.\n\nBias b = 0.1; activation = sigmoid.\n\n### To find\n\nCalculate net input z and activated output a.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.\n\nz = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.\n\n\n\n**Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator.\n\n### Solution — step by step\n\n\n\n\n\n\n**Step 1: multiply input 1 by weight 1.**\n\n\n**Step 2: multiply input 2 by weight 2.**\n\nThe minus sign belongs to the weight, so the contribution stays negative.\n\n**Step 3: multiply input 3 by weight 3.**\n\n\n**Step 4: add the three contributions.**\n\n\n**Step 5: add the bias once.**\n\nThis is the preactivation. It is not yet the requested sigmoid output.\n\n**Step 6: negate z for the exponential.**\n\n\n\n**Step 7: calculate the complete denominator.**\n\n\n**Step 8: divide one by that denominator.**\n\n**Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025.\n"},{"title":"MT1A Q3","question":["Question (3 marks). For z = [−1,0,1], calculate sigmoid, softmax and Leaky ReLU with α = 0.01 (an explicit assumption because the paper does not give the slope).
"],"answer":["Scores z = [−1,0,1].
","Calculate sigmoid, softmax and Leaky ReLU.
","The paper omits the Leaky slope: α = 0.01 is the stated practice assumption.
","Calculate every component under each of the three activations.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.
","zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.
","![\"M=\\max_j](\"assets/equations/1a279f991a1a98b5.svg\")
![\"p_j=\\frac{e^{z_j-M}}{\\sum_k](\"assets/equations/a8375be5e6a8189d.svg\")
Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Sigmoid is applied separately: σ(−1) = ![\"\\frac{1}{1+e^{1}}\"](\"assets/equations/983a154bf3030fc0.svg\")
= 0.268941421; σ(0) = 1/2 = 0.5; σ(1) = ![\"\\frac{1}{1+e^{-1}}\"](\"assets/equations/b1ad72f023278d5b.svg\")
= 0.731058579.
Step 1 stabilize: subtract the largest logit 1 from every value: [-2, -1, 0]. This does not change softmax.
","Step 2 exponentiate: exp values = [0.135335283, 0.367879441, 1].
","Step 3 denominator: sum = 1.503214724.
","Step 4 divide each value by this same sum: probabilities = [0.090030573, 0.244728471, 0.665240956]. Check: all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.
","Leaky ReLU multiplies only negative values by α: [0.01 × (−1),0,1] = [−0.01,0,1]. Softmax normalizes the entire vector; independent sigmoid values need not sum to 1.
"],"kind":"practice","id":"mt2-c10-s1","search":"MT1A Q3 **Question (3 marks).** For z = [−1,0,1], calculate sigmoid, softmax and Leaky ReLU with α = 0.01 (an explicit assumption because the paper does not give the slope).\n\n**Complete worked answer.**\n\n\n\n### Given\n\nScores z = [−1,0,1].\n\nCalculate sigmoid, softmax and Leaky ReLU.\n\nThe paper omits the Leaky slope: α = 0.01 is the stated practice assumption.\n\n### To find\n\nCalculate every component under each of the three activations.\n\n### Formula — which one and why\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.\n\nzⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\n\n\nSigmoid is applied separately: σ(−1) = \\(\\frac{1}{1+e^{1}}\\) = 0.268941421; σ(0) = 1/2 = 0.5; σ(1) = \\(\\frac{1}{1+e^{-1}}\\) = 0.731058579.\n**Step 1 stabilize: ** subtract the largest logit 1 from every value: [-2, -1, 0]. This does not change softmax.\n\n**Step 2 exponentiate: ** exp values = [0.135335283, 0.367879441, 1].\n\n**Step 3 denominator: ** sum = 1.503214724.\n\n**Step 4 divide each value by this same sum: ** probabilities = **[0.090030573, 0.244728471, 0.665240956]**. **Check: ** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.\nLeaky ReLU multiplies only negative values by α: [0.01 × (−1),0,1] = **[−0.01,0,1]**. Softmax normalizes the entire vector; independent sigmoid values need not sum to 1.\n"},{"title":"MT1A Q4","question":["Question (3 marks). Input x = [2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1]; W₂ is shown below; b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.
","Input x = [2].
","W₁ is printed with the question.
","b₁ = [0.1].
","W₂ is printed with the question.
","b₂ = [0.2].
","Hidden and output activations are ReLU.
","Target y = [1].
","Find all preactivations, hidden activations and final outputs. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/11165a2e9ad9cccb.svg\")
![\"h_{1}](\"assets/equations/51f4d0c29cd3e471.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/ef13baba40338fd0.svg\")
![\"o_{1}](\"assets/equations/965f3f37dce04d5d.svg\")
Step 2 loss convention. The question specifies half squared loss. No learning rate is supplied, so report gradients, not invented weight updates.
","Prediction minus target = [0.08]. Squared errors = [0.0064]. Half-sum squared loss = 0.0032; mean squared error = 0.0064. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/156b50e8d480457c.svg\")
The ReLU derivative is 1 here because this preactivation is positive.
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=\\frac{\\partial](\"assets/equations/267f6c273b5a3588.svg\")
![\"g_{1,1}^{(2)}=(0.08)\\times(1.1)\\approx](\"assets/equations/58dabafbc78878c9.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.8)\\times(0.08)\\approx](\"assets/equations/234bc238f68f3454.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/37af0a1e41171eca.svg\")
![\"\\delta_{h_1}](\"assets/equations/b10ee7f796aaf918.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=\\frac{\\partial](\"assets/equations/cb514e27c4d52804.svg\")
![\"g_{1,1}^{(1)}=(0.064)\\times(2)\\approx](\"assets/equations/12beed29e7a45da4.svg\")
Requested answers: dL/dw₂ = 0.088 and dL/db₂ = 0.08. Hidden gradients above explain the dependency and are additional practice.
"],"kind":"practice","id":"mt2-c10-s2","search":"MT1A Q4 **Question (3 marks).** Input x = [2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1]; W₂ is shown below; b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput x = [2].\n\nW₁ is printed with the question.\n\nb₁ = [0.1].\n\nW₂ is printed with the question.\n\nb₂ = [0.2].\n\nHidden and output activations are ReLU.\n\nTarget y = [1].\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** The question specifies half squared loss. No learning rate is supplied, so report gradients, not invented weight updates.\nPrediction minus target = [0.08]. Squared errors = [0.0064]. Half-sum squared loss = 0.0032; mean squared error = 0.0064. These are different unless the number of output components makes the scale coincide.\n**Output delta 1:**\n\nThe ReLU derivative is 1 here because this preactivation is positive.\n**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.\nh1 → o1: gradient = (0.08) × 1.1 = 0.088.\n**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.\n**Hidden 1 downstream contribution:**\n\n\n**Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output.\nx1 → h1: gradient = (0.064) × (2) = 0.128.\n**Requested answers: ** dL/dw₂ = **0.088** and dL/db₂ = **0.08**. Hidden gradients above explain the dependency and are additional practice.\n"},{"title":"MT1B Q3","question":["Question (3 marks). Calculate sigmoid, ReLU and tanh for z = [−2,0,2].
"],"answer":["Input scores = [−2,0,2].
","Required activations: sigmoid, ReLU and tanh.
","Calculate all three activation outputs for each score.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Tanh and its slope. Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².
","z = value entering tanh; a = its output. A prime means derivative, or local rate of change.
","![\"a=\\tanh](\"assets/equations/11b2c40e39b0ee66.svg\")
![\"\\frac{da}{dz}=1-a^2\"](\"assets/equations/6b6ac5e0c1df9b26.svg\")
Yaad rakhne ka cue: Tanh signed output deta hai; slope = 1 minus output ka square.
","Step 1: negate z for sigmoid.
","![\"-z=2\"](\"assets/equations/baed17557936141e.svg\")
Step 2: evaluate the exponential.
","![\"e^{-z}\\approx7.389056099\"](\"assets/equations/ee2ca92f60ab98c1.svg\")
Step 3: add one to form the denominator.
","![\"1+7.389056099=8.389056099\"](\"assets/equations/c833999cba87b62d.svg\")
Step 4: take the reciprocal for sigmoid.
","![\"\\sigma(-2)=\\frac1{8.389056099}\\approx0.119202922\"](\"assets/equations/c492fd878121169c.svg\")
Step 5: compare z with zero for ReLU.
","![\"\\mathrm{ReLU}(-2)=\\max(0,-2)=0\"](\"assets/equations/eb663510f8ab435b.svg\")
Step 6: calculate the tanh numerator.
","![\"e^{-2}-e^{2}\\approx0.135335283-7.389056099=-7.253720816\"](\"assets/equations/7d6baa3ddebec66e.svg\")
Step 7: calculate the tanh denominator.
","![\"e^{-2}+e^{2}\\approx0.135335283+7.389056099=7.524391382\"](\"assets/equations/ee253631806c17e2.svg\")
Step 8: divide for tanh.
","![\"\\tanh(-2)\\approx\\frac{-7.253720816}{7.524391382}\\approx-0.964027580\"](\"assets/equations/4bb67612e76e38e6.svg\")
Step 1: negate z for sigmoid.
","![\"-z=0\"](\"assets/equations/3aebd020c0eccec6.svg\")
Step 2: evaluate the exponential.
","![\"e^{-z}\\approx1.000000000\"](\"assets/equations/f5ab9de96f693147.svg\")
Step 3: add one to form the denominator.
","![\"1+1.000000000=2.000000000\"](\"assets/equations/29756201dd5e46c6.svg\")
Step 4: take the reciprocal for sigmoid.
","![\"\\sigma(0)=\\frac1{2.000000000}\\approx0.500000000\"](\"assets/equations/fc3feba9268d5c6f.svg\")
Step 5: compare z with zero for ReLU.
","![\"\\mathrm{ReLU}(0)=\\max(0,0)=0\"](\"assets/equations/8512edeebdecaad0.svg\")
Step 6: calculate the tanh numerator.
","![\"e^{0}-e^{0}\\approx1.000000000-1.000000000=0.000000000\"](\"assets/equations/2df025bea6def52b.svg\")
Step 7: calculate the tanh denominator.
","![\"e^{0}+e^{0}\\approx1.000000000+1.000000000=2.000000000\"](\"assets/equations/6c3fb9efcb35a389.svg\")
Step 8: divide for tanh.
","![\"\\tanh(0)\\approx\\frac{0.000000000}{2.000000000}\\approx0.000000000\"](\"assets/equations/3179ee482059cbdd.svg\")
Step 1: negate z for sigmoid.
","![\"-z=-2\"](\"assets/equations/e116d9c93aba3d51.svg\")
Step 2: evaluate the exponential.
","![\"e^{-z}\\approx0.135335283\"](\"assets/equations/63363cd12105c12d.svg\")
Step 3: add one to form the denominator.
","![\"1+0.135335283=1.135335283\"](\"assets/equations/4e918224e5cd8eaa.svg\")
Step 4: take the reciprocal for sigmoid.
","![\"\\sigma(2)=\\frac1{1.135335283}\\approx0.880797078\"](\"assets/equations/b29d72f7254a4877.svg\")
Step 5: compare z with zero for ReLU.
","![\"\\mathrm{ReLU}(2)=\\max(0,2)=2\"](\"assets/equations/c454b416d112784d.svg\")
Step 6: calculate the tanh numerator.
","![\"e^{2}-e^{-2}\\approx7.389056099-0.135335283=7.253720816\"](\"assets/equations/beea5fae2dfeb83f.svg\")
Step 7: calculate the tanh denominator.
","![\"e^{2}+e^{-2}\\approx7.389056099+0.135335283=7.524391382\"](\"assets/equations/597fbd34bf357417.svg\")
Step 8: divide for tanh.
","![\"\\tanh(2)\\approx\\frac{7.253720816}{7.524391382}\\approx0.964027580\"](\"assets/equations/6c415e8a033259d4.svg\")
Final answer: sigmoid = [0.119202922, 0.5, 0.880797078]; ReLU = [0, 0, 2]; tanh = [−0.964027580, 0, 0.964027580].
"],"kind":"practice","id":"mt2-c10-s3","search":"MT1B Q3 **Question (3 marks).** Calculate sigmoid, ReLU and tanh for z = [−2,0,2].\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput scores = [−2,0,2].\n\nRequired activations: sigmoid, ReLU and tanh.\n\n### To find\n\nCalculate all three activation outputs for each score.\n\n### Formula — which one and why\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n**Tanh and its slope.** Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².\n\nz = value entering tanh; a = its output. A prime means derivative, or local rate of change.\n\n\n\n\n\n**Yaad rakhne ka cue:** Tanh signed output deta hai; slope = 1 minus output ka square.\n\n### Solution — step by step\n\n\n\n\n\n\n### Preactivation z = -2\n\n**Step 1: negate z for sigmoid.**\n\n\n\n\n\n\n\n**Step 2: evaluate the exponential.**\n\n\n\n\n\n\n\n**Step 3: add one to form the denominator.**\n\n\n\n\n\n\n\n**Step 4: take the reciprocal for sigmoid.**\n\n\n\n\n\n\n\n**Step 5: compare z with zero for ReLU.**\n\n\n\n\n\n\n\n**Step 6: calculate the tanh numerator.**\n\n\n\n\n\n\n\n**Step 7: calculate the tanh denominator.**\n\n\n\n\n\n\n\n**Step 8: divide for tanh.**\n\n\n\n\n\n\n\n\n### Preactivation z = 0\n\n**Step 1: negate z for sigmoid.**\n\n\n\n\n\n\n\n**Step 2: evaluate the exponential.**\n\n\n\n\n\n\n\n**Step 3: add one to form the denominator.**\n\n\n\n\n\n\n\n**Step 4: take the reciprocal for sigmoid.**\n\n\n\n\n\n\n\n**Step 5: compare z with zero for ReLU.**\n\n\n\n\n\n\n\n**Step 6: calculate the tanh numerator.**\n\n\n\n\n\n\n\n**Step 7: calculate the tanh denominator.**\n\n\n\n\n\n\n\n**Step 8: divide for tanh.**\n\n\n\n\n\n\n\n\n### Preactivation z = 2\n\n**Step 1: negate z for sigmoid.**\n\n\n\n\n\n\n\n**Step 2: evaluate the exponential.**\n\n\n\n\n\n\n\n**Step 3: add one to form the denominator.**\n\n\n\n\n\n\n\n**Step 4: take the reciprocal for sigmoid.**\n\n\n\n\n\n\n\n**Step 5: compare z with zero for ReLU.**\n\n\n\n\n\n\n\n**Step 6: calculate the tanh numerator.**\n\n\n\n\n\n\n\n**Step 7: calculate the tanh denominator.**\n\n\n\n\n\n\n\n**Step 8: divide for tanh.**\n\n\n\n\n\n\n\n**Final answer:** sigmoid = [0.119202922, 0.5, 0.880797078]; ReLU = [0, 0, 2]; tanh = [−0.964027580, 0, 0.964027580].\n"},{"title":"MT1B Q4","question":["Question (3 marks). Input x = [3]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are sigmoid. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.
","Input x = [3].
","W₁ is printed with the question.
","b₁ = [0.2].
","W₂ is printed with the question.
","b₂ = [0.1].
","Hidden and output activations are sigmoid.
","Target y = [1].
","Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/624d0aa704cb40ed.svg\")
![\"h_{1}=\\frac{1}{1+e^{-(1.4)}}=0.802183889\"](\"assets/equations/c20e39027aee1353.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/f73d73bef40a3e53.svg\")
![\"o_{1}=\\frac{1}{1+e^{-(0.581310333)}}=0.641368858\"](\"assets/equations/72c83495e26a383d.svg\")
Step 2 loss convention. Use the half squared loss stated in this question. No learning rate is supplied.
","Prediction minus target = [-0.358631142]. Squared errors = [0.128616296]. Half-sum squared loss = 0.064308148; mean squared error = 0.128616296. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/4b919c902c0118b6.svg\")
The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=(-0.082490487)\\times(0.802183889)\\approx](\"assets/equations/275f10a4f3768fee.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.6)\\times(-0.082490487)\\approx](\"assets/equations/d8c0fdc1c33b51f1.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/8e5ced5b8d72aefa.svg\")
![\"\\delta_{h_1}](\"assets/equations/d94f5a36d64f33c1.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=(-0.007853997)\\times(3)\\approx](\"assets/equations/8fbb5c252349f4b6.svg\")
Requested answers: dL/dw₂ = −0.066172539; dL/db₂ = −0.082490487. Negative gradients mean ordinary gradient descent would increase those parameters.
"],"kind":"practice","id":"mt2-c10-s4","search":"MT1B Q4 **Question (3 marks).** Input x = [3]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are sigmoid. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput x = [3].\n\nW₁ is printed with the question.\n\nb₁ = [0.2].\n\nW₂ is printed with the question.\n\nb₂ = [0.1].\n\nHidden and output activations are sigmoid.\n\nTarget y = [1].\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** Use the half squared loss stated in this question. No learning rate is supplied.\nPrediction minus target = [-0.358631142]. Squared errors = [0.128616296]. Half-sum squared loss = 0.064308148; mean squared error = 0.128616296. These are different unless the number of output components makes the scale coincide.\n**Output delta 1:**\n\nThe derivative is o(1−o).\n**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.\nh1 → o1: gradient = (-0.082490487) × 0.802183889 = -0.066172539.\n**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.\n**Hidden 1 downstream contribution:**\n\n\n**Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output.\nx1 → h1: gradient = (-0.007853997) × (3) = -0.02356199.\n**Requested answers: ** dL/dw₂ = **−0.066172539**; dL/db₂ = **−0.082490487**. Negative gradients mean ordinary gradient descent would increase those parameters.\n"},{"title":"Teacher vectorized ReLU example","blocks":["Question. Input x = [2, 3]. Each row below lists the incoming weights of one receiving neuron. W₁ = ![\"\\begin{bmatrix}0.11](\"assets/equations/386eee5b3bdd13ab.svg\")
; b₁ = [0, 0]; W₂ = ![\"\\begin{bmatrix}0.14](\"assets/equations/d33649adcaa2e9c5.svg\")
; b₂ = [0]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.05. Use half squared loss and perform forward and backward passes.
Input x = [2, 3].
","W₁ = .
b₁ = [0, 0].
","W₂ = .
b₂ = [0].
","Hidden and output activations are ReLU.
","Target y = [1].
","Learning rate η = 0.05.
","Use half squared loss and perform forward and backward passes.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/89d0bf0bfbd0eee8.svg\")
![\"h_{1}](\"assets/equations/86a3de78168e02c1.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/71f2cfea44a277ff.svg\")
![\"h_{2}](\"assets/equations/b5c17f7904703c18.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/30c6287eca7be224.svg\")
![\"o_{1}](\"assets/equations/0510bc5bdad8a8b2.svg\")
Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [-0.809]. Squared errors = [0.654481]. Half-sum squared loss = 0.3272405; mean squared error = 0.654481. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/d0231cff1e5d205d.svg\")
The ReLU derivative is 1 here because this preactivation is positive.
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.0343825. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=(-0.809)\\times(0.85)\\approx](\"assets/equations/a7b158dc1aab8c85.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,1,\\mathrm{new}}=0.14-0.05(-0.68765)\\approx](\"assets/equations/5a6b892831ec0aa5.svg\")
h2 → o1:
","Subtract η × gradient = −0.019416. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,2}^{(2)}=\\frac{\\partial](\"assets/equations/2314d47ce6468a48.svg\")
![\"g_{1,2}^{(2)}=(-0.809)\\times(0.48)\\approx](\"assets/equations/d09d2ce4b3621339.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,2,\\mathrm{new}}=0.15-0.05(-0.38832)\\approx](\"assets/equations/780b675c44e8ec0a.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.14)\\times(-0.809)\\approx](\"assets/equations/9be822c811879a17.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/e31db28d376c594d.svg\")
![\"\\delta_{h_1}](\"assets/equations/0406bf395ccea595.svg\")
Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.15)\\times(-0.809)\\approx](\"assets/equations/628dab575705c1e4.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/f74169ed59976f22.svg\")
![\"\\delta_{h_2}](\"assets/equations/73149a9e0e57e742.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = −0.011326. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=(-0.11326)\\times(2)\\approx](\"assets/equations/6540c5b7941f90d3.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,1,\\mathrm{new}}=0.11-0.05(-0.22652)\\approx](\"assets/equations/857c0ad228d64fc5.svg\")
x2 → h1:
","Subtract η × gradient = −0.016989. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,2}^{(1)}=\\frac{\\partial](\"assets/equations/a7e2552eb055cb6f.svg\")
![\"g_{1,2}^{(1)}=(-0.11326)\\times(3)\\approx](\"assets/equations/93e0a3007b972754.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,2,\\mathrm{new}}=0.21-0.05(-0.33978)\\approx](\"assets/equations/3e18578432e574df.svg\")
x1 → h2:
","Subtract η × gradient = −0.012135. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,1}^{(1)}=\\frac{\\partial](\"assets/equations/d8f0b089bd0fe450.svg\")
![\"g_{2,1}^{(1)}=(-0.12135)\\times(2)\\approx](\"assets/equations/c5d0a80d06a8382c.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,1,\\mathrm{new}}=0.12-0.05(-0.2427)\\approx](\"assets/equations/f91b7787569a2882.svg\")
x2 → h2:
","Subtract η × gradient = −0.0182025. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,2}^{(1)}=\\frac{\\partial](\"assets/equations/b5ce6325e440d113.svg\")
![\"g_{2,2}^{(1)}=(-0.12135)\\times(3)\\approx](\"assets/equations/2f8d27aa1e3f5627.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,2,\\mathrm{new}}=0.08-0.05(-0.36405)\\approx](\"assets/equations/6cd428530ca786f9.svg\")
Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","![\"b_{h_1,\\mathrm{new}}=0-0.05(-0.113260000)=0.005663000\"](\"assets/equations/2bb2563809fc624e.svg\")
![\"b_{h_2,\\mathrm{new}}=0-0.05(-0.121350000)=0.006067500\"](\"assets/equations/2db77709ff0c80b8.svg\")
![\"b_{o_1,\\mathrm{new}}=0-0.05(-0.809000000)=0.040450000\"](\"assets/equations/6a95d59624552356.svg\")
These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
","Correction to the handwritten working: the hidden deltas are negative (−0.11326 and −0.12135). A positive value at that step is a sign error. Keep output error as prediction minus target throughout.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c10-s5","search":"Teacher vectorized ReLU example \n**Question.** Input x = [2, 3]. Each row below lists the incoming weights of one receiving neuron. W₁ = ; b₁ = [0, 0]; W₂ = ; b₂ = [0]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.05. Use half squared loss and perform forward and backward passes.\n\n\n\n### Given\n\nInput x = [2, 3].\n\nW₁ = .\n\nb₁ = [0, 0].\n\nW₂ = .\n\nb₂ = [0].\n\nHidden and output activations are ReLU.\n\nTarget y = [1].\n\nLearning rate η = 0.05.\n\nUse half squared loss and perform forward and backward passes.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.\nPrediction minus target = [-0.809]. Squared errors = [0.654481]. Half-sum squared loss = 0.3272405; mean squared error = 0.654481. These are different unless the number of output components makes the scale coincide.\n**Output delta 1:**\n\nThe ReLU derivative is 1 here because this preactivation is positive.\n**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.\nh1 → o1: gradient = (-0.809) × 0.85 = -0.68765. New weight = 0.14−0.05 × (-0.68765) = **0.1743825**.\nh2 → o1: gradient = (-0.809) × 0.48 = -0.38832. New weight = 0.15−0.05 × (-0.38832) = **0.169416**.\n**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.\n*"},{"title":"Teams photos full sigmoid example","blocks":["The three photographs show this 2 → 2 → 2 network and its chain rule. The photo label for w₃ is read as 0.25, consistent with the visible output/error values and the matching worked example in S05. Learning rate is not legible/specified in these photos; η = 0.5 below is an explicit practice assumption.
","Question. Input x = [0.05, 0.1]. Each row below lists the incoming weights of one receiving neuron. W₁ = ![\"\\begin{bmatrix}0.15](\"assets/equations/82d4501fc485058b.svg\")
; b₁ = [0.35, 0.35]; W₂ = ![\"\\begin{bmatrix}0.4](\"assets/equations/a6699cc1064bfaf3.svg\")
; b₂ = [0.6, 0.6]. Hidden and output activations are sigmoid. Target y = [0.01, 0.99]. Learning rate η = 0.5. Compute all outputs, total half squared loss, all eight weight gradients and one simultaneous weight update. Map weights in order: w₁ = x₁ → h₁, w₂ = x₂ → h₁, w₃ = x₁ → h₂, w₄ = x₂ → h₂; w₅ = h₁ → o₁, w₆ = h₂ → o₁, w₇ = h₁ → o₂, w₈ = h₂ → o₂.
Input x = [0.05, 0.1].
","W₁ = .
b₁ = [0.35, 0.35].
","W₂ = .
b₂ = [0.6, 0.6].
","Hidden and output activations are sigmoid.
","Target y = [0.01, 0.99].
","Learning rate η = 0.5.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/feca30d72e6cdb12.svg\")
![\"h_{1}=\\frac{1}{1+e^{-(0.3775)}}=0.593269992\"](\"assets/equations/e586b2530e4f9cf0.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/0d0fc7f16047fdb4.svg\")
![\"h_{2}=\\frac{1}{1+e^{-(0.3925)}}=0.596884378\"](\"assets/equations/78d6d0a3e42830ce.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/587365feb9b3026e.svg\")
![\"o_{1}=\\frac{1}{1+e^{-(1.105905967)}}=0.75136507\"](\"assets/equations/a028b62b987f7076.svg\")
Output neuron 2:
","![\"z_{o_2}](\"assets/equations/a3d7358ccdf52d31.svg\")
![\"o_{2}=\\frac{1}{1+e^{-(1.224921404)}}=0.772928465\"](\"assets/equations/378ac86a046b16fc.svg\")
Step 2 loss convention. The photos explicitly use ½(t₁−o₁)² + ½(t₂−o₂)². With exactly two output components, this equals their mean squared error, 0.298371109. The unhalved sum is 0.596742218.
","Prediction minus target = [0.74136507, -0.217071535]. Squared errors = [0.549622166, 0.047120051]. Half-sum squared loss = 0.298371109; mean squared error = 0.298371109. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/ffde7979414e34c0.svg\")
The derivative is o(1−o).
","Output delta 2:
","![\"\\delta_{o_2}](\"assets/equations/9e7235b42f1e0b22.svg\")
The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = 0.04108352. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=(0.138498562)\\times(0.593269992)\\approx](\"assets/equations/8d844a8676791dfd.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,1,\\mathrm{new}}=0.4-0.5(0.082167041)\\approx](\"assets/equations/3afaa45378006799.svg\")
h2 → o1:
","Subtract η × gradient = 0.041333814. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,2}^{(2)}=(0.138498562)\\times(0.596884378)\\approx](\"assets/equations/df479afdae4fe00d.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,2,\\mathrm{new}}=0.45-0.5(0.082667628)\\approx](\"assets/equations/4ad573638df0109c.svg\")
h1 → o2:
","Subtract η × gradient = −0.01130127. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{2,1}^{(2)}=\\frac{\\partial](\"assets/equations/dde4ee7e9e8f997c.svg\")
![\"g_{2,1}^{(2)}=(-0.038098237)\\times(0.593269992)\\approx](\"assets/equations/f032d0a14a103c60.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{2,1,\\mathrm{new}}=0.5-0.5(-0.02260254)\\approx](\"assets/equations/62292012970cdab4.svg\")
h2 → o2:
","Subtract η × gradient = −0.011370121. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{2,2}^{(2)}=\\frac{\\partial](\"assets/equations/57d7e754e94fe533.svg\")
![\"g_{2,2}^{(2)}=(-0.038098237)\\times(0.596884378)\\approx](\"assets/equations/02d5efeb8dafde38.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{2,2,\\mathrm{new}}=0.55-0.5(-0.022740242)\\approx](\"assets/equations/af1284429e824dd6.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.4)\\times(0.138498562)\\approx](\"assets/equations/a7ba541a2ae61050.svg\")
Contribution from output 2.
","![\"C_2=(0.5)\\times(-0.038098237)\\approx](\"assets/equations/899ef2a84df9b3cd.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1+C_2\\approx](\"assets/equations/167389750e490b23.svg\")
![\"\\delta_{h_1}](\"assets/equations/3d990b0c410497e3.svg\")
Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.45)\\times(0.138498562)\\approx](\"assets/equations/78866731f335909d.svg\")
Contribution from output 2.
","![\"C_2=(0.55)\\times(-0.038098237)\\approx](\"assets/equations/2b032a83e37967d3.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1+C_2\\approx](\"assets/equations/5c3c002092c7eca9.svg\")
![\"\\delta_{h_2}](\"assets/equations/bdbc44ac317050d2.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = 0.000219284. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=(0.008771355)\\times(0.05)\\approx](\"assets/equations/2525430dc0c2af1f.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,1,\\mathrm{new}}=0.15-0.5(0.000438568)\\approx](\"assets/equations/db7d742025f4d68e.svg\")
x2 → h1:
","Subtract η × gradient = 0.000438568. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,2}^{(1)}=(0.008771355)\\times(0.1)\\approx](\"assets/equations/753fcf33c45e8537.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,2,\\mathrm{new}}=0.2-0.5(0.000877135)\\approx](\"assets/equations/738f6089731cd94e.svg\")
x1 → h2:
","Subtract η × gradient = 0.000248856. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,1}^{(1)}=(0.009954255)\\times(0.05)\\approx](\"assets/equations/76611b6fffb6eae0.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,1,\\mathrm{new}}=0.25-0.5(0.000497713)\\approx](\"assets/equations/8c4ebc4a08766591.svg\")
x2 → h2:
","Subtract η × gradient = 0.000497712. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,2}^{(1)}=(0.009954255)\\times(0.1)\\approx](\"assets/equations/3534da4dd6514b21.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,2,\\mathrm{new}}=0.3-0.5(0.000995425)\\approx](\"assets/equations/770fd618f5ab4f7d.svg\")
Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","![\"b_{h_1,\\mathrm{new}}=0.35-0.5(0.008771355)=0.345614322\"](\"assets/equations/d0caa37a084caf21.svg\")
![\"b_{h_2,\\mathrm{new}}=0.35-0.5(0.009954255)=0.345022872\"](\"assets/equations/b81e2513786c90a9.svg\")
![\"b_{o_1,\\mathrm{new}}=0.6-0.5(0.138498562)=0.530750719\"](\"assets/equations/37ec194c9837cd70.svg\")
![\"b_{o_2,\\mathrm{new}}=0.6-0.5(-0.038098237)=0.619049118\"](\"assets/equations/2267211e36f3ce48.svg\")
These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
","Why the photos add two terms for h₁: dL/dh₁ = δo₁w₅ + δo₂w₇ = 0.138498562 × 0.4 + (−0.038098237) × 0.5 = 0.055399425−0.019049118 = 0.036350306. Now multiply by h₁(1−h₁) = 0.241300709 to obtain δh₁ = 0.008771355; multiply by x₁ = 0.05 to obtain dL/dw₁ = 0.000438568. Adding the two output paths is essential.
","Memory sequence: error → output slope → outgoing weight → add paths → hidden slope → input. If you can explain why each factor appears, changed numbers or extra output neurons are much less confusing.
","Bias convention: the photo draws one bias value per layer. If these are literally tied scalars rather than equal initial per-neuron biases, sum the relevant deltas: hidden shared b becomes 0.35−0.5(0.008771355+0.009954255) = 0.340637195; output shared b becomes 0.6−0.5(0.138498562−0.038098237) = 0.549799837. If only weights are requested, do not invent a bias update requirement.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c10-s6","search":"Teams photos full sigmoid example The three photographs show this 2 → 2 → 2 network and its chain rule. The photo label for w₃ is read as 0.25, consistent with the visible output/error values and the matching worked example in S05. Learning rate is not legible/specified in these photos; η = 0.5 below is an explicit practice assumption.\n**Question.** Input x = [0.05, 0.1]. Each row below lists the incoming weights of one receiving neuron. W₁ = ; b₁ = [0.35, 0.35]; W₂ = ; b₂ = [0.6, 0.6]. Hidden and output activations are sigmoid. Target y = [0.01, 0.99]. Learning rate η = 0.5. Compute all outputs, total half squared loss, all eight weight gradients and one simultaneous weight update. Map weights in order: w₁ = x₁ → h₁, w₂ = x₂ → h₁, w₃ = x₁ → h₂, w₄ = x₂ → h₂; w₅ = h₁ → o₁, w₆ = h₂ → o₁, w₇ = h₁ → o₂, w₈ = h₂ → o₂.\n\n\n\n### Given\n\nInput x = [0.05, 0.1].\n\nW₁ = .\n\nb₁ = [0.35, 0.35].\n\nW₂ = .\n\nb₂ = [0.6, 0.6].\n\nHidden and output activations are sigmoid.\n\nTarget y = [0.01, 0.99].\n\nLearning rate η = 0.5.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Output neuron 2: **\n\n\n**Step 2 loss convention.** The photos explicitly use "}]},{"n":11,"title":"F25 selected MT2 practice","sections":[{"title":"Start here","blocks":["The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper.
"],"kind":"read","id":"mt2-c11-intro"},{"title":"F25 Q1c","question":["Question (5 marks). Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization.
"],"answer":["Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","![\"J_{L1}=L+\\lambda\\sum_j|w_j|,\\qquad](\"assets/equations/3eb3323790998696.svg\")
L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
"],"kind":"practice","id":"mt2-c11-s0","search":"F25 Q1c **Question (5 marks).** Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization.\n\n**Complete worked answer.**\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\n"},{"title":"F25 Q1d","question":["Question (5 marks). Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","![\"h_t=\\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\\quad](\"assets/equations/72d09ad3983b6b52.svg\")
At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
"],"kind":"practice","id":"mt2-c11-s1","search":"F25 Q1d **Question (5 marks).** Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate.\n\n**Complete worked answer.**\nA recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.\n\n\nAt each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.\nFor next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.\nA basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.\n"},{"title":"F25 Q2b","question":["Question (10 marks). Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction.
"],"answer":["A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.
","![\"\\frac{\\partial](\"assets/equations/69be13fbc705ad29.svg\")
The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.
","Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].
","An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
","AlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs.
"],"kind":"practice","id":"mt2-c11-s2","search":"F25 Q2b **Question (10 marks).** Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction.\n\n**Complete worked answer.**\nA residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.\n\n\nThe identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.\nElementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.\nAlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs.\n"},{"title":"F25 Q3b","question":["Question (10 marks). Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning.
"],"answer":["Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.
","1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.
","2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.
","3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.
","4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.
","5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.
","6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.
","Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.
"],"kind":"practice","id":"mt2-c11-s3","search":"F25 Q3b **Question (10 marks).** Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning.\n\n**Complete worked answer.**\nTransfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.\n1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.\n2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.\n3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.\n4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.\n5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.\n6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.\nFeature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.\n"},{"title":"F25 Q4a","question":["Question (10 marks). For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters.
"],"answer":["N = 1,000; mini-batch size = 100.
","J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9.
","Use signed velocity.
","Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations.
","Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.
","N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.
","Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","Step 1: count full-batch updates.
","All 1,000 samples contribute to one update: 1 per epoch.
","Step 2: count stochastic updates.
","One update per sample: 1,000 per epoch.
","Step 3: count mini-batches.
","![\"1000/100=10\"](\"assets/equations/eee877d70ea14115.svg\")
Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ.
","Step 1: evaluate the gradient at the current parameter.
","![\"g_1=1\"](\"assets/equations/228fc3b3267b30c9.svg\")
Step 2: retain the previous displacement.
","Step 3: scale the current gradient.
","![\"0.1(1)=0.1\"](\"assets/equations/7cde65d80bbbce53.svg\")
Step 4: combine to obtain the new signed velocity.
","![\"v_1=0-0.1=-0.1\"](\"assets/equations/2c6e04e58352c2ce.svg\")
Step 5: add new velocity to the actual old parameter.
","Step 1: evaluate the gradient at the current parameter.
","![\"g_2=0.9\"](\"assets/equations/bc4532aff9a9a2da.svg\")
Step 2: retain the previous displacement.
","![\"0.9(-0.1)=-0.09\"](\"assets/equations/a8fd30bce103b617.svg\")
Step 3: scale the current gradient.
","![\"0.1(0.9)=0.09\"](\"assets/equations/bf93339d0945c688.svg\")
Step 4: combine to obtain the new signed velocity.
","![\"v_2=-0.09-0.09=-0.18\"](\"assets/equations/f18fc7f5dc715d05.svg\")
Step 5: add new velocity to the actual old parameter.
","Hyperparameters: the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update.
"],"kind":"practice","id":"mt2-c11-s4","search":"F25 Q4a **Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nN = 1,000; mini-batch size = 100.\n\nJ(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9.\n\nUse signed velocity.\n\n### To find\n\nCount updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations.\n\n### Formula — which one and why\n\n**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.\n\nN = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n\n**Step 1: count full-batch updates.**\n\nAll 1,000 samples contribute to one update: 1 per epoch.\n\n**Step 2: count stochastic updates.**\n\nOne update per sample: 1,000 per epoch.\n\n**Step 3: count mini-batches.**\n\n\n\n\n\n\n\n\nUse signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ.\n### Momentum iteration 1\n\n**Step 1: evaluate the gradient at the current parameter.**\n\n\n\n\n\n\n\n**Step 2: retain the previous displacement.**\n\n\n\n\n\n\n\n**Step 3: scale the current gradient.**\n\n\n\n\n\n\n\n**Step 4: combine to obtain the new signed velocity.**\n\n\n\n\n\n\n\n**Step 5: add new velocity to the actual old parameter.**\n\n\n\n\n\n\n\n\n### Momentum iteration 2\n\n**Step 1: evaluate the gradient at the current parameter.**\n\n\n\n\n\n\n\n**Step 2: retain the previous displacement.**\n\n\n\n\n\n\n\n**Step 3: scale the current gradient.**\n\n\n\n\n\n\n\n**Step 4: combine to obtain the new signed velocity.**\n\n\n\n\n\n\n\n**Step 5: add new velocity to the actual old parameter.**\n\n\n\n\n\n\n\n\n**Hyperparameters: ** the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update.\n"},{"title":"F25 Q5b","question":["Question (10 marks). Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD.
"],"answer":["AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:
","![\"G_t=G_{t-1}+g_t^2,\\quad\\theta_t=\\theta_{t-1}-\\frac{\\eta](\"assets/equations/3060020ef7a35ed4.svg\")
Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.
","Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.
","Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.
"],"kind":"practice","id":"mt2-c11-s5","search":"F25 Q5b **Question (10 marks).** Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD.\n\n**Complete worked answer.**\nAdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:\n\nFrequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.\nExample with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.\nAdvantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.\n"},{"title":"F25 Q6a","question":["Question (10 marks). Calculate every activation shape and trainable-parameter count for: Input 128 × 128 × 3. → Conv K = 3, filters = 16, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 3, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 240 units. → Dense 120 units. → Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2.
","| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Stride 2; P = 0 (no padding).
Input 128 × 128 × 3.
","Conv K = 3, filters = 16, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 3, filters = 32, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Dense 240 units.
","Dense 120 units.
","Dense 5 units.
","Also convolve the input A with the kernel K shown below, valid padding and stride 2.
","Find every layer shape, its parameter count and the network total. Find every convolution output cell.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Use output size ![\"\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\"](\"assets/equations/c0351298d1be04e6.svg\")
, Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 128 × 128 × 3. Use 16 filters of spatial size 3 × 3, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{128+2(0)-3}{1}\\right\\rfloor+1=126\"](\"assets/equations/60bad070a07deb57.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 126 × 126 × 16.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=3\\times3\\times3\\times16=432\"](\"assets/equations/cd36df36d5f6f027.svg\")
![\"N_{\\mathrm{biases}}=16\"](\"assets/equations/dc0c940eb69b41c0.svg\")
![\"P_1=432+16=448\"](\"assets/equations/fe2e71f3a8efffa4.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 126 × 126 × 16. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{126-2}{2}\\right\\rfloor+1=63\"](\"assets/equations/31d9e5cfd413812f.svg\")
Pooling acts independently on each channel. The output is 63 × 63 × 16.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 63 × 63 × 16. Use 32 filters of spatial size 3 × 3, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{63+2(0)-3}{1}\\right\\rfloor+1=61\"](\"assets/equations/d1b1c42ec3b8b44b.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 61 × 61 × 32.
","3. Count trainable values. Each filter receives all 16 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=3\\times3\\times16\\times32=4{,}608\"](\"assets/equations/fcf6aecafd4a3651.svg\")
![\"N_{\\mathrm{biases}}=32\"](\"assets/equations/1cbdbfdd12c82e10.svg\")
![\"P_3=4{,}608+32=4{,}640\"](\"assets/equations/ebc484850d4827e3.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 61 × 61 × 32. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{61-2}{2}\\right\\rfloor+1=30\"](\"assets/equations/2fa03284017c373e.svg\")
Pooling acts independently on each channel. The output is 30 × 30 × 32.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The division gives 29.5; take its floor before adding 1. A partial window outside the unpadded input is not included.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 30 × 30 × 32. Rearrange all activation values into one vector.
","![\"n_{\\mathrm{flat}}=30\\times30\\times32=28{,}800\"](\"assets/equations/f8d6a23f5595005e.svg\")
Output: a vector with 28,800 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 28,800 features. Output: 240 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=28{,}800\\times240=6{,}912{,}000\"](\"assets/equations/2e815f655162e42b.svg\")
![\"N_{\\mathrm{biases}}=240\"](\"assets/equations/237f8d226a0fb66a.svg\")
![\"P_6=6{,}912{,}000+240=6{,}912{,}240\"](\"assets/equations/a36eb80ee2f332fd.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 240 features. Output: 120 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=240\\times120=28{,}800\"](\"assets/equations/55419bb786cbe655.svg\")
![\"N_{\\mathrm{biases}}=120\"](\"assets/equations/7dde36966a653a38.svg\")
![\"P_7=28{,}800+120=28{,}920\"](\"assets/equations/780340973d3c0ce4.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 120 features. Output: 5 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=120\\times5=600\"](\"assets/equations/26857a3f5e70fd9d.svg\")
![\"N_{\\mathrm{biases}}=5\"](\"assets/equations/8917be67939f2d89.svg\")
![\"P_8=600+5=605\"](\"assets/equations/b4289ff8bd02992b.svg\")
| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | ![\"\\left\\lfloor\\frac{128+2](\"assets/equations/dbb0832b53a96b49.svg\") | 126 x 126 x 16 | 432 | 16 | 448 |
| 2 pool | ![\"\\left\\lfloor\\frac{126-2}{2}\\right\\rfloor+1\"](\"assets/equations/a6979fd0d1f6b483.svg\") | 63 x 63 x 16 | 0 | 0 | 0 |
| 3 conv | ![\"\\left\\lfloor\\frac{63+2](\"assets/equations/f9e5ee2c7c7e1bbb.svg\") | 61 x 61 x 32 | 4,608 | 32 | 4,640 |
| 4 pool | ![\"\\left\\lfloor\\frac{61-2}{2}\\right\\rfloor+1\"](\"assets/equations/1ac02853de1445dd.svg\") | 30 x 30 x 32 | 0 | 0 | 0 |
| 5 flat | 30 × 30 × 32 | 28800 | 0 | 0 | 0 |
| 6 dense | 28800 → 240 | 240 | 6,912,000 | 240 | 6,912,240 |
| 7 dense | 240 → 120 | 120 | 28,800 | 120 | 28,920 |
| 8 dense | 120 → 5 | 5 | 600 | 5 | 605 |
Parameter substitutions, in layer order:
","Layer 1: (3 x 3 x 3 + 1) x 16 = 448.
","Layer 3: (3 x 3 x 16 + 1) x 32 = 4,640.
","Layer 6: (28800 + 1) x 240 = 6,912,240.
","Layer 7: (240 + 1) x 120 = 28,920.
","Layer 8: (120 + 1) x 5 = 605.
","Total: 448 + 4,640 + 6,912,240 + 28,920 + 605 = 6,946,853 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","Separate convolution calculation:
","Step 1 shape: ![\"\\left\\lfloor\\frac{5+2](\"assets/equations/966c8fb493880239.svg\")
= 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 1–3. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 3 | 0 | −2 |
| 4 | 0 | −4 |
| 3 | 0 | −1 |
Each cell is input × matching kernel coefficient.
1 + 0 + 2 = 3
| 3 | 5 |
| 3 | −7 |
Output row 1, column 1.
Window row 1.
","![\"R_1=(3)\\times(1)+(1)\\times(0)+(2)\\times(-1)=1\"](\"assets/equations/a7b5121489bd31f6.svg\")
Window row 2.
","![\"R_2=(2)\\times(2)+(1)\\times(0)+(2)\\times(-2)=0\"](\"assets/equations/2be91abc57b10ecb.svg\")
Window row 3.
","![\"R_3=(3)\\times(1)+(2)\\times(0)+(1)\\times(-1)=2\"](\"assets/equations/e404b4617ef0e7d0.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,1}=(1)+(0)+(2)=3\"](\"assets/equations/b108d091066c8f7c.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 3–5. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 2 | 0 | −1 |
| 4 | 0 | 0 |
| 1 | 0 | −1 |
Each cell is input × matching kernel coefficient.
1 + 4 + 0 = 5
| 3 | 5 |
| 3 | −7 |
Output row 1, column 2.
Window row 1.
","![\"R_1=(2)\\times(1)+(2)\\times(0)+(1)\\times(-1)=1\"](\"assets/equations/aa4766cd874d5e7f.svg\")
Window row 2.
","![\"R_2=(2)\\times(2)+(1)\\times(0)+(0)\\times(-2)=4\"](\"assets/equations/93a6f5b5b20f2f23.svg\")
Window row 3.
","![\"R_3=(1)\\times(1)+(0)\\times(0)+(1)\\times(-1)=0\"](\"assets/equations/5f03457a49dcba68.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,2}=(1)+(4)+(0)=5\"](\"assets/equations/dd6e5a7547d2ee64.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 1–3. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 3 | 0 | −1 |
| 0 | 0 | 0 |
| 1 | 0 | 0 |
Each cell is input × matching kernel coefficient.
2 + 0 + 1 = 3
| 3 | 5 |
| 3 | −7 |
Output row 2, column 1.
Window row 1.
","![\"R_1=(3)\\times(1)+(2)\\times(0)+(1)\\times(-1)=2\"](\"assets/equations/d3a3d0394f9567ec.svg\")
Window row 2.
","![\"R_2=(0)\\times(2)+(3)\\times(0)+(0)\\times(-2)=0\"](\"assets/equations/4ab2e486b72e8236.svg\")
Window row 3.
","![\"R_3=(1)\\times(1)+(1)\\times(0)+(0)\\times(-1)=1\"](\"assets/equations/c2dca2b82f9aac21.svg\")
Add the rows to obtain one output cell.
","![\"Y_{2,1}=(2)+(0)+(1)=3\"](\"assets/equations/f077b3678b0e1f35.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 3–5. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 | −1 |
| 0 | 0 | −4 |
| 0 | 0 | −3 |
Each cell is input × matching kernel coefficient.
0 + (−4) + (−3) = −7
| 3 | 5 |
| 3 | −7 |
Output row 2, column 2.
Window row 1.
","![\"R_1=(1)\\times(1)+(0)\\times(0)+(1)\\times(-1)=0\"](\"assets/equations/8247d99177a36658.svg\")
Window row 2.
","![\"R_2=(0)\\times(2)+(1)\\times(0)+(2)\\times(-2)=-4\"](\"assets/equations/959c04625a9a0df4.svg\")
Window row 3.
","![\"R_3=(0)\\times(1)+(2)\\times(0)+(3)\\times(-1)=-3\"](\"assets/equations/20986846d13e4484.svg\")
Add the rows to obtain one output cell.
","![\"Y_{2,2}=(0)+(-4)+(-3)=-7\"](\"assets/equations/aec0342788a9224e.svg\")
Final output: ![\"\\begin{bmatrix}3](\"assets/equations/3797d8870cefe33e.svg\")
. Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
Question (10 marks). Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages.
"],"answer":["A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification.
","Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities.
","Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation.
","For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions.
"],"kind":"practice","id":"mt2-c11-s7","search":"F25 Q7b **Question (10 marks).** Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages.\n\n**Complete worked answer.**\nA convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification.\n\nConvolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities.\nCompared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation.\nFor medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions.\n\n\n"}]},{"n":12,"title":"R26 selected MT2 practice","sections":[{"title":"Start here","blocks":["The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper.
"],"kind":"read","id":"mt2-c12-intro"},{"title":"R26 Q1c","question":["Question (5 marks). Explain L1 and L2 regularization. Apply 50% dropout to outputs [0.8,−0.4,0.6,0.2] using an explicitly assumed mask.
"],"answer":["Activations = [0.8,−0.4,0.6,0.2]; dropout rate = 50%.
","The worked example assumes mask [1,0,1,0] and uses inverted dropout.
","No numerical weights or λ are supplied for the regularization explanation.
","Explain L1/L2 and calculate post-dropout activations under the stated mask.
","L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.
","λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.
","Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.
","L2 penalty with a factor of one-half. This answer uses λ/2 times the sum of squared weights. Differentiating a square supplies a factor of 2, which cancels the one-half, so the added gradient is λw. Use this same convention in the explanation below.
","λ = penalty strength; J_data = original data loss; wᵢ = weight i.
","![\"J=J_{\\mathrm{data}}+\\frac{\\lambda}{2}\\sum_iw_i^2\"](\"assets/equations/4e906f5f903db890.svg\")
![\"\\frac{\\partial](\"assets/equations/919b9b7755f3c860.svg\")
Yaad rakhne ka cue: L2 mein half diya hai: square ka 2 cancel, gradient mein lambda × weight.
","Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.
","aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.
","Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.
","Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
","Step 1: convert dropout probability to keep probability.
","![\"q=1-0.5=0.5\"](\"assets/equations/39070c1933e29986.svg\")
Step 2: state a possible mask.
","Choose [1,0,1,0]; 1 means keep and 0 means drop. The question allows an assumed mask; others give different realizations.
","Step 3: multiply each activation by its mask entry.
","![\"[0.8(1),(-0.4)(0),0.6(1),0.2(0)]=[0.8,0,0.6,0]\"](\"assets/equations/4ccd5e8324645240.svg\")
Step 4: divide the kept values by q for inverted dropout.
","![\"0.8/0.5=1.6\"](\"assets/equations/552d192c8691910e.svg\")
![\"0.6/0.5=1.2\"](\"assets/equations/2e2dc3409b6b4b5c.svg\")
Step 5: assemble the training output.
","![\"a_{\\mathrm{train}}=[1.6,0,1.2,0]\"](\"assets/equations/e92cb4c6084d0ae8.svg\")
Final answer: simple masking gives [0.8,0,0.6,0]; inverted training dropout gives [1.6,0,1.2,0]. At inference inverted dropout is disabled.
"],"kind":"practice","id":"mt2-c12-s0","search":"R26 Q1c **Question (5 marks).** Explain L1 and L2 regularization. Apply 50% dropout to outputs [0.8,−0.4,0.6,0.2] using an explicitly assumed mask.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nActivations = [0.8,−0.4,0.6,0.2]; dropout rate = 50%.\n\nThe worked example assumes mask [1,0,1,0] and uses inverted dropout.\n\nNo numerical weights or λ are supplied for the regularization explanation.\n\n### To find\n\nExplain L1/L2 and calculate post-dropout activations under the stated mask.\n\n### Formula — which one and why\n\n**L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.\n\nλ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.\n\n\n\n\n\n**Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai.\n\n**L2 penalty with a factor of one-half.** This answer uses λ/2 times the sum of squared weights. Differentiating a square supplies a factor of 2, which cancels the one-half, so the added gradient is λw. Use this same convention in the explanation below.\n\nλ = penalty strength; J_data = original data loss; wᵢ = weight i.\n\n\n\n\n\n**Yaad rakhne ka cue:** L2 mein half diya hai: square ka 2 cancel, gradient mein lambda × weight.\n\n**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.\n\naᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.\n\n\n\n\n\n**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.\n\n### Solution — step by step\n\n\n\n\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\n### Numerical with one explicitly chosen mask\n\n**Step 1: convert dropout probability to keep probability.**\n\n\n\n\n\n\n\n**Step 2: state a possible mask.**\n\nChoose [1,0,1,0]; 1 means keep and 0 means drop. The question allows an assumed mask; others give different realizations.\n\n**Step 3: multiply each activation by its mask entry.**\n\n\n\n\n\n\n\n**Step 4: divide the kept values by q for inverted dropout.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 5: assemble the training output.**\n\n\n\n\n\n\n\n**Final answer:** simpl"},{"title":"R26 Q1d","question":["Question (5 marks). Explain an RNN for positive, negative or neutral e-commerce review sentiment. Why are recurrence and word order relevant?
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","For this task select a many-to-one output: tokenize and embed the full review, update the recurrent state across its words, and classify the final sequence representation with a three-unit softmax. Train using one-hot sentiment labels and categorical cross-entropy. “Not worth buying” and “worth buying” illustrate why negation and order matter.
"],"kind":"practice","id":"mt2-c12-s1","search":"R26 Q1d **Question (5 marks).** Explain an RNN for positive, negative or neutral e-commerce review sentiment. Why are recurrence and word order relevant?\n\n**Complete worked answer.**\nA recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.\n\n\nAt each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.\nFor next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.\nA basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.\nFor this task select a many-to-one output: tokenize and embed the full review, update the recurrent state across its words, and classify the final sequence representation with a three-unit softmax. Train using one-hot sentiment labels and categorical cross-entropy. “Not worth buying” and “worth buying” illustrate why negation and order matter.\n"},{"title":"R26 Q2b","question":["Question (10 marks). Explain AlexNet regularization and training improvements; explain Inception multiscale features and 1 × 1 convolutions; recommend an architecture for large-scale image classification with justification.
"],"answer":["AlexNet uses a sequence of convolutional layers followed by fully connected classification layers. Its influential design used ReLU activations, max pooling, dropout in the fully connected part, data augmentation and GPU training. ReLU's positive derivative supports learning compared with heavily saturated sigmoid hidden units. Augmentation supplies varied training views; dropout reduces co-adaptation; GPU computation makes a large network practical to train.
","The common 227-pixel classroom layout is: Conv 11 stride 4 with 96 filters; pool 3 stride 2; Conv 5 padding 2 with 256 filters; pool 3 stride 2; three Conv 3 padding 1 layers with 384,384,256 filters; pool 3 stride 2; dense 4096; dense 4096; output 1000. Nonlinearities are applied between learned layers. Exact padding, grouping and input conventions must follow the question: the original model used channel groups in some layers, so parameter counts differ from a fully connected-channel simplification.
","AlexNet follows a mainly sequential feature-extraction path. An Inception module processes several spatial scales in parallel and concatenates them, often using 1 × 1 bottlenecks to reduce cost. Neither architecture is universally best; the task, resource budget and validation performance determine the choice.
","An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
","Recommendation: an Inception-family model is a defensible choice when varied object scales and computation efficiency matter, because parallel scales with bottlenecks provide diverse features at controlled cost. Compare held-out accuracy, latency and memory with available alternatives. This is a justified choice, not a claim that Inception always beats AlexNet on every dataset.
"],"kind":"practice","id":"mt2-c12-s2","search":"R26 Q2b **Question (10 marks).** Explain AlexNet regularization and training improvements; explain Inception multiscale features and 1 × 1 convolutions; recommend an architecture for large-scale image classification with justification.\n\n**Complete worked answer.**\nAlexNet uses a sequence of convolutional layers followed by fully connected classification layers. Its influential design used ReLU activations, max pooling, dropout in the fully connected part, data augmentation and GPU training. ReLU's positive derivative supports learning compared with heavily saturated sigmoid hidden units. Augmentation supplies varied training views; dropout reduces co-adaptation; GPU computation makes a large network practical to train.\n\nThe common 227-pixel classroom layout is: Conv 11 stride 4 with 96 filters; pool 3 stride 2; Conv 5 padding 2 with 256 filters; pool 3 stride 2; three Conv 3 padding 1 layers with 384,384,256 filters; pool 3 stride 2; dense 4096; dense 4096; output 1000. Nonlinearities are applied between learned layers. Exact padding, grouping and input conventions must follow the question: the original model used channel groups in some layers, so parameter counts differ from a fully connected-channel simplification.\nAlexNet follows a mainly sequential feature-extraction path. An Inception module processes several spatial scales in parallel and concatenates them, often using 1 × 1 bottlenecks to reduce cost. Neither architecture is universally best; the task, resource budget and validation performance determine the choice.\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.\n**Recommendation: ** an Inception-family model is a defensible choice when varied object scales and computation efficiency matter, because parallel scales with bottlenecks provide diverse features at controlled cost. Compare held-out accuracy, latency and memory with available alternatives. This is a justified choice, not a claim that Inception always beats AlexNet on every dataset.\n"},{"title":"R26 Q3b","question":["Question (10 marks). How should CNN filter counts be selected? Explain effects on feature extraction, dimensionality, computation/memory, overfitting and hierarchical learning.
"],"answer":["Choose filter counts as architecture hyperparameters using task complexity, available data, validation performance and resource limits. There is no universal correct count.
","1. More filters can represent a wider variety of learned features at a layer; redundant filters do not automatically improve accuracy.
","2. Cout filters produce Cout output channels; spatial height/width still follow kernel, padding and stride.
","3. A K × K layer uses K²CinCout weights plus Cout biases. More output channels increase current activations and the input-channel cost of the next convolution.
","4. Extra capacity can overfit limited data, so use validation, augmentation, transfer learning or regularization. Too few channels may underfit.
","5. Later layers often use more channels after spatial downsampling, allowing complex feature combinations at a smaller spatial size. This is a common design choice, not a compulsory rule.
","Example: a 3 × 3 layer with 16 input channels has (9 × 16+1) × 32 = 4,640 parameters for 32 filters, versus 9,280 for 64 filters. Output-channel count doubles while the same spatial-size formula applies.
"],"kind":"practice","id":"mt2-c12-s3","search":"R26 Q3b **Question (10 marks).** How should CNN filter counts be selected? Explain effects on feature extraction, dimensionality, computation/memory, overfitting and hierarchical learning.\n\n**Complete worked answer.**\nChoose filter counts as architecture hyperparameters using task complexity, available data, validation performance and resource limits. There is no universal correct count.\n1. More filters can represent a wider variety of learned features at a layer; redundant filters do not automatically improve accuracy.\n2. Cout filters produce Cout output channels; spatial height/width still follow kernel, padding and stride.\n3. A K × K layer uses K²CinCout weights plus Cout biases. More output channels increase current activations and the input-channel cost of the next convolution.\n4. Extra capacity can overfit limited data, so use validation, augmentation, transfer learning or regularization. Too few channels may underfit.\n5. Later layers often use more channels after spatial downsampling, allowing complex feature combinations at a smaller spatial size. This is a common design choice, not a compulsory rule.\nExample: a 3 × 3 layer with 16 input channels has (9 × 16+1) × 32 = 4,640 parameters for 32 filters, versus 9,280 for 64 filters. Output-channel count doubles while the same spatial-size formula applies.\n"},{"title":"R26 Q4a","question":["Question (10 marks). Compare batch GD and SGD for loss oscillations with a graph, memory and large datasets. Discuss learning rate extremes. For J(θ) = 0.7 θ³, θ₀ = 1, η = 0.1, perform two iterations.
"],"answer":["J(θ) = 0.7θ³; θ₀ = 1; η = 0.1.
","Calculate the derivative and perform two updates.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.
","Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.
","Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.
","Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.
","Step 1: differentiate the objective before substituting.
","![\"J(\\theta)=0.7\\theta^3\"](\"assets/equations/eaad5a7869c46a71.svg\")
![\"g(\\theta)=0.7(3)\\theta^2=2.1\\theta^2\"](\"assets/equations/2aadd5dc4445f1ba.svg\")
Step 2: evaluate gradient at the original parameter.
","![\"g_1=2.1(1)^2=2.1\"](\"assets/equations/6e21de54ead2d325.svg\")
Step 3: scale the first gradient.
","![\"0.1(2.1)=0.21\"](\"assets/equations/3ed37be87afc8239.svg\")
Step 4: subtract to obtain the first update.
","![\"\\theta_1=1-0.21=0.79\"](\"assets/equations/e9423903ada8f563.svg\")
Step 5: square the changed parameter.
","![\"0.79^2=0.6241\"](\"assets/equations/c5b6ff4c2963f39a.svg\")
Step 6: recalculate the gradient.
","![\"g_2=2.1(0.6241)=1.31061\"](\"assets/equations/c986b4d77e3de9d0.svg\")
Step 7: scale the second gradient.
","![\"0.1(1.31061)=0.131061\"](\"assets/equations/7f5b4b6bf69894db.svg\")
Step 8: subtract from the first updated parameter.
","![\"\\theta_2=0.79-0.131061=0.658939\"](\"assets/equations/11da355fe098969f.svg\")
Final answer: θ₁ = 0.79; θ₂ = 0.658939. No individual samples are supplied, so these are the stated scalar gradient updates. The cubic is unbounded below; two steps are not a convergence proof.
"],"kind":"practice","id":"mt2-c12-s4","search":"R26 Q4a **Question (10 marks).** Compare batch GD and SGD for loss oscillations with a graph, memory and large datasets. Discuss learning rate extremes. For J(θ) = 0.7 θ³, θ₀ = 1, η = 0.1, perform two iterations.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nJ(θ) = 0.7θ³; θ₀ = 1; η = 0.1.\n\n### To find\n\nCalculate the derivative and perform two updates.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\nThe learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.\n\nBatch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.\nMomentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.\nDiagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.\n**Step 1: differentiate the objective before substituting.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 2: evaluate gradient at the original parameter.**\n\n\n\n\n\n\n\n**Step 3: scale the first gradient.**\n\n\n\n\n\n\n\n**Step 4: subtract to obtain the first update.**\n\n\n\n\n\n\n\n**Step 5: square the changed parameter.**\n\n\n\n\n\n\n\n**Step 6: recalculate the gradient.**\n\n\n\n\n\n\n\n**Step 7: scale the second gradient.**\n\n\n\n\n\n\n\n**Step 8: subtract from the first updated parameter.**\n\n\n\n\n\n\n\n**Final answer:** θ₁ = 0.79; θ₂ = 0.658939. No individual samples are supplied, so these are the stated scalar gradient updates. The cubic is unbounded below; two steps are not a convergence proof.\n"},{"title":"R26 Q5a","question":["Question (10 marks). Explain feedforward limitations for sequential context and how recurrence helps. Calculate h using tanh for Wxh is shown below, Whh is shown below, x = [0,1], hprevious = [0,1], zero bias. Explain next-word probabilities.
","Wxh and Whh are printed in the question; use their receiving-neuron rows.
","Current input = [0,1]; previous hidden state = [0,1]; bias = 0.
","Activation = tanh.
","Calculate both entries of the new hidden state. Output-layer weights are not supplied.
","Recurrent forward pass. At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.
","x_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.
","Yaad rakhne ka cue: Naya input + pichhla activated state; phir activation.
","A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","Step 1: multiply the input vector into row 1.
","![\"0.6(0)+0.4(1)=0+0.4=0.4\"](\"assets/equations/10e199f12bbe9b83.svg\")
Step 2: multiply the input vector into row 2.
","![\"0.3(0)+0.7(1)=0+0.7=0.7\"](\"assets/equations/360d1cebae87086f.svg\")
Step 3: multiply previous state into recurrent row 1.
","![\"0.5(0)+0.3(1)=0+0.3=0.3\"](\"assets/equations/7c5775eea93e5ad0.svg\")
Step 4: multiply previous state into recurrent row 2.
","![\"0.2(0)+0.4(1)=0+0.4=0.4\"](\"assets/equations/818979c823c2927e.svg\")
Step 5: combine contributions for hidden unit 1.
","![\"a_1=0.4+0.3+0=0.7\"](\"assets/equations/01f98e63892e2cf5.svg\")
Step 6: combine contributions for hidden unit 2.
","![\"a_2=0.7+0.4+0=1.1\"](\"assets/equations/d5abcbc4ff3b8b0f.svg\")
Step 7: apply tanh separately to each complete sum.
","![\"h_1=\\tanh(0.7)\\approx0.604367777\"](\"assets/equations/bc5f2bb1f8425e2b.svg\")
![\"h_2=\\tanh(1.1)\\approx0.800499022\"](\"assets/equations/ceaf167b0d164375.svg\")
Final answer: h ≈ [0.604367777,0.800499022]. The absent vocabulary output weights prevent calculating a numerical next-word probability list.
"],"kind":"practice","id":"mt2-c12-s5","search":"R26 Q5a **Question (10 marks).** Explain feedforward limitations for sequential context and how recurrence helps. Calculate h using tanh for Wxh is shown below, Whh is shown below, x = [0,1], hprevious = [0,1], zero bias. Explain next-word probabilities.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nWxh and Whh are printed in the question; use their receiving-neuron rows.\n\nCurrent input = [0,1]; previous hidden state = [0,1]; bias = 0.\n\nActivation = tanh.\n\n### To find\n\nCalculate both entries of the new hidden state. Output-layer weights are not supplied.\n\n### Formula — which one and why\n\n**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.\n\nx_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.\n\n### Solution — step by step\n\n\n\n\nA recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.\n\n\nAt each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.\nFor next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.\nA basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.\n**Step 1: multiply the input vector into row 1.**\n\n\n\n\n\n\n\n**Step 2: multiply the input vector into row 2.**\n\n\n\n\n\n\n\n**Step 3: multiply previous state into recurrent row 1.**\n\n\n\n\n\n\n\n**Step 4: multiply previous state into recurrent row 2.**\n\n\n\n\n\n\n\n**Step 5: combine contributions for hidden unit 1.**\n\n\n\n\n\n\n\n**Step 6: combine contributions for hidden unit 2.**\n\n\n\n\n\n\n\n**Step 7: apply tanh separately to each complete sum.**\n\n\n\n\n\n\n\n\n\n\n\n**Final answer:** h ≈ [0.604367777,0.800499022]. The absent vocabulary output weights prevent calculating a numerical next-word probability list.\n"},{"title":"R26 Q5b","question":["Question (10 marks). Explain RMSProp, its update rule and accumulated squared gradients. Give advantages and disadvantages relative to SGD.
"],"answer":["RMSProp keeps an exponentially weighted average of squared gradients, instead of AdaGrad's ever-growing sum. For each parameter, initialize s₀ = 0 and use:
","![\"s_t=\\rho](\"assets/equations/459c86785f7d6c4f.svg\")
ρ controls how strongly earlier squared gradients are retained. A large average magnitude produces a larger denominator and a smaller step. A small average magnitude produces a relatively larger step. Squaring happens before averaging; averaging signed gradients instead would allow cancellation and would not be RMSProp.
","Example: θ₀ = 1, η = 0.1, ρ = 0.9, s₀ = 0, g₁ = 2. Then s₁ = 0.9 × 0+0.1 × 4 = 0.4. θ₁ = 1−0.2/√0.4 = 0.683772234 when ε is negligible. If g₂ = 1, s₂ = 0.9 × 0.4+0.1 × 1 = 0.46 and θ₂ = 0.683772234−0.1/√0.46 = 0.536330278.
","Benefits include adapting to different coordinate scales and avoiding AdaGrad's permanent accumulation of all past squared gradients. Costs include extra state memory and tuning η, ρ and ε; noisy or badly chosen settings can still produce poor training. Unlike standard SGD it rescales each coordinate; unlike Adam it does not include a separate first-moment estimate in this basic form.
"],"kind":"practice","id":"mt2-c12-s6","search":"R26 Q5b **Question (10 marks).** Explain RMSProp, its update rule and accumulated squared gradients. Give advantages and disadvantages relative to SGD.\n\n**Complete worked answer.**\nRMSProp keeps an exponentially weighted average of squared gradients, instead of AdaGrad's ever-growing sum. For each parameter, initialize s₀ = 0 and use:\n\nρ controls how strongly earlier squared gradients are retained. A large average magnitude produces a larger denominator and a smaller step. A small average magnitude produces a relatively larger step. Squaring happens before averaging; averaging signed gradients instead would allow cancellation and would not be RMSProp.\nExample: θ₀ = 1, η = 0.1, ρ = 0.9, s₀ = 0, g₁ = 2. Then s₁ = 0.9 × 0+0.1 × 4 = 0.4. θ₁ = 1−0.2/√0.4 = 0.683772234 when ε is negligible. If g₂ = 1, s₂ = 0.9 × 0.4+0.1 × 1 = 0.46 and θ₂ = 0.683772234−0.1/√0.46 = 0.536330278.\nBenefits include adapting to different coordinate scales and avoiding AdaGrad's permanent accumulation of all past squared gradients. Costs include extra state memory and tuning η, ρ and ε; noisy or badly chosen settings can still produce poor training. Unlike standard SGD it rescales each coordinate; unlike Adam it does not include a separate first-moment estimate in this basic form.\n"},{"title":"R26 Q6a","question":["Question (10 marks). For the binary CNN below, calculate output dimensions and trainable parameters after every layer. Input 256 × 256 × 3. → Conv K = 5, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 128, stride = 1, padding = 1. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 84 units. → Dense 32 units. → Dense 1 units. The printed output size is unspecified; assume one sigmoid output. Unspecified convolution padding is taken as valid.
"],"answer":["Input 256 × 256 × 3.
","Conv K = 5, filters = 32, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 5, filters = 64, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 5, filters = 128, stride = 1, padding = 1.
","Pool K = 2, stride = 2, padding = 0.
","Dense 84 units.
","Dense 32 units.
","Dense 1 units.
","assume one sigmoid output.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 256 × 256 × 3. Use 32 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{256+2(0)-5}{1}\\right\\rfloor+1=252\"](\"assets/equations/f4b85116af3d067a.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 252 × 252 × 32.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=5\\times5\\times3\\times32=2{,}400\"](\"assets/equations/963130ef1e615735.svg\")
![\"P_1=2{,}400+32=2{,}432\"](\"assets/equations/113b757e4d085763.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 252 × 252 × 32. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{252-2}{2}\\right\\rfloor+1=126\"](\"assets/equations/3083aa89c3311d57.svg\")
Pooling acts independently on each channel. The output is 126 × 126 × 32.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 126 × 126 × 32. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{126+2(0)-5}{1}\\right\\rfloor+1=122\"](\"assets/equations/991cd2a9cf06a3a0.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 122 × 122 × 64.
","3. Count trainable values. Each filter receives all 32 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=5\\times5\\times32\\times64=51{,}200\"](\"assets/equations/a22447ce668f8dbf.svg\")
![\"N_{\\mathrm{biases}}=64\"](\"assets/equations/dc3a74202b89cbd8.svg\")
![\"P_3=51{,}200+64=51{,}264\"](\"assets/equations/101c0d0d89af6010.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 122 × 122 × 64. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{122-2}{2}\\right\\rfloor+1=61\"](\"assets/equations/e5f058816466e626.svg\")
Pooling acts independently on each channel. The output is 61 × 61 × 64.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 61 × 61 × 64. Use 128 filters of spatial size 5 × 5, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{61+2(1)-5}{1}\\right\\rfloor+1=59\"](\"assets/equations/8321e7609e3b2886.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 59 × 59 × 128.
","3. Count trainable values. Each filter receives all 64 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=5\\times5\\times64\\times128=204{,}800\"](\"assets/equations/c52ed6d2bdad242d.svg\")
![\"N_{\\mathrm{biases}}=128\"](\"assets/equations/cf36568f06144792.svg\")
![\"P_5=204{,}800+128=204{,}928\"](\"assets/equations/f00380ddc73ffc90.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 59 × 59 × 128. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{59-2}{2}\\right\\rfloor+1=29\"](\"assets/equations/c7e04e5a610fd348.svg\")
Pooling acts independently on each channel. The output is 29 × 29 × 128.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The division gives 28.5; take its floor before adding 1. A partial window outside the unpadded input is not included.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 29 × 29 × 128. Rearrange all activation values into one vector.
","![\"n_{\\mathrm{flat}}=29\\times29\\times128=107{,}648\"](\"assets/equations/4f51a385169358d4.svg\")
Output: a vector with 107,648 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 107,648 features. Output: 84 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=107{,}648\\times84=9{,}042{,}432\"](\"assets/equations/3496f06ddb91fc91.svg\")
![\"N_{\\mathrm{biases}}=84\"](\"assets/equations/ec58135349b68114.svg\")
![\"P_8=9{,}042{,}432+84=9{,}042{,}516\"](\"assets/equations/3a73b69ea184fcd0.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 84 features. Output: 32 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=84\\times32=2{,}688\"](\"assets/equations/6439e320bc2235fb.svg\")
![\"P_9=2{,}688+32=2{,}720\"](\"assets/equations/737742757d7487f8.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 32 features. Output: 1 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=32\\times1=32\"](\"assets/equations/63bcab1d4c5ad97d.svg\")
![\"N_{\\mathrm{biases}}=1\"](\"assets/equations/9172e32c44e88eec.svg\")
![\"P_10=32+1=33\"](\"assets/equations/6095954f97f48924.svg\")
| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | ![\"\\left\\lfloor\\frac{256+2](\"assets/equations/6723e51cb832856a.svg\") | 252 x 252 x 32 | 2,400 | 32 | 2,432 |
| 2 pool | ![\"\\left\\lfloor\\frac{252-2}{2}\\right\\rfloor+1\"](\"assets/equations/88e7931306349767.svg\") | 126 x 126 x 32 | 0 | 0 | 0 |
| 3 conv | ![\"\\left\\lfloor\\frac{126+2](\"assets/equations/5b79f5df9ff78236.svg\") | 122 x 122 x 64 | 51,200 | 64 | 51,264 |
| 4 pool | ![\"\\left\\lfloor\\frac{122-2}{2}\\right\\rfloor+1\"](\"assets/equations/8713b9890edc970c.svg\") | 61 x 61 x 64 | 0 | 0 | 0 |
| 5 conv | ![\"\\left\\lfloor\\frac{61+2](\"assets/equations/54ee85e0468271d4.svg\") | 59 x 59 x 128 | 204,800 | 128 | 204,928 |
| 6 pool | ![\"\\left\\lfloor\\frac{59-2}{2}\\right\\rfloor+1\"](\"assets/equations/e80bd4fe22509760.svg\") | 29 x 29 x 128 | 0 | 0 | 0 |
| 7 flat | 29 × 29 × 128 | 107648 | 0 | 0 | 0 |
| 8 dense | 107648 → 84 | 84 | 9,042,432 | 84 | 9,042,516 |
| 9 dense | 84 → 32 | 32 | 2,688 | 32 | 2,720 |
| 10 dense | 32 → 1 | 1 | 32 | 1 | 33 |
Parameter substitutions, in layer order:
","Layer 1: (5 x 5 x 3 + 1) x 32 = 2,432.
","Layer 3: (5 x 5 x 32 + 1) x 64 = 51,264.
","Layer 5: (5 x 5 x 64 + 1) x 128 = 204,928.
","Layer 8: (107648 + 1) x 84 = 9,042,516.
","Layer 9: (84 + 1) x 32 = 2,720.
","Layer 10: (32 + 1) x 1 = 33.
","Total: 2,432 + 51,264 + 204,928 + 9,042,516 + 2,720 + 33 = 9,303,893 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","Correction: Conv 3 has (5 × 5 × 64+1) × 128 = 204,928 parameters, not 73,856 as printed in the key. Correct total is 9,303,893. If binary classification is instead implemented with two softmax outputs, the last dense layer has(32+1) × 2 = 66 parameters and the total becomes 9,303,926; explicitly state the output convention.
"],"kind":"practice","id":"mt2-c12-s7","search":"R26 Q6a **Question (10 marks).** For the binary CNN below, calculate output dimensions and trainable parameters after every layer. Input 256 × 256 × 3. → Conv K = 5, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 128, stride = 1, padding = 1. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 84 units. → Dense 32 units. → Dense 1 units. The printed output size is unspecified; assume one sigmoid output. Unspecified convolution padding is taken as valid.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput 256 × 256 × 3.\n\nConv K = 5, filters = 32, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nConv K = 5, filters = 64, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nConv K = 5, filters = 128, stride = 1, padding = 1.\n\nPool K = 2, stride = 2, padding = 0.\n\nDense 84 units.\n\nDense 32 units.\n\nDense 1 units.\n\nassume one sigmoid output.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{256+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 252 x 252 x 32 | 2,400 | 32 | 2,432 |\n| 2 pool | \\(\\left\\lfloor\\frac{252-2}{2}\\right\\rfloor+1\\) | 126 x 126 x 32 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{126+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 122 x 122 x 64 | 51,200 | 64 | 51,264 |\n| 4 pool | \\(\\left\\lfloor\\frac{122-2}{2}\\right\\rfloor+1\\) | 61 x 61 x 64 | 0 | 0 | 0 |\n| 5 conv | \\(\\left\\lfloor\\frac{61+2 \\times 1-5}{1}\\right\\rfloor+1\\) | 59 x 59 x 128 | 204,800 | 128 | 204,928 |\n| 6 pool | \\(\\left\\lfloor\\frac{59-2}{2}\\right\\rfloor+1\\) | 29 x 29 x 128 | 0 | 0 | 0 |\n| 7 flat | 29 × 29 × 128 | 107648 | 0 | 0 | 0 |\n| 8 dense | 107648 → 84 | 84 | 9,042,432 | 84 | 9,042,516 |\n| 9 dense | 84 → 32 | 32 | 2,688 | 32 | 2,720 |\n| 10 dense | 32 → 1 | 1 | 32 | 1 | 33 |\n**Parameter substitutions, in layer order: **\nLayer 1: (5 x 5 x 3 + 1) x 32 = 2,432.\nLayer 3: (5 x 5 x 32 + 1) x 64 = 51,264.\nLayer 5: (5 x 5 x 64 + 1) x 128 = 204,928.\nLayer 8: (107648 + 1) x 84 = 9,042,516.\nLayer 9: (84 + 1) x 32 = 2,720.\nLayer 10: (32 + 1) x 1 = 33.\n**Total: ** 2,432 + 51,264 + 204,928 + 9,042,516 + 2,720 + 33 = **9,303,893 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.\n**Correction: ** Conv 3 has (5 × 5 × 64+1) × 128 = **204,928** parameters, not 73,856 as printed in the key. Correct total is "},{"title":"R26 Q7b","question":["Question (10 marks). Explain transfer learning with CNNs and how to adapt a pretrained model to a new image classification task, including an example model.
"],"answer":["Transfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.
","1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together.
","2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.
","3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.
","4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.
","5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.
","6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds.
","Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.
","Concrete example: replace the 1000-class head of a pretrained ResNet with a 10-class vegetable head. Train the new head with the base frozen, then optionally fine-tune late residual blocks at a smaller rate. Input preprocessing must match the pretrained model. The 10 output probabilities sum to 1 for mutually exclusive classes.
"],"kind":"practice","id":"mt2-c12-s8","search":"R26 Q7b **Question (10 marks).** Explain transfer learning with CNNs and how to adapt a pretrained model to a new image classification task, including an example model.\n\n**Complete worked answer.**\nTransfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.\n1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together.\n2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.\n3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.\n4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.\n5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.\n6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds.\nFeature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.\n**Concrete example: ** replace the 1000-class head of a pretrained ResNet with a 10-class vegetable head. Train the new head with the base frozen, then optionally fine-tune late residual blocks at a smaller rate. Input preprocessing must match the pretrained model. The 10 output probabilities sum to 1 for mutually exclusive classes.\n\n\n"}]},{"n":13,"title":"F23 selected MT2 practice","sections":[{"title":"Start here","blocks":["The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper.
"],"kind":"read","id":"mt2-c13-intro"},{"title":"F23 Q1b","question":["Question (5 marks). Identify the hyperparameters to tune when training an ANN and explain their significance.
"],"answer":["Hyperparameters are chosen outside the ordinary weight-gradient updates. Learning rate controls step size; number of layers and units controls capacity; activation determines nonlinear behavior; batch size controls the data used per update and memory demand; epochs control training duration. Regularization strength, dropout rate and early-stopping patience affect generalization. Optimizer-specific choices include momentum β or Adam β₁, β₂,ε. Tune using validation data while keeping the final test set separate. Weights and biases are learned parameters; they are not hyperparameters merely because their initial values must be selected.
"],"kind":"practice","id":"mt2-c13-s0","search":"F23 Q1b **Question (5 marks).** Identify the hyperparameters to tune when training an ANN and explain their significance.\n\n**Complete worked answer.**\nHyperparameters are chosen outside the ordinary weight-gradient updates. Learning rate controls step size; number of layers and units controls capacity; activation determines nonlinear behavior; batch size controls the data used per update and memory demand; epochs control training duration. Regularization strength, dropout rate and early-stopping patience affect generalization. Optimizer-specific choices include momentum β or Adam β₁, β₂,ε. Tune using validation data while keeping the final test set separate. Weights and biases are learned parameters; they are not hyperparameters merely because their initial values must be selected.\n"},{"title":"F23 Q1d","question":["Question (5 marks). Compute the convolution using the input A and kernel K shown below using padding 0 and stride 1.
","| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Stride 1; P = 0 (no padding).
Compute the convolution using the input A and kernel K shown below using padding 0 and stride 1.
","Find every convolution output cell.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Step 1 shape: ![\"\\left\\lfloor\\frac{4+2](\"assets/equations/451621c0173b1850.svg\")
= 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 1–3. The window moves 1 cell per output step.
| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
Only the outlined cells belong to this output.
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Same row and column positions; no matrix multiplication.
| 10 | 9 | 5 |
| 5 | 14 | 6 |
| 3 | 3 | 2 |
Each cell is input × matching kernel coefficient.
24 + 25 + 8 = 57
| 57 | 59 |
| 40 | 50 |
Output row 1, column 1.
Window row 1.
","![\"R_1=(10)\\times(1)+(9)\\times(1)+(5)\\times(1)=24\"](\"assets/equations/9fe1b0c406527aa8.svg\")
Window row 2.
","![\"R_2=(5)\\times(1)+(7)\\times(2)+(6)\\times(1)=25\"](\"assets/equations/dec76766281c84de.svg\")
Window row 3.
","![\"R_3=(3)\\times(1)+(3)\\times(1)+(2)\\times(1)=8\"](\"assets/equations/612c6e731f469d2a.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,1}=(24)+(25)+(8)=57\"](\"assets/equations/1b718b308893f322.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 2–4. The window moves 1 cell per output step.
| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
Only the outlined cells belong to this output.
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Same row and column positions; no matrix multiplication.
| 9 | 5 | 6 |
| 7 | 12 | 7 |
| 3 | 2 | 8 |
Each cell is input × matching kernel coefficient.
20 + 26 + 13 = 59
| 57 | 59 |
| 40 | 50 |
Output row 1, column 2.
Window row 1.
","![\"R_1=(9)\\times(1)+(5)\\times(1)+(6)\\times(1)=20\"](\"assets/equations/c760b3950eb840ed.svg\")
Window row 2.
","![\"R_2=(7)\\times(1)+(6)\\times(2)+(7)\\times(1)=26\"](\"assets/equations/16eb873463fe23bb.svg\")
Window row 3.
","![\"R_3=(3)\\times(1)+(2)\\times(1)+(8)\\times(1)=13\"](\"assets/equations/17c18e38643140f3.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,2}=(20)+(26)+(13)=59\"](\"assets/equations/f077bed2b9d89eff.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–4, columns 1–3. The window moves 1 cell per output step.
| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
Only the outlined cells belong to this output.
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Same row and column positions; no matrix multiplication.
| 5 | 7 | 6 |
| 3 | 6 | 2 |
| 3 | 2 | 6 |
Each cell is input × matching kernel coefficient.
18 + 11 + 11 = 40
| 57 | 59 |
| 40 | 50 |
Output row 2, column 1.
Window row 1.
","![\"R_1=(5)\\times(1)+(7)\\times(1)+(6)\\times(1)=18\"](\"assets/equations/7ba28bdc7c806323.svg\")
Window row 2.
","![\"R_2=(3)\\times(1)+(3)\\times(2)+(2)\\times(1)=11\"](\"assets/equations/5861059147729994.svg\")
Window row 3.
","![\"R_3=(3)\\times(1)+(2)\\times(1)+(6)\\times(1)=11\"](\"assets/equations/c85bf61f7da39979.svg\")
Add the rows to obtain one output cell.
","![\"Y_{2,1}=(18)+(11)+(11)=40\"](\"assets/equations/7ba895c690daff56.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–4, columns 2–4. The window moves 1 cell per output step.
| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
Only the outlined cells belong to this output.
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Same row and column positions; no matrix multiplication.
| 7 | 6 | 7 |
| 3 | 4 | 8 |
| 2 | 6 | 7 |
Each cell is input × matching kernel coefficient.
20 + 15 + 15 = 50
| 57 | 59 |
| 40 | 50 |
Output row 2, column 2.
Window row 1.
","![\"R_1=(7)\\times(1)+(6)\\times(1)+(7)\\times(1)=20\"](\"assets/equations/05ae17650121d076.svg\")
Window row 2.
","![\"R_2=(3)\\times(1)+(2)\\times(2)+(8)\\times(1)=15\"](\"assets/equations/9dcb85c5c6766eb3.svg\")
Window row 3.
","![\"R_3=(2)\\times(1)+(6)\\times(1)+(7)\\times(1)=15\"](\"assets/equations/2745b26251055027.svg\")
Add the rows to obtain one output cell.
","![\"Y_{2,2}=(20)+(15)+(15)=50\"](\"assets/equations/90b52e79a3bf1954.svg\")
Final output: ![\"\\begin{bmatrix}57](\"assets/equations/3c78c4fdcdcef59f.svg\")
. Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
Question (10 marks). Explain key advantages of Adam for ANN training and how it differs from SGD.
"],"answer":["Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.
","![\"m_t=\\beta_1m_{t-1}+(1-\\beta_1)g_t,\\quad](\"assets/equations/903aa08b8d83755c.svg\")
![\"\\widehat](\"assets/equations/92678247acd594b0.svg\")
Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.
","For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.
","Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.
"],"kind":"practice","id":"mt2-c13-s2","search":"F23 Q3b **Question (10 marks).** Explain key advantages of Adam for ANN training and how it differs from SGD.\n\n**Complete worked answer.**\nAdam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.\n\n\nUse update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.\nFor g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.\nCompared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.\n"},{"title":"F23 Q4a","question":["Question (10 marks). Determine feature-map dimensions and parameters at every layer of the pictured AlexNet-style CNN. The normalized architecture is Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units.
"],"answer":["The normalized architecture is Input 227 × 227 × 3.
","Conv K = 11, filters = 96, stride = 4, padding = 0.
","Pool K = 3, stride = 2, padding = 0.
","Conv K = 5, filters = 256, stride = 1, padding = 2.
","Pool K = 3, stride = 2, padding = 0.
","Conv K = 3, filters = 384, stride = 1, padding = 1.
","Conv K = 3, filters = 384, stride = 1, padding = 1.
","Conv K = 3, filters = 256, stride = 1, padding = 1.
","Pool K = 3, stride = 2, padding = 0.
","Dense 4096 units.
","Dense 4096 units.
","Dense 1000 units.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 227 × 227 × 3. Use 96 filters of spatial size 11 × 11, stride 4, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{227+2(0)-11}{4}\\right\\rfloor+1=55\"](\"assets/equations/441a061296efda7a.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 55 × 55 × 96.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=11\\times11\\times3\\times96=34{,}848\"](\"assets/equations/9a11e5db4a09d338.svg\")
![\"N_{\\mathrm{biases}}=96\"](\"assets/equations/a3b19ed994769f54.svg\")
![\"P_1=34{,}848+96=34{,}944\"](\"assets/equations/ae51bc20750a112f.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 55 × 55 × 96. Use a 3 × 3 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{55-3}{2}\\right\\rfloor+1=27\"](\"assets/equations/d42806d80c07005c.svg\")
Pooling acts independently on each channel. The output is 27 × 27 × 96.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 27 × 27 × 96. Use 256 filters of spatial size 5 × 5, stride 1, and padding 2 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{27+2(2)-5}{1}\\right\\rfloor+1=27\"](\"assets/equations/7f4e102905e6bf9f.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 27 × 27 × 256.
","3. Count trainable values. Each filter receives all 96 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=5\\times5\\times96\\times256=614{,}400\"](\"assets/equations/b720c78ffc948913.svg\")
![\"N_{\\mathrm{biases}}=256\"](\"assets/equations/96c4c1de3aa70d55.svg\")
![\"P_3=614{,}400+256=614{,}656\"](\"assets/equations/29c935e56c01ee26.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 27 × 27 × 256. Use a 3 × 3 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{27-3}{2}\\right\\rfloor+1=13\"](\"assets/equations/ca522f15a3632877.svg\")
Pooling acts independently on each channel. The output is 13 × 13 × 256.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 256. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{13+2(1)-3}{1}\\right\\rfloor+1=13\"](\"assets/equations/b1fa159221b64b09.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.
","3. Count trainable values. Each filter receives all 256 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=3\\times3\\times256\\times384=884{,}736\"](\"assets/equations/492d70515e0406f9.svg\")
![\"N_{\\mathrm{biases}}=384\"](\"assets/equations/558f11bf6ad2aa25.svg\")
![\"P_5=884{,}736+384=885{,}120\"](\"assets/equations/2e9338e40808aff5.svg\")
Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 384. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.
","3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=3\\times3\\times384\\times384=1{,}327{,}104\"](\"assets/equations/3729f79bf2b69fab.svg\")
![\"P_6=1{,}327{,}104+384=1{,}327{,}488\"](\"assets/equations/4565bf3714918ad3.svg\")
Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 384. Use 256 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 256.
","3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=3\\times3\\times384\\times256=884{,}736\"](\"assets/equations/be0b63ec1f7ffb8c.svg\")
![\"P_7=884{,}736+256=884{,}992\"](\"assets/equations/c304d00633ff2322.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 13 × 13 × 256. Use a 3 × 3 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{13-3}{2}\\right\\rfloor+1=6\"](\"assets/equations/117abcc38db86957.svg\")
Pooling acts independently on each channel. The output is 6 × 6 × 256.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 6 × 6 × 256. Rearrange all activation values into one vector.
","![\"n_{\\mathrm{flat}}=6\\times6\\times256=9{,}216\"](\"assets/equations/8d9921c92554e599.svg\")
Output: a vector with 9,216 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 9,216 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=9{,}216\\times4{,}096=37{,}748{,}736\"](\"assets/equations/253a54d671a02309.svg\")
![\"N_{\\mathrm{biases}}=4{,}096\"](\"assets/equations/cc9a0e4319a0da43.svg\")
![\"P_10=37{,}748{,}736+4{,}096=37{,}752{,}832\"](\"assets/equations/3a6d14bbf5757207.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 4,096 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=4{,}096\\times4{,}096=16{,}777{,}216\"](\"assets/equations/fe4cfd667cc7b7ce.svg\")
![\"P_11=16{,}777{,}216+4{,}096=16{,}781{,}312\"](\"assets/equations/3652163d527344ad.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 4,096 features. Output: 1,000 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=4{,}096\\times1{,}000=4{,}096{,}000\"](\"assets/equations/3d7876f0d5fdb4fa.svg\")
![\"N_{\\mathrm{biases}}=1{,}000\"](\"assets/equations/d0008f88a44995b2.svg\")
![\"P_12=4{,}096{,}000+1{,}000=4{,}097{,}000\"](\"assets/equations/3adeefbc3dbe4d6b.svg\")
| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | ![\"\\left\\lfloor\\frac{227+2](\"assets/equations/74235148b2b7312a.svg\") | 55 x 55 x 96 | 34,848 | 96 | 34,944 |
| 2 pool | ![\"\\left\\lfloor\\frac{55-3}{2}\\right\\rfloor+1\"](\"assets/equations/cdac8f1c682bc40a.svg\") | 27 x 27 x 96 | 0 | 0 | 0 |
| 3 conv | ![\"\\left\\lfloor\\frac{27+2](\"assets/equations/71046f85d0cc0ca5.svg\") | 27 x 27 x 256 | 614,400 | 256 | 614,656 |
| 4 pool | ![\"\\left\\lfloor\\frac{27-3}{2}\\right\\rfloor+1\"](\"assets/equations/55802f10aac3668d.svg\") | 13 x 13 x 256 | 0 | 0 | 0 |
| 5 conv | ![\"\\left\\lfloor\\frac{13+2](\"assets/equations/e30499da6b8e9d4a.svg\") | 13 x 13 x 384 | 884,736 | 384 | 885,120 |
| 6 conv | | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 |
| 7 conv | | 13 x 13 x 256 | 884,736 | 256 | 884,992 |
| 8 pool | ![\"\\left\\lfloor\\frac{13-3}{2}\\right\\rfloor+1\"](\"assets/equations/73ab07c24eb836b2.svg\") | 6 x 6 x 256 | 0 | 0 | 0 |
| 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 |
| 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 |
| 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 |
| 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 |
Parameter substitutions, in layer order:
","Layer 1: (11 x 11 x 3 + 1) x 96 = 34,944.
","Layer 3: (5 x 5 x 96 + 1) x 256 = 614,656.
","Layer 5: (3 x 3 x 256 + 1) x 384 = 885,120.
","Layer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488.
","Layer 7: (3 x 3 x 384 + 1) x 256 = 884,992.
","Layer 10: (9216 + 1) x 4096 = 37,752,832.
","Layer 11: (4096 + 1) x 4096 = 16,781,312.
","Layer 12: (4096 + 1) x 1000 = 4,097,000.
","Total: 34,944 + 614,656 + 885,120 + 1,327,488 + 884,992 + 37,752,832 + 16,781,312 + 4,097,000 = 62,378,344 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","The count treats every listed convolution as connected to all input channels. Original grouped AlexNet implementations have different counts; do not insert grouping that is not shown in the question.
"],"kind":"practice","id":"mt2-c13-s3","search":"F23 Q4a **Question (10 marks).** Determine feature-map dimensions and parameters at every layer of the pictured AlexNet-style CNN. The normalized architecture is Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nThe normalized architecture is Input 227 × 227 × 3.\n\nConv K = 11, filters = 96, stride = 4, padding = 0.\n\nPool K = 3, stride = 2, padding = 0.\n\nConv K = 5, filters = 256, stride = 1, padding = 2.\n\nPool K = 3, stride = 2, padding = 0.\n\nConv K = 3, filters = 384, stride = 1, padding = 1.\n\nConv K = 3, filters = 384, stride = 1, padding = 1.\n\nConv K = 3, filters = 256, stride = 1, padding = 1.\n\nPool K = 3, stride = 2, padding = 0.\n\nDense 4096 units.\n\nDense 4096 units.\n\nDense 1000 units.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{227+2 \\times 0-11}{4}\\right\\rfloor+1\\) | 55 x 55 x 96 | 34,848 | 96 | 34,944 |\n| 2 pool | \\(\\left\\lfloor\\frac{55-3}{2}\\right\\rfloor+1\\) | 27 x 27 x 96 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{27+2 \\times 2-5}{1}\\right\\rfloor+1\\) | 27 x 27 x 256 | 614,400 | 256 | 614,656 |\n| 4 pool | \\(\\left\\lfloor\\frac{27-3}{2}\\right\\rfloor+1\\) | 13 x 13 x 256 | 0 | 0 | 0 |\n| 5 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 384 | 884,736 | 384 | 885,120 |\n| 6 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 |\n| 7 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 256 | 884,736 | 256 | 884,992 |\n| 8 pool | \\(\\left\\lfloor\\frac{13-3}{2}\\right\\rfloor+1\\) | 6 x 6 x 256 | 0 | 0 | 0 |\n| 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 |\n| 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 |\n| 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 |\n| 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 |\n**Parameter substitutions, in layer order: **\nLayer 1: (11 x 11 x 3 + 1) x 96 = 34,944.\nLayer 3: (5 x 5 x 96 + 1) x 256 = 614,656.\nLayer 5: (3 x 3 x 256 + 1) x 384 = 885,120.\nLayer 6: (3 x"},{"title":"F23 Q5a","question":["Question (10 marks). For Inception V 1, justify different kernel sizes in one block, explain 1 × 1 convolutions and sketch one block.
"],"answer":["An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
"],"kind":"practice","id":"mt2-c13-s4","search":"F23 Q5a **Question (10 marks).** For Inception V 1, justify different kernel sizes in one block, explain 1 × 1 convolutions and sketch one block.\n\n**Complete worked answer.**\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.\n"},{"title":"F23 Q6b","question":["Question (10 marks). Explain how ResNet addresses vanishing gradients. Compare a block output with and without a residual connection.
"],"answer":["A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.
","The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.
","Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].
"],"kind":"practice","id":"mt2-c13-s5","search":"F23 Q6b **Question (10 marks).** Explain how ResNet addresses vanishing gradients. Compare a block output with and without a residual connection.\n\n**Complete worked answer.**\nA residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.\n\n\nThe identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.\nElementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].\n"},{"title":"F23 Q7a","question":["Question (10 marks). Design and sketch a sentiment-analysis model whose input is a small paragraph. Explain how its output is generated.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","Concrete many-to-one design: tokenize paragraph → embeddings → vanilla RNN sequence processor → final hidden state → dense 3 → softmax for positive/neutral/negative. If the task specifies only two labels, use one sigmoid or two softmax outputs instead. Train with the matching cross-entropy. Keep token order, handle padding/masks consistently and evaluate held-out reviews.
","Question (10 marks). Explain the bias-variance tradeoff. Fill effects on bias and variance for weight regularization, more hidden units, dropout and more training data from the same distribution.
"],"answer":["Bias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data.
","Increasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws.
","Underfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate.
","Training and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting.
","| Change | Typical bias | Typical variance |
|---|---|---|
| Stronger weight regularization | Increases | Decreases |
| More hidden units | Decreases | Increases |
| Dropout during training | Increases | Decreases |
| More same-distribution training data | Approximately no change | Decreases |
These are the conventional exam trends, assuming other conditions are comparable. They are not guarantees for every training run.
"],"kind":"practice","id":"mt2-c13-s7","search":"F23 Q7b **Question (10 marks).** Explain the bias-variance tradeoff. Fill effects on bias and variance for weight regularization, more hidden units, dropout and more training data from the same distribution.\n\n**Complete worked answer.**\nBias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data.\n\nIncreasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws.\nUnderfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate.\nTraining and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting.\n| Change | Typical bias | Typical variance |\n| Stronger weight regularization | Increases | Decreases |\n| More hidden units | Decreases | Increases |\n| Dropout during training | Increases | Decreases |\n| More same-distribution training data | Approximately no change | Decreases |\nThese are the conventional exam trends, assuming other conditions are comparable. They are not guarantees for every training run.\n\n\n"}]},{"n":14,"title":"R24 selected MT2 practice","sections":[{"title":"Start here","blocks":["The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper.
"],"kind":"read","id":"mt2-c14-intro"},{"title":"R24 Q1c","question":["Question (5 marks). A deep network strongly overfits its training set and performs poorly on unseen data. Explain remedies.
"],"answer":["Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
","Use a representative validation set to choose the remedy. Also inspect leakage, label quality and distribution mismatch. If the final test set was repeatedly used to tune changes, reserve a fresh untouched set for final reporting.
"],"kind":"practice","id":"mt2-c14-s0","search":"R24 Q1c **Question (5 marks).** A deep network strongly overfits its training set and performs poorly on unseen data. Explain remedies.\n\n**Complete worked answer.**\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\nUse a representative validation set to choose the remedy. Also inspect leakage, label quality and distribution mismatch. If the final test set was repeatedly used to tune changes, reserve a fresh untouched set for final reporting.\n"},{"title":"R24 Q1d","question":["Question (5 marks). For feature map Matrix 1 shown below, find max and average pooling with 2 × 2 windows, stride 2.
","For feature map Matrix 1 shown below, find max and average pooling with 2 × 2 windows, stride 2.
","Find the requested pooled output(s).
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","Shape: ![\"\\left\\lfloor\\frac{4-2}{2}\\right\\rfloor+1\"](\"assets/equations/ad46d0c1b3646c71.svg\")
= 2 on each axis; channels stay unchanged.
Step 1: select the entries.
","[21, 11, 12, 9]
","Step 2: add all selected entries.
","21 + 11 + 12 + 9 = 53
","Step 3: divide by the 4 entries in this window.
","53/4 = 13.25. Maximum = 21
","Step 1: select the entries.
","[15, 10, 10, 11]
","Step 2: add all selected entries.
","15 + 10 + 10 + 11 = 46
","Step 3: divide by the 4 entries in this window.
","46/4 = 11.5. Maximum = 15
","Step 1: select the entries.
","[11, 13, 7, 4]
","Step 2: add all selected entries.
","11 + 13 + 7 + 4 = 35
","Step 3: divide by the 4 entries in this window.
","35/4 = 8.75. Maximum = 13
","Step 1: select the entries.
","[18, 12, 15, 12]
","Step 2: add all selected entries.
","18 + 12 + 15 + 12 = 57
","Step 3: divide by the 4 entries in this window.
","57/4 = 14.25. Maximum = 18
","Average output: . Max output: . Pooling normally has zero trainable parameters and reduces spatial dimensions.
Question. Compare standard momentum and Nesterov accelerated gradient using their gradient-evaluation points and update equations. Explain what may happen when momentum is extremely close to 1, for example 0.9999.
","Syllabus-aligned extract: source R24 Q2b. The current-syllabus optimizer comparison and high-momentum discussion are retained; marks for this extract are not separately stated.
"],"answer":["Use signed velocity. Standard momentum evaluates g at current θ: vnew = βv−ηg(θ), θ new = θ+vnew. Nesterov evaluates at a look-ahead point: θ look = θ+βv; vnew = βv−ηg(θ look); θ new = θ+vnew. Thus the gradient can correct a direction after anticipating part of the momentum move. Nesterov often reduces overshoot in suitable settings, but it is not universally faster for every neural network.
","With β = 0.9999, old velocity decays very slowly. The method can respond sluggishly to direction changes, overshoot or oscillate, particularly with a large learning rate. An approximate memory scale 1/(1−β) = 10,000 steps illustrates the long persistence; it is not an exact convergence time.
"],"kind":"practice","id":"mt2-c14-s2","search":"R24 Q2b **Question.** Compare standard momentum and Nesterov accelerated gradient using their gradient-evaluation points and update equations. Explain what may happen when momentum is extremely close to 1, for example 0.9999.\n\n**Syllabus-aligned extract:** source R24 Q2b. The current-syllabus optimizer comparison and high-momentum discussion are retained; marks for this extract are not separately stated.\n\n**Complete worked answer.**\nUse signed velocity. Standard momentum evaluates g at current θ: vnew = βv−ηg(θ), θ new = θ+vnew. Nesterov evaluates at a look-ahead point: θ look = θ+βv; vnew = βv−ηg(θ look); θ new = θ+vnew. Thus the gradient can correct a direction after anticipating part of the momentum move. Nesterov often reduces overshoot in suitable settings, but it is not universally faster for every neural network.\nWith β = 0.9999, old velocity decays very slowly. The method can respond sluggishly to direction changes, overshoot or oscillate, particularly with a large learning rate. An approximate memory scale 1/(1−β) = 10,000 steps illustrates the long persistence; it is not an exact convergence time.\n"},{"title":"R24 Q3b","question":["Question (10 marks). Explain why regularization matters, list techniques and elaborate one.
"],"answer":["Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
","Elaborating L2: add(λ/2)Σw² to the data loss. For weight w = .5, data gradient.2, λ = .1, η = .1, total gradient = .2+.1 × .5 = .25, so wnew = .5−.1 × .25 = .475. Relative to the data-only update.48, the penalty pulls the weight further toward 0. Too large λ can remove useful signal, so tune on validation data.
"],"kind":"practice","id":"mt2-c14-s3","search":"R24 Q3b **Question (10 marks).** Explain why regularization matters, list techniques and elaborate one.\n\n**Complete worked answer.**\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\n**Elaborating L2: ** add(λ/2)Σw² to the data loss. For weight w = .5, data gradient.2, λ = .1, η = .1, total gradient = .2+.1 × .5 = .25, so wnew = .5−.1 × .25 = .475. Relative to the data-only update.48, the penalty pulls the weight further toward 0. Too large λ can remove useful signal, so tune on validation data.\n"},{"title":"R24 Q4b","question":["Question (10 marks). Calculate weights, biases and feature-map sizes of the CNN: Input 227 × 227 × 3. → Conv K = 9, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 120 units. → Dense 84 units. → Dense 10 units.
"],"answer":["Input 227 × 227 × 3.
","Conv K = 9, filters = 32, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 5, filters = 64, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 5, filters = 64, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Dense 120 units.
","Dense 84 units.
","Dense 10 units.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 227 × 227 × 3. Use 32 filters of spatial size 9 × 9, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{227+2(0)-9}{1}\\right\\rfloor+1=219\"](\"assets/equations/0d078197de388e4c.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 219 × 219 × 32.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=9\\times9\\times3\\times32=7{,}776\"](\"assets/equations/90e2d92ad9867cbd.svg\")
![\"P_1=7{,}776+32=7{,}808\"](\"assets/equations/59616c98ab8fffe7.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 219 × 219 × 32. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{219-2}{2}\\right\\rfloor+1=109\"](\"assets/equations/3f1c898ff7560508.svg\")
Pooling acts independently on each channel. The output is 109 × 109 × 32.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The division gives 108.5; take its floor before adding 1. A partial window outside the unpadded input is not included.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 109 × 109 × 32. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{109+2(0)-5}{1}\\right\\rfloor+1=105\"](\"assets/equations/a2cc5fced7b8ab11.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 105 × 105 × 64.
","3. Count trainable values. Each filter receives all 32 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 105 × 105 × 64. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{105-2}{2}\\right\\rfloor+1=52\"](\"assets/equations/616d6791b658eeb9.svg\")
Pooling acts independently on each channel. The output is 52 × 52 × 64.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The division gives 51.5; take its floor before adding 1. A partial window outside the unpadded input is not included.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 52 × 52 × 64. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{52+2(0)-5}{1}\\right\\rfloor+1=48\"](\"assets/equations/58a6dd2bccbabd6a.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 48 × 48 × 64.
","3. Count trainable values. Each filter receives all 64 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=5\\times5\\times64\\times64=102{,}400\"](\"assets/equations/c1baccc90afa895b.svg\")
![\"P_5=102{,}400+64=102{,}464\"](\"assets/equations/0bf8643a16a94e64.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 48 × 48 × 64. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{48-2}{2}\\right\\rfloor+1=24\"](\"assets/equations/64a91a8e536e0765.svg\")
Pooling acts independently on each channel. The output is 24 × 24 × 64.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 24 × 24 × 64. Rearrange all activation values into one vector.
","![\"n_{\\mathrm{flat}}=24\\times24\\times64=36{,}864\"](\"assets/equations/5ee055840a7e6f06.svg\")
Output: a vector with 36,864 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 36,864 features. Output: 120 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=36{,}864\\times120=4{,}423{,}680\"](\"assets/equations/b963fc1e5d67938a.svg\")
![\"P_8=4{,}423{,}680+120=4{,}423{,}800\"](\"assets/equations/99a27e646d35bbda.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 120 features. Output: 84 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=120\\times84=10{,}080\"](\"assets/equations/4874c9639ae8a286.svg\")
![\"P_9=10{,}080+84=10{,}164\"](\"assets/equations/3cfadc0698f826a8.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 84 features. Output: 10 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=84\\times10=840\"](\"assets/equations/995f161a94dc7ae1.svg\")
![\"N_{\\mathrm{biases}}=10\"](\"assets/equations/8203d983cccce3a9.svg\")
![\"P_10=840+10=850\"](\"assets/equations/be85fac4796ebc34.svg\")
| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | ![\"\\left\\lfloor\\frac{227+2](\"assets/equations/8a2a9eb5376c2557.svg\") | 219 x 219 x 32 | 7,776 | 32 | 7,808 |
| 2 pool | ![\"\\left\\lfloor\\frac{219-2}{2}\\right\\rfloor+1\"](\"assets/equations/4185cd16a03f14e2.svg\") | 109 x 109 x 32 | 0 | 0 | 0 |
| 3 conv | ![\"\\left\\lfloor\\frac{109+2](\"assets/equations/e91c4793e36ecf45.svg\") | 105 x 105 x 64 | 51,200 | 64 | 51,264 |
| 4 pool | ![\"\\left\\lfloor\\frac{105-2}{2}\\right\\rfloor+1\"](\"assets/equations/02f6d3536f89b2ae.svg\") | 52 x 52 x 64 | 0 | 0 | 0 |
| 5 conv | ![\"\\left\\lfloor\\frac{52+2](\"assets/equations/6619d9b65db6ea0f.svg\") | 48 x 48 x 64 | 102,400 | 64 | 102,464 |
| 6 pool | ![\"\\left\\lfloor\\frac{48-2}{2}\\right\\rfloor+1\"](\"assets/equations/b6ef920332c6684a.svg\") | 24 x 24 x 64 | 0 | 0 | 0 |
| 7 flat | 24 × 24 × 64 | 36864 | 0 | 0 | 0 |
| 8 dense | 36864 → 120 | 120 | 4,423,680 | 120 | 4,423,800 |
| 9 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 |
| 10 dense | 84 → 10 | 10 | 840 | 10 | 850 |
Parameter substitutions, in layer order:
","Layer 1: (9 x 9 x 3 + 1) x 32 = 7,808.
","Layer 3: (5 x 5 x 32 + 1) x 64 = 51,264.
","Layer 5: (5 x 5 x 64 + 1) x 64 = 102,464.
","Layer 8: (36864 + 1) x 120 = 4,423,800.
","Layer 9: (120 + 1) x 84 = 10,164.
","Layer 10: (84 + 1) x 10 = 850.
","Total: 7,808 + 51,264 + 102,464 + 4,423,800 + 10,164 + 850 = 4,596,350 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
"],"kind":"practice","id":"mt2-c14-s4","search":"R24 Q4b **Question (10 marks).** Calculate weights, biases and feature-map sizes of the CNN: Input 227 × 227 × 3. → Conv K = 9, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 120 units. → Dense 84 units. → Dense 10 units.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput 227 × 227 × 3.\n\nConv K = 9, filters = 32, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nConv K = 5, filters = 64, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nConv K = 5, filters = 64, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nDense 120 units.\n\nDense 84 units.\n\nDense 10 units.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{227+2 \\times 0-9}{1}\\right\\rfloor+1\\) | 219 x 219 x 32 | 7,776 | 32 | 7,808 |\n| 2 pool | \\(\\left\\lfloor\\frac{219-2}{2}\\right\\rfloor+1\\) | 109 x 109 x 32 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{109+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 105 x 105 x 64 | 51,200 | 64 | 51,264 |\n| 4 pool | \\(\\left\\lfloor\\frac{105-2}{2}\\right\\rfloor+1\\) | 52 x 52 x 64 | 0 | 0 | 0 |\n| 5 conv | \\(\\left\\lfloor\\frac{52+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 48 x 48 x 64 | 102,400 | 64 | 102,464 |\n| 6 pool | \\(\\left\\lfloor\\frac{48-2}{2}\\right\\rfloor+1\\) | 24 x 24 x 64 | 0 | 0 | 0 |\n| 7 flat | 24 × 24 × 64 | 36864 | 0 | 0 | 0 |\n| 8 dense | 36864 → 120 | 120 | 4,423,680 | 120 | 4,423,800 |\n| 9 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 |\n| 10 dense | 84 → 10 | 10 | 840 | 10 | 850 |\n**Parameter substitutions, in layer order: **\nLayer 1: (9 x 9 x 3 + 1) x 32 = 7,808.\nLayer 3: (5 x 5 x 32 + 1) x 64 = 51,264.\nLayer 5: (5 x 5 x 64 + 1) x 64 = 102,464.\nLayer 8: (36864 + 1) x 120 = 4,423,800.\nLayer 9: (120 + 1) x 84 = 10,164.\nLayer 10: (84 + 1) x 10 = 850.\n**Total: ** 7,808 + 51,264 + 102,464 + 4,423,800 + 10,164 + 850 = **4,596,350 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.\n"},{"title":"R24 Q5a","question":["Question (10 marks). A GoogLeNet classifier trains slowly and vanishing gradients are suspected. Suggest an architecture change and explain its important features.
"],"answer":["A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.
","The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.
","Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].
","A ResNet is a suitable architecture to investigate because skip additions provide shorter gradient paths. First verify the diagnosis with gradients and training curves; slow training alone does not establish vanishing gradients. Match projection shortcuts when spatial size/channels change and compare validation performance after the redesign.
"],"kind":"practice","id":"mt2-c14-s5","search":"R24 Q5a **Question (10 marks).** A GoogLeNet classifier trains slowly and vanishing gradients are suspected. Suggest an architecture change and explain its important features.\n\n**Complete worked answer.**\nA residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.\n\n\nThe identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.\nElementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].\nA ResNet is a suitable architecture to investigate because skip additions provide shorter gradient paths. First verify the diagnosis with gradients and training curves; slow training alone does not establish vanishing gradients. Match projection shortcuts when spatial size/channels change and compare validation performance after the redesign.\n"},{"title":"R24 Q6b","question":["Question (10 marks). A supermarket vegetable classifier has 10 classes but few examples per class and limited training resources. Explain different ways to use a pretrained model.
"],"answer":["Transfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.
","1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together.
","2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.
","3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.
","4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.
","5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.
","6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds.
","Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.
","For vegetables, replace the source head with 10 softmax outputs. Strategy 1: freeze the base and train only this head. Strategy 2: initialize from the same pretrained model and fine-tune selected later blocks at a small rate after head training. The first is cheaper and less prone to fitting all parameters from limited examples; the second can adapt features more closely to vegetables when validation supports it.
"],"kind":"practice","id":"mt2-c14-s6","search":"R24 Q6b **Question (10 marks).** A supermarket vegetable classifier has 10 classes but few examples per class and limited training resources. Explain different ways to use a pretrained model.\n\n**Complete worked answer.**\nTransfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.\n1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together.\n2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.\n3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.\n4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.\n5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.\n6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds.\nFeature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.\nFor vegetables, replace the source head with 10 softmax outputs. Strategy 1: freeze the base and train only this head. Strategy 2: initialize from the same pretrained model and fine-tune selected later blocks at a small rate after head training. The first is cheaper and less prone to fitting all parameters from limited examples; the second can adapt features more closely to vegetables when validation supports it.\n"},{"title":"R24 Q7b","question":["Question (10 marks). Explain different RNN input/output sequence types with architecture sketches.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","One-to-many: one image representation generates a caption. Many-to-one: a review produces one sentiment. Aligned many-to-many: each video frame receives a label. Unaligned many-to-many: an encoder reads a source sentence and a decoder generates a translation of possibly different length. Specify which outputs carry losses during training; variable lengths do not require new weights for each position.
"],"kind":"practice","id":"mt2-c14-s7","search":"R24 Q7b **Question (10 marks).** Explain different RNN input/output sequence types with architecture sketches.\n\n**Complete worked answer.**\nA recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.\n\n\nAt each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.\nFor next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.\nA basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.\n\nOne-to-many: one image representation generates a caption. Many-to-one: a review produces one sentiment. Aligned many-to-many: each video frame receives a label. Unaligned many-to-many: an encoder reads a source sentence and a decoder generates a translation of possibly different length. Specify which outputs carry losses during training; variable lengths do not require new weights for each position.\n\n\n"}]},{"n":15,"title":"F22 selected MT2 practice","sections":[{"title":"Start here","blocks":["The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper.
"],"kind":"read","id":"mt2-c15-intro"},{"title":"F22 Q1a","question":["Question (4 marks). A normalized 5 × 5 image has every row [0.5,0.5,0.3,0.3,0.3]. Convolve it with a 3 × 3 filter whose every row is [1,0,−1], stride 2, no padding. Compute the output and identify the feature.
"],"answer":["A normalized 5 × 5 image has every row [0.5,0.5,0.3,0.3,0.3].
","Convolve it with a 3 × 3 filter whose every row is [1,0,−1], stride 2, no padding.
","Find every convolution output cell.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Step 1 shape: = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 1–3. The window moves 2 cells per output step.
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0.5 | 0 | −0.3 |
| 0.5 | 0 | −0.3 |
| 0.5 | 0 | −0.3 |
Each cell is input × matching kernel coefficient.
0.2 + 0.2 + 0.2 = 0.6
| 0.6 | 0 |
| 0.6 | 0 |
Output row 1, column 1.
Window row 1.
","![\"R_1=(0.5)\\times(1)+(0.5)\\times(0)+(0.3)\\times(-1)=0.2\"](\"assets/equations/46893fc8d069d1a9.svg\")
Window row 2.
","![\"R_2=(0.5)\\times(1)+(0.5)\\times(0)+(0.3)\\times(-1)=0.2\"](\"assets/equations/6a410c53ff471ca9.svg\")
Window row 3.
","![\"R_3=(0.5)\\times(1)+(0.5)\\times(0)+(0.3)\\times(-1)=0.2\"](\"assets/equations/3c0f1258f7cc1b3a.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,1}=(0.2)+(0.2)+(0.2)=0.6\"](\"assets/equations/fb7d1b049e078f58.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 3–5. The window moves 2 cells per output step.
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0.3 | 0 | −0.3 |
| 0.3 | 0 | −0.3 |
| 0.3 | 0 | −0.3 |
Each cell is input × matching kernel coefficient.
0 + 0 + 0 = 0
| 0.6 | 0 |
| 0.6 | 0 |
Output row 1, column 2.
Window row 1.
","![\"R_1=(0.3)\\times(1)+(0.3)\\times(0)+(0.3)\\times(-1)=0\"](\"assets/equations/8fffc5f555fb2f95.svg\")
Window row 2.
","![\"R_2=(0.3)\\times(1)+(0.3)\\times(0)+(0.3)\\times(-1)=0\"](\"assets/equations/a27c355bf724dc1d.svg\")
Window row 3.
","![\"R_3=(0.3)\\times(1)+(0.3)\\times(0)+(0.3)\\times(-1)=0\"](\"assets/equations/9405479984ca544c.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,2}=(0)+(0)+(0)=0\"](\"assets/equations/f3a79dadbe7e581b.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 1–3. The window moves 2 cells per output step.
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0.5 | 0 | −0.3 |
| 0.5 | 0 | −0.3 |
| 0.5 | 0 | −0.3 |
Each cell is input × matching kernel coefficient.
0.2 + 0.2 + 0.2 = 0.6
| 0.6 | 0 |
| 0.6 | 0 |
Output row 2, column 1.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","![\"Y_{2,1}=(0.2)+(0.2)+(0.2)=0.6\"](\"assets/equations/9e16ceceb6a0a438.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 3–5. The window moves 2 cells per output step.
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0.3 | 0 | −0.3 |
| 0.3 | 0 | −0.3 |
| 0.3 | 0 | −0.3 |
Each cell is input × matching kernel coefficient.
0 + 0 + 0 = 0
| 0.6 | 0 |
| 0.6 | 0 |
Output row 2, column 2.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","![\"Y_{2,2}=(0)+(0)+(0)=0\"](\"assets/equations/9d6dfc0f38e45b75.svg\")
Final output: ![\"\\begin{bmatrix}0.6](\"assets/equations/14d5a1ec1325dcc6.svg\")
. Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
The filter compares left and right neighborhoods and responds to a vertical brightness boundary. Inputs are already normalized. No additional output-normalization rule is supplied, so the determined convolution is . If an examiner explicitly requests min-max normalization of this result, divide by 0.6 to obtain ![\"\\begin{bmatrix}1](\"assets/equations/40c2bacd9336c7bb.svg\")
; that is a separate stated operation.
Question (4 marks). Weights are 0.55 and 0.65; their gradients are+0.3 and−0.25. With η = 0.1, compute gradient-descent changes and new weights.
"],"answer":["Old weights = [0.55,0.65].
","Corresponding supplied gradients = [0.3,−0.25]; η = 0.1.
","Calculate the signed weight changes and both new weights.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","Step 1: calculate the change in weight 1.
","![\"\\Delta](\"assets/equations/eaf1d1bad63a2309.svg\")
Step 2: add that change to old weight 1.
","![\"w_1^{\\mathrm{new}}=0.55+(-0.03)=0.52\"](\"assets/equations/278794747f889246.svg\")
Step 3: calculate the change in weight 2.
","![\"\\Delta](\"assets/equations/1c844e3b7026fa3a.svg\")
Step 4: add that change to old weight 2.
","![\"w_2^{\\mathrm{new}}=0.65+0.025=0.675\"](\"assets/equations/99431f20567e072c.svg\")
Final answer: changes = [−0.03, +0.025]; new weights = [0.52, 0.675]. A negative gradient increases its weight under descent.
"],"kind":"practice","id":"mt2-c15-s1","search":"F22 Q1b **Question (4 marks).** Weights are 0.55 and 0.65; their gradients are+0.3 and−0.25. With η = 0.1, compute gradient-descent changes and new weights.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nOld weights = [0.55,0.65].\n\nCorresponding supplied gradients = [0.3,−0.25]; η = 0.1.\n\n### To find\n\nCalculate the signed weight changes and both new weights.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n\n**Step 1: calculate the change in weight 1.**\n\n\n\n\n\n\n\n**Step 2: add that change to old weight 1.**\n\n\n\n\n\n\n\n**Step 3: calculate the change in weight 2.**\n\n\n\n\n\n\n\n**Step 4: add that change to old weight 2.**\n\n\n\n\n\n\n\n**Final answer:** changes = [−0.03, +0.025]; new weights = [0.52, 0.675]. A negative gradient increases its weight under descent.\n"},{"title":"F22 Q1e","question":["Question (4 marks). Suggest and sketch an RNN architecture for image captioning.
"],"answer":["Use a CNN encoder to map the image to a visual representation and a recurrent decoder (vanilla RNN) to generate words one at a time. Condition the decoder’s initial state or its inputs on image features. At each step, vocabulary softmax predicts the next word based on the image and earlier words. Train using correct next-word targets and cross-entropy; stop generation at an end token. This is a one-to-many mapping from one image to a caption sequence.
","Question (10 marks). Explain bias, variance, underfitting and overfitting. A four-feature product-texture classifier has four grades including reject. Choose output activation/loss. Diagnose training error 15% and validation error 18% and suggest remedies.
"],"answer":["Bias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data.
","Increasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws.
","Underfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate.
","Training and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting.
","Classifier: four mutually exclusive grades require 4 output scores, softmax probabilities and categorical cross-entropy. A reject class is still a labeled class in this formulation.
","Diagnosis: a modest 3 percentage-point gap with both errors appreciable suggests underfitting/high bias if a much better attainable baseline exists. Confirm learning curves and label quality. Improve relevant features or model capacity, train sufficiently, tune the optimizer and reduce excessive regularization. The percentages alone do not prove the cause.
"],"kind":"practice","id":"mt2-c15-s3","search":"F22 Q3a **Question (10 marks).** Explain bias, variance, underfitting and overfitting. A four-feature product-texture classifier has four grades including reject. Choose output activation/loss. Diagnose training error 15% and validation error 18% and suggest remedies.\n\n**Complete worked answer.**\nBias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data.\n\nIncreasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws.\nUnderfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate.\nTraining and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting.\n**Classifier: ** four mutually exclusive grades require 4 output scores, softmax probabilities and categorical cross-entropy. A reject class is still a labeled class in this formulation.\n**Diagnosis: ** a modest 3 percentage-point gap with both errors appreciable suggests underfitting/high bias if a much better attainable baseline exists. Confirm learning curves and label quality. Improve relevant features or model capacity, train sufficiently, tune the optimizer and reduce excessive regularization. The percentages alone do not prove the cause.\n"},{"title":"F22 Q3b","question":["Question (10 marks). Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2; convolution 5 × 5: 16 filters, pooling 2; dense 120, dense 84, softmax 10. Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps.
"],"answer":["Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2.
","convolution 5 × 5: 16 filters, pooling 2.
","dense 120, dense 84, softmax 10.
","Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 32 × 32 × 1. Use 6 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{32+2(0)-5}{1}\\right\\rfloor+1=28\"](\"assets/equations/a58485fe718e957c.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 28 × 28 × 6.
","3. Count trainable values. Each filter receives all 1 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=5\\times5\\times1\\times6=150\"](\"assets/equations/4503fe259db6cc31.svg\")
![\"N_{\\mathrm{biases}}=6\"](\"assets/equations/07df0417eea506bc.svg\")
![\"P_1=150+6=156\"](\"assets/equations/60d683ee579f75b5.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 28 × 28 × 6. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{28-2}{2}\\right\\rfloor+1=14\"](\"assets/equations/d66b872d3a8b7d76.svg\")
Pooling acts independently on each channel. The output is 14 × 14 × 6.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 14 × 14 × 6. Use 16 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{14+2(0)-5}{1}\\right\\rfloor+1=10\"](\"assets/equations/85506bf5dfa9eb3a.svg\")
2. Determine output channels. Each filter produces one feature map. Therefore the output is 10 × 10 × 16.
","3. Count trainable values. Each filter receives all 6 input channels. Its weights are reused at every spatial position.
","![\"N_{\\mathrm{weights}}=5\\times5\\times6\\times16=2{,}400\"](\"assets/equations/95205b29ca5fb5db.svg\")
![\"P_3=2{,}400+16=2{,}416\"](\"assets/equations/0985f033fb9785fd.svg\")
Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 10 × 10 × 16. Use a 2 × 2 window, stride 2, and zero padding.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{10-2}{2}\\right\\rfloor+1=5\"](\"assets/equations/c2393768f5c7cc6f.svg\")
Pooling acts independently on each channel. The output is 5 × 5 × 16.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 5 × 5 × 16. Rearrange all activation values into one vector.
","![\"n_{\\mathrm{flat}}=5\\times5\\times16=400\"](\"assets/equations/b7fdfd1b9a608954.svg\")
Output: a vector with 400 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 400 features. Output: 120 neurons. Every output neuron connects to every input feature and has one bias.
","![\"N_{\\mathrm{weights}}=400\\times120=48{,}000\"](\"assets/equations/142c2516a0279849.svg\")
![\"P_6=48{,}000+120=48{,}120\"](\"assets/equations/ee00c5a42d90e348.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 120 features. Output: 84 neurons. Every output neuron connects to every input feature and has one bias.
","![\"P_7=10{,}080+84=10{,}164\"](\"assets/equations/c9fa89f780aa10aa.svg\")
Every output neuron receives every input feature, with one bias per output neuron.
Input: 84 features. Output: 10 neurons. Every output neuron connects to every input feature and has one bias.
","![\"P_8=840+10=850\"](\"assets/equations/3b9609ba48ce42f1.svg\")
| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | ![\"\\left\\lfloor\\frac{32+2](\"assets/equations/77e2ec6d75d55591.svg\") | 28 x 28 x 6 | 150 | 6 | 156 |
| 2 pool | ![\"\\left\\lfloor\\frac{28-2}{2}\\right\\rfloor+1\"](\"assets/equations/0f4985548e8cd39b.svg\") | 14 x 14 x 6 | 0 | 0 | 0 |
| 3 conv | ![\"\\left\\lfloor\\frac{14+2](\"assets/equations/104aadf49e642e25.svg\") | 10 x 10 x 16 | 2,400 | 16 | 2,416 |
| 4 pool | ![\"\\left\\lfloor\\frac{10-2}{2}\\right\\rfloor+1\"](\"assets/equations/0c637a47bfb54036.svg\") | 5 x 5 x 16 | 0 | 0 | 0 |
| 5 flat | 5 × 5 × 16 | 400 | 0 | 0 | 0 |
| 6 dense | 400 → 120 | 120 | 48,000 | 120 | 48,120 |
| 7 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 |
| 8 dense | 84 → 10 | 10 | 840 | 10 | 850 |
Parameter substitutions, in layer order:
","Layer 1: (5 x 5 x 1 + 1) x 6 = 156.
","Layer 3: (5 x 5 x 6 + 1) x 16 = 2,416.
","Layer 6: (400 + 1) x 120 = 48,120.
","Layer 7: (120 + 1) x 84 = 10,164.
","Layer 8: (84 + 1) x 10 = 850.
","Total: 156 + 2,416 + 48,120 + 10,164 + 850 = 61,706 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","This is the standard classroom full-channel interpretation with nontrainable pooling and a dense 10-class softmax. The historical original LeNet used partial channel connections and other details, so its published parameter count need not equal this diagram-based count. If an input label is read as 28, the stated later 28 map after valid 5 × 5 convolution is inconsistent; record the 32 input assumption rather than silently ignoring the mismatch.
"],"kind":"practice","id":"mt2-c15-s4","search":"F22 Q3b **Question (10 marks).** Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2; convolution 5 × 5: 16 filters, pooling 2; dense 120, dense 84, softmax 10. Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nCount trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2.\n\nconvolution 5 × 5: 16 filters, pooling 2.\n\ndense 120, dense 84, softmax 10.\n\nUse the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{32+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 28 x 28 x 6 | 150 | 6 | 156 |\n| 2 pool | \\(\\left\\lfloor\\frac{28-2}{2}\\right\\rfloor+1\\) | 14 x 14 x 6 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{14+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 10 x 10 x 16 | 2,400 | 16 | 2,416 |\n| 4 pool | \\(\\left\\lfloor\\frac{10-2}{2}\\right\\rfloor+1\\) | 5 x 5 x 16 | 0 | 0 | 0 |\n| 5 flat | 5 × 5 × 16 | 400 | 0 | 0 | 0 |\n| 6 dense | 400 → 120 | 120 | 48,000 | 120 | 48,120 |\n| 7 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 |\n| 8 dense | 84 → 10 | 10 | 840 | 10 | 850 |\n**Parameter substitutions, in layer order: **\nLayer 1: (5 x 5 x 1 + 1) x 6 = 156.\nLayer 3: (5 x 5 x 6 + 1) x 16 = 2,416.\nLayer 6: (400 + 1) x 120 = 48,120.\nLayer 7: (120 + 1) x 84 = 10,164.\nLayer 8: (84 + 1) x 10 = 850.\n**Total: ** 156 + 2,416 + 48,120 + 10,164 + 850 = **61,706 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.\nThis is the standard classroom full-channel interpretation with nontrainable pooling and a dense 10-class softmax. The historical original LeNet used partial channel connections and other details, so its published parameter count need not equal this diagram-based count. If an input label is read as 28, the stated later 28 map after valid 5 × 5 convolution is inconsistent; record the 32 input assumption rather than silently ignoring the mismatch.\n"},{"title":"F22 Q4a","question":["Question (10 marks). An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved. Compute operations and explain 1 × 1 convolution.
"],"answer":["An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved.
","Find direct and bottleneck computation counts and the requested saving.
","Multiply–accumulate count. Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.
","H_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.
","![\"\\mathrm{saving\\](\"assets/equations/ea97c94764a3090c.svg\")
Yaad rakhne ka cue: Har output cell ka kaam × kitne output cells.
","Step 2: direct path.
","output stays 28 × 28 with 32 channels.
","Step 3: direct path.
","MACs = 28 × 28 × 192 × 32 × 5 × 5 = 120,422,400.
","Step 4: bottleneck path with 16 reduction channels.
","1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs.
","Step 5: bottleneck path with 16 reduction channels.
","The 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs.
","Step 6: bottleneck path with 16 reduction channels.
","Total = 12,443,648.
","Step 7: saving.
","120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = 89.666667%. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = 777,728r
","Step 8: saving.
","there is no unique numeric bottleneck cost until r is chosen.
","Step 9: calculate this quantity.
","An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","Step 10: calculate this quantity.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
"],"kind":"practice","id":"mt2-c15-s5","search":"F22 Q4a **Question (10 marks).** An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved. Compute operations and explain 1 × 1 convolution.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nAn Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved.\n\n### To find\n\nFind direct and bottleneck computation counts and the requested saving.\n\n### Formula — which one and why\n\n**Multiply–accumulate count.** Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.\n\nH_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells.\n\n### Solution — step by step\n\n\n\n\n\n\n\n\n\n\n### Direct path\n\n**Step 2: direct path.**\n\noutput stays 28 × 28 with 32 channels.\n\n**Step 3: direct path.**\n\nMACs = 28 × 28 × 192 × 32 × 5 × 5 = **120,422,400**.\n\n### Bottleneck path with 16 reduction channels\n\n**Step 4: bottleneck path with 16 reduction channels.**\n\n1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs.\n\n**Step 5: bottleneck path with 16 reduction channels.**\n\nThe 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs.\n\n**Step 6: bottleneck path with 16 reduction channels.**\n\nTotal = **12,443,648**.\n\n### Saving\n\n**Step 7: saving.**\n\n120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = **89.666667%**. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = **777,728r**\n\n**Step 8: saving.**\n\nthere is no unique numeric bottleneck cost until r is chosen.\n\n**Step 9: calculate this quantity.**\n\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\n\n\n**Step 10: calculate this quantity.**\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\n\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.\n"},{"title":"F22 Q4b","question":["Question (10 marks). Draw a 6-input, two-hidden-layer network with 5 neurons each and 4 classes, trained by AdaGrad. Identify learned parameters and tunable hyperparameters; no input-layer weights or biases.
"],"answer":["6 inputs → hidden 5 → hidden 5 → 4 output classes.
","AdaGrad is the specified training algorithm; one bias per receiving neuron.
","Draw the architecture and count its learned weights/biases. The optimizer name does not change the count.
","Dense-layer parameters. Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.
","n_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.
","Yaad rakhne ka cue: Har output ke liye saare inputs plus ek bias.
","Architecture: 6 → 5 → 5 → 4. Each non-input neuron has one bias.
","Step 1: count connections.
","![\"6(5)=30\"](\"assets/equations/9aa0cefe0fe7172a.svg\")
Step 2: count receiving-neuron biases.
","There are 5 receiving neurons, so 5 biases.
","Step 3: add weights and biases.
","![\"30+5=35\"](\"assets/equations/e27ba4eb6d98f55b.svg\")
Step 1: count connections.
","![\"5(5)=25\"](\"assets/equations/d16b109d87f9088b.svg\")
Step 2: count receiving-neuron biases.
","There are 5 receiving neurons, so 5 biases.
","Step 3: add weights and biases.
","![\"25+5=30\"](\"assets/equations/862c71dcbfedc68d.svg\")
Step 1: count connections.
","![\"5(4)=20\"](\"assets/equations/b06f6eedc0b0c61f.svg\")
Step 2: count receiving-neuron biases.
","There are 4 receiving neurons, so 4 biases.
","Step 3: add weights and biases.
","![\"20+4=24\"](\"assets/equations/92df7685dd83d921.svg\")
Total = 89. Input values and activations are not trainable parameters.
","The learned parameters are input-to-hidden, hidden-to-hidden and hidden-to-output weights plus biases at receiving neurons. AdaGrad accumulators are optimizer state, not additional predictive weights. Hyperparameters include hidden widths/depth, learning rate η,ε, batch size, epochs, activations and regularization settings. Use 4-class softmax with categorical cross-entropy.
","AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:
","Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.
","Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.
","Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.
"],"kind":"practice","id":"mt2-c15-s6","search":"F22 Q4b **Question (10 marks).** Draw a 6-input, two-hidden-layer network with 5 neurons each and 4 classes, trained by AdaGrad. Identify learned parameters and tunable hyperparameters; no input-layer weights or biases.\n\n**Complete worked answer.**\n\n\n\n### Given\n\n6 inputs → hidden 5 → hidden 5 → 4 output classes.\n\nAdaGrad is the specified training algorithm; one bias per receiving neuron.\n\n### To find\n\nDraw the architecture and count its learned weights/biases. The optimizer name does not change the count.\n\n### Formula — which one and why\n\n**Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.\n\nn_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.\n\n\n\n**Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias.\n\n### Solution — step by step\n\n\n\n\nArchitecture: 6 → 5 → 5 → 4. Each non-input neuron has one bias.\n\n### Layer 6 → 5\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 5 receiving neurons, so 5 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 5 → 5\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 5 receiving neurons, so 5 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 5 → 4\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 4 receiving neurons, so 4 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n**Total = 89**. Input values and activations are not trainable parameters.\nThe learned parameters are input-to-hidden, hidden-to-hidden and hidden-to-output weights plus biases at receiving neurons. AdaGrad accumulators are optimizer state, not additional predictive weights. Hyperparameters include hidden widths/depth, learning rate η,ε, batch size, epochs, activations and regularization settings. Use 4-class softmax with categorical cross-entropy.\nAdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:\n\nFrequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.\nExample with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.\nAdvantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.\n"},{"title":"F22 Q5b","question":["Question (10 marks). Explain DL optimization challenges and the advantage of momentum over plain mini-batch gradient descent.
"],"answer":["The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.
","Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.
","Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.
","Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.
","Momentum uses vnew = βv−ηg and θ new = θ+vnew; consistent gradients accumulate direction, whereas some alternating components cancel. Plain mini-batch descent uses only the current batch gradient. Both can be combined: momentum with mini-batch gradients is a standard coherent choice.
"],"kind":"practice","id":"mt2-c15-s7","search":"F22 Q5b **Question (10 marks).** Explain DL optimization challenges and the advantage of momentum over plain mini-batch gradient descent.\n\n**Complete worked answer.**\nThe learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.\n\nBatch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.\nMomentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.\nDiagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.\nMomentum uses vnew = βv−ηg and θ new = θ+vnew; consistent gradients accumulate direction, whereas some alternating components cancel. Plain mini-batch descent uses only the current batch gradient. Both can be combined: momentum with mini-batch gradients is a standard coherent choice.\n"},{"title":"F22 Q6a","question":["Question (10 marks). A CNN image classifier performs poorly because of vanishing gradients. Explain remedies.
"],"answer":["A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.
","The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.
","Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].
","Also inspect saturated sigmoid/tanh hidden units, initialization and gradient magnitudes. Appropriate ReLU-family activations and initialization can help preserve usable gradients. Residual blocks require shape-compatible additions or projections. Verify that the diagnosis is correct; additional training time alone cannot repair gradients that are consistently negligible.
"],"kind":"practice","id":"mt2-c15-s8","search":"F22 Q6a **Question (10 marks).** A CNN image classifier performs poorly because of vanishing gradients. Explain remedies.\n\n**Complete worked answer.**\nA residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.\n\n\nThe identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.\nElementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].\nAlso inspect saturated sigmoid/tanh hidden units, initialization and gradient magnitudes. Appropriate ReLU-family activations and initialization can help preserve usable gradients. Residual blocks require shape-compatible additions or projections. Verify that the diagnosis is correct; additional training time alone cannot repair gradients that are consistently negligible.\n"},{"title":"F22 Q6b","question":["Question (10 marks). Design a clothes classifier for 250,000 six-dimensional samples,6 classes and 2 hidden layers. Give train/validation/test split, activations/loss, actions for 15% training 20% validation error, interpret improvement to 1%/1.5%, then address 15% test error.
"],"answer":["One explicit design: 6 → 16 → 8 → 6, ReLU hidden layers,6-unit softmax output, categorical cross-entropy. The hidden widths are chosen assumptions, not supplied numerical requirements.
","Architecture: 6 → 16 → 8 → 6. Each non-input neuron has one bias.
","Step 1: count connections.
","![\"6(16)=96\"](\"assets/equations/35a3fea4574170f0.svg\")
Step 2: count receiving-neuron biases.
","There are 16 receiving neurons, so 16 biases.
","Step 3: add weights and biases.
","![\"96+16=112\"](\"assets/equations/77dd73646865b98e.svg\")
Step 1: count connections.
","![\"16(8)=128\"](\"assets/equations/3a81b7540b48b829.svg\")
Step 2: count receiving-neuron biases.
","There are 8 receiving neurons, so 8 biases.
","Step 3: add weights and biases.
","![\"128+8=136\"](\"assets/equations/c4ed13ec1c4089a9.svg\")
Step 1: count connections.
","![\"8(6)=48\"](\"assets/equations/42c41d8c1f1be799.svg\")
Step 2: count receiving-neuron biases.
","There are 6 receiving neurons, so 6 biases.
","Step 3: add weights and biases.
","![\"48+6=54\"](\"assets/equations/935190190dc4ddae.svg\")
Total = 302. Input values and activations are not trainable parameters.
","Split: a stratified 70/15/15 split gives 175,000 training,37,500 validation,37,500 test samples. Fit preprocessing on training data only and keep related duplicates within a single split.
","15%/20%: inspect learning curves and baseline difficulty. Substantial training error suggests capacity/optimization or data-quality issues; the 5 point gap also suggests a generalization issue. Improve training fit, tune regularization and collect useful representative data instead of blindly increasing capacity.
","1%/1.5%: good comparable-set results with a small gap are encouraging; verify that validation has not leaked into training.
","15% test: inspect distribution shift, preprocessing differences, label quality, leakage or repeated validation overfitting. Compare per-class errors and collection conditions. Address the actual cause and collect representative data if necessary. If test results guide retuning, use a fresh untouched final test set.
"],"kind":"practice","id":"mt2-c15-s9","search":"F22 Q6b **Question (10 marks).** Design a clothes classifier for 250,000 six-dimensional samples,6 classes and 2 hidden layers. Give train/validation/test split, activations/loss, actions for 15% training 20% validation error, interpret improvement to 1%/1.5%, then address 15% test error.\n\n**Complete worked answer.**\n**One explicit design: **6 → 16 → 8 → 6, ReLU hidden layers,6-unit softmax output, categorical cross-entropy. The hidden widths are chosen assumptions, not supplied numerical requirements.\nArchitecture: 6 → 16 → 8 → 6. Each non-input neuron has one bias.\n### Layer 6 → 16\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 16 receiving neurons, so 16 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 16 → 8\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 8 receiving neurons, so 8 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 8 → 6\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 6 receiving neurons, so 6 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n**Total = 302**. Input values and activations are not trainable parameters.\n**Split: ** a stratified 70/15/15 split gives 175,000 training,37,500 validation,37,500 test samples. Fit preprocessing on training data only and keep related duplicates within a single split.\n**15%/20%: ** inspect learning curves and baseline difficulty. Substantial training error suggests capacity/optimization or data-quality issues; the 5 point gap also suggests a generalization issue. Improve training fit, tune regularization and collect useful representative data instead of blindly increasing capacity.\n**1%/1.5%: ** good comparable-set results with a small gap are encouraging; verify that validation has not leaked into training.\n**15% test: ** inspect distribution shift, preprocessing differences, label quality, leakage or repeated validation overfitting. Compare per-class errors and collection conditions. Address the actual cause and collect representative data if necessary. If test results guide retuning, use a fresh untouched final test set.\n\n"}]},{"n":16,"title":"R23 selected MT2 practice","sections":[{"title":"Start here","blocks":["The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper.
"],"kind":"read","id":"mt2-c16-intro"},{"title":"R23 Q1a","question":["Question (4 marks). Suggest and sketch an RNN architecture to synthesize a musical tune.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","Represent notes as discrete tokens or suitable pitch-duration vectors. Train a recurrent decoder to predict the next note from preceding notes. A vocabulary softmax handles discrete note tokens; repeatedly feed generated notes to continue the tune until a stop condition. This is autoregressive sequence generation, typically a many-to-many next-token training pattern.
","Question (4 marks). Apply 3 × 3 average pooling with stride 2 to Matrix 1 shown below. Give the output and justify pooling.
","Apply 3 × 3 average pooling with stride 2 to Matrix 1 shown below.
","Find the requested pooled output(s).
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","Shape: ![\"\\left\\lfloor\\frac{5-3}{2}\\right\\rfloor+1\"](\"assets/equations/cc7bf1ef4d7c44f8.svg\")
= 2 on each axis; channels stay unchanged.
Step 1: select the entries.
","[1, 2, 3, 5, 3, 4, 7, 4, 8]
","Step 2: add all selected entries.
","1 + 2 + 3 + 5 + 3 + 4 + 7 + 4 + 8 = 37
","Step 3: divide by the 9 entries in this window.
","37/9 = 4.111111111
","Step 1: select the entries.
","[3, 2, 5, 4, 6, 3, 8, 5, 9]
","Step 2: add all selected entries.
","3 + 2 + 5 + 4 + 6 + 3 + 8 + 5 + 9 = 45
","Step 3: divide by the 9 entries in this window.
","45/9 = 5
","Step 1: select the entries.
","[7, 4, 8, 8, 5, 10, 9, 10, 3]
","Step 2: add all selected entries.
","7 + 4 + 8 + 8 + 5 + 10 + 9 + 10 + 3 = 64
","Step 3: divide by the 9 entries in this window.
","64/9 = 7.111111111
","Step 1: select the entries.
","[8, 5, 9, 10, 4, 2, 3, 4, 3]
","Step 2: add all selected entries.
","8 + 5 + 9 + 10 + 4 + 2 + 3 + 4 + 3 = 48
","Step 3: divide by the 9 entries in this window.
","48/9 = 5.333333333
","Average output: ![\"\\begin{bmatrix}4.111111111](\"assets/equations/072844f2570c1168.svg\")
. Pooling normally has zero trainable parameters and reduces spatial dimensions.
Question (4 marks). Evaluate how mini-batch size affects optimization and parallel processing.
"],"answer":["A mini-batch forms one gradient from B examples. Smaller B usually gives noisier gradients, lower activation memory and more updates per epoch. Larger B averages more examples, often uses vectorized hardware more efficiently up to resource limits, but needs more memory and gives fewer updates per epoch. N examples produce ceil(N/B) updates if the last partial batch is kept. Too-large B does not guarantee faster wall-clock progress or better generalization. Tune batch size with learning rate and resource constraints. Computation within a batch can be parallelized; successive optimizer steps remain dependent on the updated parameters.
"],"kind":"practice","id":"mt2-c16-s2","search":"R23 Q1d **Question (4 marks).** Evaluate how mini-batch size affects optimization and parallel processing.\n\n**Complete worked answer.**\nA mini-batch forms one gradient from B examples. Smaller B usually gives noisier gradients, lower activation memory and more updates per epoch. Larger B averages more examples, often uses vectorized hardware more efficiently up to resource limits, but needs more memory and gives fewer updates per epoch. N examples produce ceil(N/B) updates if the last partial batch is kept. Too-large B does not guarantee faster wall-clock progress or better generalization. Tune batch size with learning rate and resource constraints. Computation within a batch can be parallelized; successive optimizer steps remain dependent on the updated parameters.\n"},{"title":"R23 Q2a","question":["Question (10 marks). A CNN with convolution, pooling, dense and output layers receives 127 × 127 images for 5 classes and trains with Adam. Identify learned parameters and tunable hyperparameters.
"],"answer":["Convolutional kernel entries and filter biases are learned parameters. Dense weight matrices and biases are learned parameters. Pooling, flattening and ordinary activations have no learned weights. The output has 5 class scores and usually softmax with categorical cross-entropy. Adam’s m, v are running optimizer states, not extra predictive parameters.
","Hyperparameters include kernel sizes, filter counts, strides, padding, pooling configuration, dense widths, activations, learning rate, Adam β₁/β₂/ε, batch size, epochs and regularization. Architecture choices control representation capacity and resource demand; optimizer settings control update behavior. No filter counts or dense widths are specified, so a unique numerical parameter total cannot be calculated from 127 × 127 and 5 classes alone.
","Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.
","Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.
","For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.
","Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.
"],"kind":"practice","id":"mt2-c16-s3","search":"R23 Q2a **Question (10 marks).** A CNN with convolution, pooling, dense and output layers receives 127 × 127 images for 5 classes and trains with Adam. Identify learned parameters and tunable hyperparameters.\n\n**Complete worked answer.**\nConvolutional kernel entries and filter biases are learned parameters. Dense weight matrices and biases are learned parameters. Pooling, flattening and ordinary activations have no learned weights. The output has 5 class scores and usually softmax with categorical cross-entropy. Adam’s m, v are running optimizer states, not extra predictive parameters.\nHyperparameters include kernel sizes, filter counts, strides, padding, pooling configuration, dense widths, activations, learning rate, Adam β₁/β₂/ε, batch size, epochs and regularization. Architecture choices control representation capacity and resource demand; optimizer settings control update behavior. No filter counts or dense widths are specified, so a unique numerical parameter total cannot be calculated from 127 × 127 and 5 classes alone.\nAdam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.\n\n\nUse update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.\nFor g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.\nCompared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.\n"},{"title":"R23 Q2b","question":["Question (10 marks). Justify ReLU over sigmoid/tanh for deep hidden layers. For predicting brand ratings on a scale of 10, choose output activation and loss. Diagnose 2% training versus 18% validation error and suggest remedies.
"],"answer":["An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.
","Sigmoid
","![\"\\sigma(z)=\\frac{1}{1+e^{-z}},\\qquad](\"assets/equations/36359a5421bdb1af.svg\")
Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.
","Hyperbolic tangent
","![\"\\tanh(z)=\\frac{e^z-e^{-z}}{e^z+e^{-z}},\\qquad](\"assets/equations/369af2e85f2204d4.svg\")
The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.
","ReLU
","![\"\\operatorname{ReLU}(z)=\\max(0,z)=\\begin{cases}z&z\\geq0\\\\0&z<0\\end{cases}\"](\"assets/equations/f3a2f94af1e35f2f.svg\")
The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.
","Leaky ReLU
","![\"\\operatorname{LeakyReLU}(z)=\\begin{cases}z&z\\geq0\\\\\\alpha](\"assets/equations/fa7213ea51b43af7.svg\")
Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.
","Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.
","Rating regression: for a continuous score, one linear output with MSE is a standard baseline. If strict 0–10 bounds are required, use 10 × sigmoid(z) and the matching regression loss, noting saturation near the endpoints. If labels are categorical rating bins, a categorical formulation would be different; state the continuous-score assumption.
","2% versus 18%: if errors use the same comparable metric and distribution, this large generalization gap strongly suggests overfitting. Use appropriate regularization, early stopping, more representative data and possibly a smaller model; inspect leakage and preprocessing too.
","Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
"],"kind":"practice","id":"mt2-c16-s4","search":"R23 Q2b **Question (10 marks).** Justify ReLU over sigmoid/tanh for deep hidden layers. For predicting brand ratings on a scale of 10, choose output activation and loss. Diagnose 2% training versus 18% validation error and suggest remedies.\n\n**Complete worked answer.**\nAn activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.\n**Sigmoid**\n\nHere z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.\n**Hyperbolic tangent**\n\nThe numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.\n**ReLU**\n\nThe output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.\n**Leaky ReLU**\n\nAlpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.\nSigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.\n**Rating regression: ** for a continuous score, one linear output with MSE is a standard baseline. If strict 0–10 bounds are required, use 10 × sigmoid(z) and the matching regression loss, noting saturation near the endpoints. If labels are categorical rating bins, a categorical formulation would be different; state the continuous-score assumption.\n**2% versus 18%: ** if errors use the same comparable metric and distribution, this large generalization gap strongly suggests overfitting. Use appropriate regularization, early stopping, more representative data and possibly a smaller model; inspect leakage and preprocessing too.\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of u"},{"title":"R23 Q4a","question":["Question (10 marks). Design employee face-recognition attendance using a CNN with too few training images. Explain solutions.
"],"answer":["Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.
","1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.
","2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.
","3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.
","4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.
","5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.
","6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.
","Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.
"],"kind":"practice","id":"mt2-c16-s5","search":"R23 Q4a **Question (10 marks).** Design employee face-recognition attendance using a CNN with too few training images. Explain solutions.\n\n**Complete worked answer.**\nTransfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.\n1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.\n2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.\n3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.\n4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.\n5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.\n6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.\nFeature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.\n"},{"title":"R23 Q4b","question":["Question (10 marks). Estimate parameters of the AlexNet diagram. Use Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units.
"],"answer":["Use Input 227 × 227 × 3.
","Conv K = 11, filters = 96, stride = 4, padding = 0.
","Pool K = 3, stride = 2, padding = 0.
","Conv K = 5, filters = 256, stride = 1, padding = 2.
","Pool K = 3, stride = 2, padding = 0.
","Conv K = 3, filters = 384, stride = 1, padding = 1.
","Conv K = 3, filters = 384, stride = 1, padding = 1.
","Conv K = 3, filters = 256, stride = 1, padding = 1.
","Pool K = 3, stride = 2, padding = 0.
","Dense 4096 units.
","Dense 4096 units.
","Dense 1000 units.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 227 × 227 × 3. Use 96 filters of spatial size 11 × 11, stride 4, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 55 × 55 × 96.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 55 × 55 × 96. Use a 3 × 3 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 27 × 27 × 96.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 27 × 27 × 96. Use 256 filters of spatial size 5 × 5, stride 1, and padding 2 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 27 × 27 × 256.
","3. Count trainable values. Each filter receives all 96 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 27 × 27 × 256. Use a 3 × 3 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 13 × 13 × 256.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 256. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.
","3. Count trainable values. Each filter receives all 256 input channels. Its weights are reused at every spatial position.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 384. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.
","3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 384. Use 256 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 256.
","3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 13 × 13 × 256. Use a 3 × 3 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 6 × 6 × 256.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 6 × 6 × 256. Rearrange all activation values into one vector.
","Output: a vector with 9,216 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 9,216 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 4,096 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 4,096 features. Output: 1,000 neurons. Every output neuron connects to every input feature and has one bias.
","| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | | 55 x 55 x 96 | 34,848 | 96 | 34,944 |
| 2 pool | | 27 x 27 x 96 | 0 | 0 | 0 |
| 3 conv | | 27 x 27 x 256 | 614,400 | 256 | 614,656 |
| 4 pool | | 13 x 13 x 256 | 0 | 0 | 0 |
| 5 conv | | 13 x 13 x 384 | 884,736 | 384 | 885,120 |
| 6 conv | | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 |
| 7 conv | | 13 x 13 x 256 | 884,736 | 256 | 884,992 |
| 8 pool | | 6 x 6 x 256 | 0 | 0 | 0 |
| 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 |
| 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 |
| 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 |
| 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 |
Parameter substitutions, in layer order:
","Layer 1: (11 x 11 x 3 + 1) x 96 = 34,944.
","Layer 3: (5 x 5 x 96 + 1) x 256 = 614,656.
","Layer 5: (3 x 3 x 256 + 1) x 384 = 885,120.
","Layer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488.
","Layer 7: (3 x 3 x 384 + 1) x 256 = 884,992.
","Layer 10: (9216 + 1) x 4096 = 37,752,832.
","Layer 11: (4096 + 1) x 4096 = 16,781,312.
","Layer 12: (4096 + 1) x 1000 = 4,097,000.
","Total: 34,944 + 614,656 + 885,120 + 1,327,488 + 884,992 + 37,752,832 + 16,781,312 + 4,097,000 = 62,378,344 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","The diagram labels its final pooled map 6 × 6 × 256; a 3 × 3, stride 2 pool from 13 × 13 produces 6. If a small printed stride label is read as 3, it conflicts with the shown 6 × 6 map. The answer uses the consistent classic stride 2 interpretation. Original grouped AlexNet counts are a different architecture assumption.
"],"kind":"practice","id":"mt2-c16-s6","search":"R23 Q4b **Question (10 marks).** Estimate parameters of the AlexNet diagram. Use Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nUse Input 227 × 227 × 3.\n\nConv K = 11, filters = 96, stride = 4, padding = 0.\n\nPool K = 3, stride = 2, padding = 0.\n\nConv K = 5, filters = 256, stride = 1, padding = 2.\n\nPool K = 3, stride = 2, padding = 0.\n\nConv K = 3, filters = 384, stride = 1, padding = 1.\n\nConv K = 3, filters = 384, stride = 1, padding = 1.\n\nConv K = 3, filters = 256, stride = 1, padding = 1.\n\nPool K = 3, stride = 2, padding = 0.\n\nDense 4096 units.\n\nDense 4096 units.\n\nDense 1000 units.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{227+2 \\times 0-11}{4}\\right\\rfloor+1\\) | 55 x 55 x 96 | 34,848 | 96 | 34,944 |\n| 2 pool | \\(\\left\\lfloor\\frac{55-3}{2}\\right\\rfloor+1\\) | 27 x 27 x 96 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{27+2 \\times 2-5}{1}\\right\\rfloor+1\\) | 27 x 27 x 256 | 614,400 | 256 | 614,656 |\n| 4 pool | \\(\\left\\lfloor\\frac{27-3}{2}\\right\\rfloor+1\\) | 13 x 13 x 256 | 0 | 0 | 0 |\n| 5 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 384 | 884,736 | 384 | 885,120 |\n| 6 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 |\n| 7 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 256 | 884,736 | 256 | 884,992 |\n| 8 pool | \\(\\left\\lfloor\\frac{13-3}{2}\\right\\rfloor+1\\) | 6 x 6 x 256 | 0 | 0 | 0 |\n| 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 |\n| 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 |\n| 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 |\n| 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 |\n**Parameter substitutions, in layer order: **\nLayer 1: (11 x 11 x 3 + 1) x 96 = 34,944.\nLayer 3: (5 x 5 x 96 + 1) x 256 = 614,656.\nLayer 5: (3 x 3 x 256 + 1) x 384 = 885,120.\nLayer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488.\nLayer 7: (3 x 3 x 384 + 1) x 256 = 884,992.\nLayer 10: (9216 + 1) x 4096 = 3"},{"title":"R23 Q6a","question":["Question (10 marks). Suggest and justify remedies when (1) loss does not minimize after many epochs, (2) gradients are noisy and loss oscillates, (3) loss falls then rises because updates miss a minimum.
"],"answer":["The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.
","Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.
","Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.
","Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.
","Scenario 1: check correctness and gradient magnitudes, then tune learning rate, activation, initialization or optimizer. Scenario 2: consider an appropriate larger mini-batch or momentum; distinguish ordinary stochastic noise from divergence. Scenario 3: reduce learning rate or use a schedule; momentum/Nesterov can help but excessive momentum can worsen overshoot. Judge training and validation curves separately.
"],"kind":"practice","id":"mt2-c16-s7","search":"R23 Q6a **Question (10 marks).** Suggest and justify remedies when (1) loss does not minimize after many epochs, (2) gradients are noisy and loss oscillates, (3) loss falls then rises because updates miss a minimum.\n\n**Complete worked answer.**\nThe learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.\n\nBatch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.\nMomentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.\nDiagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.\n**Scenario 1: ** check correctness and gradient magnitudes, then tune learning rate, activation, initialization or optimizer. **Scenario 2: ** consider an appropriate larger mini-batch or momentum; distinguish ordinary stochastic noise from divergence. **Scenario 3: ** reduce learning rate or use a schedule; momentum/Nesterov can help but excessive momentum can worsen overshoot. Judge training and validation curves separately.\n"},{"title":"R23 Q6b","question":["Question (10 marks). A 28 × 28 × 192 input is convolved with 32 same-padded 5 × 5 filters to produce 28 × 28 × 32. Count computations. What happens if a 1 × 1 convolution precedes it? The reduction-channel count is not supplied.
"],"answer":["Input = 28 × 28 × 192; 32 same-padded 5 × 5 filters.
","A preceding 1 × 1 convolution would reduce to C channels.
","C is not supplied, so first give the symbolic bottleneck count; 16 channels is only an illustrative assumption.
","Calculate direct cost and express bottleneck cost in C; explain when the bottleneck saves computation.
","Multiply–accumulate count. Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.
","H_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.
","Yaad rakhne ka cue: Har output cell ka kaam × kitne output cells.
","Step 2: direct path.
","output stays 28 × 28 with 32 channels.
","Step 3: direct path.
","MACs = 28 × 28 × 192 × 32 × 5 × 5 = 120,422,400.
","Step 4: bottleneck path with 16 reduction channels.
","1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs.
","Step 5: bottleneck path with 16 reduction channels.
","The 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs.
","Step 6: bottleneck path with 16 reduction channels.
","Total = 12,443,648.
","Step 7: saving.
","120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = 89.666667%. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = 777,728r
","Step 8: saving.
","there is no unique numeric bottleneck cost until r is chosen.
","The 16-channel bottleneck is an explicitly chosen illustrative value, consistent with a related paper; the symbolic 777,728 r answer is required when r has not been specified.
","Step 9: calculate this quantity.
","An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","Step 10: calculate this quantity.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
"],"kind":"practice","id":"mt2-c16-s8","search":"R23 Q6b **Question (10 marks).** A 28 × 28 × 192 input is convolved with 32 same-padded 5 × 5 filters to produce 28 × 28 × 32. Count computations. What happens if a 1 × 1 convolution precedes it? The reduction-channel count is not supplied.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput = 28 × 28 × 192; 32 same-padded 5 × 5 filters.\n\nA preceding 1 × 1 convolution would reduce to C channels.\n\nC is not supplied, so first give the symbolic bottleneck count; 16 channels is only an illustrative assumption.\n\n### To find\n\nCalculate direct cost and express bottleneck cost in C; explain when the bottleneck saves computation.\n\n### Formula — which one and why\n\n**Multiply–accumulate count.** Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.\n\nH_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells.\n\n### Solution — step by step\n\n\n\n\n\n\n\n\n\n\n### Direct path\n\n**Step 2: direct path.**\n\noutput stays 28 × 28 with 32 channels.\n\n**Step 3: direct path.**\n\nMACs = 28 × 28 × 192 × 32 × 5 × 5 = **120,422,400**.\n\n### Bottleneck path with 16 reduction channels\n\n**Step 4: bottleneck path with 16 reduction channels.**\n\n1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs.\n\n**Step 5: bottleneck path with 16 reduction channels.**\n\nThe 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs.\n\n**Step 6: bottleneck path with 16 reduction channels.**\n\nTotal = **12,443,648**.\n\n### Saving\n\n**Step 7: saving.**\n\n120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = **89.666667%**. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = **777,728r**\n\n**Step 8: saving.**\n\nthere is no unique numeric bottleneck cost until r is chosen.\n\nThe 16-channel bottleneck is an explicitly chosen illustrative value, consistent with a related paper; the symbolic 777,728 r answer is required when r has not been specified.\n\n**Step 9: calculate this quantity.**\n\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\n\n\n**Step 10: calculate this quantity.**\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\n\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kern"},{"title":"R23 Q7b","question":["Question (10 marks). Training performance is good but unseen-data performance degrades. Identify the issue and explain ways to improve it.
"],"answer":["Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
","A comparable-distribution train/unseen gap suggests overfitting. Also check whether the new data have shifted, preprocessing differs or labels are inconsistent. Select regularization using validation performance, not final-test tuning. Improvement must be demonstrated on representative held-out examples.
"],"kind":"practice","id":"mt2-c16-s9","search":"R23 Q7b **Question (10 marks).** Training performance is good but unseen-data performance degrades. Identify the issue and explain ways to improve it.\n\n**Complete worked answer.**\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\nA comparable-distribution train/unseen gap suggests overfitting. Also check whether the new data have shifted, preprocessing differs or labels are inconsistent. Select regularization using validation performance, not final-test tuning. Improvement must be demonstrated on representative held-out examples.\n\n\n"}]},{"n":17,"title":"MT2 numerical practice from direct to mixed","sections":[{"title":"Start here","blocks":["These 45 original drills cover the required maths, neuron/gradient prerequisites, optimization, regularization, CNNs and basic RNN forward passes. Each has a full solution. They are practice questions, not predictions.
","First learn the method, then cover the answer and solve. Start with drills 4–6, 11, 14–39, then the mixed drills 40–45. Use the earlier arithmetic drills whenever a sign, bracket or calculator entry blocks you.
"],"kind":"read","id":"mt2-c17-intro"},{"title":"How to solve a mixed numerical without guessing","blocks":["A mixed question is a chain of familiar operations. First circle the requested final quantity. Underline the given activation, loss, padding, stride, threshold, learning rate and any fixed parameters. Draw a small box for each stage. Write the output of one box as the input to the next. Never skip a stage because its title looks unfamiliar.
","If the question says output or prediction: do only the forward chain, unless training is separately requested. For a neuron, multiply and add before activating. For a CNN, determine spatial shapes before flattening. For an RNN, carry the previous state into the next time step.
","If it says train or update: finish the entire old forward pass, calculate the stated loss derivative, move backward, store every gradient, then update together. If there are several output branches, add their effects at the shared hidden unit. For a batch, follow the stated sum or mean loss before updating.
","If it mixes dropout and a loss: apply the given training mask and scaling first; then predict from the resulting activations. The gradient to a dropped path is zero for that mask. If it also specifies L2, add its gradient to the data gradient before the optimizer step. If it also specifies momentum or Adam, carry their stored states as well as the parameter.
","If it mixes convolution, pooling and a dense classifier: output shape → next layer input shape → flatten length → dense scores → activation → loss. Count learned parameters separately from output values. Pooling and flattening have zero learned parameters in these questions.
","If it changes a familiar problem: an extra input adds another product; an extra output adds another backward path; a new channel adds another kernel sum; a new time step adds another shared-state calculation. The method stays the same. The drills below deliberately change signs, targets, loss scales and layer combinations.
","Your completion rule: solve a representative problem today without the answer, repeat it after several hours, then solve a changed-number or mixed version tomorrow. Mark the exact line where your reasoning failed. Correcting that line is what builds reliable recall; reading an open solution does not test it.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c17-s0","search":"How to solve a mixed numerical without guessing A mixed question is a chain of familiar operations. First circle the requested final quantity. Underline the given activation, loss, padding, stride, threshold, learning rate and any fixed parameters. Draw a small box for each stage. Write the output of one box as the input to the next. Never skip a stage because its title looks unfamiliar.\n**If the question says output or prediction:** do only the forward chain, unless training is separately requested. For a neuron, multiply and add before activating. For a CNN, determine spatial shapes before flattening. For an RNN, carry the previous state into the next time step.\n**If it says train or update:** finish the entire old forward pass, calculate the stated loss derivative, move backward, store every gradient, then update together. If there are several output branches, add their effects at the shared hidden unit. For a batch, follow the stated sum or mean loss before updating.\n**If it mixes dropout and a loss:** apply the given training mask and scaling first; then predict from the resulting activations. The gradient to a dropped path is zero for that mask. If it also specifies L2, add its gradient to the data gradient before the optimizer step. If it also specifies momentum or Adam, carry their stored states as well as the parameter.\n**If it mixes convolution, pooling and a dense classifier:** output shape → next layer input shape → flatten length → dense scores → activation → loss. Count learned parameters separately from output values. Pooling and flattening have zero learned parameters in these questions.\n**If it changes a familiar problem:** an extra input adds another product; an extra output adds another backward path; a new channel adds another kernel sum; a new time step adds another shared-state calculation. The method stays the same. The drills below deliberately change signs, targets, loss scales and layer combinations.\n**Your completion rule:** solve a representative problem today without the answer, repeat it after several hours, then solve a changed-number or mixed version tomorrow. Mark the exact line where your reasoning failed. Correcting that line is what builds reliable recall; reading an open solution does not test it.\n"},{"title":"Drill 1 Signs and brackets","question":["Question. Evaluate 2−0.1 × (−3), then (2−0.1) × (−3). Explain why they differ.
"],"answer":["Expressions: 2−0.1×(−3) and (2−0.1)×(−3).
","Evaluate both and explain the effect of grouping.
","Order of operations. Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.
","A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.
","Yaad rakhne ka cue: Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.
","A minus sign can mean subtraction or a negative number. Brackets tell us which part belongs together. These two expressions have different brackets, so they describe different calculations.
","Do brackets first, then multiplication, then addition or subtraction. Subtracting a negative number means adding its positive size.
","Step 1: First expression: multiply before subtracting.
","![\"2-0.1\\times(-3)\"](\"assets/equations/5b11a126e768858f.svg\")
First multiply: a positive number times a negative number is negative.
","![\"0.1\\times(-3)=-0.3\"](\"assets/equations/189f052516957693.svg\")
Put this complete result back into the expression. Keep its negative sign inside brackets.
","![\"2-(-0.3)=2+0.3=2.3\"](\"assets/equations/2d796303b17c0e32.svg\")
Step 2: Second expression: finish the bracket first.
","![\"2-0.1=1.9\"](\"assets/equations/26a467850584ea86.svg\")
Now multiply the whole bracket result by negative 3.
","![\"1.9\\times(-3)=-5.7\"](\"assets/equations/f4e294dd1ab7a8aa.svg\")
Final answer: first expression = 2.3; second expression = −5.7. In a weight update, multiply learning rate by gradient first, then subtract that product from the old weight.
"],"kind":"practice","id":"mt2-c17-s1","search":"Drill 1 Signs and brackets **Question.** Evaluate 2−0.1 × (−3), then (2−0.1) × (−3). Explain why they differ.\n\n**Full solution.**\n\n\n\n### Given\n\nExpressions: 2−0.1×(−3) and (2−0.1)×(−3).\n\n### To find\n\nEvaluate both and explain the effect of grouping.\n\n### Formula — which one and why\n\n**Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.\n\nA minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.\n\n\n\n**Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the calculation\nA minus sign can mean subtraction or a negative number. Brackets tell us which part belongs together. These two expressions have different brackets, so they describe different calculations.\n### Rule used\nDo brackets first, then multiplication, then addition or subtraction. Subtracting a negative number means adding its positive size.\n**Step 1: First expression: multiply before subtracting.**\n\nFirst multiply: a positive number times a negative number is negative.\n\nPut this complete result back into the expression. Keep its negative sign inside brackets.\n\n**Step 2: Second expression: finish the bracket first.**\n\nNow multiply the whole bracket result by negative 3.\n\n**Final answer:** first expression = 2.3; second expression = −5.7. In a weight update, multiply learning rate by gradient first, then subtract that product from the old weight.\n"},{"title":"Drill 2 Half sum versus mean","question":["Question. Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE.
"],"answer":["Predictions = [0.2, 0.7, 0.4].
","Targets = [0, 1, 1]; number of outputs n = 3.
","Find SSE, half SSE, MSE and the prediction gradients requested in the question.
","Squared-error objectives. Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.
","ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.
","Yaad rakhne ka cue: Pehle error square; phir question ke hisaab se sum, half ya mean.
","There are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i.
","![\"\\mathrm{SSE}=\\sum_{i=1}^{n}e_i^2\"](\"assets/equations/5312f21f7180b663.svg\")
![\"L_{\\mathrm{half}}=\\frac12\\sum_{i=1}^{n}e_i^2=\\frac{\\mathrm{SSE}}2\"](\"assets/equations/8370703d0e22f365.svg\")
![\"\\mathrm{MSE}=\\frac1n\\sum_{i=1}^{n}e_i^2=\\frac{\\mathrm{SSE}}n\"](\"assets/equations/bde24be7fa074ec1.svg\")
SSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively.
","| Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual |
|---|---|---|---|---|
| 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 |
| 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 |
| 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 |
Square each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel.
","For half SSE, the denominator is 2:
","![\"L_{\\mathrm{half}}=\\frac{0.49}{2}=0.245\"](\"assets/equations/3d10f476bd1cf13c.svg\")
For MSE, the denominator is the number of pairs, n = 3:
","![\"\\mathrm{MSE}=\\frac{0.49}{3}=0.163333333\\ldots\"](\"assets/equations/d4cd1db7c5158eb6.svg\")
Final answer: SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question.
"],"kind":"practice","id":"mt2-c17-s2","search":"Drill 2 Half sum versus mean \n**Question.** Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE.\n\n**Full solution.**\n\n\n\n### Given\n\nPredictions = [0.2, 0.7, 0.4].\n\nTargets = [0, 1, 1]; number of outputs n = 3.\n\n### To find\n\nFind SSE, half SSE, MSE and the prediction gradients requested in the question.\n\n### Formula — which one and why\n\n**Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.\n\nŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean.\n\n### Solution — step by step\n\n\n\n\n\n### Given values and loss definitions\nThere are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i.\n\n\n\nSSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively.\n\n### Step 1: calculate each residual and its square\n| Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual |\n| 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 |\n| 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 |\n| 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 |\n\n### Step 2: sum the squared residuals\n\nSquare each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel.\n\n### Step 3: apply each requested normalization\nFor half SSE, the denominator is 2:\n\nFor MSE, the denominator is the number of pairs, n = 3:\n\n\n**Final answer:** SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question.\n\n"},{"title":"Drill 3 Natural log and probability","question":["Question. Compute −ln(0.9),−ln(0.1), and ![\"e^{-\\ln](\"assets/equations/b7f5712356b73942.svg\")
.
Probabilities: 0.9 and 0.1.
","Exponential expression: exp(−ln 2).
","Find −ln(0.9), −ln(0.1) and exp(−ln 2).
","Natural logarithms. The natural logarithm reverses the exponential. A negative log turns a small correct-class probability into a larger positive loss.
","ln uses base e ≈ 2.71828. exp(a) means e raised to the power a. Keep the minus outside ln when the question says −ln(p).
","![\"-\\ln](\"assets/equations/45890ca9be189629.svg\")
![\"e^{\\ln](\"assets/equations/04d92df0f222846e.svg\")
![\"e^{-\\ln](\"assets/equations/5953def5a7690c0e.svg\")
Yaad rakhne ka cue: exp banata hai power; ln us power ko wapas nikaalta hai.
","The natural logarithm ln tells us the power of e needed to obtain a number. It reverses the exponential operation. A calculator evaluates ln; it is not ordinary division.
","![\"\\ln(e^a)=a,\\qquad](\"assets/equations/794f58bf694147ad.svg\")
![\"e^{-a}=\\frac1{e^a}\"](\"assets/equations/1ee2a4da74df0c61.svg\")
![\"\\ln(0.9)\\approx-0.105360516\"](\"assets/equations/5deadd8f45d24fdd.svg\")
![\"-\\ln(0.9)=-(-0.105360516)=0.105360516\"](\"assets/equations/0b17db90c0b1eab0.svg\")
![\"\\ln(0.1)\\approx-2.302585093\"](\"assets/equations/5ccbedb0aa22add9.svg\")
![\"-\\ln(0.1)=-(-2.302585093)=2.302585093\"](\"assets/equations/bc081502a0ccb2a9.svg\")
When these are correct-class probabilities, 0.1 represents a worse prediction than 0.9, so its negative-log loss is larger.
","![\"e^{-\\ln2}=\\frac1{e^{\\ln2}}=\\frac12=0.5\"](\"assets/equations/b612f5289fa6fa44.svg\")
Final answers: 0.105360516, 2.302585093 and 0.5. Use the ln key; base-10 log gives different first and second answers.
"],"kind":"practice","id":"mt2-c17-s3","search":"Drill 3 Natural log and probability **Question.** Compute −ln(0.9),−ln(0.1), and \\(e^{-\\ln 2}\\).\n\n**Full solution.**\n\n\n\n### Given\n\nProbabilities: 0.9 and 0.1.\n\nExponential expression: exp(−ln 2).\n\n### To find\n\nFind −ln(0.9), −ln(0.1) and exp(−ln 2).\n\n### Formula — which one and why\n\n**Natural logarithms.** The natural logarithm reverses the exponential. A negative log turns a small correct-class probability into a larger positive loss.\n\nln uses base e ≈ 2.71828. exp(a) means e raised to the power a. Keep the minus outside ln when the question says −ln(p).\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** exp banata hai power; ln us power ko wapas nikaalta hai.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe natural logarithm ln tells us the power of e needed to obtain a number. It reverses the exponential operation. A calculator evaluates ln; it is not ordinary division.\n\n\n### Step 1: calculate the first logarithm, then its minus sign\n\n\n### Step 2: calculate the second logarithm\n\n\nWhen these are correct-class probabilities, 0.1 represents a worse prediction than 0.9, so its negative-log loss is larger.\n### Step 3: simplify the exponential using the inverse rule\n\n**Final answers:** 0.105360516, 2.302585093 and 0.5. Use the ln key; base-10 log gives different first and second answers.\n"},{"title":"Drill 4 Dot product and bias","question":["Question. Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU.
"],"answer":["Inputs x = [2, −1, 0.5].
","Weights w = [0.3, 0.4, −0.2]; bias b = 0.1.
","Activation: ReLU.
","Find the weighted sum z and ReLU output.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero.
","![\"z=w_1x_1+w_2x_2+w_3x_3+b,\\qquad](\"assets/equations/d429a2592c8f926e.svg\")
Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output.
","![\"w_1x_1=0.3\\times2=0.6\"](\"assets/equations/e6092f2a718fe356.svg\")
![\"w_2x_2=0.4\\times(-1)=-0.4\"](\"assets/equations/d39443d0dbfaaa2b.svg\")
![\"w_3x_3=(-0.2)\\times0.5=-0.1\"](\"assets/equations/6f257d4f869d8fa2.svg\")
![\"z=0.6+(-0.4)+(-0.1)+0.1\"](\"assets/equations/72dd56dcc4bb45b6.svg\")
![\"z=0.2-0.1+0.1=0.2\"](\"assets/equations/f3991f5f0c7df172.svg\")
![\"a=\\max(0,0.2)=0.2\"](\"assets/equations/2b62624e146d1934.svg\")
Final answer: weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive.
"],"kind":"practice","id":"mt2-c17-s4","search":"Drill 4 Dot product and bias **Question.** Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU.\n\n**Full solution.**\n\n\n\n### Given\n\nInputs x = [2, −1, 0.5].\n\nWeights w = [0.3, 0.4, −0.2]; bias b = 0.1.\n\nActivation: ReLU.\n\n### To find\n\nFind the weighted sum z and ReLU output.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nA dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero.\n\nHere x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output.\n### Step 1: multiply the matching pairs\n\n\n\n### Step 2: add all products and add bias once\n\n\n### Step 3: apply the activation\n\n**Final answer:** weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive.\n"},{"title":"Drill 5 Matrix rows receive inputs","question":["Question. For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU.
","W = ![\"\\begin{bmatrix}1](\"assets/equations/47c671c1553de484.svg\")
.
Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ.
","Activation: ReLU on each result.
","Find Wx+b and the activated output.
","Row–column multiplication. Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.
","Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.
","Yaad rakhne ka cue: Ek row × poora input column = ek receiving neuron ka answer.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row.
","![\"z=Wx+b,\\qquad](\"assets/equations/aa52f339dd0d5a91.svg\")
![\"W=\\begin{bmatrix}1&2\\\\-1&3\\end{bmatrix},\\quad](\"assets/equations/4e990284b43290d2.svg\")
![\"z_1=(1\\times2)+(2\\times1)+0.5\"](\"assets/equations/eaa341baa6e16b1f.svg\")
![\"z_1=2+2+0.5=4.5\"](\"assets/equations/50386fe91979c412.svg\")
![\"z_2=((-1)\\times2)+(3\\times1)+(-0.5)\"](\"assets/equations/b0b4c769cba96d49.svg\")
![\"z_2=-2+3-0.5=0.5\"](\"assets/equations/8bdba182228536bc.svg\")
![\"a_1=\\max(0,4.5)=4.5,\\qquad](\"assets/equations/43cee11d40484528.svg\")
![\"z=\\begin{bmatrix}4.5\\\\0.5\\end{bmatrix},\\qquad](\"assets/equations/b93dfe6da27f9705.svg\")
The shape is 2 rows by 1 column: one output for each receiving row.
"],"kind":"practice","id":"mt2-c17-s5","search":"Drill 5 Matrix rows receive inputs **Question.** For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU.\n\n\n\n**Full solution.**\n\n\n\n### Given\n\nW = .\n\nInput column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ.\n\nActivation: ReLU on each result.\n\n### To find\n\nFind Wx+b and the activated output.\n\n### Formula — which one and why\n\n**Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.\n\nWⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.\n\n\n\n**Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formula\nEach row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row.\n\n\n### Step 1: calculate receiving neuron 1 from row 1\n\n\n### Step 2: calculate receiving neuron 2 from row 2\n\n\n### Step 3: apply ReLU to each entry\n\n\nThe shape is 2 rows by 1 column: one output for each receiving row.\n"},{"title":"Drill 6 Chain rule on a tiny loss","question":["Question. Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1.
"],"answer":["Model: o = wx+b; loss L = ½(o−y)².
","x = 3; y = 1; w = 0.2; b = 0.1.
","Learning rate η = 0.1.
","Find ∂L/∂w, ∂L/∂b and the updated weight and bias.
","Chain rule for a linear neuron. The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.
","z = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.
","Yaad rakhne ka cue: Ek path mein multiply; alag paths mil rahe hon toh add.
","The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects.
","![\"o=wx+b,\\qquad](\"assets/equations/90379cab01aa101e.svg\")
![\"\\frac{\\partial](\"assets/equations/7ae2da1d297dac40.svg\")
![\"\\frac{\\partial](\"assets/equations/a63ae410089093d5.svg\")
The learning rate η says how much of the gradient to subtract.
","![\"w_{\\mathrm{new}}=w-\\eta\\frac{\\partial](\"assets/equations/9cca82416f4ee903.svg\")
![\"o=(0.2\\times3)+0.1=0.6+0.1=0.7\"](\"assets/equations/7d8d618e9d2cfc7d.svg\")
![\"o-y=0.7-1=-0.3\"](\"assets/equations/f0bd8184760ad600.svg\")
![\"L=\\frac12(-0.3)^2=\\frac12(0.09)=0.045\"](\"assets/equations/21397d642a37130f.svg\")
![\"\\frac{\\partial](\"assets/equations/ed97f79ab4475569.svg\")
![\"\\frac{\\partial](\"assets/equations/80efe080f6cbf4d3.svg\")
![\"w_{\\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29\"](\"assets/equations/187fe0232f317977.svg\")
![\"b_{\\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13\"](\"assets/equations/15129640706302ac.svg\")
Final answer: weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass.
"],"kind":"practice","id":"mt2-c17-s6","search":"Drill 6 Chain rule on a tiny loss **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1.\n\n**Full solution.**\n\n\n\n### Given\n\nModel: o = wx+b; loss L = ½(o−y)².\n\nx = 3; y = 1; w = 0.2; b = 0.1.\n\nLearning rate η = 0.1.\n\n### To find\n\nFind ∂L/∂w, ∂L/∂b and the updated weight and bias.\n\n### Formula — which one and why\n\n**Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.\n\nz = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.\n\n\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects.\n\n\n\nThe learning rate η says how much of the gradient to subtract.\n\n### Step 1: obtain the current prediction and error\n\n\n\n### Step 2: calculate both gradients from this same prediction\n\n\n### Step 3: update the weight\n\n### Step 4: update the bias\n\n**Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass.\n"},{"title":"Drill 7 Sigmoid with binary cross entropy","question":["Question. A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z.
"],"answer":["Logit z = 1; binary target y = 1.
","Output: sigmoid; loss: binary cross-entropy.
","Find probability, BCE and the loss derivative with respect to z.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Binary cross-entropy. For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.
","y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.
","![\"\\frac{\\partial](\"assets/equations/2bbef5a5d1c7047f.svg\")
Yaad rakhne ka cue: Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).
","A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome.
","![\"p=\\frac1{1+e^{-z}}\"](\"assets/equations/06236932ea7e53d0.svg\")
![\"L=-\\left[y\\ln](\"assets/equations/e6b608fdfba7ba2f.svg\")
For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form:
","![\"1+e^{-1}\\approx1.367879441\"](\"assets/equations/f1586fb666c5061e.svg\")
![\"p=\\frac1{1.367879441}\\approx0.731058579\"](\"assets/equations/2f55393e129c6628.svg\")
![\"L=-\\left[1\\ln(0.731058579)+(1-1)\\ln(1-0.731058579)\\right]\"](\"assets/equations/fca94e6b25348fdc.svg\")
The second term has multiplier zero, so it contributes zero.
","![\"L=-\\ln(0.731058579)\\approx0.313261688\"](\"assets/equations/f41a1bed5052539d.svg\")
![\"\\frac{\\partial](\"assets/equations/d503567052ec89af.svg\")
Final answer: probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice.
"],"kind":"practice","id":"mt2-c17-s7","search":"Drill 7 Sigmoid with binary cross entropy **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z.\n\n**Full solution.**\n\n\n\n### Given\n\nLogit z = 1; binary target y = 1.\n\nOutput: sigmoid; loss: binary cross-entropy.\n\n### To find\n\nFind probability, BCE and the loss derivative with respect to z.\n\n### Formula — which one and why\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.\n\ny is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.\n\n\n\n\n\n**Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nA logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome.\n\n\nFor sigmoid combined with this loss, the gradient with respect to the logit has the simplified form:\n\n### Step 1: convert logit 1 into probability\n\n\n\n### Step 2: substitute the target in the full loss formula\n\nThe second term has multiplier zero, so it contributes zero.\n\n### Step 3: calculate the combined gradient\n\n**Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice.\n"},{"title":"Drill 8 Activation contrast","question":["Question. For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0.
"],"answer":["Inputs z = [−3,0,2].
","Leaky-ReLU negative slope α = 0.1.
","For the sigmoid derivative, use z = 0.
","Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","ReLU compares the input with zero and returns the larger value.
","Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.
","![\"\\max(0,-3)=0,\\quad](\"assets/equations/3202f6e374a86c4c.svg\")
ReLU output = [0, 0, 2].
","The zero and positive entries remain 0 and 2. Leaky ReLU output = [−0.3, 0, 2].
","![\"\\sigma(z)=\\frac1{1+e^{-z}},\\qquad](\"assets/equations/7727ce70c548ec20.svg\")
![\"\\sigma(0)=\\frac1{1+e^0}=\\frac1{1+1}=0.5\"](\"assets/equations/1ef72ba6a1deae0f.svg\")
![\"\\sigma'(0)=0.5(1-0.5)=0.5\\times0.5=0.25\"](\"assets/equations/c6f6bc13772ea829.svg\")
The sigmoid value is 0.5 but its slope is 0.25. They are different quantities.
"],"kind":"practice","id":"mt2-c17-s8","search":"Drill 8 Activation contrast **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0.\n\n**Full solution.**\n\n\n\n### Given\n\nInputs z = [−3,0,2].\n\nLeaky-ReLU negative slope α = 0.1.\n\nFor the sigmoid derivative, use z = 0.\n\n### To find\n\nFind ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.\n\n### Formula — which one and why\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n### Solution — step by step\n\n\n\n\n\n### Formulas and their meanings\nReLU compares the input with zero and returns the larger value.\n\nLeaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.\n\n### Step 1: apply ReLU separately to all three inputs\n\n**ReLU output = [0, 0, 2].**\n### Step 2: use α = 0.1 only on the negative input\n\nThe zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].**\n### Step 3: calculate sigmoid before its derivative\n\n\n\nThe sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities.\n"},{"title":"Drill 9 Softmax and true class loss","question":["Question. Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy.
"],"answer":["Logits = [1,2,0].
","One-hot target = [0,1,0]: class 2 is correct.
","Find all three softmax probabilities and CCE.
","Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.
","zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.
","Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.
","Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.
","yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.
","Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.
","The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities.
","![\"p_i=\\frac{e^{z_i-c}}{\\sum_j](\"assets/equations/dd0c8ddf71d7e5e7.svg\")
Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials.
","![\"L_{\\mathrm{CCE}}=-\\sum_i](\"assets/equations/6a21d413c89b5a80.svg\")
The largest of [1,2,0] is 2.
","![\"[1-2,\\;2-2,\\;0-2]=[-1,0,-2]\"](\"assets/equations/72ceba57b734c648.svg\")
![\"e^{-1}=0.367879441,\\quad](\"assets/equations/b2c277f81e3d9881.svg\")
![\"D=0.367879441+1+0.135335283=1.503214724\"](\"assets/equations/dc4d03dcae0355c5.svg\")
![\"p_1=\\frac{0.367879441}{1.503214724}=0.244728471\"](\"assets/equations/aa686bb14848b0d1.svg\")
![\"p_2=\\frac{1}{1.503214724}=0.665240956\"](\"assets/equations/361efa91829285c7.svg\")
![\"p_3=\\frac{0.135335283}{1.503214724}=0.090030573\"](\"assets/equations/54c91e5ed1921172.svg\")
The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability.
","![\"L=-[0\\ln](\"assets/equations/cc77914badc24790.svg\")
![\"L=-\\ln(0.665240956)=0.407605964\"](\"assets/equations/65db0db7f14d7ab8.svg\")
Final answer: probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606.
"],"kind":"practice","id":"mt2-c17-s9","search":"Drill 9 Softmax and true class loss **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy.\n\n**Full solution.**\n\n\n\n### Given\n\nLogits = [1,2,0].\n\nOne-hot target = [0,1,0]: class 2 is correct.\n\n### To find\n\nFind all three softmax probabilities and CCE.\n\n### Formula — which one and why\n\n**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.\n\nzⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.\n\n**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.\n\nyⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.\n\n\n\n\n\n**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities.\n\nSubtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials.\n\n### Step 1: subtract the largest score\nThe largest of [1,2,0] is 2.\n\n### Step 2: calculate the three exponentials\n\n### Step 3: add them to get one common denominator\n\n### Step 4: divide each exponential by this denominator\n\n\n\nThe probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability.\n### Step 5: use the probability of the correct class in CCE\n\n\n**Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606.\n"},{"title":"Drill 10 Sigmoid neuron gradient","question":["Question. x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update.
"],"answer":["x = 2; w = 0.5; b = 0; target y = 1.
","Activation: sigmoid; loss: half squared error.
","Learning rate η = 0.1.
","Find the output, delta, weight/bias gradients and one update.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly.
","![\"z=wx+b,\\quad](\"assets/equations/72569e3d1cf18215.svg\")
The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z.
","![\"\\delta=(p-y)p(1-p)\"](\"assets/equations/cf8f40120fc7de9a.svg\")
![\"\\frac{\\partial](\"assets/equations/6678df97903ca342.svg\")
![\"w_{\\mathrm{new}}=w-\\eta\\delta](\"assets/equations/9e125df8665bd715.svg\")
![\"z=0.5\\times2+0=1\"](\"assets/equations/88322b08817273b7.svg\")
![\"p=\\frac1{1+e^{-1}}=0.731058579\"](\"assets/equations/496f72f75e4bdba9.svg\")
![\"p-y=0.731058579-1=-0.268941421\"](\"assets/equations/95ba7a4e62bbaee0.svg\")
![\"p(1-p)=0.731058579\\times0.268941421=0.196611933\"](\"assets/equations/fe81a8ae8e93cc3b.svg\")
![\"\\delta=(-0.268941421)\\times0.196611933=-0.052877093\"](\"assets/equations/ca75a0d20742f652.svg\")
The weight receives x = 2. The bias effectively receives the constant input 1.
","![\"\\frac{\\partial](\"assets/equations/909d6c4ba3b4dba5.svg\")
![\"\\frac{\\partial](\"assets/equations/385f9ff4016d4aa5.svg\")
![\"w_{\\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419\"](\"assets/equations/48c8e5f15b7994a8.svg\")
![\"b_{\\mathrm{new}}=0-0.1(-0.052877093)=0.005287709\"](\"assets/equations/b7af577aaec3d7cf.svg\")
Final answer: new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here.
"],"kind":"practice","id":"mt2-c17-s10","search":"Drill 10 Sigmoid neuron gradient **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update.\n\n**Full solution.**\n\n\n\n### Given\n\nx = 2; w = 0.5; b = 0; target y = 1.\n\nActivation: sigmoid; loss: half squared error.\n\nLearning rate η = 0.1.\n\n### To find\n\nFind the output, delta, weight/bias gradients and one update.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nHere the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly.\n\nThe symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z.\n\n\n\n### Step 1: weighted sum and sigmoid\n\n\n### Step 2: error and activation slope\n\n\n### Step 3: multiply the linked effects to obtain delta\n\n### Step 4: multiply delta by the input to each parameter\nThe weight receives x = 2. The bias effectively receives the constant input 1.\n\n\n### Step 5: subtract learning rate times each gradient\n\n\n**Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here.\n"},{"title":"Drill 11 Full ReLU backpropagation","question":["Question. Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases.
","Input x = [1, 2].
","W₁ is printed with the question.
","b₁ = [0.1, 0.2].
","W₂ is printed with the question.
","b₂ = [0.1].
","Hidden and output activations are ReLU.
","Target y = [1].
","Learning rate η = 0.1.
","Use half squared loss and update all weights and biases.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.
","![\"z_j=\\sum_iw_{ji}a_i+b_j,\\quad](\"assets/equations/dc1bda021c501564.svg\")
Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.
","![\"\\delta_{o_k}=(o_k-y_k)f'(z_{o_k})\"](\"assets/equations/e56bc6ef11f21beb.svg\")
![\"\\delta_{h_j}=\\left(\\sum_kw_{kj}\\delta_{o_k}\\right)f'(z_{h_j})\"](\"assets/equations/6086310bbc701ed1.svg\")
The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.
","![\"\\frac{\\partial](\"assets/equations/6f883fcc3e95816b.svg\")
![\"w_{ji,\\mathrm{new}}=w_{ji}-\\eta\\delta_j](\"assets/equations/a7c0c7e8cef31635.svg\")
For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/d12052a5268cb30c.svg\")
![\"h_{1}](\"assets/equations/b78961f402336b21.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/0eb4dbe7db30c72c.svg\")
![\"h_{2}](\"assets/equations/3ca4b1ac8771a1a8.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/24202b454fb4dad2.svg\")
![\"o_{1}](\"assets/equations/e6f7822c28a4fdd9.svg\")
Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/08c65e290585835f.svg\")
The ReLU derivative is 1 here because this preactivation is positive.
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.0275. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=(-0.55)\\times(0.5)\\approx](\"assets/equations/3a95215144e52b49.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,1,\\mathrm{new}}=0.4-0.1(-0.275)\\approx](\"assets/equations/5bb285fe6a5d5ca6.svg\")
h2 → o1:
","Subtract η × gradient = −0.0165. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,2}^{(2)}=(-0.55)\\times(0.3)\\approx](\"assets/equations/9dd543c1a4cfe977.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,2,\\mathrm{new}}=0.5-0.1(-0.165)\\approx](\"assets/equations/6b95f80af95c8ca3.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.4)\\times(-0.55)\\approx](\"assets/equations/b6b2e13098fb66e7.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/b39852d5faaf11fb.svg\")
![\"\\delta_{h_1}](\"assets/equations/f452f757c3f0207e.svg\")
Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.5)\\times(-0.55)\\approx](\"assets/equations/7150a7153bb561f5.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/d346b0df4b69bd95.svg\")
![\"\\delta_{h_2}](\"assets/equations/7ad50eed6d4c12e2.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = −0.022. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=(-0.22)\\times(1)\\approx](\"assets/equations/0c1b77c7df67c595.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,1,\\mathrm{new}}=0.2-0.1(-0.22)\\approx](\"assets/equations/e91b335304219c5c.svg\")
x2 → h1:
","Subtract η × gradient = −0.044. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,2}^{(1)}=(-0.22)\\times(2)\\approx](\"assets/equations/828d4390f7c40527.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,2,\\mathrm{new}}=0.1-0.1(-0.44)\\approx](\"assets/equations/d0bf76ef563b87e1.svg\")
x1 → h2:
","Subtract η × gradient = −0.0275. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,1}^{(1)}=(-0.275)\\times(1)\\approx](\"assets/equations/a0689c05816a3a3a.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,1,\\mathrm{new}}=0.3-0.1(-0.275)\\approx](\"assets/equations/950cba298eb25531.svg\")
x2 → h2:
","Subtract η × gradient = −0.055. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,2}^{(1)}=(-0.275)\\times(2)\\approx](\"assets/equations/27b6c549a9eafc2d.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,2,\\mathrm{new}}=-0.1-0.1(-0.55)\\approx](\"assets/equations/c7fab0773a9a18b8.svg\")
Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","![\"b_{h_1,\\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000\"](\"assets/equations/31940cedcf20decc.svg\")
![\"b_{h_2,\\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000\"](\"assets/equations/6350b7de1a7fd962.svg\")
![\"b_{o_1,\\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000\"](\"assets/equations/ce690df86543a5bc.svg\")
These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
"],"kind":"practice","id":"mt2-c17-s11","search":"Drill 11 Full ReLU backpropagation **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases.\n\n\n\n\n\n**Full solution.**\n\n\n\n### Given\n\nInput x = [1, 2].\n\nW₁ is printed with the question.\n\nb₁ = [0.1, 0.2].\n\nW₂ is printed with the question.\n\nb₂ = [0.1].\n\nHidden and output activations are ReLU.\n\nTarget y = [1].\n\nLearning rate η = 0.1.\n\nUse half squared loss and update all weights and biases.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used before substituting numbers\nFor every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.\n\nDelta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.\n\n\nThe gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.\n\n\nFor sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.\nPrediction minus target = [-0.55]."},{"title":"Drill 12 Two output sigmoid backpropagation","question":["Question. Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters.
","Input x = [0.2, 0.4].
","W₁ is printed with the question.
","b₁ = [0.1, 0.1].
","W₂ is printed with the question.
","b₂ = [0, 0.1].
","Hidden and output activations are sigmoid.
","Target y = [1, 0].
","Learning rate η = 0.2.
","Use half summed squared loss and update all parameters.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.
","Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.
","The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.
","For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/b34f23de35a7cf0c.svg\")
![\"h_{1}=\\frac{1}{1+e^{-(0.2)}}=0.549833997\"](\"assets/equations/acdd5499eb345eef.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/6178667245d07427.svg\")
![\"h_{2}=\\frac{1}{1+e^{-(0.18)}}=0.544878892\"](\"assets/equations/63a52c4f6d59b8f1.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/897314316dee8982.svg\")
![\"o_{1}=\\frac{1}{1+e^{-(0.05547891)}}=0.513866171\"](\"assets/equations/247adbbc93580b44.svg\")
Output neuron 2:
","![\"z_{o_2}](\"assets/equations/0e0191df53ed3741.svg\")
![\"o_{2}=\\frac{1}{1+e^{-(0.482901756)}}=0.618432848\"](\"assets/equations/9a8b37a15ece040e.svg\")
Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/f1f5b1787fc59def.svg\")
The derivative is o(1−o).
","Output delta 2:
","![\"\\delta_{o_2}](\"assets/equations/6546421fef60e0c5.svg\")
The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.013354367. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=(-0.121439988)\\times(0.549833997)\\approx](\"assets/equations/8335141cc6234897.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,1,\\mathrm{new}}=0.2-0.2(-0.066771834)\\approx](\"assets/equations/4c9c4df3cf58176a.svg\")
h2 → o1:
","Subtract η × gradient = −0.013234017. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,2}^{(2)}=(-0.121439988)\\times(0.544878892)\\approx](\"assets/equations/e6cb2fb79ea00bc6.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,2,\\mathrm{new}}=-0.1-0.2(-0.066170086)\\approx](\"assets/equations/1f30020d32cef10b.svg\")
h1 → o2:
","Subtract η × gradient = 0.01604788. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{2,1}^{(2)}=(0.145933863)\\times(0.549833997)\\approx](\"assets/equations/16c40d7d34614dcb.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{2,1,\\mathrm{new}}=0.3-0.2(0.080239399)\\approx](\"assets/equations/5660f8c2c8b62a6f.svg\")
h2 → o2:
","Subtract η × gradient = 0.015903256. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{2,2}^{(2)}=(0.145933863)\\times(0.544878892)\\approx](\"assets/equations/2bf8871d8771b0c8.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{2,2,\\mathrm{new}}=0.4-0.2(0.079516282)\\approx](\"assets/equations/d6e1bdf461c23702.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.2)\\times(-0.121439988)\\approx](\"assets/equations/7be9181956492e01.svg\")
Contribution from output 2.
","![\"C_2=(0.3)\\times(0.145933863)\\approx](\"assets/equations/0025ef2e4f3926ed.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1+C_2\\approx](\"assets/equations/6abb0857809f8db8.svg\")
![\"\\delta_{h_1}](\"assets/equations/3348b1ba7e091393.svg\")
Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(-0.1)\\times(-0.121439988)\\approx](\"assets/equations/909a9296285664eb.svg\")
Contribution from output 2.
","![\"C_2=(0.4)\\times(0.145933863)\\approx](\"assets/equations/14154a8890e8abb8.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1+C_2\\approx](\"assets/equations/1538baedb3bcff3c.svg\")
![\"\\delta_{h_2}](\"assets/equations/b6511e93ffa2060f.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = 0.000192985. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=(0.004824633)\\times(0.2)\\approx](\"assets/equations/a8abe88d39c4a4c9.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,1,\\mathrm{new}}=0.1-0.2(0.000964927)\\approx](\"assets/equations/e34dab3cfb2bef31.svg\")
x2 → h1:
","Subtract η × gradient = 0.000385971. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,2}^{(1)}=(0.004824633)\\times(0.4)\\approx](\"assets/equations/a45450793aa1fbee.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,2,\\mathrm{new}}=0.2-0.2(0.001929853)\\approx](\"assets/equations/ff165bc32cce96bf.svg\")
x1 → h2:
","Subtract η × gradient = 0.000699494. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,1}^{(1)}=(0.017487356)\\times(0.2)\\approx](\"assets/equations/5a519701097922e5.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,1,\\mathrm{new}}=-0.2-0.2(0.003497471)\\approx](\"assets/equations/251cd8c98d77e90c.svg\")
x2 → h2:
","Subtract η × gradient = 0.001398988. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,2}^{(1)}=(0.017487356)\\times(0.4)\\approx](\"assets/equations/253913dce1e316cf.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,2,\\mathrm{new}}=0.3-0.2(0.006994942)\\approx](\"assets/equations/30eae54a2d6333bf.svg\")
Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","![\"b_{h_1,\\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073\"](\"assets/equations/4d95f499c8cf92af.svg\")
![\"b_{h_2,\\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529\"](\"assets/equations/574987d80e93f51f.svg\")
![\"b_{o_1,\\mathrm{new}}=0-0.2(-0.121439988)=0.024287998\"](\"assets/equations/c0bf69b79bc6be7c.svg\")
![\"b_{o_2,\\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227\"](\"assets/equations/6ab34abf185daf06.svg\")
These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
"],"kind":"practice","id":"mt2-c17-s12","search":"Drill 12 Two output sigmoid backpropagation **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters.\n\n\n\n\n\n**Full solution.**\n\n\n\n### Given\n\nInput x = [0.2, 0.4].\n\nW₁ is printed with the question.\n\nb₁ = [0.1, 0.1].\n\nW₂ is printed with the question.\n\nb₂ = [0, 0.1].\n\nHidden and output activations are sigmoid.\n\nTarget y = [1, 0].\n\nLearning rate η = 0.2.\n\nUse half summed squared loss and update all parameters.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used before substituting numbers\nFor every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.\n\nDelta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.\n\n\nThe gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.\n\n\nFor sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each "},{"title":"Drill 13 A negative ReLU blocks this path","question":["Question. x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2.
"],"answer":["x = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2.
","Hidden activation: ReLU; output activation: linear.
","Target y = 1; loss: half squared error.
","Find the gradients of w₁ and w₂ and explain any zero gradients.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","The hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output.
","![\"z_h=w_1x+b_1,\\quad](\"assets/equations/2107914ba4f232f1.svg\")
![\"L=\\frac12(o-y)^2,\\quad](\"assets/equations/160d78635e6efcbf.svg\")
![\"\\frac{\\partial](\"assets/equations/a9eda432984ec8da.svg\")
![\"z_h=(-0.4)\\times2+0.1=-0.8+0.1=-0.7\"](\"assets/equations/81ce8e4429aab67a.svg\")
![\"h=\\max(0,-0.7)=0\"](\"assets/equations/0f4049c82bf618fd.svg\")
Because the raw hidden score is negative, ReLU's derivative at this point is 0.
","![\"o=0.5\\times0+0.2=0.2\"](\"assets/equations/8775ad02ed7eaf6c.svg\")
![\"\\delta_o=0.2-1=-0.8\"](\"assets/equations/ff295007085f430d.svg\")
![\"\\frac{\\partial](\"assets/equations/400d3f8e82bf09fc.svg\")
This is zero because the input to that connection, h, is zero.
","![\"\\delta_h=(-0.8)\\times0.5\\times0=0\"](\"assets/equations/1721f576f8f22211.svg\")
![\"\\frac{\\partial](\"assets/equations/a131ac401273e09f.svg\")
Final answer: both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero.
"],"kind":"practice","id":"mt2-c17-s13","search":"Drill 13 A negative ReLU blocks this path **Question.** x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2.\n\n**Full solution.**\n\n\n\n### Given\n\nx = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2.\n\nHidden activation: ReLU; output activation: linear.\n\nTarget y = 1; loss: half squared error.\n\n### To find\n\nFind the gradients of w₁ and w₂ and explain any zero gradients.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used\nThe hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output.\n\n\n\n### Step 1: hidden weighted sum and activation\n\n\nBecause the raw hidden score is negative, ReLU's derivative at this point is 0.\n### Step 2: output prediction and output delta\n\n\n### Step 3: output weight gradient\n\nThis is zero because the input to that connection, h, is zero.\n### Step 4: hidden delta and input weight gradient\n\n\n**Final answer:** both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero.\n"},{"title":"Drill 14 Batch and sequential SGD","question":["Question. ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch.
"],"answer":["Model ŷ = wx; initial w = 0; η = 0.1.
","Samples in order: (x,y) = (1,1), then (2,2).
","Batch objective: mean half squared loss.
","Compare the weight after one batch update and one sequential SGD epoch.
","Batch mean versus sequential SGD. A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.
","B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.
","Yaad rakhne ka cue: Batch: same old weight par average; SGD: har sample ke baad weight change.
","For one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the same old weight. Sequential SGD updates after each sample and uses that new weight for the next sample.
","![\"\\hat](\"assets/equations/e02d92789136f341.svg\")
![\"g_{\\mathrm{batch}}=\\frac{g_1+g_2}{2},\\qquad](\"assets/equations/7783293f367f0a09.svg\")
Step 1: Part A: one batch update from w = 0.
","For sample (1,1), prediction = 0 × 1 = 0, so:
","![\"g_1=(0-1)\\times1=-1\"](\"assets/equations/65b3616ae354de6a.svg\")
For sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so:
","![\"g_2=(0-2)\\times2=-4\"](\"assets/equations/b5e54f46939905d8.svg\")
Average these two gradients, then update once.
","![\"g_{\\mathrm{batch}}=\\frac{-1+(-4)}2=-2.5\"](\"assets/equations/ae3a9f41f9f31680.svg\")
![\"w_{\\mathrm{batch}}=0-0.1(-2.5)=0.25\"](\"assets/equations/568002eaa70aedb2.svg\")
Step 2: Part B: one SGD epoch, restarting at w = 0.
","The first sample has gradient −1, just as above.
","![\"w_1=0-0.1(-1)=0.1\"](\"assets/equations/f2b5721e0e1c67b4.svg\")
For the second sample use this new weight, so prediction is:
","![\"\\hat](\"assets/equations/905f0951780ee22a.svg\")
![\"g_2=(0.2-2)\\times2=(-1.8)\\times2=-3.6\"](\"assets/equations/f262bc5d6c22d8bb.svg\")
![\"w_2=0.1-0.1(-3.6)=0.1+0.36=0.46\"](\"assets/equations/f8c5ce8387c2e254.svg\")
Final answer: batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates.
"],"kind":"practice","id":"mt2-c17-s14","search":"Drill 14 Batch and sequential SGD **Question.** ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch.\n\n**Full solution.**\n\n\n\n### Given\n\nModel ŷ = wx; initial w = 0; η = 0.1.\n\nSamples in order: (x,y) = (1,1), then (2,2).\n\nBatch objective: mean half squared loss.\n\n### To find\n\nCompare the weight after one batch update and one sequential SGD epoch.\n\n### Formula — which one and why\n\n**Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.\n\nB = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nFor one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the **same old weight**. Sequential SGD updates after each sample and uses that new weight for the next sample.\n\n\n**Step 1: Part A: one batch update from w = 0.**\nFor sample (1,1), prediction = 0 × 1 = 0, so:\n\nFor sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so:\n\nAverage these two gradients, then update once.\n\n\n**Step 2: Part B: one SGD epoch, restarting at w = 0.**\nThe first sample has gradient −1, just as above.\n\nFor the second sample use this new weight, so prediction is:\n\n\n\n**Final answer:** batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates.\n"},{"title":"Drill 15 Incomplete mini batch","question":["Question. N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it.
"],"answer":["N = 103 samples; B = 20 samples per mini-batch.
","E = 3 epochs.
","Compare both remainder policies: keep or drop the final 3 examples.
","Find total updates for full-batch GD, SGD, and both mini-batch policies.
","Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.
","N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.
","Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.
","An epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs.
","![\"U_{\\mathrm{batch}}=E,\\qquad](\"assets/equations/e0f5f4a6b7d3af9c.svg\")
![\"U_{\\mathrm{keep}}=E\\left\\lceil\\frac](\"assets/equations/147e19d72acc4203.svg\")
Ceiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained.
","All 103 samples form one batch per epoch.
","![\"U_{\\mathrm{batch}}=3\\times1=3\"](\"assets/equations/11f14a41e22f1916.svg\")
Each sample gets its own update.
","![\"U_{\\mathrm{SGD}}=103\\times3=309\"](\"assets/equations/4c08a8b839ca4b99.svg\")
![\"103=5\\times20+3\"](\"assets/equations/7600fce77b11dfac.svg\")
The batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give:
","![\"U_{\\mathrm{keep}}=6\\times3=18\"](\"assets/equations/64422d140052bf5c.svg\")
Only the five complete batches are used each epoch.
","![\"U_{\\mathrm{drop}}=5\\times3=15\"](\"assets/equations/6b34106919fd6639.svg\")
Final answer: 3, 309, 18 and 15 updates respectively.
"],"kind":"practice","id":"mt2-c17-s15","search":"Drill 15 Incomplete mini batch **Question.** N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it.\n\n**Full solution.**\n\n\n\n### Given\n\nN = 103 samples; B = 20 samples per mini-batch.\n\nE = 3 epochs.\n\nCompare both remainder policies: keep or drop the final 3 examples.\n\n### To find\n\nFind total updates for full-batch GD, SGD, and both mini-batch policies.\n\n### Formula — which one and why\n\n**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.\n\nN = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nAn epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs.\n\n\nCeiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained.\n### Step 1: batch GD\nAll 103 samples form one batch per epoch.\n\n### Step 2: SGD\nEach sample gets its own update.\n\n### Step 3: mini-batches with the remainder\n\nThe batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give:\n\n### Step 4: mini-batches without the remainder\nOnly the five complete batches are used each epoch.\n\n**Final answer:** 3, 309, 18 and 15 updates respectively.\n"},{"title":"Drill 16 GD variation 1","question":["Question. J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates.
"],"answer":["Objective: J(θ) = ½θ².
","Initial θ₀ = 1; η = 0.1; two iterations.
","Find every stored state needed by the method and the parameter after each of two updates.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","![\"J(\\theta)=\\frac12(\\theta-0)^2,\\qquad](\"assets/equations/aec670e56abfcf00.svg\")
The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.
","![\"g_t=J\\prime(\\theta_{t-1}),\\quad](\"assets/equations/b3d142fa7d569702.svg\")
Step 1: Update 1.
","![\"g_1=1-(0)=1\"](\"assets/equations/51e0e1de442cadbf.svg\")
![\"\\theta_1=1-0.1(1)=0.9\"](\"assets/equations/1f113e7fc6ecc1f4.svg\")
Step 2: Update 2.
","![\"g_2=0.9-(0)=0.9\"](\"assets/equations/acfaeec4e6af0c51.svg\")
![\"\\theta_2=0.9-0.1(0.9)=0.81\"](\"assets/equations/1708353d6ee40680.svg\")
Final parameter after two updates: 0.81. The second gradient is evaluated again using the parameter obtained after the first update.
"],"kind":"practice","id":"mt2-c17-s16","search":"Drill 16 GD variation 1 **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates. \n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½θ².\n\nInitial θ₀ = 1; η = 0.1; two iterations.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.\n### Update formulas\n\n**Step 1: Update 1.**\n\n\n**Step 2: Update 2.**\n\n\n**Final parameter after two updates: 0.81.** The second gradient is evaluated again using the parameter obtained after the first update.\n"},{"title":"Drill 17 GD variation 2","question":["Question. J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates.
"],"answer":["Objective: J(θ) = ½(θ−1)².
","Initial θ₀ = −0.5; η = 0.1; two iterations.
","Find every stored state needed by the method and the parameter after each of two updates.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","![\"J(\\theta)=\\frac12(\\theta-1)^2,\\qquad](\"assets/equations/6e49b01175757fe7.svg\")
The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.
","Step 1: Update 1.
","![\"g_1=-0.5-(1)=-1.5\"](\"assets/equations/ce4034cb1a64ebf6.svg\")
![\"\\theta_1=-0.5-0.1(-1.5)=-0.35\"](\"assets/equations/2c2dd4efda1f26eb.svg\")
Step 2: Update 2.
","![\"g_2=-0.35-(1)=-1.35\"](\"assets/equations/7e8a70b34a625603.svg\")
![\"\\theta_2=-0.35-0.1(-1.35)=-0.215\"](\"assets/equations/354b5f7978875963.svg\")
Final parameter after two updates: -0.215. The second gradient is evaluated again using the parameter obtained after the first update.
"],"kind":"practice","id":"mt2-c17-s17","search":"Drill 17 GD variation 2 **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates. \n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½(θ−1)².\n\nInitial θ₀ = −0.5; η = 0.1; two iterations.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.\n### Update formulas\n\n**Step 1: Update 1.**\n\n\n**Step 2: Update 2.**\n\n\n**Final parameter after two updates: -0.215.** The second gradient is evaluated again using the parameter obtained after the first update.\n"},{"title":"Drill 18 Momentum variation 1","question":["Question. J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.
"],"answer":["Objective: J(θ) = ½θ².
","Initial θ₀ = 1; η = 0.1; two iterations.
","β = 0.9; initial signed velocity v₀ = 0.
","Find every stored state needed by the method and the parameter after each of two updates.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.
","v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.
","![\"v_t=\\beta](\"assets/equations/cfd2c8e117828e21.svg\")
Step 1: Update 1.
","Step 2: Update 2.
","Final parameter after two updates: 0.72. The second gradient is evaluated again using the parameter obtained after the first update.
"],"kind":"practice","id":"mt2-c17-s18","search":"Drill 18 Momentum variation 1 **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.\n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½θ².\n\nInitial θ₀ = 1; η = 0.1; two iterations.\n\nβ = 0.9; initial signed velocity v₀ = 0.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.\n### Update formulas\nv stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.\n\n\n**Step 1: Update 1.**\n\n\n\n**Step 2: Update 2.**\n\n\n\n**Final parameter after two updates: 0.72.** The second gradient is evaluated again using the parameter obtained after the first update.\n"},{"title":"Drill 19 Momentum variation 2","question":["Question. J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.
"],"answer":["Objective: J(θ) = ½(θ−1)².
","Initial θ₀ = −0.5; η = 0.1; two iterations.
","β = 0.9; initial signed velocity v₀ = 0.
","Find every stored state needed by the method and the parameter after each of two updates.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.
","v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.
","Step 1: Update 1.
","![\"v_1=0.9(0)-0.1(-1.5)=0.15\"](\"assets/equations/9e63b276a509c75c.svg\")
![\"\\theta_1=-0.5+(0.15)=-0.35\"](\"assets/equations/3de31f4085d1f9eb.svg\")
Step 2: Update 2.
","![\"v_2=0.9(0.15)-0.1(-1.35)=0.27\"](\"assets/equations/0b81155b82e98b2d.svg\")
![\"\\theta_2=-0.35+(0.27)=-0.08\"](\"assets/equations/e5bc5d38d05f5667.svg\")
Final parameter after two updates: -0.08. The second gradient is evaluated again using the parameter obtained after the first update.
"],"kind":"practice","id":"mt2-c17-s19","search":"Drill 19 Momentum variation 2 **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.\n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½(θ−1)².\n\nInitial θ₀ = −0.5; η = 0.1; two iterations.\n\nβ = 0.9; initial signed velocity v₀ = 0.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.\n### Update formulas\nv stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.\n\n\n**Step 1: Update 1.**\n\n\n\n**Step 2: Update 2.**\n\n\n\n**Final parameter after two updates: -0.08.** The second gradient is evaluated again using the parameter obtained after the first update.\n"},{"title":"Drill 20 Nesterov variation 1","question":["Question. J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.
"],"answer":["Objective: J(θ) = ½θ².
","Initial θ₀ = 1; η = 0.1; two iterations.
","β = 0.9; initial signed velocity v₀ = 0.
","Find every stored state needed by the method and the parameter after each of two updates.
","Nesterov lookahead. First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.
","θ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.
","Yaad rakhne ka cue: Pehle look ahead; gradient wahan; update original position se.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.
","v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.
","![\"\\theta_{\\mathrm{look}}=\\theta_{t-1}+\\beta](\"assets/equations/122d449c66d53648.svg\")
Step 1: Update 1.
","![\"\\theta_{\\mathrm{look}}=1+0.9(0)=1\"](\"assets/equations/44a5155fc196e4c6.svg\")
Step 2: Update 2.
","![\"\\theta_{\\mathrm{look}}=0.9+0.9(-0.1)=0.81\"](\"assets/equations/2f8c51c39237a8f3.svg\")
![\"g_2=0.81-(0)=0.81\"](\"assets/equations/f61619e280f936a0.svg\")
Final parameter after two updates: 0.729. The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.
"],"kind":"practice","id":"mt2-c17-s20","search":"Drill 20 Nesterov variation 1 **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.\n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½θ².\n\nInitial θ₀ = 1; η = 0.1; two iterations.\n\nβ = 0.9; initial signed velocity v₀ = 0.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.\n\nθ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.\n### Update formulas\nv stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.\n\n\n**Step 1: Update 1.**\n\n\n\n\n**Step 2: Update 2.**\n\n\n\n\n**Final parameter after two updates: 0.729.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.\n"},{"title":"Drill 21 Nesterov variation 2","question":["Question. J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.
"],"answer":["Objective: J(θ) = ½(θ−1)².
","Initial θ₀ = −0.5; η = 0.1; two iterations.
","β = 0.9; initial signed velocity v₀ = 0.
","Find every stored state needed by the method and the parameter after each of two updates.
","Nesterov lookahead. First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.
","θ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.
","Yaad rakhne ka cue: Pehle look ahead; gradient wahan; update original position se.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.
","v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.
","Step 1: Update 1.
","![\"\\theta_{\\mathrm{look}}=-0.5+0.9(0)=-0.5\"](\"assets/equations/a4363d91dbba8cc0.svg\")
Step 2: Update 2.
","![\"\\theta_{\\mathrm{look}}=-0.35+0.9(0.15)=-0.215\"](\"assets/equations/f623b6f68ec5b35a.svg\")
![\"g_2=-0.215-(1)=-1.215\"](\"assets/equations/f77fca68d44fde4d.svg\")
![\"v_2=0.9(0.15)-0.1(-1.215)=0.2565\"](\"assets/equations/afa161b36559b605.svg\")
![\"\\theta_2=-0.35+(0.2565)=-0.0935\"](\"assets/equations/cc919e70a6113df8.svg\")
Final parameter after two updates: -0.0935. The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.
"],"kind":"practice","id":"mt2-c17-s21","search":"Drill 21 Nesterov variation 2 **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.\n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½(θ−1)².\n\nInitial θ₀ = −0.5; η = 0.1; two iterations.\n\nβ = 0.9; initial signed velocity v₀ = 0.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.\n\nθ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.\n### Update formulas\nv stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.\n\n\n**Step 1: Update 1.**\n\n\n\n\n**Step 2: Update 2.**\n\n\n\n\n**Final parameter after two updates: -0.0935.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.\n"},{"title":"Drill 22 AdaGrad variation 1","question":["Question. θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation.
"],"answer":["Initial θ₀ = 1; η = 0.1.
","Supplied gradients: [2,1].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","Find every stored state needed by the method and the parameter after each of two updates.
","AdaGrad. Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.
","G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.
","Yaad rakhne ka cue: Square karo, purane sum mein jodo, root se scale karo.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0.
","![\"\\theta_t=\\theta_{t-1}-\\frac{\\eta](\"assets/equations/49a0512d42c905d8.svg\")
Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient 2.
","Add the new squared gradient to all previous squares.
","![\"G_1=0+(2)^2=4\"](\"assets/equations/a2f78c9b3d52d653.svg\")
![\"\\sqrt{G_1}=\\sqrt{4}=2\"](\"assets/equations/7923d1e84dd761ae.svg\")
![\"\\theta_1=1-\\frac{0.1\\times(2)}{\\sqrt{4}}=0.9\"](\"assets/equations/e1122171b0bd236e.svg\")
Step 2: Iteration 2 with supplied gradient 1.
","Add the new squared gradient to all previous squares.
","![\"G_2=4+(1)^2=5\"](\"assets/equations/dd892c081c748bf2.svg\")
![\"\\sqrt{G_2}=\\sqrt{5}=2.236067977\"](\"assets/equations/33c414269f4796af.svg\")
![\"\\theta_2=0.9-\\frac{0.1\\times(1)}{\\sqrt{5}}=0.85527864\"](\"assets/equations/c2a79e598e1b306b.svg\")
Question. θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation.
"],"answer":["Initial θ₀ = −0.5; η = 0.1.
","Supplied gradients: [−1,2].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","Find every stored state needed by the method and the parameter after each of two updates.
","AdaGrad. Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.
","G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.
","Yaad rakhne ka cue: Square karo, purane sum mein jodo, root se scale karo.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient -1.
","Add the new squared gradient to all previous squares.
","![\"G_1=0+(-1)^2=1\"](\"assets/equations/bba453205f160315.svg\")
![\"\\sqrt{G_1}=\\sqrt{1}=1\"](\"assets/equations/1112f0b4d6224a38.svg\")
![\"\\theta_1=-0.5-\\frac{0.1\\times(-1)}{\\sqrt{1}}=-0.4\"](\"assets/equations/37395f4e33d5e1e8.svg\")
Step 2: Iteration 2 with supplied gradient 2.
","Add the new squared gradient to all previous squares.
","![\"G_2=1+(2)^2=5\"](\"assets/equations/4e7c36432792b91c.svg\")
![\"\\theta_2=-0.4-\\frac{0.1\\times(2)}{\\sqrt{5}}=-0.489442719\"](\"assets/equations/1161742779014e25.svg\")
Question. θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9.
"],"answer":["Initial θ₀ = 1; η = 0.1.
","Supplied gradients: [2,1].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","ρ = 0.9.
","Find every stored state needed by the method and the parameter after each of two updates.
","RMSProp. Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.
","s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.
","Yaad rakhne ka cue: Saare purane squares nahi—unki fading average.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient 2.
","Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.
","![\"s_1=0.9(0)+0.1(2)^2=0.4\"](\"assets/equations/a0575b9596aea909.svg\")
![\"\\sqrt{s_1}=\\sqrt{0.4}=0.632455532\"](\"assets/equations/8724fbf30b88d3b5.svg\")
![\"\\theta_1=1-\\frac{0.1\\times(2)}{\\sqrt{0.4}}=0.683772234\"](\"assets/equations/84ca434a6eea16db.svg\")
Step 2: Iteration 2 with supplied gradient 1.
","Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.
","![\"s_2=0.9(0.4)+0.1(1)^2=0.46\"](\"assets/equations/99dd3340957517b7.svg\")
![\"\\sqrt{s_2}=\\sqrt{0.46}=0.678232998\"](\"assets/equations/7bc15fc56ff7b9f2.svg\")
![\"\\theta_2=0.683772234-\\frac{0.1\\times(1)}{\\sqrt{0.46}}=0.536330278\"](\"assets/equations/dec405b6648724b8.svg\")
Question. θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9.
"],"answer":["Initial θ₀ = −0.5; η = 0.1.
","Supplied gradients: [−1,2].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","ρ = 0.9.
","Find every stored state needed by the method and the parameter after each of two updates.
","RMSProp. Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.
","s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.
","Yaad rakhne ka cue: Saare purane squares nahi—unki fading average.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient -1.
","Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.
","![\"s_1=0.9(0)+0.1(-1)^2=0.1\"](\"assets/equations/81d06c696eada47a.svg\")
![\"\\sqrt{s_1}=\\sqrt{0.1}=0.316227766\"](\"assets/equations/8e43eda4dcf6bcb7.svg\")
![\"\\theta_1=-0.5-\\frac{0.1\\times(-1)}{\\sqrt{0.1}}=-0.183772234\"](\"assets/equations/1da7d1cdeda02890.svg\")
Step 2: Iteration 2 with supplied gradient 2.
","Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.
","![\"s_2=0.9(0.1)+0.1(2)^2=0.49\"](\"assets/equations/2c7113e75687c23e.svg\")
![\"\\sqrt{s_2}=\\sqrt{0.49}=0.7\"](\"assets/equations/e14a6fd4eb271def.svg\")
![\"\\theta_2=-0.183772234-\\frac{0.1\\times(2)}{\\sqrt{0.49}}=-0.46948652\"](\"assets/equations/dbbd9b6c37d99e9e.svg\")
Question. θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction.
"],"answer":["Initial θ₀ = 1; η = 0.1.
","Supplied gradients: [2,1].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","β₁ = 0.9; β₂ = 0.999; use bias correction.
","Find every stored state needed by the method and the parameter after each of two updates.
","Adam with bias correction. Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update.
","m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence.
","Yaad rakhne ka cue: Direction memory + square memory; hats lagao, phir update.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999.
","![\"v_t=\\beta_2v_{t-1}+(1-\\beta_2)g_t^2\"](\"assets/equations/caedeeee27912e58.svg\")
![\"\\hat](\"assets/equations/637a35db2d99492a.svg\")
![\"\\theta_t=\\theta_{t-1}-\\frac{\\eta\\hat](\"assets/equations/2f5e5116ef55c3b3.svg\")
Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient 2.
","First calculate the signed-gradient mean m and squared-gradient mean v.
","![\"v_1=0.999(0)+0.001(2)^2=0.004\"](\"assets/equations/b00fce2cf8ae5a5b.svg\")
Correct each zero-initialized mean using this iteration number in the power.
","![\"\\widehat](\"assets/equations/f8975b50fed0845b.svg\")
![\"\\widehat](\"assets/equations/f16a835e4241c7ed.svg\")
Divide the corrected direction by the corrected root, then subtract the scaled step.
","Step 2: Iteration 2 with supplied gradient 1.
","First calculate the signed-gradient mean m and squared-gradient mean v.
","![\"v_2=0.999(0.004)+0.001(1)^2=0.004996\"](\"assets/equations/308792cf26a22040.svg\")
Correct each zero-initialized mean using this iteration number in the power.
","![\"\\widehat](\"assets/equations/2b1a3823c4815fe2.svg\")
![\"\\widehat](\"assets/equations/3922f5c0c13f3037.svg\")
Divide the corrected direction by the corrected root, then subtract the scaled step.
","![\"\\theta_2=0.9-\\frac{0.1\\times(1.473684211)}{\\sqrt{2.499249625}}=0.806782036\"](\"assets/equations/4d6c4212f9eead9c.svg\")
Question. θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction.
"],"answer":["Initial θ₀ = −0.5; η = 0.1.
","Supplied gradients: [−1,2].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","β₁ = 0.9; β₂ = 0.999; use bias correction.
","Find every stored state needed by the method and the parameter after each of two updates.
","Adam with bias correction. Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update.
","m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence.
","Yaad rakhne ka cue: Direction memory + square memory; hats lagao, phir update.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient -1.
","First calculate the signed-gradient mean m and squared-gradient mean v.
","![\"m_1=0.9(0)+0.1(-1)=-0.1\"](\"assets/equations/ff0acbc11515087e.svg\")
![\"v_1=0.999(0)+0.001(-1)^2=0.001\"](\"assets/equations/e44b2300355f9498.svg\")
Correct each zero-initialized mean using this iteration number in the power.
","![\"\\widehat](\"assets/equations/721803042ed359b9.svg\")
![\"\\widehat](\"assets/equations/b4d865fbfcf2b328.svg\")
Divide the corrected direction by the corrected root, then subtract the scaled step.
","Step 2: Iteration 2 with supplied gradient 2.
","First calculate the signed-gradient mean m and squared-gradient mean v.
","![\"m_2=0.9(-0.1)+0.1(2)=0.11\"](\"assets/equations/89d9785f6f04be8b.svg\")
![\"v_2=0.999(0.001)+0.001(2)^2=0.004999\"](\"assets/equations/0838089126c8ebdc.svg\")
Correct each zero-initialized mean using this iteration number in the power.
","![\"\\widehat](\"assets/equations/e970513a5c078389.svg\")
![\"\\widehat](\"assets/equations/a6410e8fada39e27.svg\")
Divide the corrected direction by the corrected root, then subtract the scaled step.
","![\"\\theta_2=-0.4-\\frac{0.1\\times(0.578947368)}{\\sqrt{2.500750375}}=-0.436610353\"](\"assets/equations/3e2be14fb02aba94.svg\")
Question. Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw².
"],"answer":["Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1.
","For L2, the question includes the factor ½.
","Find each penalty and the total L1/L2 objective.
","L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.
","λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.
","Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.
","L2 penalty with a factor of one-half. Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.
","λ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.
","![\"J=J_{\\mathrm{data}}+\\frac\\lambda2\\sum_iw_i^2\"](\"assets/equations/29419cadd5578572.svg\")
![\"g_i=g_{\\mathrm{data},i}+\\lambda](\"assets/equations/79fc02963f72332f.svg\")
![\"w_i^{\\mathrm{new}}=w_i-\\eta](\"assets/equations/616c8cbb90430248.svg\")
Yaad rakhne ka cue: Half wala L2 derivative mein lambda × weight deta hai.
","The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty.
","![\"J_{L1}=L_{\\mathrm{data}}+\\lambda\\sum_i|w_i|\"](\"assets/equations/14053316f92ab54d.svg\")
![\"J_{L2}=L_{\\mathrm{data}}+\\frac\\lambda2\\sum_iw_i^2\"](\"assets/equations/c3cf9af2ddfa197c.svg\")
Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations.
","Step 1: Part A: L1 objective.
","![\"|-2|=2,\\quad](\"assets/equations/fcceddf63f9341b2.svg\")
![\"\\sum_i|w_i|=2+1+0.5=3.5\"](\"assets/equations/fb36584739f81886.svg\")
![\"\\text{L1](\"assets/equations/fc6e588ea36fd61a.svg\")
![\"J_{L1}=0.4+0.35=0.75\"](\"assets/equations/e3ee4cac7da2daf6.svg\")
Step 2: Part B: L2 objective with the given half.
","![\"(-2)^2=4,\\quad](\"assets/equations/ef16358fc81a0cdc.svg\")
![\"\\sum_iw_i^2=4+1+0.25=5.25\"](\"assets/equations/18bd7cc9173340e8.svg\")
![\"\\frac\\lambda2=\\frac{0.1}2=0.05\"](\"assets/equations/482c5bfd76d38e6c.svg\")
![\"\\text{L2](\"assets/equations/722ddb9c14ea7480.svg\")
![\"J_{L2}=0.4+0.2625=0.6625\"](\"assets/equations/6399f9c64b91d83f.svg\")
Final answer: L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values.
"],"kind":"practice","id":"mt2-c17-s28","search":"Drill 28 Regularized objective **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw².\n\n**Full solution.**\n\n\n\n### Given\n\nData loss = 0.4; weights = [−2,1,0.5]; λ = 0.1.\n\nFor L2, the question includes the factor ½.\n\n### To find\n\nFind each penalty and the total L1/L2 objective.\n\n### Formula — which one and why\n\n**L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.\n\nλ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.\n\n\n\n\n\n**Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai.\n\n**L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.\n\nλ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty.\n\n\nAbsolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations.\n**Step 1: Part A: L1 objective.**\n\n\n\n\n**Step 2: Part B: L2 objective with the given half.**\n\n\n\n\n\n**Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values.\n"},{"title":"Drill 29 L1 update on positive and negative weights","question":["Question. w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero.
"],"answer":["Old weights = [−0.5,0.3].
","Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2.
","Find total gradients and the two updated weights.
","L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.
","λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.
","Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight.
","![\"g_{j,\\mathrm{total}}=g_{j,\\mathrm{data}}+\\lambda\\operatorname{sign}(w_j)\"](\"assets/equations/edfc42b95078bb2c.svg\")
![\"w_{j,\\mathrm{new}}=w_j-\\eta](\"assets/equations/8e2c2fa59e985f4e.svg\")
Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed.
","Step 1: Weight 1: old value −0.5.
","![\"\\lambda\\operatorname{sign}(-0.5)=0.1(-1)=-0.1\"](\"assets/equations/56f95d7675b65e54.svg\")
![\"g_{1,\\mathrm{total}}=0.2+(-0.1)=0.1\"](\"assets/equations/ee9041d822361870.svg\")
![\"w_{1,\\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52\"](\"assets/equations/6d4d841ab160b740.svg\")
Step 2: Weight 2: old value 0.3.
","![\"\\lambda\\operatorname{sign}(0.3)=0.1(1)=0.1\"](\"assets/equations/cce8304f0261d4a5.svg\")
![\"g_{2,\\mathrm{total}}=-0.1+0.1=0\"](\"assets/equations/09ef0e91f63c60f6.svg\")
![\"w_{2,\\mathrm{new}}=0.3-0.2(0)=0.3\"](\"assets/equations/39fa74bf610f2797.svg\")
Final answer: [−0.52, 0.3]. L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero.
"],"kind":"practice","id":"mt2-c17-s29","search":"Drill 29 L1 update on positive and negative weights **Question.** w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero.\n\n**Full solution.**\n\n\n\n### Given\n\nOld weights = [−0.5,0.3].\n\nData gradients = [0.2,−0.1]; λ = 0.1; η = 0.2.\n\n### To find\n\nFind total gradients and the two updated weights.\n\n### Formula — which one and why\n\n**L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.\n\nλ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.\n\n\n\n\n\n**Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai.\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formula\nThe supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight.\n\n\nBoth weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed.\n**Step 1: Weight 1: old value −0.5.**\n\n\n\n**Step 2: Weight 2: old value 0.3.**\n\n\n\n**Final answer: [−0.52, 0.3].** L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero.\n"},{"title":"Drill 30 L2 update","question":["Question. w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw².
"],"answer":["Old weights = [0.5,−0.2]; data gradients = [0.1,0.3].
","λ = 0.2; η = 0.1; penalty = (λ/2)Σw².
","Find penalty gradients, total gradients and updated weights.
","L2 penalty with a factor of one-half. Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.
","λ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.
","Yaad rakhne ka cue: Half wala L2 derivative mein lambda × weight deta hai.
","The derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating.
","![\"g_{j,\\mathrm{total}}=g_{j,\\mathrm{data}}+\\lambda](\"assets/equations/6dece7c528660c2d.svg\")
Step 1: Weight 1.
","![\"\\lambda](\"assets/equations/26564fe78f783309.svg\")
![\"g_{1,\\mathrm{total}}=0.1+0.1=0.2\"](\"assets/equations/e611707728d90ba8.svg\")
![\"w_{1,\\mathrm{new}}=0.5-0.1(0.2)=0.48\"](\"assets/equations/86b47a5bd5373b33.svg\")
Step 2: Weight 2.
","![\"\\lambda](\"assets/equations/d1d1a2ba29bb3862.svg\")
![\"g_{2,\\mathrm{total}}=0.3+(-0.04)=0.26\"](\"assets/equations/eeb80cfd582d91ed.svg\")
![\"w_{2,\\mathrm{new}}=-0.2-0.1(0.26)=-0.226\"](\"assets/equations/b559be4dc46e47e0.svg\")
Final answer: [0.48, −0.226]. The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative.
"],"kind":"practice","id":"mt2-c17-s30","search":"Drill 30 L2 update **Question.** w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw².\n\n**Full solution.**\n\n\n\n### Given\n\nOld weights = [0.5,−0.2]; data gradients = [0.1,0.3].\n\nλ = 0.2; η = 0.1; penalty = (λ/2)Σw².\n\n### To find\n\nFind penalty gradients, total gradients and updated weights.\n\n### Formula — which one and why\n\n**L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.\n\nλ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai.\n\n### Solution — step by step\n\n\n\n\n\n### Formula and its reason\nThe derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating.\n\n\n**Step 1: Weight 1.**\n\n\n\n**Step 2: Weight 2.**\n\n\n\n**Final answer: [0.48, −0.226].** The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative.\n"},{"title":"Drill 31 Inverted dropout","question":["Question. Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference.
"],"answer":["Activations = [2,−1,0,3].
","Keep probability q = 0.75; keep mask = [1,0,1,1].
","Find training outputs and state the inference rule.
","Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.
","aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.
","Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.
","The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved.
","![\"a'_j=\\frac{m_j](\"assets/equations/4210eb7af01249ba.svg\")
The multiplication is entry by entry; there is no dot-product sum.
","![\"[1(2),\\;0(-1),\\;1(0),\\;1(3)]=[2,0,0,3]\"](\"assets/equations/f0ff08cea46f6163.svg\")
![\"a'_1=\\frac2{0.75}=2.666666667\"](\"assets/equations/4257cce51e38bfe0.svg\")
![\"a'_2=\\frac0{0.75}=0,\\qquad](\"assets/equations/93312a45b7b42272.svg\")
![\"a'_4=\\frac3{0.75}=4\"](\"assets/equations/d5f3e01a5c059a83.svg\")
Training output = [2.666666667, 0, 0, 4]. A kept activation that was already zero stays zero.
","Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training.
"],"kind":"practice","id":"mt2-c17-s31","search":"Drill 31 Inverted dropout **Question.** Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference.\n\n**Full solution.**\n\n\n\n### Given\n\nActivations = [2,−1,0,3].\n\nKeep probability q = 0.75; keep mask = [1,0,1,1].\n\n### To find\n\nFind training outputs and state the inference rule.\n\n### Formula — which one and why\n\n**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.\n\naᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.\n\n\n\n\n\n**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formula\nThe mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved.\n\nThe multiplication is entry by entry; there is no dot-product sum.\n### Step 1: apply the mask\n\n### Step 2: divide each result by 0.75\n\n\n\n**Training output = [2.666666667, 0, 0, 4].** A kept activation that was already zero stays zero.\n### During inference\nDisable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training.\n"},{"title":"Drill 32 Convolution with a negative coefficient","question":["Question. Use the input A and kernel K shown below, stride 1, padding 0. Compute all outputs.
","| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
| 1 | −1 |
| 0 | 1 |
Stride 1; P = 0 (no padding).
Input A and kernel K are printed with the question.
","Stride S = 1; padding P = 0; bias = 0.
","Use the kernel without flipping it.
","Calculate all nine output cells.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation.
","![\"Y_{ij}=\\sum_{u=0}^{K_h-1}\\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b\"](\"assets/equations/1a1aa2b10ea1816e.svg\")
Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability.
","N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Step 1 shape: ![\"\\left\\lfloor\\frac{4+2](\"assets/equations/99ba828c16946a06.svg\")
= 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 1 | −2 |
| 0 | 1 |
Each cell is input × matching kernel coefficient.
(−1) + 1 = 0
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 1, column 1.
Window row 1.
","![\"R_1=(1)\\times(1)+(2)\\times(-1)=-1\"](\"assets/equations/d0c50421371c4116.svg\")
Window row 2.
","![\"R_2=(0)\\times(0)+(1)\\times(1)=1\"](\"assets/equations/8c103dd6ecdec24c.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,1}=(-1)+(1)=0\"](\"assets/equations/23722ac152853b27.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 2 | 0 |
| 0 | 3 |
Each cell is input × matching kernel coefficient.
2 + 3 = 5
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 1, column 2.
Window row 1.
","![\"R_1=(2)\\times(1)+(0)\\times(-1)=2\"](\"assets/equations/92e95617002bb2ff.svg\")
Window row 2.
","![\"R_2=(1)\\times(0)+(3)\\times(1)=3\"](\"assets/equations/89af9e3096091f5c.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,2}=(2)+(3)=5\"](\"assets/equations/e073412e219f2310.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 0 | −1 |
| 0 | 2 |
Each cell is input × matching kernel coefficient.
(−1) + 2 = 1
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 1, column 3.
Window row 1.
","![\"R_1=(0)\\times(1)+(1)\\times(-1)=-1\"](\"assets/equations/37f1662710810d02.svg\")
Window row 2.
","![\"R_2=(3)\\times(0)+(2)\\times(1)=2\"](\"assets/equations/1986dc904d99f31e.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,3}=(-1)+(2)=1\"](\"assets/equations/48b778ac3ad46a58.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 0 | −1 |
| 0 | 1 |
Each cell is input × matching kernel coefficient.
(−1) + 1 = 0
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 2, column 1.
Window row 1.
","Window row 2.
","![\"R_2=(2)\\times(0)+(1)\\times(1)=1\"](\"assets/equations/e63832c06fcb7aed.svg\")
Add the rows to obtain one output cell.
","![\"Y_{2,1}=(-1)+(1)=0\"](\"assets/equations/7cb2a77b737bf455.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 1 | −3 |
| 0 | 0 |
Each cell is input × matching kernel coefficient.
(−2) + 0 = −2
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 2, column 2.
Window row 1.
","![\"R_1=(1)\\times(1)+(3)\\times(-1)=-2\"](\"assets/equations/540cabd018cf9dd5.svg\")
Window row 2.
","![\"R_2=(1)\\times(0)+(0)\\times(1)=0\"](\"assets/equations/75388bfbb51edc5c.svg\")
Add the rows to obtain one output cell.
","![\"Y_{2,2}=(-2)+(0)=-2\"](\"assets/equations/6a1f7599e33d0692.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 3 | −2 |
| 0 | 1 |
Each cell is input × matching kernel coefficient.
1 + 1 = 2
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 2, column 3.
Window row 1.
","![\"R_1=(3)\\times(1)+(2)\\times(-1)=1\"](\"assets/equations/941c8bddd3a4fd90.svg\")
Window row 2.
","Add the rows to obtain one output cell.
","![\"Y_{2,3}=(1)+(1)=2\"](\"assets/equations/ba68e9967c8a5ec6.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 2 | −1 |
| 0 | 0 |
Each cell is input × matching kernel coefficient.
1 + 0 = 1
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 3, column 1.
Window row 1.
","![\"R_1=(2)\\times(1)+(1)\\times(-1)=1\"](\"assets/equations/ed9de22182a739e4.svg\")
Window row 2.
","Add the rows to obtain one output cell.
","![\"Y_{3,1}=(1)+(0)=1\"](\"assets/equations/108e2edb8fe7afbe.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 |
| 0 | 2 |
Each cell is input × matching kernel coefficient.
1 + 2 = 3
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 3, column 2.
Window row 1.
","![\"R_1=(1)\\times(1)+(0)\\times(-1)=1\"](\"assets/equations/248757728423ee8c.svg\")
Window row 2.
","![\"R_2=(0)\\times(0)+(2)\\times(1)=2\"](\"assets/equations/8b2b9a5baccb2ea6.svg\")
Add the rows to obtain one output cell.
","![\"Y_{3,2}=(1)+(2)=3\"](\"assets/equations/50d4d84e0be67f18.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 0 | −1 |
| 0 | 3 |
Each cell is input × matching kernel coefficient.
(−1) + 3 = 2
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 3, column 3.
Window row 1.
","Window row 2.
","![\"R_2=(2)\\times(0)+(3)\\times(1)=3\"](\"assets/equations/c18dea868e99c1c8.svg\")
Add the rows to obtain one output cell.
","![\"Y_{3,3}=(-1)+(3)=2\"](\"assets/equations/5248561d79f5d37a.svg\")
Final output: ![\"\\begin{bmatrix}0](\"assets/equations/1659d27513ae9f09.svg\")
. Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
Question. Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total.
"],"answer":["Input shape = 31 × 31 × 3.
","Filter size = 5 × 5; number of filters = 8.
","Stride S = 2; padding P = 1 per side; one bias per filter.
","Find output shape, weights, biases and total parameters.
","Convolution/pooling spatial size. Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.
","N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.
","Yaad rakhne ka cue: Input + dono sides ki padding − kernel; stride se divide; floor + 1.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","The input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels.
","![\"\\text{weights}=K^2C_{\\mathrm{in}}C_{\\mathrm{out}},\\quad](\"assets/equations/4fb085a2eda022c5.svg\")
![\"\\text{parameters}=\\text{weights}+\\text{biases}\"](\"assets/equations/35882bbb02ddb8aa.svg\")
Padding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{31+2(1)-5}{2}\\right\\rfloor+1\"](\"assets/equations/f73307dcf9d893e8.svg\")
![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{28}{2}\\right\\rfloor+1=14+1=15\"](\"assets/equations/c0ed9983d10f2a99.svg\")
The height uses the same calculation. Output shape = 15 × 15 × 8.
","Each filter covers 5 × 5 positions in each of 3 channels.
","![\"5\\times5\\times3=75\\text{](\"assets/equations/4ce380d19eaf1a9e.svg\")
![\"\\text{weights}=75\\times8=600\"](\"assets/equations/19223176d175b12b.svg\")
![\"\\text{biases}=8,\\qquad](\"assets/equations/e580536b54e8c233.svg\")
Final answer: shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights.
"],"kind":"practice","id":"mt2-c17-s33","search":"Drill 33 Padding stride and channels **Question.** Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total.\n\n**Full solution.**\n\n\n\n### Given\n\nInput shape = 31 × 31 × 3.\n\nFilter size = 5 × 5; number of filters = 8.\n\nStride S = 2; padding P = 1 per side; one bias per filter.\n\n### To find\n\nFind output shape, weights, biases and total parameters.\n\n### Formula — which one and why\n\n**Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.\n\nN = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.\n\n\n\n**Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of each given value\nThe input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels.\n### Formulas used\n\n\n\n### Step 1: find the output height and width\nPadding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33.\n\n\nThe height uses the same calculation. **Output shape = 15 × 15 × 8.**\n### Step 2: count weights in one filter\nEach filter covers 5 × 5 positions in each of 3 channels.\n\n### Step 3: count all filters and their biases\n\n\n**Final answer:** shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights.\n"},{"title":"Drill 34 Max and average pooling","question":["Question. Pool the input A shown below with 2 × 2 windows and stride 2. Find max and average outputs.
","Input A is printed with the question.
","Window size = 2 × 2; stride = 2; no padding.
","Find the complete max-pooling and average-pooling outputs.
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer.
","![\"p_{\\mathrm{max}}=\\max(a_1,a_2,a_3,a_4)\"](\"assets/equations/870fc5e76c2ff285.svg\")
![\"p_{\\mathrm{avg}}=\\frac{a_1+a_2+a_3+a_4}{4}\"](\"assets/equations/cf572bd2a565d88a.svg\")
![\"N_{\\mathrm{pool}}=\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\"](\"assets/equations/22a9854190bb7c9a.svg\")
Use the input size at the pooling layer. The output keeps the same channel count.
","Shape: = 2 on each axis; channels stay unchanged.
Step 1: select the entries.
","[1, 4, 0, 2]
","Step 2: add all selected entries.
","1 + 4 + 0 + 2 = 7
","Step 3: divide by the 4 entries in this window.
","7/4 = 1.75. Maximum = 4
","Step 1: select the entries.
","[2, 3, 5, 1]
","Step 2: add all selected entries.
","2 + 3 + 5 + 1 = 11
","Step 3: divide by the 4 entries in this window.
","11/4 = 2.75. Maximum = 5
","Step 1: select the entries.
","[6, 3, 1, 7]
","Step 2: add all selected entries.
","6 + 3 + 1 + 7 = 17
","Step 3: divide by the 4 entries in this window.
","17/4 = 4.25. Maximum = 7
","Step 1: select the entries.
","[2, 0, 4, 8]
","Step 2: add all selected entries.
","2 + 0 + 4 + 8 = 14
","Step 3: divide by the 4 entries in this window.
","14/4 = 3.5. Maximum = 8
","Average output: ![\"\\begin{bmatrix}1.75](\"assets/equations/c4e510493e329d60.svg\")
. Max output: ![\"\\begin{bmatrix}4](\"assets/equations/71895dac90bdedf0.svg\")
. Pooling normally has zero trainable parameters and reduces spatial dimensions.
Question. Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters.
"],"answer":["Input = 10 × 10 × 1.
","Conv: 3 × 3, 4 filters, stride 1, valid padding.
","Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs.
","Include one bias per filter/dense neuron.
","Find all output shapes, layer parameter counts and their sum.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","“Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters.
","![\"n_{\\mathrm{flat}}=HWC,\\qquad](\"assets/equations/d4064c7b5aff30cf.svg\")
![\"N_{\\mathrm{conv}}=\\frac{10+0-3}{1}+1=7+1=8\"](\"assets/equations/79853ff0cb463822.svg\")
Four filters produce four channels: 8 × 8 × 4.
","![\"\\text{weights}=3\\times3\\times1\\times4=36\"](\"assets/equations/d2206cecca6115a8.svg\")
![\"\\text{biases}=4,\\qquad](\"assets/equations/d97de20b29f5a9fb.svg\")
Pooling now receives size 8, not the original size 10.
","![\"N_{\\mathrm{pool}}=\\left\\lfloor\\frac{8-2}{2}\\right\\rfloor+1=3+1=4\"](\"assets/equations/4578ff92c09a54ec.svg\")
Channels remain 4: 4 × 4 × 4. Pooling parameters = 0.
","![\"n_{\\mathrm{flat}}=4\\times4\\times4=64\"](\"assets/equations/b42325b09a6929a4.svg\")
Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0.
","![\"\\text{weights}=64\\times3=192\"](\"assets/equations/738d2c475096e79a.svg\")
![\"\\text{biases}=3,\\qquad](\"assets/equations/f927a7fa1a913d20.svg\")
![\"P_{\\mathrm{total}}=40+0+0+195=235\"](\"assets/equations/468553004a2eb33c.svg\")
Final answer: shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters.
"],"kind":"practice","id":"mt2-c17-s35","search":"Drill 35 Complete small CNN **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters.\n\n**Full solution.**\n\n\n\n### Given\n\nInput = 10 × 10 × 1.\n\nConv: 3 × 3, 4 filters, stride 1, valid padding.\n\nPool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs.\n\nInclude one bias per filter/dense neuron.\n\n### To find\n\nFind all output shapes, layer parameter counts and their sum.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas and assumptions\n“Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters.\n\n\n\n### Step 1: convolution shape\n\nFour filters produce four channels: **8 × 8 × 4**.\n### Step 2: convolution parameters\n\n\n### Step 3: pooling shape\nPooling now receives size 8, not the original size 10.\n\nChannels remain 4: **4 × 4 × 4**. Pooling parameters = 0.\n### Step 4: flatten\n\nFlatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0.\n### Step 5: dense layer with 3 outputs\n\n\n### Step 6: add learned parameters\n\n**Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters.\n"},{"title":"Drill 36 Reverse a spatial formula","question":["Question. A stride-1, valid 3 × 3 convolution outputs 18 × 18 × 12. What input spatial size is needed? Is input channel count determined?
"],"answer":["Output shape = 18 × 18 × 12.
","Kernel = 3 × 3; stride = 1; valid padding P = 0.
","Input channel count is not supplied.
","Solve for input height/width; state whether input channels and parameter count are uniquely determined.
","Convolution/pooling spatial size. Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.
","N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.
","Yaad rakhne ka cue: Input + dono sides ki padding − kernel; stride se divide; floor + 1.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","For stride 1 and valid padding, the output width is input width minus kernel width plus one.
","![\"N_{\\mathrm{out}}=N-K+1\"](\"assets/equations/6074916da488a8bc.svg\")
![\"18=N-3+1=N-2\"](\"assets/equations/73b9e54fad32ca52.svg\")
Add 2 to both sides. Doing the same operation to both sides preserves the equality.
","![\"18+2=N-2+2\"](\"assets/equations/cfbfbce776e71cb1.svg\")
![\"N=20\"](\"assets/equations/c758e056cb2f1688.svg\")
The 12 output channels tell us there are 12 filters. They do not say how many channels the input has. An RGB input and a many-channel intermediate map could both feed 12 filters.
","With one bias per filter, the parameter formula would be:
","![\"P=(3\\times3\\times](\"assets/equations/f0b530e411679c54.svg\")
Final answer: input spatial shape 20 × 20; input channel count is undetermined from these givens.
"],"kind":"practice","id":"mt2-c17-s36","search":"Drill 36 Reverse a spatial formula **Question.** A stride-1, valid 3 × 3 convolution outputs 18 × 18 × 12. What input spatial size is needed? Is input channel count determined?\n\n**Full solution.**\n\n\n\n### Given\n\nOutput shape = 18 × 18 × 12.\n\nKernel = 3 × 3; stride = 1; valid padding P = 0.\n\nInput channel count is not supplied.\n\n### To find\n\nSolve for input height/width; state whether input channels and parameter count are uniquely determined.\n\n### Formula — which one and why\n\n**Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.\n\nN = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.\n\n\n\n**Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n\n### Formula used\nFor stride 1 and valid padding, the output width is input width minus kernel width plus one.\n\n### Step 1: substitute the known output and kernel sizes\n\n### Step 2: isolate the unknown input size\nAdd 2 to both sides. Doing the same operation to both sides preserves the equality.\n\n\n### Step 3: interpret the channel information\nThe 12 output channels tell us there are 12 filters. They do not say how many channels the input has. An RGB input and a many-channel intermediate map could both feed 12 filters.\nWith one bias per filter, the parameter formula would be:\n\n**Final answer:** input spatial shape 20 × 20; input channel count is undetermined from these givens.\n"},{"title":"Drill 37 Bottleneck cost with new values","question":["Question. A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first.
"],"answer":["Input = 16 × 16 × 64.
","Direct branch: 32 same-padded 3 × 3 filters.
","Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer.
","Spatial size remains 16 × 16; exclude bias additions.
","Find both MAC counts, absolute saving and percentage saving.
","Multiply–accumulate count. Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.
","H_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.
","Yaad rakhne ka cue: Har output cell ka kaam × kitne output cells.
","A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer.
","Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs.
","![\"\\mathrm{MACs}_{\\mathrm{direct}}=8192\\times576=4{,}718{,}592\"](\"assets/equations/673cd31676ab87d1.svg\")
![\"\\mathrm{MACs}_{1\\times1}=16\\times16\\times8\\times1\\times1\\times64=131{,}072\"](\"assets/equations/0bcc137de649f71d.svg\")
![\"\\mathrm{MACs}_{3\\times3}=16\\times16\\times32\\times3\\times3\\times8=589{,}824\"](\"assets/equations/da27ae2a69680cb6.svg\")
![\"\\mathrm{MACs}_{\\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896\"](\"assets/equations/741f21ae6e5bfc83.svg\")
![\"\\mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696\"](\"assets/equations/005360b3f69dc10a.svg\")
![\"\\mathrm{saving\\](\"assets/equations/2a96913f36caca06.svg\")
Final answer: direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32.
"],"kind":"practice","id":"mt2-c17-s37","search":"Drill 37 Bottleneck cost with new values **Question.** A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first.\n\n**Full solution.**\n\n\n\n### Given\n\nInput = 16 × 16 × 64.\n\nDirect branch: 32 same-padded 3 × 3 filters.\n\nAlternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer.\n\nSpatial size remains 16 × 16; exclude bias additions.\n\n### To find\n\nFind both MAC counts, absolute saving and percentage saving.\n\n### Formula — which one and why\n\n**Multiply–accumulate count.** Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.\n\nH_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formula\nA MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer.\n\n### Step 1: direct 3 × 3 convolution\nEach output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs.\n\n### Step 2: 1 × 1 reduction from 64 to 8 channels\n\n### Step 3: 3 × 3 convolution now receives only 8 channels\n\n### Step 4: add the two bottleneck stages\n\n### Step 5: calculate the saving\n\n\n**Final answer:** direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32.\n"},{"title":"Drill 38 Residual addition","question":["Question. x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block.
"],"answer":["x = [2,−3,1]; F(x) = [−1,1,0.5].
","Find x+F(x), apply ReLU, and compare with ReLU(F(x)).
","Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.
","x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.
","Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join.
","![\"H(x)=x+F(x),\\qquad](\"assets/equations/78deb7e05ee3749d.svg\")
![\"H_1=2+(-1)=1\"](\"assets/equations/1a26d4fa253f0177.svg\")
![\"H_2=-3+1=-2\"](\"assets/equations/2b33025f78e725f0.svg\")
![\"H_3=1+0.5=1.5\"](\"assets/equations/4f50029ce8c5be41.svg\")
The residual output before ReLU is [1, −2, 1.5].
","![\"[\\max(0,1),\\max(0,-2),\\max(0,1.5)]=[1,0,1.5]\"](\"assets/equations/09e521346431d0a8.svg\")
Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is [0, 1, 0.5].
","Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation.
"],"kind":"practice","id":"mt2-c17-s38","search":"Drill 38 Residual addition **Question.** x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block.\n\n**Full solution.**\n\n\n\n### Given\n\nx = [2,−3,1]; F(x) = [−1,1,0.5].\n\n### To find\n\nFind x+F(x), apply ReLU, and compare with ReLU(F(x)).\n\n### Formula — which one and why\n\n**Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.\n\nx = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.\n\n\n\n**Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join.\n\n### Step 1: add each pair\n\n\n\nThe residual output before ReLU is **[1, −2, 1.5]**.\n### Step 2: apply ReLU\n\n### Step 3: compare with the plain branch\nWithout a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is **[0, 1, 0.5]**.\nResidual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation.\n"},{"title":"Drill 39 Two RNN steps","question":["Question. h₀ = 0, Wxh = 0.5, Whh = 0.2, b = 0, x₁ = 1, x₂ = 2, tanh activation. Find h₁, h₂.
"],"answer":["Initial h₀ = 0; input weight Wxh = 0.5; recurrent weight Whh = 0.2; b = 0.
","Inputs: x₁ = 1, x₂ = 2; activation: tanh.
","Calculate h₁, then use it to calculate h₂.
","Recurrent forward pass. At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.
","x_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.
","Yaad rakhne ka cue: Naya input + pichhla activated state; phir activation.
","At time t, the RNN combines the current input xₜ and the previous hidden state hₜ₋₁. The same two weights are reused at both times.
","![\"a_t=W_{xh}x_t+W_{hh}h_{t-1}+b,\\qquad](\"assets/equations/53aa19b3509081a2.svg\")
The raw sum a is not the hidden state. Tanh converts that sum into the state that is carried forward.
","![\"\\tanh(a)=\\frac{e^{2a}-1}{e^{2a}+1}\"](\"assets/equations/994b9a9f57dad438.svg\")
![\"h_1=\\tanh(0.5)=\\frac{e^1-1}{e^1+1}\"](\"assets/equations/a56af3f825290564.svg\")
![\"h_1=\\frac{2.718281828-1}{2.718281828+1}=0.462117157\"](\"assets/equations/8658aac9b96930a3.svg\")
![\"a_2=0.5(2)+0.2(0.462117157)+0\"](\"assets/equations/05a3c9bcda834dc2.svg\")
![\"a_2=1+0.092423431=1.092423431\"](\"assets/equations/7019e7bc4f1316c8.svg\")
![\"h_2=\\tanh(1.092423431)=0.797760954\"](\"assets/equations/b1963e3f6225d34f.svg\")
Final answer: h₁ ≈ 0.462117157; h₂ ≈ 0.797760954. Pass h₁ to the next step, not the unactivated value 0.5.
"],"kind":"practice","id":"mt2-c17-s39","search":"Drill 39 Two RNN steps **Question.** h₀ = 0, Wxh = 0.5, Whh = 0.2, b = 0, x₁ = 1, x₂ = 2, tanh activation. Find h₁, h₂.\n\n**Full solution.**\n\n\n\n### Given\n\nInitial h₀ = 0; input weight Wxh = 0.5; recurrent weight Whh = 0.2; b = 0.\n\nInputs: x₁ = 1, x₂ = 2; activation: tanh.\n\n### To find\n\nCalculate h₁, then use it to calculate h₂.\n\n### Formula — which one and why\n\n**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.\n\nx_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nAt time t, the RNN combines the current input xₜ and the previous hidden state hₜ₋₁. The same two weights are reused at both times.\n\nThe raw sum a is not the hidden state. Tanh converts that sum into the state that is carried forward.\n\n### Step 1: calculate the first raw sum\n\n### Step 2: activate it to obtain the first state\n\n\n### Step 3: use that state in the second raw sum\n\n\n### Step 4: activate the second sum\n\n**Final answer:** h₁ ≈ 0.462117157; h₂ ≈ 0.797760954. Pass h₁ to the next step, not the unactivated value 0.5.\n"},{"title":"Drill 40 Mixed convolution pool dense and loss","question":["Question. A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE.
","| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
| 1 | 0 |
| 0 | −1 |
Stride 1; P = 0 (no padding).
Input A and kernel Matrix 2 are printed with the question.
","Conv: stride 1, valid padding; follow with ReLU.
","Max pool: 2 × 2, stride 1; flatten row by row.
","Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output.
","Binary target = 1.
","Find convolution, activated map, pooled map, flattened values, prediction and BCE.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Binary cross-entropy. For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.
","y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.
","Yaad rakhne ka cue: Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).
","A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation.
","Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability.
","N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one.
","Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer.
","Use the input size at the pooling layer. The output keeps the same channel count.
","Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p).
","![\"z=\\sum_iw_ix_i+b,\\qquad](\"assets/equations/a593381e0c37ce0b.svg\")
Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Step 1 shape: = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 |
| 0 | −1 |
Each cell is input × matching kernel coefficient.
1 + (−1) = 0
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 1, column 1.
Window row 1.
","![\"R_1=(1)\\times(1)+(2)\\times(0)=1\"](\"assets/equations/10a146899dccf86a.svg\")
Window row 2.
","![\"R_2=(0)\\times(0)+(1)\\times(-1)=-1\"](\"assets/equations/a91d42dac9ba1633.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,1}=(1)+(-1)=0\"](\"assets/equations/059edda10c2ac26e.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 2 | 0 |
| 0 | −3 |
Each cell is input × matching kernel coefficient.
2 + (−3) = −1
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 1, column 2.
Window row 1.
","![\"R_1=(2)\\times(1)+(0)\\times(0)=2\"](\"assets/equations/13b97b59677316ed.svg\")
Window row 2.
","![\"R_2=(1)\\times(0)+(3)\\times(-1)=-3\"](\"assets/equations/9cf98567181f7899.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,2}=(2)+(-3)=-1\"](\"assets/equations/b6828ab91aa100a9.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0 | 0 |
| 0 | −2 |
Each cell is input × matching kernel coefficient.
0 + (−2) = −2
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 1, column 3.
Window row 1.
","![\"R_1=(0)\\times(1)+(1)\\times(0)=0\"](\"assets/equations/9602643188947584.svg\")
Window row 2.
","![\"R_2=(3)\\times(0)+(2)\\times(-1)=-2\"](\"assets/equations/49c3d4aac1f50f22.svg\")
Add the rows to obtain one output cell.
","![\"Y_{1,3}=(0)+(-2)=-2\"](\"assets/equations/77839b654a5d343b.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0 | 0 |
| 0 | −1 |
Each cell is input × matching kernel coefficient.
0 + (−1) = −1
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 2, column 1.
Window row 1.
","Window row 2.
","![\"R_2=(2)\\times(0)+(1)\\times(-1)=-1\"](\"assets/equations/825cd1ca9fbe27a2.svg\")
Add the rows to obtain one output cell.
","![\"Y_{2,1}=(0)+(-1)=-1\"](\"assets/equations/74f7aefb8b0d75bd.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 |
| 0 | 0 |
Each cell is input × matching kernel coefficient.
1 + 0 = 1
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 2, column 2.
Window row 1.
","![\"R_1=(1)\\times(1)+(3)\\times(0)=1\"](\"assets/equations/50ba62bd05697a22.svg\")
Window row 2.
","![\"R_2=(1)\\times(0)+(0)\\times(-1)=0\"](\"assets/equations/f7d66e56559f9de7.svg\")
Add the rows to obtain one output cell.
","![\"Y_{2,2}=(1)+(0)=1\"](\"assets/equations/e353ed5c5344d9f0.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 3 | 0 |
| 0 | −1 |
Each cell is input × matching kernel coefficient.
3 + (−1) = 2
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 2, column 3.
Window row 1.
","![\"R_1=(3)\\times(1)+(2)\\times(0)=3\"](\"assets/equations/239b4ccf3eb098a0.svg\")
Window row 2.
","Add the rows to obtain one output cell.
","![\"Y_{2,3}=(3)+(-1)=2\"](\"assets/equations/bc81876afa80a85e.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 2 | 0 |
| 0 | 0 |
Each cell is input × matching kernel coefficient.
2 + 0 = 2
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 3, column 1.
Window row 1.
","![\"R_1=(2)\\times(1)+(1)\\times(0)=2\"](\"assets/equations/ca82f5af7cbb2e71.svg\")
Window row 2.
","Add the rows to obtain one output cell.
","![\"Y_{3,1}=(2)+(0)=2\"](\"assets/equations/d3322f48fabaf515.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 |
| 0 | −2 |
Each cell is input × matching kernel coefficient.
1 + (−2) = −1
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 3, column 2.
Window row 1.
","![\"R_1=(1)\\times(1)+(0)\\times(0)=1\"](\"assets/equations/1b9538ffab8a7d30.svg\")
Window row 2.
","![\"R_2=(0)\\times(0)+(2)\\times(-1)=-2\"](\"assets/equations/1baae78cba2e069c.svg\")
Add the rows to obtain one output cell.
","![\"Y_{3,2}=(1)+(-2)=-1\"](\"assets/equations/43f244bc80788fd1.svg\")
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0 | 0 |
| 0 | −3 |
Each cell is input × matching kernel coefficient.
0 + (−3) = −3
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 3, column 3.
Window row 1.
","Window row 2.
","![\"R_2=(2)\\times(0)+(3)\\times(-1)=-3\"](\"assets/equations/04e972a19789fc08.svg\")
Add the rows to obtain one output cell.
","![\"Y_{3,3}=(0)+(-3)=-3\"](\"assets/equations/73f07f2c01400713.svg\")
Final output: ![\"\\begin{bmatrix}0](\"assets/equations/78836b0ad068fd57.svg\")
. Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
ReLU turns negative entries into 0: ![\"\\begin{bmatrix}0](\"assets/equations/5203ca394b677586.svg\")
.
Shape: ![\"\\left\\lfloor\\frac{3-2}{1}\\right\\rfloor+1\"](\"assets/equations/65d999fa66cd37e1.svg\")
= 2 on each axis; channels stay unchanged.
Step 1: select the entries.
","[0, 0, 0, 1]
","Step 2: add all selected entries.
","0 + 0 + 0 + 1 = 1
","Step 3: divide by the 4 entries in this window.
","1/4 = 0.25. Maximum = 1
","Step 1: select the entries.
","[0, 0, 1, 2]
","Step 2: add all selected entries.
","0 + 0 + 1 + 2 = 3
","Step 3: divide by the 4 entries in this window.
","3/4 = 0.75. Maximum = 2
","Step 1: select the entries.
","[0, 1, 2, 0]
","Step 2: add all selected entries.
","0 + 1 + 2 + 0 = 3
","Step 3: divide by the 4 entries in this window.
","3/4 = 0.75. Maximum = 2
","Step 1: select the entries.
","[1, 2, 0, 0]
","Step 2: add all selected entries.
","1 + 2 + 0 + 0 = 3
","Step 3: divide by the 4 entries in this window.
","3/4 = 0.75. Maximum = 2
","Average output: ![\"\\begin{bmatrix}0.25](\"assets/equations/386a0646a3bb55b5.svg\")
. Max output: ![\"\\begin{bmatrix}1](\"assets/equations/4fb14ffabb3fef49.svg\")
. Pooling normally has zero trainable parameters and reduces spatial dimensions.
Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = 0.139386758. Average pool values were computed for contrast; the question requires max pool.
"],"kind":"practice","id":"mt2-c17-s40","search":"Drill 40 Mixed convolution pool dense and loss **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE.\n\n\n\n\n**Full solution.**\n\n\n\n### Given\n\nInput A and kernel Matrix 2 are printed with the question.\n\nConv: stride 1, valid padding; follow with ReLU.\n\nMax pool: 2 × 2, stride 1; flatten row by row.\n\nDense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output.\n\nBinary target = 1.\n\n### To find\n\nFind convolution, activated map, pooled map, flattened values, prediction and BCE.\n\n### Formula — which one and why\n\n**Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.\n\nX = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.\n\n\n\n**Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar.\n\n**Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.\n\nV = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.\n\n\n\n\n\n**Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count.\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.\n\ny is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.\n\n\n\n\n\n**Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).\n\n### Solution — step by step\n\n\n\n\n\n### Formulas and the operation we are performing\nA filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation.\n\nHere i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability.\n\nN is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one.\n### Formulas and meaning\nEach 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the g"},{"title":"Drill 41 Mixed backpropagation and L2","question":["Question. Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight.
"],"answer":["Prediction ŷ = wx; x = 2; w = 0.4; target y = 1.
","Data loss = ½(ŷ−y)²; penalty = (λ/2)w².
","λ = 0.1; η = 0.2.
","Find total objective, its weight gradient and the new weight.
","L2 penalty with a factor of one-half. Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.
","λ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.
","Yaad rakhne ka cue: Half wala L2 derivative mein lambda × weight deta hai.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once.
","![\"\\hat](\"assets/equations/9807919ec463b336.svg\")
![\"\\frac{\\partial](\"assets/equations/fccbb24d2fd5ed97.svg\")
![\"\\hat](\"assets/equations/3414961b88098bc1.svg\")
![\"L_{\\mathrm{data}}=\\frac12(-0.2)^2=\\frac12(0.04)=0.02\"](\"assets/equations/f025cfcfa02c965d.svg\")
![\"R=\\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008\"](\"assets/equations/c4fc429067a3f156.svg\")
![\"J=0.02+0.008=0.028\"](\"assets/equations/72c65e08a565a4c1.svg\")
![\"g_{\\mathrm{data}}=(-0.2)\\times2=-0.4\"](\"assets/equations/20e7a6699528fa05.svg\")
![\"g_{\\mathrm{penalty}}=0.1\\times0.4=0.04\"](\"assets/equations/ed23175a058690ac.svg\")
![\"g_{\\mathrm{total}}=-0.4+0.04=-0.36\"](\"assets/equations/2c43044f63438eb7.svg\")
![\"w_{\\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472\"](\"assets/equations/93c291aebdd73770.svg\")
Final answer: total loss 0.028; gradient −0.36; new weight 0.472.
"],"kind":"practice","id":"mt2-c17-s41","search":"Drill 41 Mixed backpropagation and L2 **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight.\n\n**Full solution.**\n\n\n\n### Given\n\nPrediction ŷ = wx; x = 2; w = 0.4; target y = 1.\n\nData loss = ½(ŷ−y)²; penalty = (λ/2)w².\n\nλ = 0.1; η = 0.2.\n\n### To find\n\nFind total objective, its weight gradient and the new weight.\n\n### Formula — which one and why\n\n**L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.\n\nλ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai.\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThis objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once.\n\n\n### Step 1: prediction and error\n\n### Step 2: data loss\n\n### Step 3: regularization penalty and total objective\n\n\n### Step 4: calculate the two gradient contributions\n\n\n\n### Step 5: update\n\n**Final answer:** total loss 0.028; gradient −0.36; new weight 0.472.\n"},{"title":"Drill 42 Mixed dropout and a momentum update","question":["Question. Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1.
"],"answer":["Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5.
","Linear-output weights = [0.2,0.3]; bias = 0; target = 1.
","Half squared loss; update only the output weights.
","Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1.
","Find dropped/scaled activations, prediction, gradients, new velocities and new output weights.
","Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.
","aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.
","Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients.
","![\"a'_j=\\frac{m_ja_j}{q},\\quad](\"assets/equations/0da6165a5d02d6ea.svg\")
The signed-velocity momentum convention is:
","![\"v_{j,\\mathrm{new}}=\\beta](\"assets/equations/fc93fbe5bcc88056.svg\")
![\"a'_1=\\frac{1\\times1}{0.5}=2,\\qquad](\"assets/equations/446657a723f0a489.svg\")
![\"o=0.2(2)+0.3(0)+0=0.4\"](\"assets/equations/a323109c2640591d.svg\")
![\"o-y=0.4-1=-0.6\"](\"assets/equations/e63213278aa7de98.svg\")
![\"g_1=(-0.6)(2)=-1.2,\\qquad](\"assets/equations/7f44ece1dbabee9e.svg\")
![\"v_{1,\\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21\"](\"assets/equations/6463456a76ac5311.svg\")
![\"v_{2,\\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09\"](\"assets/equations/66eba48d514886f3.svg\")
![\"w_{1,\\mathrm{new}}=0.2+0.21=0.41\"](\"assets/equations/ad8445391f1a6df2.svg\")
![\"w_{2,\\mathrm{new}}=0.3-0.09=0.21\"](\"assets/equations/7dc7d2ca77c91e6c.svg\")
Final answer: [0.41, 0.21]. The second weight moves even with zero current gradient because its old momentum is nonzero.
"],"kind":"practice","id":"mt2-c17-s42","search":"Drill 42 Mixed dropout and a momentum update **Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1.\n\n**Full solution.**\n\n\n\n### Given\n\nHidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5.\n\nLinear-output weights = [0.2,0.3]; bias = 0; target = 1.\n\nHalf squared loss; update only the output weights.\n\nOld signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1.\n\n### To find\n\nFind dropped/scaled activations, prediction, gradients, new velocities and new output weights.\n\n### Formula — which one and why\n\n**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.\n\naᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.\n\n\n\n\n\n**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used\nDropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients.\n\nThe signed-velocity momentum convention is:\n\n### Step 1: apply inverted dropout\n\n### Step 2: prediction and error\n\n\n### Step 3: current weight gradients\n\n### Step 4: remembered movement plus new correction\n\n\n### Step 5: add velocities to old weights\n\n\n**Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero.\n"},{"title":"Drill 43 Mixed CNN dimensions softmax and loss","question":["Question. Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE.
"],"answer":["Input = 8 × 8 × 3.
","Conv: kernel 3, stride 1, padding 1, 2 filters.
","Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs.
","Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]).
","Find shapes, parameter counts, probabilities and CCE.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.
","zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.
","Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.
","Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.
","yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.
","Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.
","Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride.
","The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class.
","![\"p_i=\\frac{e^{z_i-c}}{\\sum_j](\"assets/equations/1ffdc701f4381a15.svg\")
Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = 155.
","Step 1 stabilize: subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax.
","Step 2 exponentiate: exp values = [0.367879441, 1, 0.135335283].
","Step 3 denominator: sum = 1.503214724.
","Step 4 divide each value by this same sum: probabilities = [0.244728471, 0.665240956, 0.090030573]. Check: all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.
","Correct class probability = 0.665240956; CCE = −ln(0.665240956) = 0.407605964.
"],"kind":"practice","id":"mt2-c17-s43","search":"Drill 43 Mixed CNN dimensions softmax and loss **Question.** Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE.\n\n**Full solution.**\n\n\n\n### Given\n\nInput = 8 × 8 × 3.\n\nConv: kernel 3, stride 1, padding 1, 2 filters.\n\nPool: kernel 2, stride 2; flatten; dense layer with 3 outputs.\n\nLogits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]).\n\n### To find\n\nFind shapes, parameter counts, probabilities and CCE.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.\n\nzⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.\n\n**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.\n\nyⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.\n\n\n\n\n\n**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used\nSeparate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride.\n\n\n\nThe +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class.\n\n\nConv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = **155**.\n**Step 1 stabilize: ** subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax.\n\n**Step 2 exponentiate: ** exp values = [0.367879441, 1, 0.135335283].\n\n**Step 3 denominator: ** sum = 1.503214724.\n\n**Step 4 divide each value by this same sum: ** probabilities = **[0.244728471, 0.665240956, 0.090030573]**. **Check: ** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.\nCorrect class probability = 0.665240956; CCE = −ln(0.665240956) = **0.407605964**.\n"},{"title":"Drill 44 Inception concatenation and parameter count","question":["Question. Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters.
"],"answer":["Input = 10 × 10 × 8.
","Branch A: 1 × 1 convolution, 4 filters.
","Branch B: 3 × 3 same convolution, 6 filters.
","Branch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters.
","Find each branch shape and parameters, then the concatenated shape and total parameters.
","Inception channel concatenation. Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches.
","C₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions.
","![\"C_{\\mathrm{out}}=\\sum_k](\"assets/equations/5fcf90049e052b72.svg\")
![\"P_{\\mathrm{total}}=\\sum_k](\"assets/equations/c6958d2f8ded49d5.svg\")
Yaad rakhne ka cue: Channels jodo; height-width same rakho.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","Each parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values.
","![\"C_{\\mathrm{combined}}=C_A+C_B+C_C\"](\"assets/equations/2245536fdd10a080.svg\")
Assume one bias per filter. Pooling has no learned parameters.
","Step 1: Branch A.
","Its 4 filters produce shape 10 × 10 × 4.
","![\"P_A=(1\\times1\\times8+1)\\times4=9\\times4=36\"](\"assets/equations/ff251b0ed25a7ae7.svg\")
Step 2: Branch B.
","Its 6 filters produce shape 10 × 10 × 6.
","![\"P_B=(3\\times3\\times8+1)\\times6=73\\times6=438\"](\"assets/equations/d6970788f463cb74.svg\")
Step 3: Branch C.
","Pooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2.
","![\"P_C=0+(1\\times1\\times8+1)\\times2=18\"](\"assets/equations/bdab800ad8bc0347.svg\")
Step 4: Join the shapes and add the parameters.
","![\"C_{\\mathrm{combined}}=4+6+2=12\"](\"assets/equations/567c2c4512c2cb27.svg\")
![\"P_{\\mathrm{total}}=36+438+18=492\"](\"assets/equations/f9a71aaf4ed27273.svg\")
Final answer: combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches.
"],"kind":"practice","id":"mt2-c17-s44","search":"Drill 44 Inception concatenation and parameter count **Question.** Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters.\n\n**Full solution.**\n\n\n\n### Given\n\nInput = 10 × 10 × 8.\n\nBranch A: 1 × 1 convolution, 4 filters.\n\nBranch B: 3 × 3 same convolution, 6 filters.\n\nBranch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters.\n\n### To find\n\nFind each branch shape and parameters, then the concatenated shape and total parameters.\n\n### Formula — which one and why\n\n**Inception channel concatenation.** Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches.\n\nC₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions.\n\n\n\n\n\n**Yaad rakhne ka cue:** Channels jodo; height-width same rakho.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nEach parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values.\n\n\nAssume one bias per filter. Pooling has no learned parameters.\n**Step 1: Branch A.**\nIts 4 filters produce shape 10 × 10 × 4.\n\n**Step 2: Branch B.**\nIts 6 filters produce shape 10 × 10 × 6.\n\n**Step 3: Branch C.**\nPooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2.\n\n**Step 4: Join the shapes and add the parameters.**\n\n\n**Final answer:** combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches.\n"},{"title":"Drill 45 Projection shortcut","question":["Question. A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters.
"],"answer":["Residual input = 16 × 16 × 8; main-path output = 8 × 8 × 16.
","Shortcut kernel = 1 × 1; stride 2; padding 0.
","Use one bias per projection filter.
","Choose the number of filters and find projection output shape and parameters.
","Convolution/pooling spatial size. Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.
","N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.
","Yaad rakhne ka cue: Input + dono sides ki padding − kernel; stride se divide; floor + 1.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","Residual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16.
","![\"P_{\\mathrm{projection}}=(K^2C_{\\mathrm{in}}+1)C_{\\mathrm{out}}\"](\"assets/equations/8abfcf2c3c8a28be.svg\")
One filter creates one channel, so 16 filters give the required 16 output channels.
","![\"N_{\\mathrm{out}}=\\left\\lfloor\\frac{16+0-1}{2}\\right\\rfloor+1\"](\"assets/equations/ecbcadd42ab07705.svg\")
![\"N_{\\mathrm{out}}=\\lfloor7.5\\rfloor+1=7+1=8\"](\"assets/equations/55b23544c4917d4f.svg\")
The floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1.
","![\"\\text{weights}=1\\times1\\times8\\times16=128\"](\"assets/equations/dfae1039bf042104.svg\")
![\"\\text{biases}=16,\\qquad](\"assets/equations/040d4fe2bfbf253e.svg\")
Final answer: projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid.
"],"kind":"practice","id":"mt2-c17-s45","search":"Drill 45 Projection shortcut **Question.** A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters.\n\n**Full solution.**\n\n\n\n### Given\n\nResidual input = 16 × 16 × 8; main-path output = 8 × 8 × 16.\n\nShortcut kernel = 1 × 1; stride 2; padding 0.\n\nUse one bias per projection filter.\n\n### To find\n\nChoose the number of filters and find projection output shape and parameters.\n\n### Formula — which one and why\n\n**Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.\n\nN = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.\n\n\n\n**Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nResidual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16.\n\n\n### Step 1: choose 16 filters\nOne filter creates one channel, so 16 filters give the required 16 output channels.\n### Step 2: use the specified 1 × 1 kernel and stride 2\n\n\nThe floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1.\n### Step 3: count weights and biases\n\n\n**Final answer:** projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid.\n"},{"title":"Revising solved examples","blocks":["Revisit a direct example and then its changed-number version. Write the given values, the formula and the substitutions on separate lines. If an answer differs, compare from the first line until the difference appears. Use the mixed questions after the individual methods make sense.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c17-s46","search":"Revising solved examples Revisit a direct example and then its changed-number version. Write the given values, the formula and the substitutions on separate lines. If an answer differs, compare from the first line until the difference appears. Use the mixed questions after the individual methods make sense.\n\n"}]},{"n":18,"title":"Two complete MT2 mock papers","sections":[{"title":"Start here","blocks":["These mocks follow the teacher's Unit 3–4 guidance and clarification relayed on 12 October. The 2-mark parts can be theory or a theory–numerical combination. For Q2–Q4, practise both stated arrangements: two numerical questions plus one theory question; or one mixed theory–numerical question, one numerical question and one theory question. Prepare the theory of both units as well as the numerical methods.
","Each rehearsal uses the established 10-mark layout: Q1(a) and Q1(b), 2 marks each, are compulsory; answer any TWO of Q2–Q4, 3 marks each. Allow 60 minutes. Mock A uses numerical + numerical + theory in Q2–Q4, including two Unit 4 numericals. Mock B uses mixed + numerical + theory, including Unit 3 optimization. The topic assigned to each question, and the marks split inside Mock B's mixed question, are practice choices rather than a fixed official blueprint. Unit 5 RNN and standalone Unit 2 ANN questions are not included here.
","Every answer is immediately below its question. The worked answers deliberately show more explanation than you need to copy during the exam. In the exam, retain the method, substitutions, intermediate values and labelled final answer.
"],"kind":"read","id":"mt2-c18-intro"},{"title":"MT2 Mock A: full questions and answers","blocks":["60-minute rehearsal; attempt 10 marks. Q1(a) and Q1(b) are compulsory. Answer any TWO of Q2–Q4. This practice arrangement contains Unit 3 theory and two Unit 4 long numericals.
","Q1a compulsory (2 marks). Explain the key difference between the AdaGrad and RMSProp accumulators. Why can AdaGrad's effective learning rate become very small during prolonged training?
","Answer — step by step.
","Let gₜ be the current gradient and Gₜ the stored sum of squared gradients. AdaGrad updates that sum as follows:
","Every added square is nonnegative, so G cannot decrease. All past squared gradients remain in the sum.
","Let sₜ be the stored squared-gradient average and ρ the fraction of the previous average retained.
","RMSProp gradually reduces the influence of older squared gradients. Its accumulator can rise or fall, depending on the new squared gradient.
","AdaGrad's effective learning rate is:
","![\"\\eta_{effective}=\\frac{\\eta}{\\sqrt{G_t}+\\epsilon}\"](\"assets/equations/6f335dbb7b791f6d.svg\")
Here η is the base learning rate and ε is a small positive number that stabilizes division. When G becomes large, its square root becomes large, so dividing by it produces a smaller effective learning rate. RMSProp's decaying average avoids this particular permanent-accumulation problem.
","Final answer: AdaGrad uses a nondecreasing sum of all squared gradients; RMSProp uses a decaying average. AdaGrad's growing denominator can make future updates very small.
","Yaad rakho: AdaGrad sab squares jodta rehta hai; RMSProp purane squares ka asar dheere kam karta hai.
","Q1b compulsory (2 marks). Explain local connectivity and parameter sharing in a CNN. How do they reduce the number of learned parameters compared with a fully connected mapping of an image?
","Answer — step by step.
","Each convolutional output depends on a small spatial neighbourhood of the input, called its receptive field, rather than every pixel in the entire image. The filter spans the input channels within that neighbourhood. This limits the connections used to calculate each output.
","The same filter weights and bias are reused at every spatial position where that filter is applied. A CNN therefore learns one set of parameters for that filter, rather than a separate set for each image location.
","An edge-detecting filter can identify a vertical boundary near the left of an image and then slide to look for a similar boundary near the right. Both locations use the same learned filter. The network does not need to learn an independent edge detector for every position.
","A fully connected layer generally learns a separate weight for every input-to-output connection. Convolution reduces this requirement in two ways: local connectivity restricts each output's input region, while parameter sharing reuses one filter across locations. More output positions create more activations and computation, but do not create new parameters for that shared filter.
","Final answer: local connectivity uses a limited input neighbourhood; parameter sharing reuses the filter's learned values across spatial positions. Together they can require far fewer parameters than a fully connected image mapping.
","Yaad rakho: Filter ke numbers ek baar seekhte hain; image par baar-baar wahi use hote hain.
","Q2 optional (3 marks). An input of size 7 × 7 × 3 passes through a convolutional layer with two 3 × 3 filters, stride 2 and padding 1 on each side; then a 2 × 2 pooling layer with stride 2 and no padding; then flattening; then a dense layer with three output units. Find the output shape after every layer and the total trainable parameters. Include one bias per convolutional filter and per dense unit.
","Answer — step by step.
","- Input: height 7, width 7, channels 3.
","- Convolution: kernel 3 × 3, filters 2, stride 2, padding 1 per side.
","- Pooling: window 2 × 2, stride 2, padding 0.
","- Flattening followed by 3 dense output units.
","- Include all specified biases. Pooling and flattening have no learned parameters.
","The convolution output, pooling output, flatten length, dense output length and total trainable parameter count.
","For either spatial axis, use the output-size formula. N is that axis's input size, K the window size, S its stride and P padding per side. The floor means discard any fractional part because a full window must fit.
","Each convolution filter spans all input channels; each dense unit connects to every flattened input. Therefore:
","![\"P_{\\mathrm{conv}}=(K^2C_{\\mathrm{in}}+1)C_{\\mathrm{out}},\\qquad](\"assets/equations/683bcf7c089c902f.svg\")
Flattening simply counts all values in the preceding feature maps:
","![\"N_{\\mathrm{conv}}=\\left\\lfloor\\frac{7+2(1)-3}{2}\\right\\rfloor+1=3+1=4\"](\"assets/equations/a2592712257363b0.svg\")
The numerator is 7 + 2 − 3 = 6. Divide by stride 2 to get 3, then add 1 to get 4 positions. The same calculation applies to height and width. Two filters create two output channels.
","Convolution output: 4 × 4 × 2.
","![\"\\text{weights}=3\\times3\\times3\\times2=54\"](\"assets/equations/e6c2c8a35c15f9ce.svg\")
![\"\\text{biases}=2,\\qquad](\"assets/equations/9fa35646db3c8aec.svg\")
The pooling input size is now 4, not the original 7.
","![\"N_{\\mathrm{pool}}=\\left\\lfloor\\frac{4-2}{2}\\right\\rfloor+1=1+1=2\"](\"assets/equations/75bb76e8c3f74574.svg\")
Pooling operates separately on each channel, so the two channels remain two.
","Pooling output: 2 × 2 × 2. Pooling parameters: 0.
","![\"n_{\\mathrm{flat}}=2\\times2\\times2=8\"](\"assets/equations/4eb23baff49f82cf.svg\")
There are eight values to send to the dense layer. Flattening rearranges them into a list and learns no parameters.
","Flatten output: 8 values. Flatten parameters: 0.
","Each of the three dense units receives eight inputs and has one bias.
","![\"\\text{weights}=8\\times3=24,\\quad\\text{biases}=3,\\quad](\"assets/equations/5d064152bee760f3.svg\")
Dense output: 3 values.
","![\"P_{\\mathrm{total}}=56+0+0+27=83\"](\"assets/equations/8013f82d7c04ad92.svg\")
Final answer: 4 × 4 × 2 → 2 × 2 × 2 → 8 → 3. Total: 83 trainable parameters.
","Yaad rakho: Pehle shape; phir weights. Ek layer ka output hi next layer ka input hai.
","Q3 optional (3 marks). Apply the printed kernel K to input I using the deep-learning cross-correlation convention, with stride 1, no padding and zero bias. Do not flip K. Apply ReLU to the resulting map, followed by 2 × 2 max pooling with stride 2 and no padding. Show every convolution entry and find the final output.
","![\"I=\\begin{bmatrix}1&2&1\\\\0&1&3\\\\2&1&0\\end{bmatrix},\\quad](\"assets/equations/9bf8a81efcf4cdbd.svg\")
Answer — step by step.
","- Input I is the printed 3 × 3 matrix; kernel K is the printed 2 × 2 matrix.
","- Convolution stride 1, padding 0, bias 0; use K in its printed orientation.
","- Activation: ReLU.
","- Pooling: max, 2 × 2 window, stride 2, padding 0.
","All four convolution values, the ReLU output matrix and the final pooled output.
","For each input window, multiply entries at matching positions, then add all products and the bias once.
","X is the input, S is stride, u and v select a kernel entry, and b is bias. This formula uses zero-based array indices; the written solution labels positions from 1 for readability.
","ReLU is required because the question requests activation after convolution. It replaces a negative value by zero.
","Max pooling selects the largest value in its current window V.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","For each axis: (3 − 2)/1 + 1 = 2. Therefore calculate a 2 × 2 output matrix.
","The window is ![\"\\begin{bmatrix}1](\"assets/equations/eed57e81393ab435.svg\")
.
Multiply matching cells: 1 × 1 = 1; 2 × (−1) = −2; 0 × 0 = 0; 1 × 1 = 1.
","Add the products and bias: 1 − 2 + 0 + 1 + 0 = 0.
","Move the window one column to the right. The window is ![\"\\begin{bmatrix}2](\"assets/equations/85c116687119a006.svg\")
.
Multiply matching cells: 2 × 1 = 2; 1 × (−1) = −1; 1 × 0 = 0; 3 × 1 = 3.
","Add: 2 − 1 + 0 + 3 + 0 = 4.
","The window is ![\"\\begin{bmatrix}0](\"assets/equations/e06165b70c0dec06.svg\")
.
Multiply matching cells: 0 × 1 = 0; 1 × (−1) = −1; 2 × 0 = 0; 1 × 1 = 1.
","Add: 0 − 1 + 0 + 1 + 0 = 0.
","The window is ![\"\\begin{bmatrix}1](\"assets/equations/9f01f837180dd993.svg\")
.
Multiply matching cells: 1 × 1 = 1; 3 × (−1) = −3; 1 × 0 = 0; 0 × 1 = 0.
","Add: 1 − 3 + 0 + 0 + 0 = −2.
","Convolution map: ![\"\\begin{bmatrix}0](\"assets/equations/f2f63d345efa8255.svg\")
.
- max(0,0) = 0.
","- max(0,4) = 4.
","- max(0,0) = 0.
","- max(0,−2) = 0.
","ReLU map: ![\"\\begin{bmatrix}0](\"assets/equations/ca987c6e03219656.svg\")
.
The 2 × 2 pooling window covers all four activated entries. Compare 0, 4, 0 and 0. Their maximum is 4.
","Final answer: convolution → ReLU → pooled output ![\"\\begin{bmatrix}4\\end{bmatrix}\"](\"assets/equations/eaf8e658e6675411.svg\")
, a 1 × 1 map.
Yaad rakho: Multiply-add se convolution; negative hatao ReLU; biggest uthao max pool.
","Q4 optional (3 marks). What is overfitting? Explain how dropout and early stopping help improve generalization. State which dataset should select the stopping point.
","Answer — step by step.
","Overfitting occurs when a model learns training-specific details or noise that do not generalize to new data. It commonly shows low training error together with substantially higher validation error. Generalization means performing well on new examples drawn from the task's data distribution.
","During training, dropout randomly sets selected activations to zero. The active subnetwork changes across training passes, discouraging reliance on a fixed combination of hidden units. Under inverted dropout, retained activations are divided by the keep probability to preserve their expected scale. During ordinary inference, the masking is disabled and all units are used without that training-time rescaling.
","Dropout is a regularizer; it can reduce overfitting, but excessive dropout can cause underfitting.
","Train on the training set and monitor performance on a separate validation set. Save the model whenever validation loss improves. Stop after the chosen number of consecutive non-improving evaluations, called patience, then restore the saved best model.
","For example, validation losses 0.8, 0.6, 0.5, 0.53 and 0.55 have their minimum at epoch 3. With patience 2, stop after epoch 5 and restore epoch 3. The stopping epoch and saved-model epoch are different.
","Final answer: dropout restricts dependence on particular training-time activations; early stopping limits training after validation performance stops improving. Choose the stopping point using validation data and reserve the test set for final evaluation.
","Yaad rakho: Training se seekho, validation se best model chuno, test se final result batao.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c18-s0","search":"MT2 Mock A: full questions and answers \n60-minute rehearsal; attempt 10 marks. Q1(a) and Q1(b) are compulsory. Answer any TWO of Q2–Q4. This practice arrangement contains Unit 3 theory and two Unit 4 long numericals.\n\n### Q1a\n**Q1a compulsory (2 marks). Explain the key difference between the AdaGrad and RMSProp accumulators. Why can AdaGrad's effective learning rate become very small during prolonged training?**\n\n**Answer — step by step.**\n\n### 1. AdaGrad stores a running sum\n\nLet gₜ be the current gradient and Gₜ the stored sum of squared gradients. AdaGrad updates that sum as follows:\n\n\n\nEvery added square is nonnegative, so G cannot decrease. All past squared gradients remain in the sum.\n\n### 2. RMSProp stores a decaying average\n\nLet sₜ be the stored squared-gradient average and ρ the fraction of the previous average retained.\n\n\n\nRMSProp gradually reduces the influence of older squared gradients. Its accumulator can rise or fall, depending on the new squared gradient.\n\n### 3. Explain the effect on step size\n\nAdaGrad's effective learning rate is:\n\n\n\nHere η is the base learning rate and ε is a small positive number that stabilizes division. When G becomes large, its square root becomes large, so dividing by it produces a smaller effective learning rate. RMSProp's decaying average avoids this particular permanent-accumulation problem.\n\n**Final answer:** AdaGrad uses a nondecreasing sum of all squared gradients; RMSProp uses a decaying average. AdaGrad's growing denominator can make future updates very small.\n\n**Yaad rakho:** AdaGrad sab squares jodta rehta hai; RMSProp purane squares ka asar dheere kam karta hai.\n\n### Q1b\n**Q1b compulsory (2 marks). Explain local connectivity and parameter sharing in a CNN. How do they reduce the number of learned parameters compared with a fully connected mapping of an image?**\n\n**Answer — step by step.**\n\n### 1. Local connectivity\n\nEach convolutional output depends on a small spatial neighbourhood of the input, called its receptive field, rather than every pixel in the entire image. The filter spans the input channels within that neighbourhood. This limits the connections used to calculate each output.\n\n### 2. Parameter sharing\n\nThe same filter weights and bias are reused at every spatial position where that filter is applied. A CNN therefore learns one set of parameters for that filter, rather than a separate set for each image location.\n\n### 3. Concrete image example\n\nAn edge-detecting filter can identify a vertical boundary near the left of an image and then slide to look for a similar boundary near the right. Both locations use the same learned filter. The network does not need to learn an independent edge detector for every position.\n\n### 4. Comparison with a fully connected mapping\n\nA fully connected layer generally learns a separate weight for every input-to-output connection. Convolution reduces this requirement in two ways: local connectivity restricts each output's input region, while parameter sharing reuses one filter across locations. More output positions create more activations and computation, but do not create new parameters for that shared filter.\n\n**Final answer:** local connectivity uses a limited input neighbourhood; parameter sharing reuses the filter's learned values across spatial positions. Together they can require far fewer parameters than a fully connected image mapping.\n\n**Yaad rakho:** Filter ke numbers ek baar seekhte hain; image par baar-baar wahi use hote hain.\n\n### Q2"},{"title":"MT2 Mock B: full questions and answers","blocks":["60-minute rehearsal; attempt 10 marks. Q1(a) and Q1(b) are compulsory. Answer any TWO of Q2–Q4. Q2 is mixed theory + numerical, Q3 is numerical and Q4 is theory. The 1 + 2 split within Q2 is a practice allocation. The specific topics are not predictions of the actual paper.
","Q1a compulsory (2 marks). Explain the purpose of scaling retained activations in inverted dropout. For activations [1,2,3,4], keep probability 0.5 and mask [1,0,1,0], calculate the activations used during this training pass. A mask value of 1 means keep and 0 means drop.
","Answer — step by step.
","Randomly dropping units would reduce the average activation. Inverted dropout divides every retained activation by the keep probability. Averaged over the random masks, this preserves the original activation's expected value. A dropped activation is still zero.
","- Original activations a = [1,2,3,4].
","- Keep probability q = 0.5, meaning a unit has a 50% chance of being kept.
","- This pass's supplied mask m = [1,0,1,0].
","The four post-dropout activations used in this training pass.
","Multiply the original activation aᵢ by its mask mᵢ, then divide by keep probability q.
","Mask 1 keeps the value. Multiply 1 × 1 = 1; divide 1 by 0.5 to get 2.
","Mask 0 drops the value. Multiply 0 × 2 = 0; divide 0 by 0.5 to get 0.
","Mask 1 keeps the value. Multiply 1 × 3 = 3; divide 3 by 0.5 to get 6.
","Mask 0 drops the value. Multiply 0 × 4 = 0; divide 0 by 0.5 to get 0.
","Final answer: [2,0,6,0]. This is the training-pass result. In ordinary inference with inverted dropout, disable the mask and its scaling.
","Yaad rakho: Mask lagao; bache hue values ko keep probability se divide karo.
","Q1b compulsory (2 marks). Explain a ResNet shortcut. Why must the two tensors being added have matching shapes, and what can be done if their shapes differ?
","Answer — step by step.
","A residual block combines a learned transformation F(x) with a shortcut carrying the input x:
","F(x) is the output of the learned residual branch. Adding x provides a direct path for information and gradients, helping optimize deep networks. If the branch produces zero, the addition stage returns x. A post-addition activation, if specified by the architecture, is applied afterward.
","The addition is entry by entry. Each element in F(x) needs a corresponding element in x, so height, width and channel count must match. For example, a 14 × 14 × 32 tensor cannot be directly added to a 14 × 14 × 64 tensor in a standard residual shortcut.
","A learned projection, commonly a 1 × 1 convolution with an appropriate number of filters and stride, maps the shortcut to the required shape. Then add the projected shortcut and the residual branch. This is addition, not channel concatenation.
","Final answer: a shortcut adds input information to the learned residual branch. Use an identity shortcut when shapes match and a suitable projection when they do not.
","Yaad rakho: Same-size entries jodte hain; mismatch ho toh shortcut ka size pehle match karo.
","Q2 optional (3 marks). (a) Explain why Adam uses bias correction for its initially zero moment estimates. (1 mark) (b) Adam starts with θ₀ = 1 and m₀ = v₀ = 0. Use learning rate η = 0.1, β₁ = 0.9 and β₂ = 0.99. The supplied gradients are g₁ = 2 and g₂ = −1. Perform two bias-corrected updates. Show the raw first and second moments, corrected moments and updated parameter after each iteration. Set ε = 0 only for this nonzero-denominator hand calculation. (2 marks)
","Answer — step by step.
","Adam initializes both moving averages to zero. At early iterations, too little gradient history has accumulated, so the raw estimates are biased toward zero. Dividing mₜ by 1 − β₁ᵗ and vₜ by 1 − β₂ᵗ corrects this initialization bias before the parameter update. The power uses the current iteration number t. As t increases, these denominators approach 1.
","- Starting parameter θ₀ = 1.
","- Starting first moment m₀ = 0 and second moment v₀ = 0.
","- Learning rate η = 0.1.
","- Decay factors β₁ = 0.9 and β₂ = 0.99.
","- Supplied gradients, in order: 2, then −1.
","- ε = 0 for this exercise; both denominators will be nonzero.
","For iterations 1 and 2, calculate m, v, their bias-corrected values and the new parameter θ.
","Adam keeps two moving averages: m stores signed gradients and v stores squared gradients. Zero initialization makes these averages initially too small, so divide by the correction factors for the current iteration t.
","The hat marks a corrected value. Carry the raw m and v into the next moving-average recurrence, not the corrected values. Keep full calculator precision until the final answer.
","The current gradient is 2. Retain 90% of the old first moment and add 10% of this gradient.
","Square the gradient first: 2² = 4. Retain 99% of the old second moment and add 1% of that square.
","![\"v_1=0.99(0)+0.01(2)^2=0.04\"](\"assets/equations/4af117593c0c60f1.svg\")
Use t = 1. The correction denominators are 1 − 0.9 = 0.1 and 1 − 0.99 = 0.01.
","![\"\\widehat](\"assets/equations/ecdf53b308191b04.svg\")
The square root of the corrected second moment is √4 = 2. The step is 0.1 × 2 ÷ 2 = 0.1.
","Use supplied gradient −1, previous raw m = 0.2 and previous raw v = 0.04.
","![\"m_2=0.9(0.2)+0.1(-1)=0.08\"](\"assets/equations/85e3c1c423ab1151.svg\")
The two contributions are 0.18 and −0.1; their sum is 0.08.
","The square of −1 is +1.
","![\"v_2=0.99(0.04)+0.01(-1)^2=0.0496\"](\"assets/equations/9fa8fd0cf3a03d97.svg\")
The two contributions are 0.0396 and 0.01; their sum is 0.0496.
","Use t = 2. The correction denominators are 1 − 0.9² = 0.19 and 1 − 0.99² = 0.0199.
","![\"\\widehat](\"assets/equations/ad611d8dfc7121b6.svg\")
![\"\\widehat](\"assets/equations/77a755525cf83c35.svg\")
Start from the updated parameter 0.9. Take the square root of the corrected second moment, divide the corrected first moment by it, multiply by 0.1 and subtract.
","![\"\\theta_2=0.9-\\frac{0.1\\times(0.421052632)}{\\sqrt{2.492462312}}=0.873330057\"](\"assets/equations/a3614192f516725e.svg\")
The displayed decimals are rounded; evaluate using the unrounded fractions when possible.
","Final answer: θ₁ = 0.9 and θ₂ ≈ 0.873330057. The second supplied gradient is negative, but the first-moment estimate is still positive because it retains part of the previous positive gradient.
","Yaad rakho: Do raw memories banao → current t se hats nikalo → update; next round mein raw memories le jao.
","Q3 optional (3 marks). A 16 × 16 × 64 feature map is processed by 32 filters of size 3 × 3 using stride 1 and same padding. Compare the direct convolution's MAC count with a bottleneck that first applies eight 1 × 1 filters, then the same 32-filter 3 × 3 convolution. Both stages keep spatial size 16 × 16. Find both costs, the absolute saving and percentage saving. Count one multiply–accumulate as one MAC and omit bias additions.
","Answer — step by step.
","- Input map: 16 × 16 with 64 channels.
","- Direct layer: 32 filters, each 3 × 3, receiving 64 channels.
","- Bottleneck stage 1: eight 1 × 1 filters, receiving 64 channels.
","- Bottleneck stage 2: 32 filters, each 3 × 3, now receiving only 8 channels.
","- Every layer has output height and width 16; ignore bias additions.
","Direct MACs, both bottleneck-stage MACs, their total, absolute saving and saving percentage.
","Each output entry uses K²Cᵢₙ weight–input pairs. There are HₒᵤₜWₒᵤₜCₒᵤₜ output entries. Multiply these counts:
","The bottleneck has two layers, so calculate each layer's work and add them. Saving is direct work minus reduced work, expressed as a fraction of direct work.
","Each output entry uses 3 × 3 × 64 = 576 weight–input pairs.
","There are 16 × 16 × 32 = 8,192 output entries.
","Each reduction output uses 1 × 1 × 64 = 64 weight–input pairs.
","There are 16 × 16 × 8 = 2,048 reduction outputs.
","The new map is 16 × 16 × 8. This changes the next layer's input channels to 8.
","Each output now uses 3 × 3 × 8 = 72 weight–input pairs, instead of 576.
","The number of outputs remains 16 × 16 × 32 = 8,192.
","Include both stages; do not compare the direct layer against only the second stage.
","Divide the saving by the original direct cost, then multiply by 100.
","Final answer: direct 4,718,592 MACs; bottleneck 720,896 MACs; saving 3,997,696 MACs, approximately 84.72%. Both routes end in a 16 × 16 × 32 output, though they are different learned transformations.
","Yaad rakho: 1 × 1 se channels kam; expensive 3 × 3 ka kaam kam. Dono stages ka cost jodna hai.
","Q4 optional (3 marks). Explain transfer learning for image classification with limited labelled data. Distinguish feature extraction from fine-tuning and describe a suitable training procedure.
","Answer — step by step.
","Transfer learning reuses knowledge learned on a source task to help solve a target task. For image classification, a pretrained CNN can provide visual features that a new classifier uses for the target classes. This can reduce the amount of target data and training needed, provided the source features are useful for the target task.
","| Method | Which parameters learn? | Main effect |
|---|---|---|
| Feature extraction | Keep pretrained backbone weights fixed; train a new classification head | Learn to classify using fixed pretrained features |
| Fine-tuning | Update selected pretrained layers as well as the new head | Adapt representations to the target task |
Freezing a layer stops its parameter updates; it does not remove its stored weights. Fine-tuning typically exposes more trainable parameters and therefore needs particular care with small datasets.
","1. Create separate training, validation and test splits for the target problem.
","2. Choose a suitable pretrained CNN and apply its required input size and preprocessing.
","3. Replace the original classifier with a new head matching the target classes.
","4. Freeze the backbone and train the new head on the training set.
","5. If validation results justify it, unfreeze selected later layers and continue with an appropriately smaller learning rate.
","6. Select the best checkpoint using validation performance; evaluate the final model on the held-out test set.
","For a small dataset containing cats and dogs, reuse a pretrained image backbone and replace its original classifier with a binary-classification head. First learn the new head with the backbone frozen. Later, selected high-level layers may be fine-tuned if this improves validation performance.
","Final answer: feature extraction learns a new classifier over fixed features; fine-tuning also adapts selected pretrained features. The correct choice depends on target data quantity, similarity to the source data and validation results.
","Yaad rakho: Freeze mein purane features fixed; fine-tune mein selected purane features bhi seekhte hain.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c18-s1","search":"MT2 Mock B: full questions and answers \n60-minute rehearsal; attempt 10 marks. Q1(a) and Q1(b) are compulsory. Answer any TWO of Q2–Q4. Q2 is mixed theory + numerical, Q3 is numerical and Q4 is theory. The 1 + 2 split within Q2 is a practice allocation. The specific topics are not predictions of the actual paper.\n\n### Q1a\n**Q1a compulsory (2 marks). Explain the purpose of scaling retained activations in inverted dropout. For activations [1,2,3,4], keep probability 0.5 and mask [1,0,1,0], calculate the activations used during this training pass. A mask value of 1 means keep and 0 means drop.**\n\n**Answer — step by step.**\n\n### Theory: why scale retained values?\n\nRandomly dropping units would reduce the average activation. Inverted dropout divides every retained activation by the keep probability. Averaged over the random masks, this preserves the original activation's expected value. A dropped activation is still zero.\n\n### Given\n\n- Original activations a = [1,2,3,4].\n- Keep probability q = 0.5, meaning a unit has a 50% chance of being kept.\n- This pass's supplied mask m = [1,0,1,0].\n\n### To find\n\nThe four post-dropout activations used in this training pass.\n\n### Formula — which one and why\n\nMultiply the original activation aᵢ by its mask mᵢ, then divide by keep probability q.\n\n\n\n### Solution — step by step\n\n### Step 1: first activation\n\nMask 1 keeps the value. Multiply 1 × 1 = 1; divide 1 by 0.5 to get **2**.\n\n### Step 2: second activation\n\nMask 0 drops the value. Multiply 0 × 2 = 0; divide 0 by 0.5 to get **0**.\n\n### Step 3: third activation\n\nMask 1 keeps the value. Multiply 1 × 3 = 3; divide 3 by 0.5 to get **6**.\n\n### Step 4: fourth activation\n\nMask 0 drops the value. Multiply 0 × 4 = 0; divide 0 by 0.5 to get **0**.\n\n**Final answer: [2,0,6,0].** This is the training-pass result. In ordinary inference with inverted dropout, disable the mask and its scaling.\n\n**Yaad rakho:** Mask lagao; bache hue values ko keep probability se divide karo.\n\n### Q1b\n**Q1b compulsory (2 marks). Explain a ResNet shortcut. Why must the two tensors being added have matching shapes, and what can be done if their shapes differ?**\n\n**Answer — step by step.**\n\n### 1. Residual addition\n\nA residual block combines a learned transformation F(x) with a shortcut carrying the input x:\n\n\n\nF(x) is the output of the learned residual branch. Adding x provides a direct path for information and gradients, helping optimize deep networks. If the branch produces zero, the addition stage returns x. A post-addition activation, if specified by the architecture, is applied afterward.\n\n### 2. Why matching shapes are needed\n\nThe addition is entry by entry. Each element in F(x) needs a corresponding element in x, so height, width and channel count must match. For example, a 14 × 14 × 32 tensor cannot be directly added to a 14 × 14 × 64 tensor in a standard residual shortcut.\n\n### 3. How a projection helps\n\nA learned projection, commonly a 1 × 1 convolution with an appropriate number of filters and stride, maps the shortcut to the required shape. Then add the projected shortcut and the residual branch. This is addition, not channel concatenation.\n\n**Final answer:** a shortcut adds input information to the learned residual branch. Use an identity shortcut when shapes match and a suitable projection when they do not.\n\n**Yaad rakho:** Same-size entries jodte hain; mismatch ho toh shortcut ka size pehle match karo.\n\n### Q2\n**Q2 optional (3 marks). (a) Explain why Adam uses bias correction "}]},{"n":19,"title":"Formula reference and exam revision","sections":[{"title":"The one page method map","blocks":["| Question clue | First action | Core formula or check |
|---|---|---|
| Neuron output | Multiply each input by its matching weight | , then activate |
| Network prediction | Finish one whole layer before the next | ![\"h=f(Wx+b)\"](\"assets/equations/5c938cf38724d627.svg\") |
| Backpropagation | State loss then prediction-minus-target | delta times sending activation |
| Optimizer update | Identify current gradient and stored state | old parameter minus update |
| CNN dimensions | Write N K S P and channels | |
| CNN parameters | Count one filter then all filters | ![\"(K^2C_{\\mathrm{in}}+1)C_{\\mathrm{out}}\"](\"assets/equations/cfa2f04f36764d25.svg\") |
| Pooling | Draw every valid window | Maximum, or mean of the window |
| Dense parameters | Count connections and receiving biases | ![\"(N_{\\mathrm{in}}+1)N_{\\mathrm{out}}\"](\"assets/equations/641d99b5a929f5f6.svg\") |
These are the formulas used by this course. Each entry names its symbols and the type of calculation it serves. Numerical practice in the teaching chapters shows the longer working. Follow any different convention explicitly stated in the examination.
","![\"\\mu=\\frac{1}{n}\\sum_i](\"assets/equations/7bccad9589b45084.svg\")
Symbols: n is the count; mu is the mean; v is the population variance; s is the standard deviation.
","Use: Use when calculating a population mean/variance or standardization. Sample-variance estimates may use n−1 when explicitly requested.
","Substitution example: For [2,4]: mean = 3; deviations = −1,1; squares = 1,1; variance = 1; standard deviation = 1.
","![\"x_{scaled}=\\frac{x-\\mu_{train}}{s_{train}}\"](\"assets/equations/6a0a4f4a38fa7007.svg\")
Symbols: Use the mean and standard deviation estimated from training data.
","Use: Use to express a value relative to the training distribution. The standard deviation must be nonzero.
","Substitution example: For x = 7, mean = 5, standard deviation = 2: (7−5)/2 = 1.
","![\"\\frac{d}{dx}(cx^n)=cnx^{n-1},\\quad\\frac{d}{dx}c=0,\\quad](\"assets/equations/dffe83aa3b00d40c.svg\")
Symbols: c is a constant, x is the changing variable and n is the power.
","Use: Use to differentiate the loss supplied in a gradient-descent question.
","Substitution example: For 0.7x³: derivative = 0.7 × 3 × x² = 2.1x².
","Symbols: z is the intermediate value linking w to L.
","Use: Multiply derivatives along a path; add contributions from distinct paths that meet.
","Substitution example: For z = 3w+1 and L = z² at w = 1: z = 4; derivative = (2×4)×3 = 24.
","![\"z=\\sum_i](\"assets/equations/e4620ff07e410bce.svg\")
Symbols: x is the input; w is its matching weight; b is one bias; z is the weighted sum; a is the activated output.
","Use: Multiply each matching pair, add the products, add bias once, then activate.
","Substitution example: x = [2,1], w = [0.5,−0.2], b = 0.1: z = 1−0.2+0.1 = 0.9.
","![\"(AB)_{ij}=\\sum_k](\"assets/equations/47f627ca6464a106.svg\")
Symbols: i selects a row of A; j selects a column of B; k runs along the matching entries.
","Use: Use row-times-column sums. A with shape m×n and B with n×p produces m×p.
","Substitution example: Row [1,2] times column [3,4] gives 1×3+2×4 = 11.
","Symbols: l labels the layer; the previous layer activations become inputs to the next layer.
","Use: For receiving-neuron rows, compute each row dot product and add that receiving neuron’s bias. Finish the whole layer before moving on.
","![\"\\sigma(z)=\\frac{1}{1+e^{-z}},\\quad](\"assets/equations/e03a4bc95b918203.svg\")
Symbols: e is approximately 2.71828; sigma is the output.
","Use: Use for a sigmoid forward pass and for its local backpropagation derivative.
","Substitution example: At z = 0: output = 1/(1+1) = 0.5; derivative = 0.5×0.5 = 0.25.
","![\"\\tanh(z)=\\frac{e^{2z}-1}{e^{2z}+1},\\quad](\"assets/equations/6e538ee72147c838.svg\")
Symbols: Tanh gives an output between −1 and 1.
","Use: Use for a stated tanh activation, commonly an RNN hidden state or recurrent candidate.
","Substitution example: At z = 0: numerator = 0, denominator = 2; output = 0; derivative = 1.
","![\"f(z)=\\max(0,z),\\quad](\"assets/equations/3bc788a99b96d468.svg\")
Symbols: max means choose the larger value.
","Use: Use for ReLU. At zero the derivative is mathematically undefined; use the convention stated by the question or implementation.
","Substitution example: For −2, output = 0 and derivative = 0. For 3, output = 3 and derivative = 1.
","![\"f(z)=\\begin{cases}z&z\\geq0\\\\az&z<0\\end{cases}\"](\"assets/equations/d3faffb967fafdf3.svg\")
Symbols: a is the small negative-side slope, commonly 0.01 unless specified.
","Use: For negative z multiply by a; the output remains negative.
","Substitution example: For z = −2 and a = 0.01: output = −0.02; derivative on this side is 0.01.
","![\"p_i=\\frac{e^{z_i-c}}{\\sum_j](\"assets/equations/c492ec558e0b1e23.svg\")
Symbols: z values are class scores; p values are probabilities.
","Use: Subtract the same maximum from every score, exponentiate, add, divide each by that total.
","Substitution example: For [0,0]: exponentials = [1,1]; total = 2; probabilities = [0.5,0.5].
","![\"\\mathrm{SSE}=\\sum_i(o_i-y_i)^2\"](\"assets/equations/a288e1ba23bc9347.svg\")
![\"L_{\\mathrm{half}}=\\frac{\\mathrm{SSE}}2\"](\"assets/equations/1bdd5bbb9ca64f61.svg\")
![\"L_{\\mathrm{MSE}}=\\frac{\\mathrm{SSE}}n\"](\"assets/equations/bdee7885d635e78c.svg\")
![\"L_{\\mathrm{MAE}}=\\frac1n\\sum_i|o_i-y_i|\"](\"assets/equations/42ae3654ddbdf33e.svg\")
Symbols: o is prediction; y is target; n is the number of averaged terms.
","Use: Read whether the question uses a half sum, a mean, or an unhalved sum. The scaling changes gradients.
","Substitution example: For errors [−1,2]: half sum = (1+4)/2 = 2.5; MSE = 2.5; MAE = (1+2)/2 = 1.5. The first two coincide only because n = 2.
","Symbols: y is 0 or 1; p is the predicted probability for class 1; ln is the natural logarithm.
","Use: Use for a sigmoid binary prediction. If y = 1 use −ln(p); if y = 0 use −ln(1−p).
","Substitution example: For y = 1 and p = 0.8: loss = 0.223144.
","![\"H(p,q)=-\\sum_i](\"assets/equations/4be60e1a1e5d9ec8.svg\")
For a soft target distribution, retain every nonzero target-weighted term. Natural logs give nats; base-2 logs give bits.
","![\"L=-\\sum_i](\"assets/equations/ea93bf47748eb958.svg\")
Symbols: The final equality assumes one-hot labels, with exactly one y equal to 1.
","Use: Use for mutually exclusive multiclass labels. Select the probability of the true class.
","Substitution example: True class 2 and probabilities [0.1,0.8,0.1]: loss = −ln(0.8) = 0.223144.
","![\"\\delta^{out}=(o-y)f\\prime(z)\"](\"assets/equations/dc1948e719d0e70b.svg\")
Symbols: Delta is the loss derivative with respect to the preactivation z.
","Use: Use with half squared loss. For unhalved one-output MSE multiply by 2; for an MSE averaged over m outputs multiply the half-sum delta by 2/m.
","Substitution example: Sigmoid output 0.5 and target 1: delta = (−0.5)×0.25 = −0.125.
","![\"\\delta^{out}=p-y\"](\"assets/equations/154ed49b87099426.svg\")
Symbols: p and y can be vectors for softmax.
","Use: This is the combined derivative with respect to the logits. Do not multiply by the activation derivative again.
","Substitution example: Binary p = 0.8, y = 1: delta = −0.2.
","![\"\\delta_j^h=f\\prime(z_j^h)\\sum_k](\"assets/equations/f960780a672401ea.svg\")
Symbols: a_i is the activation sending a signal along the weight; delta_j belongs to the receiving neuron.
","Use: Use OLD downstream weights throughout one backward pass. Add all outgoing-path contributions.
","Substitution example: If receiving delta = −0.2 and sending activation = 3: weight gradient = −0.6; bias gradient = −0.2.
","![\"\\theta_{new}=\\theta-\\eta](\"assets/equations/0d35498ffd4122e2.svg\")
Symbols: theta is the parameter; eta is learning rate; g is the current gradient.
","Use: Recalculate g at the current parameter unless the question supplies gradients directly.
","Substitution example: theta = 1, eta = 0.1, g = −2 gives 1−0.1(−2) = 1.2.
","![\"g_{batch}=\\frac1B\\sum_{i=1}^{B}g_i,\\quad](\"assets/equations/01bdd22eb4365aa9.svg\")
Symbols: B is batch size; N is sample count; U is updates per epoch when the final incomplete batch is retained.
","Use: SGD uses B = 1. Full batch uses B = N. Average only when the stated objective is a mean.
","Substitution example: N = 105, B = 20: six updates, with five samples in the last batch.
","Symbols: v stores signed movement; beta retains part of old movement.
","Use: Use this convention consistently. Other textbooks may define an unsigned gradient accumulator and subtract it instead.
","Substitution example: beta = 0.8, old v = −0.2, eta = 0.1, g = 1.8 gives v = −0.34.
","![\"\\theta_{look}=\\theta+\\beta](\"assets/equations/29d3ddbf9730bc3a.svg\")
Symbols: The look-ahead point is temporary; the update is added to the original theta.
","Use: Evaluate the gradient at the look-ahead point, not at the old theta.
","Substitution example: theta = 1.8, beta = 0.8 and v = −0.2 gives look-ahead = 1.64.
","Symbols: G is accumulated squared gradient; epsilon prevents division by zero.
","Use: Accumulate every squared gradient, separately for each parameter. Use the epsilon placement stated by the problem.
","Substitution example: Old G = 4 and g = −3 gives G = 13.
","Symbols: rho is the decay; s is a decaying mean of squared gradients.
","Use: Calculate the new square memory first, then update the parameter.
","Substitution example: rho = 0.9, old s = 4, g = −3 gives s = 3.6+0.9 = 4.5.
","Symbols: m remembers signed gradients; v remembers squared gradients.
","Use: Start with the initial states specified. Common defaults are beta1 = 0.9 and beta2 = 0.999.
","Symbols: t counts updates starting at 1, not 0.
","Use: Correct both zero-initialized memories before the update; do not multiply by the gradient a second time.
","Substitution example: At t = 1, m = 0.2 and beta1 = 0.9: corrected m = 0.2/0.1 = 2.
","![\"J=L+\\lambda\\sum_j|w_j|,\\quad](\"assets/equations/f35555b2b4b6691c.svg\")
Symbols: lambda is regularization strength; sign(w) is +1 for positive w and −1 for negative w.
","Use: At zero, L1 has a subgradient rather than a unique derivative; use the specified convention. Include only parameters named in the penalty.
","Substitution example: w = −2, lambda = 0.1, data gradient = 0 gives penalty gradient = −0.1.
","![\"J=L+\\frac\\lambda2\\sum_jw_j^2,\\quad](\"assets/equations/a21ef40f9211941d.svg\")
Symbols: The half cancels the 2 produced by differentiating a square.
","Use: If the objective uses lambda times the sum of squares WITHOUT a half, use 2 lambda w instead.
","Substitution example: w = 2, lambda = 0.1, data gradient = 0.3 gives total gradient = 0.5.
","![\"q=1-p,\\quad](\"assets/equations/d4fe7ae66a3ded01.svg\")
Symbols: p is drop probability; q is keep probability; mask m contains zeros and ones; the circled dot means elementwise multiplication.
","Use: During training, zero dropped entries and divide kept entries by q. Disable dropout at inference.
","Substitution example: a = 3, q = 0.5, mask = 1 gives 3/0.5 = 6.
","![\"N_{out}=\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\"](\"assets/equations/f467d926f905df6d.svg\")
Symbols: N is input height or width; P is padding on each side; K is kernel size; S is stride; floor rounds down.
","Use: Apply separately to height and width. Output channels equal the number of filters. This formula assumes dilation 1.
","Substitution example: N = 7, P = 1, K = 3, S = 2 gives floor(6/2)+1 = 4.
","![\"Y_{ij,o}=\\sum_c\\sum_u\\sum_v](\"assets/equations/e5afb61a6d5f3879.svg\")
Symbols: c runs over input channels; u,v run over kernel positions; o selects one filter.
","Use: This is the usual deep-learning cross-correlation operation. Use zeros outside the padded image. Multiply matching cells, add all channels, then add the filter bias once.
","Min pooling keeps the smallest entry in each window. For a four-entry window:
","![\"y_{max}=\\max_{(i,j)\\in](\"assets/equations/edf81d51873a0dfc.svg\")
Symbols: W is one pooling window; its size is the number of included cells.
","Use: Move the window by the stated stride; process channels separately. For padding, follow the stated average-pooling denominator convention.
","Substitution example: For [1,4,2,3]: maximum = 4; average = 10/4 = 2.5.
","![\"P_{conv}=(K_hK_wC_{in}+1)C_{out}\"](\"assets/equations/8a253f24fdf99d8e.svg\")
Symbols: Kernel height and width times input channels gives weights per filter; +1 is one bias; Cout is filter count.
","Use: Use for a fully connected, ungrouped convolution with bias. Output height and width do not multiply the parameter count.
","Substitution example: 3×3 kernel, 3 input channels, 2 filters: (27+1)×2 = 56.
","![\"N_{flat}=HWC,\\qquad](\"assets/equations/56dad01ac3222ddd.svg\")
Symbols: H,W,C are activation dimensions; Nin and Nout are dense-layer input and output counts.
","Use: Flatten has no trainable parameters. Dense biases belong to receiving neurons.
","Substitution example: 2×2×2 flattens to 8; dense 3 then has (8+1)×3 = 27 parameters.
","![\"MAC=H_{out}W_{out}K_hK_wC_{in}C_{out}\"](\"assets/equations/7d20bfcc57cb122e.svg\")
Symbols: Each output value uses kernel area times input-channel count multiplications.
","Use: Used for Inception bottleneck comparisons. This counts one multiply-accumulate as one MAC and ignores bias additions. Some FLOP conventions count multiply and add separately.
","![\"C_{concat}=\\sum_b](\"assets/equations/588297b667afd636.svg\")
Symbols: b labels branches with matching spatial dimensions.
","Use: Inception concatenates channels; residual addition requires matching full shapes. Use a projection when needed.
","Substitution example: Channels 16+32+8+8 give 64. Adding two 14×14×64 tensors keeps 14×14×64.
","![\"h_t=\\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\\quad](\"assets/equations/a2d32a1185192d80.svg\")
Symbols: x_t is current input; h_previous is previous state; the same matrices are reused at every time step.
","Use: Calculate the two matrix-vector products, add bias, then tanh. Apply the stated output activation to o_t when required.
","Use during training to normalize intermediate activations before a learned scale and shift. m is the number of values; μ_B and σ²_B are the batch mean and population variance; γ and β are learned. Epsilon is inside the square root. Ordinary inference uses stored running statistics. The linked full lesson calculates every stage for [2,4].
","Take a separate average for each full spatial feature map. H and W count rows and columns; c selects one of C channels; a is an activation value. Output has C values. The linked lesson computes two channel averages and the following dense-layer parameters.
","N is the total number of evaluated examples. C_k counts those whose true class is among the k highest-scoring predictions. Count failures, divide by total, then convert to percent. The worked lesson contrasts top-1 and top-5 using the same 100 images.
","The reported model name is “Casio FX ES 990”; the exact model is not confirmed. Use your calculator’s actual function labels. On common ES/ES Plus models, exp(x) is usually the shifted ln key, while the EXP/ × 10ˣ key enters scientific notation and is NOT the same as exp(x). Check the exact key labeling on your device.
","For sigmoid at 2.2, type 1÷(1+exp(−2.2)). Keep the negative sign inside the exponential. Expected 0.900249511. For a negative gradient update, type old−rate × (negative gradient); use parentheses around the negative value. For cross-entropy, use −ln(probability), not log 10. For tanh, use the hyperbolic function or its exponential fraction if that function is difficult to find.
","Retain 6–9 digits in intermediate results or use stored values; round final answers to 4–6 decimal places unless the paper says otherwise. Work line by line on paper even if the calculator can evaluate the whole expression. Check exp(0) = 1, ln(1) = 0, sigmoid(0) = 0.5, tanh(0) = 0 before relying on the chosen key sequence.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c19-s2","search":"Calculator sequence without model guesswork The reported model name is “Casio FX ES 990”; the exact model is not confirmed. Use your calculator’s actual function labels. On common ES/ES Plus models, exp(x) is usually the shifted ln key, while the EXP/ × 10ˣ key enters scientific notation and is NOT the same as exp(x). Check the exact key labeling on your device.\nFor sigmoid at 2.2, type 1÷(1+exp(−2.2)). Keep the negative sign inside the exponential. Expected 0.900249511. For a negative gradient update, type old−rate × (negative gradient); use parentheses around the negative value. For cross-entropy, use −ln(probability), not log 10. For tanh, use the hyperbolic function or its exponential fraction if that function is difficult to find.\nRetain 6–9 digits in intermediate results or use stored values; round final answers to 4–6 decimal places unless the paper says otherwise. Work line by line on paper even if the calculator can evaluate the whole expression. Check exp(0) = 1, ln(1) = 0, sigmoid(0) = 0.5, tanh(0) = 0 before relying on the chosen key sequence."},{"title":"What to write if the exam page feels blank","blocks":["Pause briefly, put both feet down and take one slow breath. Then write only the GIVEN values and what is ASKED. Classify the question using the method map. Write its general formula before inserting numbers. One correct first line is a smaller task than “solve the whole question”.
","For CNN, draw the shape chain. For back prop, draw arrows and label the loss. For an optimizer, draw columns for step, gradient, state, new parameter. If you remain stuck after a short attempt, leave visible working space and move to a question you can start. Return after collecting those marks.
","The goal is to reduce the next action to something concrete and familiar. Practice it during mocks so it is available under time pressure.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c19-s3","search":"What to write if the exam page feels blank Pause briefly, put both feet down and take one slow breath. Then write only the GIVEN values and what is ASKED. Classify the question using the method map. Write its general formula before inserting numbers. One correct first line is a smaller task than “solve the whole question”.\nFor CNN, draw the shape chain. For back prop, draw arrows and label the loss. For an optimizer, draw columns for step, gradient, state, new parameter. If you remain stuck after a short attempt, leave visible working space and move to a question you can start. Return after collecting those marks.\nThe goal is to reduce the next action to something concrete and familiar. Practice it during mocks so it is available under time pressure."},{"title":"Answer lengths and partial credit habits","blocks":["For 2 marks, give a precise definition plus the key distinction/example. For 5 marks, give the definition, a labeled diagram or formula, working/principle and a relevant implication. For 10 marks, answer every subpart: architecture or setup, main equations, stepwise explanation, required numerical working, and limitations or justification where asked. Do not pad a numerical with unrelated theory.
","Always label assumptions when inputs, loss, threshold or matrix orientation are missing. Box the requested final result, include shapes or units, and retain working. An unsupported final number makes it harder to award method credit. Never say an optimizer guarantees global convergence or an architecture always outperforms another unless a specific theorem and its conditions actually apply.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c19-s4","search":"Answer lengths and partial credit habits For 2 marks, give a precise definition plus the key distinction/example. For 5 marks, give the definition, a labeled diagram or formula, working/principle and a relevant implication. For 10 marks, answer every subpart: architecture or setup, main equations, stepwise explanation, required numerical working, and limitations or justification where asked. Do not pad a numerical with unrelated theory.\nAlways label assumptions when inputs, loss, threshold or matrix orientation are missing. Box the requested final result, include shapes or units, and retain working. An unsupported final number makes it harder to award method credit. Never say an optimizer guarantees global convergence or an architecture always outperforms another unless a specific theorem and its conditions actually apply."},{"title":"Methods to revise before the examination","blocks":["This is a revision digest after studying the lessons. Full solutions remain beside each topic and under Solved Questions. If a step here is unclear, return to that worked example; the final day is for recovering the method, not memorizing unexplained final numbers.
","Machine learning estimates a model from data. Deep learning uses multiple trainable layers to learn successive representations. Inputs describe a sample; targets are the required answers; predictions are the model's answers. Weights and biases are learned parameters. Learning rate, batch size and architecture are chosen hyperparameters.
","Training changes parameters using training examples. Validation selects settings/stopping time. Testing evaluates the final choice on held-out data. Overfitting means strong training performance with weak generalization; underfitting means the model has not captured enough of the relationship.
","For one neuron, multiply each input by its own weight, add all products, add the bias once and apply the activation. A layer repeats this for several receiving neurons. Each weight-matrix row belongs to one receiving neuron. Transpose swaps rows and columns; it does not change signs.
","![\"z=\\sum_iw_ix_i+b,\\qquad](\"assets/equations/1489ef71177ccc4a.svg\")
Memory cue: multiply → sum → bias → activate. In a full network, finish one complete layer before the next.
","Sigmoid maps to (0,1) and is used for binary-class probabilities. Tanh maps to (−1,1), centred on zero. Both can saturate, producing small derivatives. ReLU retains positive values and returns zero on the negative side; its negative-side gradient is zero. Leaky ReLU retains a specified small negative-side slope. Softmax normalizes a complete list of logits into mutually exclusive class probabilities.
","![\"\\sigma(z)=\\frac1{1+e^{-z}},\\qquad\\sigma\\prime(z)=\\sigma(z)[1-\\sigma(z)]\"](\"assets/equations/fc390cc0a677c2cc.svg\")
![\"\\tanh\\prime(z)=1-\\tanh^2(z)\"](\"assets/equations/bc7dbcac123b9229.svg\")
For squared errors, SSE sums squares, half SSE divides that sum by 2, and MSE divides it by the number of scalar predictions. Their derivatives differ. For one-hot categorical cross-entropy, only the correct-class probability contributes. For binary cross-entropy, p means probability of class 1, not automatically the true class.
","![\"L_{\\mathrm{half}}=\\frac12\\sum_i(\\hat](\"assets/equations/38b125f3e664a9e5.svg\")
![\"\\mathrm{MSE}=\\frac1n\\sum_i(\\hat](\"assets/equations/52c8252c5d3ef107.svg\")
![\"L_{\\mathrm{BCE}}=-[y\\ln](\"assets/equations/519958f461b019f9.svg\")
![\"L_{\\mathrm{CCE}}=-\\ln](\"assets/equations/266c4f8c9aa94e08.svg\")
Memory cue: loss ka denominator dekho before differentiating. Sigmoid output is not its derivative.
","Write the given loss and activation first. Compute all forward sums/activations with the original parameters. Differentiate loss with respect to each output, multiply by the output activation derivative, and call the result the output delta.
","![\"\\delta_o=\\frac{\\partial](\"assets/equations/0f696548e5b34122.svg\")
For a hidden neuron, add the outgoing weighted deltas first and then multiply by the hidden activation derivative. Each connection gradient is receiving delta × sending activation. The bias gradient is the receiving delta. Use old weights for all gradients before making the simultaneous update.
","![\"\\frac{\\partial](\"assets/equations/c82c3235c49eee6f.svg\")
![\"\\theta_{\\mathrm{new}}=\\theta-\\eta\\frac{\\partial](\"assets/equations/a5c419cc72b98572.svg\")
Memory cue: path mein multiply; paths milne par add; update sab gradients ke baad. For sigmoid+BCE or softmax+CCE, the logit delta simplifies to prediction−target for the stated single-example sum convention.
","Full-batch GD uses the entire dataset for one update; SGD uses one sample; mini-batch uses a subset. An epoch is one pass through the data. With a final incomplete batch included, updates per epoch are ceiling(N/B).
","| Method | State and calculation order | Detail that changes the answer |
|---|---|---|
| GD | Current gradient → subtract ηg | Recompute at the new parameter unless gradients are explicitly supplied |
| Momentum | Retain βv → subtract ηg → add new velocity to parameter | Signed velocity and gradient-average conventions must not be mixed |
| Nesterov | Compute lookahead θ+βv → gradient there → new velocity → update actual θ | Do not add new velocity to the lookahead point |
| AdaGrad | Add g² to all-past sum G → divide gradient by √G+ε | G never decreases |
| RMSProp | Decay old squared state → add recent g² → normalize | Use the specified decay factor |
| Adam | Update signed mean m and squared mean v → correct both → normalize | Carry raw states forward; correction powers use iteration t |
Here v means signed velocity in momentum and squared-gradient state in Adam; the method defines the symbol. Memory cue: AdaGrad sab squares; RMSProp recent squares; Adam direction + squares + correction.
","L1 adds λΣ|w| and a nonzero-weight derivative λsign(w), encouraging sparsity. L2 with the half convention adds (λ/2)Σw² and derivative λw. Without that half, the derivative is 2λw. Regularization gradients add to data gradients before updating the weight.
","Inverted dropout uses keep probability q: retain a sampled activation with scale 1/q, or replace it by zero. Inference uses the full activation without that training-time rescaling. Drop probability p gives q = 1−p. Early stopping tracks validation performance and restores the best checkpoint under the stated patience rule.
","![\"a'_i=\\frac{m_i](\"assets/equations/6e936750aa2b4998.svg\")
Memory cue: dropout probability aur keep probability alag; L2 mein half hai ya nahi, pehle dekho.
","Use the deep-learning cross-correlation convention unless the question explicitly requires kernel flipping. At each output position, multiply matching input-window and kernel entries, sum over spatial entries and all input channels, and add that filter's bias once. A new filter creates a new output channel.
","Apply the shape formula separately to height and width. Pooling changes spatial dimensions but preserves channels; ordinary pooling and flattening have zero learned parameters. Maximum pooling chooses the largest window value; average pooling adds its values and divides by the number of entries. A filter's spatial reuse changes computation, not its parameter count.
","For a complete architecture, make one row per layer: input shape → operation → output shape → weights → biases → total. Flatten uses the size after the final convolution/pool. Memory cue: shape count aur parameter count ko kabhi mix mat karo.
","CNNs use local connectivity and weight sharing, reducing parameters compared with unrestricted dense image mappings. Convolution learns features; activations introduce nonlinearity; pooling summarizes local responses; dense or global-pooling/output stages make the final task prediction.
","AlexNet: stacked convolutions with ReLU, pooling and dense classification; dropout in dense layers and data augmentation help control overfitting. Inception: parallel spatial scales, 1 × 1 reductions and channel concatenation. ResNet: learns a residual F(x) and adds a shortcut x; use projection when shapes differ. Concatenation adds channels; residual addition requires equal shapes and retains their count.
","Transfer learning reuses a pretrained feature extractor, replaces the task head, trains the head and optionally fine-tunes selected pretrained layers at a suitable smaller learning rate. Explain classification, detection and segmentation as different outputs, with an appropriate concrete application.
","Revise all theory from optimization, regularization, CNN operations and CNN architectures. For 2 marks, answer exactly the requested definition, distinction or small calculation. For 3 marks, show the complete requested numerical method or explain the mechanism clearly. The paper may combine both.
","Prepare Unit 3 numericals and especially the two Unit 4 numerical opportunities highlighted by the faculty. Their exact question numbers are not specified. RNN, LSTM and GRU remain in the TEE material; they are not part of this focused MT2 revision.
","For every numerical: full question → Given → To find → governing formula and why → separate numbered steps → final answer. Carry extra calculator precision; round at the end. Label each layer, iteration and connection so the examiner can follow the calculation.
","For a definition question, state the technical meaning and the requested distinction/example. For a longer mechanism question, include a labelled diagram or equations, explain their sequence, then give the relevant benefits, limitations or comparison. Marks printed beside a historical question are its actual marks; individual steps here are not assigned invented official marks.
","For this 60-minute MT2, a workable allocation is 5 minutes to read and select, 15 minutes for Q1(a,b), 15 minutes for each selected 3-mark question and 10 minutes to review. Adjust if one calculation needs more working. If stuck, write the givens and governing formula first, complete the solvable parts, then return to the unfinished calculation.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c19-s5","search":"Methods to revise before the examination \n### Last-day revision: read in this order\nThis is a revision digest after studying the lessons. Full solutions remain beside each topic and under Solved Questions. If a step here is unclear, return to that worked example; the final day is for recovering the method, not memorizing unexplained final numbers.\n\n### 1. Learning and the neuron\nMachine learning estimates a model from data. Deep learning uses multiple trainable layers to learn successive representations. Inputs describe a sample; targets are the required answers; predictions are the model's answers. Weights and biases are learned parameters. Learning rate, batch size and architecture are chosen hyperparameters.\n\nTraining changes parameters using training examples. Validation selects settings/stopping time. Testing evaluates the final choice on held-out data. Overfitting means strong training performance with weak generalization; underfitting means the model has not captured enough of the relationship.\n\nFor one neuron, multiply each input by its own weight, add all products, add the bias once and apply the activation. A layer repeats this for several receiving neurons. Each weight-matrix row belongs to one receiving neuron. Transpose swaps rows and columns; it does not change signs.\n\n**Memory cue:** multiply → sum → bias → activate. In a full network, finish one complete layer before the next.\n\n### 2. Activation and loss choices\nSigmoid maps to (0,1) and is used for binary-class probabilities. Tanh maps to (−1,1), centred on zero. Both can saturate, producing small derivatives. ReLU retains positive values and returns zero on the negative side; its negative-side gradient is zero. Leaky ReLU retains a specified small negative-side slope. Softmax normalizes a complete list of logits into mutually exclusive class probabilities.\n\n\n\nFor squared errors, SSE sums squares, half SSE divides that sum by 2, and MSE divides it by the number of scalar predictions. Their derivatives differ. For one-hot categorical cross-entropy, only the correct-class probability contributes. For binary cross-entropy, p means probability of class 1, not automatically the true class.\n\n\n\n\n**Memory cue:** loss ka denominator dekho before differentiating. Sigmoid output is not its derivative.\n\n### 3. Backpropagation: the repeatable method\nWrite the given loss and activation first. Compute all forward sums/activations with the original parameters. Differentiate loss with respect to each output, multiply by the output activation derivative, and call the result the output delta.\n\nFor a hidden neuron, add the outgoing weighted deltas first and then multiply by the hidden activation derivative. Each connection gradient is receiving delta × sending activation. The bias gradient is the receiving delta. Use old weights for all gradients before making the simultaneous update.\n\n\n\n**Memory cue:** path mein multiply; paths milne par add; update sab gradients ke baad. For sigmoid+BCE or softmax+CCE, the logit delta simplifies to prediction−target for the stated single-example sum convention.\n\n### 4. Optimizers: identify the stored state\nFull-batch GD uses the entire dataset for one update; SGD uses one sample; mini-batch uses a subset. An epoch is one pass through the data. With a final incomplete batch included, updates per epoch are ceiling(N/B).\n\n| Method | State and calculation order | Detail that changes the answer |\n| GD | Current gradient → subtract ηg | Recompute at the new pa"}]},{"n":20,"title":"Course vocabulary in plain words","sections":[{"title":"Learning and the numbers we adjust","blocks":["| Term | Meaning | Tiny example |
|---|---|---|
| Input / feature | A number given to the model | Image pixel intensity |
| Sample | One example processed by the model | One labeled image |
| Dataset | A collection of examples | 1,000 training images |
| Target / label | Correct answer supplied for training | Class 2 is correct |
| Prediction | Model's current answer | Class probabilities |
| Weight | Learned multiplier on a connection | 0.4 multiplying an input |
| Bias | Learned value added to a weighted sum | z = wx + b |
| Parameter | A value learned during training | Weight or bias |
| Hyperparameter | A setting chosen for training | Learning rate or batch size |
| Activation | Transformation of a neuron's raw score | ReLU or sigmoid |
| Loss | Numerical measure of prediction error | Mean squared error |
| Gradient | Local rate of loss change for parameters | Positive g means locally uphill |
| Optimizer | Rule using gradients to update parameters | Momentum or Adam |
| Epoch | One pass through the training dataset | All 100 samples processed |
| Iteration | One parameter update in these exercises | One mini-batch update |
| Inference | Applying the trained model | Predicting a new image's class |
| Term | Meaning | What to keep separate |
|---|---|---|
| Tensor | Array of numbers with one or more axes | Its shape and its values |
| Channel | One feature map within an image-like tensor | Channels versus spatial size |
| Kernel / filter | Learned weights sliding over a map | Kernel size versus filter count |
| Stride | Step between adjacent window positions | Stride versus kernel width |
| Padding | Extra border cells around an input | Per-side P versus total 2P |
| Feature map | Responses from a filter across positions | Activations versus parameters |
| Pooling | Window-based reduction of feature maps | Maximum versus average |
| Flatten | Rearrange H × W × C entries as a vector | Rearrangement has no parameters |
| Logit | Raw score before an output activation | Score versus probability |
| Hidden state | Recurrent features carried through time | Previous state versus current input |
| Shared weights | One learned rule reused at many positions/times | Reuse versus extra parameters |
“Calculate” asks for arithmetic and a result. Write the formula and substitutions before the final number.
","“Explain” asks how and why a method works. Give its definition, describe the steps, and connect each step to its purpose.
","“Compare” asks for matched differences. Discuss the same attributes for both methods, such as stored state, update rule, benefit and limitation.
","“Draw” asks for a labeled structure. Label inputs, layers or time steps, arrows, activations and the output. A diagram and explanation should describe the same computation.
","“Design” asks for a justified arrangement. State the task, input and output, chosen layers, activation/loss pair and training procedure. State any assumption needed because the question leaves a value unspecified.
"],"question":[],"answer":[],"kind":"read","id":"mt2-c20-s2","search":"Reading a question precisely “Calculate” asks for arithmetic and a result. Write the formula and substitutions before the final number.\n“Explain” asks how and why a method works. Give its definition, describe the steps, and connect each step to its purpose.\n“Compare” asks for matched differences. Discuss the same attributes for both methods, such as stored state, update rule, benefit and limitation.\n“Draw” asks for a labeled structure. Label inputs, layers or time steps, arrows, activations and the output. A diagram and explanation should describe the same computation.\n“Design” asks for a justified arrangement. State the task, input and output, chosen layers, activation/loss pair and training procedure. State any assumption needed because the question leaves a value unspecified."}]}],"tee":[{"n":1,"title":"TEE preparation and examination plan","sections":[{"title":"Coverage and examination structure","blocks":["The TEE route covers all five units of course 702AI0C036: introduction and neural-network foundations, optimization and regularization, CNNs and their architectures, and recurrent models including BPTT, LSTM and GRU. Machine-learning prerequisites are explained within the course. Basic algebra and calculus are assumed, with a concise revision chapter available when required.
","The supplied course policy specifies a three-hour, 100-mark TEE: Q1 is compulsory, followed by any four of Q2–Q7; each main question carries 20 marks. TEE marks are scaled to 50. Read the instructions printed on the actual examination paper.
"],"question":[],"answer":[],"kind":"read","id":"tee-c1-s0","search":"Coverage and examination structure The TEE route covers all five units of course 702AI0C036: introduction and neural-network foundations, optimization and regularization, CNNs and their architectures, and recurrent models including BPTT, LSTM and GRU. Machine-learning prerequisites are explained within the course. Basic algebra and calculus are assumed, with a concise revision chapter available when required.\n\nThe supplied course policy specifies a three-hour, 100-mark TEE: Q1 is compulsory, followed by any four of Q2–Q7; each main question carries 20 marks. TEE marks are scaled to 50. Read the instructions printed on the actual examination paper.\n"},{"title":"Recommended reading order","blocks":["1. Neural-network foundations: learning tasks, data partitions, neuron models, perceptrons, activations, losses and backpropagation.
","2. Optimization: gradient descent variants, momentum, Nesterov, AdaGrad, RMSProp and Adam, including their states and update conventions.
","3. Regularization and generalization: overfitting, L1/L2 objectives, dropout, early stopping and remedies selected from training behavior.
","4. CNNs: convolution and pooling operations, output dimensions, trainable parameter counts and complete network calculations.
","5. CNN architectures: AlexNet, Inception, ResNet, bottlenecks and transfer learning.
","6. Recurrent models: state equations, BPTT, vanishing/exploding gradients, LSTM and GRU.
","7. Examination preparation: use the formula reference, complete solved questions and full TEE paper rehearsal.
","Use the Mathematics revision chapter for notation, matrix operations, differentiation, probability and logarithms. It is a reference rather than the compulsory starting point.
"],"question":[],"answer":[],"kind":"read","id":"tee-c1-s1","search":"Recommended reading order 1. **Neural-network foundations:** learning tasks, data partitions, neuron models, perceptrons, activations, losses and backpropagation.\n2. **Optimization:** gradient descent variants, momentum, Nesterov, AdaGrad, RMSProp and Adam, including their states and update conventions.\n3. **Regularization and generalization:** overfitting, L1/L2 objectives, dropout, early stopping and remedies selected from training behavior.\n4. **CNNs:** convolution and pooling operations, output dimensions, trainable parameter counts and complete network calculations.\n5. **CNN architectures:** AlexNet, Inception, ResNet, bottlenecks and transfer learning.\n6. **Recurrent models:** state equations, BPTT, vanishing/exploding gradients, LSTM and GRU.\n7. **Examination preparation:** use the formula reference, complete solved questions and full TEE paper rehearsal.\n\nUse the Mathematics revision chapter for notation, matrix operations, differentiation, probability and logarithms. It is a reference rather than the compulsory starting point.\n"},{"title":"Theory answers and numerical solutions","blocks":["For theory, write a precise definition, equations or a labelled architecture, the working mechanism, and the relevant properties or limitations. For comparison questions, compare the same aspects on both sides. A table should support the explanation rather than replace it with disconnected keywords.
","For numericals, state the given values and conventions, show the governing formula, calculate each intermediate state, and identify the final result. Keep unrounded intermediate values when using the calculator. In backpropagation, calculate all gradients using the same forward-pass parameters before updating them.
"],"question":[],"answer":[],"kind":"read","id":"tee-c1-s2","search":"Theory answers and numerical solutions For theory, write a precise definition, equations or a labelled architecture, the working mechanism, and the relevant properties or limitations. For comparison questions, compare the same aspects on both sides. A table should support the explanation rather than replace it with disconnected keywords.\n\nFor numericals, state the given values and conventions, show the governing formula, calculate each intermediate state, and identify the final result. Keep unrounded intermediate values when using the calculator. In backpropagation, calculate all gradients using the same forward-pass parameters before updating them.\n"},{"title":"Practice resources","blocks":["The reading flow places worked numericals and relevant examination questions beside their concepts. Solved Questions also provides separate lists of practice numericals, past examination questions and theory answers. Every question has its answer directly below it.
","The supplied historical papers are filtered against the current syllabus. The TEE mock uses the December 2025 paper as a three-hour rehearsal with full solutions. This is historical practice; it is not a prediction of the forthcoming paper.
","The course policy allocates 10 marks to Class Test 2. The shared confirmed pattern is Q1(a,b), 2 marks each compulsory, followed by any two of Q2–Q4, 3 marks each, in one hour. The faculty message relayed on 12 October identifies Unit 3 and Unit 4 theory and numericals, highlighting two Unit 4 numerical questions. It describes alternative theory/numerical combinations without fixing their exact question numbers. Use the MT2 route for that focused preparation; the TEE route covers all five units.
","TEE has 100 marks in three hours. Q1 is compulsory, then answer any four of Q2–Q7. Each main question carries 20 marks. The supplied 2023–2026 end papers use four 5-mark parts in Q1 and two 10-mark parts in later main questions. The older 2022/January 2023 papers use five 4-mark parts in Q1. This is an observed historical change, not a prediction of the next paper's subparts.
","| Asked question family | Examples in the supplied papers | What the answer must contain |
|---|---|---|
| Neuron / activation calculation | MT1A Q2–Q3 (3 marks each); R26 Q6b (10 marks, mixed calculation and explanation) | Matching products, bias once, activation, requested derivative/probabilities |
| Forward pass and backpropagation | MT1A Q4 (3); F25 Q2a, R26 Q2a, R24 Q2a (10 each) | Network data and loss convention, forward values, chain-rule factors, requested gradients and updates |
| Perceptron training | F25 Q3a, R26 Q3a, R24 Q4a (10 each) | Threshold and target convention, every sample in order, update only as required, final weights |
| Optimizer and update-count questions | F25 Q4a and R26 Q4a (10 each) | Differentiate when needed, initialize state, show each iteration; recompute or use supplied gradients correctly |
| Regularization / overfitting | R26 Q1c (5); R24 Q3b and R23 Q7b (10 each) | Definition, mechanism, equations where relevant, training/inference distinction, suitable remedy |
| CNN map / shape / parameter calculations | F25 Q6a, R26 Q6a, R24 Q4b (10 each) | Every map entry if requested, each layer shape, filter weights and biases, pooling zero parameters, dense count |
| Architectures and comparison | F25 Q2b, F23 Q5a, R23 Q4a (10 each) | Labelled structure, working mechanism, reason for design, advantages and limitations |
| Recurrent state / LSTM / GRU | R26 Q5a, F25 Q5a, R24 Q6a (10 each) | State/gate definitions, correct activation, carried state, requested calculations and comparison |
These are examples of question types, not an official marking rubric or an assurance that topics repeat. The TEE route retains 108 mapped question parts across the two MT1 sets and six end-semester papers. Two historical parts are restricted to their current-syllabus content. A supplied model answer does not override an explicit loss, dimension or convention in the question.
"],"question":[],"answer":[],"kind":"read","id":"tee-c1-s3","search":"Practice resources The reading flow places worked numericals and relevant examination questions beside their concepts. Solved Questions also provides separate lists of practice numericals, past examination questions and theory answers. Every question has its answer directly below it.\n\nThe supplied historical papers are filtered against the current syllabus. The TEE mock uses the December 2025 paper as a three-hour rehearsal with full solutions. This is historical practice; it is not a prediction of the forthcoming paper.\n\n\n### How the supplied papers ask questions\nThe course policy allocates **10 marks to Class Test 2**. The shared confirmed pattern is Q1(a,b), 2 marks each compulsory, followed by any two of Q2–Q4, 3 marks each, in one hour. The faculty message relayed on 12 October identifies **Unit 3 and Unit 4 theory and numericals**, highlighting two Unit 4 numerical questions. It describes alternative theory/numerical combinations without fixing their exact question numbers. Use the MT2 route for that focused preparation; the TEE route covers all five units.\n\nTEE has **100 marks in three hours**. Q1 is compulsory, then answer any four of Q2–Q7. Each main question carries 20 marks. The supplied 2023–2026 end papers use four 5-mark parts in Q1 and two 10-mark parts in later main questions. The older 2022/January 2023 papers use five 4-mark parts in Q1. This is an observed historical change, not a prediction of the next paper's subparts.\n\n| Asked question family | Examples in the supplied papers | What the answer must contain |\n| Neuron / activation calculation | MT1A Q2–Q3 (3 marks each); R26 Q6b (10 marks, mixed calculation and explanation) | Matching products, bias once, activation, requested derivative/probabilities |\n| Forward pass and backpropagation | MT1A Q4 (3); F25 Q2a, R26 Q2a, R24 Q2a (10 each) | Network data and loss convention, forward values, chain-rule factors, requested gradients and updates |\n| Perceptron training | F25 Q3a, R26 Q3a, R24 Q4a (10 each) | Threshold and target convention, every sample in order, update only as required, final weights |\n| Optimizer and update-count questions | F25 Q4a and R26 Q4a (10 each) | Differentiate when needed, initialize state, show each iteration; recompute or use supplied gradients correctly |\n| Regularization / overfitting | R26 Q1c (5); R24 Q3b and R23 Q7b (10 each) | Definition, mechanism, equations where relevant, training/inference distinction, suitable remedy |\n| CNN map / shape / parameter calculations | F25 Q6a, R26 Q6a, R24 Q4b (10 each) | Every map entry if requested, each layer shape, filter weights and biases, pooling zero parameters, dense count |\n| Architectures and comparison | F25 Q2b, F23 Q5a, R23 Q4a (10 each) | Labelled structure, working mechanism, reason for design, advantages and limitations |\n| Recurrent state / LSTM / GRU | R26 Q5a, F25 Q5a, R24 Q6a (10 each) | State/gate definitions, correct activation, carried state, requested calculations and comparison |\n\nThese are examples of question types, not an official marking rubric or an assurance that topics repeat. The TEE route retains 108 mapped question parts across the two MT1 sets and six end-semester papers. Two historical parts are restricted to their current-syllabus content. A supplied model answer does not override an explicit loss, dimension or convention in the question.\n\n"},{"id":"tee-pyq-analysis","title":"All recovered TEE papers: unit, topic and marks map","kind":"read","blocks":["Start with the newest final and re-exam, then use older papers to practise variations. The table links each original lettered part to its complete worked answer. Repeated questions are history, not a prediction. A syllabus topic stays required even when no recovered paper asks it.
Marks: 100 marks are attempted: compulsory Q1 (20), then any four other 20-mark questions. Most recovered papers offer140 marks; NLP December2023 offers120 because the complete original ends at Q6. A mixed question retains one mark allocation; its units are not counted as separate questions. Blank individual allocations remain unknown.
140 marks offered; 100 attempted.
| Question [marks] | Current unit and topic | Question form | Answer / scope |
|---|---|---|---|
| Q1a [5] | Unit 1 · ML versus DL task choices | case | Read question + answer |
| Q1b [5] | Unit 2 · Activation functions | theory | Read question + answer |
| Q1c [5] | Unit 3 · L1/L2 and specified-rate dropout | mixed | Read question + answer |
| Q1d [5] | Unit 5 · RNN sentiment classification | case | Read question + answer |
| Q2a [10] | Unit 2 · Two-output sigmoid backpropagation | numerical | Read question + answer · Loss omitted in original; answer states assumed half summed squared loss. |
| Q2b [10] | Unit 4 · AlexNet, Inception and architecture recommendation | case | Read question + answer |
| Q3a [10] | Unit 2 · Bipolar perceptron epoch | numerical | Read question + answer |
| Q3b [10] | Unit 4 · Five effects of CNN filter count | theory | Read question + answer |
| Q4a [10] | Unit 3 · Batch/SGD, learning rate and cubic-objective updates | mixed | Read question + answer |
| Q4b [10] | Unit 5 · LSTM and GRU; legacy bidirectional-RNN subpart | theory | Read question + answer · Current-syllabus extract; original part also contains Bidirectional-RNN advantages (iv) |
| Q5a [10] | Unit 5 · RNN hidden-state matrix calculation | numerical | Read question + answer |
| Q5b [10] | Unit 3 · RMSProp mechanism and SGD comparison | theory | Read question + answer |
| Q6a [10] | Unit 4 · CNN sizes and counts with binary-output ambiguity | numerical | Read question + answer · Binary head unspecified: one sigmoid or two softmax units give different parameter totals; both are stated. |
| Q6b [10] | Unit 2 · Sigmoid value/derivative and softmax | numerical | Read question + answer |
| Q7a [10] | Unit 2 · Feedforward roles and sigmoid forward pass | mixed | Read question + answer |
| Q7b [10] | Unit 4 · Transfer learning and its application | case | Read question + answer |
Original offered marks by mapped topic, grouped without double counting: Unit 1: 1 parts, 5 known marks; Unit 2: 5 parts, 45 known marks; Unit 3: 3 parts, 25 known marks; Unit 5: 3 parts, 25 known marks; Unit 4: 4 parts, 40 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint.
Current-syllabus qualification: Q4b: mixed current and legacy tasks; the retained-only marks cannot be determined from the printed allocation. The original whole-part marks remain visible so the paper is faithfully represented.
140 marks offered; 100 attempted.
| Question [marks] | Current unit and topic | Question form | Answer / scope |
|---|---|---|---|
| Q1a [5] | Unit 1 · ML versus DL and three task choices | case | Read question + answer |
| Q1b [5] | Unit 2 · Loss choice and categorical cross-entropy | mixed | Read question + answer |
| Q1c [5] | Unit 3 · L1 sparsity, overfitting and dropout | case | Read question + answer |
| Q1d [5] | Unit 5 · RNN context and variable-length translation | case | Read question + answer |
| Q2a [10] | Unit 2 · Sigmoid network: forward pass and backpropagation | numerical | Read question + answer · Original omits inputs and loss. Answer explicitly assumes x=[1,1] and half squared loss; the result is conditional on those assumptions. |
| Q2b [10] | Unit 4 · ResNet; Inception versus AlexNet | theory | Read question + answer |
| Q3a [10] | Unit 2 · Biological neuron and perceptron epoch | mixed | Read question + answer |
| Q3b [10] | Unit 4 · Transfer learning for face attendance | case | Read question + answer |
| Q4a [10] | Unit 3 · Batch counts, momentum iterations and hyperparameters | mixed | Read question + answer |
| Q4b [10] | Unit 5 · Vanishing gradients and LSTM forget contribution | mixed | Read question + answer |
| Q5a [10] | Unit 5 · LSTM gates, cell state and hidden state | mixed | Read question + answer |
| Q5b [10] | Unit 3 · AdaGrad mechanism and comparison with SGD | theory | Read question + answer |
| Q6a [10] | Unit 4 · CNN sizes and parameters; stride-2 convolution | numerical | Read question + answer |
| Q6b [10] | Unit 2 · Activation functions and their roles | theory | Read question + answer |
| Q7a [10] | Unit 2 · Feedforward network roles and ReLU forward pass | mixed | Read question + answer · Receiving-neuron rows give 0.55; the supplied key uses a transposed convention (0.61). Both interpretations are explained. |
| Q7b [10] | Unit 4 · CNN feature hierarchy and applications | theory | Read question + answer |
Original offered marks by mapped topic, grouped without double counting: Unit 1: 1 parts, 5 known marks; Unit 2: 5 parts, 45 known marks; Unit 3: 3 parts, 25 known marks; Unit 5: 3 parts, 25 known marks; Unit 4: 4 parts, 40 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint.
140 marks offered; 100 attempted. Combined-year archive filename contains one six-page exam, not two independent papers.
| Question [marks] | Current unit and topic | Question form | Answer / scope |
|---|---|---|---|
| Q1a [5] | Unit 1, 4, 5 · ML versus DL and architecture choice | case | Read question + answer |
| Q1b [5] | Unit 3 · Training and validation errors | case | Read question + answer |
| Q1c [5] | Unit 4 · CNN hierarchy and stride-two convolution | mixed | Read question + answer · Padding is not specified; valid/no-padding is declared. CNN cross-correlation convention is used without kernel reversal. |
| Q1d [5] | Unit 5 · BPTT and recurrent strengths | theory | Read question + answer |
| Q2a [10] | Unit 2 · Activation, ReLU and numeric outputs | mixed | Read question + answer |
| Q2b [10] | Unit 5 · Vanishing/exploding gradients and GRU | theory | Read question + answer |
| Q3a [10] | Unit 2, 3 · 4–3–2 forward pass and dropout | numerical | Read question + answer · Activation/loss/dropout scaling missing from original; result is conditional on explicit standard classroom assumptions. |
| Q3b [10] | Unit 4 · Four CNN design explanations | theory | Read question + answer |
| Q4a [10] | Unit 4 · 64×64 CNN shape and parameter table | numerical | Read question + answer |
| Q4b [10] | Unit 2 · Bipolar AND perceptron | numerical | Read question + answer · Initialization, order and threshold convention explicitly supplied as assumptions; multiple equivalent separating weights exist. |
| Q5a [10] | Unit 4 · ResNet shortcut and Inception1×1 | theory | Read question + answer |
| Q5b [10] | Unit 5 · ReLU RNN state with declared orientation | numerical | Read question + answer · Square matrices and printed horizontal vectors leave orientation ambiguous; both possible readings are explained, main answer follows row-vector form. |
| Q6a [10] | Unit 2 · 3–2–1 forward pass and update w7,w8 | numerical | Read question + answer · Original does not specify loss scaling; half-SSE is declared and alternatives distinguished. |
| Q6b [10] | Unit 3 · AdaGrad stagnation versus RMSProp | case | Read question + answer |
| Q7a [10] | Unit 5 · Long-text sentiment architecture | case | Read question + answer |
| Q7b [5] | Unit 4 · Transfer learning | theory | Read question + answer |
| Q7c [5] | Unit 2, 3 · Learning rate and two losses | theory | Read question + answer |
Original offered marks by mapped topic, grouped without double counting: Unit 1, 4, 5: 1 parts, 5 known marks; Unit 3: 2 parts, 15 known marks; Unit 4: 5 parts, 40 known marks; Unit 5: 4 parts, 35 known marks; Unit 2: 3 parts, 30 known marks; Unit 2, 3: 2 parts, 15 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint.
140 marks offered; 100 attempted.
| Question [marks] | Current unit and topic | Question form | Answer / scope |
|---|---|---|---|
| Q1a [5] | Unit 1 · Drivers of deep learning adoption | theory | Read question + answer |
| Q1b [5] | Unit 2 · 10-5-5-5-4 dense network parameter count | numerical | Read question + answer |
| Q1c [5] | Unit 3 · Overfitting diagnosis and remedies | case | Read question + answer |
| Q1d [5] | Unit 4 · Max and average pooling | numerical | Read question + answer |
| Q2a [10] | Unit 2 · Four-hidden-neuron sigmoid MSE and update | numerical | Read question + answer |
| Q2b [10] | Unit 3 · Momentum/Nesterov; high momentum; excluded convex rates | theory | Read question + answer · Current-syllabus extract; original part also contains Asymptotic convergence-rate comparison on convex problems (part of i) |
| Q3a [10] | Unit 2 · Activation comparisons and softmax | mixed | Read question + answer |
| Q3b [10] | Unit 3 · Regularization importance and techniques | theory | Read question + answer |
| Q4a [10] | Unit 2 · Perceptron training for two epochs | numerical | Read question + answer |
| Q4b [10] | Unit 4 · CNN sizes, weights and biases | numerical | Read question + answer |
| Q5a [10] | Unit 4 · Replacing architecture to improve gradient flow | case | Read question + answer |
| Q5b [10] | Unit 1, 5 · X-ray DL justification; RNN pros and cons | case | Read question + answer |
| Q6a [10] | Unit 5 · GRU update and reset gates | theory | Read question + answer |
| Q6b [10] | Unit 4 · Pretrained vegetable classifier | case | Read question + answer |
| Q7a [10] | Unit 5 · LSTM architecture | theory | Read question + answer |
| Q7b [10] | Unit 5 · RNN input/output sequence architectures | theory | Read question + answer |
Original offered marks by mapped topic, grouped without double counting: Unit 1: 1 parts, 5 known marks; Unit 2: 4 parts, 35 known marks; Unit 3: 3 parts, 25 known marks; Unit 4: 4 parts, 35 known marks; Unit 1, 5: 1 parts, 10 known marks; Unit 5: 3 parts, 30 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint.
Current-syllabus qualification: Q2b: mixed current and legacy tasks; the retained-only marks cannot be determined from the printed allocation. The original whole-part marks remain visible so the paper is faithfully represented.
140 marks offered; 100 attempted.
| Question [marks] | Current unit and topic | Question form | Answer / scope |
|---|---|---|---|
| Q1a [5] | Unit 1 · Deep learning features and limitations | theory | Read question + answer |
| Q1b [5] | Unit 2, 3 · ANN hyperparameters and significance | theory | Read question + answer |
| Q1c [5] | Unit 5 · Unrolled RNN and BPTT | theory | Read question + answer |
| Q1d [5] | Unit 4 · Convolution map | numerical | Read question + answer |
| Q2a [10] | Unit 2 · Binary-input step-activation network | numerical | Read question + answer |
| Q2b [10] | Unit 2, 1 · Activation necessity; ML/DL facial-recognition choice | mixed | Read question + answer |
| Q3a [10] | Unit 2 · Sigmoid output-weight backpropagation | numerical | Read question + answer |
| Q3b [10] | Unit 3 · Adam advantages versus SGD | theory | Read question + answer |
| Q4a [10] | Unit 4 · AlexNet-style layer dimensions and parameters | numerical | Read question + answer |
| Q4b [10] | Unit 5 · LSTM autocomplete architecture and token probabilities | case | Read question + answer |
| Q5a [10] | Unit 4 · Inception branches and 1x1 convolution | theory | Read question + answer |
| Q5b [10] | Unit 2 · Linear separability; XOR and XNOR construction | mixed | Read question + answer |
| Q6a [10] | Unit 5 · LSTM gates; source mixes reset/update terminology | theory | Read question + answer · Printed LSTM diagram uses update-gate wording and asks reset equations. Answer separates LSTM and GRU rather than inventing an LSTM reset gate. |
| Q6b [10] | Unit 4 · ResNet gradient flow and block outputs | theory | Read question + answer |
| Q7a [10] | Unit 5 · Paragraph sentiment model | case | Read question + answer |
| Q7b [10] | Unit 3 · Bias-variance tradeoff and intervention table | theory | Read question + answer |
Original offered marks by mapped topic, grouped without double counting: Unit 1: 1 parts, 5 known marks; Unit 2, 3: 1 parts, 5 known marks; Unit 5: 4 parts, 35 known marks; Unit 4: 4 parts, 35 known marks; Unit 2: 3 parts, 30 known marks; Unit 2, 1: 1 parts, 10 known marks; Unit 3: 2 parts, 20 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint.
140 marks offered; 100 attempted.
| Question [marks] | Current unit and topic | Question form | Answer / scope |
|---|---|---|---|
| Q1a [4] | Unit 5 · Music-generation RNN architecture | case | Read question + answer |
| Q1b [4] | Unit 4 · Stride-2 average pooling and purpose | numerical | Read question + answer |
| Q1c [4] | Unit 1 · DL advantages and limitations | theory | Read question + answer |
| Q1d [4] | Unit 3 · Mini-batch size and parallelism | theory | Read question + answer |
| Q1e [4] | Unit 2 · AND perceptron design | numerical | Read question + answer |
| Q2a [10] | Unit 2, 3, 4 · CNN parameters, hyperparameters and Adam | theory | Read question + answer |
| Q2b [10] | Unit 2, 3 · ReLU, rating regression and overfitting | case | Read question + answer |
| Q3a [10] | Unit 5 · RNN language model and per-token probabilities | case | Read question + answer |
| Q3b [10] | Unit 2 · ReLU/sigmoid forward pass and four output-weight updates | numerical | Read question + answer |
| Q4a [10] | Unit 4 · Small-data face-attendance transfer learning | case | Read question + answer |
| Q4b [10] | Unit 4 · AlexNet parameter count | numerical | Read question + answer |
| Q5a [10] | Unit 5 · Vanishing-gradient RNN remedies | case | Read question + answer |
| Q5b [10] | Unit 2 · Fraud classifier design and parameter count | mixed | Read question + answer |
| Q6a [10] | Unit 3 · Three optimization-failure scenarios | case | Read question + answer |
| Q6b [10] | Unit 4 · Convolution cost and unspecified bottleneck width | numerical | Read question + answer · The 1x1 reduction width is not specified. Answer gives a symbolic count and an explicitly assumed numerical example. |
| Q7a [10] | Unit 5 · LSTM sentence memory and gates | case | Read question + answer |
| Q7b [10] | Unit 3 · Overfitting diagnosis and remedies | case | Read question + answer |
Original offered marks by mapped topic, grouped without double counting: Unit 5: 4 parts, 34 known marks; Unit 4: 4 parts, 34 known marks; Unit 1: 1 parts, 4 known marks; Unit 3: 3 parts, 24 known marks; Unit 2: 3 parts, 24 known marks; Unit 2, 3, 4: 1 parts, 10 known marks; Unit 2, 3: 1 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint.
140 marks offered; 100 attempted.
| Question [marks] | Current unit and topic | Question form | Answer / scope |
|---|---|---|---|
| Q1a [4] | Unit 4 · Stride-2 convolution and detected feature | numerical | Read question + answer |
| Q1b [4] | Unit 2 · Signed gradient-descent update | numerical | Read question + answer |
| Q1c [4] | Unit 1 · ML and DL comparison | theory | Read question + answer |
| Q1d [4] | Unit 2 · Sigmoid versus tanh | theory | Read question + answer |
| Q1e [4] | Unit 5 · Image-captioning RNN architecture | case | Read question + answer |
| Q2a [10] | Unit 5 · GRU advantages and subject memory in a sentence | case | Read question + answer |
| Q2b [10] | Unit 2 · Two-sample sigmoid forward pass and mean-MSE backprop | numerical | Read question + answer |
| Q3a [10] | Unit 2, 3 · Bias/variance, four-class activation/loss and diagnosis | case | Read question + answer |
| Q3b [10] | Unit 4 · LeNet-style parameter count | numerical | Read question + answer |
| Q4a [10] | Unit 4 · Inception bottleneck computation count | numerical | Read question + answer |
| Q4b [10] | Unit 2, 3 · 6-5-5-4 network; AdaGrad parameters and hyperparameters | theory | Read question + answer |
| Q5a [10] | Unit 5 · RNN model and BPTT updates | theory | Read question + answer |
| Q5b [10] | Unit 3 · Optimization challenges and momentum | theory | Read question + answer |
| Q6a [10] | Unit 4 · CNN vanishing-gradient remedies | case | Read question + answer |
| Q6b [10] | Unit 2, 3 · Clothing classifier design and six diagnostic tasks | case | Read question + answer |
| Q7a [10] | Unit 5 · LSTM long-sequence language model | case | Read question + answer |
| Q7b [10] | Unit 2 · Perceptron non-linear separability and solution | theory | Read question + answer |
Original offered marks by mapped topic, grouped without double counting: Unit 4: 4 parts, 34 known marks; Unit 2: 4 parts, 28 known marks; Unit 1: 1 parts, 4 known marks; Unit 5: 4 parts, 34 known marks; Unit 2, 3: 3 parts, 30 known marks; Unit 3: 1 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint.
The latest papers show the form and depth of questions, while the official current syllabus sets the learning scope. Authored drills, explanatory numbers and optional extensions are not counted as past questions. Source assumptions and ambiguous data are stated in their answers.
"],"question":[],"answer":[]}]},{"n":2,"title":"Mathematics revision","sections":[{"title":"Before maths: what are we trying to learn?","blocks":["Imagine a tiny machine that estimates a bill. You tell it how many notebooks you bought. This information going INTO the machine is called the INPUT. Abhi bas naam samjho: input = andar diya hua number.
","Calculation: Input = 3 notebooks
","Suppose the machine says the bill is 30 rupees. This answer coming OUT is called the OUTPUT. Input aur output alag jobs hain: 3 was the quantity; 30 is the predicted bill.
","Calculation: Input: 3 notebooks → Output: 30 rupees
","If each notebook costs 10 rupees, three notebooks cost 10 + 10 + 10 = 30. Multiplication is just the shorter calculation: 3 × 10. In this simple model the multiplying number is called a WEIGHT.
","Calculation: 3 × 10 = 30
","Keep the input 3. Change only the weight from 10 to 8. The output becomes 3 × 8 = 24. A weight controls how strongly an input affects the answer. Neural-network weights can also be negative or fractions.
","Calculation: Old: 3 × 10 = 30. New: 3 × 8 = 24.
","The real bill is 30, but the machine predicted 24. The correct answer supplied for a training example is its TARGET. Our prediction is 6 too small because 30 − 24 = 6.
","Calculation: Target = 30. Prediction = 24. Difference = 6.
","For this tiny example, changing the weight from 8 to 10 changes the prediction from 24 to the correct 30. Learning means using examples to adjust weights and reduce prediction error. Later, gradient descent will calculate HOW to adjust them.
","Calculation: 3 × 8 = 24 → adjust weight → 3 × 10 = 30
","Now suppose there is a fixed packing fee of 2 rupees. First calculate the notebook cost: 3 × 10 = 30. Then add 2. That extra adjustable number is called a BIAS. In a neural network, it lets the result shift independently of the input.
","Calculation: 3 × 10 + 2 = 30 + 2 = 32
","Instead of writing input, weight and bias every time, books use short names. x means input; w means weight; b means bias. So x × w + b says exactly the calculation you already did. A letter is a named box holding a number.
","Calculation: x = 3, w = 10, b = 2 → x × w + b = 32
","A neural network connects many small calculating units called neurons. They multiply inputs by weights, add a bias and apply an activation rule. Several layers can learn more complex relationships. First we will learn the small calculation; phir layers jodenge.
","Calculation: Input → calculations in layers → prediction
","Question. A toy model has input 4, weight 5 and bias 2. Find its output using input × weight + bias.
","A toy model has input 4, weight 5 and bias 2.
","Find the preactivation and the requested neuron output.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Step 1: multiply input by weight: 4 × 5 = 20.
","Step 2: add the bias: 20 + 2 = 22. Final output = 22. The bias is added once, after multiplication.
"],"question":[],"answer":[],"kind":"read","id":"tee-c2-s0","search":"Before maths: what are we trying to learn? \n### Machine ko ek number do.\nImagine a tiny machine that estimates a bill. You tell it how many notebooks you bought. This information going INTO the machine is called the INPUT. Abhi bas naam samjho: input = andar diya hua number.\n\n\n\n**Calculation:** Input = 3 notebooks\n\n### Machine se ek answer niklega.\nSuppose the machine says the bill is 30 rupees. This answer coming OUT is called the OUTPUT. Input aur output alag jobs hain: 3 was the quantity; 30 is the predicted bill.\n\n\n\n**Calculation:** Input: 3 notebooks → Output: 30 rupees\n\n### Multiply kyun karte hain?\nIf each notebook costs 10 rupees, three notebooks cost 10 + 10 + 10 = 30. Multiplication is just the shorter calculation: 3 × 10. In this simple model the multiplying number is called a WEIGHT.\n\n\n\n**Calculation:** 3 × 10 = 30\n\n### Weight badla, answer badla.\nKeep the input 3. Change only the weight from 10 to 8. The output becomes 3 × 8 = 24. A weight controls how strongly an input affects the answer. Neural-network weights can also be negative or fractions.\n\n\n\n**Calculation:** Old: 3 × 10 = 30. New: 3 × 8 = 24.\n\n### Correct answer ko target kehte hain.\nThe real bill is 30, but the machine predicted 24. The correct answer supplied for a training example is its TARGET. Our prediction is 6 too small because 30 − 24 = 6.\n\n\n\n**Calculation:** Target = 30. Prediction = 24. Difference = 6.\n\n### Learning ka matlab weight improve karna.\nFor this tiny example, changing the weight from 8 to 10 changes the prediction from 24 to the correct 30. Learning means using examples to adjust weights and reduce prediction error. Later, gradient descent will calculate HOW to adjust them.\n\n\n\n**Calculation:** 3 × 8 = 24 → adjust weight → 3 × 10 = 30\n\n### Bias ek extra add hone wala number hai.\nNow suppose there is a fixed packing fee of 2 rupees. First calculate the notebook cost: 3 × 10 = 30. Then add 2. That extra adjustable number is called a BIAS. In a neural network, it lets the result shift independently of the input.\n\n\n\n**Calculation:** 3 × 10 + 2 = 30 + 2 = 32\n\n### Letters numbers ke naam hain.\nInstead of writing input, weight and bias every time, books use short names. x means input; w means weight; b means bias. So x × w + b says exactly the calculation you already did. A letter is a named box holding a number.\n\n\n\n**Calculation:** x = 3, w = 10, b = 2 → x × w + b = 32\n\n### Deep learning mein aage kya hoga?\nA neural network connects many small calculating units called neurons. They multiply inputs by weights, add a bias and apply an activation rule. Several layers can learn more complex relationships. First we will learn the small calculation; phir layers jodenge.\n\n\n\n**Calculation:** Input → calculations in layers → prediction\n\n**Question.** A toy model has input 4, weight 5 and bias 2. Find its output using input × weight + bias.\n\n\n\n### Given\n\nA toy model has input 4, weight 5 and bias 2.\n\n### To find\n\nFind the preactivation and the requested neuron output.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n### Solution — step by step\n\n\n\n\n**Step 1:** multiply input by weight: 4 × 5 = 20.\n\n*"},{"title":"Negative numbers are direction","blocks":["Zero is the reference point. Positive numbers such as 1, 2 and 3 lie to its right on a number line. Negative numbers such as −1, −2 and −3 lie to its left. The number −3 is three units below zero; it is not an instruction to subtract until another number is supplied.
","In 5 − 2, the minus sign is an operation: subtract 2 from 5. In 5 + (−2), the sign inside the brackets belongs to the number negative 2. Both expressions finish at 3. Brackets keep the signed number together.
","Start at 5. Adding positive 2 moves two places right, to 7. Adding negative 2 moves two places left, to 3.
","If both numbers are negative, keep moving left. Start at −3 and add −2: the answer is −5.
","Subtracting positive 2 moves left. Subtracting negative 2 reverses a leftward movement, so it moves right. A money example is removing a debt of 2: your balance increases by 2.
","The shortcut “two negatives make a positive” is not a rule for every calculation. Negative 3 plus negative 2 is still −5. First identify the operation: addition, subtraction, multiplication or division.
","For multiplication and division, equal signs give a positive result and different signs give a negative result. Calculate the ordinary size, then attach the correct sign.
","| Signs being multiplied | Complete calculation | Result sign |
|---|---|---|
| Positive × positive | 3 × 2 = 6 | Positive |
| Negative × positive | (−3) × 2 = −6 | Negative |
| Positive × negative | 3 × (−2) = −6 | Negative |
| Negative × negative | (−3) × (−2) = 6 | Positive |
Division uses the same sign pattern: (−6) ÷ 2 = −3 and (−6) ÷ (−2) = 3. The divisor must be nonzero.
","Question. Calculate −3 + 2, −3 − 2 and −3 − (−2).
","Answer — step by step.
","Expressions: −3+2; −3−2; −3−(−2).
","Evaluate the three signed-number expressions.
","Order of operations. Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.
","A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.
","Yaad rakhne ka cue: Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.
","Step 1: evaluate the first expression. start at −3 and move two places right. The answer is −1.
","Step 2: evaluate the second expression. start at −3 and move two places left. The answer is −5.
","Step 3: evaluate the third expression. replace subtract-negative-2 with add-positive-2. The answer is −1.
","Later, a weight update subtracts a correction that can itself be negative. The third expression explains exactly why that update can increase a weight.
"],"question":[],"answer":[],"kind":"read","id":"tee-c2-s1","search":"Negative numbers are direction ### 1. The minus sign can describe a number\nZero is the reference point. Positive numbers such as 1, 2 and 3 lie to its right on a number line. Negative numbers such as −1, −2 and −3 lie to its left. The number −3 is three units below zero; it is not an instruction to subtract until another number is supplied.\n\nIn 5 − 2, the minus sign is an operation: subtract 2 from 5. In 5 + (−2), the sign inside the brackets belongs to the number negative 2. Both expressions finish at 3. Brackets keep the signed number together.\n### 2. Adding a negative moves left\nStart at 5. Adding positive 2 moves two places right, to 7. Adding negative 2 moves two places left, to 3.\n\nIf both numbers are negative, keep moving left. Start at −3 and add −2: the answer is −5.\n\n### 3. Subtracting reverses that movement\nSubtracting positive 2 moves left. Subtracting negative 2 reverses a leftward movement, so it moves right. A money example is removing a debt of 2: your balance increases by 2.\n\nThe shortcut “two negatives make a positive” is not a rule for every calculation. Negative 3 plus negative 2 is still −5. First identify the operation: addition, subtraction, multiplication or division.\n### 4. Multiplication has its own sign rule\nFor multiplication and division, equal signs give a positive result and different signs give a negative result. Calculate the ordinary size, then attach the correct sign.\n| Signs being multiplied | Complete calculation | Result sign |\n| Positive × positive | 3 × 2 = 6 | Positive |\n| Negative × positive | (−3) × 2 = −6 | Negative |\n| Positive × negative | 3 × (−2) = −6 | Negative |\n| Negative × negative | (−3) × (−2) = 6 | Positive |\nDivision uses the same sign pattern: (−6) ÷ 2 = −3 and (−6) ÷ (−2) = 3. The divisor must be nonzero.\n### 5. Fully worked arithmetic\n**Question.** Calculate −3 + 2, −3 − 2 and −3 − (−2).\n**Answer — step by step.**\n\n\n\n### Given\n\nExpressions: −3+2; −3−2; −3−(−2).\n\n### To find\n\nEvaluate the three signed-number expressions.\n\n### Formula — which one and why\n\n**Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.\n\nA minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.\n\n\n\n**Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.\n\n### Solution — step by step\n\n\n\n\n**Step 1: evaluate the first expression.** start at −3 and move two places right. The answer is −1.\n\n**Step 2: evaluate the second expression.** start at −3 and move two places left. The answer is −5.\n\n**Step 3: evaluate the third expression.** replace subtract-negative-2 with add-positive-2. The answer is −1.\n\nLater, a weight update subtracts a correction that can itself be negative. The third expression explains exactly why that update can increase a weight.\n"},{"title":"Fractions decimals and percentages","blocks":["The upper number is the numerator. The lower number is the denominator. In one-quarter, split one whole into four equal parts and take one part. The fraction bar means division.
","A decimal and a fraction can name the same amount. One-half is 0.5, one-tenth is 0.1, and one-hundredth is 0.01. The denominator cannot be zero: division by zero is undefined.
","Multiplying by 0.1 means taking one-tenth. Multiplying by 0.2 means taking two-tenths. For 0.2 × 0.3, express both numbers as fractions. Multiply the numerators and multiply the denominators.
","This also explains a calculation used later: 0.1 × (−0.6) is one-tenth of negative 0.6, which is −0.06. The negative sign is kept because the factors have different signs.
","Two halves fit inside 1. Four halves fit inside 2. Therefore dividing 2 by one-half gives 4, even though division by a number greater than 1 would make 2 smaller.
","The reciprocal flips a nonzero fraction: the reciprocal of 1/2 is 2/1. Dividing by a fraction means multiplying by its reciprocal. For decimals, multiplying both the numerator and denominator by the same nonzero number preserves their ratio.
","To turn a percentage into a decimal, divide by 100. To turn a decimal into a percentage, multiply by 100.
","If 20% is removed, 80% remains. As decimals: 1 − 0.2 = 0.8. The number 1 represents the whole amount, or 100%. This same arithmetic will be used for dropout later.
","For 2, 4 and 9, first add all three numbers: 2 + 4 + 9 = 15. Divide by the number of values, which is 3. The mean is 5. Do not divide by 9 just because it is the largest value.
","Here the sum is 15 and the count is 3. Apply those two quantities to the formula.
","A weighted mean gives different shares of importance to the values. With shares 0.9 and 0.1, the first value receives 90% of the importance and the second receives 10%. The shares add to 1.
","Question. Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively.
","Answer — step by step.
","Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively.
","Find the weighted mean.
","Weighted sum. Multiply each value by its own coefficient and then add. For a weighted mean, divide by the sum of the weights unless that sum is already 1.
","xᵢ = value i; aᵢ = its weight. Each input keeps its own matching coefficient.
","Yaad rakhne ka cue: Har number apne weight ke saath multiply; phir sab add.
","Step 1: multiply the first value by its weight: 0.9 × 10 = 9.
","Step 2: multiply the second value by its weight: 0.1 × 20 = 2.
","Step 3: add the contributions: 9 + 2 = 11. The weighted mean is 11. It is closer to 10 because 10 received most of the importance.
","If importance weights do not sum to 1, divide their weighted sum by the sum of the weights to obtain the weighted mean. A neuron's general weighted sum does not require weights to sum to 1 and is not automatically a mean.
"],"question":[],"answer":[],"kind":"read","id":"tee-c2-s2","search":"Fractions decimals and percentages ### 1. A fraction tells us to divide\nThe upper number is the **numerator**. The lower number is the **denominator**. In one-quarter, split one whole into four equal parts and take one part. The fraction bar means division.\n\n\nA decimal and a fraction can name the same amount. One-half is 0.5, one-tenth is 0.1, and one-hundredth is 0.01. The denominator cannot be zero: division by zero is undefined.\n\n### 2. Why small decimals become smaller when multiplied\nMultiplying by 0.1 means taking one-tenth. Multiplying by 0.2 means taking two-tenths. For 0.2 × 0.3, express both numbers as fractions. Multiply the numerators and multiply the denominators.\n\nThis also explains a calculation used later: 0.1 × (−0.6) is one-tenth of negative 0.6, which is −0.06. The negative sign is kept because the factors have different signs.\n### 3. Dividing by one-half asks how many halves fit\nTwo halves fit inside 1. Four halves fit inside 2. Therefore dividing 2 by one-half gives 4, even though division by a number greater than 1 would make 2 smaller.\n\nThe **reciprocal** flips a nonzero fraction: the reciprocal of 1/2 is 2/1. Dividing by a fraction means multiplying by its reciprocal. For decimals, multiplying both the numerator and denominator by the same nonzero number preserves their ratio.\n\n### 4. Percent means out of one hundred\nTo turn a percentage into a decimal, divide by 100. To turn a decimal into a percentage, multiply by 100.\n\nIf 20% is removed, 80% remains. As decimals: 1 − 0.2 = 0.8. The number 1 represents the whole amount, or 100%. This same arithmetic will be used for dropout later.\n### 5. A mean is a fair-share amount\nFor 2, 4 and 9, first add all three numbers: 2 + 4 + 9 = 15. Divide by the number of values, which is 3. The mean is 5. Do not divide by 9 just because it is the largest value.\n\nHere the sum is 15 and the count is 3. Apply those two quantities to the formula.\n\nA **weighted mean** gives different shares of importance to the values. With shares 0.9 and 0.1, the first value receives 90% of the importance and the second receives 10%. The shares add to 1.\n**Question.** Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively.\n**Answer — step by step.**\n\n\n\n### Given\n\nFind the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively.\n\n### To find\n\nFind the weighted mean.\n\n### Formula — which one and why\n\n**Weighted sum.** Multiply each value by its own coefficient and then add. For a weighted mean, divide by the sum of the weights unless that sum is already 1.\n\nxᵢ = value i; aᵢ = its weight. Each input keeps its own matching coefficient.\n\n\n\n**Yaad rakhne ka cue:** Har number apne weight ke saath multiply; phir sab add.\n\n### Solution — step by step\n\n\n\n\n**Step 1:** multiply the first value by its weight: 0.9 × 10 = 9.\n**Step 2:** multiply the second value by its weight: 0.1 × 20 = 2.\n**Step 3:** add the contributions: 9 + 2 = 11. The weighted mean is **11**. It is closer to 10 because 10 received most of the importance.\n\nIf importance weights do not sum to 1, divide their weighted sum by the sum of the weights to obtain the weighted mean. A neuron's general weighted sum does not require weights to sum to 1 and is not automatically a mean.\n"},{"title":"Brackets before everything else","blocks":["An expression is a written calculation. Brackets group part of that calculation. In 1 ÷ (1 + 3), the whole quantity 1 + 3 is the divisor, so calculate it before dividing.
","Removing the brackets changes the calculation. In 1 ÷ 1 + 3, division comes before addition: 1 ÷ 1 = 1, then 1 + 3 = 4.
","First calculate inside brackets. Then evaluate powers and roots. Then do multiplication and division, reading left to right. Finally do addition and subtraction, again reading left to right. Multiplication does not have priority over division; they share a level.
","For 8 ÷ 2 × 3, calculate 8 ÷ 2 = 4 and then 4 × 3 = 12. Do not change it into 8 ÷ (2 × 3).
","Question. Evaluate (2 + 3)² ÷ 5.
","Answer — step by step.
","Evaluate (2 + 3)² ÷ 5.
","Evaluate each expression and state its value.
","Order of operations. Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.
","A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.
","Yaad rakhne ka cue: Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.
","Step 1: finish the brackets: 2 + 3 = 5. The expression becomes 5² ÷ 5.
","Step 2: square the bracket result: 5² = 25. The expression becomes 25 ÷ 5.
","Step 3: divide: 25 ÷ 5 = 5.
","Question. Evaluate 2 + 3² × 4.
","Answer — step by step.
","Evaluate 2 + 3² × 4.
","Evaluate each expression and state its value.
","Order of operations. Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.
","A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.
","Yaad rakhne ka cue: Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.
","Step 1: calculate the power: 3² = 9. Keep the other numbers where they are: 2 + 9 × 4.
","Step 2: multiply: 9 × 4 = 36. The expression becomes 2 + 36.
","Step 3: add: 2 + 36 = 38.
","In a printed fraction, everything below the bar belongs to the denominator. When entering that fraction in a single calculator line, put brackets around the complete denominator. A printed fraction with 1 above and 1 + 3 below must be entered as 1 ÷ (1 + 3).
"],"question":[],"answer":[],"kind":"read","id":"tee-c2-s3","search":"Brackets before everything else ### 1. Brackets tell us what must stay together\nAn expression is a written calculation. Brackets group part of that calculation. In 1 ÷ (1 + 3), the whole quantity 1 + 3 is the divisor, so calculate it before dividing.\n\n\nRemoving the brackets changes the calculation. In 1 ÷ 1 + 3, division comes before addition: 1 ÷ 1 = 1, then 1 + 3 = 4.\n\n### 2. Use this order for the expressions in this course\nFirst calculate inside brackets. Then evaluate powers and roots. Then do multiplication and division, reading left to right. Finally do addition and subtraction, again reading left to right. Multiplication does not have priority over division; they share a level.\nFor 8 ÷ 2 × 3, calculate 8 ÷ 2 = 4 and then 4 × 3 = 12. Do not change it into 8 ÷ (2 × 3).\n### 3. A full example with brackets and a power\n**Question.** Evaluate (2 + 3)² ÷ 5.\n**Answer — step by step.**\n\n\n\n### Given\n\nEvaluate (2 + 3)² ÷ 5.\n\n### To find\n\nEvaluate each expression and state its value.\n\n### Formula — which one and why\n\n**Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.\n\nA minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.\n\n\n\n**Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.\n\n### Solution — step by step\n\n\n\n\n**Step 1:** finish the brackets: 2 + 3 = 5. The expression becomes 5² ÷ 5.\n**Step 2:** square the bracket result: 5² = 25. The expression becomes 25 ÷ 5.\n**Step 3:** divide: 25 ÷ 5 = **5**.\n\n### 4. A full example without grouping the addition\n**Question.** Evaluate 2 + 3² × 4.\n**Answer — step by step.**\n\n\n\n### Given\n\nEvaluate 2 + 3² × 4.\n\n### To find\n\nEvaluate each expression and state its value.\n\n### Formula — which one and why\n\n**Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.\n\nA minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.\n\n\n\n**Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.\n\n### Solution — step by step\n\n\n\n\n**Step 1:** calculate the power: 3² = 9. Keep the other numbers where they are: 2 + 9 × 4.\n**Step 2:** multiply: 9 × 4 = 36. The expression becomes 2 + 36.\n**Step 3:** add: 2 + 36 = **38**.\n\n### 5. A fraction bar groups the whole denominator\nIn a printed fraction, everything below the bar belongs to the denominator. When entering that fraction in a single calculator line, put brackets around the complete denominator. A printed fraction with 1 above and 1 + 3 below must be entered as 1 ÷ (1 + 3).\n"},{"title":"Powers roots and very small numbers","blocks":["In 2³, 2 is the base and 3 is the exponent. For a positive whole-number exponent, multiply that many copies of the base. It does not mean 2 × 3.
","The exponent 2 is called a square: 3² = 3 × 3 = 9. Three rows of three cells make nine cells.
","Squaring negative 2 means multiplying negative 2 by negative 2. The equal signs of the factors give a positive result.
","Without brackets, −2² means first calculate 2², then apply the minus sign: −(4) = −4. Enter brackets when the entire negative number is the base.
","For decimals, use the same rule. Negative 0.3 squared is positive 0.09 because 3/10 multiplied by 3/10 is 9/100.
","Because 3 × 3 = 9, the principal square root of 9 is 3. Although (−3)² also equals 9, the symbol √9 by itself means the nonnegative root, 3.
","The absolute value of a number is its distance from zero: |−4| = 4 and |4| = 4. Squaring and taking a square root returns this nonnegative size.
","Start with powers of 10. Each time the exponent decreases by 1, divide the previous value by 10: 1000 → 100 → 10 → 1 → 0.1 → 0.01. This explains both the zero power and the negative powers.
","| Power | Calculation | Value |
|---|---|---|
| 10³ | 10 × 10 × 10 | 1000 |
| 10² | 1000 ÷ 10 | 100 |
| 10¹ | 100 ÷ 10 | 10 |
| 10⁰ | 10 ÷ 10 | 1 |
| 10⁻¹ | 1 ÷ 10 | 0.1 |
| 10⁻² | 0.1 ÷ 10 | 0.01 |
| 10⁻³ | 0.01 ÷ 10 | 0.001 |
For a nonzero base, power zero is 1. A negative exponent means the reciprocal of the corresponding positive power. It does not mean a negative answer.
","The notation 1 × 10⁻⁸ equals 0.00000001. Calculators may display it as 1E−8. In optimizer formulas, ε, pronounced “epsilon”, often names a small positive constant added to a denominator. Read its supplied value before calculating.
","Question. Calculate 1 − 0.9².
","Expression: 1−0.9².
","Evaluate the power first, then the subtraction.
","A square before subtraction. The exponent applies to 0.9. Squaring is repeated multiplication; perform it before subtracting the result from 1.
","q denotes the number being squared. Parentheses would be required to square the whole difference.
","Yaad rakhne ka cue: Jis number pe power hai, sirf uska square; subtraction baad mein.
","Step 1: square only the 0.9.
","Step 2: subtract the result from 1.
","Final answer: 0.19. The square acts on 0.9, not on the entire difference.
","These notes assume familiarity with algebra and basic calculus. The revision here covers the notation used in neural-network equations and the conventions that affect numerical answers.
","| Notation | Technical meaning |
|---|---|
| x, y, ŷ | Input feature vector, target, model prediction |
| W, b, θ | Weight matrix, bias vector, complete trainable parameter vector |
| z, a | Preactivation (affine input) and activation (layer output) |
| L, J | Per-example loss and aggregate training objective |
| ∂L/∂w | Partial derivative with respect to one parameter |
| ∇θJ | Gradient: vector of partial derivatives with respect to θ |
| η, λ, ε | Learning rate, regularization coefficient, numerical stabilizer |
| Wᵀ, u⊙v | Transpose and elementwise multiplication |
| θₜ, W⁽ˡ⁾ | Update index t and layer index l; neither denotes a power |
Vectors are columns unless a question specifies otherwise. A weight matrix has one row per receiving neuron and one column per incoming feature. Write matrix dimensions before multiplying. A bias is added once per receiving neuron or output feature map.
","Keep at least six significant digits during intermediate calculations and round the final result as requested. The loss reduction matters: a sum, mean and half-sum produce different gradient factors. The location of ε also matters: a denominator √s + ε differs from √(s + ε).
","The sign of a gradient belongs to the derivative, not to the loss value. For example, w = 0.4, η = 0.1 and gradient −2 give the update below.
","In later calculations, state the optimizer convention, initial state and iteration number before substituting values. This prevents mixing momentum velocity with Adam's second-moment accumulator.
"],"question":[],"answer":[],"kind":"read","id":"tee-c2-s5","search":"Variables functions and symbols These notes assume familiarity with algebra and basic calculus. The revision here covers the notation used in neural-network equations and the conventions that affect numerical answers.\n\n| Notation | Technical meaning |\n| x, y, ŷ | Input feature vector, target, model prediction |\n| W, b, θ | Weight matrix, bias vector, complete trainable parameter vector |\n| z, a | Preactivation (affine input) and activation (layer output) |\n| L, J | Per-example loss and aggregate training objective |\n| ∂L/∂w | Partial derivative with respect to one parameter |\n| ∇θJ | Gradient: vector of partial derivatives with respect to θ |\n| η, λ, ε | Learning rate, regularization coefficient, numerical stabilizer |\n| Wᵀ, u⊙v | Transpose and elementwise multiplication |\n| θₜ, W⁽ˡ⁾ | Update index t and layer index l; neither denotes a power |\n\n### Conventions used in numerical solutions\nVectors are columns unless a question specifies otherwise. A weight matrix has one row per receiving neuron and one column per incoming feature. Write matrix dimensions before multiplying. A bias is added once per receiving neuron or output feature map.\n\nKeep at least six significant digits during intermediate calculations and round the final result as requested. The loss reduction matters: a sum, mean and half-sum produce different gradient factors. The location of ε also matters: a denominator √s + ε differs from √(s + ε).\n\nThe sign of a gradient belongs to the derivative, not to the loss value. For example, w = 0.4, η = 0.1 and gradient −2 give the update below.\n\nIn later calculations, state the optimizer convention, initial state and iteration number before substituting values. This prevents mixing momentum velocity with Adam's second-moment accumulator.\n"},{"title":"Exponentials and natural logarithms","blocks":["Definitions. The exponential function exp(x) = eˣ uses Euler's constant e. The natural logarithm ln is its inverse on positive real values. These functions occur in sigmoid, tanh, softmax and cross-entropy.
","For positive constants and valid domains, exponent and log rules simplify expressions before a calculator is needed. For example, e^(−ln 2) = 1/2. Use natural ln for the loss formulas, not the calculator's base-10 log.
","The minus sign in the first expression comes from differentiating the inner function −z. Applying the chain rule correctly here gives the sigmoid derivative derived in the activation section.
","Integration reverses differentiation up to an additive constant. The basic relevant identities are:
","For angles in radians, d(sin x)/dx = cos x and d(cos x)/dx = −sin x. The identity sin²x + cos²x = 1 concerns ordinary trigonometric functions. Hyperbolic tanh is a different function; using tan instead of tanh gives an incorrect recurrent-network answer.
","The supplied MT2 calculation families mainly require derivatives, matrix products, logarithms and elementary statistics. Extended integration or trigonometric exercises are not needed for those families.
","At z = 2, sigmoid = 1/(1 + e⁻²) ≈ 0.880797. At z = 1, tanh ≈ 0.761594. Keep unrounded values when a later gradient uses these activations.
"],"question":[],"answer":[],"kind":"read","id":"tee-c2-s6","search":"Exponentials and natural logarithms **Definitions.** The exponential function exp(x) = eˣ uses Euler's constant e. The natural logarithm ln is its inverse on positive real values. These functions occur in sigmoid, tanh, softmax and cross-entropy.\n\n\nFor positive constants and valid domains, exponent and log rules simplify expressions before a calculator is needed. For example, e^(−ln 2) = 1/2. Use natural ln for the loss formulas, not the calculator's base-10 log.\n\n### Derivatives used in neural networks\n\nThe minus sign in the first expression comes from differentiating the inner function −z. Applying the chain rule correctly here gives the sigmoid derivative derived in the activation section.\n\n### Brief integration and trigonometry reminder\nIntegration reverses differentiation up to an additive constant. The basic relevant identities are:\n\n\nFor angles in radians, d(sin x)/dx = cos x and d(cos x)/dx = −sin x. The identity sin²x + cos²x = 1 concerns ordinary trigonometric functions. Hyperbolic tanh is a different function; using tan instead of tanh gives an incorrect recurrent-network answer.\n\nThe supplied MT2 calculation families mainly require derivatives, matrix products, logarithms and elementary statistics. Extended integration or trigonometric exercises are not needed for those families.\n\n### Calculator values for activation calculations\nAt z = 2, sigmoid = 1/(1 + e⁻²) ≈ 0.880797. At z = 1, tanh ≈ 0.761594. Keep unrounded values when a later gradient uses these activations.\n"},{"title":"Calculator routine","blocks":["You reported a Casio ES-series calculator. The exact model string is not confirmed. The functions below are mathematical entries; use the keys labelled on your model. On fx-991 ES/fx-991 ES PLUS models, eˣ is normally the SHIFT function of ln. Do not confuse eˣ with the × 10ˣ scientific-notation key. Use the function labels on your own calculator for these entries.
","1. Use ordinary computation mode. Enter complete brackets around a denominator or negative exponent.
","2. For sigmoid at z = 2, enter 1 ÷ (1 + e^(−2)). Result: 0.880797078. Keep the full calculator value; write about six decimals during intermediate work.
","3. For tanh at z = 1, use the hyperbolic tanh function if available. A universal alternative is (e² − 1) ÷ (e² + 1), giving 0.761594156. Ordinary tan is a different function; do not use it.
","4. For cross-entropy with true probability 0.8, enter −ln(0.8), giving 0.223143551.
","5. For softmax [2.3, 4.5, 3.2], subtract 4.5 first, then calculate e⁻²·², 1, e⁻¹·³. Add them, then divide each exponential by the same total.
","6. If a result displays as a fraction, use your model’s fraction/decimal conversion key, often S⇔D. Do not reset the calculator unnecessarily.
","Five-minute warm-up: compute sigmoid(0) = 0.5; sigmoid(−2) ≈ 0.119203; tanh(0) = 0; −ln(0.5) ≈ 0.693147; √0.25 = 0.5. A different result signals an entry mistake, not a reason to memorize a wrong value.
"],"question":[],"answer":[],"kind":"read","id":"tee-c2-s7","search":"Calculator routine You reported a Casio ES-series calculator. The exact model string is not confirmed. The functions below are mathematical entries; use the keys labelled on your model. On fx-991 ES/fx-991 ES PLUS models, eˣ is normally the SHIFT function of ln. Do not confuse eˣ with the × 10ˣ scientific-notation key. Use the function labels on your own calculator for these entries.\n\n\n\n1. Use ordinary computation mode. Enter complete brackets around a denominator or negative exponent.\n2. For sigmoid at z = 2, enter 1 ÷ (1 + e^(−2)). Result: 0.880797078. Keep the full calculator value; write about six decimals during intermediate work.\n3. For tanh at z = 1, use the hyperbolic tanh function if available. A universal alternative is (e² − 1) ÷ (e² + 1), giving 0.761594156. Ordinary tan is a different function; do not use it.\n4. For cross-entropy with true probability 0.8, enter −ln(0.8), giving 0.223143551.\n5. For softmax [2.3, 4.5, 3.2], subtract 4.5 first, then calculate e⁻²·², 1, e⁻¹·³. Add them, then divide each exponential by the same total.\n6. If a result displays as a fraction, use your model’s fraction/decimal conversion key, often S⇔D. Do not reset the calculator unnecessarily.\n**Five-minute warm-up: ** compute sigmoid(0) = 0.5; sigmoid(−2) ≈ 0.119203; tanh(0) = 0; −ln(0.5) ≈ 0.693147; √0.25 = 0.5. A different result signals an entry mistake, not a reason to memorize a wrong value."},{"title":"A complete arithmetic calculation","blocks":["Evaluate 0.4 − 0.1 × (−2). The multiplication has priority over the subtraction.
","Step 1: multiply 0.1 by −2, giving −0.2.
","Step 2: substitute that entire result, including its sign: 0.4 − (−0.2).
","Step 3: subtracting a negative means adding its positive value: 0.4 + 0.2 = 0.6.
","The answer is 0.6. This calculation will later appear inside a weight update; the arithmetic is exactly the same.
"],"question":[],"answer":[],"kind":"read","id":"tee-c2-s8","search":"A complete arithmetic calculation Evaluate 0.4 − 0.1 × (−2). The multiplication has priority over the subtraction.\n\n\n\n**Step 1:** multiply 0.1 by −2, giving −0.2.\n**Step 2:** substitute that entire result, including its sign: 0.4 − (−0.2).\n**Step 3:** subtracting a negative means adding its positive value: 0.4 + 0.2 = 0.6.\nThe answer is **0.6**. This calculation will later appear inside a weight update; the arithmetic is exactly the same.\n"},{"title":"Loss: how one wrong prediction becomes a number","blocks":["Definition. A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative.
","For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective.
","When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual:
","The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight.
","Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.
","Predictions = [0.2,0.7,0.4].
","Targets = [0,1,1]; n = 3 scalar predictions.
","Find SSE, half SSE and MSE, then all three prediction-gradient lists.
","Squared-error objectives. Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.
","ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.
","Yaad rakhne ka cue: Pehle error square; phir question ke hisaab se sum, half ya mean.
","Step 1: calculate the three signed prediction errors.
","Step 2: square every error.
","Step 3: add the three squared errors.
","Step 4: apply the requested loss divisors.
","Step 5: multiply every error by 2 for the SSE gradient.
","Step 6: use each error itself for the half-SSE gradient.
","Step 7: multiply every error by 2/3 for the MSE gradient.
","The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate.
"],"question":[],"answer":[],"kind":"read","id":"tee-c2-s9","search":"Loss: how one wrong prediction becomes a number \n**Definition.** A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative.\n\nFor this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective.\n\n### Derivatives and their scaling\nWhen differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual:\n\nThe factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight.\n\n**Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.**\n\n\n\n### Given\n\nPredictions = [0.2,0.7,0.4].\n\nTargets = [0,1,1]; n = 3 scalar predictions.\n\n### To find\n\nFind SSE, half SSE and MSE, then all three prediction-gradient lists.\n\n### Formula — which one and why\n\n**Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.\n\nŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: calculate the three signed prediction errors.**\n\n\n**Step 2: square every error.**\n\n\n**Step 3: add the three squared errors.**\n\n\n**Step 4: apply the requested loss divisors.**\n\n\n**Step 5: multiply every error by 2 for the SSE gradient.**\n\n\n**Step 6: use each error itself for the half-SSE gradient.**\n\n\n**Step 7: multiply every error by 2/3 for the MSE gradient.**\n\n\nThe residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate.\n\n"}]},{"n":3,"title":"The maths behind a network","sections":[{"title":"A vector is a list and a matrix is a table","blocks":["The input [2, 3] is a vector containing two features. A matrix is a rectangle of numbers. “2 by 3” means two rows and three columns. A column vector with two entries has shape 2 by 1. Always write rows before columns.
","In this course, unless a question’s diagram explicitly uses another convention, each row of a weight matrix belongs to one receiving neuron. Thus W has shape number of output neurons by number of input features, and the input is a column vector.
","Do it slowly: row 1 says 0.11 × 2 = 0.22 and 0.21 × 3 = 0.63, so its answer is 0.85. Row 2 says 0.12 × 2 = 0.24 and 0.08 × 3 = 0.24, so its answer is 0.48. Matrix multiplication is several weighted sums written together.
","Shape check: (2 by 2) × (2 by 1) gives (2 by 1). The inside numbers must match; the outside numbers are the answer shape. This check catches many transposed-weight errors before arithmetic begins.
","Transpose swaps rows and columns. Elementwise multiplication does not do row-by-column sums: [2, 3] ⊙ [4, 5] = [8, 15]. Dot product does add: [2, 3] · [4, 5] = 8 + 15 = 23. LSTM gates use elementwise multiplication, not a dot product.
","Practice. Multiply W = by x = [2, 1] as a column. Answer: first row 1 × 2 + 2 × 1 = 4; second row 3 × 2 + 4 × 1 = 10. With b = [1, −1], Wx + b = [5, 9].
A row runs horizontally, from left to right. A column runs vertically, from top to bottom. A matrix is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows.
","A vector is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors.
","The superscript T means transpose, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m.
","The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1.
","Yaad rakhne ka cue: transpose = row ko column bana; number aur sign ko haath mat laga.
","Question. Find the transpose of the following matrix A and state the size of both matrices.
","Aᵀ and both matrix sizes.
","A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.
","A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.
","Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.
","Step 1: write the original size.
","A has 1 row(s) and 3 column(s), so its size is 1 × 3.
","Step 2: move original row 1 into new column 1.
","Copy [2, 1, 3] down column 1, in the same order.
","Step 3: write the transposed matrix.
","Step 4: state its new size.
","The result has 3 row(s) and 1 column(s): 3 × 1.
","Final answer: Aᵀ has size 3 × 1. Transposing it again gives the original A.
","| 2 | 1 | 3 |
Read each colored row left to right.
| 2 |
| 1 |
| 3 |
Each original row is now the matching colored column.
The entry at row i, column j moves to row j, column i. Its value and sign do not change.
Question. Find the transpose of the following matrix A and state the size of both matrices.
","Aᵀ and both matrix sizes.
","A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.
","A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.
","Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.
","Step 1: write the original size.
","A has 2 row(s) and 3 column(s), so its size is 2 × 3.
","Step 2: move original row 1 into new column 1.
","Copy [1, 2, 3] down column 1, in the same order.
","Step 3: move original row 2 into new column 2.
","Copy [4, 5, 6] down column 2, in the same order.
","Step 4: write the transposed matrix.
","Step 5: state its new size.
","The result has 3 row(s) and 2 column(s): 3 × 2.
","Final answer: Aᵀ has size 3 × 2. Transposing it again gives the original A.
","| 1 | 2 | 3 |
| 4 | 5 | 6 |
Read each colored row left to right.
| 1 | 4 |
| 2 | 5 |
| 3 | 6 |
Each original row is now the matching colored column.
The entry at row i, column j moves to row j, column i. Its value and sign do not change.
Question. Find the transpose of the following matrix A and state the size of both matrices.
","Aᵀ and both matrix sizes.
","A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.
","A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.
","Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.
","Step 1: write the original size.
","A has 2 row(s) and 2 column(s), so its size is 2 × 2.
","Step 2: move original row 1 into new column 1.
","Copy [1, -2] down column 1, in the same order.
","Step 3: move original row 2 into new column 2.
","Copy [0, 3] down column 2, in the same order.
","Step 4: write the transposed matrix.
","Step 5: state its new size.
","The result has 2 row(s) and 2 column(s): 2 × 2.
","Final answer: Aᵀ has size 2 × 2. Transposing it again gives the original A.
","| 1 | −2 |
| 0 | 3 |
Read each colored row left to right.
| 1 | 0 |
| −2 | 3 |
Each original row is now the matching colored column.
The entry at row i, column j moves to row j, column i. Its value and sign do not change.
Question. Find the transpose of the following matrix A and state the size of both matrices.
","Aᵀ and both matrix sizes.
","A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication.
","A is the original matrix; Aᵀ is its transpose; i and j are row/column positions.
","Yaad rakhne ka cue: Row seedha column mein; double transpose se original wapas.
","Step 1: write the original size.
","A has 3 row(s) and 1 column(s), so its size is 3 × 1.
","Step 2: move original row 1 into new column 1.
","Copy [-1] down column 1, in the same order.
","Step 3: move original row 2 into new column 2.
","Copy [4] down column 2, in the same order.
","Step 4: move original row 3 into new column 3.
","Copy [2] down column 3, in the same order.
","Step 5: write the transposed matrix.
","Step 6: state its new size.
","The result has 1 row(s) and 3 column(s): 1 × 3.
","Final answer: Aᵀ has size 1 × 3. Transposing it again gives the original A.
","| −1 |
| 4 |
| 2 |
Read each colored row left to right.
| −1 | 4 | 2 |
Each original row is now the matching colored column.
The entry at row i, column j moves to row j, column i. Its value and sign do not change.
A dot product turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed.
","This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products.
","Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.
","Weights w = [2, 3].
","Matching inputs x = [4, 1].
","Both lists contain 2 numbers.
","One scalar value wᵀx.
","A dot product is a sum of matching products. There is no bias in this question.
","Σ means add; the index i visits each matching pair.
","Yaad rakhne ka cue: Same position multiply; sab products add.
","Step 1: multiply matching pair 1.
","Step 2: multiply matching pair 2.
","Step 3: add the products.
","Final answer: wᵀx = 11.
","Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.
","Weights w = [1, -2].
","Matching inputs x = [2, 3].
","Both lists contain 2 numbers.
","One scalar value wᵀx.
","A dot product is a sum of matching products. There is no bias in this question.
","Σ means add; the index i visits each matching pair.
","Yaad rakhne ka cue: Same position multiply; sab products add.
","Step 1: multiply matching pair 1.
","Step 2: multiply matching pair 2.
","Step 3: add the products.
","Final answer: wᵀx = -4.
","Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.
","Weights w = [0.5, 1, -1].
","Matching inputs x = [4, 2, 3].
","Both lists contain 3 numbers.
","One scalar value wᵀx.
","A dot product is a sum of matching products. There is no bias in this question.
","Σ means add; the index i visits each matching pair.
","Yaad rakhne ka cue: Same position multiply; sab products add.
","Step 1: multiply matching pair 1.
","Step 2: multiply matching pair 2.
","Step 3: multiply matching pair 3.
","Step 4: add the products.
","Final answer: wᵀx = 1.
","To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B.
","For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once.
","Yaad rakhne ka cue: multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column.
","Definition. A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation.
","For d input features and m receiving neurons, the dimensions are:
","The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product.
","Question. Given W = , x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b.
Weights W = .
Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ.
","Find both components of z = Wx+b.
","Row–column multiplication. Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.
","Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.
","Yaad rakhne ka cue: Ek row × poora input column = ek receiving neuron ka answer.
","Step 1: verify compatible dimensions.
","W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column.
","Step 2: calculate the first receiving row products.
","Step 3: sum row 1 and add its bias.
","Step 4: calculate the second receiving row products.
","Step 5: sum row 2 and add its bias.
","| 1 | −2 |
| 0.5 | 3 |
| 2 |
| 1 |
| 2 | −2 |
First × first; second × second.
One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component.
| 1 | −2 |
| 0.5 | 3 |
| 2 |
| 1 |
| 1 | 3 |
First × first; second × second.
One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component.
An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W.
","Question. Multiply A = by B = . Show all four output entries.
The 2 × 2 matrix C = AB.
","Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication.
","Cᵢⱼ is the entry in output row i, column j.
","Yaad rakhne ka cue: Output ki position dekho: wahi row, wahi column.
","Step 1: check the dimensions.
","(2 × 2)(2 × 2) is valid; output size = 2 × 2.
","Step 2: row 1 with column 1.
","Step 3: row 1 with column 2.
","Step 4: row 2 with column 1.
","Step 5: row 2 with column 2.
","Step 6: assemble the output by position.
","Final answer: C = , with size 2 × 2.
Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.
","a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5.
","The 2 × 1 entry-by-entry product and the 2 × 2 outer product.
","⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons.
","⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row.
","Yaad rakhne ka cue: ⊙ mein same jagah; outer product mein har pair.
","Step 1: multiply matching entries.
","Step 2: transpose b.
","Step 3: form row 1 of the outer product.
","Step 4: form row 2 of the outer product.
","Step 5: write both rows.
","Final answer: Entry-by-entry result = [8,15]ᵀ. Outer product = .
A graph shows how one value changes with another. Put the weight w on the horizontal axis and the loss L on the vertical axis. A slope is change in height divided by change in horizontal position. If L rises by 6 when w rises by 2, the average slope is 3.
","A derivative is the slope for an extremely small change at one location. A positive derivative says increasing the weight initially increases loss. A negative derivative says increasing the weight initially decreases loss. Gradient descent moves against that local slope. Gradient batata hai loss kis taraf badhega; hum ulte taraf chhota step lete hain.
","At w = 3, slope = 6. With learning rate 0.1, the update is 3 − 0.1 × 6 = 2.4. Loss falls from 9 to 5.76. At w = −3, slope = −6, so the update is −3 − 0.1 × (−6) = −2.4. Both moves approach zero.
","Definition. A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector.
","The rules needed for the supplied neural-network numericals are:
","| Function or operation | Derivative |
|---|---|
| Constant c | 0 |
| xⁿ | nxⁿ⁻¹ |
| c f(x) | c f′(x) |
| f(x) + g(x) | f′(x) + g′(x) |
| f(x)g(x) | f′(x)g(x) + f(x)g′(x) |
| eˣ | eˣ |
| ln x, x > 0 | 1/x |
| f(g(x)) | f′(g(x))g′(x) |
For z = wx + b, the variable named in the denominator determines which derivative is required.
","At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities.
","For a small parameter displacement Δθ, first-order Taylor expansion gives:
","The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large.
","Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.
","J(w) = 3w^2+5; w = 2.
","The function value and its derivative value.
","The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.
","c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.
","Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.
","Step 1: evaluate the function itself.
","Step 2: differentiate symbolically.
","Step 3: put the given w into the derivative.
","Final answer: J = 17; derivative = 12.
","Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.
","J(w) = 0.7w^3+0; w = 1.
","The function value and its derivative value.
","The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.
","c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.
","Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.
","Step 1: evaluate the function itself.
","Step 2: differentiate symbolically.
","Step 3: put the given w into the derivative.
","Final answer: J = 0.7; derivative = 2.1.
","Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.
","J(w) = 2w^1+4; w = -1.
","The function value and its derivative value.
","The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.
","c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.
","Yaad rakhne ka cue: Power neeche multiply; exponent ek kam; constant ka zero.
","Step 1: evaluate the function itself.
","Step 2: differentiate symbolically.
","Step 3: put the given w into the derivative.
","Final answer: J = 2; derivative = 2.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s3","search":"Derivatives without a calculus course **Definition.** A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector.\n\nThe rules needed for the supplied neural-network numericals are:\n\n| Function or operation | Derivative |\n| Constant c | 0 |\n| xⁿ | nxⁿ⁻¹ |\n| c f(x) | c f′(x) |\n| f(x) + g(x) | f′(x) + g′(x) |\n| f(x)g(x) | f′(x)g(x) + f(x)g′(x) |\n| eˣ | eˣ |\n| ln x, x > 0 | 1/x |\n| f(g(x)) | f′(g(x))g′(x) |\n\n### A parameter derivative is not an input derivative\nFor z = wx + b, the variable named in the denominator determines which derivative is required.\n\nAt x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities.\n\n### Gradient direction\nFor a small parameter displacement Δθ, first-order Taylor expansion gives:\n\nThe gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large.\n\n\n\n### Differentiate before substituting: 3w^2+5\n\n**Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.**\n\n\n\n### Given\n\nJ(w) = 3w^2+5; w = 2.\n\n### To find\n\nThe function value and its derivative value.\n\n### Formula — which one and why\n\nThe power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.\n\nc is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.\n\n\n\n**Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero.\n\n### Solution — step by step\n\n\n\n**Step 1: evaluate the function itself.**\n\n\n\n\n\n\n\n**Step 2: differentiate symbolically.**\n\n\n\n\n\n\n\n**Step 3: put the given w into the derivative.**\n\n\n\n\n\n\n\n**Final answer:** J = 17; derivative = 12.\n\n\n\n### Differentiate before substituting: 0.7w^3+0\n\n**Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.**\n\n\n\n### Given\n\nJ(w) = 0.7w^3+0; w = 1.\n\n### To find\n\nThe function value and its derivative value.\n\n### Formula — which one and why\n\nThe power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.\n\nc is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.\n\n\n\n**Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero.\n\n### Solution — step by step\n\n\n\n**Step 1: evaluate the function itself.**\n\n\n\n\n\n\n\n**Step 2: differentiate symbolically.**\n\n\n\n\n\n\n\n**Step 3: put the given w into the derivative.**\n\n\n\n\n\n\n\n**Final answer:** J = 0.7; derivative = 2.1.\n\n\n\n### Differentiate before substituting: 2w^1+4\n\n**Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.**\n\n\n\n### Given\n\nJ(w) = 2w^1+4; w = -1.\n\n### To find\n\nThe function value and its derivative value.\n\n### Formula — which one and why\n\nThe power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero.\n\nc is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities.\n\n\n\n**Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero.\n\n### Solution — step by step\n\n\n\n**Step 1: evaluate the function itself.**\n\n\n\n\n\n\n\n**Step 2: differentiate symbolically.**\n\n\n\n\n\n\n\n**S"},{"title":"Derivatives: from an ordinary change to a local slope","blocks":["Let L(w) = w². At w = 3, loss is 9. If we increase w to 4, loss becomes 16. The loss change is 16 − 9 = 7 and the weight change is 4 − 3 = 1. Their ratio is 7. This is an average slope over a whole interval.
","For the same starting weight 3, use an increase of 0.1. The new weight is 3.1 and its square is 9.61. The slope estimate is (9.61 − 9)/0.1 = 6.1. With an increase of 0.01, it is (9.0601 − 9)/0.01 = 6.01. Smaller intervals approach 6.
","| Increase in w | New w | New loss w² | Change in loss | Change ratio |
|---|---|---|---|---|
| 1 | 4 | 16 | 7 | 7 |
| 0.1 | 3.1 | 9.61 | 0.61 | 6.1 |
| 0.01 | 3.01 | 9.0601 | 0.0601 | 6.01 |
The derivative is the limiting local slope as that interval approaches zero. We do not substitute a literal zero into a division by the interval.
","Call a small nonzero change Δw, read “delta w”. Expanding the new square gives:
","Subtract the old square w² and divide by the change Δw:
","As Δw approaches zero, this approaches 2w. That is the origin of the rule d(w²)/dw = 2w. At w = 3 it gives 6, matching the table's trend.
","In dL/dw, the top names the quantity whose response we study, and the bottom names what we vary. Read it as “the derivative of L with respect to w”. The notation is not asking you to divide the current loss 9 by the current weight 3; that would give 3 and answer a different question.
","For a function of several variables, ∂L/∂w is a partial derivative: change w while holding the other independent variables fixed. The gradient groups those partial derivatives for all parameters.
","If the derivative is positive, increasing w locally increases loss, so we try a small decrease. If it is negative, increasing w locally decreases loss, so we try a small increase. The subtraction in gradient descent automatically handles both signs.
","This is local information. A very large step can cross a minimum and make loss worse. The learning rate η keeps control of the step; the derivative does not promise that any step size is safe.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s4","search":"Derivatives: from an ordinary change to a local slope ### 1. Function value and change are different numbers\nLet L(w) = w². At w = 3, loss is 9. If we increase w to 4, loss becomes 16. The loss change is 16 − 9 = 7 and the weight change is 4 − 3 = 1. Their ratio is 7. This is an average slope over a whole interval.\n\n\n\n\n### 2. Shrink the interval\nFor the same starting weight 3, use an increase of 0.1. The new weight is 3.1 and its square is 9.61. The slope estimate is (9.61 − 9)/0.1 = 6.1. With an increase of 0.01, it is (9.0601 − 9)/0.01 = 6.01. Smaller intervals approach 6.\n| Increase in w | New w | New loss w² | Change in loss | Change ratio |\n| 1 | 4 | 16 | 7 | 7 |\n| 0.1 | 3.1 | 9.61 | 0.61 | 6.1 |\n| 0.01 | 3.01 | 9.0601 | 0.0601 | 6.01 |\nThe derivative is the limiting local slope as that interval approaches zero. We do not substitute a literal zero into a division by the interval.\n\n### 3. Why the square has derivative 2w\nCall a small nonzero change Δw, read “delta w”. Expanding the new square gives:\n\nSubtract the old square w² and divide by the change Δw:\n\nAs Δw approaches zero, this approaches 2w. That is the origin of the rule d(w²)/dw = 2w. At w = 3 it gives 6, matching the table's trend.\n\n### 4. Reading derivative notation\nIn dL/dw, the top names the quantity whose response we study, and the bottom names what we vary. Read it as “the derivative of L with respect to w”. The notation is not asking you to divide the current loss 9 by the current weight 3; that would give 3 and answer a different question.\nFor a function of several variables, ∂L/∂w is a partial derivative: change w while holding the other independent variables fixed. The gradient groups those partial derivatives for all parameters.\n\n### 5. Why subtract the gradient?\nIf the derivative is positive, increasing w locally increases loss, so we try a small decrease. If it is negative, increasing w locally decreases loss, so we try a small increase. The subtraction in gradient descent automatically handles both signs.\n\nThis is local information. A very large step can cross a minimum and make loss worse. The learning rate η keeps control of the step; the derivative does not promise that any step size is safe.\n"},{"title":"Partial derivatives hold other things still","blocks":["For z = wx + b, the derivative with respect to w is x, because x is held fixed. The derivative with respect to b is 1. The derivative with respect to x is w. This is the entire local maths of a neuron’s weighted sum.
","Question: z = 3w + b, w = 2, b = 1. Find z, ∂z/∂w and ∂z/∂b.
","z = 3w+b; w = 2; b = 1.
","Find z, ∂z/∂w and ∂z/∂b.
","Partial derivatives. Differentiate only with respect to the named variable; all other independent inputs and parameters remain fixed.
","∂z/∂w reads “rate of change of z when w changes”. For z = wx+b, x is the coefficient of w, while b has coefficient 1.
","Yaad rakhne ka cue: Neeche jis letter ka naam hai, sirf usko change samjho.
","Step 1: calculate the function value.
","Step 2: differentiate with respect to w while b is fixed.
","Step 3: differentiate with respect to b while w is fixed.
","Final answer: z = 7; weight derivative = 3; bias derivative = 1. A function value and its rate of change are different quantities.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s5","search":"Partial derivatives hold other things still For z = wx + b, the derivative with respect to w is x, because x is held fixed. The derivative with respect to b is 1. The derivative with respect to x is w. This is the entire local maths of a neuron’s weighted sum.\n\n\n\n\n**Question: ** z = 3w + b, w = 2, b = 1. Find z, ∂z/∂w and ∂z/∂b.\n\n\n\n### Given\n\nz = 3w+b; w = 2; b = 1.\n\n### To find\n\nFind z, ∂z/∂w and ∂z/∂b.\n\n### Formula — which one and why\n\n**Partial derivatives.** Differentiate only with respect to the named variable; all other independent inputs and parameters remain fixed.\n\n∂z/∂w reads “rate of change of z when w changes”. For z = wx+b, x is the coefficient of w, while b has coefficient 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Neeche jis letter ka naam hai, sirf usko change samjho.\n\n### Solution — step by step\n\n\n\n\n**Step 1: calculate the function value.**\n\n\n\n\n\n\n\n**Step 2: differentiate with respect to w while b is fixed.**\n\n\n\n\n\n\n\n**Step 3: differentiate with respect to b while w is fixed.**\n\n\n\n\n\n\n\n**Final answer:** z = 7; weight derivative = 3; bias derivative = 1. A function value and its rate of change are different quantities."},{"title":"The chain rule is multiplying linked effects","blocks":["Definition. The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions.
","Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency.
","The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w.
","Yaad rakhne ka cue: ek raaste par slopes multiply; alag raaston ke full answers add.
","Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.
","w = 1; u = 3w+1; v = 2u.
","The two function values and the rate dv/dw.
","v depends on w through u. Both links must be included, so multiply the two local derivatives.
","du/dw is the slope of 3w+1. dv/du is the slope of 2u.
","Yaad rakhne ka cue: Andar ka slope × bahar ka slope.
","Step 1: calculate the intermediate value.
","Step 2: calculate the final value.
","Step 3: differentiate the first link.
","Step 4: differentiate the second link.
","Step 5: multiply along the path.
","Final answer: u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different.
","Question. A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1.
","Input x = 2; target y = 4.
","Old weight w = 1; old bias b = 0.
","Linear prediction z = wx+b; loss L = ½(z−y)².
","Learning rate η = 0.1; update both parameters once.
","Find the loss gradient for w and for b, then w_new and b_new. Finally evaluate the changed prediction and loss.
","Chain rule for a linear neuron. The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.
","z = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.
","Yaad rakhne ka cue: Ek path mein multiply; alag paths mil rahe hon toh add.
","Step 1 — evaluate the original model. Use the given parameters before applying any update.
","The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide.
","Step 2 — differentiate the loss, keeping the target constant. First differentiate the square, then the error inside it.
","The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection.
","Step 3 — differentiate the neuron's affine expression. When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1.
","Step 4 — multiply along each parameter's path.
","Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate.
","Step 5 — update both parameters simultaneously. Gradient descent subtracts the learning rate times the corresponding gradient.
","Both updates use gradients from the same original forward pass. Re-evaluate the model afterward.
Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure.
","Step 6 — perform a new forward pass to evaluate the update.
","Final answer: weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3.
","A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward.
","Question. A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed.
","h = wx+b; outputs o₁ = 2h and o₂ = −h.
","x = 2; w = 0.5; b = 0.1; targets = [1,0].
","Loss = half the sum of the two squared output errors.
","Output multipliers 2 and −1 stay fixed.
","Find total loss and its derivatives with respect to h, w and b.
","Chain rule with more than one path. The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b.
","oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path.
","Yaad rakhne ka cue: Har raaste ka effect pehle multiply, junction par sab add.
","Step 1 — forward values.
","Step 2 — output derivatives. Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output.
","Step 3 — calculate the contribution of each complete path. The derivative of 2h with respect to h is 2; the derivative of −h is −1.
","Step 4 — add the two contributions at the shared hidden value.
","Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path.
","Step 5 — continue to the hidden weight and bias. Since h = wx + b, its derivatives with respect to w and b are x and 1.
","Final answer: L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation.
","Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.
","w = 1; z = 3w+1; L = z².
","z, L and the derivative of L with respect to w.
","The square acts on z, not directly on w. Differentiate the square first and include the change of z with w.
","dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression.
","Yaad rakhne ka cue: Square ka 2z; andar ka 3; dono multiply.
","Step 1: evaluate the inner expression.
","Step 2: evaluate the loss expression.
","Step 3: differentiate the square and use z = 4.
","Step 4: differentiate the inner expression.
","Step 5: multiply, because the changes occur in sequence.
","Final answer: z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s6","search":"The chain rule is multiplying linked effects **Definition.** The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions.\n\n### First understand why the rates multiply\nSuppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency.\n\nThe notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w.\n\n**Yaad rakhne ka cue:** ek raaste par slopes multiply; alag raaston ke full answers add.\n\n\n### Chain-rule example 1: two linear links\n\n**Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.**\n\n\n\n### Given\n\nw = 1; u = 3w+1; v = 2u.\n\n### To find\n\nThe two function values and the rate dv/dw.\n\n### Formula — which one and why\n\nv depends on w through u. Both links must be included, so multiply the two local derivatives.\n\ndu/dw is the slope of 3w+1. dv/du is the slope of 2u.\n\n\n\n**Yaad rakhne ka cue:** Andar ka slope × bahar ka slope.\n\n### Solution — step by step\n\n\n\n**Step 1: calculate the intermediate value.**\n\n\n\n\n\n\n\n**Step 2: calculate the final value.**\n\n\n\n\n\n\n\n**Step 3: differentiate the first link.**\n\n\n\n\n\n\n\n**Step 4: differentiate the second link.**\n\n\n\n\n\n\n\n**Step 5: multiply along the path.**\n\n\n\n\n\n\n\n**Final answer:** u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different.\n\n### 2. One output: calculate both parameter gradients\n**Question.** A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1.\n\n\n\n### Given\n\nInput x = 2; target y = 4.\n\nOld weight w = 1; old bias b = 0.\n\nLinear prediction z = wx+b; loss L = ½(z−y)².\n\nLearning rate η = 0.1; update both parameters once.\n\n### To find\n\nFind the loss gradient for w and for b, then w_new and b_new. Finally evaluate the changed prediction and loss.\n\n### Formula — which one and why\n\n**Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.\n\nz = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.\n\n\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add.\n\n### Solution — step by step\n\n\n\n\n\n\n\n**Step 1 — evaluate the original model.** Use the given parameters before applying any update.\n\n\n\nThe prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide.\n\n**Step 2 — differentiate the loss, keeping the target constant.** First differentiate the square, then the error inside it.\n\n\n\nThe symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection.\n\n**Step 3 — differentiate the neuron's "},{"title":"The derivative of prediction error","blocks":["For the half squared loss, use prediction minus target. Let target be y and prediction be ŷ.
","The 2 from differentiating a square cancels the half. Writing the loss as ½(y − ŷ)² gives the same derivative: the derivative of the inside y − ŷ adds a minus sign. This is why “target minus prediction” without the extra minus gives the wrong gradient.
","If the question uses plain squared error, its derivative is 2(ŷ − y). If it averages errors over n values, also divide by n. Read the stated loss before using a memorized gradient.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s7","search":"The derivative of prediction error For the half squared loss, use prediction minus target. Let target be y and prediction be ŷ.\n\n\n\n\nThe 2 from differentiating a square cancels the half. Writing the loss as ½(y − ŷ)² gives the same derivative: the derivative of the inside y − ŷ adds a minus sign. This is why “target minus prediction” without the extra minus gives the wrong gradient.\nIf the question uses plain squared error, its derivative is 2(ŷ − y). If it averages errors over n values, also divide by n. Read the stated loss before using a memorized gradient."},{"title":"Loss scaling: derive the gradient instead of guessing a factor","blocks":["Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square:
","Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below.
","| Stated loss over n outputs | Derivative with respect to one prediction aⱼ |
|---|---|
| SSE: sum of all squared errors | 2(aⱼ − yⱼ) |
| Half SSE: half of that sum | aⱼ − yⱼ |
| MSE: that sum divided by n | 2(aⱼ − yⱼ)/n |
| Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n |
Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor.
","Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages.
","If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore:
","For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question.
","For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042.
","The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s8","search":"Loss scaling: derive the gradient instead of guessing a factor ### 1. One error, three possible objectives\nLet a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square:\n\n\n\n\nMultiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below.\n\n| Stated loss over n outputs | Derivative with respect to one prediction aⱼ |\n| SSE: sum of all squared errors | 2(aⱼ − yⱼ) |\n| Half SSE: half of that sum | aⱼ − yⱼ |\n| MSE: that sum divided by n | 2(aⱼ − yⱼ)/n |\n| Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n |\n\n### 2. Why only the matching output appears\nSuppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor.\nLater, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages.\n\n### 3. The activation contributes another factor\nIf a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore:\n\nFor a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question.\n\n### 4. A numerical comparison\nFor a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042.\nThe forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used.\n"},{"title":"Probability mean and variance in plain language","blocks":["Definitions. A probability assigns a value between 0 and 1 to an event. A discrete probability distribution assigns probabilities to mutually exclusive outcomes whose probabilities sum to 1. Expectation is a probability-weighted average. Variance measures expected squared deviation from the mean.
","P(A|B) is the probability of A conditional on B, defined when P(B) > 0. Bayes' rule relates it to the reverse conditional probability.
","For classification, the network estimates P(class | input). A softmax output is normalized across mutually exclusive classes. Independent sigmoid outputs are used for multiple labels that can be present together.
","For N observed values, use the stated variance convention. These notes use division by N for the population-style variance in the numerical; an unbiased sample-variance estimate uses N − 1.
","For values [3, 3, 7, 7], μ = 5, squared deviations are [4, 4, 4, 4], σ² = 16/4 = 4 and σ = 2. Thus a value 7 standardizes to (7 − 5)/2 = 1: one standard deviation above the mean. For preprocessing, fit μ and σ on training data and reuse them for validation/test data; σ must be nonzero.
","If the correct class has predicted probability p, its negative-log loss is −ln p. For p = 0.8, this is approximately 0.223144. For p = 0.1, it is approximately 2.302585. Increasing correct-class probability lowers this loss. Over independent examples, maximizing the product of correct-class probabilities is equivalent to minimizing the sum of negative log probabilities.
","Question. For the population [1, 2, 3], find mean, population variance and standard deviation.
","Values = [1, 2, 3]; N = 3. Use population division by N.
","Mean μ, variance σ² and standard deviation σ.
","The question specifies a population, so use N rather than the N−1 sample-variance divisor.
","μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.
","Yaad rakhne ka cue: Mean, distance, square, mean again, then root.
","Step 1: add all values.
","Step 2: divide by the number of values.
","Step 3: find and square deviation 1.
","Step 4: find and square deviation 2.
","Step 5: find and square deviation 3.
","Step 6: average the squared deviations.
","Step 7: take the nonnegative square root.
","Final answer: μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581.
","Question. For the population [2, 2, 2], find mean, population variance and standard deviation.
","Values = [2, 2, 2]; N = 3. Use population division by N.
","Mean μ, variance σ² and standard deviation σ.
","The question specifies a population, so use N rather than the N−1 sample-variance divisor.
","μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.
","Yaad rakhne ka cue: Mean, distance, square, mean again, then root.
","Step 1: add all values.
","Step 2: divide by the number of values.
","Step 3: find and square deviation 1.
","Step 4: find and square deviation 2.
","Step 5: find and square deviation 3.
","Step 6: average the squared deviations.
","Step 7: take the nonnegative square root.
","Final answer: μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s9","search":"Probability mean and variance in plain language **Definitions.** A probability assigns a value between 0 and 1 to an event. A discrete probability distribution assigns probabilities to mutually exclusive outcomes whose probabilities sum to 1. Expectation is a probability-weighted average. Variance measures expected squared deviation from the mean.\n\n\n\n### Conditional probability and classification\nP(A|B) is the probability of A conditional on B, defined when P(B) > 0. Bayes' rule relates it to the reverse conditional probability.\n\nFor classification, the network estimates P(class | input). A softmax output is normalized across mutually exclusive classes. Independent sigmoid outputs are used for multiple labels that can be present together.\n\n### Empirical mean, variance and standardization\nFor N observed values, use the stated variance convention. These notes use division by N for the population-style variance in the numerical; an unbiased sample-variance estimate uses N − 1.\n\n\nFor values [3, 3, 7, 7], μ = 5, squared deviations are [4, 4, 4, 4], σ² = 16/4 = 4 and σ = 2. Thus a value 7 standardizes to (7 − 5)/2 = 1: one standard deviation above the mean. For preprocessing, fit μ and σ on training data and reuse them for validation/test data; σ must be nonzero.\n\n### Likelihood and cross-entropy\nIf the correct class has predicted probability p, its negative-log loss is −ln p. For p = 0.8, this is approximately 0.223144. For p = 0.1, it is approximately 2.302585. Increasing correct-class probability lowers this loss. Over independent examples, maximizing the product of correct-class probabilities is equivalent to minimizing the sum of negative log probabilities.\n\n### Population mean and spread: [1, 2, 3]\n\n**Question. For the population [1, 2, 3], find mean, population variance and standard deviation.**\n\n\n\n### Given\n\nValues = [1, 2, 3]; N = 3. Use population division by N.\n\n### To find\n\nMean μ, variance σ² and standard deviation σ.\n\n### Formula — which one and why\n\nThe question specifies a population, so use N rather than the N−1 sample-variance divisor.\n\nμ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Mean, distance, square, mean again, then root.\n\n### Solution — step by step\n\n\n\n**Step 1: add all values.**\n\n\n\n\n\n\n\n**Step 2: divide by the number of values.**\n\n\n\n\n\n\n\n**Step 3: find and square deviation 1.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 4: find and square deviation 2.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 5: find and square deviation 3.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 6: average the squared deviations.**\n\n\n\n\n\n\n\n**Step 7: take the nonnegative square root.**\n\n\n\n\n\n\n\n**Final answer:** μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581.\n\n\n\n\n\n### Population mean and spread: [2, 2, 2]\n\n**Question. For the population [2, 2, 2], find mean, population variance and standard deviation.**\n\n\n\n### Given\n\nValues = [2, 2, 2]; N = 3. Use population division by N.\n\n### To find\n\nMean μ, variance σ² and standard deviation σ.\n\n### Formula — which one and why\n\nThe question specifies a population, so use N rather than the N−1 sample-variance divisor.\n\nμ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Mean, distance, square, mean again, then root.\n\n### Solution — step by step\n\n\n\n**Step 1: add all values.**\n\n\n\n\n\n\n\n**Step 2: divide by the number of values.**\n\n\n\n\n\n\n\n**Step 3: find and square deviation 1.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 4: find and square deviation 2.**\n\n\n"},{"title":"Mean, variance and standard deviation from ordinary arithmetic","blocks":["Definition. The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units.
","These are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable.
","Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].
","Observations = [3,3,7,7]; N = 4.
","Use population variance (divide by N).
","Find the population mean μ, variance σ² and standard deviation σ.
","Mean and spread. The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.
","N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.
","Yaad rakhne ka cue: Mean = sabka total ÷ count; variance = squared distances ka mean.
","Step 1: add all four observations.
","Step 2: divide by the count to obtain the mean.
","Step 3: subtract the mean from each observation.
","Step 4: square each deviation.
","Step 5: add the squared deviations.
","Step 6: divide by four for population variance.
","Step 7: take the nonnegative square root.
","Answer: mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s10","search":"Mean, variance and standard deviation from ordinary arithmetic \n**Definition.** The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units.\n\nThese are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable.\n\n**Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].**\n\n\n\n### Given\n\nObservations = [3,3,7,7]; N = 4.\n\nUse population variance (divide by N).\n\n### To find\n\nFind the population mean μ, variance σ² and standard deviation σ.\n\n### Formula — which one and why\n\n**Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.\n\nN = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: add all four observations.**\n\n\n\n\n\n\n\n**Step 2: divide by the count to obtain the mean.**\n\n\n\n\n\n\n\n**Step 3: subtract the mean from each observation.**\n\n\n\n\n\n\n\n**Step 4: square each deviation.**\n\n\n\n\n\n\n\n**Step 5: add the squared deviations.**\n\n\n\n\n\n\n\n**Step 6: divide by four for population variance.**\n\n\n\n\n\n\n\n**Step 7: take the nonnegative square root.**\n\n\n\n\n\n\n\n**Answer:** mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3.\n\n"},{"title":"Standardization: what the answer 1 actually means","blocks":["Definition. Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units.
","Subtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1].
","Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.
","Training observations = [3,3,7,7].
","New observation = 9.
","Estimate the population standard deviation from the training values.
","Find the training mean and standard deviation; standardize all four training values and the new observation.
","Mean and spread. The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.
","N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.
","Yaad rakhne ka cue: Mean = sabka total ÷ count; variance = squared distances ka mean.
","Standardization. Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units.
","x = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value.
","Yaad rakhne ka cue: Pehle centre hatao, phir spread se divide karo.
","Step 1: add all four observations.
","Step 2: divide by the count to obtain the mean.
","Step 3: subtract the mean from each observation.
","Step 4: square each deviation.
","Step 5: add the squared deviations.
","Step 6: divide by four for population variance.
","Step 7: take the nonnegative square root.
","Step 8: standardize each original training observation.
","The repeated observations give [-1, −1, 1, 1].
","Step 9: subtract the training mean from the new observation.
","Step 10: divide by the training standard deviation.
","Answer: the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy.
","Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.
","x = 3; training mean μ = 5; training standard deviation σ = 2.
","Standardized value z and its meaning.
","Subtract the training centre, then measure that distance in units of the training standard deviation.
","A negative z means below the training mean; positive means above it; zero means equal to it.
","Yaad rakhne ka cue: Centre hatao, spread se divide; answer batata hai kitne σ door.
","Step 1: find distance from the mean.
","Step 2: divide by one standard-deviation unit.
","Step 3: interpret sign and size.
","The value is 1 standard-deviation unit(s) below the training mean.
","Final answer: z = -1. Use the same training statistics when transforming a new test input.
","Question. Training mean is 5 and training standard deviation is 2. Standardize x = 5 and interpret the result.
","x = 5; training mean μ = 5; training standard deviation σ = 2.
","Standardized value z and its meaning.
","Subtract the training centre, then measure that distance in units of the training standard deviation.
","A negative z means below the training mean; positive means above it; zero means equal to it.
","Yaad rakhne ka cue: Centre hatao, spread se divide; answer batata hai kitne σ door.
","Step 1: find distance from the mean.
","Step 2: divide by one standard-deviation unit.
","Step 3: interpret sign and size.
","The value is 0 standard-deviation unit(s) away from the training mean.
","Final answer: z = 0. Use the same training statistics when transforming a new test input.
","Question. Training mean is 5 and training standard deviation is 2. Standardize x = 9 and interpret the result.
","x = 9; training mean μ = 5; training standard deviation σ = 2.
","Standardized value z and its meaning.
","Subtract the training centre, then measure that distance in units of the training standard deviation.
","A negative z means below the training mean; positive means above it; zero means equal to it.
","Yaad rakhne ka cue: Centre hatao, spread se divide; answer batata hai kitne σ door.
","Step 1: find distance from the mean.
","Step 2: divide by one standard-deviation unit.
","Step 3: interpret sign and size.
","The value is 2 standard-deviation unit(s) above the training mean.
","Final answer: z = 2. Use the same training statistics when transforming a new test input.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s11","search":"Standardization: what the answer 1 actually means \n**Definition.** Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units.\n\nSubtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1].\n\n**Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.**\n\n\n\n### Given\n\nTraining observations = [3,3,7,7].\n\nNew observation = 9.\n\nEstimate the population standard deviation from the training values.\n\n### To find\n\nFind the training mean and standard deviation; standardize all four training values and the new observation.\n\n### Formula — which one and why\n\n**Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root.\n\nN = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean.\n\n**Standardization.** Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units.\n\nx = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value.\n\n\n\n**Yaad rakhne ka cue:** Pehle centre hatao, phir spread se divide karo.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: add all four observations.**\n\n\n\n\n\n\n\n**Step 2: divide by the count to obtain the mean.**\n\n\n\n\n\n\n\n**Step 3: subtract the mean from each observation.**\n\n\n\n\n\n\n\n**Step 4: square each deviation.**\n\n\n\n\n\n\n\n**Step 5: add the squared deviations.**\n\n\n\n\n\n\n\n**Step 6: divide by four for population variance.**\n\n\n\n\n\n\n\n**Step 7: take the nonnegative square root.**\n\n\n\n\n\n\n\n**Step 8: standardize each original training observation.**\n\n\n\n\n\n\n\n\n\nThe repeated observations give [-1, −1, 1, 1].\n\n**Step 9: subtract the training mean from the new observation.**\n\n\n\n\n\n\n\n**Step 10: divide by the training standard deviation.**\n\n\n\n\n\n\n\n**Answer:** the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy.\n\n### Standardization with x = 3\n\n**Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.**\n\n\n\n### Given\n\nx = 3; training mean μ = 5; training standard deviation σ = 2.\n\n### To find\n\nStandardized value z and its meaning.\n\n### Formula — which one and why\n\nSubtract the training centre, then measure that distance in units of the training standard deviation.\n\nA negative z means below the training mean; positive means above it; zero means equal to it.\n\n\n\n**Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door.\n\n### Solution — step by step\n\n\n\n**Step 1: find distance from the mean.**\n\n\n\n\n\n\n\n**Step 2: divide by one standard-deviation unit.**\n\n\n\n\n\n\n\n"},{"title":"Chain rule: the complete calculation","blocks":["When one value depends on another, a change passes through both calculations. The total derivative multiplies the two local derivatives. It does not add them. Here w is the number being changed, z is an intermediate value, and L is the final value.
","Let z = 3w + 1 and L = z². We will calculate the derivative of L with respect to w at w = 1.
","Step 1: find the intermediate value. Replace w by 1: z = 3 × 1 + 1 = 4.
","Step 2: differentiate the last calculation. For L = z², the power rule gives dL/dz = 2z. At z = 4, this is 2 × 4 = 8. This measures how L responds to z.
","Step 3: differentiate the first calculation. For z = 3w + 1, the derivative of 3w is 3 and the derivative of the constant 1 is 0. Therefore dz/dw = 3. This measures how z responds to w.
","Step 4: connect the two effects. Multiply 8 by 3. The final derivative is 24.
","The function value is L = 4² = 16. The derivative is 24. These have different meanings: 16 is the current value of L; 24 is its local rate of change per unit change in w.
"],"question":[],"answer":[],"kind":"read","id":"tee-c3-s12","search":"Chain rule: the complete calculation When one value depends on another, a change passes through both calculations. The total derivative multiplies the two local derivatives. It does not add them. Here w is the number being changed, z is an intermediate value, and L is the final value.\n\n\n\nLet z = 3w + 1 and L = z². We will calculate the derivative of L with respect to w at w = 1.\n**Step 1: find the intermediate value.** Replace w by 1: z = 3 × 1 + 1 = 4.\n**Step 2: differentiate the last calculation.** For L = z², the power rule gives dL/dz = 2z. At z = 4, this is 2 × 4 = 8. This measures how L responds to z.\n**Step 3: differentiate the first calculation.** For z = 3w + 1, the derivative of 3w is 3 and the derivative of the constant 1 is 0. Therefore dz/dw = 3. This measures how z responds to w.\n**Step 4: connect the two effects.** Multiply 8 by 3. The final derivative is 24.\n\nThe function value is L = 4² = 16. The derivative is 24. These have different meanings: 16 is the current value of L; 24 is its local rate of change per unit change in w.\n"}]},{"n":4,"title":"Neural-network foundations","sections":[{"title":"What does learning from data mean","blocks":["Definition. Machine learning develops models from data so that they can perform a task on new examples. A model is a calculation rule with adjustable numbers. Deep learning uses neural networks with multiple trainable layers; each layer transforms the previous layer's outputs.
","A sample means one example in the dataset. Suppose one example says: an input number is 2 and the required answer is 4. We write the input as x = 2 and the correct answer, or target, as y = 4. These letters are labels for the two numbers.
","An input feature is one recorded property used to make a prediction. One house might have area, number of rooms and age as three features. The three values together describe one sample. An input vector is just an ordered list of those feature values. Order matters: the first model weight must always multiply the first feature.
","The model's calculated answer is the prediction, written ŷ, read “y hat”. The hat helps us distinguish the calculated answer ŷ from the required answer y. A prediction is a probability only when the task and output function give it that interpretation.
","In the following model, x is the input, w is a weight and b is a bias. A weight multiplies the input. A bias is added once after multiplication. Both w and b are adjustable model numbers, called parameters.
","The model first multiplies w and x; then it adds b. For several inputs it uses one matching weight per input, adds their products, and then adds the bias.
","Yaad rakhne ka cue: x diya hua input; y sahi answer; ŷ model ka answer; w aur b training mein badlenge.
","Question. A linear model uses ŷ = wx+b. For x = 2, w = 1 and b = 0, calculate its prediction.
","Input x = 2.
","Weight w = 1; bias b = 0.
","The output is linear: no extra activation is specified.
","The predicted value ŷ.
","Use the stated model. The multiplication gives the input contribution; the bias shifts the result.
","x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.
","Yaad rakhne ka cue: Pehle input × weight, phir bias add.
","Step 1: multiply the input by its weight.
","Step 2: add the bias once.
","Final answer: ŷ = 2. This is a prediction, not a loss or a learned weight.
","Question. A linear model uses ŷ = wx+b. For x = 2, w = 1.5 and b = 1, calculate its prediction.
","Input x = 2.
","Weight w = 1.5; bias b = 1.
","The output is linear: no extra activation is specified.
","The predicted value ŷ.
","Use the stated model. The multiplication gives the input contribution; the bias shifts the result.
","x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.
","Yaad rakhne ka cue: Pehle input × weight, phir bias add.
","Step 1: multiply the input by its weight.
","Step 2: add the bias once.
","Final answer: ŷ = 4. This is a prediction, not a loss or a learned weight.
","Question. A linear model uses ŷ = wx+b. For x = 3, w = 1.5 and b = 1, calculate its prediction.
","Input x = 3.
","Weight w = 1.5; bias b = 1.
","The output is linear: no extra activation is specified.
","The predicted value ŷ.
","Use the stated model. The multiplication gives the input contribution; the bias shifts the result.
","x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.
","Yaad rakhne ka cue: Pehle input × weight, phir bias add.
","Step 1: multiply the input by its weight.
","Step 2: add the bias once.
","Final answer: ŷ = 5.5. This is a prediction, not a loss or a learned weight.
","Question. A linear model uses ŷ = wx+b. For x = 2, w = -0.5 and b = 3, calculate its prediction.
","Input x = 2.
","Weight w = -0.5; bias b = 3.
","The output is linear: no extra activation is specified.
","The predicted value ŷ.
","Use the stated model. The multiplication gives the input contribution; the bias shifts the result.
","x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.
","Yaad rakhne ka cue: Pehle input × weight, phir bias add.
","Step 1: multiply the input by its weight.
","Step 2: add the bias once.
","Final answer: ŷ = 2. This is a prediction, not a loss or a learned weight.
","An initial model can predict 2 when the target is 4. A loss function measures the discrepancy. Training calculates how changing each parameter changes the loss; that rate is a gradient. An optimizer uses the gradient to adjust the parameters. The learning rate controls the size of this adjustment. The loss, derivative and complete update are explained with full examples in the next lessons.
","The learning rate is a hyperparameter: a setting chosen for training. A learned weight is a parameter: an adjustable number inside the model. During ordinary inference, the trained weights stay fixed and the model calculates an output for a new input.
","For many training samples, the objective normally averages the individual losses. θ (theta) means the collection of adjustable parameters; N is the number of training examples; Lᵢ is the loss for example i.
","Σ is an instruction to add the listed terms. For three losses it means L₁ + L₂ + L₃, followed by division by 3. Regularization may add a penalty; the regularization chapter explains that separate term.
","A layer contains several neurons that calculate outputs from the same incoming values. In an image model, earlier layers can respond to simple patterns such as edges; later layers combine earlier responses into more complex features. This hierarchy is learned from data. A deep network is not guaranteed to learn useful features merely because it has many layers; training data, architecture and optimization still matter.
","Generalization means performing well on new examples. Training data adjusts weights; validation data helps choose settings and stopping time; held-out test data evaluates the chosen model. Small training loss alone does not establish good generalization.
","Yaad rakhne ka cue: training = answer nikalo → loss nikalo → gradients nikalo → parameters update karo; inference = trained model se answer nikalo.
","Question. Three training examples have losses 2, 0.5 and 0.5. Calculate their mean training loss.
","L₁ = 2; L₂ = 0.5; L₃ = 0.5; N = 3.
","The mean loss J.
","The question asks for the mean, so add all sample losses and divide by the number of samples.
","Lᵢ means loss of example i. N counts examples.
","Yaad rakhne ka cue: Mean = total loss ÷ examples.
","Step 1: add the losses.
","Step 2: divide by the number of examples.
","Final answer: Mean training loss = 1.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s0","search":"What does learning from data mean **Definition.** Machine learning develops models from data so that they can perform a task on new examples. A model is a calculation rule with adjustable numbers. Deep learning uses neural networks with multiple trainable layers; each layer transforms the previous layer's outputs.\n\n### Start with one complete training example\nA **sample** means one example in the dataset. Suppose one example says: an input number is 2 and the required answer is 4. We write the input as x = 2 and the correct answer, or **target**, as y = 4. These letters are labels for the two numbers.\n\nAn **input feature** is one recorded property used to make a prediction. One house might have area, number of rooms and age as three features. The three values together describe one sample. An **input vector** is just an ordered list of those feature values. Order matters: the first model weight must always multiply the first feature.\n\nThe model's calculated answer is the **prediction**, written ŷ, read “y hat”. The hat helps us distinguish the calculated answer ŷ from the required answer y. A prediction is a probability only when the task and output function give it that interpretation.\n\n### The simplest model: multiply, then add\nIn the following model, x is the input, w is a weight and b is a bias. A **weight** multiplies the input. A **bias** is added once after multiplication. Both w and b are adjustable model numbers, called **parameters**.\n\nThe model first multiplies w and x; then it adds b. For several inputs it uses one matching weight per input, adds their products, and then adds the bias.\n\n**Yaad rakhne ka cue:** x diya hua input; y sahi answer; ŷ model ka answer; w aur b training mein badlenge.\n\n\n\n\n### Model example 1: calculate a prediction\n\n**Question. A linear model uses ŷ = wx+b. For x = 2, w = 1 and b = 0, calculate its prediction.**\n\n\n\n### Given\n\nInput x = 2.\n\nWeight w = 1; bias b = 0.\n\nThe output is linear: no extra activation is specified.\n\n### To find\n\nThe predicted value ŷ.\n\n### Formula — which one and why\n\nUse the stated model. The multiplication gives the input contribution; the bias shifts the result.\n\nx is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.\n\n\n\n**Yaad rakhne ka cue:** Pehle input × weight, phir bias add.\n\n### Solution — step by step\n\n\n\n**Step 1: multiply the input by its weight.**\n\n\n\n\n\n\n\n**Step 2: add the bias once.**\n\n\n\n\n\n\n\n**Final answer:** ŷ = 2. This is a prediction, not a loss or a learned weight.\n\n\n\n### Model example 2: calculate a prediction\n\n**Question. A linear model uses ŷ = wx+b. For x = 2, w = 1.5 and b = 1, calculate its prediction.**\n\n\n\n### Given\n\nInput x = 2.\n\nWeight w = 1.5; bias b = 1.\n\nThe output is linear: no extra activation is specified.\n\n### To find\n\nThe predicted value ŷ.\n\n### Formula — which one and why\n\nUse the stated model. The multiplication gives the input contribution; the bias shifts the result.\n\nx is the input; w is its multiplier; b is the added offset; ŷ is the calculated output.\n\n\n\n**Yaad rakhne ka cue:** Pehle input × weight, phir bias add.\n\n### Solution — step by step\n\n\n\n**Step 1: multiply the input by its weight.**\n\n\n\n\n\n\n\n**Step 2: add the bias once.**\n\n\n\n\n\n\n\n**Final answer:** ŷ = 4. This is a prediction, not a loss or a learned weight.\n\n\n\n### Model example 3: calculate a prediction\n\n**Question. A linear model uses ŷ = wx+b. For x = 3, w = 1.5 and b = 1, calculate its prediction.**\n\n\n\n### Given\n\nInput x = 3.\n\nWeight w = 1.5; bias b"},{"title":"Your first complete training step","blocks":["Question. A linear model is defined by . For the training example x = 2, y = 4, use initial weight w₀ = 1, learning rate η = 0.1 and half squared loss. Perform two gradient-descent updates and report the prediction and loss after each update.
Model: ŷ = wx; bias is absent and is not updated.
","x = 2; y = 4; initial w₀ = 1; η = 0.1.
","Loss = half squared error; perform two updates.
","Find the weight, prediction and loss after each update.
","Chain rule with only a trainable weight. The weight changes the prediction and the prediction changes the half squared loss. Multiply both rates, then subtract the scaled gradient. The stated model has no bias parameter.
","ŷ = prediction; y = target; x = input; w = weight; η = learning rate; g = loss derivative with respect to w.
","Yaad rakhne ka cue: Prediction minus target, phir input se multiply; naya weight = purana − ηg.
","The input and target remain fixed; only the weight is trainable. There is no bias parameter in this question.
","Apply the chain rule through the prediction:
","The factor 1/2 in the objective cancels the factor 2 from differentiating the square. The remaining factor x comes from differentiating wx with respect to w.
","Step 1: forward evaluation at the current parameter.
","Step 2: gradient at the current parameter.
","Step 3: parameter update.
","Step 4: evaluate the updated model.
","Use w₁ = 1.4 and recompute the gradient; reusing the previous gradient would not be gradient descent on this objective.
","| Model state | Weight | Prediction | Half squared loss |
|---|---|---|---|
| Initial | 1 | 2 | 2 |
| After update 1 | 1.4 | 2.8 | 0.72 |
| After update 2 | 1.64 | 3.28 | 0.2592 |
Interpretation. Both gradients are negative, so the updates increase the weight toward the optimum w = y/x = 2. The loss decreases in these iterations. Gradient descent does not guarantee a decrease for every learning rate; step size matters.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s1","search":"Your first complete training step \n**Question.** A linear model is defined by \\(\\hat y=wx\\). For the training example x = 2, y = 4, use initial weight w₀ = 1, learning rate η = 0.1 and half squared loss. Perform two gradient-descent updates and report the prediction and loss after each update.\n\n\n\n### Given\n\nModel: ŷ = wx; bias is absent and is not updated.\n\nx = 2; y = 4; initial w₀ = 1; η = 0.1.\n\nLoss = half squared error; perform two updates.\n\n### To find\n\nFind the weight, prediction and loss after each update.\n\n### Formula — which one and why\n\n**Chain rule with only a trainable weight.** The weight changes the prediction and the prediction changes the half squared loss. Multiply both rates, then subtract the scaled gradient. The stated model has no bias parameter.\n\nŷ = prediction; y = target; x = input; w = weight; η = learning rate; g = loss derivative with respect to w.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Prediction minus target, phir input se multiply; naya weight = purana − ηg.\n\n### Solution — step by step\n\n\n\n\n\n### Model, objective and gradient\nThe input and target remain fixed; only the weight is trainable. There is no bias parameter in this question.\n\nApply the chain rule through the prediction:\n\nThe factor 1/2 in the objective cancels the factor 2 from differentiating the square. The remaining factor x comes from differentiating wx with respect to w.\n\n\n\n### Iteration 1\n**Step 1: forward evaluation at the current parameter.**\n\n\n**Step 2: gradient at the current parameter.**\n\n**Step 3: parameter update.**\n\n**Step 4: evaluate the updated model.**\n\n\n### Iteration 2\nUse w₁ = 1.4 and recompute the gradient; reusing the previous gradient would not be gradient descent on this objective.\n\n\n\n| Model state | Weight | Prediction | Half squared loss |\n| Initial | 1 | 2 | 2 |\n| After update 1 | 1.4 | 2.8 | 0.72 |\n| After update 2 | 1.64 | 3.28 | 0.2592 |\n**Interpretation.** Both gradients are negative, so the updates increase the weight toward the optimum w = y/x = 2. The loss decreases in these iterations. Gradient descent does not guarantee a decrease for every learning rate; step size matters.\n\n"},{"title":"Classification regression and learning types","blocks":["Definition. A learning problem is specified by its data, desired output and objective. Classification estimates a discrete label or class distribution; regression estimates numerical responses. The supervision type describes the feedback available during training.
","Regression predicts a numerical quantity, such as temperature, price or a continuous rating. Classification predicts a category, such as defective/non-defective or cat/dog/bird. Binary classification has two classes. Multiclass classification chooses one of several classes. Multilabel classification can choose several labels together, such as “beach” and “sunset” for the same image.
","Supervised learning uses known input–target pairs. Most examination numericals in this course use it. Unsupervised learning searches for patterns without supplied target labels, for example grouping similar samples. Reinforcement learning learns actions using rewards from an environment. You only need these distinctions as vocabulary here; detailed algorithms from last semester are not prerequisites for this course.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s2","search":"Classification regression and learning types **Definition.** A learning problem is specified by its data, desired output and objective. Classification estimates a discrete label or class distribution; regression estimates numerical responses. The supervision type describes the feedback available during training.\n\n### Explanation and working\n**Regression** predicts a numerical quantity, such as temperature, price or a continuous rating. **Classification** predicts a category, such as defective/non-defective or cat/dog/bird. Binary classification has two classes. Multiclass classification chooses one of several classes. Multilabel classification can choose several labels together, such as “beach” and “sunset” for the same image.\n\n\n\n**Supervised learning** uses known input–target pairs. Most examination numericals in this course use it. **Unsupervised learning** searches for patterns without supplied target labels, for example grouping similar samples. **Reinforcement learning** learns actions using rewards from an environment. You only need these distinctions as vocabulary here; detailed algorithms from last semester are not prerequisites for this book."},{"title":"Training validation and test data","blocks":["Definition. Dataset partitioning separates parameter estimation, model selection and final performance assessment. This separation is essential to evaluate generalization without contaminating the evaluation with selection decisions.
","Training data changes the weights. Validation data helps choose settings such as layer size, learning rate and the stopping epoch. Test data is held back for the final estimate of performance on unseen samples. A possible split is 70% training, 15% validation, 15% testing, but the best split depends on the task and dataset size.
","For 1,000 independent samples, this split gives 700 training, 150 validation and 150 testing. Avoid putting duplicates or near-identical views of the same subject across splits. For time series, later observations normally belong after earlier observations rather than being mixed randomly without considering time.
","Data leakage occurs when information that would not be available at prediction time enters training or model selection. Fitting a normalization rule to the whole dataset before splitting can leak information. Estimate preprocessing quantities from training data and apply the same rule to validation and test data.
","Normalization example. To map a training range 0 to 100 into 0 to 1, divide by 100. A value 40 becomes 0.4. Standardization instead subtracts the training mean and divides by the training standard deviation. Scaling can make gradients easier to optimize; it does not create labels or guarantee accuracy.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s3","search":"Training validation and test data **Definition.** Dataset partitioning separates parameter estimation, model selection and final performance assessment. This separation is essential to evaluate generalization without contaminating the evaluation with selection decisions.\n\n### Explanation and working\nTraining data changes the weights. Validation data helps choose settings such as layer size, learning rate and the stopping epoch. Test data is held back for the final estimate of performance on unseen samples. A possible split is 70% training, 15% validation, 15% testing, but the best split depends on the task and dataset size.\n\n\n\nFor 1,000 independent samples, this split gives 700 training, 150 validation and 150 testing. Avoid putting duplicates or near-identical views of the same subject across splits. For time series, later observations normally belong after earlier observations rather than being mixed randomly without considering time.\n**Data leakage** occurs when information that would not be available at prediction time enters training or model selection. Fitting a normalization rule to the whole dataset before splitting can leak information. Estimate preprocessing quantities from training data and apply the same rule to validation and test data.\n**Normalization example.** To map a training range 0 to 100 into 0 to 1, divide by 100. A value 40 becomes 0.4. Standardization instead subtracts the training mean and divides by the training standard deviation. Scaling can make gradients easier to optimize; it does not create labels or guarantee accuracy."},{"title":"Parameters and hyperparameters","blocks":["Definition. Model parameters are quantities estimated during training, whereas hyperparameters specify the architecture or training procedure and are selected separately. Optimizer states are algorithmic accumulators associated with the training procedure.
","Weights and biases are parameters learned during training. Learning rate, batch size, number of epochs, network architecture, dropout rate and regularization strength are hyperparameters chosen or tuned using validation performance. Adam’s moving averages are optimizer state: they are updated by the algorithm but are not extra network connection weights.
","An epoch is one complete pass through the training dataset. A batch is the group used to calculate one update. An iteration here means one update. If there are 100 samples and batch size 10, one epoch has 10 updates. One epoch does not generally mean one update.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s4","search":"Parameters and hyperparameters **Definition.** Model parameters are quantities estimated during training, whereas hyperparameters specify the architecture or training procedure and are selected separately. Optimizer states are algorithmic accumulators associated with the training procedure.\n\n### Explanation and working\nWeights and biases are parameters learned during training. Learning rate, batch size, number of epochs, network architecture, dropout rate and regularization strength are hyperparameters chosen or tuned using validation performance. Adam’s moving averages are optimizer state: they are updated by the algorithm but are not extra network connection weights.\n\n\n\nAn **epoch** is one complete pass through the training dataset. A **batch** is the group used to calculate one update. An **iteration** here means one update. If there are 100 samples and batch size 10, one epoch has 10 updates. One epoch does not generally mean one update."},{"title":"Artificial intelligence machine learning and deep learning","blocks":["Definition. Deep learning is a subfield of machine learning that learns representations through multiple trainable processing layers. Each layer transforms the preceding representation, allowing the complete model to learn a composition of functions from data.
","Artificial intelligence studies systems that perform tasks such as perception, reasoning and decision-making. Machine learning develops models whose parameters or behavior are estimated from data. Deep learning uses layered neural models within machine learning.
","![\"\\text{Deep](\"assets/equations/862fde9ce2f6cd54.svg\")
Representation learning. Hand-designing every useful image or speech feature is difficult. A deep model can learn intermediate representations together with the final prediction objective. In a CNN, early layers may respond to local edges, while later layers combine information over larger regions. These patterns depend on training and are not guaranteed to have a particular human meaning.
","Nonlinear relationships. Real tasks often depend on combinations of input variables. Composing affine transformations with nonlinear activations can represent interactions that a single linear decision boundary cannot. Depth supplies repeated transformations; increasing only the number of neurons in one layer changes width instead.
","End-to-end optimization. The final task loss supplies gradients to the preceding trainable layers. The feature extractor and prediction head can therefore be adjusted for the same task. Preprocessing and human decisions about architecture, data and evaluation still remain necessary.
","| Factor | How it contributes |
|---|---|
| Larger datasets | Provide more examples of the variations a complex model must learn |
| GPUs and parallel computation | Accelerate the large matrix and convolution operations used in training |
| Improved training methods | Backpropagation, suitable activations, initialization and optimizers make fitting deep models more practical |
| Pretrained models | Reuse learned features so a new task may require less labeled data and training effort |
An application answer should identify the input, the prediction and the role of the learned representation.
","| Domain | Example input | Required output | Role of deep learning |
|---|---|---|---|
| Computer vision | Photograph or video frame | Image class, object locations or pixel labels | Learn visual features with a CNN and an appropriate task head |
| Language | Review, sentence or conversation | Sentiment label, translated words or a response | Learn representations of text and its context |
| Speech | Audio sequence | Words or a speaker label | Model patterns in acoustic features over time |
| Healthcare | Medical image or patient measurements | A task-specific class or numerical estimate | Learn associations from labeled examples; assess results on suitable held-out data |
| Autonomous systems and robotics | Camera and other sensor measurements | Perception estimates used by a larger control system | Recognize objects and relevant environmental features |
| Financial services | Transaction features or a time-ordered record | Fraud score, risk estimate or forecast | Model patterns and interactions among observations |
| Aspect | Traditional machine learning often uses | Deep learning often uses |
|---|---|---|
| Representation | Selected or engineered input features | Features learned jointly across layers |
| Useful architectures | Linear models, decision trees, SVMs | MLPs, CNNs, RNNs and gated recurrent models |
| Training from scratch | Can be effective with modest structured datasets | Large models often need substantial data and computation |
| Interpretation | Some models have directly inspectable rules or coefficients | Learned internal representations can be harder to interpret |
Feature learning, flexible nonlinear modeling and reuse of pretrained representations are major advantages. Limitations include computation and memory costs, overfitting with insufficient representative data, difficulty interpreting predictions, and reduced reliability when deployment data differs from training data. High training accuracy does not establish generalization.
","These are tendencies, not absolute divisions. Traditional ML can also learn representations; deep models can process tabular data. A pretrained deep model may be useful even with a small target dataset. Select the model according to the task, available data, resources and validation results.
","Question. Explain the motivation for deep learning and describe applications in at least three domains.
","Answer. Deep learning is motivated by the need to learn useful hierarchical representations and nonlinear relationships directly from complex data. Multiple trainable layers compose feature transformations, and the task loss can train those transformations jointly with the predictor. Larger datasets, GPU computation and improved optimization methods have made this practical. In computer vision, CNNs learn visual features for image classification and object localization. In language processing, neural sequence models use word representations and context for sentiment analysis or translation. In speech processing, neural models map acoustic patterns to words. The benefits must be balanced against data requirements, computation, overfitting and interpretability limitations.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s5","search":"Artificial intelligence machine learning and deep learning **Definition.** Deep learning is a subfield of machine learning that learns representations through multiple trainable processing layers. Each layer transforms the preceding representation, allowing the complete model to learn a composition of functions from data.\n\n### Position within artificial intelligence\nArtificial intelligence studies systems that perform tasks such as perception, reasoning and decision-making. Machine learning develops models whose parameters or behavior are estimated from data. Deep learning uses layered neural models within machine learning.\n\n\n\n### Motivation: why use multiple learned layers?\n**Representation learning.** Hand-designing every useful image or speech feature is difficult. A deep model can learn intermediate representations together with the final prediction objective. In a CNN, early layers may respond to local edges, while later layers combine information over larger regions. These patterns depend on training and are not guaranteed to have a particular human meaning.\n\n**Nonlinear relationships.** Real tasks often depend on combinations of input variables. Composing affine transformations with nonlinear activations can represent interactions that a single linear decision boundary cannot. Depth supplies repeated transformations; increasing only the number of neurons in one layer changes width instead.\n\n**End-to-end optimization.** The final task loss supplies gradients to the preceding trainable layers. The feature extractor and prediction head can therefore be adjusted for the same task. Preprocessing and human decisions about architecture, data and evaluation still remain necessary.\n\n### Factors that made deep learning practical\n| Factor | How it contributes |\n| Larger datasets | Provide more examples of the variations a complex model must learn |\n| GPUs and parallel computation | Accelerate the large matrix and convolution operations used in training |\n| Improved training methods | Backpropagation, suitable activations, initialization and optimizers make fitting deep models more practical |\n| Pretrained models | Reuse learned features so a new task may require less labeled data and training effort |\n\n### Applications in different domains\nAn application answer should identify the input, the prediction and the role of the learned representation.\n| Domain | Example input | Required output | Role of deep learning |\n| Computer vision | Photograph or video frame | Image class, object locations or pixel labels | Learn visual features with a CNN and an appropriate task head |\n| Language | Review, sentence or conversation | Sentiment label, translated words or a response | Learn representations of text and its context |\n| Speech | Audio sequence | Words or a speaker label | Model patterns in acoustic features over time |\n| Healthcare | Medical image or patient measurements | A task-specific class or numerical estimate | Learn associations from labeled examples; assess results on suitable held-out data |\n| Autonomous systems and robotics | Camera and other sensor measurements | Perception estimates used by a larger control system | Recognize objects and relevant environmental features |\n| Financial services | Transaction features or a time-ordered record | Fraud score, risk estimate or forecast | Model patterns and interactions among observations |\n\n### Advantages, limitations and comparison with traditional ML\n| Aspect | Traditional machine learni"},{"title":"What one artificial neuron does","blocks":["Definition. An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output.
","An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive.
","The bias is one extra adjustable number added to the total. The total before activation is called z or preactivation. The activation function is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the activation.
","Yaad rakhne ka cue: inputs × apne weights → products ka sum → ek bias → activation → output.
","| Symbol | Meaning and role |
|---|---|
| xᵢ | Input component i; the neuron receives d such components |
| wᵢ | Trainable weight multiplying that input component |
| b | Trainable bias added once to this neuron's total |
| z | Preactivation: the result before applying the activation function |
| f | Chosen activation function, such as sigmoid or ReLU |
| a | Activation: the neuron's output after applying f |
The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs.
","The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin.
","The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values.
","Question. A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output.
","Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5].
","Bias = 0.1; activation = sigmoid.
","Find the preactivation z, then the sigmoid output.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid output. The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.
","z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.
","Yaad rakhne ka cue: Minus-z ki e power, plus one, phir one divided by whole denominator.
","Step 1: multiply input 1 by weight 1.
","Step 2: multiply input 2 by weight 2.
","The minus sign belongs to the weight, so the contribution stays negative.
","Step 3: multiply input 3 by weight 3.
","Step 4: add the three contributions.
","Step 5: add the bias once.
","This is the preactivation. It is not yet the requested sigmoid output.
","Step 6: negate z for the exponential.
","Step 7: calculate the complete denominator.
","Step 8: divide one by that denominator.
","Final answer: sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025.
","With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron:
","Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together.
","Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.
","Inputs = [2].
","Weights = [3]; bias = 1.
","Activation = ReLU.
","z before activation and a after activation.
","The weighted sum combines the inputs. The output must then use the activation explicitly requested.
","z names the sum; a names the activated result. They are not always equal.
","Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.
","Step 1: multiply input 1 by its own weight.
","Step 2: add the input contributions.
","Step 3: add the bias once.
","Step 4: apply the stated activation.
","Final answer: z = 7; a = 7.
","Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.
","Inputs = [2, 1].
","Weights = [-1, 0.5]; bias = 0.2.
","Activation = ReLU.
","z before activation and a after activation.
","The weighted sum combines the inputs. The output must then use the activation explicitly requested.
","z names the sum; a names the activated result. They are not always equal.
","Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.
","Step 1: multiply input 1 by its own weight.
","Step 2: multiply input 2 by its own weight.
","Step 3: add the input contributions.
","Step 4: add the bias once.
","Step 5: apply the stated activation.
","Final answer: z = -1.3; a = 0.
","Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.
","Inputs = [1, 2].
","Weights = [0.2, -0.1]; bias = 0.
","Activation = sigmoid.
","z before activation and a after activation.
","The weighted sum combines the inputs. The output must then use the activation explicitly requested.
","z names the sum; a names the activated result. They are not always equal.
","Yaad rakhne ka cue: Sum alag, activation alag; activation lagana mat bhool.
","Step 1: multiply input 1 by its own weight.
","Step 2: multiply input 2 by its own weight.
","Step 3: add the input contributions.
","Step 4: add the bias once.
","Step 5: apply the stated activation.
","Final answer: z = 0; a = 0.5.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s6","search":"What one artificial neuron does **Definition.** An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output.\n\n### What goes into a neuron?\nAn input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive.\n\nThe **bias** is one extra adjustable number added to the total. The total before activation is called z or **preactivation**. The **activation function** is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the **activation**.\n\n**Yaad rakhne ka cue:** inputs × apne weights → products ka sum → ek bias → activation → output.\n\n### Mathematical model and notation\n\n\n| Symbol | Meaning and role |\n| xᵢ | Input component i; the neuron receives d such components |\n| wᵢ | Trainable weight multiplying that input component |\n| b | Trainable bias added once to this neuron's total |\n| z | Preactivation: the result before applying the activation function |\n| f | Chosen activation function, such as sigmoid or ReLU |\n| a | Activation: the neuron's output after applying f |\n\nThe dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs.\n\n### Why both a bias and an activation are needed\nThe bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin.\n\nThe activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values.\n\n\n\n### Worked numerical\n**Question.** A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output.\n\n\n\n### Given\n\nInputs = [2,1,3]; matching weights = [0.4,−0.2,0.5].\n\nBias = 0.1; activation = sigmoid.\n\n### To find\n\nFind the preactivation z, then the sigmoid output.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.\n\nz = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.\n\n\n\n**Yaad rakhne ka cue:** Minus-z ki e power, plus"},{"title":"Model answer: explain an artificial neuron","question":["Question. Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example.
"],"answer":["An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers.
","| Part | Meaning | What it does |
|---|---|---|
| x₁, x₂, … | Input features | Supply the information for this calculation |
| w₁, w₂, … | Weights | Control the contribution of each matching input |
| b | Bias | Shifts the weighted sum independently of the inputs |
| z | Net input | The number before activation |
| f | Activation rule | Transforms z, often nonlinearly |
| a | Activated output | The number sent onward |
Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.
","Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU.
","Preactivation z and activated output a.
","Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total.
","z means before activation; a means after activation.
","Yaad rakhne ka cue: Product, product, sum, bias, phir activation.
","Step 1: multiply the first pair.
","Step 2: multiply the second pair.
","Step 3: add both contributions.
","Step 4: add the bias once.
","Step 5: apply ReLU.
","Final answer: z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged.
","The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once.
"],"kind":"theory","id":"tee-c4-s7","search":"Model answer: explain an artificial neuron **Question.** Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example.\n\n**Model answer.**\nAn artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers.\n\n\n\n\n### Formula and symbols\n\n| Part | Meaning | What it does |\n| x₁, x₂, … | Input features | Supply the information for this calculation |\n| w₁, w₂, … | Weights | Control the contribution of each matching input |\n| b | Bias | Shifts the weighted sum independently of the inputs |\n| z | Net input | The number before activation |\n| f | Activation rule | Transforms z, often nonlinearly |\n| a | Activated output | The number sent onward |\n\n\n\n### Worked ReLU neuron example\n\n**Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.**\n\n\n\n### Given\n\nInputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU.\n\n### To find\n\nPreactivation z and activated output a.\n\n### Formula — which one and why\n\nEach input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total.\n\nz means before activation; a means after activation.\n\n\n\n\n\n**Yaad rakhne ka cue:** Product, product, sum, bias, phir activation.\n\n### Solution — step by step\n\n\n\n**Step 1: multiply the first pair.**\n\n\n\n\n\n\n\n**Step 2: multiply the second pair.**\n\n\n\n\n\n\n\n**Step 3: add both contributions.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply ReLU.**\n\n\n\n\n\n\n\n**Final answer:** z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged.\n\n\n\nThe weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once.\n"},{"title":"Perceptrons and the threshold","blocks":["A perceptron is a simple neuron used for classification with a threshold decision. If a question says output 1 for z ≥ 0 and 0 otherwise, then z = 0 gives 1. If it says output 1 for z > 0 and 0 otherwise, then z = 0 gives 0. That single equality sign can change an entire training table.
","Question. Implement binary AND with w₁ = w₂ = 0.6, b = −0.8 and output 1 when z ≥ 0.
","w₁ = w₂ = 0.6; bias b = −0.8.
","Binary inputs; output 1 when z ≥ 0.
","Calculate the output for 00,01,10,11 and show that the result is AND.
","Threshold logic. Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.
","step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.
","![\"\\operatorname{step}(z)=\\begin{cases}1&z\\ge0\\\\0&z<0\\end{cases}\"](\"assets/equations/ecb2a58d23e67edf.svg\")
Yaad rakhne ka cue: Har input row ka sum nikalo, phir threshold lagao.
","Step 1: multiply input 1 by its weight.
","![\"0.6(0)=0\"](\"assets/equations/b4145d1d293ebdf3.svg\")
Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","![\"s=0+0=0\"](\"assets/equations/e725afda84fc9ed3.svg\")
Step 4: add the bias once.
","![\"z=0+(-0.8)=-0.8\"](\"assets/equations/d15ca461c07dd695.svg\")
Step 5: apply the stated threshold.
","z = -0.8; output = 0 because output is 1 exactly when z ≥ 0.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","![\"0.6(1)=0.6\"](\"assets/equations/6eb084da38b1ada3.svg\")
Step 3: add the two products.
","![\"s=0+0.6=0.6\"](\"assets/equations/8544b9e49d354ea3.svg\")
Step 4: add the bias once.
","![\"z=0.6+(-0.8)=-0.2\"](\"assets/equations/f10c0127832440d1.svg\")
Step 5: apply the stated threshold.
","z = -0.2; output = 0 because output is 1 exactly when z ≥ 0.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","![\"s=0.6+0=0.6\"](\"assets/equations/5398d7a271c52472.svg\")
Step 4: add the bias once.
","Step 5: apply the stated threshold.
","z = -0.2; output = 0 because output is 1 exactly when z ≥ 0.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","![\"s=0.6+0.6=1.2\"](\"assets/equations/379a82d19e479abf.svg\")
Step 4: add the bias once.
","![\"z=1.2+(-0.8)=0.4\"](\"assets/equations/63a2763d058f9aec.svg\")
Step 5: apply the stated threshold.
","z = 0.4; output = 1 because output is 1 exactly when z ≥ 0.
","Final answer: outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND.
","| Inputs | Full net calculation | Net z | Output | AND target |
|---|---|---|---|---|
| 0, 0 | 0.6(0) + 0.6(0) − 0.8 | −0.8 | 0 | 0 |
| 0, 1 | 0.6(0) + 0.6(1) − 0.8 | −0.2 | 0 | 0 |
| 1, 0 | 0.6(1) + 0.6(0) − 0.8 | −0.2 | 0 | 0 |
| 1, 1 | 0.6(1) + 0.6(1) − 0.8 | 0.4 | 1 | 1 |
The neuron fires only when both inputs are 1. No weight update was requested, so this is a forward evaluation, not a training problem.
","For binary targets, a common learning rule is w ← w + η(t − a)x and b ← b + η(t − a). For bipolar targets −1 and +1, the class notes use a different rule: on a mismatch, w ← w + η tx and b ← b + ηt; otherwise keep them. The notes give output 0 at zero threshold, +1 above it and −1 below it. Never combine the activation from one convention with an unexplained update from another.
","Training table routine: copy current weights → calculate z → apply the stated threshold → compare output with target → compute every weight change → update bias → carry new values to the next row. Reset nothing between rows or epochs unless instructed.
","Question. A binary perceptron has x = [1,2], weights [-0.2, 0.1], bias 0 and target 1. Use output 1 for net ≥ 0 and learning rate 0.1. Perform one update.
","x = [1,2]; w = [-0.2, 0.1]; b = 0; target = 1; η = 0.1.
","Net, threshold output, correction and new weights/bias.
","This binary learning rule uses target minus output. Carry the same correction into every input weight and the bias.
","t is target; o is threshold output; Δ means the amount added to the current parameter.
","![\"\\Delta](\"assets/equations/3870c6199f4ce08e.svg\")
![\"\\Delta](\"assets/equations/998f813758f64544.svg\")
Yaad rakhne ka cue: Target minus output; correct prediction ho toh correction zero.
","Step 1: calculate the net input.
","![\"z=(-0.2)(1)+(0.1)(2)+0=0\"](\"assets/equations/ad825aadcb48f967.svg\")
Step 2: apply the threshold.
","Net = 0; output = 1 under the stated ≥ 0 rule.
","Step 3: calculate the correction.
","![\"t-o=1-1=0\"](\"assets/equations/ec562da655acc385.svg\")
Step 4: update weight 1.
","![\"w_1^{\\mathrm{new}}=-0.2+0.1(0)(1)=-0.2\"](\"assets/equations/35b1f47fa8ef32d6.svg\")
Step 5: update weight 2.
","![\"w_2^{\\mathrm{new}}=0.1+0.1(0)(2)=0.1\"](\"assets/equations/a1cc84ebf8de3e99.svg\")
Step 6: update the bias.
","![\"b_{\\mathrm{new}}=0+0.1(0)=0\"](\"assets/equations/920794cfde95f28a.svg\")
Final answer: New weights = [-0.2,0.1]; new bias = 0.
","Question. A binary perceptron has x = [1,2], weights [0.2, 0.1], bias 0 and target 0. Use output 1 for net ≥ 0 and learning rate 0.1. Perform one update.
","x = [1,2]; w = [0.2, 0.1]; b = 0; target = 0; η = 0.1.
","Net, threshold output, correction and new weights/bias.
","This binary learning rule uses target minus output. Carry the same correction into every input weight and the bias.
","t is target; o is threshold output; Δ means the amount added to the current parameter.
","Yaad rakhne ka cue: Target minus output; correct prediction ho toh correction zero.
","Step 1: calculate the net input.
","![\"z=(0.2)(1)+(0.1)(2)+0=0.4\"](\"assets/equations/cc07c4020befc673.svg\")
Step 2: apply the threshold.
","Net = 0.4; output = 1 under the stated ≥ 0 rule.
","Step 3: calculate the correction.
","![\"t-o=0-1=-1\"](\"assets/equations/84c2ca9c5cec76e2.svg\")
Step 4: update weight 1.
","![\"w_1^{\\mathrm{new}}=0.2+0.1(-1)(1)=0.1\"](\"assets/equations/e7f1347350326319.svg\")
Step 5: update weight 2.
","![\"w_2^{\\mathrm{new}}=0.1+0.1(-1)(2)=-0.1\"](\"assets/equations/ced16326d7d6ffec.svg\")
Step 6: update the bias.
","![\"b_{\\mathrm{new}}=0+0.1(-1)=-0.1\"](\"assets/equations/d6c4895766052790.svg\")
Final answer: New weights = [0.1,-0.1]; new bias = -0.1.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s8","search":"Perceptrons and the threshold A perceptron is a simple neuron used for classification with a threshold decision. If a question says output 1 for z ≥ 0 and 0 otherwise, then z = 0 gives 1. If it says output 1 for z > 0 and 0 otherwise, then z = 0 gives 0. That single equality sign can change an entire training table.\n\n\n\n**Question.** Implement binary AND with w₁ = w₂ = 0.6, b = −0.8 and output 1 when z ≥ 0.\n\n\n\n### Given\n\nw₁ = w₂ = 0.6; bias b = −0.8.\n\nBinary inputs; output 1 when z ≥ 0.\n\n### To find\n\nCalculate the output for 00,01,10,11 and show that the result is AND.\n\n### Formula — which one and why\n\n**Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.\n\nstep(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao.\n\n### Solution — step by step\n\n\n\n\n### Input (0, 0)\n\n**Step 1: multiply input 1 by its weight.**\n\n\n\n\n\n\n\n**Step 2: multiply input 2 by its weight.**\n\n\n\n\n\n\n\n**Step 3: add the two products.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply the stated threshold.**\n\nz = -0.8; output = 0 because output is 1 exactly when z ≥ 0.\n\n\n### Input (0, 1)\n\n**Step 1: multiply input 1 by its weight.**\n\n\n\n\n\n\n\n**Step 2: multiply input 2 by its weight.**\n\n\n\n\n\n\n\n**Step 3: add the two products.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply the stated threshold.**\n\nz = -0.2; output = 0 because output is 1 exactly when z ≥ 0.\n\n\n### Input (1, 0)\n\n**Step 1: multiply input 1 by its weight.**\n\n\n\n\n\n\n\n**Step 2: multiply input 2 by its weight.**\n\n\n\n\n\n\n\n**Step 3: add the two products.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply the stated threshold.**\n\nz = -0.2; output = 0 because output is 1 exactly when z ≥ 0.\n\n\n### Input (1, 1)\n\n**Step 1: multiply input 1 by its weight.**\n\n\n\n\n\n\n\n**Step 2: multiply input 2 by its weight.**\n\n\n\n\n\n\n\n**Step 3: add the two products.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply the stated threshold.**\n\nz = 0.4; output = 1 because output is 1 exactly when z ≥ 0.\n\n**Final answer:** outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND.\n\n### Summary of the four cases\n\n| Inputs | Full net calculation | Net z | Output | AND target |\n| 0, 0 | 0.6(0) + 0.6(0) − 0.8 | −0.8 | 0 | 0 |\n| 0, 1 | 0.6(0) + 0.6(1) − 0.8 | −0.2 | 0 | 0 |\n| 1, 0 | 0.6(1) + 0.6(0) − 0.8 | −0.2 | 0 | 0 |\n| 1, 1 | 0.6(1) + 0.6(1) − 0.8 | 0.4 | 1 | 1 |\nThe neuron fires only when both inputs are 1. No weight update was requested, so this is a forward evaluation, not a training problem.\nFor binary targets, a common learning rule is w ← w + η(t − a)x and b ← b + η(t − a). For bipolar targets −1 and +1, the class notes use a different rule: on a mismatch, w ← w + η tx and b ← b + ηt; otherwise keep them. The notes give output 0 at zero threshold, +1 above it and −1 below it. Never combine the activation from one convention with an unexplained update from another.\n**Training table routine: ** copy current weights → calculate z → apply the stated threshold → compare output with target → compute every weight change → update bias → carry new values to the next row. Reset nothing between rows or epochs unless instructed.\n\n\n\n### Binary perceptron: target 1\n\n**Question. A binary perceptron has x = [1,2], weights [-0.2, 0.1], "},{"title":"Why XOR needs more than one perceptron","blocks":["XOR outputs 1 for [0,1] and [1,0], but 0 for [0,0] and [1,1]. On a graph the positive points occupy opposite corners. A single straight line cannot put both positive points on one side and both negative points on the other. A single perceptron has only one linear decision boundary.
","A two-layer construction solves it. Let h₁ = step(x₁ + x₂ − 0.5), which is OR. Let h₂ = step(x₁ + x₂ − 1.5), which is AND. With step(z) = 1 for z ≥ 0, output XOR = step(h₁ − 2h₂ − 0.5). Output XNOR = step(−h₁ + 2h₂ + 0.5).
","| x₁ x₂ | h₁ OR | h₂ AND | XOR net | XOR | XNOR |
|---|---|---|---|---|---|
| 0 0 | 0 | 0 | −0.5 | 0 | 1 |
| 0 1 | 1 | 0 | 0.5 | 1 | 0 |
| 1 0 | 1 | 0 | 0.5 | 1 | 0 |
| 1 1 | 1 | 1 | −1.5 | 0 | 1 |
More layers help only if appropriate nonlinearities are used. Two purely affine layers collapse into one affine map: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + W₂b₁ + b₂. Affine means linear plus a bias.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s9","search":"Why XOR needs more than one perceptron XOR outputs 1 for [0,1] and [1,0], but 0 for [0,0] and [1,1]. On a graph the positive points occupy opposite corners. A single straight line cannot put both positive points on one side and both negative points on the other. A single perceptron has only one linear decision boundary.\n\nA two-layer construction solves it. Let h₁ = step(x₁ + x₂ − 0.5), which is OR. Let h₂ = step(x₁ + x₂ − 1.5), which is AND. With step(z) = 1 for z ≥ 0, output XOR = step(h₁ − 2h₂ − 0.5). Output XNOR = step(−h₁ + 2h₂ + 0.5).\n| x₁ x₂ | h₁ OR | h₂ AND | XOR net | XOR | XNOR |\n| 0 0 | 0 | 0 | −0.5 | 0 | 1 |\n| 0 1 | 1 | 0 | 0.5 | 1 | 0 |\n| 1 0 | 1 | 0 | 0.5 | 1 | 0 |\n| 1 1 | 1 | 1 | −1.5 | 0 | 1 |\nMore layers help only if appropriate nonlinearities are used. Two purely affine layers collapse into one affine map: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + W₂b₁ + b₂. Affine means linear plus a bias."},{"title":"Activation functions and how to calculate them","blocks":["A neuron first produces the weighted sum z. The activation function f is a rule that transforms that sum into the output a = f(z). Applying an activation does not change the stored weights; it calculates a new output from the current sum.
","If every layer only multiplies and adds, several layers still combine into a single affine transformation. A nonlinear activation lets the network represent more complicated relationships. The output activation must also suit the task: sigmoid for a binary-class probability, softmax for mutually exclusive class probabilities, and often a linear output for unrestricted regression.
","The activation value says what the neuron outputs. The activation derivative says how quickly that output changes when z changes, which is needed during backpropagation. For sigmoid at z = 0, the value is 0.5 but the derivative is 0.25. They serve different purposes.
","Yaad rakhne ka cue: z = activation se pehle ka sum; a = uske baad ka answer; f′ = answer kitni tezi se badlega.
","Definition. An activation function maps a neuron's preactivation to its output. Nonlinear activations allow a multilayer network to represent nonlinear functions; the output activation also encodes the prediction type. Without a nonlinear activation, a composition of affine layers remains affine.
","Sigmoid converts any finite real input into a value strictly between 0 and 1. At zero it is 0.5. It is useful for a binary-class probability or independent multilabel outputs. At very positive or negative inputs it saturates, so its derivative becomes small. Its derivative is at most 0.25.
","Tanh lies between −1 and +1 and is zero-centered. It is common in recurrent hidden states and LSTM candidates. It also saturates. “Zero-centered” does not mean “no vanishing gradients”.
","ReLU returns the input if positive and zero if negative. It is cheap to compute and has derivative 1 on its positive side. Units can become inactive on all relevant inputs, known as dying ReLU. At z = 0 the mathematical derivative is undefined; a convention such as 0 is used in implementations and this course’s exercises when needed.
","The small negative slope a lets some gradient through for negative inputs. At z = −2 and a = 0.01, output = −0.02. Do not take the absolute value. PReLU learns the negative slope; Leaky ReLU normally fixes it.
","Softmax acts on a group of class scores, called logits. All probabilities are positive and sum to 1. Subtracting the same maximum c from all logits leaves the result unchanged and avoids very large exponentials. It is suited to mutually exclusive multiclass output.
","Full softmax example. For [−1, 0, 1], exponentials are [0.367879, 1, 2.718282]. Sum = 4.086161. Divide each by that same sum: [0.090031, 0.244728, 0.665241]. Check their sum is approximately 1. Three separate sigmoids would give [0.268941, 0.5, 0.731059], which are different and do not sum to 1.
","Write sigmoid as a negative power. The outer derivative contributes −1, and differentiating the inner exponential contributes a second negative sign:
","Separate the expression into two factors:
","The second factor is 1−σ(z) because subtracting 1/(1+e⁻ᶻ) from 1 leaves e⁻ᶻ/(1+e⁻ᶻ). Once a forward pass has supplied σ(z), its derivative requires no further exponential evaluation.
","Use the quotient rule on the exponential definition. Here u = e²ᶻ−1 and v = e²ᶻ+1, with u′ = v′ = 2e²ᶻ.
","The identity 1−tanh²(z) gives the same expression, since the numerator becomes (e²ᶻ+1)²−(e²ᶻ−1)² = 4e²ᶻ. Therefore the backward-pass factor is 1−a² when a = tanh(z).
","For two affine layers without a nonlinear activation:
","The composition is still one affine transformation. A nonlinear activation between layers changes the family of functions the network can represent. This is the mathematical reason for using nonlinear hidden activations; merely adding affine layers does not provide this expressive advantage.
","Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.
","z = -2; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","Step 2: calculate the exponential.
","Step 3: add one in the denominator.
","Step 4: divide one by the complete denominator.
","Step 5: use the activated output in the slope.
","Final answer: a ≈ 0.119202922. Derivative ≈ 0.104993585.
","Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.
","z = 0; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","Step 2: calculate the exponential.
","Step 3: add one in the denominator.
","Step 4: divide one by the complete denominator.
","Step 5: use the activated output in the slope.
","Final answer: a ≈ 0.500000000. Derivative ≈ 0.250000000.
","Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.
","z = 2; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","Step 2: calculate the exponential.
","Step 3: add one in the denominator.
","Step 4: divide one by the complete denominator.
","Step 5: use the activated output in the slope.
","Final answer: a ≈ 0.880797078. Derivative ≈ 0.104993585.
","Question. For preactivation z = -2, calculate the tanh output and state its derivative.
","z = -2; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","Step 4: divide.
","Step 5: square the activated output for its slope.
","Final answer: a ≈ -0.964027580. Derivative ≈ 0.070650825.
","Question. For preactivation z = 0, calculate the tanh output and state its derivative.
","z = 0; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","Step 4: divide.
","Step 5: square the activated output for its slope.
","Final answer: a ≈ 0.000000000. Derivative ≈ 1.000000000.
","Question. For preactivation z = 2, calculate the tanh output and state its derivative.
","z = 2; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","Step 4: divide.
","Step 5: square the activated output for its slope.
","Final answer: a ≈ 0.964027580. Derivative ≈ 0.070650825.
","Question. For preactivation z = -2, calculate the ReLU output and state its derivative.
","z = -2; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","-2 is negative, so use the negative branch.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 0.000000000. Derivative ≈ 0.000000000.
","Question. For preactivation z = 0, calculate the ReLU output and state its derivative.
","z = 0; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","Both branches meet at zero. The value is zero, but the left and right slopes differ.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.
","Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.
","Question. For preactivation z = 2, calculate the ReLU output and state its derivative.
","z = 2; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","2 is positive, so retain it.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.
","Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = -2; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","-2 is negative, so use the negative branch.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ -0.200000000. Derivative ≈ 0.100000000.
","Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = 0; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","Both branches meet at zero. The value is zero, but the left and right slopes differ.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.
","Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.
","Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = 2; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","2 is positive, so retain it.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.
","Question. Find stable softmax probabilities for logits [0, 0, 0].
","Logits = [0, 0, 0]; there are three mutually exclusive classes.
","All three class probabilities.
","Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.
","A logit is a raw class score. c is the largest score; pᵢ is probability for class i.
","Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.
","Step 1: subtract the largest score from every entry.
","Step 2: exponentiate shifted entry 1.
","Step 3: exponentiate shifted entry 2.
","Step 4: exponentiate shifted entry 3.
","Step 5: add all exponentials into one denominator.
","Step 6: divide entry 1 by the same denominator.
","Step 7: divide entry 2 by the same denominator.
","Step 8: divide entry 3 by the same denominator.
","Final answer: Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding.
","Question. Find stable softmax probabilities for logits [1, 2, 3].
","Logits = [1, 2, 3]; there are three mutually exclusive classes.
","All three class probabilities.
","Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.
","A logit is a raw class score. c is the largest score; pᵢ is probability for class i.
","Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.
","Step 1: subtract the largest score from every entry.
","Step 2: exponentiate shifted entry 1.
","Step 3: exponentiate shifted entry 2.
","Step 4: exponentiate shifted entry 3.
","Step 5: add all exponentials into one denominator.
","Step 6: divide entry 1 by the same denominator.
","Step 7: divide entry 2 by the same denominator.
","Step 8: divide entry 3 by the same denominator.
","Final answer: Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s10","search":"Activation functions and how to calculate them \n### What an activation function does, and why it is needed\nA neuron first produces the weighted sum z. The activation function f is a rule that transforms that sum into the output a = f(z). Applying an activation does not change the stored weights; it calculates a new output from the current sum.\n\nIf every layer only multiplies and adds, several layers still combine into a single affine transformation. A nonlinear activation lets the network represent more complicated relationships. The output activation must also suit the task: sigmoid for a binary-class probability, softmax for mutually exclusive class probabilities, and often a linear output for unrestricted regression.\n\nThe **activation value** says what the neuron outputs. The **activation derivative** says how quickly that output changes when z changes, which is needed during backpropagation. For sigmoid at z = 0, the value is 0.5 but the derivative is 0.25. They serve different purposes.\n\n**Yaad rakhne ka cue:** z = activation se pehle ka sum; a = uske baad ka answer; f′ = answer kitni tezi se badlega.\n\n**Definition.** An activation function maps a neuron's preactivation to its output. Nonlinear activations allow a multilayer network to represent nonlinear functions; the output activation also encodes the prediction type. Without a nonlinear activation, a composition of affine layers remains affine.\n\n### Explanation and working\n\nSigmoid converts any finite real input into a value strictly between 0 and 1. At zero it is 0.5. It is useful for a binary-class probability or independent multilabel outputs. At very positive or negative inputs it saturates, so its derivative becomes small. Its derivative is at most 0.25.\n\n\n\n\nTanh lies between −1 and +1 and is zero-centered. It is common in recurrent hidden states and LSTM candidates. It also saturates. “Zero-centered” does not mean “no vanishing gradients”.\n\nReLU returns the input if positive and zero if negative. It is cheap to compute and has derivative 1 on its positive side. Units can become inactive on all relevant inputs, known as dying ReLU. At z = 0 the mathematical derivative is undefined; a convention such as 0 is used in implementations and this book’s exercises when needed.\n\nThe small negative slope a lets some gradient through for negative inputs. At z = −2 and a = 0.01, output = −0.02. Do not take the absolute value. PReLU learns the negative slope; Leaky ReLU normally fixes it.\n\nSoftmax acts on a group of class scores, called logits. All probabilities are positive and sum to 1. Subtracting the same maximum c from all logits leaves the result unchanged and avoids very large exponentials. It is suited to mutually exclusive multiclass output.\n**Full softmax example.** For [−1, 0, 1], exponentials are [0.367879, 1, 2.718282]. Sum = 4.086161. Divide each by that same sum: [0.090031, 0.244728, 0.665241]. Check their sum is approximately 1. Three separate sigmoids would give [0.268941, 0.5, 0.731059], which are different and do not sum to 1.\n\n### Derivation of the sigmoid derivative\nWrite sigmoid as a negative power. The outer derivative contributes −1, and differentiating the inner exponential contributes a second negative sign:\n\n\nSeparate the expression into two factors:\n\nThe second factor is 1−σ(z) because subtracting 1/(1+e⁻ᶻ) from 1 leaves e⁻ᶻ/(1+e⁻ᶻ). Once a forward pass has supplied σ(z), its derivative requires no further exponential evaluation.\n\n### Derivation of the tanh "},{"title":"Model answer: why a network needs nonlinear activation","question":["Question. Why are activation functions required in a multilayer neural network? Compare sigmoid, tanh and ReLU, including their limitations.
"],"answer":["An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary.
","For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second:
","This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent.
","| Activation | Output range | Useful property | Main limitation |
|---|---|---|---|
| Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small |
| Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients |
| ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient |
Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score.
","Exam conclusion: select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction.
"],"kind":"theory","id":"tee-c4-s11","search":"Model answer: why a network needs nonlinear activation **Question.** Why are activation functions required in a multilayer neural network? Compare sigmoid, tanh and ReLU, including their limitations.\n\n**Model answer.**\nAn activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary.\n\n\n\n\n### Why two linear layers are still linear in the input\nFor the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second:\n\nThis is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent.\n\n| Activation | Output range | Useful property | Main limitation |\n| Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small |\n| Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients |\n| ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient |\n\nSigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score.\n\n**Exam conclusion:** select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction.\n"},{"title":"Choosing a loss","blocks":["Definition. A loss function quantifies the discrepancy between a prediction and its target. The training objective aggregates per-example losses and may add regularization. The loss must be compatible with the task and output parameterization.
","Activation produces the prediction. Loss measures how wrong that prediction is. An optimizer uses gradients to update parameters. These are three different roles.
","Question. Targets are [2, 4], predictions are [1, 6]. Find MSE and MAE.
","Targets = [2,4]; predictions = [1,6]; n = 2.
","Find mean squared error and mean absolute error.
","Squared-error objectives. Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.
","ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.
","Yaad rakhne ka cue: Pehle error square; phir question ke hisaab se sum, half ya mean.
","Mean absolute error. Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes.
","n = number of predictions. |e| means the nonnegative magnitude of error e.
","Yaad rakhne ka cue: MAE mein magnitude; MSE mein square.
","Step 1: calculate prediction minus target.
","Step 2: square both errors.
","Step 3: add squares and divide by two observations.
","Step 4: take the absolute value of each original error.
","Step 5: add absolute errors and divide by two.
","Final answer: MSE = 2.5; MAE = 1.5. Squaring gives larger errors more influence; MAE is not differentiable at zero error.
","For binary y = 1, BCE becomes −ln(p). For y = 0, it becomes −ln(1 − p). For one-hot multiclass labels, CCE is simply minus the natural log of the predicted probability assigned to the correct class.
","Question. True class is Mild, encoded [0,1,0], and probabilities are [0.1,0.8,0.1]. Find CCE.
","True class = Mild; one-hot target = [0,1,0].
","Predicted probabilities = [0.1,0.8,0.1].
","Find the categorical cross-entropy.
","Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.
","yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.
","Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.
","Step 1: write one weighted log term for each class.
","Step 2: remove terms with zero target weight.
","Step 3: evaluate the natural logarithm.
","Step 4: apply the outer minus sign.
","Final answer: CCE ≈ 0.223144. Only the correct-class probability contributes for this one-hot target.
","| Task | Typical output | Typical loss |
|---|---|---|
| Unbounded numerical regression | One linear output | MSE or MAE |
| Binary classification | One sigmoid output | Binary cross-entropy |
| Exactly one of K classes | K softmax outputs | Categorical cross-entropy |
| Several independent labels | K sigmoid outputs | Binary cross-entropy per label |
A continuous rating restricted to 0–10 can use a linear output with a stated range policy, or 10 × sigmoid with an appropriate regression loss. If ratings are treated as discrete ordered categories, modelling choices differ. State the interpretation instead of selecting a loss merely from the number 10.
","For n included scalar predictions, let eᵢ = ŷᵢ−yᵢ. Three frequently used objectives are:
","Differentiating with respect to one prediction produces different factors:
","Use the exact objective in the question. The teacher's two-output backpropagation example uses half SSE. Do not automatically insert the MSE factor into that example.
","For a one-hot target, the likelihood of the observed class is the probability assigned to it. Maximizing that likelihood is equivalent to minimizing its negative logarithm. For independent training examples, multiplying probabilities becomes adding their logarithms. A low correct-class probability therefore produces a large loss.
","Let p = σ(z). First differentiate the loss with respect to p, holding y constant:
","Then multiply by the sigmoid derivative:
","Thus the logit delta is p−y for this paired output and loss. With sigmoid and half squared error it is instead (p−y)p(1−p). Confusing these two cases changes every subsequent gradient.
","For a normalized target vector, the derivative of softmax component pᵢ with respect to logit zⱼ is pᵢ times (δᵢⱼ−pⱼ), where δᵢⱼ is 1 if i = j and 0 otherwise. Therefore:
","This result assumes the target components sum to 1. For a mean loss over a batch, include the corresponding averaging factor as well.
","Question. Predictions are [1, 3] and targets are [2, 3]. Find SSE, half SSE and MSE.
","Predictions = [1, 3].
","Targets = [2, 3].
","Number of scalar predictions n = 2.
","SSE, half SSE and MSE separately.
","All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention.
","ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers.
","Yaad rakhne ka cue: Square sab mein; half mein 2, mean mein n se divide.
","Step 1: calculate error 1.
","Step 2: square error 1.
","Step 3: calculate error 2.
","Step 4: square error 2.
","Step 5: add squared errors.
","Step 6: divide by 2 for half SSE.
","Step 7: divide by the number of predictions for MSE.
","Final answer: SSE = 1; half SSE = 0.5; MSE ≈ 0.500000000.
","Question. Predictions are [2, 5, 1] and targets are [1, 3, 2]. Find SSE, half SSE and MSE.
","Predictions = [2, 5, 1].
","Targets = [1, 3, 2].
","Number of scalar predictions n = 3.
","SSE, half SSE and MSE separately.
","All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention.
","ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers.
","Yaad rakhne ka cue: Square sab mein; half mein 2, mean mein n se divide.
","Step 1: calculate error 1.
","Step 2: square error 1.
","Step 3: calculate error 2.
","Step 4: square error 2.
","Step 5: calculate error 3.
","Step 6: square error 3.
","Step 7: add squared errors.
","Step 8: divide by 2 for half SSE.
","Step 9: divide by the number of predictions for MSE.
","Final answer: SSE = 6; half SSE = 3; MSE ≈ 2.000000000.
","Question. Predictions are [4] and targets are [1]. Find SSE, half SSE and MSE.
","Predictions = [4].
","Targets = [1].
","Number of scalar predictions n = 1.
","SSE, half SSE and MSE separately.
","All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention.
","ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers.
","Yaad rakhne ka cue: Square sab mein; half mein 2, mean mein n se divide.
","Step 1: calculate error 1.
","Step 2: square error 1.
","Step 3: add squared errors.
","Step 4: divide by 2 for half SSE.
","Step 5: divide by the number of predictions for MSE.
","Final answer: SSE = 9; half SSE = 4.5; MSE ≈ 9.000000000.
","Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.
","Target y = 1; p = P(y=1) = 0.8; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","Final answer: Loss ≈ 0.223143551 nats.
","Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.
","Target y = 0; p = P(y=1) = 0.8; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","Final answer: Loss ≈ 1.609437912 nats.
","Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.
","Target y = 1; p = P(y=1) = 0.5; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","Final answer: Loss ≈ 0.693147181 nats.
","Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.
","p = 0.8; y = 1; single-example BCE; p = sigmoid(z).
","dL/dz, not dL/dp.
","Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.
","p is an activated probability; z is its raw score; the requested derivative is with respect to z.
","Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.
","Step 1: calculate the sigmoid slope.
","Step 2: differentiate BCE with respect to p.
","Step 3: multiply the two local derivatives.
","Final answer: dL/dz = -0.2. No further sigmoid derivative is needed.
","Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.
","p = 0.8; y = 0; single-example BCE; p = sigmoid(z).
","dL/dz, not dL/dp.
","Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.
","p is an activated probability; z is its raw score; the requested derivative is with respect to z.
","Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.
","Step 1: calculate the sigmoid slope.
","Step 2: differentiate BCE with respect to p.
","Step 3: multiply the two local derivatives.
","Final answer: dL/dz = 0.8. No further sigmoid derivative is needed.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s12","search":"Choosing a loss **Definition.** A loss function quantifies the discrepancy between a prediction and its target. The training objective aggregates per-example losses and may add regularization. The loss must be compatible with the task and output parameterization.\n\n### Explanation and working\nActivation produces the prediction. Loss measures how wrong that prediction is. An optimizer uses gradients to update parameters. These are three different roles.\n\n\n\n\n**Question.** Targets are [2, 4], predictions are [1, 6]. Find MSE and MAE.\n\n\n\n### Given\n\nTargets = [2,4]; predictions = [1,6]; n = 2.\n\n### To find\n\nFind mean squared error and mean absolute error.\n\n### Formula — which one and why\n\n**Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.\n\nŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean.\n\n**Mean absolute error.** Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes.\n\nn = number of predictions. |e| means the nonnegative magnitude of error e.\n\n\n\n**Yaad rakhne ka cue:** MAE mein magnitude; MSE mein square.\n\n### Solution — step by step\n\n\n\n\n**Step 1: calculate prediction minus target.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 2: square both errors.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 3: add squares and divide by two observations.**\n\n\n\n\n\n\n\n**Step 4: take the absolute value of each original error.**\n\n\n\n\n\n\n\n**Step 5: add absolute errors and divide by two.**\n\n\n\n\n\n\n\n**Final answer:** MSE = 2.5; MAE = 1.5. Squaring gives larger errors more influence; MAE is not differentiable at zero error.\n\nFor binary y = 1, BCE becomes −ln(p). For y = 0, it becomes −ln(1 − p). For one-hot multiclass labels, CCE is simply minus the natural log of the predicted probability assigned to the correct class.\n**Question.** True class is Mild, encoded [0,1,0], and probabilities are [0.1,0.8,0.1]. Find CCE.\n\n\n\n### Given\n\nTrue class = Mild; one-hot target = [0,1,0].\n\nPredicted probabilities = [0.1,0.8,0.1].\n\n### To find\n\nFind the categorical cross-entropy.\n\n### Formula — which one and why\n\n**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.\n\nyⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.\n\n\n\n\n\n**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.\n\n### Solution — step by step\n\n\n\n\n**Step 1: write one weighted log term for each class.**\n\n\n\n\n\n\n\n**Step 2: remove terms with zero target weight.**\n\n\n\n\n\n\n\n**Step 3: evaluate the natural logarithm.**\n\n\n\n\n\n\n\n**Step 4: apply the outer minus sign.**\n\n\n\n\n\n\n\n**Final answer:** CCE ≈ 0.223144. Only the correct-class probability contributes for this one-hot target.\n| Task | Typical output | Typical loss |\n| Unbounded numerical regression | One linear output | MSE or MAE |\n| Binary classification | One sigmoid output | Binary cross-entropy |\n| Exactly one of K classes | K softmax outputs | Categorical cross-entropy |\n| Several independent labels | K sigmoid outputs | Binary cross-entropy per label |\nA continuous rating restricted to 0–10 can us"},{"title":"Forward propagation through multiple neurons","blocks":["Definitions. A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations.
","The input layer supplies the feature vector. A hidden layer learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The output layer produces the prediction in the form needed by the task.
","In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections.
","Let a⁽⁰⁾ = x. For layer l:
","If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j.
","For one hidden layer, these computations compose to give:
","The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer.
","Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture.
","Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.
A feedforward network receives x = [1, 0]ᵀ.
","Its hidden-layer weights are W¹ = and biases are b¹ = [0.1, 0.2]ᵀ.
The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1.
","Use ReLU in both layers and one row per receiving neuron.
","Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Step 1: identify the matrix dimensions. W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output.
","Step 2: calculate the first hidden preactivation. Multiply the first row by x, then add the first hidden bias.
","Step 3: calculate the second hidden preactivation.
","Step 4: activate the hidden layer. Both values are positive, so ReLU retains them.
","Step 5: calculate the output preactivation. Use the activated hidden values as this layer's inputs.
","Step 6: apply the output activation.
","Final answer: hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s13","search":"Forward propagation through multiple neurons **Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations.\n\n### Input, hidden and output layers\nThe **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task.\n\nIn a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections.\n\n\n\n### General layer equations and dimensions\nLet a⁽⁰⁾ = x. For layer l:\n\n\nIf the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j.\n\nFor one hidden layer, these computations compose to give:\n\n\nThe hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer.\n\n### Forward propagation versus training\nForward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture.\n\n### Complete two-layer numerical\n**Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.**\n\n\n\n### Given\n\nA feedforward network receives x = [1, 0]ᵀ.\n\nIts hidden-layer weights are W¹ = and biases are b¹ = [0.1, 0.2]ᵀ.\n\nThe output weights are W² = [0.5, 0.6] and output bias is b² = 0.1.\n\nUse ReLU in both layers and one row per receiving neuron.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota"},{"title":"Backpropagation with one hidden neuron","blocks":["Definition. Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters.
","Consider one input, one hidden neuron and one output neuron:
","For the half squared loss used in this example:
","Define a neuron's delta as the derivative of the loss with respect to its preactivation, not with respect to its output:
","The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂:
","To reach the hidden neuron, continue backwards through the old output weight and the hidden activation:
","The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass.
","Question. x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1.
","x = 2, y = 1.
","w₁ = 0.5, b₁ = 0.1.
","w₂ = 0.8, b₂ = 0.2.
","Both neurons use ReLU.
","Use half squared loss.
","Find the output and all four parameter gradients, then update all four parameters using η = 0.1.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1: forward propagation through the hidden neuron.
","Step 2: forward propagation through the output neuron.
","ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here.
","Step 3: calculate the loss and output delta.
","Both preactivations are positive, so both ReLU derivatives are 1.
","Step 4: calculate the output-layer parameter gradients.
","Step 5: propagate the derivative to the hidden neuron.
","Use w₂ = 0.8, its value during the forward pass:
","Step 6: update each parameter using its own gradient.
","| Parameter | Gradient from the original forward pass | Updated value |
|---|---|---|
| w₁ | 0.128 | 0.4872 |
| b₁ | 0.064 | 0.0936 |
| w₂ | 0.088 | 0.7912 |
| b₂ | 0.08 | 0.192 |
Final answer: the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s14","search":"Backpropagation with one hidden neuron \n**Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters.\n\n### Network equations and the meaning of delta\nConsider one input, one hidden neuron and one output neuron:\n\n\nFor the half squared loss used in this example:\n\nDefine a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output:\n\nThe output-weight derivative contains the hidden activation because zₒ = w₂h + b₂:\n\nTo reach the hidden neuron, continue backwards through the old output weight and the hidden activation:\n\n\nThe bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass.\n\n### Complete worked numerical\n**Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1.\n\n\n\n### Given\n\nx = 2, y = 1.\n\nw₁ = 0.5, b₁ = 0.1.\n\nw₂ = 0.8, b₂ = 0.2.\n\nBoth neurons use ReLU.\n\nUse half squared loss.\n\nFind the output and all four parameter gradients, then update all four parameters using η = 0.1.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n\n\n**Step 1: forward propagation through the hidden neuron.**\n\n\n\n**Step 2: forward propagation through the output neuron.**\n\n\nReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here.\n\n**Step 3: calculate the loss and output delta.**\n\nBoth preactivations are positive, so bot"},{"title":"More outputs and vectorized backpropagation","blocks":["Definition. Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path.
","If hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives:
","For any number of connected outputs, sum their contributions and then differentiate the hidden activation:
","Use the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss.
","Use column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes half sum of squared errors and an elementwise output activation f₂:
","Here ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an outer product: every receiving delta is multiplied by every sending activation.
","| Quantity | Shape | Reason |
|---|---|---|
| W₁ and its gradient | m × d | m receiving hidden neurons, d inputs |
| W₂ and its gradient | k × m | k receiving output neurons, m hidden activations |
| δ₂ | k × 1 | One derivative per output preactivation |
| W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit |
| δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ |
For MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target:
","That simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required.
","For a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s15","search":"More outputs and vectorized backpropagation \n**Definition.** Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path.\n\n### Why the hidden derivative is a sum\nIf hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives:\n\nFor any number of connected outputs, sum their contributions and then differentiate the hidden activation:\n\nUse the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss.\n\n\n\n### Matrix equations for one training sample\nUse column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes **half sum of squared errors** and an elementwise output activation f₂:\n\n\n\n\n\nHere ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an **outer product**: every receiving delta is multiplied by every sending activation.\n\n| Quantity | Shape | Reason |\n| W₁ and its gradient | m × d | m receiving hidden neurons, d inputs |\n| W₂ and its gradient | k × m | k receiving output neurons, m hidden activations |\n| δ₂ | k × 1 | One derivative per output preactivation |\n| W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit |\n| δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ |\n\n### How the loss changes the output delta\nFor MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target:\n\nThat simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required.\n\nFor a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question.\n\n"},{"title":"Model answer: forward propagation, backpropagation and updating","question":["Question. Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations.
"],"answer":["Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs.
","| Operation | Direction or action | Result |
|---|---|---|
| Forward propagation | Inputs move through successive layers | Predictions and intermediate activations |
| Loss calculation | Compare predictions with targets | A scalar objective for the example or batch |
| Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters |
| Optimizer update | Use those gradients to change parameters | A new set of weights and biases |
At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next.
","Use the loss specified by the task. For one regression output, half squared loss is one possible choice:
","Here a is the prediction and y is the target. Other stated losses produce different derivatives.
","Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation.
","Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it.
","Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations.
"],"kind":"theory","id":"tee-c4-s16","search":"Model answer: forward propagation, backpropagation and updating **Question.** Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations.\n\n**Model answer.**\nTraining repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs.\n\n\n\n\n| Operation | Direction or action | Result |\n| Forward propagation | Inputs move through successive layers | Predictions and intermediate activations |\n| Loss calculation | Compare predictions with targets | A scalar objective for the example or batch |\n| Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters |\n| Optimizer update | Use those gradients to change parameters | A new set of weights and biases |\n\n### 1. Forward propagation\nAt each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next.\n\n### 2. Loss\nUse the loss specified by the task. For one regression output, half squared loss is one possible choice:\n\nHere a is the prediction and y is the target. Other stated losses produce different derivatives.\n### 3. Backpropagation\nMultiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation.\n\nHere δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it.\n### 4. Update and repeat\n\nCompute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations.\n"},{"title":"Counting parameters in a dense network","blocks":["Definition. A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases.
","For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters:
","Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram.
","Question. A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters.
","Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs.
","Every hidden/output neuron has one bias.
","Count weights and biases at each receiving layer, then the total.
","Dense-layer parameters. Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.
","n_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.
","Yaad rakhne ka cue: Har output ke liye saare inputs plus ek bias.
","Step 1: identify the consecutive layer widths.
","There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters.
","Step 2: count weights and biases for each transformation.
","| Transformation | Weights | Biases | Layer total |
|---|---|---|---|
| Input → hidden 1 | 10 × 5 = 50 | 5 | 55 |
| Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 |
| Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 |
| Hidden 3 → output | 5 × 4 = 20 | 4 | 24 |
Step 3: add the layer totals.
","Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139.
","Final answer: 139 trainable parameters. If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s17","search":"Counting parameters in a dense network \n**Definition.** A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases.\n\n### Deriving the parameter count\nFor m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters:\n\nActivation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram.\n\n\n\n### Complete numerical\n**Question.** A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters.\n\n\n\n### Given\n\nLayer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs.\n\nEvery hidden/output neuron has one bias.\n\n### To find\n\nCount weights and biases at each receiving layer, then the total.\n\n### Formula — which one and why\n\n**Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.\n\nn_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.\n\n\n\n**Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: identify the consecutive layer widths.**\n\nThere are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters.\n\n**Step 2: count weights and biases for each transformation.**\n| Transformation | Weights | Biases | Layer total |\n| Input → hidden 1 | 10 × 5 = 50 | 5 | 55 |\n| Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 |\n| Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 |\n| Hidden 3 → output | 5 × 4 = 20 | 4 | 24 |\n\n**Step 3: add the layer totals.**\n\nEquivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139.\n\n**Final answer: 139 trainable parameters.** If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network.\n\n"},{"title":"What you should understand before optimization","blocks":["You should be able to explain why a target and prediction differ, compute a weighted sum, choose an activation, measure a loss, find a one-neuron gradient and update a weight. If these steps are weak, repeat the first training example and the ReLU backprop example. They contain the entire calculation pattern that the next chapters extend.
","Question. A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update.
","x = 2; w = 0; b = 0; y = 1.
","Sigmoid output; half squared loss; η = 0.1.
","Find the weight-gradient sign, exact weight update and resulting weight.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 forward: net = 0(2) + 0 = 0; sigmoid output = 0.5.
","Step 2 backward: prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125.
","Step 3 gradients and update: weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s19","search":"Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update.\n\n\n\n### Given\n\nx = 2; w = 0; b = 0; y = 1.\n\nSigmoid output; half squared loss; η = 0.1.\n\n### To find\n\nFind the weight-gradient sign, exact weight update and resulting weight.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5.\n\n\n\n**Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125.\n**Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree.\n"},{"title":"Cross-entropy for probability distributions","blocks":["Definition. For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities.
","Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms.
","Natural logarithms produce nats; base-2 logarithms produce bits. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2:
","For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities.
","Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.
","A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5].
","Calculate cross-entropy using natural logarithms and using base-2 logarithms.
","Find cross-entropy in the requested units.
","Cross-entropy between distributions. Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute.
","pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits.
","Yaad rakhne ka cue: Target probability batati hai kis log-loss ka kitna hissa lena hai.
","Step 1: write the two terms. Both target entries are nonzero, so neither term can be discarded.
","Step 2: calculate the logarithms.
","Step 3: weight each negative logarithm by its target probability.
","Step 4: add the contributions, retaining calculator precision.
","Step 5: repeat with base 2. Since 2⁻¹ = 0.5, log₂(0.5) = −1.
","Final answer: 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale.
","These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio.
","For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus.
","Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.
","For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.
","Find entropy and KL divergence in nats.
","Entropy and relative entropy. Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p.
","p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence.
","Yaad rakhne ka cue: KL = cross-entropy minus target ki own entropy.
","Step 1: calculate the target entropy.
","Step 2: subtract target entropy from cross-entropy.
","Final answer: H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s20","search":"Cross-entropy for probability distributions **Definition.** For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities.\n\nHere pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms.\n\n### Logarithm base and units\nNatural logarithms produce **nats**; base-2 logarithms produce **bits**. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2:\n\nFor a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities.\n\n### A target distribution with two nonzero entries\n**Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.**\n\n\n\n### Given\n\nA two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5].\n\nCalculate cross-entropy using natural logarithms and using base-2 logarithms.\n\n### To find\n\nFind cross-entropy in the requested units.\n\n### Formula — which one and why\n\n**Cross-entropy between distributions.** Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute.\n\npᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits.\n\n\n\n\n\n**Yaad rakhne ka cue:** Target probability batati hai kis log-loss ka kitna hissa lena hai.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: write the two terms.** Both target entries are nonzero, so neither term can be discarded.\n\n\n**Step 2: calculate the logarithms.**\n\n\n**Step 3: weight each negative logarithm by its target probability.**\n\n\n\n**Step 4: add the contributions, retaining calculator precision.**\n\n\n**Step 5: repeat with base 2.** Since 2⁻¹ = 0.5, log₂(0.5) = −1.\n\n**Final answer:** 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale.\n\n### Cross-entropy, entropy and relative entropy\nThese names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio.\n\n\nFor fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus.\n\n**Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.**\n\n\n\n### Given\n\nFor the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.\n\n### To find\n\nFind entropy and KL divergence in nats.\n\n### Formula — which one and why\n\n**Entropy and relative entropy.** Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p.\n\np = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence.\n\n\n\n\n\n**Yaad rakhne ka cue:** KL"},{"title":"Biological neurons and the McCulloch-Pitts model","blocks":["Definition. An artificial neuron is a simplified computational model inspired by the collection, combination and transmission of signals in biological neurons. This analogy motivates weighted inputs and an output response; it is not a full simulation of a living neuron.
","| Biological component | Simplified artificial counterpart | Function in the analogy |
|---|---|---|
| Dendrites | Input connections | Receive incoming signals |
| Synapses | Connection weights | Influence how strongly each incoming signal contributes |
| Cell body or soma | Aggregation stage | Combines the weighted signals |
| Firing behavior | Threshold or activation function | Determines the model's output response |
| Axon | Output connection | Passes the resulting signal onward |
For an artificial threshold unit, a threshold θ on the weighted sum can be written as a bias b = −θ, followed by a threshold at zero. The equality convention must be preserved.
","![\"\\sum_i](\"assets/equations/bd6e61f8065942de.svg\")
The McCulloch–Pitts model is an early binary threshold model. Its logical connections and threshold are prescribed to implement a desired behavior; it has no data-driven weight-update procedure. The original formulation also distinguishes excitatory inputs from inhibitory inputs, which can prevent firing. In elementary logic-gate examples using only excitatory inputs, a threshold on their sum is sufficient.
","A perceptron introduces adjustable connection weights and a learning rule. It calculates a weighted sum, applies a threshold and updates parameters when its classification disagrees with the target. Both single threshold units have a linear decision boundary; neither single unit solves XOR in the original two-input space.
","Question. Implement AND and OR for binary inputs x₁, x₂ using the sum x₁ + x₂ and output 1 when the sum is at least θ. Use θ = 2 for AND and θ = 1 for OR.
","Implement AND and OR for binary inputs x₁, x₂ using the sum x₁ + x₂ and output 1 when the sum is at least θ.
","Use θ = 2 for AND and θ = 1 for OR.
","Write every row of the truth table and the requested network output.
","Threshold logic. Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.
","step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.
","Yaad rakhne ka cue: Har input row ka sum nikalo, phir threshold lagao.
","Calculate the sum once for each input pair, then compare it with the relevant threshold.
","Step 1: add the inputs.
","Step 2: compare with AND threshold 2.
","0 ≥ 2 is false; AND output = 0.
","Step 3: compare with OR threshold 1.
","0 ≥ 1 is false; OR output = 0.
","Step 1: add the inputs.
","![\"s=0+1=1\"](\"assets/equations/4ae62ee9eceeaffe.svg\")
Step 2: compare with AND threshold 2.
","1 ≥ 2 is false; AND output = 0.
","Step 3: compare with OR threshold 1.
","1 ≥ 1 is true; OR output = 1.
","Step 1: add the inputs.
","![\"s=1+0=1\"](\"assets/equations/2707337aabb06170.svg\")
Step 2: compare with AND threshold 2.
","1 ≥ 2 is false; AND output = 0.
","Step 3: compare with OR threshold 1.
","1 ≥ 1 is true; OR output = 1.
","Step 1: add the inputs.
","![\"s=1+1=2\"](\"assets/equations/1c8553f39604f98c.svg\")
Step 2: compare with AND threshold 2.
","2 ≥ 2 is true; AND output = 1.
","Step 3: compare with OR threshold 1.
","2 ≥ 1 is true; OR output = 1.
","| x₁ | x₂ | x₁ + x₂ | AND: sum ≥ 2 | OR: sum ≥ 1 |
|---|---|---|---|---|
| 0 | 0 | 0 + 0 = 0 | 0 | 0 |
| 0 | 1 | 0 + 1 = 1 | 0 | 1 |
| 1 | 0 | 1 + 0 = 1 | 0 | 1 |
| 1 | 1 | 1 + 1 = 2 | 1 | 1 |
The parameters have been chosen to implement the truth tables. This calculation is a logic-gate evaluation, not a perceptron training epoch.
"],"question":[],"answer":[],"kind":"read","id":"tee-c4-s21","search":"Biological neurons and the McCulloch-Pitts model **Definition.** An artificial neuron is a simplified computational model inspired by the collection, combination and transmission of signals in biological neurons. This analogy motivates weighted inputs and an output response; it is not a full simulation of a living neuron.\n\n### Biological and artificial components\n| Biological component | Simplified artificial counterpart | Function in the analogy |\n| Dendrites | Input connections | Receive incoming signals |\n| Synapses | Connection weights | Influence how strongly each incoming signal contributes |\n| Cell body or soma | Aggregation stage | Combines the weighted signals |\n| Firing behavior | Threshold or activation function | Determines the model's output response |\n| Axon | Output connection | Passes the resulting signal onward |\n\nFor an artificial threshold unit, a threshold θ on the weighted sum can be written as a bias b = −θ, followed by a threshold at zero. The equality convention must be preserved.\n\n\n### McCulloch–Pitts model and perceptron\nThe McCulloch–Pitts model is an early binary threshold model. Its logical connections and threshold are prescribed to implement a desired behavior; it has no data-driven weight-update procedure. The original formulation also distinguishes excitatory inputs from inhibitory inputs, which can prevent firing. In elementary logic-gate examples using only excitatory inputs, a threshold on their sum is sufficient.\n\nA perceptron introduces adjustable connection weights and a learning rule. It calculates a weighted sum, applies a threshold and updates parameters when its classification disagrees with the target. Both single threshold units have a linear decision boundary; neither single unit solves XOR in the original two-input space.\n\n**Question. Implement AND and OR for binary inputs x₁, x₂ using the sum x₁ + x₂ and output 1 when the sum is at least θ. Use θ = 2 for AND and θ = 1 for OR.**\n\n\n\n### Given\n\nImplement AND and OR for binary inputs x₁, x₂ using the sum x₁ + x₂ and output 1 when the sum is at least θ.\n\nUse θ = 2 for AND and θ = 1 for OR.\n\n### To find\n\nWrite every row of the truth table and the requested network output.\n\n### Formula — which one and why\n\n**Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.\n\nstep(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao.\n\n### Solution — step by step\n\n\n\n\n\nCalculate the sum once for each input pair, then compare it with the relevant threshold.\n### Binary input (0,0)\n\n**Step 1: add the inputs.**\n\n\n\n\n\n\n\n**Step 2: compare with AND threshold 2.**\n\n0 ≥ 2 is false; AND output = 0.\n\n**Step 3: compare with OR threshold 1.**\n\n0 ≥ 1 is false; OR output = 0.\n\n\n### Binary input (0,1)\n\n**Step 1: add the inputs.**\n\n\n\n\n\n\n\n**Step 2: compare with AND threshold 2.**\n\n1 ≥ 2 is false; AND output = 0.\n\n**Step 3: compare with OR threshold 1.**\n\n1 ≥ 1 is true; OR output = 1.\n\n\n### Binary input (1,0)\n\n**Step 1: add the inputs.**\n\n\n\n\n\n\n\n**Step 2: compare with AND threshold 2.**\n\n1 ≥ 2 is false; AND output = 0.\n\n**Step 3: compare with OR threshold 1.**\n\n1 ≥ 1 is true; OR output = 1.\n\n\n### Binary input (1,1)\n\n**Step 1: add the inputs.**\n\n\n\n\n\n\n\n**Step 2: compare with AND threshold 2.**\n\n2 ≥ 2 is tru"}]},{"n":5,"title":"Optimization algorithms","sections":[{"title":"Reading an optimizer formula without getting lost","blocks":["An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated.
","At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position.
","η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition.
","An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares.
","If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square.
","The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum.
","For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g².
","ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s0","search":"Reading an optimizer formula without getting lost An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated.\n\n\n\n### Old, current and new\nAt update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position.\n\nη, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition.\n### What a stored state means\nAn optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares.\n### A weighted average from ordinary arithmetic\nIf the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square.\n\n\nThe coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum.\n### Squared memory and signed direction\nFor gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g².\n### Epsilon and hats\nε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second.\n"},{"title":"Optimizer notation: old value, memory, new value","blocks":["θ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries.
","At time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values.
","If the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead.
","For θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64.
","Both 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square.
","If s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero.
","Write one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1.
","The worked examples below use this exact order, with every state shown.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s1","search":"Optimizer notation: old value, memory, new value ### 1. A parameter and a stored state are different\nθ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries.\n\n\n\nAt time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values.\n\n### 2. A supplied gradient versus a supplied objective\nIf the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead.\nFor θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64.\n\n### 3. A squared gradient keeps size and loses sign\nBoth 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square.\n\n### 4. Square root and epsilon belong to the denominator\nIf s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero.\n\n### 5. A table prevents mixing iterations\nWrite one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1.\nThe worked examples below use this exact order, with every state shown.\n"},{"title":"What an optimizer is trying to do","blocks":["Definition. Optimization is the process of selecting model parameters to minimize a scalar objective. In neural-network training, a gradient-based optimizer updates weights and biases using derivatives of the training objective.
","For empirical risk with an optional regularizer:
","The loss value indicates the current objective. Its gradient indicates local sensitivity to parameters. Backpropagation computes this gradient efficiently; the optimizer determines how the parameters are updated from it. A gradient and a loss value cannot be interchanged in the update equation.
","For a small displacement Δθ, first-order Taylor expansion approximates the objective by:
","Choose Δθ = −η∇J(θ), with η > 0. Substitution gives:
","The squared norm is nonnegative. Therefore the negative-gradient direction is locally a descent direction when the gradient is nonzero. The approximation explains the sign; it does not guarantee that an arbitrarily large finite step decreases loss.
","A local minimum has objective value no larger than nearby points. A global minimum is minimal over the whole feasible domain. At a saddle point, the objective increases in some directions and decreases in others. A small gradient alone does not establish a minimum: a stationary point can also be a maximum or saddle.
","Neural-network objectives are generally nonconvex. Training can encounter flat regions, ill-conditioned valleys, noisy gradient estimates and poor step sizes. Momentum modifies the update direction using previous steps; adaptive methods rescale parameter-specific updates. These methods improve particular optimization behavior, without guaranteeing the global optimum or best test performance.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s2","search":"What an optimizer is trying to do **Definition.** Optimization is the process of selecting model parameters to minimize a scalar objective. In neural-network training, a gradient-based optimizer updates weights and biases using derivatives of the training objective.\n\n### Objective and gradient\nFor empirical risk with an optional regularizer:\n\nThe loss value indicates the current objective. Its gradient indicates local sensitivity to parameters. Backpropagation computes this gradient efficiently; the optimizer determines how the parameters are updated from it. A gradient and a loss value cannot be interchanged in the update equation.\n\n### Why gradient descent subtracts the gradient\nFor a small displacement Δθ, first-order Taylor expansion approximates the objective by:\n\nChoose Δθ = −η∇J(θ), with η > 0. Substitution gives:\n\nThe squared norm is nonnegative. Therefore the negative-gradient direction is locally a descent direction when the gradient is nonzero. The approximation explains the sign; it does not guarantee that an arbitrarily large finite step decreases loss.\n\n\n\n### Local minima, global minima and saddle points\nA local minimum has objective value no larger than nearby points. A global minimum is minimal over the whole feasible domain. At a saddle point, the objective increases in some directions and decreases in others. A small gradient alone does not establish a minimum: a stationary point can also be a maximum or saddle.\n\nNeural-network objectives are generally nonconvex. Training can encounter flat regions, ill-conditioned valleys, noisy gradient estimates and poor step sizes. Momentum modifies the update direction using previous steps; adaptive methods rescale parameter-specific updates. These methods improve particular optimization behavior, without guaranteeing the global optimum or best test performance.\n"},{"title":"The ordinary gradient descent update","blocks":["Definition. Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate.
","Here update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent.
","Read it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction.
","Question. J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations.
","J(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1.
","Differentiate J and perform two gradient-descent iterations.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","Step 1. Differentiate. g(θ) = 0.7 × 3 × θ² = 2.1 θ².
","Step 2. First update. g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79.
","Step 3. Second gradient. Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061.
","Step 4. Second update. θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = 0.658939.
","The paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum.
","Common error: using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s3","search":"The ordinary gradient descent update **Definition.** Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate.\n\n### Explanation and working\n\n\nHere update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent.\nRead it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction.\n\n\n\n**Question.** J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations.\n\n\n\n### Given\n\nJ(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1.\n\n### To find\n\nDifferentiate J and perform two gradient-descent iterations.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1. Differentiate.** g(θ) = 0.7 × 3 × θ² = 2.1 θ².\n**Step 2. First update.** g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79.\n**Step 3. Second gradient.** Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061.\n**Step 4. Second update.** θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = **0.658939**.\nThe paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum.\n**Common error: ** using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients."},{"title":"Batch stochastic and mini batch gradient descent","blocks":["Definition. Batch, stochastic and mini-batch gradient descent differ in the subset of training examples used to estimate the gradient for each parameter update. Their update equation is the same; their gradient estimates and update frequency differ.
","All three use the same update idea. They differ in how much data contributes to one gradient.
","| Method | Data for one update | Updates in one epoch with N samples | Typical behaviour |
|---|---|---|---|
| Batch GD | All N samples | 1 | Stable full-data gradient; costly updates |
| Stochastic GD | One sample | N | Frequent noisy updates |
| Mini-batch GD | B samples | Ceiling of N/B if final partial batch is retained | Efficient parallel calculation with moderate noise |
Question. N = 1,000 and B = 100. Find updates per epoch.
","N = 1,000 examples; mini-batch size B = 100; one epoch.
","Count full-batch, stochastic and mini-batch updates per epoch.
","Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.
","N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.
","Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.
","Step 1: count full-batch updates.
","All 1,000 samples form one group, so one epoch has 1 update.
","Step 2: count stochastic updates.
","One sample per update gives 1,000 updates per epoch.
","Step 3: count mini-batches.
","Final answer: batch = 1; SGD = 1,000; mini-batch = 10 updates per epoch.
","Question. A dataset contains 105 samples and the mini-batch size is 20. Keeping the incomplete last batch, find the updates per epoch.
","N = 105; B = 20; retain the final incomplete batch.
","The number of updates per epoch.
","Every processed group causes one update. A remaining nonempty group also counts.
","The ceiling symbol rounds upward to the next integer.
","Yaad rakhne ka cue: Bacha hua batch process kiya toh ek aur update.
","Step 1: count complete batches.
","Step 2: count remaining examples.
","Step 3: include the nonempty last batch.
","Final answer: 6 updates. If a question explicitly drops the final incomplete batch, it instead gives 5.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s4","search":"Batch stochastic and mini batch gradient descent **Definition.** Batch, stochastic and mini-batch gradient descent differ in the subset of training examples used to estimate the gradient for each parameter update. Their update equation is the same; their gradient estimates and update frequency differ.\n\n### Explanation and working\nAll three use the same update idea. They differ in how much data contributes to one gradient.\n\n\n\n| Method | Data for one update | Updates in one epoch with N samples | Typical behaviour |\n| Batch GD | All N samples | 1 | Stable full-data gradient; costly updates |\n| Stochastic GD | One sample | N | Frequent noisy updates |\n| Mini-batch GD | B samples | Ceiling of N/B if final partial batch is retained | Efficient parallel calculation with moderate noise |\n**Question.** N = 1,000 and B = 100. Find updates per epoch.\n\n\n\n### Given\n\nN = 1,000 examples; mini-batch size B = 100; one epoch.\n\n### To find\n\nCount full-batch, stochastic and mini-batch updates per epoch.\n\n### Formula — which one and why\n\n**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.\n\nN = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.\n\n### Solution — step by step\n\n\n\n\n**Step 1: count full-batch updates.**\n\nAll 1,000 samples form one group, so one epoch has 1 update.\n\n**Step 2: count stochastic updates.**\n\nOne sample per update gives 1,000 updates per epoch.\n\n**Step 3: count mini-batches.**\n\n\n\n\n\n\n\n**Final answer:** batch = 1; SGD = 1,000; mini-batch = 10 updates per epoch.\n\n\n### Mini-batch calculation with a remainder\n\n**Question. A dataset contains 105 samples and the mini-batch size is 20. Keeping the incomplete last batch, find the updates per epoch.**\n\n\n\n### Given\n\nN = 105; B = 20; retain the final incomplete batch.\n\n### To find\n\nThe number of updates per epoch.\n\n### Formula — which one and why\n\nEvery processed group causes one update. A remaining nonempty group also counts.\n\nThe ceiling symbol rounds upward to the next integer.\n\n\n\n**Yaad rakhne ka cue:** Bacha hua batch process kiya toh ek aur update.\n\n### Solution — step by step\n\n\n\n**Step 1: count complete batches.**\n\n\n\n\n\n\n\n**Step 2: count remaining examples.**\n\n\n\n\n\n\n\n**Step 3: include the nonempty last batch.**\n\n\n\n\n\n\n\n**Final answer:** 6 updates. If a question explicitly drops the final incomplete batch, it instead gives 5.\n\n"},{"title":"Model answer: batch, stochastic and mini-batch training","question":["Question. Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration.
"],"answer":["All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update.
","A batch is the group of examples used together for an update. An iteration, in this context, is one optimizer update. An epoch is one complete pass through the chosen training dataset. One epoch usually contains several iterations.
","| Basis | Batch GD | Stochastic GD | Mini-batch GD |
|---|---|---|---|
| Examples in one gradient | Entire training dataset | One example | A smaller group of B examples |
| Updates per epoch | One | N | Ceiling of N/B if the remainder is retained |
| Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging |
| Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation |
With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch.
","The basic parameter update is unchanged:
","For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples.
"],"kind":"theory","id":"tee-c5-s5","search":"Model answer: batch, stochastic and mini-batch training **Question.** Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration.\n\n**Model answer.**\nAll three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update.\n\n\n\n\nA **batch** is the group of examples used together for an update. An **iteration**, in this context, is one optimizer update. An **epoch** is one complete pass through the chosen training dataset. One epoch usually contains several iterations.\n\n| Basis | Batch GD | Stochastic GD | Mini-batch GD |\n| Examples in one gradient | Entire training dataset | One example | A smaller group of B examples |\n| Updates per epoch | One | N | Ceiling of N/B if the remainder is retained |\n| Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging |\n| Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation |\n\n### Example to distinguish the three counts\nWith 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch.\n\nThe basic parameter update is unchanged:\n\nFor batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples.\n"},{"title":"A tiny dataset shows why the methods differ","blocks":["Question. Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order.
","Model ŷ = wx; initial w = 1; η = 0.1.
","Samples in order: (1,2), (2,4).
","Mean half squared loss for the batch.
","Compare one batch update with a sequential SGD epoch.
","Batch mean versus sequential SGD. A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.
","B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.
","Yaad rakhne ka cue: Batch: same old weight par average; SGD: har sample ke baad weight change.
","Step 1: predict sample 1 with the original weight.
","Step 2: calculate its error.
","Step 3: multiply its error by its input.
","Step 4: predict sample 2 with the same original weight.
","Step 5: calculate its error.
","Step 6: multiply its error by its input.
","Step 7: average the two gradients.
","Step 8: update once.
","Step 1: use the first sample gradient.
","Step 2: update after the first sample.
","Step 3: predict sample 2 using the changed weight.
","Step 4: find sample 2 error.
","Step 5: find the new sample 2 gradient.
","Step 6: update after sample 2.
","Final answer: batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples.
","Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s6","search":"A tiny dataset shows why the methods differ **Question.** Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order.\n\n\n\n### Given\n\nModel ŷ = wx; initial w = 1; η = 0.1.\n\nSamples in order: (1,2), (2,4).\n\nMean half squared loss for the batch.\n\n### To find\n\nCompare one batch update with a sequential SGD epoch.\n\n### Formula — which one and why\n\n**Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.\n\nB = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change.\n\n### Solution — step by step\n\n\n\n\n### One batch update\n\n**Step 1: predict sample 1 with the original weight.**\n\n\n\n\n\n\n\n**Step 2: calculate its error.**\n\n\n\n\n\n\n\n**Step 3: multiply its error by its input.**\n\n\n\n\n\n\n\n**Step 4: predict sample 2 with the same original weight.**\n\n\n\n\n\n\n\n**Step 5: calculate its error.**\n\n\n\n\n\n\n\n**Step 6: multiply its error by its input.**\n\n\n\n\n\n\n\n**Step 7: average the two gradients.**\n\n\n\n\n\n\n\n**Step 8: update once.**\n\n\n\n\n\n\n\n\n\n\n\n### One sequential SGD epoch\n\n**Step 1: use the first sample gradient.**\n\n\n\n\n\n\n\n**Step 2: update after the first sample.**\n\n\n\n\n\n\n\n**Step 3: predict sample 2 using the changed weight.**\n\n\n\n\n\n\n\n**Step 4: find sample 2 error.**\n\n\n\n\n\n\n\n**Step 5: find the new sample 2 gradient.**\n\n\n\n\n\n\n\n**Step 6: update after sample 2.**\n\n\n\n\n\n\n\n**Final answer:** batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples.\nLarger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim."},{"title":"Learning rate and common training problems","blocks":["Definition. The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss.
","A very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate.
","| Observation | Possible reason | Sensible first action |
|---|---|---|
| Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation |
| Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate |
| Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum |
| Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping |
| All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights |
Convergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s7","search":"Learning rate and common training problems **Definition.** The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss.\n\n### Explanation and working\nA very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate.\n\n| Observation | Possible reason | Sensible first action |\n| Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation |\n| Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate |\n| Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum |\n| Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping |\n| All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights |\nConvergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum."},{"title":"A moving average remembers the recent past","blocks":["Definition. An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients.
","For m₀ = 0, expanding the recurrence gives:
","The most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history.
","An average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form.
","Question. Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂.
","β = 0.9; initial m₀ = 0.
","Gradients in order: g₁ = 2, g₂ = −1.
","Find m₁ and m₂, preserving the sign of each gradient.
","Exponential moving average. Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state.
","β (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state.
","Yaad rakhne ka cue: Beta purani memory rakhta hai; 1−beta naya gradient leta hai.
","Step 1: find the recent-gradient coefficient.
","Step 2: retain the old memory for step 1.
","Step 3: weight the first gradient.
","Step 4: add to obtain the first memory.
","Step 5: retain the first memory for step 2.
","Step 6: weight the second gradient.
","Step 7: add to obtain the second memory.
","Final answer: m₁ = 0.2; m₂ = 0.08.
","The second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s8","search":"A moving average remembers the recent past **Definition.** An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients.\n\nFor m₀ = 0, expanding the recurrence gives:\n\nThe most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history.\n\n### Signed gradients versus squared gradients\nAn average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form.\n\n**Question.** Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂.\n\n\n\n### Given\n\nβ = 0.9; initial m₀ = 0.\n\nGradients in order: g₁ = 2, g₂ = −1.\n\n### To find\n\nFind m₁ and m₂, preserving the sign of each gradient.\n\n### Formula — which one and why\n\n**Exponential moving average.** Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state.\n\nβ (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state.\n\n\n\n**Yaad rakhne ka cue:** Beta purani memory rakhta hai; 1−beta naya gradient leta hai.\n\n### Solution — step by step\n\n\n\n\n**Step 1: find the recent-gradient coefficient.**\n\n\n\n\n\n\n\n**Step 2: retain the old memory for step 1.**\n\n\n\n\n\n\n\n**Step 3: weight the first gradient.**\n\n\n\n\n\n\n\n**Step 4: add to obtain the first memory.**\n\n\n\n\n\n\n\n**Step 5: retain the first memory for step 2.**\n\n\n\n\n\n\n\n**Step 6: weight the second gradient.**\n\n\n\n\n\n\n\n**Step 7: add to obtain the second memory.**\n\n\n\n\n\n\n\n**Final answer:** m₁ = 0.2; m₂ = 0.08.\nThe second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable.\n"},{"title":"Momentum remembers movement","blocks":["Definition. Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions.
","This course uses signed displacement velocity v. It already includes the learning rate, and the parameter update adds it to the old parameter:
","θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters.
","Question. J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity.
","J(θ) = ½θ²; θ₀ = 1; v₀ = 0.
","η = 0.1; β = 0.9; signed-displacement velocity.
","Find the gradient, velocity and actual parameter after each of two iterations.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","Derivative used: differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration.
","Step 1: Iteration 1: gradient, velocity, parameter.
","Step 2: Iteration 2: use θ₁ and v₁.
","| Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter |
|---|---|---|---|---|
| 1 | 1 | 1 | −0.1 | 0.9 |
| 2 | 0.9 | 0.9 | −0.18 | 0.72 |
Final answer: θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient.
","Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective.
","Some texts store a positive gradient accumulator u instead:
","With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s9","search":"Momentum remembers movement \n**Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions.\n\n### Update equations and convention\nThis course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter:\n\n\n\nθ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters.\n\n\n\n### Complete numerical: two iterations\n**Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity.\n\n\n\n### Given\n\nJ(θ) = ½θ²; θ₀ = 1; v₀ = 0.\n\nη = 0.1; β = 0.9; signed-displacement velocity.\n\n### To find\n\nFind the gradient, velocity and actual parameter after each of two iterations.\n\n### Formula — which one and why\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n**Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration.\n\n**Step 1: Iteration 1: gradient, velocity, parameter.**\n\n\n\n\n**Step 2: Iteration 2: use θ₁ and v₁.**\n\n\n\n\n| Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter |\n| 1 | 1 | 1 | −0.1 | 0.9 |\n| 2 | 0.9 | 0.9 | −0.18 | 0.72 |\n\n**Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient.\n\n### Properties and alternative notation\nMomentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective.\n\nSome texts store a positive gradient accumulator u instead:\n\nWith constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions.\n\n"},{"title":"Nesterov looks before correcting","blocks":["Definition. Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update.
","Ordinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured:
","The lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁.
","Question. J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity.
","J(θ) = ½θ²; θ₀ = 1; v₀ = 0.
","η = 0.1; β = 0.9; signed-displacement velocity.
","Find the gradient, velocity and actual parameter after each of two iterations.
","Nesterov lookahead. First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.
","θ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.
","Yaad rakhne ka cue: Pehle look ahead; gradient wahan; update original position se.
","The derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective.
","Iteration 1 — step 1: locate the lookahead point.
","Step 2: evaluate the gradient there.
","Step 3: calculate the new velocity and actual parameter.
","Iteration 2 — step 1: use the previous actual parameter and velocity.
","Step 2: evaluate the gradient at 0.81, not at 0.9.
","Step 3: calculate the new velocity.
","Step 4: update the old actual parameter.
","| Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ |
|---|---|---|---|---|---|
| 1 | 1 | 1 | 1 | −0.1 | 0.9 |
| 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 |
Final answer: θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s10","search":"Nesterov looks before correcting \n**Definition.** Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update.\n\n### What changes relative to momentum\nOrdinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured:\n\n\n\n\nThe lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁.\n\n\n\n### Complete numerical: two iterations\n**Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity.\n\n\n\n### Given\n\nJ(θ) = ½θ²; θ₀ = 1; v₀ = 0.\n\nη = 0.1; β = 0.9; signed-displacement velocity.\n\n### To find\n\nFind the gradient, velocity and actual parameter after each of two iterations.\n\n### Formula — which one and why\n\n**Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.\n\nθ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se.\n\n### Solution — step by step\n\n\n\n\n\nThe derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective.\n\n**Iteration 1 — step 1: locate the lookahead point.**\n\n**Step 2: evaluate the gradient there.**\n\n**Step 3: calculate the new velocity and actual parameter.**\n\n\n\n**Iteration 2 — step 1: use the previous actual parameter and velocity.**\n\n**Step 2: evaluate the gradient at 0.81, not at 0.9.**\n\n**Step 3: calculate the new velocity.**\n\n**Step 4: update the old actual parameter.**\n\n\n| Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ |\n| 1 | 1 | 1 | 1 | −0.1 | 0.9 |\n| 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 |\n\n**Final answer:** θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice.\n\n"},{"title":"Model answer: momentum and Nesterov","question":["Question. Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ?
"],"answer":["Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions.
","This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained.
","Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter.
","The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point.
","| Basis | Momentum | Nesterov |
|---|---|---|
| Gradient evaluated at | Current parameter | Lookahead parameter |
| Information retained | Previous velocity | Previous velocity |
| Intuition | Continue useful movement | Inspect the slope where old momentum points |
| Initial zero velocity | No movement contribution yet | First lookahead equals current parameter |
Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently.
"],"kind":"theory","id":"tee-c5-s11","search":"Model answer: momentum and Nesterov **Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ?\n\n**Model answer.**\nMomentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions.\n\n\n\n\n### Momentum formulas\nThis answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained.\n\n\n\n### Nesterov's change\nNesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter.\n\n\nThe velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point.\n\n| Basis | Momentum | Nesterov |\n| Gradient evaluated at | Current parameter | Lookahead parameter |\n| Information retained | Previous velocity | Previous velocity |\n| Intuition | Continue useful movement | Inspect the slope where old momentum points |\n| Initial zero velocity | No movement contribution yet | First lookahead equals current parameter |\n\nBoth require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently.\n"},{"title":"AdaGrad adds every squared gradient","blocks":["Definition. AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator.
","For one parameter θ, let gₜ be its current gradient and s₀ = 0:
","s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign.
","Question. Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero.
","θ₀ = 1; initial state(s) = 0; η = 0.1.
","Supplied gradients g₁ = 2, g₂ = 1.
","ε = 0 for this hand calculation.
","Calculate all required states and the parameter after each of two updates.
","AdaGrad. Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.
","G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.
","Yaad rakhne ka cue: Square karo, purane sum mein jodo, root se scale karo.
","The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter.
","Iteration 1 — step 1: update the accumulator.
","Step 2: calculate the effective learning rate.
","Step 3: apply the signed gradient.
","Iteration 2 — step 1: retain s₁ and add the new squared gradient.
","Step 2: calculate the new coefficient.
","Step 3: update from θ₁.
","Final answer: θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5.
","Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes.
","ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s12","search":"AdaGrad adds every squared gradient \n**Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator.\n\n### State, effective learning rate and update\nFor one parameter θ, let gₜ be its current gradient and s₀ = 0:\n\n\n\ns stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign.\n\n\n\n### Complete numerical: supplied gradients\n**Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero.\n\n\n\n### Given\n\nθ₀ = 1; initial state(s) = 0; η = 0.1.\n\nSupplied gradients g₁ = 2, g₂ = 1.\n\nε = 0 for this hand calculation.\n\n### To find\n\nCalculate all required states and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.\n\nG = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.\n\n\n\n\n\n**Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo.\n\n### Solution — step by step\n\n\n\n\n\nThe gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter.\n\n**Iteration 1 — step 1: update the accumulator.**\n\n**Step 2: calculate the effective learning rate.**\n\n**Step 3: apply the signed gradient.**\n\n\n**Iteration 2 — step 1: retain s₁ and add the new squared gradient.**\n\n**Step 2: calculate the new coefficient.**\n\n**Step 3: update from θ₁.**\n\n\n**Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5.\n\n### Advantage, limitation and epsilon convention\nCoordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes.\n\nε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical.\n\n"},{"title":"RMSProp remembers recent squared gradients","blocks":["Definition. RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator.
","Using decay β and zero initial state:
","β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update.
","If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever.
","Question. θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation.
","θ₀ = 1; initial state(s) = 0; η = 0.1.
","Supplied gradients g₁ = 2, g₂ = 1.
","ε = 0 for this hand calculation.
","β = 0.9.
","Calculate all required states and the parameter after each of two updates.
","RMSProp. Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.
","s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.
","Yaad rakhne ka cue: Saare purane squares nahi—unki fading average.
","Iteration 1 — step 1: calculate the squared-gradient average.
","Step 2: calculate the denominator and signed correction.
","Step 3: update the parameter.
","Iteration 2 — step 1: use s₁, not the original zero state.
","Step 2: calculate the denominator and signed correction.
","Step 3: update from θ₁, retaining calculator precision.
","Final answer: θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46.
","AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s13","search":"RMSProp remembers recent squared gradients \n**Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator.\n\n### Recurrence and interpretation\nUsing decay β and zero initial state:\n\n\nβ is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update.\n\nIf recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever.\n\n\n\n### Complete numerical: two iterations\n**Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation.\n\n\n\n### Given\n\nθ₀ = 1; initial state(s) = 0; η = 0.1.\n\nSupplied gradients g₁ = 2, g₂ = 1.\n\nε = 0 for this hand calculation.\n\nβ = 0.9.\n\n### To find\n\nCalculate all required states and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.\n\ns = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.\n\n\n\n\n\n**Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average.\n\n### Solution — step by step\n\n\n\n\n\n**Iteration 1 — step 1: calculate the squared-gradient average.**\n\n**Step 2: calculate the denominator and signed correction.**\n\n\n**Step 3: update the parameter.**\n\n\n**Iteration 2 — step 1: use s₁, not the original zero state.**\n\n**Step 2: calculate the denominator and signed correction.**\n\n\n**Step 3: update from θ₁, retaining calculator precision.**\n\n\n**Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46.\n\n### Comparison with AdaGrad and Adam\nAdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum.\n\n"},{"title":"Adam remembers both direction and squared size","blocks":["Definition. Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates.
","| State or coefficient | Meaning |
|---|---|
| mₜ | Moving average of signed gradients; retains directional information |
| sₜ | Moving average of squared gradients; estimates their uncentered second moment |
| β₁ | Decay coefficient for the first-moment estimate |
| β₂ | Decay coefficient for the second-moment estimate |
| t | Update number, starting at 1 for the first update |
| η and ε | Base learning rate and small denominator stabilizer |
The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity.
","Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight:
","The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator:
","The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ.
","Question. θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic.
","θ₀ = 1; initial state(s) = 0; η = 0.1.
","Supplied gradients g₁ = 2, g₂ = 1.
","ε = 0 for this hand calculation.
","β₁ = 0.9; β₂ = 0.999; use bias correction.
","Calculate all required states and the parameter after each of two updates.
","Adam with bias correction. Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update.
","m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence.
","Yaad rakhne ka cue: Direction memory + square memory; hats lagao, phir update.
","Iteration 1 — step 1: update the two raw states.
","Step 2: correct both states using t = 1.
","Step 3: update the parameter.
","Start with the current gradient and the two states saved from the previous iteration.
Carry forward raw states: m = 0.2, s = 0.004. The corrected values are used for this update.
Iteration 2 — step 1: carry forward the raw states m₁ and s₁.
","The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4.
","Step 2: correct both states using t = 2.
","Step 3: divide the corrected first moment by the corrected root.
","Step 4: update from θ₁.
","Start with the current gradient and the two states saved from the previous iteration.
Carry forward raw states: m = 0.28, s = 0.004996. The corrected values are used for this update.
| State | After iteration 1 | After iteration 2 |
|---|---|---|
| Raw first moment m | 0.2 | 0.28 |
| Raw second moment s | 0.004 | 0.004996 |
| Corrected first moment | 2 | 1.473684211 |
| Corrected second moment | 4 | 2.499249625 |
| Parameter θ | 0.9 | 0.806782036 |
Final answer: θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update.
","Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s14","search":"Adam remembers both direction and squared size \n**Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates.\n\n### What the two states represent\n| State or coefficient | Meaning |\n| mₜ | Moving average of signed gradients; retains directional information |\n| sₜ | Moving average of squared gradients; estimates their uncentered second moment |\n| β₁ | Decay coefficient for the first-moment estimate |\n| β₂ | Decay coefficient for the second-moment estimate |\n| t | Update number, starting at 1 for the first update |\n| η and ε | Base learning rate and small denominator stabilizer |\n\nThe second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity.\n\n\n\n### Recurrences, bias correction and update\n\n\nZero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight:\n\n\nThe update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator:\n\nThe derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ.\n\n### Complete numerical: changing gradient magnitude\n**Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic.\n\n\n\n### Given\n\nθ₀ = 1; initial state(s) = 0; η = 0.1.\n\nSupplied gradients g₁ = 2, g₂ = 1.\n\nε = 0 for this hand calculation.\n\nβ₁ = 0.9; β₂ = 0.999; use bias correction.\n\n### To find\n\nCalculate all required states and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update.\n\nm = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update.\n\n### Solution — step by step\n\n\n\n\n\n**Iteration 1 — step 1: update the two raw states.**\n\n\n**Step 2: correct both states using t = 1.**\n\n\n**Step 3: update the parameter.**\n\n\n**Iteration 2 — step 1: carry forward the raw states m₁ and s₁.**\nThe recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4.\n\n\n**Step 2: correct both states using t = 2.**\n\n\n\n**Step 3: divide the corrected first moment by the corrected root.**\n\n**Step 4: update from θ₁.**\n\n\n| State | After iteration 1 | After iteration 2 |\n| Raw first moment m | 0.2 | 0.28 |\n| Raw second moment s | 0.004 | 0.004996 |\n| Corrected first moment | 2 | 1.473684211 |\n| Corrected second moment | 4 | 2.499249625 |\n| Parameter θ | 0.9 | 0.806782036 |\n\n**Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update.\n\n### Properties and limitations\nAdam combines directional "},{"title":"Adam bias correction: why the denominator is one minus beta to a power","blocks":["Definition. Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate.
","Adam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization.
","At the second step, substitute the first expression into the recurrence:
","After the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term:
","After t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit.
","The squared-gradient memory uses its own decay β₂ and the same reasoning:
","Let both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38.
","Both corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient.
","At update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators.
","The parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows.
","This derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s15","search":"Adam bias correction: why the denominator is one minus beta to a power **Definition.** Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate.\n\n### Explanation and working\n### 1. What the moving average actually contains\nAdam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization.\n\n\n\n\nAt the second step, substitute the first expression into the recurrence:\n\n### 2. Add the coefficients\nAfter the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term:\n\nAfter t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit.\n\nThe squared-gradient memory uses its own decay β₂ and the same reasoning:\n\n\n### 3. A constant-gradient example makes the purpose visible\nLet both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38.\n\n\nBoth corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient.\n\n### 4. Two betas mean two separate powers\nAt update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators.\n\n### 5. Correct first, then combine\nThe parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows.\nThis derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude.\n"},{"title":"Model answer: AdaGrad, RMSProp and Adam","question":["Question. Compare AdaGrad, RMSProp and Adam. Explain what each method stores and how that memory changes a parameter update.
"],"answer":["AdaGrad, RMSProp and Adam are adaptive gradient-based optimizers. Each stores separate state for each trainable parameter and uses that state to scale its update. Let gₜ be the current gradient for parameter θ, η the base learning rate and ε a small positive stabilizer. The equations below put ε outside the square root.
","Starting at s₀ = 0, every squared gradient contributes permanently. Parameters with frequent large gradients acquire smaller effective learning rates. This can be useful for sparse features, but the growing denominator can make later updates too small. The base learning rate still has to be chosen.
","Old contributions decay by repeated multiplication by β. The state can decrease when recent squared gradients are small, avoiding AdaGrad's permanent accumulation. Basic RMSProp has one squared-gradient state and no separate signed-gradient average.
","Initialize m₀ = s₀ = 0 and count the first update as t = 1:
","The first moment stores signed directional information; the second stores squared-gradient scale. Correction compensates for their zero initialization. The corrected first moment is already the numerator's gradient estimate, so the update is not multiplied by gₜ again. Raw moments are carried into the next iteration.
","| Aspect | AdaGrad | RMSProp | Adam |
|---|---|---|---|
| Squared-gradient history | Cumulative sum | Decaying average | Decaying average |
| Separate signed-gradient average | No | No in the basic method | Yes |
| Standard initialization correction | No | No in the basic method | Corrects both moments |
| State values per parameter | One | One | Two |
| Main concern | Effective rate can become very small | Learning rate and decay need tuning | More state; hyperparameters still matter |
All three adapt updates coordinate by coordinate. None removes the need for validation or guarantees a global minimum for a general neural-network objective.
"],"kind":"theory","id":"tee-c5-s16","search":"Model answer: AdaGrad, RMSProp and Adam \n**Question.** Compare AdaGrad, RMSProp and Adam. Explain what each method stores and how that memory changes a parameter update.\n\n**Model answer.**\nAdaGrad, RMSProp and Adam are adaptive gradient-based optimizers. Each stores separate state for each trainable parameter and uses that state to scale its update. Let gₜ be the current gradient for parameter θ, η the base learning rate and ε a small positive stabilizer. The equations below put ε outside the square root.\n\n### AdaGrad: cumulative squared-gradient history\n\n\nStarting at s₀ = 0, every squared gradient contributes permanently. Parameters with frequent large gradients acquire smaller effective learning rates. This can be useful for sparse features, but the growing denominator can make later updates too small. The base learning rate still has to be chosen.\n\n### RMSProp: a decaying squared-gradient average\n\n\nOld contributions decay by repeated multiplication by β. The state can decrease when recent squared gradients are small, avoiding AdaGrad's permanent accumulation. Basic RMSProp has one squared-gradient state and no separate signed-gradient average.\n\n### Adam: first and second moments with bias correction\nInitialize m₀ = s₀ = 0 and count the first update as t = 1:\n\n\n\n\nThe first moment stores signed directional information; the second stores squared-gradient scale. Correction compensates for their zero initialization. The corrected first moment is already the numerator's gradient estimate, so the update is not multiplied by gₜ again. Raw moments are carried into the next iteration.\n\n| Aspect | AdaGrad | RMSProp | Adam |\n| Squared-gradient history | Cumulative sum | Decaying average | Decaying average |\n| Separate signed-gradient average | No | No in the basic method | Yes |\n| Standard initialization correction | No | No in the basic method | Corrects both moments |\n| State values per parameter | One | One | Two |\n| Main concern | Effective rate can become very small | Learning rate and decay need tuning | More state; hyperparameters still matter |\n\nAll three adapt updates coordinate by coordinate. None removes the need for validation or guarantees a global minimum for a general neural-network objective.\n\n"},{"title":"One comparison to rebuild all optimizer formulas","blocks":["| Method | What is remembered | What changes the step |
|---|---|---|
| GD or SGD | No optimizer memory in the basic rule | Current gradient × η |
| Momentum | Previous movement | Old velocity plus current correction |
| Nesterov | Previous movement | Gradient evaluated at lookahead |
| AdaGrad | Sum of all squared gradients | Divide current gradient by accumulated root |
| RMSProp | Moving average of squared gradients | Divide current gradient by recent root |
| Adam | Moving averages of gradients and squares | Divide corrected mean by corrected root |
Working order. Start each method on a separate line with its states initialized. The solved examples use the same supplied gradients to show how their stored states differ. They do not have to give the same parameter value. The prescribed gradient sequence is practice data; if a question instead gives J(θ), recalculate its gradient at the appropriate current or lookahead parameter.
","The gradient has a sign, but its square is nonnegative. Both methods store squared gradients to control update size. The signed gradient is still used separately in the update numerator to preserve direction.
","For an old stored state of 4 and a new gradient of −3, first square the gradient: (−3) × (−3) = 9.
","AdaGrad: add the new square to the entire old sum: 4 + 9 = 13.
","RMSProp with decay 0.9: keep 0.9 × 4 = 3.6 of the old state; add (1 − 0.9) × 9 = 0.1 × 9 = 0.9 of the new square. The new state is 3.6 + 0.9 = 4.5.
","AdaGrad retains all earlier squares in a growing sum. RMSProp gradually discounts earlier squares. This difference explains why their effective step sizes change differently during training.
"],"question":[],"answer":[],"kind":"read","id":"tee-c5-s18","search":"AdaGrad and RMSProp with a negative gradient The gradient has a sign, but its square is nonnegative. Both methods store squared gradients to control update size. The signed gradient is still used separately in the update numerator to preserve direction.\n\n\n\nFor an old stored state of 4 and a new gradient of −3, first square the gradient: (−3) × (−3) = 9.\n**AdaGrad:** add the new square to the entire old sum: 4 + 9 = 13.\n**RMSProp with decay 0.9:** keep 0.9 × 4 = 3.6 of the old state; add (1 − 0.9) × 9 = 0.1 × 9 = 0.9 of the new square. The new state is 3.6 + 0.9 = 4.5.\nAdaGrad retains all earlier squares in a growing sum. RMSProp gradually discounts earlier squares. This difference explains why their effective step sizes change differently during training.\n"}]},{"n":6,"title":"Regularization and generalization","sections":[{"title":"Recognizing overfitting and underfitting","blocks":["Definitions. Generalization is a model's ability to perform on unseen examples from the relevant task distribution. Underfitting occurs when the model fails to capture the task's important relationships. Overfitting occurs when fitting the training data produces substantially poorer generalization, often by modeling noise or incidental details.
","Underfitting commonly gives high training and validation errors. Overfitting commonly gives low training error with a substantially higher validation error, or validation loss that rises while training loss continues to fall. These patterns are diagnostic clues; distribution shift, leakage and an unrepresentative split can also affect the curves.
","| Observation | Plausible explanation | Appropriate response |
|---|---|---|
| Both errors remain high | Insufficient capacity, weak features or inadequate optimization | Improve representation, train appropriately, reconsider architecture or reduce excessive regularization |
| Training error low; validation error high | Overfitting or a distribution/split problem | Inspect the split; use suitable regularization, more representative data or early stopping |
| Both losses fluctuate or diverge | Optimization instability | Examine learning rate, data scaling and gradient magnitude |
Insufficient model flexibility can create high approximation bias. A highly flexible model can be sensitive to the particular training sample, increasing variance. Regularization trades some training fit for constraints that can improve generalization. A more complex network is not automatically more accurate on unseen data.
","The validation set is used to choose hyperparameters and checkpoints. The test set should remain outside these choices. Repeatedly selecting models from test results turns the test data into part of the selection process and weakens its role as an unbiased final evaluation.
"],"question":[],"answer":[],"kind":"read","id":"tee-c6-s0","search":"Recognizing overfitting and underfitting **Definitions.** Generalization is a model's ability to perform on unseen examples from the relevant task distribution. Underfitting occurs when the model fails to capture the task's important relationships. Overfitting occurs when fitting the training data produces substantially poorer generalization, often by modeling noise or incidental details.\n\n### Interpreting learning curves\nUnderfitting commonly gives high training and validation errors. Overfitting commonly gives low training error with a substantially higher validation error, or validation loss that rises while training loss continues to fall. These patterns are diagnostic clues; distribution shift, leakage and an unrepresentative split can also affect the curves.\n\n\n\n| Observation | Plausible explanation | Appropriate response |\n| Both errors remain high | Insufficient capacity, weak features or inadequate optimization | Improve representation, train appropriately, reconsider architecture or reduce excessive regularization |\n| Training error low; validation error high | Overfitting or a distribution/split problem | Inspect the split; use suitable regularization, more representative data or early stopping |\n| Both losses fluctuate or diverge | Optimization instability | Examine learning rate, data scaling and gradient magnitude |\n\n### Bias–variance perspective\nInsufficient model flexibility can create high approximation bias. A highly flexible model can be sensitive to the particular training sample, increasing variance. Regularization trades some training fit for constraints that can improve generalization. A more complex network is not automatically more accurate on unseen data.\n\nThe validation set is used to choose hyperparameters and checkpoints. The test set should remain outside these choices. Repeatedly selecting models from test results turns the test data into part of the selection process and weakens its role as an unbiased final evaluation.\n"},{"title":"L1 and L2 add a cost for large weights","blocks":["Definition. Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss.
","Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit.
","Question. Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention.
","Data loss = 0.5; weights = [2,−1]; λ = 0.1.
","The L2 convention here has no factor ½.
","Find L1/L2 penalties and the two regularized objective values.
","L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.
","λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.
","Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.
","L2 penalty without a factor of one-half. This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective.
","λ = penalty strength; J_data = original data loss; wᵢ = weight i.
","Yaad rakhne ka cue: Half nahi diya toh L2 derivative mein 2 lambda × weight.
","Step 1: take absolute values for L1.
","Step 2: add the absolute values.
","Step 3: multiply by lambda.
","Step 4: add the data loss.
","Step 5: square both weights for L2.
","Step 6: add the squares.
","Step 7: multiply by lambda without adding a half.
","Step 8: add the data loss.
","For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective.
","Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.
","w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½.
","Penalty gradient, total gradient and updated weight.
","The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent.
","λ is penalty strength; η is learning rate; g is the full objective gradient.
","Yaad rakhne ka cue: Penalty ka gradient data gradient mein jodo, phir update.
","Step 1: calculate the penalty gradient.
","Step 2: add the data gradient.
","Step 3: scale by the learning rate.
","Step 4: subtract from the old weight.
","Final answer: Updated weight = 1.993. Bias penalties are omitted unless the objective includes them.
","Under the objective written above, the data gradient and penalty gradient add:
","For ordinary gradient descent, regrouping the update gives:
","This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay.
"],"question":[],"answer":[],"kind":"read","id":"tee-c6-s1","search":"L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss.\n\n### Explanation and working\n\nLambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit.\n\n\n\n**Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention.\n\n\n\n### Given\n\nData loss = 0.5; weights = [2,−1]; λ = 0.1.\n\nThe L2 convention here has no factor ½.\n\n### To find\n\nFind L1/L2 penalties and the two regularized objective values.\n\n### Formula — which one and why\n\n**L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.\n\nλ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.\n\n\n\n\n\n**Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai.\n\n**L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective.\n\nλ = penalty strength; J_data = original data loss; wᵢ = weight i.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight.\n\n### Solution — step by step\n\n\n\n\n**Step 1: take absolute values for L1.**\n\n\n\n\n\n\n\n**Step 2: add the absolute values.**\n\n\n\n\n\n\n\n**Step 3: multiply by lambda.**\n\n\n\n\n\n\n\n**Step 4: add the data loss.**\n\n\n\n\n\n\n\n\n**Step 5: square both weights for L2.**\n\n\n\n\n\n\n\n**Step 6: add the squares.**\n\n\n\n\n\n\n\n**Step 7: multiply by lambda without adding a half.**\n\n\n\n\n\n\n\n**Step 8: add the data loss.**\n\n\n\n\n\n\n\n\nFor this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective.\n\n\n### L2 gradient update without a half factor\n\n**Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.**\n\n\n\n### Given\n\nw = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½.\n\n### To find\n\nPenalty gradient, total gradient and updated weight.\n\n### Formula — which one and why\n\nThe derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent.\n\nλ is penalty strength; η is learning rate; g is the full objective gradient.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update.\n\n### Solution — step by step\n\n\n\n**Step 1: calculate the penalty gradient.**\n\n\n\n\n\n\n\n**Step 2: add the data gradient.**\n\n\n\n\n\n\n\n**Step 3: scale by the learning rate.**\n\n\n\n\n\n\n\n**Step 4: subtract from the old weight.**\n\n\n\n\n\n\n\n**Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them.\n\n\n\n### Deriving the L2 update\nUnder the objective written above, the data gradient and penalty gradient add:\n\nFor ordinary gradient descent, regrouping the update gives:\n\nThis shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ."},{"title":"Model answer: regularization and the L1–L2 difference","question":["Question. What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength.
"],"answer":["Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost.
","Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question.
","| Basis | L1 | L2 |
|---|---|---|
| Quantity penalized | Absolute weight magnitude | Squared weight magnitude |
| Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention |
| Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes |
| Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights |
The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total.
","Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective.
"],"kind":"theory","id":"tee-c6-s2","search":"Model answer: regularization and the L1–L2 difference **Question.** What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength.\n\n**Model answer.**\nRegularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost.\n\n\n\n\n### Objectives used in this guide\n\n\nJdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question.\n\n| Basis | L1 | L2 |\n| Quantity penalized | Absolute weight magnitude | Squared weight magnitude |\n| Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention |\n| Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes |\n| Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights |\n\nThe total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total.\n\nIncreasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective.\n"},{"title":"Dropout removes activations during training","blocks":["Definition. Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training.
","Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count.
","Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions.
","Question. Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs.
","Activations = [0.8,−0.4,0.6,0.2].
","Dropout rate = 50%, so keep probability q = 0.5.
","Chosen mask = [1,0,1,0].
","Find training outputs under inverted dropout.
","Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.
","aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.
","Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.
","Step 1. q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key.
","Step 2. For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = [1.6,0,1.2,0]. Inference then uses the full network without that extra scaling. State the convention in an answer.
","Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition.
"],"question":[],"answer":[],"kind":"read","id":"tee-c6-s3","search":"Dropout removes activations during training **Definition.** Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training.\n\n### Explanation and working\nDropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count.\n\n\n\nLet drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions.\n\n\n**Question.** Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs.\n\n\n\n### Given\n\nActivations = [0.8,−0.4,0.6,0.2].\n\nDropout rate = 50%, so keep probability q = 0.5.\n\nChosen mask = [1,0,1,0].\n\n### To find\n\nFind training outputs under inverted dropout.\n\n### Formula — which one and why\n\n**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.\n\naᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.\n\n\n\n\n\n**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.\n\n### Solution — step by step\n\n\n\n\n**Step 1.** q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key.\n**Step 2.** For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = **[1.6,0,1.2,0]**. Inference then uses the full network without that extra scaling. State the convention in an answer.\nRandom dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition."},{"title":"Early stopping uses the best validation point","blocks":["Definition. Early stopping is a regularization and model-selection procedure that terminates training when a monitored validation criterion no longer improves according to a specified rule, typically retaining the best checkpoint.
","Monitor validation loss after epochs. When it stops improving for a chosen patience interval, stop training and restore the best validation checkpoint. Training loss can fluctuate; it does not always decrease monotonically. Never select the stopping epoch by repeatedly consulting the test set.
","Question. Validation losses after epochs 1–5 are [0.8,0.6,0.5,0.53,0.55]. Patience is 2 with no required minimum improvement. When do you stop and what do you restore?
","Validation losses at epochs 1–5 = [0.8,0.6,0.5,0.53,0.55].
","Patience = 2; no required minimum improvement.
","Find the stop epoch and the best checkpoint to restore.
","Early stopping rule. Compare each validation loss with the best one so far. A new best resets the waiting count; a non-improvement increases it. Stop when the count reaches patience and restore the best checkpoint.
","Patience = allowed consecutive epochs without improvement. With minimum improvement zero, a strictly smaller validation loss counts as improvement.
","Yaad rakhne ka cue: Best validation save; patience khatam toh stop aur best restore.
","Step 1: process epoch 1.
","Loss 0.8 establishes the first best checkpoint; wait = 0.
","Step 2: process epoch 2.
","0.6 < 0.8: save epoch 2; wait = 0.
","Step 3: process epoch 3.
","0.5 < 0.6: save epoch 3; wait = 0.
","Step 4: process epoch 4.
","0.53 > 0.5: no improvement; wait = 1.
","Step 5: process epoch 5.
","0.55 > 0.5: no improvement; wait = 2, equal to patience. Stop.
","Step 6: restore the best checkpoint.
","The smallest observed validation loss is 0.5 at epoch 3. Restore its weights.
","Final answer: stop after epoch 5; restore epoch 3. These are different epochs.
","Question. Validation losses are [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.
","Validation losses in epoch order = [0.8, 0.7, 0.72, 0.68, 0.69, 0.7].
","Patience = 2; smaller loss is better; ties are not improvements.
","The stopping epoch and best checkpoint.
","Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs.
","Patience counts consecutive non-improvements. A new best resets the count.
","Yaad rakhne ka cue: Naya best mila toh save aur wait zero; patience khatam toh best restore.
","Step 1: process epoch 1.
","Validation loss = 0.8; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.8.
","Step 2: process epoch 2.
","Validation loss = 0.7; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 2, loss 0.7.
","Step 3: process epoch 3.
","Validation loss = 0.72; no strict improvement: wait count becomes 1. Best checkpoint is epoch 2, loss 0.7.
","Step 4: process epoch 4.
","Validation loss = 0.68; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 4, loss 0.68.
","Step 5: process epoch 5.
","Validation loss = 0.69; no strict improvement: wait count becomes 1. Best checkpoint is epoch 4, loss 0.68.
","Step 6: process epoch 6.
","Validation loss = 0.7; no strict improvement: wait count becomes 2. Best checkpoint is epoch 4, loss 0.68.
","Final answer: Stop after epoch 6; restore epoch 4, with validation loss 0.68.
","Question. Validation losses are [0.6, 0.6, 0.61, 0.62]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.
","Validation losses in epoch order = [0.6, 0.6, 0.61, 0.62].
","Patience = 2; smaller loss is better; ties are not improvements.
","The stopping epoch and best checkpoint.
","Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs.
","Patience counts consecutive non-improvements. A new best resets the count.
","Yaad rakhne ka cue: Naya best mila toh save aur wait zero; patience khatam toh best restore.
","Step 1: process epoch 1.
","Validation loss = 0.6; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.6.
","Step 2: process epoch 2.
","Validation loss = 0.6; no strict improvement: wait count becomes 1. Best checkpoint is epoch 1, loss 0.6.
","Step 3: process epoch 3.
","Validation loss = 0.61; no strict improvement: wait count becomes 2. Best checkpoint is epoch 1, loss 0.6.
","Final answer: Stop after epoch 3; restore epoch 1, with validation loss 0.6.
"],"question":[],"answer":[],"kind":"read","id":"tee-c6-s4","search":"Early stopping uses the best validation point **Definition.** Early stopping is a regularization and model-selection procedure that terminates training when a monitored validation criterion no longer improves according to a specified rule, typically retaining the best checkpoint.\n\n### Explanation and working\nMonitor validation loss after epochs. When it stops improving for a chosen patience interval, stop training and restore the best validation checkpoint. Training loss can fluctuate; it does not always decrease monotonically. Never select the stopping epoch by repeatedly consulting the test set.\n\n\n\n**Question.** Validation losses after epochs 1–5 are [0.8,0.6,0.5,0.53,0.55]. Patience is 2 with no required minimum improvement. When do you stop and what do you restore?\n\n\n\n### Given\n\nValidation losses at epochs 1–5 = [0.8,0.6,0.5,0.53,0.55].\n\nPatience = 2; no required minimum improvement.\n\n### To find\n\nFind the stop epoch and the best checkpoint to restore.\n\n### Formula — which one and why\n\n**Early stopping rule.** Compare each validation loss with the best one so far. A new best resets the waiting count; a non-improvement increases it. Stop when the count reaches patience and restore the best checkpoint.\n\nPatience = allowed consecutive epochs without improvement. With minimum improvement zero, a strictly smaller validation loss counts as improvement.\n\n**Yaad rakhne ka cue:** Best validation save; patience khatam toh stop aur best restore.\n\n### Solution — step by step\n\n\n\n\n**Step 1: process epoch 1.**\n\nLoss 0.8 establishes the first best checkpoint; wait = 0.\n\n**Step 2: process epoch 2.**\n\n0.6 < 0.8: save epoch 2; wait = 0.\n\n**Step 3: process epoch 3.**\n\n0.5 < 0.6: save epoch 3; wait = 0.\n\n**Step 4: process epoch 4.**\n\n0.53 > 0.5: no improvement; wait = 1.\n\n**Step 5: process epoch 5.**\n\n0.55 > 0.5: no improvement; wait = 2, equal to patience. Stop.\n\n**Step 6: restore the best checkpoint.**\n\nThe smallest observed validation loss is 0.5 at epoch 3. Restore its weights.\n\n**Final answer:** stop after epoch 5; restore epoch 3. These are different epochs.\n\n### Early stopping with [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]\n\n**Question. Validation losses are [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.**\n\n\n\n### Given\n\nValidation losses in epoch order = [0.8, 0.7, 0.72, 0.68, 0.69, 0.7].\n\nPatience = 2; smaller loss is better; ties are not improvements.\n\n### To find\n\nThe stopping epoch and best checkpoint.\n\n### Formula — which one and why\n\nSave an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs.\n\nPatience counts consecutive non-improvements. A new best resets the count.\n\n**Yaad rakhne ka cue:** Naya best mila toh save aur wait zero; patience khatam toh best restore.\n\n### Solution — step by step\n\n\n\n**Step 1: process epoch 1.**\n\nValidation loss = 0.8; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.8.\n\n**Step 2: process epoch 2.**\n\nValidation loss = 0.7; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 2, loss 0.7.\n\n**Step 3: process epoch 3.**\n\nValidation loss = 0.72; no strict improvement: wait count becomes 1. Best checkpoint is epoch 2, loss 0.7.\n\n**Step 4: process epoch 4.**\n\nValidation loss = 0.68; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is ep"},{"title":"Model answer: generalization and early stopping","question":["Question. Distinguish underfitting and overfitting. Explain how validation data and early stopping help control overfitting.
"],"answer":["Underfitting occurs when a model has not captured enough useful structure in the training data. Both training and validation errors are usually high relative to a sensible task baseline. Causes can include insufficient capacity, inadequate features, excessive regularization or incomplete optimization.
","Overfitting occurs when a model fits training-specific details or noise that do not generalize. Training error is low while validation error is substantially higher. A mismatch between training and validation data can produce a similar gap, so the split must also be examined.
","1. Train using the training set; do not learn parameters from the validation targets as training samples.
","2. After an epoch or another chosen interval, evaluate validation loss.
","3. Save the parameters whenever validation loss reaches a new best value.
","4. Count consecutive intervals with no qualifying improvement.
","5. When the count reaches the chosen patience, stop and restore the saved best parameters.
","For validation losses 0.8, 0.6, 0.5, 0.53 and 0.55, epoch 3 is best. With patience 2, epochs 4 and 5 supply two non-improvements: stop after epoch 5 and restore epoch 3. The stopping epoch and the retained model epoch are different.
","Other remedies for overfitting include appropriate L1/L2 penalties, dropout, augmentation and more representative training data. Underfitting may instead require a better model, adequate training or less regularization. Use the test set for final evaluation after those choices, not for repeatedly choosing the stopping point.
"],"kind":"theory","id":"tee-c6-s5","search":"Model answer: generalization and early stopping **Question.** Distinguish underfitting and overfitting. Explain how validation data and early stopping help control overfitting.\n\n**Model answer.**\nUnderfitting occurs when a model has not captured enough useful structure in the training data. Both training and validation errors are usually high relative to a sensible task baseline. Causes can include insufficient capacity, inadequate features, excessive regularization or incomplete optimization.\n\n\n\n\nOverfitting occurs when a model fits training-specific details or noise that do not generalize. Training error is low while validation error is substantially higher. A mismatch between training and validation data can produce a similar gap, so the split must also be examined.\n\n### How early stopping works\n1. Train using the training set; do not learn parameters from the validation targets as training samples.\n2. After an epoch or another chosen interval, evaluate validation loss.\n3. Save the parameters whenever validation loss reaches a new best value.\n4. Count consecutive intervals with no qualifying improvement.\n5. When the count reaches the chosen patience, stop and restore the saved best parameters.\n\nFor validation losses 0.8, 0.6, 0.5, 0.53 and 0.55, epoch 3 is best. With patience 2, epochs 4 and 5 supply two non-improvements: stop after epoch 5 and restore epoch 3. The stopping epoch and the retained model epoch are different.\n\nOther remedies for overfitting include appropriate L1/L2 penalties, dropout, augmentation and more representative training data. Underfitting may instead require a better model, adequate training or less regularization. Use the test set for final evaluation after those choices, not for repeatedly choosing the stopping point.\n"},{"title":"Choosing a remedy","blocks":["For overfitting: try L1/L2, dropout, suitable data augmentation, a smaller model, more representative training data and early stopping. For underfitting: improve features or architecture, train adequately, check optimization, or reduce excessive regularization. Evaluate changes on validation data.
","Data augmentation makes label-preserving variations of training examples, such as suitable crops or brightness changes in images. It must respect the task: flipping a digit or a medical image may change meaning. Augmentation happens within the training split; do not leak augmented copies into validation.
","| Change | Usual bias tendency | Usual variance tendency |
|---|---|---|
| Stronger weight regularization | Increase | Decrease |
| More hidden units with other factors fixed | Decrease | Increase |
| Dropout | Increase | Decrease |
| More representative data from same distribution | Roughly unchanged model bias | Decrease |
These are classical tendencies for exam discussion, not exact laws for every deep-network training run. Distinguish model capacity from optimization failure.
"],"question":[],"answer":[],"kind":"read","id":"tee-c6-s6","search":"Choosing a remedy For overfitting: try L1/L2, dropout, suitable data augmentation, a smaller model, more representative training data and early stopping. For underfitting: improve features or architecture, train adequately, check optimization, or reduce excessive regularization. Evaluate changes on validation data.\n\n\n\nData augmentation makes label-preserving variations of training examples, such as suitable crops or brightness changes in images. It must respect the task: flipping a digit or a medical image may change meaning. Augmentation happens within the training split; do not leak augmented copies into validation.\n| Change | Usual bias tendency | Usual variance tendency |\n| Stronger weight regularization | Increase | Decrease |\n| More hidden units with other factors fixed | Decrease | Increase |\n| Dropout | Increase | Decrease |\n| More representative data from same distribution | Roughly unchanged model bias | Decrease |\nThese are classical tendencies for exam discussion, not exact laws for every deep-network training run. Distinguish model capacity from optimization failure.\n\n"},{"title":"Inverted dropout during training and inference","blocks":["Definition. Inverted dropout divides retained training activations by their keep probability so that their expectation equals the original activation. At inference, dropout is disabled and no compensating activation scaling is required.
","Inverted dropout first chooses which activations to keep, then divides each kept activation by the keep probability. This compensates for the missing activations in expectation. A dropped activation becomes zero.
","For activation 3 and keep probability q = 0.5, a kept value becomes 3 ÷ 0.5 = 6. A dropped value becomes 0. Since half the masks keep it, the average is 0.5 × 6 + 0.5 × 0 = 3, equal to the original activation.
","At inference, dropout is disabled and the ordinary activation 3 is used. Under this inverted-dropout convention, no extra multiplication by q is required at inference.
","For a fixed activation a and a Bernoulli keep mask m with P(m = 1) = q:
","The equality concerns the activation's expectation over masks. It does not say every sampled layer preserves its sum, nor does it generally make a nonlinear network's complete stochastic prediction equal to its deterministic inference prediction.
"],"question":[],"answer":[],"kind":"read","id":"tee-c6-s7","search":"Inverted dropout during training and inference **Definition.** Inverted dropout divides retained training activations by their keep probability so that their expectation equals the original activation. At inference, dropout is disabled and no compensating activation scaling is required.\n\n### Explanation and working\nInverted dropout first chooses which activations to keep, then divides each kept activation by the keep probability. This compensates for the missing activations in expectation. A dropped activation becomes zero.\n\n\n\nFor activation 3 and keep probability q = 0.5, a kept value becomes 3 ÷ 0.5 = 6. A dropped value becomes 0. Since half the masks keep it, the average is 0.5 × 6 + 0.5 × 0 = 3, equal to the original activation.\nAt inference, dropout is disabled and the ordinary activation 3 is used. Under this inverted-dropout convention, no extra multiplication by q is required at inference.\n\n### Expectation calculation\nFor a fixed activation a and a Bernoulli keep mask m with P(m = 1) = q:\n\nThe equality concerns the activation's expectation over masks. It does not say every sampled layer preserves its sum, nor does it generally make a nonlinear network's complete stochastic prediction equal to its deterministic inference prediction.\n"},{"title":"Model answer: dropout with training and inference explained","question":["Question. Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling.
"],"answer":["Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights.
","Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop.
","A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks.
","For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is:
","Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions.
","Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data.
"],"kind":"theory","id":"tee-c6-s8","search":"Model answer: dropout with training and inference explained **Question.** Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling.\n\n**Model answer.**\nDropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights.\n\n\n\n\nLet p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop.\n\n### Inverted dropout during training\n\nA kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks.\n\nFor a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is:\n\n### Inference\nDisable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions.\n\nDropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data.\n"}]},{"n":7,"title":"Convolutional neural networks","sections":[{"title":"Reading image shapes before doing CNN arithmetic","blocks":["Definition. An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension.
","The shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors.
","A filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels.
","A 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels.
","The spatial-size formula computes height and width. It does not compute the channel count.
","N = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position.
","Shape counts the arrangement of output values. Parameters count learned weights and biases. Computation counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights.
","In a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s0","search":"Reading image shapes before doing CNN arithmetic **Definition.** An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension.\n\n### Explanation and working\n### Height, width and channels\nThe shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors.\n\n\n\n### One position versus the whole map\nA filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels.\n### Why filter depth matches input channels\nA 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels.\n### Reading a complete shape calculation\nThe spatial-size formula computes height and width. It does not compute the channel count.\n\nN = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position.\n### Shape, parameters and computation are three different counts\n**Shape** counts the arrangement of output values. **Parameters** count learned weights and biases. **Computation** counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights.\nIn a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count.\n"},{"title":"A computer sees an image as numbers","blocks":["Definition. A digital image is a spatially arranged numerical tensor. In a CNN, convolutions transform input channels into learned feature maps using local connectivity and spatially shared kernel parameters.
","A pixel is one position in an image. A grayscale image stores one intensity per position. A colour RGB image usually stores three numbers per position: red, green and blue. Therefore 32 × 32 × 3 means height 32, width 32 and 3 input channels. It contains 32 × 32 × 3 = 3,072 numbers, not 32 neurons.
","A feature is a useful pattern, such as an edge or texture. A feature map is a grid of responses that says where a learned filter matches the input. Early CNN layers often detect local simple patterns; later layers combine them into parts and more complex patterns.
","A filter, also called a kernel in many exam questions, is a small set of learned weights. Slide it across the image, multiply corresponding numbers, and add. The same weights are reused at every spatial position. This is weight sharing.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s1","search":"A computer sees an image as numbers **Definition.** A digital image is a spatially arranged numerical tensor. In a CNN, convolutions transform input channels into learned feature maps using local connectivity and spatially shared kernel parameters.\n\n### Explanation and working\nA pixel is one position in an image. A grayscale image stores one intensity per position. A colour RGB image usually stores three numbers per position: red, green and blue. Therefore 32 × 32 × 3 means height 32, width 32 and 3 input channels. It contains 32 × 32 × 3 = 3,072 numbers, not 32 neurons.\n\n\n\nA **feature** is a useful pattern, such as an edge or texture. A **feature map** is a grid of responses that says where a learned filter matches the input. Early CNN layers often detect local simple patterns; later layers combine them into parts and more complex patterns.\nA filter, also called a kernel in many exam questions, is a small set of learned weights. Slide it across the image, multiply corresponding numbers, and add. The same weights are reused at every spatial position. This is **weight sharing**."},{"title":"The convolution calculation","blocks":["Definition. A convolutional layer calculates local weighted sums with a spatially shared kernel and optional bias. Standard deep-learning implementations ordinarily use cross-correlation, without flipping the kernel; examination calculations must follow the stated convention.
","In deep-learning software, the operation normally called convolution is cross-correlation: use the kernel as written, without flipping it. This is the convention in this course’s CNN numericals and the supplied latest answer key. Mathematical convolution would rotate the kernel; state that convention if a question explicitly requests it.
","Question. Apply a 2 × 2 kernel Matrix 1 shown below to the 3 × 3 image Matrix 2 shown below, with stride 1, no padding and zero bias.
","Input X = .
Kernel K = .
Stride = 1; padding = 0; bias = 0; no kernel flip.
","Find the 2 × 2 output map.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Step 1. Top-left window. 1(1) + 2(0) + 4(0) + 5(−1) = 1 − 5 = −4.
","Step 2. Move right by one. 2(1) + 3(0) + 5(0) + 6(−1) = 2 − 6 = −4.
","Step 3. Move down and return left. 4(1) + 5(0) + 7(0) + 8(−1) = 4 − 8 = −4.
","Step 4. Bottom-right window. 5(1) + 6(0) + 8(0) + 9(−1) = 5 − 9 = −4. Output is . If ReLU follows, that next layer changes all four entries to zero; do not silently apply it when it was not requested.
Definition. Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output.
","Stride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension.
","N is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: = 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels.
Why plus one? Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1.
","Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.
","N = 6; K = 3; S = 2; P = 0; dilation = 1.
","The integer output side length.
","The formula counts complete kernel placements. Flooring removes an incomplete final placement.
","N is input side; P is padding on each side; K is kernel side; S is stride.
","Yaad rakhne ka cue: Padding dono side; stride se divide; floor; phir +1.
","Step 1: include padding and subtract kernel size.
","Step 2: divide by the stride.
","Step 3: take the floor.
","Step 4: count the first placement.
","Final answer: Output side = 2; a square input gives a 2 × 2 map per filter.
","Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.
","N = 5; K = 3; S = 1; P = 1 on each side.
","Padded input and output dimensions.
","Padding adds P entries on each of two opposite sides; the output counts complete kernel placements.
","N = unpadded side; K = kernel side; S = stride; P = per-side padding.
","Yaad rakhne ka cue: Padded input aur output ka size alag cheez hain.
","Step 1: count both padding edges.
","Step 2: calculate the padded input side.
","Step 3: subtract kernel width.
","Step 4: divide by stride and take the floor.
","Step 5: include the first placement.
","Final answer: Padded input = 7 × 7; output = 5 × 5 per filter.
","“Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s3","search":"Stride padding and floor **Definition.** Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output.\n\n### Explanation and working\nStride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension.\n\n\n\n\nN is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: \\(\\left\\lfloor 2.5\\right\\rfloor\\) = 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels.\n**Why plus one?** Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1.\n\n\n### Output dimension with a fractional intermediate result\n\n**Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.**\n\n\n\n### Given\n\nN = 6; K = 3; S = 2; P = 0; dilation = 1.\n\n### To find\n\nThe integer output side length.\n\n### Formula — which one and why\n\nThe formula counts complete kernel placements. Flooring removes an incomplete final placement.\n\nN is input side; P is padding on each side; K is kernel side; S is stride.\n\n\n\n**Yaad rakhne ka cue:** Padding dono side; stride se divide; floor; phir +1.\n\n### Solution — step by step\n\n\n\n**Step 1: include padding and subtract kernel size.**\n\n\n\n\n\n\n\n**Step 2: divide by the stride.**\n\n\n\n\n\n\n\n**Step 3: take the floor.**\n\n\n\n\n\n\n\n**Step 4: count the first placement.**\n\n\n\n\n\n\n\n**Final answer:** Output side = 2; a square input gives a 2 × 2 map per filter.\n\n\n\n\n### Padding on both sides\n\n**Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.**\n\n\n\n### Given\n\nN = 5; K = 3; S = 1; P = 1 on each side.\n\n### To find\n\nPadded input and output dimensions.\n\n### Formula — which one and why\n\nPadding adds P entries on each of two opposite sides; the output counts complete kernel placements.\n\nN = unpadded side; K = kernel side; S = stride; P = per-side padding.\n\n\n\n\n\n**Yaad rakhne ka cue:** Padded input aur output ka size alag cheez hain.\n\n### Solution — step by step\n\n\n\n**Step 1: count both padding edges.**\n\n\n\n\n\n\n\n**Step 2: calculate the padded input side.**\n\n\n\n\n\n\n\n**Step 3: subtract kernel width.**\n\n\n\n\n\n\n\n**Step 4: divide by stride and take the floor.**\n\n\n\n\n\n\n\n**Step 5: include the first placement.**\n\n\n\n\n\n\n\n**Final answer:** Padded input = 7 × 7; output = 5 × 5 per filter.\n\n\n“Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case."},{"title":"Convolution over several channels","blocks":["Definition. A standard convolutional filter spans every input channel. At one spatial position, channel-wise weighted sums are accumulated and one filter bias is added, producing one scalar output in that filter's feature map.
","A 3 × 3 filter receiving RGB input has depth 3, so it contains 3 × 3 × 3 weights. At one position, compute a weighted sum in each of the three channels, add those sums, and then add the filter’s one bias. One filter produces one output channel. Sixteen filters produce sixteen output channels.
","Tiny example. At one position the red, green and blue channel sums are 2, −1 and 4. With bias 0.5, the output is 2 − 1 + 4 + 0.5 = 5.5. Do not keep three separate outputs for this one filter; they have been added.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s4","search":"Convolution over several channels **Definition.** A standard convolutional filter spans every input channel. At one spatial position, channel-wise weighted sums are accumulated and one filter bias is added, producing one scalar output in that filter's feature map.\n\n### Explanation and working\nA 3 × 3 filter receiving RGB input has depth 3, so it contains 3 × 3 × 3 weights. At one position, compute a weighted sum in each of the three channels, add those sums, and then add the filter’s one bias. One filter produces one output channel. Sixteen filters produce sixteen output channels.\n\n\n\n**Tiny example.** At one position the red, green and blue channel sums are 2, −1 and 4. With bias 0.5, the output is 2 − 1 + 4 + 0.5 = 5.5. Do not keep three separate outputs for this one filter; they have been added."},{"title":"One filter over multiple channels: all contributions go into one cell","blocks":["Definition. Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels.
","Imagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3.
","The three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels.
","A 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias.
","The channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified.
","Each filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth.
","For three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters.
","A 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed.
","This is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s5","search":"One filter over multiple channels: all contributions go into one cell **Definition.** Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels.\n\n### Explanation and working\n### 1. Separate spatial size from depth\nImagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3.\n\n\n\n\n\nThe three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels.\n\n### 2. A larger filter has one spatial grid per input channel\nA 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias.\nThe channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified.\n\n### 3. What changes when there are several filters?\nEach filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth.\nFor three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters.\n\n### 4. One-by-one does not mean one weight\nA 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed.\nThis is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately.\n"},{"title":"Choosing more or fewer filters","blocks":["Definition. The number of convolutional filters determines the number of output channels. Kernel height, kernel width, stride and padding determine spatial processing; input-channel count determines each filter's depth.
","Each filter produces one output channel and can learn a different pattern. More filters increase representational capacity, output depth, parameter count and activation memory. They may improve performance but can overfit small datasets or exceed a compute budget. Choose counts using validation results and resource constraints. More filters in one layer increase its width, not the number of network layers.
","Definition. Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases.
","Max pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged.
","Question. Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding:
","The 4 × 4 input map is printed below.
","Window = 2 × 2; stride = 2; padding = 0.
","Find the pooled output by the operation(s) requested.
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","Step 1: select the four entries.
","[21, 11, 12, 9]
","Step 2: choose the largest entry.
","Step 3: add all entries.
","Step 4: divide the sum by four for the average.
","Step 1: select the four entries.
","[15, 10, 10, 11]
","Step 2: choose the largest entry.
","Step 3: add all entries.
","Step 4: divide the sum by four for the average.
","Step 1: select the four entries.
","[11, 13, 7, 4]
","Step 2: choose the largest entry.
","Step 3: add all entries.
","Step 4: divide the sum by four for the average.
","Step 1: select the four entries.
","[18, 12, 15, 12]
","Step 2: choose the largest entry.
","Step 3: add all entries.
","Step 4: divide the sum by four for the average.
","Outputs are max and average .
Pooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s7","search":"Pooling needs comparisons or averages **Definition.** Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases.\n\n### Explanation and working\nMax pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged.\n**Question.** Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding:\n\n\n\n### Given\n\nThe 4 × 4 input map is printed below.\n\nWindow = 2 × 2; stride = 2; padding = 0.\n\n### To find\n\nFind the pooled output by the operation(s) requested.\n\n### Formula — which one and why\n\n**Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.\n\nV = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.\n\n\n\n\n\n**Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count.\n\n### Solution — step by step\n\n\n\n\n\n### Top-left window\n\n**Step 1: select the four entries.**\n\n[21, 11, 12, 9]\n\n**Step 2: choose the largest entry.**\n\n\n\n\n\n\n\n**Step 3: add all entries.**\n\n\n\n\n\n\n\n**Step 4: divide the sum by four for the average.**\n\n\n\n\n\n\n\n\n### Top-right window\n\n**Step 1: select the four entries.**\n\n[15, 10, 10, 11]\n\n**Step 2: choose the largest entry.**\n\n\n\n\n\n\n\n**Step 3: add all entries.**\n\n\n\n\n\n\n\n**Step 4: divide the sum by four for the average.**\n\n\n\n\n\n\n\n\n### Bottom-left window\n\n**Step 1: select the four entries.**\n\n[11, 13, 7, 4]\n\n**Step 2: choose the largest entry.**\n\n\n\n\n\n\n\n**Step 3: add all entries.**\n\n\n\n\n\n\n\n**Step 4: divide the sum by four for the average.**\n\n\n\n\n\n\n\n\n### Bottom-right window\n\n**Step 1: select the four entries.**\n\n[18, 12, 15, 12]\n\n**Step 2: choose the largest entry.**\n\n\n\n\n\n\n\n**Step 3: add all entries.**\n\n\n\n\n\n\n\n**Step 4: divide the sum by four for the average.**\n\n\n\n\n\n\n\n\nOutputs are max and average .\n\nPooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance."},{"title":"Parameters are different from activation values","blocks":["Definition. Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters.
","An activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights.
","The +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation.
","Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.
","Kernel = 3 × 3; input channels = 3; filters = 16; one bias per filter.
","Weights, biases, total trainable parameters and output channels.
","Each filter spans every input channel and owns one bias. Spatial reuse does not create extra weights.
","Kₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters.
","Yaad rakhne ka cue: Ek filter ka count banao, phir filters se multiply.
","Step 1: count weights in one filter.
","Step 2: include its one bias.
","Step 3: count all weights.
","Step 4: count all biases.
","Step 5: add and state channel count.
","Each filter produces one channel, so output channels = 16.
","Final answer: 432 weights + 16 biases = 448 trainable parameters; 16 output channels.
","Flatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s8","search":"Parameters are different from activation values **Definition.** Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters.\n\n### Explanation and working\nAn activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights.\n\n\n\n\nThe +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation.\n\n\n### One filter, then all filters\n\n**Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.**\n\n\n\n### Given\n\nKernel = 3 × 3; input channels = 3; filters = 16; one bias per filter.\n\n### To find\n\nWeights, biases, total trainable parameters and output channels.\n\n### Formula — which one and why\n\nEach filter spans every input channel and owns one bias. Spatial reuse does not create extra weights.\n\nKₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters.\n\n\n\n**Yaad rakhne ka cue:** Ek filter ka count banao, phir filters se multiply.\n\n### Solution — step by step\n\n\n\n**Step 1: count weights in one filter.**\n\n\n\n\n\n\n\n**Step 2: include its one bias.**\n\n\n\n\n\n\n\n**Step 3: count all weights.**\n\n\n\n\n\n\n\n**Step 4: count all biases.**\n\n\n\n\n\n\n\n**Step 5: add and state channel count.**\n\n\n\n\n\nEach filter produces one channel, so output channels = 16.\n\n**Final answer:** 432 weights + 16 biases = 448 trainable parameters; 16 output channels.\n\n\n\nFlatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values."},{"title":"One complete CNN pipeline","blocks":["Question. Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron.
","Input = 128 × 128 × 3.
","Conv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2.
","Conv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2.
","Flatten → dense 240 → dense 120 → dense 5.
","One bias per learned filter/dense neuron.
","Find every output shape and layer parameter count, then the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Step 1. Conv 1 size. (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448.
","Step 2. Pool 1 size. = 63; output 63 × 63 × 16. Parameters = 0.
Step 3. Conv 2 size. (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640.
","Step 4. Pool 2 size. = + 1 = 30; output 30 × 30 × 32. Parameters = 0.
Step 5. Flatten. 30 × 30 × 32 = 28,800 values. Parameters = 0.
","Step 6. Dense 240. 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240.
","Step 7. Dense 120. (240 + 1) × 120 = 28,920.
","Step 8. Classifier. (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer.
","Step 9. Total. 448 + 4,640 + 6,912,240 + 28,920 + 605 = 6,946,853. Most parameters come from the first dense layer.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s9","search":"One complete CNN pipeline **Question.** Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron.\n\n\n\n### Given\n\nInput = 128 × 128 × 3.\n\nConv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2.\n\nConv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2.\n\nFlatten → dense 240 → dense 120 → dense 5.\n\nOne bias per learned filter/dense neuron.\n\n### To find\n\nFind every output shape and layer parameter count, then the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n**Step 1. Conv 1 size.** (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448.\n\n\n\n**Step 2. Pool 1 size.** \\(\\left\\lfloor\\frac{126 - 2}{2}\\right\\rfloor+1\\) = 63; output 63 × 63 × 16. Parameters = 0.\n**Step 3. Conv 2 size.** (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640.\n**Step 4. Pool 2 size.** \\(\\left\\lfloor\\frac{61 - 2}{2}\\right\\rfloor+1\\) = \\(\\left\\lfloor 29.5\\right\\rfloor\\) + 1 = 30; output 30 × 30 × 32. Parameters = 0.\n**Step 5. Flatten.** 30 × 30 × 32 = 28,800 values. Parameters = 0.\n**Step 6. Dense 240.** 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240.\n**Step 7. Dense 120.** (240 + 1) × 120 = 28,920.\n**Step 8. Classifier.** (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer.\n**Step 9. Total.** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853**. Most parameters come from the first dense layer."},{"title":"CNN versus a fully connected image network","blocks":["Definition. A convolutional neural network uses local receptive fields and spatial parameter sharing, unlike a dense layer's independent connection weights. These architectural constraints exploit spatial structure and can greatly reduce parameter counts.
","A fully connected layer can learn image features, but flattening pixels and connecting every pixel to every neuron ignores the useful local structure built into a CNN and can require many parameters. CNNs exploit local connections and shared filters. They learn hierarchical features and reuse a pattern detector at many locations.
","For a 32 × 32 × 3 input, a dense layer with 100 neurons has (3,072 + 1) × 100 = 307,300 parameters. A convolution with 100 filters of size 3 × 3 has (27 + 1) × 100 = 2,800 parameters. The outputs are different shapes, so this is a parameter-efficiency illustration, not two identical computational tasks.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s10","search":"CNN versus a fully connected image network **Definition.** A convolutional neural network uses local receptive fields and spatial parameter sharing, unlike a dense layer's independent connection weights. These architectural constraints exploit spatial structure and can greatly reduce parameter counts.\n\n### Explanation and working\nA fully connected layer can learn image features, but flattening pixels and connecting every pixel to every neuron ignores the useful local structure built into a CNN and can require many parameters. CNNs exploit local connections and shared filters. They learn hierarchical features and reuse a pattern detector at many locations.\n\n\n\nFor a 32 × 32 × 3 input, a dense layer with 100 neurons has (3,072 + 1) × 100 = 307,300 parameters. A convolution with 100 filters of size 3 × 3 has (27 + 1) × 100 = 2,800 parameters. The outputs are different shapes, so this is a parameter-efficiency illustration, not two identical computational tasks.\n"},{"title":"Model answer: CNN structure and parameter sharing","question":["Question. Explain the main layers of a convolutional neural network and why CNNs are suitable for image data. Compare them with a fully connected image network.
"],"answer":["A CNN processes spatially arranged data using filters that examine local regions. The same filter weights are reused at different positions. This combination of local connectivity and weight sharing builds an image-specific structure into the model and can greatly reduce the number of learned weights.
","Input → convolution → nonlinear activation → optional pooling → further feature extraction → classifier.
","| Layer | Operation | Learned parameters? |
|---|---|---|
| Convolution | Multiply a local window by a filter, sum and add its bias | Filter weights and usually one bias per filter |
| ReLU | Replace negative values by zero | No ordinary ReLU parameters |
| Pooling | Summarize each window by maximum or average | None for ordinary max/average pooling |
| Flatten | Rearrange H × W × C values into a vector | None |
| Dense layer | Form weighted combinations of all incoming values | Weights and receiving-neuron biases |
| Softmax | Convert class scores into probabilities | No additional weights of its own |
Early filters can learn local patterns such as edges. Later layers combine features over larger receptive fields. A filter responding at several image positions uses one shared set of weights, not a new set at every location.
","Flattening an image and using a dense layer does not enforce this local, shared structure. A dense layer has a separate connection weight for each input–output pair. CNNs can therefore be more parameter-efficient for visual patterns, although they still need suitable data and training. Pooling can provide some robustness to small shifts; it does not guarantee complete invariance to every translation.
"],"kind":"theory","id":"tee-c7-s11","search":"Model answer: CNN structure and parameter sharing **Question.** Explain the main layers of a convolutional neural network and why CNNs are suitable for image data. Compare them with a fully connected image network.\n\n**Model answer.**\nA CNN processes spatially arranged data using filters that examine local regions. The same filter weights are reused at different positions. This combination of local connectivity and weight sharing builds an image-specific structure into the model and can greatly reduce the number of learned weights.\n\n\n\n\n### Typical layer sequence\n**Input → convolution → nonlinear activation → optional pooling → further feature extraction → classifier.**\n\n| Layer | Operation | Learned parameters? |\n| Convolution | Multiply a local window by a filter, sum and add its bias | Filter weights and usually one bias per filter |\n| ReLU | Replace negative values by zero | No ordinary ReLU parameters |\n| Pooling | Summarize each window by maximum or average | None for ordinary max/average pooling |\n| Flatten | Rearrange H × W × C values into a vector | None |\n| Dense layer | Form weighted combinations of all incoming values | Weights and receiving-neuron biases |\n| Softmax | Convert class scores into probabilities | No additional weights of its own |\n\nEarly filters can learn local patterns such as edges. Later layers combine features over larger receptive fields. A filter responding at several image positions uses one shared set of weights, not a new set at every location.\n\nFlattening an image and using a dense layer does not enforce this local, shared structure. A dense layer has a separate connection weight for each input–output pair. CNNs can therefore be more parameter-efficient for visual patterns, although they still need suitable data and training. Pooling can provide some robustness to small shifts; it does not guarantee complete invariance to every translation.\n"},{"title":"Output size and parameter count are different","blocks":["Definition. Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters.
","Question. A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias.
","Input = 7 × 7 × 3; two 3 × 3 filters.
","Stride = 2; padding = 1; include biases.
","Find the output shape and trainable parameter count.
","Convolution/pooling spatial size. Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.
","N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.
","Yaad rakhne ka cue: Input + dono sides ki padding − kernel; stride se divide; floor + 1.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","Step 1 spatial size: padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2.
","Step 2 parameters: one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s12","search":"Output size and parameter count are different **Definition.** Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters.\n\n### Explanation and working\n**Question.** A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias.\n\n\n\n### Given\n\nInput = 7 × 7 × 3; two 3 × 3 filters.\n\nStride = 2; padding = 1; include biases.\n\n### To find\n\nFind the output shape and trainable parameter count.\n\n### Formula — which one and why\n\n**Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.\n\nN = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.\n\n\n\n**Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n**Step 1 spatial size:** padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2.\n\n\n\n**Step 2 parameters:** one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters.\n\n"},{"title":"Model answer: shape, activations and parameters","question":["Question. Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them.
"],"answer":["Output shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions.
","For an ordinary convolution with dilation 1:
","N is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters.
","This is the number of output values for one example, excluding any batch dimension.
","Each filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions.
","A 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels.
"],"kind":"theory","id":"tee-c7-s13","search":"Model answer: shape, activations and parameters **Question.** Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them.\n\n**Model answer.**\nOutput shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions.\n\n\n\n\n### Spatial output shape\nFor an ordinary convolution with dilation 1:\n\nN is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters.\n\n### Output activations\n\nThis is the number of output values for one example, excluding any batch dimension.\n\n### Trainable parameters\n\nEach filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions.\n\n### Small example\nA 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels.\n"},{"title":"Faculty example: a complete binary-image convolution","blocks":["Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)
","Image X = .
Filter K = . These are the values in the Unit 4 lecture example.
The output shape and all nine feature-map values.
","The output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input.
","Here N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients.
","For this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always top-left, top-right, centre, bottom-left, bottom-right within the current window.
","Step 1: find the shape. (5 − 3)/1 + 1 = 3. The output is 3 × 3.
","Step 2: top-left output, showing all nine products. 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = 4.
","Step 3: top-middle output. Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = 3.
","Step 4: top-right output. Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = 4.
","Step 5: middle-left output. Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = 2.
","Step 6: middle-middle output. Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = 4.
","Step 7: middle-right output. Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = 3.
","Step 8: bottom-left output. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = 2.
","Step 9: bottom-middle output. Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = 3.
","Step 10: bottom-right output. Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = 4.
","Final answer: Y = . All nine outputs use the same nine filter weights. They do not introduce nine different filters.
Yaad rakho: Window badalta hai; filter ke numbers same rehte hain.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s14","search":"Faculty example: a complete binary-image convolution **Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)**\n\n### Given\nImage X = .\n\nFilter K = . These are the values in the Unit 4 lecture example.\n\n### To find\nThe output shape and all nine feature-map values.\n\n### Formula — which one and why\nThe output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input.\n\n\n\nHere N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients.\n\n\n\nFor this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always **top-left, top-right, centre, bottom-left, bottom-right** within the current window.\n\n\n\n### Solution — step by step\n**Step 1: find the shape.** (5 − 3)/1 + 1 = 3. The output is 3 × 3.\n\n**Step 2: top-left output, showing all nine products.** 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = **4**.\n\n**Step 3: top-middle output.** Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = **3**.\n\n**Step 4: top-right output.** Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = **4**.\n\n**Step 5: middle-left output.** Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**.\n\n**Step 6: middle-middle output.** Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = **4**.\n\n**Step 7: middle-right output.** Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = **3**.\n\n**Step 8: bottom-left output.** Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**.\n\n**Step 9: bottom-middle output.** Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = **3**.\n\n**Step 10: bottom-right output.** Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = **4**.\n\n**Final answer:** Y = . All nine outputs use the same nine filter weights. They do not introduce nine different filters.\n\n**Yaad rakho:** Window badalta hai; filter ke numbers same rehte hain.\n"},{"title":"Faculty example: combine three RGB channel sums","blocks":["Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)
","Red patch = ; red kernel = .
Green patch = ; green kernel = .
Blue patch = ; blue kernel = . Bias = 1.
These are three slices of one full-depth filter, not three independent output filters.
","The contribution of each input channel and the final scalar output at this position.
","Multiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions.
","The quantities s_R, s_G and s_B are intermediate weighted sums, not separate final output channels.
","Step 1: red-channel products. The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1.
","Step 2: red-channel sum. 156 − 155 + 153 + 154 = 308.
","Step 3: green-channel products. The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1).
","Step 4: green-channel sum. −167 −166 +0 −165 = −498.
","Step 5: blue-channel products. The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1.
","Step 6: blue-channel sum. 163 +0 −160 +161 = 164.
","Step 7: add the channel contributions. 308 −498 +164 = −26.
","Step 8: add one filter bias. −26 +1 = −25.
","Final answer: The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages.
","Yaad rakho: Teen channels ka contribution jodo; ek filter ka bias ek baar jodo.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s15","search":"Faculty example: combine three RGB channel sums **Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)**\n\n### Given\nRed patch = ; red kernel = .\n\nGreen patch = ; green kernel = .\n\nBlue patch = ; blue kernel = . Bias = 1.\n\nThese are three slices of **one** full-depth filter, not three independent output filters.\n\n### To find\nThe contribution of each input channel and the final scalar output at this position.\n\n### Formula — which one and why\nMultiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions.\n\n\n\nThe quantities s_R, s_G and s_B are intermediate weighted sums, not separate final output channels.\n\n### Solution — step by step\n**Step 1: red-channel products.** The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1.\n\n**Step 2: red-channel sum.** 156 − 155 + 153 + 154 = **308**.\n\n**Step 3: green-channel products.** The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1).\n\n**Step 4: green-channel sum.** −167 −166 +0 −165 = **−498**.\n\n**Step 5: blue-channel products.** The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1.\n\n**Step 6: blue-channel sum.** 163 +0 −160 +161 = **164**.\n\n**Step 7: add the channel contributions.** 308 −498 +164 = −26.\n\n**Step 8: add one filter bias.** −26 +1 = **−25**.\n\n**Final answer:** The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages.\n\n**Yaad rakho:** Teen channels ka contribution jodo; ek filter ka bias ek baar jodo.\n"},{"title":"Max, min and average pooling on the same lecture grid","blocks":["Definition. Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases.
","Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)
","X = . Window = 2 × 2; stride = 2; padding = 0.
The shared output shape and each of the four cells for each pooling method.
","Each output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap.
","Step 1: output size. (4 −2)/2 +1 = 2. Each output has shape 2 × 2.
","Step 2: top-left window [12,20,8,12]. Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13.
","Step 3: top-right window [30,0,2,0]. Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8.
","Step 4: bottom-left window [34,70,112,100]. Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79.
","Step 5: bottom-right window [37,4,25,12]. Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = 19.5. Do not silently round this to 20.
","Step 6: put each result back in its window's position. Max = . Min = . Average = .
Final answer: Three 2 × 2 maps as above; zero trainable parameters for each pooling operation.
","Max retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour.
","Pooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter.
","Yaad rakho: Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s16","search":"Max, min and average pooling on the same lecture grid **Definition.** Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases.\n\n**Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)**\n\n### Given\nX = . Window = 2 × 2; stride = 2; padding = 0.\n\n### To find\nThe shared output shape and each of the four cells for each pooling method.\n\n### Formula — which one and why\nEach output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap.\n\n\n\n\n\n\n\n### Solution — step by step\n**Step 1: output size.** (4 −2)/2 +1 = 2. Each output has shape 2 × 2.\n\n**Step 2: top-left window [12,20,8,12].** Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13.\n\n**Step 3: top-right window [30,0,2,0].** Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8.\n\n**Step 4: bottom-left window [34,70,112,100].** Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79.\n\n**Step 5: bottom-right window [37,4,25,12].** Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = **19.5**. Do not silently round this to 20.\n\n**Step 6: put each result back in its window's position.** Max = . Min = . Average = .\n\n**Final answer:** Three 2 × 2 maps as above; zero trainable parameters for each pooling operation.\n\n### Why the choice matters\nMax retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour.\n\nPooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter.\n\n**Yaad rakho:** Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide.\n"},{"title":"Faculty CNN table: shapes, activations and 62958 parameters","blocks":["Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)
","Input = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question.
","Output shape and number of output values for each layer, its trainable parameters, and the total parameters.
","Use the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias.
","A counts produced activations. P_conv and P_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here.
","Step 1: input. Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0.
","Step 2: Conv1 spatial size. (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272.
","Step 3: Conv1 parameters. One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = 608.
","Step 4: Pool1. (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight.
","Step 5: Conv2 spatial size. (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600.
","Step 6: Conv2 parameters. Its input has eight channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = 3,216.
","Step 7: Pool2 and flatten. (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = 400. Neither operation adds parameters.
","Step 8: dense 120. Weights = 400×120 = 48,000. Biases = 120. Parameters = 48,120. Output contains 120 activations.
","Step 9: dense 84. Weights = 120×84 = 10,080. Biases = 84. Parameters = 10,164. Output contains 84 activations.
","Step 10: dense 10 and softmax. Weights = 84×10 = 840. Biases = 10. Dense parameters = 850. Softmax converts the ten logits into ten probabilities and adds zero parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation.
","Step 11: add only the trainable counts. 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = 62,958.
","| Layer | Output shape | Activations | Parameters |
|---|---|---|---|
| Input | 32 × 32 × 3 | 3,072 | 0 |
| Conv1 | 28 × 28 × 8 | 6,272 | 608 |
| Pool1 | 14 × 14 × 8 | 1,568 | 0 |
| Conv2 | 10 × 10 × 16 | 1,600 | 3,216 |
| Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 |
| Dense 120 | 120 | 120 | 48,120 |
| Dense 84 | 84 | 84 | 10,164 |
| Dense 10 → softmax | 10 | 10 | 850 + 0 |
Final answer: The network has 62,958 trainable parameters. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network.
","Yaad rakho: Shape aage carry karo; weights aur activations ko alag columns mein rakho.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s17","search":"Faculty CNN table: shapes, activations and 62958 parameters **Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)**\n\n### Given\nInput = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question.\n\n### To find\nOutput shape and number of output values for each layer, its trainable parameters, and the total parameters.\n\n### Formula — which one and why\nUse the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias.\n\n\n\n\n\n\nA counts produced activations. P_conv and P_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here.\n\n### Solution — step by step\n**Step 1: input.** Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0.\n\n**Step 2: Conv1 spatial size.** (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272.\n\n**Step 3: Conv1 parameters.** One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = **608**.\n\n**Step 4: Pool1.** (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight.\n\n**Step 5: Conv2 spatial size.** (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600.\n\n**Step 6: Conv2 parameters.** Its input has **eight** channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = **3,216**.\n\n**Step 7: Pool2 and flatten.** (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = **400**. Neither operation adds parameters.\n\n**Step 8: dense 120.** Weights = 400×120 = 48,000. Biases = 120. Parameters = **48,120**. Output contains 120 activations.\n\n**Step 9: dense 84.** Weights = 120×84 = 10,080. Biases = 84. Parameters = **10,164**. Output contains 84 activations.\n\n**Step 10: dense 10 and softmax.** Weights = 84×10 = 840. Biases = 10. Dense parameters = **850**. Softmax converts the ten logits into ten probabilities and adds **zero** parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation.\n\n**Step 11: add only the trainable counts.** 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = **62,958**.\n\n| Layer | Output shape | Activations | Parameters |\n|---|---|---|---|\n| Input | 32 × 32 × 3 | 3,072 | 0 |\n| Conv1 | 28 × 28 × 8 | 6,272 | 608 |\n| Pool1 | 14 × 14 × 8 | 1,568 | 0 |\n| Conv2 | 10 × 10 × 16 | 1,600 | 3,216 |\n| Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 |\n| Dense 120 | 120 | 120 | 48,120 |\n| Dense 84 | 84 | 84 | 10,164 |\n| Dense 10 → softmax | 10 | 10 | 850 + 0 |\n\n**Final answer:** The network has **62,958 trainable parameters**. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network.\n\n**Yaad rakho:** Shape aage carry karo; weights aur activati"},{"title":"Even-sized kernels: SAME padding is not always symmetric","blocks":["Definition. A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides.
","For a 2 by 2 kernel at stride 1, preserving the input size requires one padding entry in total per axis, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention.
","Hinglish cue: P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho.
","Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)
","Input height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter.
","The padded input dimensions and the convolution output dimensions.
","The two opposite borders contribute P entries each. The output formula counts complete filter placements after padding.
","N is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel.
","Step 1: count the two padding borders.
","Step 2: find the padded side length.
","The padded input is 7 × 7 × 1.
","Step 3: subtract the kernel size.
","Step 4: divide by stride, take the floor and count the first placement.
","Step 5: write height, width and channels.
","Both spatial dimensions equal 6. There is one filter, so there is one output channel.
","Final answer: padded input = 7 × 7 × 1; output = 6 × 6 × 1. This explicitly specified padding does not implement SAME for this even kernel.
","Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)
","Input side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found.
","Padding above, below, left and right; padded-input shape; final output shape.
","For unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis.
","At stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1.
","Step 1: calculate total padding along the height.
","Only one extra row in total is needed.
","Step 2: place the row using the stated convention.
","Padding above = 0; padding below = 1.
","Step 3: apply the same calculation to width.
","Only one extra column in total is needed. Padding left = 0; padding right = 1.
","Step 4: find the padded-input shape.
","The padded input is 6 × 6 × 1.
","Step 5: count the output positions along each axis.
","Step 6: include the output channel.
","One filter creates one output channel.
","Final answer: add 0 above, 1 below, 0 left and 1 right; padded input = 6 × 6 × 1; output = 5 × 5 × 1.
","The two questions have the same input and kernel but different padding instructions. Read the stated boundary padding before applying the formula.
"],"question":[],"answer":[],"kind":"read","id":"tee-c7-s18","search":"Even-sized kernels: SAME padding is not always symmetric \n**Definition.** A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides.\n\nFor a 2 by 2 kernel at stride 1, preserving the input size requires **one padding entry in total per axis**, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention.\n\n**Hinglish cue:** P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho.\n\n### Example 1: one entry on every side\n\n**Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)**\n\n### Given\n\nInput height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter.\n\n### To find\n\nThe padded input dimensions and the convolution output dimensions.\n\n### Formula — which one and why\n\nThe two opposite borders contribute P entries each. The output formula counts complete filter placements after padding.\n\n\n\n\n\nN is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel.\n\n### Solution — step by step\n\n**Step 1: count the two padding borders.**\n\n\n\n**Step 2: find the padded side length.**\n\n\n\nThe padded input is 7 × 7 × 1.\n\n**Step 3: subtract the kernel size.**\n\n\n\n**Step 4: divide by stride, take the floor and count the first placement.**\n\n\n\n**Step 5: write height, width and channels.**\n\nBoth spatial dimensions equal 6. There is one filter, so there is one output channel.\n\n**Final answer:** padded input = **7 × 7 × 1**; output = **6 × 6 × 1**. This explicitly specified padding does not implement SAME for this even kernel.\n\n### Example 2: preserve the size with SAME padding\n\n**Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)**\n\n### Given\n\nInput side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found.\n\n### To find\n\nPadding above, below, left and right; padded-input shape; final output shape.\n\n### Formula — which one and why\n\nFor unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis.\n\n\n\nAt stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1.\n\n\n\n### Solution — step by step\n\n**Step 1: calculate total padding along the height.**\n\n\n\nOnly one extra row in total is needed.\n\n**Step 2: place the row using the stated convention.**\n\nPadding above = 0; padding below = 1.\n\n**Step 3: apply the same calculation to width.**\n\nOnly one extra column in total is needed. Padding left = 0; padding right = 1.\n\n**St"}]},{"n":8,"title":"CNN architectures and transfer learning","sections":[{"title":"AlexNet and the reason for its components","blocks":["Definition. AlexNet is a deep convolutional architecture introduced for ImageNet classification, conventionally described as five convolutional layers followed by three fully connected layers. Its historical significance is the effective combination of learned convolutional features, nonlinear activations, GPU training and regularization at large scale.
","Input image → convolution and ReLU stages → spatial pooling → fully connected representations → class scores and softmax. The original model used local response normalization in selected early stages and overlapping max pooling. Grouped computation across two GPUs was part of the original implementation. Exact sizes must follow the architecture specified in an examination question.
","ReLU supplies a nonlinear transformation and avoids sigmoid-style saturation for positive preactivations. Convolution uses local connectivity and spatial weight sharing. Pooling reduces spatial dimensions and increases tolerance to small local changes. Dropout regularizes the fully connected layers. Data augmentation supplies transformed training examples to reduce dependence on a fixed view. These components perform different jobs and should not be described as interchangeable methods of reducing overfitting.
","The dense layers contribute a large number of parameters; the model has substantial memory and computation requirements. AlexNet is much shallower than later residual architectures. Its original use of local response normalization is different from batch normalization. For a comparison answer, identify its sequential convolutional design, then contrast it with Inception's parallel branches and ResNet's residual addition.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s0","search":"AlexNet and the reason for its components **Definition.** AlexNet is a deep convolutional architecture introduced for ImageNet classification, conventionally described as five convolutional layers followed by three fully connected layers. Its historical significance is the effective combination of learned convolutional features, nonlinear activations, GPU training and regularization at large scale.\n\n### Architecture and information flow\nInput image → convolution and ReLU stages → spatial pooling → fully connected representations → class scores and softmax. The original model used local response normalization in selected early stages and overlapping max pooling. Grouped computation across two GPUs was part of the original implementation. Exact sizes must follow the architecture specified in an examination question.\n\n\n\n### Role of the components\n**ReLU** supplies a nonlinear transformation and avoids sigmoid-style saturation for positive preactivations. **Convolution** uses local connectivity and spatial weight sharing. **Pooling** reduces spatial dimensions and increases tolerance to small local changes. **Dropout** regularizes the fully connected layers. **Data augmentation** supplies transformed training examples to reduce dependence on a fixed view. These components perform different jobs and should not be described as interchangeable methods of reducing overfitting.\n\n### Limitations and exam distinction\nThe dense layers contribute a large number of parameters; the model has substantial memory and computation requirements. AlexNet is much shallower than later residual architectures. Its original use of local response normalization is different from batch normalization. For a comparison answer, identify its sequential convolutional design, then contrast it with Inception's parallel branches and ResNet's residual addition.\n"},{"title":"Inception examines several spatial scales","blocks":["Definition. An Inception module is a convolutional building block that processes the same input through parallel branches using different receptive-field operations and concatenates their output feature maps along the channel dimension.
","A conventional early Inception module includes 1 × 1, 3 × 3 and 5 × 5 convolution branches and a pooling branch. Suitable stride and padding preserve compatible spatial dimensions. The branch outputs are then concatenated. If branch channel counts are c₁, c₂, c₃ and c₄, the output depth is c₁ + c₂ + c₃ + c₄.
","Different branch receptive fields can capture patterns at different spatial scales. A 1 × 1 convolution mixes channel information at each position. Before an expensive larger kernel, it can reduce the number of input channels and therefore the parameter count and multiplication cost. The bottleneck calculation is developed in the following section.
","Concatenation increases channel depth and retains each branch's feature maps separately. Elementwise addition requires equal shapes and combines corresponding values. Inception commonly uses concatenation; a standard residual block uses addition. A question giving branch sizes tests this distinction as well as the convolution formula.
","Early Inception designs reduce reliance on a large fully connected classifier through global average pooling. The parallel design still needs compatible branch shapes and carefully chosen channel widths; increasing the number of branches or filters increases computation.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s1","search":"Inception examines several spatial scales **Definition.** An Inception module is a convolutional building block that processes the same input through parallel branches using different receptive-field operations and concatenates their output feature maps along the channel dimension.\n\n### Structure and working\nA conventional early Inception module includes 1 × 1, 3 × 3 and 5 × 5 convolution branches and a pooling branch. Suitable stride and padding preserve compatible spatial dimensions. The branch outputs are then concatenated. If branch channel counts are c₁, c₂, c₃ and c₄, the output depth is c₁ + c₂ + c₃ + c₄.\n\n\n\n### Why use several branches?\nDifferent branch receptive fields can capture patterns at different spatial scales. A 1 × 1 convolution mixes channel information at each position. Before an expensive larger kernel, it can reduce the number of input channels and therefore the parameter count and multiplication cost. The bottleneck calculation is developed in the following section.\n\n### Concatenation is not addition\nConcatenation increases channel depth and retains each branch's feature maps separately. Elementwise addition requires equal shapes and combines corresponding values. Inception commonly uses concatenation; a standard residual block uses addition. A question giving branch sizes tests this distinction as well as the convolution formula.\n\nEarly Inception designs reduce reliance on a large fully connected classifier through global average pooling. The parallel design still needs compatible branch shapes and carefully chosen channel widths; increasing the number of branches or filters increases computation.\n"},{"title":"Why one by one convolution saves work","blocks":["Definition. A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts.
","Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.
","Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size.
","Direct MACs, total two-stage bottleneck MACs and percentage saved.
","Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added.
","One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs.
","Step 1: direct layer. Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs.
","Step 2: 1 × 1 reduction. There are 16 receiving filters, each spanning all 192 original channels.
","Step 3: reduced-depth spatial layer. Its 32 filters now span 16 channels.
","Step 4: add the two bottleneck costs.
","Step 5: calculate the saving.
","Final answer: direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption.
","Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.
","Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1.
","Direct MACs, bottleneck MACs and saving.
","Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers.
","Step 1: direct spatial convolution.
","Step 2: 1 × 1 reduction.
","Step 3: reduced-depth 3 × 3 convolution.
","Step 4: add the stages and subtract from the direct cost.
","Final answer: direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s2","search":"Why one by one convolution saves work **Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts.\n\n\n\n**Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.**\n\n### Given\nBoth paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size.\n\n### To find\nDirect MACs, total two-stage bottleneck MACs and percentage saved.\n\n### Formula — which one and why\nEach output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added.\n\nOne MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs.\n\n### Solution — step by step\n**Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs.\n\n\n**Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels.\n\n\n**Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels.\n\n\n**Step 4: add the two bottleneck costs.**\n\n\n**Step 5: calculate the saving.**\n\n\n\n**Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption.\n\n### A second bottleneck with different dimensions\n**Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.**\n\n### Given\nSpatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1.\n\n### To find\nDirect MACs, bottleneck MACs and saving.\n\n### Formula — which one and why\nUse the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers.\n\n### Solution — step by step\n**Step 1: direct spatial convolution.**\n\n\n**Step 2: 1 × 1 reduction.**\n\n\n**Step 3: reduced-depth 3 × 3 convolution.**\n\n\n**Step 4: add the stages and subtract from the direct cost.**\n\n\n**Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values.\n"},{"title":"ResNet adds a shortcut","blocks":["Definition. A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x.
","If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch.
","For the identity-shortcut formulation, the derivative includes a direct identity contribution.
","This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients.
","If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride.
","An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition.
","Question. Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum.
","x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3].
","Find the three components of x+F(x).
","Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.
","x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.
","Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.
","Step 1: add the first matching pair.
","Step 2: add the second matching pair.
","Step 3: add the third matching pair.
","Step 4: assemble in the original order.
","The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s3","search":"ResNet adds a shortcut **Definition.** A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x.\n\nIf the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch.\n\n\n\n### Effect on gradient propagation\nFor the identity-shortcut formulation, the derivative includes a direct identity contribution.\n\nThis provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients.\n\n### Projection when dimensions differ\nIf channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride.\n\nAn identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition.\n\n**Question.** Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum.\n\n\n\n### Given\n\nx = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3].\n\n### To find\n\nFind the three components of x+F(x).\n\n### Formula — which one and why\n\n**Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.\n\nx = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.\n\n\n\n**Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye.\n\n### Solution — step by step\n\n\n\n\n**Step 1: add the first matching pair.**\n\n\n\n\n\n\n\n**Step 2: add the second matching pair.**\n\n\n\n\n\n\n\n**Step 3: add the third matching pair.**\n\n\n\n\n\n\n\n**Step 4: assemble in the original order.**\n\n\n\n\n\n\n\n\nThe vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum.\n"},{"title":"Transfer learning from a small dataset","blocks":["Definition. Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers.
","Transfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort.
","Full scenario. Build a face-attendance model with few images per student. Explain a training plan.
","Step 1. Define and split the data. Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization.
","Step 2. Choose a suitable pretrained backbone. For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available.
","Step 3. Replace the classifier. Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy.
","Step 4. Freeze and train. Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring.
","Step 5. Fine-tune if justified. Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint.
","Step 6. Evaluate. Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition.
","Feature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s4","search":"Transfer learning from a small dataset **Definition.** Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers.\n\n### Explanation and working\nTransfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort.\n\n\n\n**Full scenario.** Build a face-attendance model with few images per student. Explain a training plan.\n**Step 1. Define and split the data.** Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization.\n**Step 2. Choose a suitable pretrained backbone.** For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available.\n**Step 3. Replace the classifier.** Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy.\n**Step 4. Freeze and train.** Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring.\n**Step 5. Fine-tune if justified.** Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint.\n**Step 6. Evaluate.** Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition.\nFeature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario."},{"title":"Model answer: feature extraction and fine-tuning","question":["Question. Explain transfer learning for image classification with limited labeled data. Distinguish feature extraction from fine-tuning and describe the training procedure.
"],"answer":["Transfer learning reuses knowledge represented by a model trained on a source task to support a related target task. A pretrained CNN can provide useful visual representations, reducing the target data and training effort needed compared with learning all features from random initialization.
","1. Define the target classes and create separate training, validation and test splits. Prevent duplicate or closely related captures from leaking across splits.
","2. Choose a suitable pretrained backbone and apply its required input size and preprocessing.
","3. Replace the original classifier with a new head whose output count matches the target classes.
","4. Freeze the backbone and train the new head. Only the unfrozen parameters receive optimizer updates.
","5. If validation results justify it, unfreeze selected later backbone layers and continue training with a suitably smaller learning rate.
","6. Select hyperparameters and checkpoints using validation data. Evaluate the selected model on the held-out test set.
","| Feature extraction | Fine-tuning |
|---|---|
| Pretrained backbone weights remain fixed | Some or all selected pretrained weights are updated |
| New classifier learns from fixed representations | Representations can adapt to the target task |
| Usually fewer trainable parameters | Usually more trainable parameters and greater risk of overfitting on a small dataset |
The benefit depends on how useful the source features are for the target data. Transfer is not guaranteed to help every unrelated task. For a numerical, count all stored parameters separately from the trainable subset; freezing a layer does not delete its weights.
"],"kind":"theory","id":"tee-c8-s5","search":"Model answer: feature extraction and fine-tuning **Question.** Explain transfer learning for image classification with limited labeled data. Distinguish feature extraction from fine-tuning and describe the training procedure.\n\n**Model answer.**\nTransfer learning reuses knowledge represented by a model trained on a source task to support a related target task. A pretrained CNN can provide useful visual representations, reducing the target data and training effort needed compared with learning all features from random initialization.\n\n\n\n\n### Procedure\n1. Define the target classes and create separate training, validation and test splits. Prevent duplicate or closely related captures from leaking across splits.\n2. Choose a suitable pretrained backbone and apply its required input size and preprocessing.\n3. Replace the original classifier with a new head whose output count matches the target classes.\n4. Freeze the backbone and train the new head. Only the unfrozen parameters receive optimizer updates.\n5. If validation results justify it, unfreeze selected later backbone layers and continue training with a suitably smaller learning rate.\n6. Select hyperparameters and checkpoints using validation data. Evaluate the selected model on the held-out test set.\n\n| Feature extraction | Fine-tuning |\n| Pretrained backbone weights remain fixed | Some or all selected pretrained weights are updated |\n| New classifier learns from fixed representations | Representations can adapt to the target task |\n| Usually fewer trainable parameters | Usually more trainable parameters and greater risk of overfitting on a small dataset |\n\nThe benefit depends on how useful the source features are for the target data. Transfer is not guaranteed to help every unrelated task. For a numerical, count all stored parameters separately from the trainable subset; freezing a layer does not delete its weights.\n"},{"title":"How to draw the architecture diagrams","blocks":["For AlexNet, draw a sequence of convolution/pooling stages followed by dense layers and classifier. For Inception, draw four parallel branches and a channel-concatenation join. For ResNet, draw one transformation branch and one shortcut meeting at a plus sign. For transfer learning, draw frozen pretrained base → new trainable head, then mark selected base layers optionally unfrozen for fine-tuning.
","Definition. Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths.
","Concatenation joins feature channels; elementwise addition sums matching entries. They are different operations.
","Four Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64.
","A residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s7","search":"Inception concatenation and residual addition **Definition.** Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths.\n\n### Explanation and working\nConcatenation joins feature channels; elementwise addition sums matching entries. They are different operations.\n\n\n\nFour Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64.\nA residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition.\n"},{"title":"Model answer: AlexNet, Inception and ResNet","question":["Question. Compare the main ideas of AlexNet, Inception and ResNet. Explain how to recognize their architecture diagrams.
"],"answer":["All three are CNN architectures, but their characteristic structures differ.
","| Architecture | Main idea | What to show in a diagram |
|---|---|---|
| AlexNet | Successive convolutional feature extraction followed by dense classification | Five convolutional layers and three fully connected layers, with the stated pooling/normalization stages |
| Inception v1 | Parallel branches analyze features at different spatial scales | 1 × 1, reduced 3 × 3, reduced 5 × 5 and pooling branches joined by channel concatenation |
| ResNet | Learn a residual transformation and add a shortcut | A transformation branch and input shortcut meeting at a plus sign |
AlexNet's exam-relevant features include ReLU, GPU-based training, augmentation and dropout in the fully connected part. A parameter calculation must follow the given channel connectivity; a grouped implementation and a simplified fully connected-channel diagram need not have the same count.
","Inception uses 1 × 1 convolutions to mix channels and, when a smaller number of filters is chosen, reduce the channel depth before expensive spatial filters. Branch outputs must have compatible height and width for concatenation. Concatenation adds channel counts by joining their lists.
","ResNet computes a learned transformation F(x) and combines it with the shortcut:
","This is entrywise addition. The two shapes must match; a projection can adapt a shortcut when they do not. The shortcut supplies an additional route for information and gradients and makes identity-like mappings easy to represent.
","Key distinction: an Inception join concatenates feature maps; a residual join adds corresponding values. Their diagram symbols and output-channel calculations must therefore differ.
"],"kind":"theory","id":"tee-c8-s8","search":"Model answer: AlexNet, Inception and ResNet **Question.** Compare the main ideas of AlexNet, Inception and ResNet. Explain how to recognize their architecture diagrams.\n\n**Model answer.**\nAll three are CNN architectures, but their characteristic structures differ.\n\n\n\n\n| Architecture | Main idea | What to show in a diagram |\n| AlexNet | Successive convolutional feature extraction followed by dense classification | Five convolutional layers and three fully connected layers, with the stated pooling/normalization stages |\n| Inception v1 | Parallel branches analyze features at different spatial scales | 1 × 1, reduced 3 × 3, reduced 5 × 5 and pooling branches joined by channel concatenation |\n| ResNet | Learn a residual transformation and add a shortcut | A transformation branch and input shortcut meeting at a plus sign |\n\nAlexNet's exam-relevant features include ReLU, GPU-based training, augmentation and dropout in the fully connected part. A parameter calculation must follow the given channel connectivity; a grouped implementation and a simplified fully connected-channel diagram need not have the same count.\n\nInception uses 1 × 1 convolutions to mix channels and, when a smaller number of filters is chosen, reduce the channel depth before expensive spatial filters. Branch outputs must have compatible height and width for concatenation. Concatenation adds channel counts by joining their lists.\n\nResNet computes a learned transformation F(x) and combines it with the shortcut:\n\nThis is entrywise addition. The two shapes must match; a projection can adapt a shortcut when they do not. The shortcut supplies an additional route for information and gradients and makes identity-like mappings easy to represent.\n\n**Key distinction:** an Inception join concatenates feature maps; a residual join adds corresponding values. Their diagram symbols and output-channel calculations must therefore differ.\n"},{"title":"Applications of convolutional neural networks","blocks":["Definition. A CNN application combines a convolutional feature extractor with an output head designed for a particular prediction task. Shared local filters learn spatial patterns; the output head determines what is predicted and the loss determines how it is trained.
","| Task | Required output | What distinguishes it |
|---|---|---|
| Image classification | One label or class distribution for the image | Answers what category the whole image belongs to |
| Object detection | Object classes and bounding-box coordinates | Also identifies where each detected object is located |
| Semantic segmentation | A class label or distribution for each pixel | Produces a spatial label map rather than one image-wide class |
| Face recognition | Identity class or a learned face representation | Compares or classifies learned visual features |
A softmax image classifier alone does not produce boxes or pixel labels. Detection and segmentation require appropriate task heads, targets and objectives. In an applications question, explain the prediction required instead of listing model names without their purpose.
","Medical imaging: classify an image or use a segmentation model to identify regions of interest. The target may be an image-level label or a pixel mask, so the architecture and loss must match that choice.
","Autonomous perception: use camera images for tasks such as detecting pedestrians and vehicles or segmenting road regions. These outputs are inputs to a larger perception and decision system.
","Manufacturing: classify product images as defective or non-defective, or localize defects when position information is needed.
","Document analysis: classify cropped handwritten characters; a complete document-reading system must also locate and order text regions.
","Remote sensing: classify land-use regions or segment different surface categories in satellite images. Spatial context and texture can help distinguish classes.
","Question. Describe how a CNN can classify product photographs into three classes: acceptable, scratched and cracked. Explain the input, network stages, output activation, loss, training and evaluation.
","Step 1: define data and labels. Each example contains an image and one of the three class labels. Use separate training, validation and test sets, keeping near-duplicate photographs of the same item out of different splits. Apply the chosen model's input size and scaling consistently.
","Step 2: extract spatial features. Convolutional layers calculate local responses using shared kernels. Nonlinear activations allow more complex combinations of those responses. Further convolution stages combine local patterns; pooling may reduce spatial dimensions.
","Step 3: produce three class scores. Convert the final representation into a feature vector, using flattening or an appropriate global pooling operation, and apply a dense layer with three outputs. These raw outputs are logits.
","Step 4: calculate class probabilities and loss. For mutually exclusive classes, use softmax and categorical cross-entropy. A one-hot label selects the probability of the correct class.
","Step 5: train the parameters. Run forward propagation, evaluate the loss, backpropagate derivatives and update the trainable convolutional and dense parameters using the selected optimizer. Use validation results for hyperparameters and stopping decisions.
","Step 6: evaluate the selected model. Assess the final model on held-out photographs. Report overall classification performance and inspect which defects are confused. With limited data, a pretrained CNN can provide features and selected layers can be fine-tuned.
","Numerical extension. If the last pooled feature vector has 64 components, how many parameters does the three-class dense head have, including bias?
","Step 1: each output receives 64 weights, so three outputs require 64 × 3 = 192 weights.
","Step 2: each of the three outputs has one bias, giving 3 biases.
","Final answer: 195 trainable parameters in this head. Softmax adds no trainable parameters. The backbone's parameters are counted separately.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s9","search":"Applications of convolutional neural networks **Definition.** A CNN application combines a convolutional feature extractor with an output head designed for a particular prediction task. Shared local filters learn spatial patterns; the output head determines what is predicted and the loss determines how it is trained.\n\n### Classification, detection and segmentation\n| Task | Required output | What distinguishes it |\n| Image classification | One label or class distribution for the image | Answers what category the whole image belongs to |\n| Object detection | Object classes and bounding-box coordinates | Also identifies where each detected object is located |\n| Semantic segmentation | A class label or distribution for each pixel | Produces a spatial label map rather than one image-wide class |\n| Face recognition | Identity class or a learned face representation | Compares or classifies learned visual features |\n\nA softmax image classifier alone does not produce boxes or pixel labels. Detection and segmentation require appropriate task heads, targets and objectives. In an applications question, explain the prediction required instead of listing model names without their purpose.\n\n### Examples across domains\n**Medical imaging:** classify an image or use a segmentation model to identify regions of interest. The target may be an image-level label or a pixel mask, so the architecture and loss must match that choice.\n\n**Autonomous perception:** use camera images for tasks such as detecting pedestrians and vehicles or segmenting road regions. These outputs are inputs to a larger perception and decision system.\n\n**Manufacturing:** classify product images as defective or non-defective, or localize defects when position information is needed.\n\n**Document analysis:** classify cropped handwritten characters; a complete document-reading system must also locate and order text regions.\n\n**Remote sensing:** classify land-use regions or segment different surface categories in satellite images. Spatial context and texture can help distinguish classes.\n\n### Complete exam-style application answer\n**Question. Describe how a CNN can classify product photographs into three classes: acceptable, scratched and cracked. Explain the input, network stages, output activation, loss, training and evaluation.**\n\n**Step 1: define data and labels.** Each example contains an image and one of the three class labels. Use separate training, validation and test sets, keeping near-duplicate photographs of the same item out of different splits. Apply the chosen model's input size and scaling consistently.\n\n**Step 2: extract spatial features.** Convolutional layers calculate local responses using shared kernels. Nonlinear activations allow more complex combinations of those responses. Further convolution stages combine local patterns; pooling may reduce spatial dimensions.\n\n**Step 3: produce three class scores.** Convert the final representation into a feature vector, using flattening or an appropriate global pooling operation, and apply a dense layer with three outputs. These raw outputs are logits.\n\n**Step 4: calculate class probabilities and loss.** For mutually exclusive classes, use softmax and categorical cross-entropy. A one-hot label selects the probability of the correct class.\n\n\n**Step 5: train the parameters.** Run forward propagation, evaluate the loss, backpropagate derivatives and update the trainable convolutional and dense parameters using the selected optimizer. Us"},{"title":"Reading LeNet and AlexNet layer diagrams","blocks":["A layer diagram describes two different things: the size of the data moving through the network and the trainable operations that transform that data. A number such as 9,216 beside a flatten operation describes data length; it does not automatically mean another dense layer with 9,216 neurons.
","The classroom LeNet layout begins with a 32 × 32 grayscale image. A valid 5 × 5 convolution with six filters produces 28 × 28 × 6. A 2 × 2 pooling operation with stride 2 produces 14 × 14 × 6. The next valid 5 × 5 convolution produces 10 × 10 × 16, and pooling reduces this to 5 × 5 × 16. A 5 × 5 convolution across those 16 channels produces 1 × 1 × 120. The final representation passes through 84 units and then ten class outputs.
","| Stage | What happens | Output shape |
|---|---|---|
| Input | One grayscale image | 32 × 32 × 1 |
| C1 | Six 5 × 5 filters, stride 1, no padding | 28 × 28 × 6 |
| S2 | 2 × 2 average pooling, stride 2 | 14 × 14 × 6 |
| C3 | Sixteen 5 × 5 filters, stride 1, no padding | 10 × 10 × 16 |
| S4 | 2 × 2 average pooling, stride 2 | 5 × 5 × 16 |
| C5 | 120 filters covering the complete 5 × 5 spatial map | 1 × 1 × 120 |
| F6 | Dense transformation | 84 values |
| Classifier | Modern classroom softmax classifier | Ten probabilities |
The original LeNet-5 used details such as partial channel connectivity, trainable subsampling and an RBF output. A modern full-channel implementation with ordinary pooling and softmax is a different parameter-count convention. For a numerical, follow the question's diagram and state the convention. The complete LeNet parameter numerical in the solved-paper section uses the modern classroom convention.
","For the faculty's 227 × 227 × 3 classroom input, the main shape sequence is 55 × 55 × 96 → 27 × 27 × 96 → 27 × 27 × 256 → 13 × 13 × 256 → 13 × 13 × 384 → 13 × 13 × 384 → 13 × 13 × 256 → 6 × 6 × 256. Flattening gives 9,216 values; the three dense outputs are 4,096, 4,096 and 1,000. There is no extra learned 9,216-unit layer between flattening and the first 4,096-unit layer.
","Question (practice, 3 marks). An AlexNet-style first layer receives a 227 × 227 RGB image and uses 96 filters of size 11 × 11, stride 4 and no padding. Find its output shape, trainable parameters including bias, and the number of output activation values.
","Input height and width = 227; input channels = 3; kernel = 11; stride = 4; padding = 0; output filters = 96. One bias is used per filter.
","The three-dimensional output shape, parameter count and activation count.
","The output-size formula counts valid window positions. Parameter count counts stored filter coefficients, reused at every position. Activation count counts the values produced for one image.
","Here N is input side length, P is padding per side, K is kernel side length, S is stride, and C denotes channel count. The symbol P in the output formula means padding; the labelled quantity P_conv means parameters.
","Step 1: calculate one spatial side.
","Step 2: attach the output depth. There is one output map per filter, so the output is 55 × 55 × 96.
","Step 3: count the weights in one filter.
","Step 4: add one bias and multiply by 96 filters.
","Step 5: count activation values for one image.
","Final answer: output = 55 × 55 × 96; parameters = 34,944; activations = 290,400. The weights are reused, so parameters and activations are different counts. The full faculty-style ungrouped AlexNet table totals 62,378,344 parameters; the original grouped model uses different channel connectivity.
","Yaad rakho: shape = kitne answers nikle; parameters = kitne numbers seekhe.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s10","search":"Reading LeNet and AlexNet layer diagrams \n**A layer diagram describes two different things:** the size of the data moving through the network and the trainable operations that transform that data. A number such as 9,216 beside a flatten operation describes data length; it does not automatically mean another dense layer with 9,216 neurons.\n\n### LeNet: follow the image as it becomes a class prediction\n\nThe classroom LeNet layout begins with a 32 × 32 grayscale image. A valid 5 × 5 convolution with six filters produces 28 × 28 × 6. A 2 × 2 pooling operation with stride 2 produces 14 × 14 × 6. The next valid 5 × 5 convolution produces 10 × 10 × 16, and pooling reduces this to 5 × 5 × 16. A 5 × 5 convolution across those 16 channels produces 1 × 1 × 120. The final representation passes through 84 units and then ten class outputs.\n\n| Stage | What happens | Output shape |\n| Input | One grayscale image | 32 × 32 × 1 |\n| C1 | Six 5 × 5 filters, stride 1, no padding | 28 × 28 × 6 |\n| S2 | 2 × 2 average pooling, stride 2 | 14 × 14 × 6 |\n| C3 | Sixteen 5 × 5 filters, stride 1, no padding | 10 × 10 × 16 |\n| S4 | 2 × 2 average pooling, stride 2 | 5 × 5 × 16 |\n| C5 | 120 filters covering the complete 5 × 5 spatial map | 1 × 1 × 120 |\n| F6 | Dense transformation | 84 values |\n| Classifier | Modern classroom softmax classifier | Ten probabilities |\n\nThe original LeNet-5 used details such as partial channel connectivity, trainable subsampling and an RBF output. A modern full-channel implementation with ordinary pooling and softmax is a different parameter-count convention. For a numerical, follow the question's diagram and state the convention. The complete LeNet parameter numerical in the solved-paper section uses the modern classroom convention.\n\n### AlexNet: identify the flatten step correctly\n\n\n\nFor the faculty's 227 × 227 × 3 classroom input, the main shape sequence is 55 × 55 × 96 → 27 × 27 × 96 → 27 × 27 × 256 → 13 × 13 × 256 → 13 × 13 × 384 → 13 × 13 × 384 → 13 × 13 × 256 → 6 × 6 × 256. Flattening gives 9,216 values; the three dense outputs are 4,096, 4,096 and 1,000. There is no extra learned 9,216-unit layer between flattening and the first 4,096-unit layer.\n\n**Question (practice, 3 marks). An AlexNet-style first layer receives a 227 × 227 RGB image and uses 96 filters of size 11 × 11, stride 4 and no padding. Find its output shape, trainable parameters including bias, and the number of output activation values.**\n\n### Given\nInput height and width = 227; input channels = 3; kernel = 11; stride = 4; padding = 0; output filters = 96. One bias is used per filter.\n\n### To find\nThe three-dimensional output shape, parameter count and activation count.\n\n### Formula — which one and why\nThe output-size formula counts valid window positions. Parameter count counts stored filter coefficients, reused at every position. Activation count counts the values produced for one image.\n\n\n\nHere N is input side length, P is padding per side, K is kernel side length, S is stride, and C denotes channel count. The symbol P in the output formula means padding; the labelled quantity P_conv means parameters.\n\n### Solution — step by step\n**Step 1: calculate one spatial side.**\n\n\n**Step 2: attach the output depth.** There is one output map per filter, so the output is **55 × 55 × 96**.\n\n**Step 3: count the weights in one filter.**\n\n\n**Step 4: add one bias and multiply by 96 filters.**\n\n\n**Step 5: count activation values for one image.**\n\n\n**Final answer:** output = 55 × 55 ×"},{"title":"Top-1 and top-5 error: what an ImageNet result means","blocks":["Top-1 prediction is the class with the highest score. A top-1 mistake occurs when that class is wrong. Top-5 prediction is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models.
","For an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures.
","Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.
","Total images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes.
","The percentage of mistakes under each rule.
","Subtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage.
","N means total evaluated images; C_k means images whose correct class appears among the top k predictions.
","Step 1: top-1 mistakes. 100 − 78 = 22.
","Step 2: top-1 error percentage.
","Step 3: top-5 mistakes. 100 − 92 = 8.
","Step 4: top-5 error percentage.
","Final answer: top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions.
","Yaad rakho: top-five mein sahi naam mil gaya toh top-five success.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s11","search":"Top-1 and top-5 error: what an ImageNet result means \n**Top-1 prediction** is the class with the highest score. A top-1 mistake occurs when that class is wrong. **Top-5 prediction** is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models.\n\nFor an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures.\n\n**Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.**\n\n### Given\nTotal images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes.\n\n### To find\nThe percentage of mistakes under each rule.\n\n### Formula — which one and why\nSubtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage.\n\nN means total evaluated images; C_k means images whose correct class appears among the top k predictions.\n\n### Solution — step by step\n**Step 1: top-1 mistakes.** 100 − 78 = 22.\n\n**Step 2: top-1 error percentage.**\n\n\n**Step 3: top-5 mistakes.** 100 − 92 = 8.\n\n**Step 4: top-5 error percentage.**\n\n\n**Final answer:** top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions.\n\n**Yaad rakho:** top-five mein sahi naam mil gaya toh top-five success.\n"},{"title":"VGG16: small filters in a regular sequence","blocks":["Definition. VGG16 is a convolutional architecture with 13 convolutional layers and three dense layers. Its standard convolution blocks contain 2, 2, 3, 3 and 3 convolutions. Pooling, activations and flattening are not counted in the name's 16 weight layers.
","Each convolution uses a 3 × 3 kernel, stride 1 and padding 1, preserving height and width. Each block ends with 2 × 2 max pooling of stride 2. The final classifier is 4,096 → 4,096 → 1,000 for the original 1,000-class task.
","| Block | Convolutions | Channels after its convolutions | Size after its pooling |
|---|---|---|---|
| 1 | Two | 64 | 112 × 112 × 64 |
| 2 | Two | 128 | 56 × 56 × 128 |
| 3 | Three | 256 | 28 × 28 × 256 |
| 4 | Three | 512 | 14 × 14 × 512 |
| 5 | Three | 512 | 7 × 7 × 512 |
This table starts with 224 × 224 × 3. The flattened length is 7 × 7 × 512 = 25,088. Flattening has zero learned parameters. Stacking small filters provides repeated nonlinear transformations while progressively combining spatial information. A limitation is the large dense classifier and its memory cost.
","Question (practice, 3 marks). VGG16's first block contains two same-padded 3 × 3 convolutions with 64 output filters each, followed by 2 × 2 max pooling with stride 2. Input is 224 × 224 × 3. Find both convolution parameter counts including bias and the pooled output shape.
","Conv1: 3 input channels, 64 output channels. Conv2: 64 input channels, 64 output channels. Both have kernel 3, stride 1 and padding 1. Pooling has kernel 2, stride 2 and no padding.
","Trainable parameters in each convolution and the output after pooling.
","Use each layer's own input depth; the second layer receives the first layer's 64 maps. Use the spatial-output formula for pooling. Max pooling introduces no trainable parameters.
","Step 1: first convolution. Each filter has 3 × 3 × 3 = 27 weights and one bias.
","Step 2: second convolution. Each filter now spans 64 channels, giving 3 × 3 × 64 = 576 weights.
","Step 3: add parameters.
","Step 4: calculate pooled side length. The same-padded convolutions retained side length 224.
","Final answer: 1,792 and 36,928 parameters; total = 38,720; pooled shape = 112 × 112 × 64.
","Exam convention: use the architecture explicitly supplied in a calculation. For the standard VGG16 architecture, remember 2–2–3–3–3 convolutions and 2 × 2 pooling; 25,088 is the flattened feature length.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s12","search":"VGG16: small filters in a regular sequence \n\n\n**Definition.** VGG16 is a convolutional architecture with 13 convolutional layers and three dense layers. Its standard convolution blocks contain 2, 2, 3, 3 and 3 convolutions. Pooling, activations and flattening are not counted in the name's 16 weight layers.\n\nEach convolution uses a 3 × 3 kernel, stride 1 and padding 1, preserving height and width. Each block ends with 2 × 2 max pooling of stride 2. The final classifier is 4,096 → 4,096 → 1,000 for the original 1,000-class task.\n\n| Block | Convolutions | Channels after its convolutions | Size after its pooling |\n| 1 | Two | 64 | 112 × 112 × 64 |\n| 2 | Two | 128 | 56 × 56 × 128 |\n| 3 | Three | 256 | 28 × 28 × 256 |\n| 4 | Three | 512 | 14 × 14 × 512 |\n| 5 | Three | 512 | 7 × 7 × 512 |\n\nThis table starts with 224 × 224 × 3. The flattened length is 7 × 7 × 512 = 25,088. Flattening has zero learned parameters. Stacking small filters provides repeated nonlinear transformations while progressively combining spatial information. A limitation is the large dense classifier and its memory cost.\n\n**Question (practice, 3 marks). VGG16's first block contains two same-padded 3 × 3 convolutions with 64 output filters each, followed by 2 × 2 max pooling with stride 2. Input is 224 × 224 × 3. Find both convolution parameter counts including bias and the pooled output shape.**\n\n### Given\nConv1: 3 input channels, 64 output channels. Conv2: 64 input channels, 64 output channels. Both have kernel 3, stride 1 and padding 1. Pooling has kernel 2, stride 2 and no padding.\n\n### To find\nTrainable parameters in each convolution and the output after pooling.\n\n### Formula — which one and why\n\nUse each layer's own input depth; the second layer receives the first layer's 64 maps. Use the spatial-output formula for pooling. Max pooling introduces no trainable parameters.\n\n### Solution — step by step\n**Step 1: first convolution.** Each filter has 3 × 3 × 3 = 27 weights and one bias.\n\n\n**Step 2: second convolution.** Each filter now spans 64 channels, giving 3 × 3 × 64 = 576 weights.\n\n\n**Step 3: add parameters.**\n\n\n**Step 4: calculate pooled side length.** The same-padded convolutions retained side length 224.\n\n\n**Final answer:** 1,792 and 36,928 parameters; total = 38,720; pooled shape = **112 × 112 × 64**.\n\n**Exam convention:** use the architecture explicitly supplied in a calculation. For the standard VGG16 architecture, remember **2–2–3–3–3 convolutions and 2 × 2 pooling**; 25,088 is the flattened feature length.\n"},{"title":"Global average pooling: one average for each channel","blocks":["Definition. Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number.
","The letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier.
","Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = and channel B = . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.
H = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs.
","The GAP result and dense-head parameter count.
","Use one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias.
","Step 1: add channel A. 1 + 3 + 5 + 7 = 16.
","Step 2: average channel A. 16 ÷ 4 = 4.
","Step 3: add channel B. 2 + 4 + 6 + 8 = 20.
","Step 4: average channel B. 20 ÷ 4 = 5. Thus GAP outputs [4,5].
","Step 5: count the classifier's weights. 2 inputs × 3 outputs = 6 weights.
","Step 6: add three biases. 6 + 3 = 9 parameters. GAP itself contributes zero.
","Final answer: GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters.
","Yaad rakho: ek channel ka poora average; channels ko aapas mein mat milao.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s13","search":"Global average pooling: one average for each channel \n\n\n**Definition.** Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number.\n\n\n\nThe letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier.\n\n**Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = and channel B = . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.**\n\n### Given\nH = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs.\n\n### To find\nThe GAP result and dense-head parameter count.\n\n### Formula — which one and why\nUse one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias.\n\n\n### Solution — step by step\n**Step 1: add channel A.** 1 + 3 + 5 + 7 = 16.\n\n**Step 2: average channel A.** 16 ÷ 4 = 4.\n\n**Step 3: add channel B.** 2 + 4 + 6 + 8 = 20.\n\n**Step 4: average channel B.** 20 ÷ 4 = 5. Thus GAP outputs **[4,5]**.\n\n**Step 5: count the classifier's weights.** 2 inputs × 3 outputs = 6 weights.\n\n**Step 6: add three biases.** 6 + 3 = **9 parameters**. GAP itself contributes zero.\n\n**Final answer:** GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters.\n\n**Yaad rakho:** ek channel ka poora average; channels ko aapas mein mat milao.\n"},{"title":"The faculty's 480-channel bottleneck numerical","blocks":["Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.
","Both paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters.
","MACs for the direct path, both bottleneck stages, their total and percentage saved.
","M is the MAC count. H_oW_oC_o counts output values; K²C_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck.
","Step 1: direct output count. 14 × 14 × 48 = 9,408.
","Step 2: products per direct output. 5 × 5 × 480 = 12,000.
","Step 3: direct MACs.
","Step 4: reduction-layer MACs. Each of 14 × 14 × 16 outputs combines 480 values.
","Step 5: reduced spatial-layer MACs. Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values.
","Step 6: add both bottleneck costs.
","Step 7: calculate the saving.
","Final answer: direct = 112.896 million MACs; bottleneck = 5.26848 million MACs; saving ≈ 95.33%. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s14","search":"The faculty's 480-channel bottleneck numerical \n\n\n**Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.**\n\n### Given\nBoth paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters.\n\n### To find\nMACs for the direct path, both bottleneck stages, their total and percentage saved.\n\n### Formula — which one and why\n\nM is the MAC count. H_oW_oC_o counts output values; K²C_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck.\n\n\n### Solution — step by step\n**Step 1: direct output count.** 14 × 14 × 48 = 9,408.\n\n**Step 2: products per direct output.** 5 × 5 × 480 = 12,000.\n\n**Step 3: direct MACs.**\n\n\n**Step 4: reduction-layer MACs.** Each of 14 × 14 × 16 outputs combines 480 values.\n\n\n**Step 5: reduced spatial-layer MACs.** Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values.\n\n\n**Step 6: add both bottleneck costs.**\n\n\n**Step 7: calculate the saving.**\n\n\n\n**Final answer:** direct = **112.896 million MACs**; bottleneck = **5.26848 million MACs**; saving ≈ **95.33%**. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone.\n"},{"title":"Batch normalization inside a residual network","blocks":["Definition. Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training.
","For one feature with m values in a training batch:
","The batch mean μ_B is the centre; batch variance σ²_B measures squared spread. Epsilon ε is a small positive stability constant inside the square root. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch.
","Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.
","x₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5.
","Batch mean, variance, normalized values and scale-and-shift outputs.
","Use the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale.
","Step 1: mean.
","Step 2: deviations. 2 − 3 = −1 and 4 − 3 = 1.
","Step 3: squared deviations and variance.
","Step 4: stable denominator.
","Step 5: normalize separately.
","Step 6: scale the first normalized value, then shift it.
","Step 7: scale the second normalized value, then shift it.
","Final answer: output ≈ [−1.49999,2.49999]. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here.
","Yaad rakho: centre hatao → spread se divide → learned scale lagao → shift add karo.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s15","search":"Batch normalization inside a residual network \n**Definition.** Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training.\n\nFor one feature with m values in a training batch:\n\n\n\n\n\nThe batch mean μ_B is the centre; batch variance σ²_B measures squared spread. Epsilon ε is a small positive stability constant **inside the square root**. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch.\n\n**Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.**\n\n### Given\nx₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5.\n\n### To find\nBatch mean, variance, normalized values and scale-and-shift outputs.\n\n### Formula — which one and why\nUse the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale.\n\n### Solution — step by step\n**Step 1: mean.**\n\n\n**Step 2: deviations.** 2 − 3 = −1 and 4 − 3 = 1.\n\n**Step 3: squared deviations and variance.**\n\n\n**Step 4: stable denominator.**\n\n\n**Step 5: normalize separately.**\n\n\n\n**Step 6: scale the first normalized value, then shift it.**\n\n\n**Step 7: scale the second normalized value, then shift it.**\n\n\n**Final answer:** output ≈ **[−1.49999,2.49999]**. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here.\n\n**Yaad rakho:** centre hatao → spread se divide → learned scale lagao → shift add karo.\n"},{"title":"Residual-block order and architecture qualifications","blocks":["The original post-activation basic block has this sequence: convolution → BN → ReLU → convolution → BN → add shortcut → ReLU. Thus the second branch convolution is not followed by a separate ReLU before the addition in this particular block. A pre-activation ResNet is another design; use the stated diagram.
","An identity shortcut has no learned parameters. A projection shortcut changes the shape using learned weights. Original ImageNet ResNets include an initial max-pooling operation; a basic residual block does not need a max-pooling layer inside it. ResNet-18/34 use two-convolution basic blocks, whereas deeper standard variants use bottleneck blocks. Avoid treating every ResNet as the same two-layer diagram.
","The main advantage is easier optimization of deep networks: shortcuts provide an additional information and gradient path. This reduces the degradation problem but does not prove immunity to every vanishing/exploding-gradient case. Large networks still need compute, memory, suitable data and regularization. Reusing early features can help; it does not guarantee that every deeper model generalizes better.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s16","search":"Residual-block order and architecture qualifications \nThe original post-activation basic block has this sequence: **convolution → BN → ReLU → convolution → BN → add shortcut → ReLU**. Thus the second branch convolution is not followed by a separate ReLU before the addition in this particular block. A pre-activation ResNet is another design; use the stated diagram.\n\n\n\nAn identity shortcut has no learned parameters. A projection shortcut changes the shape using learned weights. Original ImageNet ResNets include an initial max-pooling operation; a basic residual block does not need a max-pooling layer inside it. ResNet-18/34 use two-convolution basic blocks, whereas deeper standard variants use bottleneck blocks. Avoid treating every ResNet as the same two-layer diagram.\n\nThe main advantage is easier optimization of deep networks: shortcuts provide an additional information and gradient path. This reduces the degradation problem but does not prove immunity to every vanishing/exploding-gradient case. Large networks still need compute, memory, suitable data and regularization. Reusing early features can help; it does not guarantee that every deeper model generalizes better.\n"},{"title":"Choosing frozen layers from dataset size and similarity","blocks":["Early image layers commonly learn reusable edges and textures. Later layers become more specialized to source-task categories. “Earlier” means closer to the input, regardless of whether a drawing places those layers at the top or bottom.
","| Target dataset | Sensible starting choice | Reason |
|---|---|---|
| Small and similar | Freeze the backbone; train a new head first | Useful features already exist, while few examples make overfitting likely |
| Large and similar | Train a new head, then fine-tune more layers if validation improves | More data can support more trainable parameters |
| Small and different | Try a more relevant pretrained model or fixed earlier/intermediate features; tune cautiously | Later source features may be unsuitable, but small data still makes extensive tuning risky |
| Large and different | Compare broad fine-tuning with training from scratch | Sufficient data permits stronger adaptation; the value of source features must be tested |
These are starting strategies, not universal rules. Fine-tuning an early layer means updating its pretrained weights, not automatically learning it “from scratch”. With very little data, simply unfreezing early layers because the task differs can overfit. Choose changes using validation performance and apply the required preprocessing consistently.
","Example. With 100 labelled microscopic images, an ImageNet model may provide reusable low-level patterns, but its animal/object classifier is inappropriate. Replace the head, test fixed representations and monitor validation before increasing the number of trainable layers. With a large representative image dataset, broader fine-tuning becomes a more reasonable candidate.
","Yaad rakho: data kam toh kam weights hilao; task alag toh features ki suitability pehle dekho.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s17","search":"Choosing frozen layers from dataset size and similarity \nEarly image layers commonly learn reusable edges and textures. Later layers become more specialized to source-task categories. “Earlier” means closer to the input, regardless of whether a drawing places those layers at the top or bottom.\n\n| Target dataset | Sensible starting choice | Reason |\n| Small and similar | Freeze the backbone; train a new head first | Useful features already exist, while few examples make overfitting likely |\n| Large and similar | Train a new head, then fine-tune more layers if validation improves | More data can support more trainable parameters |\n| Small and different | Try a more relevant pretrained model or fixed earlier/intermediate features; tune cautiously | Later source features may be unsuitable, but small data still makes extensive tuning risky |\n| Large and different | Compare broad fine-tuning with training from scratch | Sufficient data permits stronger adaptation; the value of source features must be tested |\n\nThese are starting strategies, not universal rules. Fine-tuning an early layer means updating its pretrained weights, not automatically learning it “from scratch”. With very little data, simply unfreezing early layers because the task differs can overfit. Choose changes using validation performance and apply the required preprocessing consistently.\n\n**Example.** With 100 labelled microscopic images, an ImageNet model may provide reusable low-level patterns, but its animal/object classifier is inappropriate. Replace the head, test fixed representations and monitor validation before increasing the number of trainable layers. With a large representative image dataset, broader fine-tuning becomes a more reasonable candidate.\n\n**Yaad rakho:** data kam toh kam weights hilao; task alag toh features ki suitability pehle dekho.\n"},{"title":"Four ways to reuse a pretrained model","blocks":["Definition. A pretrained model has already learned parameters from an earlier training task. We can reuse its complete classifier, reuse its feature extractor, or use its learned weights as the starting point for further training. The choice determines which output labels are available and which parameters change.
","First identify two parts. The backbone converts an input image into useful features. The head converts those features into the required prediction, such as class scores. A new task can reuse the backbone while replacing the head.
","Yaad rakhne ka cue: Purane model ka poora answer use karna, sirf features lena, ya uske weights ko aage train karna—yeh teen decisions alag hain.
","The faculty reference divides these choices into the following four uses. The two feature-extractor uses differ in whether features are saved separately or computed inside the new model.
","| Use | What happens to the pretrained part? | What learns for the target task? | Concrete example |
|---|---|---|---|
| Direct classifier | Keep the complete model and its existing output head. Run it on new inputs. | Nothing is updated during ordinary prediction. | A model already trained to classify cats and dogs labels new cat/dog photographs. It can do this because the required labels match its existing outputs. |
| Standalone feature extractor | Run images through the fixed backbone and save the resulting feature vectors. A feature vector is an ordered list of numbers describing the image. | Train a separate classifier using these saved vectors. | Extract one vector per product photograph, then train a classifier to distinguish acceptable, scratched and cracked products. The extractor stays unchanged. |
| Integrated feature extractor | Place the pretrained backbone inside the new network and freeze it. Add a new head. The backbone calculates features whenever the new model runs. | Train the new head while the backbone's weights remain fixed. | Use a frozen image backbone followed by a new three-output head for acceptable, scratched and cracked products. The complete image-to-class pipeline is one model. |
| Weight initialization followed by fine-tuning | Start from the pretrained weights, then allow selected pretrained layers to update. Replace or adapt the head as required. | Train the new head and the unfrozen part of the backbone. | After first training a product-defect head, unfreeze the last backbone block and train it with a smaller learning rate to adapt to the factory's photographs. |
A cat/dog head has two outputs with learned meanings: cat and dog. Giving it a photograph of a scratched product does not create a new output called “scratched.” For the product task, use an appropriate new head and train it on labeled product examples. Direct reuse of the original classifier is suitable only when its output meanings and preprocessing match the required task. When the task and data distribution are unchanged, this is ordinary inference rather than adaptation to a new task.
","Both methods can keep the backbone frozen. In the standalone method, feature vectors are calculated first and saved as a separate dataset; a later classifier reads that dataset. In the integrated method, each image passes through the backbone and head within one pipeline. Freezing means that the backbone parameters receive no training updates; it does not mean that the backbone stops performing calculations or that its stored weights disappear.
","It is useful when labeled target data are limited and a pretrained model has learned representations relevant to the target task. It can reduce training time and the amount of target data needed. Improvement is not guaranteed: unrelated source features, unsuitable preprocessing or excessive fine-tuning on a small dataset can hurt performance. Choose the frozen or fine-tuned configuration using validation data, then evaluate the chosen model on the held-out test set.
","Exam answer structure: define transfer learning → identify the backbone and head → explain the selected reuse mode → state which weights remain fixed and which learn → give a matching example and a limitation.
"],"question":[],"answer":[],"kind":"read","id":"tee-c8-s18","search":"Four ways to reuse a pretrained model \n**Definition.** A pretrained model has already learned parameters from an earlier training task. We can reuse its complete classifier, reuse its feature extractor, or use its learned weights as the starting point for further training. The choice determines which output labels are available and which parameters change.\n\n**First identify two parts.** The **backbone** converts an input image into useful features. The **head** converts those features into the required prediction, such as class scores. A new task can reuse the backbone while replacing the head.\n\n**Yaad rakhne ka cue:** Purane model ka poora answer use karna, sirf features lena, ya uske weights ko aage train karna—yeh teen decisions alag hain.\n\nThe faculty reference divides these choices into the following four uses. The two feature-extractor uses differ in whether features are saved separately or computed inside the new model.\n\n| Use | What happens to the pretrained part? | What learns for the target task? | Concrete example |\n| Direct classifier | Keep the complete model and its existing output head. Run it on new inputs. | Nothing is updated during ordinary prediction. | A model already trained to classify cats and dogs labels new cat/dog photographs. It can do this because the required labels match its existing outputs. |\n| Standalone feature extractor | Run images through the fixed backbone and save the resulting feature vectors. A feature vector is an ordered list of numbers describing the image. | Train a separate classifier using these saved vectors. | Extract one vector per product photograph, then train a classifier to distinguish acceptable, scratched and cracked products. The extractor stays unchanged. |\n| Integrated feature extractor | Place the pretrained backbone inside the new network and freeze it. Add a new head. The backbone calculates features whenever the new model runs. | Train the new head while the backbone's weights remain fixed. | Use a frozen image backbone followed by a new three-output head for acceptable, scratched and cracked products. The complete image-to-class pipeline is one model. |\n| Weight initialization followed by fine-tuning | Start from the pretrained weights, then allow selected pretrained layers to update. Replace or adapt the head as required. | Train the new head and the unfrozen part of the backbone. | After first training a product-defect head, unfreeze the last backbone block and train it with a smaller learning rate to adapt to the factory's photographs. |\n\n### Why the existing class labels matter\n\nA cat/dog head has two outputs with learned meanings: cat and dog. Giving it a photograph of a scratched product does not create a new output called “scratched.” For the product task, use an appropriate new head and train it on labeled product examples. Direct reuse of the original classifier is suitable only when its output meanings and preprocessing match the required task. When the task and data distribution are unchanged, this is ordinary inference rather than adaptation to a new task.\n\n### Standalone versus integrated features\n\nBoth methods can keep the backbone frozen. In the standalone method, feature vectors are calculated first and saved as a separate dataset; a later classifier reads that dataset. In the integrated method, each image passes through the backbone and head within one pipeline. Freezing means that the backbone parameters receive no training updates; it does not mean that "}]},{"n":9,"title":"Recurrent neural networks","sections":[{"title":"Complete the entire sequence chapter","blocks":["All sections in this chapter belong to the TEE preparation route. Learn vanilla RNNs first, then the shared-gradient calculation, then LSTM and GRU. Do not skip the gates or BPTT because the MT2 route ends earlier.
","Definition. A recurrent neural network (RNN) is a neural architecture that models sequential data by updating a hidden state from the current input and the previous state, using shared parameters across time steps.
","The state hₜ is a learned representation of information available up to time t. It is not a guaranteed lossless record of the entire sequence. The initial state is supplied, initialized to zero or learned, according to the model or question.
","In language, audio and time series, the meaning of an observation can depend on earlier observations and their order. A basic feedforward network treating each input independently has no recurrent state. An RNN allows the output at a time step to depend on previous processing through hₜ₋₁.
","The unrolled cells use the same Wₓₕ, Wₕₕ and bₕ at every time step. A longer input sequence increases the number of evaluations, not the number of independently learned copies of these parameters. Each new hidden state must be calculated before it is used by the next step.
","For TEE, study the state equation, time-step computation, shared parameters and sequence-to-output configurations, then BPTT, vanishing/exploding gradients, LSTM and GRU. All are part of Unit 5. The faculty update received on 12 October focuses MT2 on Units 3 and 4; this complete recurrent-network chapter is TEE preparation.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s1","search":"What makes a sequence different **Definition.** A recurrent neural network (RNN) is a neural architecture that models sequential data by updating a hidden state from the current input and the previous state, using shared parameters across time steps.\n\nThe state hₜ is a learned representation of information available up to time t. It is not a guaranteed lossless record of the entire sequence. The initial state is supplied, initialized to zero or learned, according to the model or question.\n\n### Why order matters\nIn language, audio and time series, the meaning of an observation can depend on earlier observations and their order. A basic feedforward network treating each input independently has no recurrent state. An RNN allows the output at a time step to depend on previous processing through hₜ₋₁.\n\n\n\n### Parameter sharing\nThe unrolled cells use the same Wₓₕ, Wₕₕ and bₕ at every time step. A longer input sequence increases the number of evaluations, not the number of independently learned copies of these parameters. Each new hidden state must be calculated before it is used by the next step.\n\n### MT2 interpretation\nFor TEE, study the state equation, time-step computation, shared parameters and sequence-to-output configurations, then BPTT, vanishing/exploding gradients, LSTM and GRU. All are part of Unit 5. The faculty update received on 12 October focuses MT2 on Units 3 and 4; this complete recurrent-network chapter is TEE preparation.\n"},{"title":"Model answer: why recurrent networks use a hidden state","question":["Question. What is a recurrent neural network? Explain how a hidden state allows it to process a sequence and distinguish it from a feedforward network.
"],"answer":["An RNN processes ordered inputs using a hidden state that is carried from one time step to the next. The current state combines information from the new input with information retained from earlier inputs. The same recurrent weights are reused across time steps.
","For a simple tanh RNN:
","| Symbol | Meaning |
|---|---|
| xₜ | Input at the current time step |
| hₜ₋₁ | Hidden state passed from the previous step |
| Wxh | Weights connecting the current input to the hidden state |
| Whh | Weights connecting the previous state to the current state |
| bₕ | Hidden-state bias |
| hₜ | New state after the activation |
For words in a sentence, earlier words affect the state used when the next word arrives. Changing the order can therefore change the final representation. The state is a learned summary; it is not a guaranteed exact copy of every earlier input.
","A feedforward network maps inputs through its layers without this recurrent state transition. An RNN can accept a sequence, reuse its parameters at each step and produce either an output at each step or an output from a final state, depending on the task. Examples include sentiment classification, next-word prediction and time-series modeling.
","Unrolling an RNN draws its repeated use at t = 1, 2, 3 and so on. Those boxes share parameters; drawing three time steps does not create three independently learned copies of Whh.
"],"kind":"theory","id":"tee-c9-s2","search":"Model answer: why recurrent networks use a hidden state **Question.** What is a recurrent neural network? Explain how a hidden state allows it to process a sequence and distinguish it from a feedforward network.\n\n**Model answer.**\nAn RNN processes ordered inputs using a hidden state that is carried from one time step to the next. The current state combines information from the new input with information retained from earlier inputs. The same recurrent weights are reused across time steps.\n\n\n\n\nFor a simple tanh RNN:\n\n| Symbol | Meaning |\n| xₜ | Input at the current time step |\n| hₜ₋₁ | Hidden state passed from the previous step |\n| Wxh | Weights connecting the current input to the hidden state |\n| Whh | Weights connecting the previous state to the current state |\n| bₕ | Hidden-state bias |\n| hₜ | New state after the activation |\n\nFor words in a sentence, earlier words affect the state used when the next word arrives. Changing the order can therefore change the final representation. The state is a learned summary; it is not a guaranteed exact copy of every earlier input.\n\nA feedforward network maps inputs through its layers without this recurrent state transition. An RNN can accept a sequence, reuse its parameters at each step and produce either an output at each step or an output from a final state, depending on the task. Examples include sentiment classification, next-word prediction and time-series modeling.\n\nUnrolling an RNN draws its repeated use at t = 1, 2, 3 and so on. Those boxes share parameters; drawing three time steps does not create three independently learned copies of Whh.\n"},{"title":"Reading recurrent memory and gates","blocks":["t = 1 is the first input; t = 2 is the second. The state h₀ is the starting memory, often given as zero. At each step the input changes and the state changes, while the learned weights are reused.
","First multiply and add to form a raw score. Then apply tanh or the activation specified in the question. The activated result is the hidden state passed to the next time step.
","![\"a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h,\\qquad](\"assets/equations/3fed90dc3ade6a15.svg\")
Wxh means the weight mapping input x to hidden state h. Whh maps the previous hidden state to the new hidden calculation. These labels describe a connection; they do not mean multiply the letters together.
","A gate is a multiplier controlling how much of a value passes through. Gate 0 keeps none of that contribution; gate 1 keeps all; gate 0.25 keeps a quarter. For example, 0.25 × 8 = 2. Sigmoid-generated gates lie between 0 and 1 for finite raw scores.
","If a gate's raw score is 0, sigmoid gives 1/(1 + 1) = 0.5. A supplied gate value 0, however, is already zero and must not be passed through sigmoid again. Read whether the question gives preactivations or gate outputs.
","The gate may be nonnegative while the value being gated is negative. Multiplying 0.2 by −0.5 gives −0.1. This is normal: the gate chooses an amount, and the candidate content supplies its sign.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s3","search":"Reading recurrent memory and gates ### Time means position in the sequence\nt = 1 is the first input; t = 2 is the second. The state h₀ is the starting memory, often given as zero. At each step the input changes and the state changes, while the learned weights are reused.\n\n\n\n### Raw score and activated state\nFirst multiply and add to form a raw score. Then apply tanh or the activation specified in the question. The activated result is the hidden state passed to the next time step.\n\nWxh means the weight mapping input x to hidden state h. Whh maps the previous hidden state to the new hidden calculation. These labels describe a connection; they do not mean multiply the letters together.\n### What a gate does\nA gate is a multiplier controlling how much of a value passes through. Gate 0 keeps none of that contribution; gate 1 keeps all; gate 0.25 keeps a quarter. For example, 0.25 × 8 = 2. Sigmoid-generated gates lie between 0 and 1 for finite raw scores.\n### A gate is different from its raw score\nIf a gate's raw score is 0, sigmoid gives 1/(1 + 1) = 0.5. A supplied gate value 0, however, is already zero and must not be passed through sigmoid again. Read whether the question gives **preactivations** or **gate outputs**.\n### Memory values can be negative\nThe gate may be nonnegative while the value being gated is negative. Multiplying 0.2 by −0.5 gives −0.1. This is normal: the gate chooses an amount, and the candidate content supplies its sign.\n"},{"title":"The vanilla RNN equations from scratch","blocks":["Definition. A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction.
","For a tanh recurrent layer:
","aₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit.
","For classification at time t, an output layer can produce logits and probabilities:
","A regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation.
","Question. Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation.
","h₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0.
","x₁ = 1; x₂ = 2; activation = tanh.
","Find h₁ and h₂ in order.
","Recurrent forward pass. At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.
","x_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.
","Yaad rakhne ka cue: Naya input + pichhla activated state; phir activation.
","Step 1: Time step 1 — calculate the preactivation.
","Apply the recurrent activation.
","This activated value, not a₁ = 0.5, is passed to the next time step.
","Step 2: Time step 2 — combine the new input and previous state.
","Apply tanh to the complete new preactivation.
","| Time step | Current input | Previous state | Preactivation | New state |
|---|---|---|---|---|
| 1 | 1 | 0 | 0.5 | 0.462117157 |
| 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 |
Final answer: h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s4","search":"The vanilla RNN equations from scratch \n**Definition.** A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction.\n\n### Hidden-state and output equations\nFor a tanh recurrent layer:\n\n\naₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit.\n\nFor classification at time t, an output layer can produce logits and probabilities:\n\nA regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation.\n\n\n\n### Complete numerical: two recurrent time steps\n**Question.** Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation.\n\n\n\n### Given\n\nh₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0.\n\nx₁ = 1; x₂ = 2; activation = tanh.\n\n### To find\n\nFind h₁ and h₂ in order.\n\n### Formula — which one and why\n\n**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.\n\nx_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: Time step 1 — calculate the preactivation.**\n\n**Apply the recurrent activation.**\n\nThis activated value, not a₁ = 0.5, is passed to the next time step.\n\n**Step 2: Time step 2 — combine the new input and previous state.**\n\n\n**Apply tanh to the complete new preactivation.**\n\n\n| Time step | Current input | Previous state | Preactivation | New state |\n| 1 | 1 | 0 | 0.5 | 0.462117157 |\n| 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 |\n\n**Final answer:** h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update.\n\n"},{"title":"A complete vector RNN calculation","blocks":["Question from R26 Q5a. xₜ = [0,1], hₜ₋₁ = [0,1], no bias is supplied, Wxh is shown below, Whh is shown below. Use tanh and rows as receiving neurons.
","Input x = [0,1]ᵀ; previous h = [0,1]ᵀ.
","Wxh = .
Whh = ; bias = 0.
Apply tanh separately to the two sums.
","Find the new two-component hidden state; output-layer parameters are not supplied.
","Recurrent forward pass. At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.
","x_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.
","Yaad rakhne ka cue: Naya input + pichhla activated state; phir activation.
","Step 1 new input: Wxh x = [0.6 × 0 + 0.4 × 1, 0.3 × 0 + 0.7 × 1] = [0.4,0.7].
","Step 2 old memory: Whh hprevious = [0.5 × 0 + 0.3 × 1, 0.2 × 0 + 0.4 × 1] = [0.3,0.4].
","Step 3 add: a = [0.4+0.3, 0.7+0.4] = [0.7,1.1].
","Step 4 activate each element: h = [tanh(0.7), tanh(1.1)] = [0.604368,0.800499]. An output-layer prediction cannot be numerically calculated without its weights and bias.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s5","search":"A complete vector RNN calculation **Question from R26 Q5a.** xₜ = [0,1], hₜ₋₁ = [0,1], no bias is supplied, Wxh is shown below, Whh is shown below. Use tanh and rows as receiving neurons.\n\n\n\n### Given\n\nInput x = [0,1]ᵀ; previous h = [0,1]ᵀ.\n\nWxh = .\n\nWhh = ; bias = 0.\n\nApply tanh separately to the two sums.\n\n### To find\n\nFind the new two-component hidden state; output-layer parameters are not supplied.\n\n### Formula — which one and why\n\n**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.\n\nx_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.\n\n### Solution — step by step\n\n\n\n\n\n\n**Step 1 new input: ** Wxh x = [0.6 × 0 + 0.4 × 1, 0.3 × 0 + 0.7 × 1] = [0.4,0.7].\n**Step 2 old memory: ** Whh hprevious = [0.5 × 0 + 0.3 × 1, 0.2 × 0 + 0.4 × 1] = [0.3,0.4].\n**Step 3 add: ** a = [0.4+0.3, 0.7+0.4] = [0.7,1.1].\n**Step 4 activate each element: ** h = [tanh(0.7), tanh(1.1)] = **[0.604368,0.800499]**. An output-layer prediction cannot be numerically calculated without its weights and bias."},{"title":"How an RNN is trained","blocks":["Definition. Backpropagation through time (BPTT) differentiates the loss of an RNN by applying backpropagation to its temporally unrolled computational graph and summing contributions to shared parameters.
","Unroll the recurrent network for the sequence length. Run the forward calculation in time order. Calculate the specified loss. Differentiate backwards through the unrolled graph. Add contributions to each shared weight from every time it was used. Update each shared weight once for the selected sequence or batch. This is backpropagation through time, or BPTT.
","For tanh, let δₜ mean the gradient with respect to the hidden preactivation. The following form allows losses at several time steps; a missing output loss contributes zero.
","![\"\\delta_t=\\left(W_{hy}^{T}\\delta_t^{out}+W_{hh}^{T}\\delta_{t+1}\\right)\\odot(1-h_t^2)\"](\"assets/equations/05416576e2064832.svg\")
![\"\\frac{\\partial](\"assets/equations/1b44f91dfbf3ef75.svg\")
The symbol ⊙ means pairwise multiplication. The transpose T turns a column into a row. δnext carries the effect of future time steps. The summation is essential because these are shared parameters.
","Question. In a scalar tanh RNN, h_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.
","Activated h_t = 0.5; q_t = 0.2; δ_(t+1) = 0.3; recurrent weight u = 0.4.
","Current preactivation delta δ_t.
","Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.
","q_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.
","![\"\\delta_t=(q_t+u\\delta_{t+1})(1-h_t^2)\"](\"assets/equations/9b43734482c48bdd.svg\")
Yaad rakhne ka cue: Future aur current dono add; tanh slope uske baad.
","Step 1: calculate the future path contribution.
","![\"0.4(0.3)=0.12\"](\"assets/equations/bb48ef38d6d22273.svg\")
Step 2: add the current output contribution.
","![\"0.2+0.12=0.32\"](\"assets/equations/fdbe3cda286c3c16.svg\")
Step 3: calculate the current tanh slope.
","![\"1-(0.5)^2=0.75\"](\"assets/equations/7f60b4793b26f591.svg\")
Step 4: multiply by that slope.
","![\"\\delta_t=(0.32)(0.75)=0.24\"](\"assets/equations/e726c1bb32ed233e.svg\")
Final answer: δ_t = 0.24.
","Question. In a scalar tanh RNN, h_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.
","Activated h_t = -0.2; q_t = 0.1; δ_(t+1) = 0; recurrent weight u = 0.5.
","Current preactivation delta δ_t.
","Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.
","q_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.
","Yaad rakhne ka cue: Future aur current dono add; tanh slope uske baad.
","Step 1: calculate the future path contribution.
","![\"0.5(0)=0\"](\"assets/equations/1ad7c98370e2604f.svg\")
Step 2: add the current output contribution.
","![\"0.1+0=0.1\"](\"assets/equations/9d31df4938e5db9a.svg\")
Step 3: calculate the current tanh slope.
","![\"1-(-0.2)^2=0.96\"](\"assets/equations/fb671dc68ac70d16.svg\")
Step 4: multiply by that slope.
","![\"\\delta_t=(0.1)(0.96)=0.096\"](\"assets/equations/c494eb0b53453545.svg\")
Final answer: δ_t = 0.096.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s6","search":"How an RNN is trained **Definition.** Backpropagation through time (BPTT) differentiates the loss of an RNN by applying backpropagation to its temporally unrolled computational graph and summing contributions to shared parameters.\n\n### Explanation and working\nUnroll the recurrent network for the sequence length. Run the forward calculation in time order. Calculate the specified loss. Differentiate backwards through the unrolled graph. Add contributions to each shared weight from every time it was used. Update each shared weight once for the selected sequence or batch. This is backpropagation through time, or BPTT.\n\n\n\nFor tanh, let δₜ mean the gradient with respect to the hidden preactivation. The following form allows losses at several time steps; a missing output loss contributes zero.\n\n\nThe symbol ⊙ means pairwise multiplication. The transpose T turns a column into a row. δnext carries the effect of future time steps. The summation is essential because these are shared parameters.\n\n\n\n### One BPTT delta at h = 0.5\n\n**Question. In a scalar tanh RNN, h_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.**\n\n\n\n### Given\n\nActivated h_t = 0.5; q_t = 0.2; δ_(t+1) = 0.3; recurrent weight u = 0.4.\n\n### To find\n\nCurrent preactivation delta δ_t.\n\n### Formula — which one and why\n\nTwo paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.\n\nq_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.\n\n\n\n**Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad.\n\n### Solution — step by step\n\n\n\n**Step 1: calculate the future path contribution.**\n\n\n\n\n\n\n\n**Step 2: add the current output contribution.**\n\n\n\n\n\n\n\n**Step 3: calculate the current tanh slope.**\n\n\n\n\n\n\n\n**Step 4: multiply by that slope.**\n\n\n\n\n\n\n\n**Final answer:** δ_t = 0.24.\n\n\n\n### One BPTT delta at h = -0.2\n\n**Question. In a scalar tanh RNN, h_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.**\n\n\n\n### Given\n\nActivated h_t = -0.2; q_t = 0.1; δ_(t+1) = 0; recurrent weight u = 0.5.\n\n### To find\n\nCurrent preactivation delta δ_t.\n\n### Formula — which one and why\n\nTwo paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.\n\nq_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.\n\n\n\n**Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad.\n\n### Solution — step by step\n\n\n\n**Step 1: calculate the future path contribution.**\n\n\n\n\n\n\n\n**Step 2: add the current output contribution.**\n\n\n\n\n\n\n\n**Step 3: calculate the current tanh slope.**\n\n\n\n\n\n\n\n**Step 4: multiply by that slope.**\n\n\n\n\n\n\n\n**Final answer:** δ_t = 0.096.\n\n\n"},{"title":"BPTT with every arithmetic step visible","blocks":["Question. A scalar RNN uses the linear state equation hₜ = u xₜ + v hₜ₋₁ + b. Its prediction is ŷ = h₂ and its loss is L = ½(ŷ−1)². Given u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1 and x₂ = 2, calculate the gradients of u, v and b using BPTT, then update them once using learning rate η = 0.1.
","hₜ = uxₜ+vhₜ₋₁+b; prediction = h₂.
","u = 0.2; v = 0.5; b = 0; h₀ = 0; x₁ = 1; x₂ = 2.
","Target = 1; half squared loss; η = 0.1.
","Find gradients for u,v,b and update each once.
","Backpropagation through time. Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once.
","For h_t=ux_t+vh_(t−1)+b with loss only at the last state: δ_t = loss derivative at that state. The linear recurrence has activation slope 1.
","![\"\\delta_T=h_T-y\"](\"assets/equations/61b2086787d829b8.svg\")
![\"\\delta_t=v\\delta_{t+1}\"](\"assets/equations/ba589b262a27bc88.svg\")
![\"g_u=\\sum_t\\delta_tx_t\"](\"assets/equations/ad879502d644f051.svg\")
![\"g_v=\\sum_t\\delta_th_{t-1}\"](\"assets/equations/3065785a32d83c15.svg\")
![\"g_b=\\sum_t\\delta_t\"](\"assets/equations/cc44b288566e90ca.svg\")
Yaad rakhne ka cue: Time ke har use ka gradient jodo; shared weight ko ek baar update karo.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","u is the input weight, v is the recurrent weight and b is the hidden bias. Each is shared by both time steps. The hidden activation is linear in this question, so its derivative is 1. There is a loss only at time step 2; time step 1 influences that loss through h₂.
","![\"h_1=ux_1+vh_0+b\"](\"assets/equations/9a1296eaa212dfb3.svg\")
![\"h_1=0.2(1)+0.5(0)+0=0.2\"](\"assets/equations/81bb99ddeeaa7079.svg\")
![\"h_2=ux_2+vh_1+b\"](\"assets/equations/aa5a122e6af7b01d.svg\")
![\"h_2=0.2(2)+0.5(0.2)+0=0.5\"](\"assets/equations/ca0b29f546ae57d0.svg\")
![\"L=\\frac12(0.5-1)^2=0.125\"](\"assets/equations/03fd42e4fe3cfc09.svg\")
Because the output is h₂ and the state activation is linear:
","![\"\\delta_2=\\frac{\\partial](\"assets/equations/44dd9c1e67b5f92d.svg\")
The local derivative of h₂ with respect to h₁ is the recurrent weight v = 0.5. Applying the chain rule:
","![\"\\delta_1=\\frac{\\partial](\"assets/equations/5911c2f4d8e13a0a.svg\")
![\"\\delta_1=\\delta_2v=(-0.5)(0.5)=-0.25\"](\"assets/equations/59ca9544049d91dc.svg\")
There is no separate direct loss at time step 1 to add here. Both deltas use the original recurrent weight.
","u appears in both h₁ and h₂, so its gradient receives one contribution from each use:
","![\"\\frac{\\partial](\"assets/equations/94543a0839df0ed3.svg\")
![\"\\frac{\\partial](\"assets/equations/a96bfa656867b634.svg\")
For v, the sending activation is the previous hidden state at the corresponding time:
","![\"\\frac{\\partial](\"assets/equations/4f377c9f2aefec11.svg\")
![\"\\frac{\\partial](\"assets/equations/5d9bcac5813d6549.svg\")
The same additive bias is used at both steps. Its two local derivatives are 1:
","![\"\\frac{\\partial](\"assets/equations/bc0231e316221e07.svg\")
| Shared parameter | Contribution at time 1 | Contribution at time 2 | Total gradient |
|---|---|---|---|
| u | −0.25 × 1 = −0.25 | −0.5 × 2 = −1 | −1.25 |
| v | −0.25 × 0 = 0 | −0.5 × 0.2 = −0.1 | −0.1 |
| b | −0.25 | −0.5 | −0.75 |
![\"u_{\\mathrm{new}}=0.2-0.1(-1.25)=0.325\"](\"assets/equations/30e22de1103f6dec.svg\")
![\"v_{\\mathrm{new}}=0.5-0.1(-0.1)=0.51\"](\"assets/equations/4822c14a12ff6b68.svg\")
![\"b_{\\mathrm{new}}=0-0.1(-0.75)=0.075\"](\"assets/equations/0fa40afe6a0cd296.svg\")
Final answer: gradients are −1.25, −0.1 and −0.75; updated parameters are u = 0.325, v = 0.51 and b = 0.075.
","Restart from the same h₀ = 0 and use all updated parameters together:
","![\"h_{1,\\mathrm{new}}=0.325(1)+0.51(0)+0.075=0.4\"](\"assets/equations/af9ceb62c0e1f59e.svg\")
![\"h_{2,\\mathrm{new}}=0.325(2)+0.51(0.4)+0.075=0.929\"](\"assets/equations/cef917ae20dca74f.svg\")
![\"L_{\\mathrm{new}}=\\frac12(0.929-1)^2=0.0025205\"](\"assets/equations/8da79c9221b5024a.svg\")
The loss decreases in this particular update. That is consistent with the computed gradient, but a decrease is not guaranteed for every choice of learning rate.
","For tanh, the derivative at each state transition also contains 1−hₜ². A trainable output layer contributes its own local derivatives. If losses occur at several time steps, add the direct output-loss contribution and the future-state contribution before applying the local activation derivative. Truncated BPTT limits the number of time steps traversed backward; it does not create separate copies of the recurrent parameters.
","Question. A scalar RNN uses hₜ = tanh(uxₜ + vhₜ₋₁ + b). Given u = 0.4, v = 0.3, b = 0.1, h₀ = 0 and inputs x₁ = 1, x₂ = −1, its final output is h₂ and target is 0.2. Use only the final-time loss L = ½(h₂ − 0.2)². Calculate both states, all three shared-parameter gradients and one simultaneous update at η = 0.1.
","Shared input weight u = 0.4; shared recurrent weight v = 0.3; shared bias b = 0.1. Initial state h₀ = 0; x₁ = 1, x₂ = −1. Hidden activation = tanh; final target = 0.2; learning rate η = 0.1. There is no separate output layer and no loss at time 1.
","Forward states h₁ and h₂, final loss, gradients with respect to u, v and b, and all three updated parameters.
","Backpropagation must pass through each tanh derivative. Because u, v and b are shared, add their local contributions from both time steps before updating them.
","![\"z_t=ux_t+vh_{t-1}+b,\\qquad](\"assets/equations/f064c725161475e9.svg\")
![\"\\delta_2=(h_2-y)(1-h_2^2),\\qquad](\"assets/equations/4b5081051ef8a38b.svg\")
![\"\\frac{\\partial](\"assets/equations/e7070aed32f74b9a.svg\")
Here δₜ means ∂L/∂zₜ, not ∂L/∂hₜ. The factor v in δ₁ is the old recurrent weight 0.3. The derivative of tanh is 1 − hₜ², using that step's activated state.
","z₁ = 0.4(1) + 0.3(0) + 0.1 = 0.5.
","h₁ = tanh(0.5) = 0.462117157.
","z₂ = 0.4(−1) + 0.3(0.462117157) + 0.1 = −0.161364853.
","h₂ = tanh(−0.161364853) = −0.159978716.
","Prediction error h₂ − y = −0.159978716 − 0.2 = −0.359978716.
","L = ½(−0.359978716)² = 0.064792338.
","δ₂ = (−0.359978716)[1 − (−0.159978716)²] = −0.350765712.
","There is no direct loss at time 1. Its gradient arrives entirely through time 2:
","δ₁ = 0.3(−0.350765712)[1 − (0.462117157)²] = −0.082757670.
","| Parameter | Time contributions | Total gradient |
|---|---|---|
| u | (−0.082757670)(1) + (−0.350765712)(−1) | 0.268008043 |
| v | (−0.082757670)(0) + (−0.350765712)(0.462117157) | −0.162094854 |
| b | −0.082757670 + (−0.350765712) | −0.433523382 |
For v, the sending state is hₜ₋₁: h₀ for time 1, h₁ for time 2. The negative second input makes its contribution to the u gradient positive.
","u_new = 0.4 − 0.1(0.268008043) = 0.373199196.
","v_new = 0.3 − 0.1(−0.162094854) = 0.316209485.
","b_new = 0.1 − 0.1(−0.433523382) = 0.143352338.
","Using the new parameters from h₀ = 0 gives h₁,new = tanh(0.373199196 + 0.143352338) = 0.475034096.
","h₂,new = tanh(−0.373199196 + 0.316209485(0.475034096) + 0.143352338) = −0.079468645.
","The new loss is ½(−0.079468645 − 0.2)² = 0.039051362.
","Final answer: gradients (u,v,b) = (0.268008043, −0.162094854, −0.433523382) and updated parameters = (0.373199196, 0.316209485, 0.143352338). This step moves the final prediction toward 0.2 and reduces loss from 0.064792338 to 0.039051362. Both time steps contributed to learning the same parameters.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s7","search":"BPTT with every arithmetic step visible **Question.** A scalar RNN uses the linear state equation hₜ = u xₜ + v hₜ₋₁ + b. Its prediction is ŷ = h₂ and its loss is L = ½(ŷ−1)². Given u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1 and x₂ = 2, calculate the gradients of u, v and b using BPTT, then update them once using learning rate η = 0.1.\n\n\n\n### Given\n\nhₜ = uxₜ+vhₜ₋₁+b; prediction = h₂.\n\nu = 0.2; v = 0.5; b = 0; h₀ = 0; x₁ = 1; x₂ = 2.\n\nTarget = 1; half squared loss; η = 0.1.\n\n### To find\n\nFind gradients for u,v,b and update each once.\n\n### Formula — which one and why\n\n**Backpropagation through time.** Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once.\n\nFor h_t=ux_t+vh_(t−1)+b with loss only at the last state: δ_t = loss derivative at that state. The linear recurrence has activation slope 1.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Time ke har use ka gradient jodo; shared weight ko ek baar update karo.\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Model and dependency structure\nu is the input weight, v is the recurrent weight and b is the hidden bias. Each is shared by both time steps. The hidden activation is linear in this question, so its derivative is 1. There is a loss only at time step 2; time step 1 influences that loss through h₂.\n\n\n\n### Step 1: complete the forward pass\n\n\n\n\n\n\n### Step 2: differentiate the loss at the final step\nBecause the output is h₂ and the state activation is linear:\n\n\n### Step 3: propagate the derivative to the earlier state\nThe local derivative of h₂ with respect to h₁ is the recurrent weight v = 0.5. Applying the chain rule:\n\n\nThere is no separate direct loss at time step 1 to add here. Both deltas use the original recurrent weight.\n\n### Step 4: sum the contributions to each shared parameter\nu appears in both h₁ and h₂, so its gradient receives one contribution from each use:\n\n\nFor v, the sending activation is the previous hidden state at the corresponding time:\n\n\nThe same additive bias is used at both steps. Its two local derivatives are 1:\n\n\n| Shared parameter | Contribution at time 1 | Contribution at time 2 | Total gradient |\n| u | −0.25 × 1 = −0.25 | −0.5 × 2 = −1 | −1.25 |\n| v | −0.25 × 0 = 0 | −0.5 × 0.2 = −0.1 | −0.1 |\n| b | −0.25 | −0.5 | −0.75 |\n\n### Step 5: update each shared parameter once\n\n\n\n**Final answer:** gradients are −1.25, −0.1 and −0.75; updated parameters are u = 0.325, v = 0.51 and b = 0.075.\n\n### Evaluate the updated model\nRestart from the same h₀ = 0 and use all updated parameters together:\n\n\n\nThe loss decreases in this particular update. That is consistent with the computed gradient, but a decrease is not guaranteed for every choice of learning rate.\n\n### Extension to a tanh RNN\nFor tanh, the derivative at each state transition also contains 1−hₜ². A trainable output layer contributes its own local derivatives. If losses occur at several time steps, add the direct output-loss contribution and the future-state contribution before applying the local activation derivative. Truncated BPTT limits the numbe"},{"title":"Vanishing and exploding gradients","blocks":["Definition. Vanishing and exploding gradients describe the attenuation or amplification of backward derivatives through repeated Jacobian products. In recurrent networks, these products extend across time and affect long-range credit assignment.
","In BPTT, the gradient repeatedly multiplies recurrent weights and activation derivatives. Products of many small factors can approach zero; early information then receives very little learning signal. Large repeated factors can make the gradient explode. These are about training signals, not an instruction that every hidden-state value must become zero or infinity.
","Numerical intuition: 0.5¹⁰ = 0.0009765625, whereas 2¹⁰ = 1,024. Sigmoid's derivative is at most 0.25 and becomes even smaller in saturation. Tanh also saturates for large magnitude inputs.
","Useful remedies include LSTM or GRU gates and cell paths, suitable initialization and learning rate, and gradient clipping for explosion. Clipping does not by itself solve vanishing gradients. LSTM makes long-range learning easier; it does not guarantee perfect unlimited memory.
","For a vanilla tanh RNN, each hidden-state transition contributes a Jacobian. The derivative through several transitions contains a product of these Jacobians:
","![\"\\frac{\\partial](\"assets/equations/78674f482f3097e9.svg\")
![\"\\frac{\\partial](\"assets/equations/382feba776054adf.svg\")
The product is ordered from the latest transition toward the earliest. Small activation derivatives and recurrent transformations can attenuate the signal; repeated amplification can make it grow. A one-dimensional product illustrates the issue, but a vector RNN depends on the full matrix products, not one weight alone.
","Norm clipping rescales an oversized gradient while retaining its direction. It limits the update size caused by an exploding gradient.
","![\"g_{\\mathrm{clipped}}=g\\min\\left(1,\\frac{\\tau}{\\lVert](\"assets/equations/bc3f87fdb9a197b4.svg\")
If g is the zero vector, leave it unchanged. The threshold τ is positive. This is global norm clipping for the vector being considered; elementwise clipping is a different operation.
","Question. A gradient is g = [6, 8]ᵀ. Clip its Euclidean norm to at most τ = 5, then update w = [1, 1]ᵀ with learning rate η = 0.1.
","Gradient g = [6,8]ᵀ; norm threshold τ = 5.
","Old weights w = [1,1]ᵀ; η = 0.1.
","Find the clipped gradient and updated weights.
","Gradient norm clipping. If the vector norm exceeds the allowed threshold, multiply every component by the same scale. This changes magnitude while preserving direction.
","g = gradient vector; τ (tau) = maximum norm; ||g||₂ = square root of the sum of squared components; η = learning rate.
","![\"\\lVert](\"assets/equations/f0dbe784a7be0b2f.svg\")
![\"g_{\\mathrm{clip}}=g\\min\\left(1,\\frac\\tau{\\lVert](\"assets/equations/dbdee9367720972d.svg\")
![\"w_{\\mathrm{new}}=w-\\eta](\"assets/equations/1b334e0c9202e5b4.svg\")
Yaad rakhne ka cue: Bada gradient? Sab components ko ek hi factor se chhota karo.
","Step 1: calculate the original norm.
","![\"\\lVert](\"assets/equations/5cf7611fc86c5694.svg\")
Step 2: find the scale. Since 10 > 5, scale the vector by 5/10 = 0.5.
","![\"g_{\\mathrm{clipped}}=0.5\\begin{bmatrix}6\\\\8\\end{bmatrix}=\\begin{bmatrix}3\\\\4\\end{bmatrix}\"](\"assets/equations/55639775a78bd93d.svg\")
Step 3: verify the clipped norm.
","![\"\\sqrt{3^2+4^2}=\\sqrt{9+16}=5\"](\"assets/equations/4cd7b08ae98b6682.svg\")
Step 4: update each parameter with the clipped gradient.
","![\"w_{1,\\mathrm{new}}=1-0.1(3)=0.7\"](\"assets/equations/22080c63aaa74ec0.svg\")
![\"w_{2,\\mathrm{new}}=1-0.1(4)=0.6\"](\"assets/equations/1268aa744d549860.svg\")
Final answer: clipped gradient [3, 4]ᵀ and new weights [0.7, 0.6]ᵀ. Clipping does not increase a vanishing gradient and does not replace the need for an appropriate recurrent architecture.
","Question. Clip gradient g = [3, 4] to maximum L2 norm 10. Use global norm clipping.
","g = [3, 4]; threshold τ = 10.
","The norm, scaling factor and clipped gradient.
","Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold.
","τ is the maximum permitted norm, not a per-entry bound.
","![\"\\|g\\|_2=\\sqrt{\\sum_i](\"assets/equations/c087afaaeb69ebf5.svg\")
![\"g_{\\mathrm{clip}}=g\\min(1,\\tau/\\|g\\|_2)\"](\"assets/equations/fbf72036323a5940.svg\")
Yaad rakhne ka cue: Norm bada ho tab sab entries ko same ratio se chhota karo.
","Step 1: calculate the squared norm.
","![\"3^2+4^2=25\"](\"assets/equations/9e1425dca608e703.svg\")
Step 2: take the square root.
","![\"\\|g\\|_2=\\sqrt{25}=5\"](\"assets/equations/31e01f04f62bf975.svg\")
Step 3: choose the shared scale.
","![\"c=\\min(1,10/5)=1\"](\"assets/equations/93428ecf1a9721ce.svg\")
Step 4: scale every entry.
","![\"g_1^{\\mathrm{clip}}=3(1)=3\"](\"assets/equations/af049506394d9137.svg\")
![\"g_2^{\\mathrm{clip}}=4(1)=4\"](\"assets/equations/9c36bcaa7f2c9e3b.svg\")
Final answer: Clipped gradient = [3, 4].
","Question. Clip gradient g = [0, 12] to maximum L2 norm 3. Use global norm clipping.
","g = [0, 12]; threshold τ = 3.
","The norm, scaling factor and clipped gradient.
","Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold.
","τ is the maximum permitted norm, not a per-entry bound.
","Yaad rakhne ka cue: Norm bada ho tab sab entries ko same ratio se chhota karo.
","Step 1: calculate the squared norm.
","![\"0^2+12^2=144\"](\"assets/equations/0d5fe5de098f7a47.svg\")
Step 2: take the square root.
","![\"\\|g\\|_2=\\sqrt{144}=12\"](\"assets/equations/723310d10d7d6089.svg\")
Step 3: choose the shared scale.
","![\"c=\\min(1,3/12)=0.25\"](\"assets/equations/185f7122d90f817c.svg\")
Step 4: scale every entry.
","![\"g_1^{\\mathrm{clip}}=0(0.25)=0\"](\"assets/equations/d479ef7213e9c007.svg\")
![\"g_2^{\\mathrm{clip}}=12(0.25)=3\"](\"assets/equations/451a7aae4fe2ab76.svg\")
Final answer: Clipped gradient = [0.0, 3.0].
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s8","search":"Vanishing and exploding gradients **Definition.** Vanishing and exploding gradients describe the attenuation or amplification of backward derivatives through repeated Jacobian products. In recurrent networks, these products extend across time and affect long-range credit assignment.\n\n### Explanation and working\nIn BPTT, the gradient repeatedly multiplies recurrent weights and activation derivatives. Products of many small factors can approach zero; early information then receives very little learning signal. Large repeated factors can make the gradient explode. These are about training signals, not an instruction that every hidden-state value must become zero or infinity.\n\n\n\n**Numerical intuition: ** 0.5¹⁰ = 0.0009765625, whereas 2¹⁰ = 1,024. Sigmoid's derivative is at most 0.25 and becomes even smaller in saturation. Tanh also saturates for large magnitude inputs.\nUseful remedies include LSTM or GRU gates and cell paths, suitable initialization and learning rate, and gradient clipping for explosion. Clipping does not by itself solve vanishing gradients. LSTM makes long-range learning easier; it does not guarantee perfect unlimited memory.\n\n### Why the effect grows with sequence length\nFor a vanilla tanh RNN, each hidden-state transition contributes a Jacobian. The derivative through several transitions contains a product of these Jacobians:\n\n\nThe product is ordered from the latest transition toward the earliest. Small activation derivatives and recurrent transformations can attenuate the signal; repeated amplification can make it grow. A one-dimensional product illustrates the issue, but a vector RNN depends on the full matrix products, not one weight alone.\n\n### Gradient norm clipping: a complete numerical\nNorm clipping rescales an oversized gradient while retaining its direction. It limits the update size caused by an exploding gradient.\n\nIf g is the zero vector, leave it unchanged. The threshold τ is positive. This is global norm clipping for the vector being considered; elementwise clipping is a different operation.\n\n**Question. A gradient is g = [6, 8]ᵀ. Clip its Euclidean norm to at most τ = 5, then update w = [1, 1]ᵀ with learning rate η = 0.1.**\n\n\n\n### Given\n\nGradient g = [6,8]ᵀ; norm threshold τ = 5.\n\nOld weights w = [1,1]ᵀ; η = 0.1.\n\n### To find\n\nFind the clipped gradient and updated weights.\n\n### Formula — which one and why\n\n**Gradient norm clipping.** If the vector norm exceeds the allowed threshold, multiply every component by the same scale. This changes magnitude while preserving direction.\n\ng = gradient vector; τ (tau) = maximum norm; ||g||₂ = square root of the sum of squared components; η = learning rate.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Bada gradient? Sab components ko ek hi factor se chhota karo.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: calculate the original norm.**\n\n\n**Step 2: find the scale.** Since 10 > 5, scale the vector by 5/10 = 0.5.\n\n\n**Step 3: verify the clipped norm.**\n\n\n**Step 4: update each parameter with the clipped gradient.**\n\n\n**Final answer:** clipped gradient [3, 4]ᵀ and new weights [0.7, 0.6]ᵀ. Clipping does not increase a vanishing gradient and does not replace the need for an appropriate recurrent architecture.\n\n### Gradient norm clipping: [3, 4]\n\n**Question. Clip gradient g = [3, 4] to maximum L2 norm 10. Use global norm clipping.**\n\n\n\n### Given\n\ng = [3, 4]; threshold τ = 10.\n\n### To find\n\nThe norm, scaling factor and clipped gradient.\n\n### Formula — which one and why\n\nGlobal norm clipping sc"},{"title":"Model answer: backpropagation through time","question":["Question. Explain backpropagation through time and the vanishing/exploding gradient problems in a recurrent neural network.
"],"answer":["Backpropagation through time, or BPTT, applies backpropagation to an RNN unrolled across a sequence. The forward pass computes the state and any output at each time step. The backward pass follows how later losses depend on earlier states and shared parameters.
","1. Start from the supplied initial state and compute h₁, h₂ and all subsequent states in order.
","2. Calculate the loss at the output positions specified by the task.
","3. Propagate loss derivatives backward through the output computations and recurrent state transitions.
","4. Add contributions from all time steps that use the same parameter.
","5. Apply the optimizer update to the shared parameter after its gradient contributions have been accumulated.
","The chain rule repeatedly multiplies derivatives through time. A simple illustration is multiplying by 0.5 ten times, giving about 0.000977, versus multiplying by 2 ten times, giving 1,024. Real RNNs involve matrices and activation derivatives, so these scalars illustrate the mechanism rather than define a universal threshold.
","Vanishing gradients make distant earlier inputs receive very weak learning signals. Exploding gradients can produce extremely large updates and unstable training. Gradient clipping limits gradient magnitude before an optimizer update and is useful for exploding gradients. It does not by itself restore already vanished information.
","LSTM/GRU gating, suitable initialization and architecture choices can improve long-range learning. Truncated BPTT limits how far the backward computation runs for computational reasons; it does not mean that each unrolled box has independent parameters.
"],"kind":"theory","id":"tee-c9-s9","search":"Model answer: backpropagation through time **Question.** Explain backpropagation through time and the vanishing/exploding gradient problems in a recurrent neural network.\n\n**Model answer.**\nBackpropagation through time, or BPTT, applies backpropagation to an RNN unrolled across a sequence. The forward pass computes the state and any output at each time step. The backward pass follows how later losses depend on earlier states and shared parameters.\n\n\n\n\n### Working procedure\n1. Start from the supplied initial state and compute h₁, h₂ and all subsequent states in order.\n2. Calculate the loss at the output positions specified by the task.\n3. Propagate loss derivatives backward through the output computations and recurrent state transitions.\n4. Add contributions from all time steps that use the same parameter.\n5. Apply the optimizer update to the shared parameter after its gradient contributions have been accumulated.\n\n### Why gradients can vanish or explode\nThe chain rule repeatedly multiplies derivatives through time. A simple illustration is multiplying by 0.5 ten times, giving about 0.000977, versus multiplying by 2 ten times, giving 1,024. Real RNNs involve matrices and activation derivatives, so these scalars illustrate the mechanism rather than define a universal threshold.\n\nVanishing gradients make distant earlier inputs receive very weak learning signals. Exploding gradients can produce extremely large updates and unstable training. Gradient clipping limits gradient magnitude before an optimizer update and is useful for exploding gradients. It does not by itself restore already vanished information.\n\nLSTM/GRU gating, suitable initialization and architecture choices can improve long-range learning. Truncated BPTT limits how far the backward computation runs for computational reasons; it does not mean that each unrolled box has independent parameters.\n"},{"title":"LSTM a memory path with three gates","blocks":["Definition. Long short-term memory (LSTM) is a gated recurrent architecture with a cell state and hidden state. Forget, input and output gates regulate state retention, incorporation of candidate information and exposure of the updated cell.
","Long short-term memory, or LSTM, keeps a cell state Cₜ and a hidden state hₜ. A sigmoid gate produces values from 0 to 1: approximately 0 blocks a component, approximately 1 passes it. Intermediate values pass a fraction. Gates are learned from the previous hidden state and current input, not manually assigned by the user during normal inference.
","![\"f_t=\\sigma(W_f[h_{t-1},x_t]+b_f),\\quad](\"assets/equations/0c991bf8824254bc.svg\")
![\"\\widetilde](\"assets/equations/c08c8ddc953628c6.svg\")
![\"C_t=f_t\\odot](\"assets/equations/8751dee634f91753.svg\")
![\"o_t=\\sigma(W_o[h_{t-1},x_t]+b_o),\\qquad](\"assets/equations/932f49e3f8a32474.svg\")
The brackets [h, x] concatenate two vectors: [a, b] joined with [c, d] gives [a, b, c, d]. Concatenation is not addition. f is the forget gate: how much old cell content remains. i is the input gate: how much new candidate enters. The candidate is proposed content. o is the output gate: how much of the activated cell becomes visible as the hidden state.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s10","search":"LSTM a memory path with three gates **Definition.** Long short-term memory (LSTM) is a gated recurrent architecture with a cell state and hidden state. Forget, input and output gates regulate state retention, incorporation of candidate information and exposure of the updated cell.\n\n### Explanation and working\nLong short-term memory, or LSTM, keeps a cell state Cₜ and a hidden state hₜ. A sigmoid gate produces values from 0 to 1: approximately 0 blocks a component, approximately 1 passes it. Intermediate values pass a fraction. Gates are learned from the previous hidden state and current input, not manually assigned by the user during normal inference.\n\n\n\n\n\nThe brackets [h, x] concatenate two vectors: [a, b] joined with [c, d] gives [a, b, c, d]. Concatenation is not addition. f is the forget gate: how much old cell content remains. i is the input gate: how much new candidate enters. The candidate is proposed content. o is the output gate: how much of the activated cell becomes visible as the hidden state."},{"title":"Reading an LSTM equation one operation at a time","blocks":["The cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other.
","For a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention.
","An input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive.
","With those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel.
","![\"c_t=f_tc_{t-1}+i_t\\widetilde](\"assets/equations/2b17315e0dde1dde.svg\")
This example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update.
","Apply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0.
","![\"h_t=o_t\\tanh(c_t)\"](\"assets/equations/88b1929e93128b1e.svg\")
If c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c.
","When f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s11","search":"Reading an LSTM equation one operation at a time ### 1. Two memories on the page do not mean the same value\nThe cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other.\n\n\n\n\n### 2. Gate values are multipliers\nFor a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention.\nAn input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive.\n\n### 3. Add old and new contributions\nWith those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel.\n\nThis example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update.\n\n### 4. Calculate the hidden state afterward\nApply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0.\n\nIf c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c.\n\n### 5. Supplied gates and calculated gates are different numerical types\nWhen f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition.\n"},{"title":"LSTM complete numerical with supplied gates","blocks":["Question. An LSTM cell has input gate iₜ = 0.7, forget gate fₜ = 0.4, output gate oₜ = 0.8, candidate cell content C̃ₜ = 0.9 and previous cell state Cₜ₋₁ = 0.5. Calculate the new cell state Cₜ and hidden state hₜ.
","Activated input gate i = 0.7; forget gate f = 0.4; output gate o = 0.8.
","Candidate = 0.9; old cell = 0.5.
","Find the new cell C and hidden state h.
","LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.
","f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.
","![\"C=f\\odot](\"assets/equations/baccbadbb58af40d.svg\")
![\"h=o\\odot\\tanh](\"assets/equations/2bb414a00be2c088.svg\")
Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.
","The question supplies activated gate outputs and the candidate content. Do not apply sigmoid or tanh to those supplied values again. First update the cell, then calculate the hidden state:
","![\"C_t=f_tC_{t-1}+i_t\\widetilde](\"assets/equations/5c31e372051573ef.svg\")
![\"h_t=o_t\\tanh(C_t)\"](\"assets/equations/694789d6807a2931.svg\")
The first equation combines retained old content and admitted candidate content. The second equation is a separate transformation of the updated cell.
","Step 1: calculate the retained old-cell contribution.
","![\"f_tC_{t-1}=0.4(0.5)=0.2\"](\"assets/equations/c8c446684892171e.svg\")
Step 2: calculate the admitted candidate contribution.
","![\"i_t\\widetilde](\"assets/equations/83577513b17d4dfa.svg\")
Step 3: add the two contributions to obtain the new cell.
","![\"C_t=0.2+0.63=0.83\"](\"assets/equations/f6bfe5d9884a48c2.svg\")
Step 4: activate the updated cell.
","![\"\\tanh(C_t)=\\tanh(0.83)\\approx0.680476006\"](\"assets/equations/01c5de218bff4e7f.svg\")
Step 5: multiply by the output gate.
","![\"h_t=0.8\\tanh(0.83)\\approx0.544380805\"](\"assets/equations/cf4e11309aadd94a.svg\")
Final answer: Cₜ = 0.83 and hₜ ≈ 0.544381. Multiplying the output gate directly by Cₜ would give 0.664 and omit the required tanh operation.
","Question. Given forget gate f = [0.1, 0.9] and previous cell C = [2, 1], calculate the retained old-cell contribution.
","Forget gate = [0.1,0.9]; old cell = [2,1].
","Input-gate and candidate values are not supplied.
","Find only the retained old-cell contribution.
","LSTM retained old memory. Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell.
","f = activated forget gate; C_old = previous cell state; ⊙ means multiply matching entries.
","![\"C_{\\mathrm{retained}}=f\\odot](\"assets/equations/35cedc4f6bc20c8a.svg\")
Yaad rakhne ka cue: Forget gate × old cell = bacha hua purana hissa.
","Step 1: retain the first old-cell component.
","Step 2: retain the second old-cell component.
","![\"0.9(1)=0.9\"](\"assets/equations/ba3003dba6d22d16.svg\")
Step 3: assemble the two retained components.
","Keep their original order. This gives the following column:
","![\"f\\odot](\"assets/equations/ece27e6278683163.svg\")
Answer: [0.2, 0.9]. This is the retained contribution, not necessarily the final new cell. The candidate contribution must also be known to calculate the full cell update.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s12","search":"LSTM complete numerical with supplied gates \n**Question.** An LSTM cell has input gate iₜ = 0.7, forget gate fₜ = 0.4, output gate oₜ = 0.8, candidate cell content C̃ₜ = 0.9 and previous cell state Cₜ₋₁ = 0.5. Calculate the new cell state Cₜ and hidden state hₜ.\n\n\n\n### Given\n\nActivated input gate i = 0.7; forget gate f = 0.4; output gate o = 0.8.\n\nCandidate = 0.9; old cell = 0.5.\n\n### To find\n\nFind the new cell C and hidden state h.\n\n### Formula — which one and why\n\n**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.\n\nf,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.\n\n\n\n\n\n**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.\n\n### Solution — step by step\n\n\n\n\n\n### Equations and supplied-value convention\nThe question supplies **activated gate outputs** and the candidate content. Do not apply sigmoid or tanh to those supplied values again. First update the cell, then calculate the hidden state:\n\n\nThe first equation combines retained old content and admitted candidate content. The second equation is a separate transformation of the updated cell.\n\n\n\n**Step 1: calculate the retained old-cell contribution.**\n\n**Step 2: calculate the admitted candidate contribution.**\n\n**Step 3: add the two contributions to obtain the new cell.**\n\n**Step 4: activate the updated cell.**\n\n**Step 5: multiply by the output gate.**\n\n\n**Final answer:** Cₜ = 0.83 and hₜ ≈ 0.544381. Multiplying the output gate directly by Cₜ would give 0.664 and omit the required tanh operation.\n\n### Related vector calculation: retention alone\n**Question.** Given forget gate f = [0.1, 0.9] and previous cell C = [2, 1], calculate the retained old-cell contribution.\n\n\n\n### Given\n\nForget gate = [0.1,0.9]; old cell = [2,1].\n\nInput-gate and candidate values are not supplied.\n\n### To find\n\nFind only the retained old-cell contribution.\n\n### Formula — which one and why\n\n**LSTM retained old memory.** Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell.\n\nf = activated forget gate; C_old = previous cell state; ⊙ means multiply matching entries.\n\n\n\n**Yaad rakhne ka cue:** Forget gate × old cell = bacha hua purana hissa.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: retain the first old-cell component.**\n\n\n\n\n\n\n\n**Step 2: retain the second old-cell component.**\n\n\n\n\n\n\n\n**Step 3: assemble the two retained components.**\n\nKeep their original order. This gives the following column:\n\n\n\n**Answer:** [0.2, 0.9]. This is the retained contribution, not necessarily the final new cell. The candidate contribution must also be known to calculate the full cell update.\n\n"},{"title":"LSTM numerical when gates must first be calculated","blocks":["Question. At an LSTM time step, xₜ = 1, hₜ₋₁ = 0 and Cₜ₋₁ = 0.2. The affine preactivations of the forget, input and output gates are all 0. The candidate preactivation is 0.5. Calculate the activated gates, candidate cell content, new cell state and new hidden state.
","x = 1; old hidden state = 0; old cell = 0.2.
","Forget/input/output preactivations are all 0.
","Candidate preactivation = 0.5.
","Activate all gates and candidate, then find C and h.
","Activate raw gate scores first. A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.
","z_f,z_i,z_o = gate preactivations; z_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.
","![\"f=\\sigma(z_f),\\quad](\"assets/equations/82486bbd71caa406.svg\")
![\"\\widetilde](\"assets/equations/b3be31d6f9a03979.svg\")
Yaad rakhne ka cue: Raw score pehle activate; activated gate ko dobara sigmoid nahi.
","LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.
","f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.
","Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.
","The affine transformations have already been evaluated in this question. Their outputs are preactivations, so activation functions are still required. Sigmoid is used for f, i and o; tanh is used for the candidate.
","![\"\\sigma(a)=\\frac1{1+e^{-a}}\"](\"assets/equations/d65b06a34ee57b5a.svg\")
![\"C_t=f_tC_{t-1}+i_t\\widetilde](\"assets/equations/3aff429544aa57f2.svg\")
Step 1: activate the three gates.
","![\"f_t=i_t=o_t=\\sigma(0)=\\frac1{1+e^0}=0.5\"](\"assets/equations/5b2fa847b2d2be9a.svg\")
Step 2: activate the candidate preactivation.
","![\"\\widetilde](\"assets/equations/b03e93e6315982fa.svg\")
Step 3: calculate the old and new cell contributions separately.
","![\"f_tC_{t-1}=0.5(0.2)=0.1\"](\"assets/equations/dbc91b31be7f6188.svg\")
![\"i_t\\widetilde](\"assets/equations/3c7c75923a5c216c.svg\")
Step 4: calculate the updated cell.
","![\"C_t=0.1+0.5\\tanh(0.5)\\approx0.331058579\"](\"assets/equations/c56ba22ea129ddab.svg\")
Step 5: calculate the hidden output from the updated cell.
","![\"h_t=0.5\\tanh(C_t)\\approx0.159735818\"](\"assets/equations/39bac92ae8b9ef62.svg\")
Final answer: fₜ = iₜ = oₜ = 0.5; C̃ₜ ≈ 0.462117157; Cₜ ≈ 0.331058579; hₜ ≈ 0.159735818. Retain unrounded calculator values between steps. A gate preactivation of 0 is different from a supplied gate output of 0: the former becomes 0.5 under sigmoid, while the latter would suppress its contribution completely.
","Question. An LSTM has forget, input and output gate raw scores all equal to 0, candidate raw score 0, and old cell 1. Calculate all gates, candidate, new cell and hidden state.
","Raw scores z_f = z_i = z_o = z_candidate = 0.
","Old cell C_old = 1; gate activation sigmoid; candidate activation tanh.
","f,i,o,candidate,C_new,h_new.
","Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it.
","f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory.
","![\"f=i=o=\\sigma(z)\"](\"assets/equations/9b53d0d62e0b3dde.svg\")
![\"\\widetilde](\"assets/equations/a911824e1fcf0c02.svg\")
![\"C_{\\mathrm{new}}=fC_{\\mathrm{old}}+i\\widetilde](\"assets/equations/a0929338a45c9e2f.svg\")
![\"h_{\\mathrm{new}}=o\\tanh(C_{\\mathrm{new}})\"](\"assets/equations/56c0f06510c31697.svg\")
Yaad rakhne ka cue: Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate.
","Step 1: calculate the shared gate value.
","![\"\\sigma(0)=\\frac1{1+e^{-0}}\\approx0.500000000\"](\"assets/equations/4a59e93b5287041f.svg\")
Step 2: calculate the candidate.
","![\"\\widetilde](\"assets/equations/289a06589f8d4dfc.svg\")
Step 3: retain the old-cell part.
","![\"fC_{\\mathrm{old}}=(0.500000000)(1)\\approx0.500000000\"](\"assets/equations/65b5c67b809b7ce0.svg\")
Step 4: admit the new content.
","![\"i\\widetilde](\"assets/equations/9178374ae7a038b8.svg\")
Step 5: add the memory contributions.
","![\"C_{\\mathrm{new}}=0.500000000+(0.000000000)\\approx0.500000000\"](\"assets/equations/d66b879a42e6911e.svg\")
Step 6: apply tanh to the cell.
","![\"\\tanh(C_{\\mathrm{new}})\\approx0.462117157\"](\"assets/equations/55fd6c15079562c2.svg\")
Step 7: apply the output gate.
","![\"h_{\\mathrm{new}}=(0.500000000)(0.462117157)\\approx0.231058579\"](\"assets/equations/9bcf24dd3ef65cfd.svg\")
Final answer: f=i=o≈0.500000000; candidate≈0.000000000; cell≈0.500000000; hidden≈0.231058579.
","Question. An LSTM has forget, input and output gate raw scores all equal to 1, candidate raw score 1, and old cell -0.5. Calculate all gates, candidate, new cell and hidden state.
","Raw scores z_f = z_i = z_o = z_candidate = 1.
","Old cell C_old = -0.5; gate activation sigmoid; candidate activation tanh.
","f,i,o,candidate,C_new,h_new.
","Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it.
","f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory.
","Yaad rakhne ka cue: Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate.
","Step 1: calculate the shared gate value.
","![\"\\sigma(1)=\\frac1{1+e^{-1}}\\approx0.731058579\"](\"assets/equations/0ac3af3fea75043b.svg\")
Step 2: calculate the candidate.
","![\"\\widetilde](\"assets/equations/37268273eb5c85e7.svg\")
Step 3: retain the old-cell part.
","![\"fC_{\\mathrm{old}}=(0.731058579)(-0.5)\\approx-0.365529289\"](\"assets/equations/c8908718c1a2b52e.svg\")
Step 4: admit the new content.
","![\"i\\widetilde](\"assets/equations/0e17a5e53a3e2bd2.svg\")
Step 5: add the memory contributions.
","![\"C_{\\mathrm{new}}=-0.365529289+(0.556769941)\\approx0.191240652\"](\"assets/equations/62b242322f0e2241.svg\")
Step 6: apply tanh to the cell.
","![\"\\tanh(C_{\\mathrm{new}})\\approx0.188942847\"](\"assets/equations/404fb8d8c547f3b7.svg\")
Step 7: apply the output gate.
","![\"h_{\\mathrm{new}}=(0.731058579)(0.188942847)\\approx0.138128289\"](\"assets/equations/d1aa738ae9a7c24f.svg\")
Final answer: f=i=o≈0.731058579; candidate≈0.761594156; cell≈0.191240652; hidden≈0.138128289.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s13","search":"LSTM numerical when gates must first be calculated \n**Question.** At an LSTM time step, xₜ = 1, hₜ₋₁ = 0 and Cₜ₋₁ = 0.2. The affine preactivations of the forget, input and output gates are all 0. The candidate preactivation is 0.5. Calculate the activated gates, candidate cell content, new cell state and new hidden state.\n\n\n\n### Given\n\nx = 1; old hidden state = 0; old cell = 0.2.\n\nForget/input/output preactivations are all 0.\n\nCandidate preactivation = 0.5.\n\n### To find\n\nActivate all gates and candidate, then find C and h.\n\n### Formula — which one and why\n\n**Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.\n\nz_f,z_i,z_o = gate preactivations; z_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.\n\n\n\n\n\n**Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi.\n\n**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.\n\nf,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.\n\n\n\n\n\n**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.\n\n### Solution — step by step\n\n\n\n\n\n### Distinguish preactivations from gate outputs\nThe affine transformations have already been evaluated in this question. Their outputs are preactivations, so activation functions are still required. Sigmoid is used for f, i and o; tanh is used for the candidate.\n\n\n\n\n\n**Step 1: activate the three gates.**\n\n**Step 2: activate the candidate preactivation.**\n\n**Step 3: calculate the old and new cell contributions separately.**\n\n\n**Step 4: calculate the updated cell.**\n\n**Step 5: calculate the hidden output from the updated cell.**\n\n\n**Final answer:** fₜ = iₜ = oₜ = 0.5; C̃ₜ ≈ 0.462117157; Cₜ ≈ 0.331058579; hₜ ≈ 0.159735818. Retain unrounded calculator values between steps. A gate preactivation of 0 is different from a supplied gate output of 0: the former becomes 0.5 under sigmoid, while the latter would suppress its contribution completely.\n\n\n\n### LSTM gates from raw score 0\n\n**Question. An LSTM has forget, input and output gate raw scores all equal to 0, candidate raw score 0, and old cell 1. Calculate all gates, candidate, new cell and hidden state.**\n\n\n\n### Given\n\nRaw scores z_f = z_i = z_o = z_candidate = 0.\n\nOld cell C_old = 1; gate activation sigmoid; candidate activation tanh.\n\n### To find\n\nf,i,o,candidate,C_new,h_new.\n\n### Formula — which one and why\n\nRaw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it.\n\nf retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate.\n\n### Solution — step by step\n\n\n\n**Step 1: calculate the shared gate value.**\n\n\n\n\n\n\n\n**Step 2: calculate the candidate.**\n\n\n\n\n\n\n\n**Step 3: retain the old-cell part.**\n\n\n\n\n\n\n\n**Step 4: admit the new content.**\n\n\n\n\n\n\n\n**Step 5: add the memory contributions.**\n\n\n\n\n\n\n\n**Step 6: apply tanh to the cell.**\n\n\n\n\n\n\n\n**Step 7: apply the output gate.**\n\n\n\n\n\n\n\n**Final answer:** f=i=o≈0.500000000; candidate≈0.000000000; cell≈0.500000000; hidden≈0.231058579.\n\n\n"},{"title":"GRU two gates and one state","blocks":["Definition. A gated recurrent unit (GRU) is a recurrent architecture that uses reset and update gates to control its hidden-state transition. Unlike a standard LSTM, it does not carry a separate cell state alongside the hidden state.
","The reset gate r controls how the previous hidden state enters candidate formation. The update gate z controls the mixture of the old state and the candidate. Both are calculated from the current input and previous hidden state:
","![\"r_t=\\sigma(W_rx_t+U_rh_{t-1}+b_r)\"](\"assets/equations/70d5c123a1a8d5b7.svg\")
![\"z_t=\\sigma(W_zx_t+U_zh_{t-1}+b_z)\"](\"assets/equations/facdae4e8c80fb1f.svg\")
The candidate equation used in these notes applies reset before the recurrent candidate transformation:
","![\"\\widetilde](\"assets/equations/3cb42e5f674e4241.svg\")
These notes define z as the fraction of new candidate admitted:
","![\"h_t=(1-z_t)\\odot](\"assets/equations/949c126edafa8f0a.svg\")
Thus z near 0 favors retention of the old hidden state, while z near 1 favors the candidate. The reset gate affects the candidate calculation; it does not directly multiply the retained old-state term in the final mixture.
","Question. A scalar GRU has hₜ₋₁ = 0.4, reset gate rₜ = 0.5, update gate zₜ = 0.25, xₜ = 1, Wₕ = 0.2, Uₕ = 0.5 and bₕ = 0. Calculate the candidate and new hidden state using hₜ = (1−zₜ)hₜ₋₁ + zₜh̃ₜ.
","Old hidden state = 0.4; r = 0.5; z = 0.25.
","x = 1; W = 0.2; U = 0.5; b = 0.
","Here z weights the candidate in the final mixture.
","Find candidate and new hidden state.
","GRU reset and state mixture. The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate.
","r = reset gate; z = update gate; h_old = previous state; h_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state.
","![\"\\widetilde](\"assets/equations/0823781665f17f77.svg\")
![\"h=(1-z)\\odot](\"assets/equations/27ea37a678ab26e6.svg\")
Yaad rakhne ka cue: Reset candidate ko control; update gate old-new mixture ko control.
","Step 1: apply the reset gate to the previous hidden state.
","![\"r_th_{t-1}=0.5(0.4)=0.2\"](\"assets/equations/1a4ee6cc7b942d74.svg\")
The gates are already supplied as outputs, so no further sigmoid calculation is needed.
","Step 2: calculate the candidate preactivation.
","![\"a_{\\mathrm{candidate}}=W_hx_t+U_h(r_th_{t-1})+b_h\"](\"assets/equations/ca995973009650d3.svg\")
![\"a_{\\mathrm{candidate}}=0.2(1)+0.5(0.2)+0=0.3\"](\"assets/equations/2f40a657a883a8d8.svg\")
Step 3: apply tanh to obtain the candidate.
","![\"\\widetilde](\"assets/equations/2e3e84d154de9f24.svg\")
Step 4: calculate the two mixture contributions.
","![\"](\"assets/equations/56b9a2466f5d57e7.svg\")
![\"z_t\\widetilde](\"assets/equations/8512106d268f635a.svg\")
Step 5: add the contributions.
","![\"h_t=0.3+0.25\\tanh(0.3)\\approx0.372828153\"](\"assets/equations/14fb84144802870c.svg\")
Final answer: candidate ≈ 0.291313 and new hidden state ≈ 0.372828. The new scalar state lies between the old state 0.4 and the candidate 0.291313, as expected for this convex mixture.
","Some texts define z as the fraction of the old state retained:
","![\"h_t=z_th_{t-1}+(1-z_t)\\widetilde](\"assets/equations/360ddf17a95ce9ee.svg\")
Under that explicitly different convention, the same numerical value z = 0.25 gives:
","![\"h_t=0.25(0.4)+0.75\\tanh(0.3)\\approx0.318484459\"](\"assets/equations/e8fba95c9469ec76.svg\")
This is not the answer to the worked question above, which states the first convention. To represent an identical mixture while switching conventions, the gate values must be complemented as well. Always identify the final mixture equation before substituting values.
","A basic GRU has three affine transformations: reset, update and candidate. A standard LSTM has four: forget, input, output and candidate. With equal input and hidden dimensions and the same bias convention, GRU therefore has fewer recurrent parameters. This is a parameter-count comparison, not a guarantee of higher accuracy or faster execution for every task.
","Question. Use h_new = z h_old+(1−z) candidate. Given h_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.
","h_old = 0.4; z = 0; r = 0.5; x = 1.
","W = 0.2; U = 0.5; b = 0; z is the retain-old gate.
","Candidate and final hidden state.
","The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.
","The question explicitly makes z multiply the old state. r and z do different jobs.
","![\"\\widetilde](\"assets/equations/2a33ca0d61aaaa89.svg\")
![\"h_{\\mathrm{new}}=zh_{\\mathrm{old}}+(1-z)\\widetilde](\"assets/equations/88df46d24a37114e.svg\")
Yaad rakhne ka cue: Reset sirf candidate mein; update mix mein original old state.
","Step 1: reset the old state for candidate formation.
","![\"rh_{\\mathrm{old}}=0.5(0.4)=0.2\"](\"assets/equations/298840de1c01f101.svg\")
Step 2: calculate the candidate raw sum.
","![\"a=0.2(1)+0.5(0.2)+0=0.3\"](\"assets/equations/708a856537eccf89.svg\")
Step 3: apply tanh.
","![\"\\widetilde](\"assets/equations/c5d5bc478f8a3566.svg\")
Step 4: retain the old-state fraction.
","![\"0(0.4)=0\"](\"assets/equations/81c714ea0d0f3d30.svg\")
Step 5: calculate the candidate fraction.
","![\"(1-0)(0.291312612)\\approx0.291312612\"](\"assets/equations/5b85f83251c9025c.svg\")
Step 6: add the two fractions.
","![\"h_{\\mathrm{new}}=0+(0.291312612)\\approx0.291312612\"](\"assets/equations/fb12f65fb3a6f99d.svg\")
Final answer: Candidate ≈ 0.291312612; new state ≈ 0.291312612.
","Question. Use h_new = z h_old+(1−z) candidate. Given h_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.
","h_old = -0.2; z = 0.8; r = 0.5; x = 1.
","W = 0.2; U = 0.5; b = 0; z is the retain-old gate.
","Candidate and final hidden state.
","The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.
","The question explicitly makes z multiply the old state. r and z do different jobs.
","Yaad rakhne ka cue: Reset sirf candidate mein; update mix mein original old state.
","Step 1: reset the old state for candidate formation.
","![\"rh_{\\mathrm{old}}=0.5(-0.2)=-0.1\"](\"assets/equations/c07f11824db74c75.svg\")
Step 2: calculate the candidate raw sum.
","![\"a=0.2(1)+0.5(-0.1)+0=0.15\"](\"assets/equations/177d51be8807173f.svg\")
Step 3: apply tanh.
","![\"\\widetilde](\"assets/equations/22b0ab9fe31c5dbb.svg\")
Step 4: retain the old-state fraction.
","![\"0.8(-0.2)=-0.16\"](\"assets/equations/8d5d62808c22a4a6.svg\")
Step 5: calculate the candidate fraction.
","![\"(1-0.8)(0.148885034)\\approx0.029777007\"](\"assets/equations/47661b22efc0daeb.svg\")
Step 6: add the two fractions.
","![\"h_{\\mathrm{new}}=-0.16+(0.029777007)\\approx-0.130222993\"](\"assets/equations/a559a88735fecf3e.svg\")
Final answer: Candidate ≈ 0.148885034; new state ≈ -0.130222993.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s14","search":"GRU two gates and one state \n**Definition.** A gated recurrent unit (GRU) is a recurrent architecture that uses reset and update gates to control its hidden-state transition. Unlike a standard LSTM, it does not carry a separate cell state alongside the hidden state.\n\n### Gate equations and the convention used here\nThe reset gate r controls how the previous hidden state enters candidate formation. The update gate z controls the mixture of the old state and the candidate. Both are calculated from the current input and previous hidden state:\n\n\nThe candidate equation used in these notes applies reset before the recurrent candidate transformation:\n\nThese notes define z as the fraction of **new candidate** admitted:\n\nThus z near 0 favors retention of the old hidden state, while z near 1 favors the candidate. The reset gate affects the candidate calculation; it does not directly multiply the retained old-state term in the final mixture.\n\n\n\n### Complete numerical with supplied gate values\n**Question.** A scalar GRU has hₜ₋₁ = 0.4, reset gate rₜ = 0.5, update gate zₜ = 0.25, xₜ = 1, Wₕ = 0.2, Uₕ = 0.5 and bₕ = 0. Calculate the candidate and new hidden state using hₜ = (1−zₜ)hₜ₋₁ + zₜh̃ₜ.\n\n\n\n### Given\n\nOld hidden state = 0.4; r = 0.5; z = 0.25.\n\nx = 1; W = 0.2; U = 0.5; b = 0.\n\nHere z weights the candidate in the final mixture.\n\n### To find\n\nFind candidate and new hidden state.\n\n### Formula — which one and why\n\n**GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate.\n\nr = reset gate; z = update gate; h_old = previous state; h_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state.\n\n\n\n\n\n**Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1: apply the reset gate to the previous hidden state.**\n\nThe gates are already supplied as outputs, so no further sigmoid calculation is needed.\n\n**Step 2: calculate the candidate preactivation.**\n\n\n\n**Step 3: apply tanh to obtain the candidate.**\n\n\n**Step 4: calculate the two mixture contributions.**\n\n\n\n**Step 5: add the contributions.**\n\n**Final answer:** candidate ≈ 0.291313 and new hidden state ≈ 0.372828. The new scalar state lies between the old state 0.4 and the candidate 0.291313, as expected for this convex mixture.\n\n### If the question uses the other update-gate convention\nSome texts define z as the fraction of the **old state retained**:\n\nUnder that explicitly different convention, the same numerical value z = 0.25 gives:\n\nThis is not the answer to the worked question above, which states the first convention. To represent an identical mixture while switching conventions, the gate values must be complemented as well. Always identify the final mixture equation before substituting values.\n\n### Relationship to LSTM\nA basic GRU has three affine transformations: reset, update and candidate. A standard LSTM has four: forget, input, output and candidate. With equal input and hidden dimensions and the same bias convention, GRU therefore has fewer recurrent parameters. This is a parameter-count comparison, not a guarantee of higher accuracy or faster execution for every task.\n\n\n\n### Retain-old GRU with gate 0\n\n**Question. Use h_new = z h_old+(1−z) candidate. Given h_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new "},{"title":"Comparing the three recurrent models","blocks":["| Feature | Vanilla RNN | LSTM | GRU |
|---|---|---|---|
| Main state | h | h and C | h |
| Gates | None in vanilla cell | Forget input output | Reset update |
| Long dependencies | Often difficult to learn | Designed to help retain useful long context | Designed to help retain useful long context |
| Parameters at equal dimensions | Usually fewest | Usually most of these three | Usually between vanilla RNN and LSTM |
| Typical choice | Simple short sequences | Complex long-context sequences | Efficient gated sequence baseline |
With input size d and hidden size H, a basic single-bias vanilla recurrent layer has H(d+H+1) parameters; a standard LSTM has 4H(d+H+1); a basic GRU has 3H(d+H+1). These omit output layers; library versions with extra bias vectors differ. The factor four in LSTM counts three gates plus one candidate transform.
","Question. Compare a vanilla RNN, an LSTM and a GRU. Explain the roles of the LSTM gates and state the GRU convention used in this course.
"],"answer":["A vanilla RNN repeatedly transforms the current input and previous hidden state. LSTMs and GRUs add learned gates that control the retention and replacement of information. Gates usually use sigmoid, so their entries lie between 0 and 1; gate multiplication controls how much of a corresponding value is passed.
","An LSTM carries a cell state c and a hidden state h. Its forget gate f controls the old cell contribution. Its input gate i controls admission of a new candidate c-tilde. Its output gate o controls the exposed hidden state.
","![\"c_t=f_t\\odot](\"assets/equations/d70555acf8f3a0de.svg\")
![\"h_t=o_t\\odot\\tanh(c_t)\"](\"assets/equations/be4b02b5e68c7633.svg\")
The candidate is commonly formed using tanh, so it can be negative. Gate multiplication is elementwise. The updated cell is not the same as the hidden output; the second equation is another required step.
","A GRU carries one hidden state rather than a separate LSTM-style cell state. A reset gate r controls how previous hidden information contributes to candidate formation. An update gate z mixes the old state and candidate. This guide uses:
","Thus z near 1 favors the new candidate under this convention. Some sources exchange z and 1 − z; always follow the equation printed in the question.
","| Model | Carried state | Main gates | Typical parameter implication for the same dimensions |
|---|---|---|---|
| Vanilla RNN | Hidden state | None of these memory gates | Fewest affine transformations |
| LSTM | Cell and hidden states | Forget, input and output gates | Four affine transformations including candidate |
| GRU | Hidden state | Reset and update gates | Three affine transformations including candidate |
Gated models improve control over information flow but do not guarantee perfect indefinite memory. Their benefit must be evaluated for the sequence task.
"],"kind":"theory","id":"tee-c9-s16","search":"Model answer: LSTM and GRU memory **Question.** Compare a vanilla RNN, an LSTM and a GRU. Explain the roles of the LSTM gates and state the GRU convention used in this guide.\n\n**Model answer.**\nA vanilla RNN repeatedly transforms the current input and previous hidden state. LSTMs and GRUs add learned gates that control the retention and replacement of information. Gates usually use sigmoid, so their entries lie between 0 and 1; gate multiplication controls how much of a corresponding value is passed.\n\n\n\n\n### LSTM\nAn LSTM carries a cell state c and a hidden state h. Its forget gate f controls the old cell contribution. Its input gate i controls admission of a new candidate c-tilde. Its output gate o controls the exposed hidden state.\n\n\nThe candidate is commonly formed using tanh, so it can be negative. Gate multiplication is elementwise. The updated cell is not the same as the hidden output; the second equation is another required step.\n\n### GRU\nA GRU carries one hidden state rather than a separate LSTM-style cell state. A reset gate r controls how previous hidden information contributes to candidate formation. An update gate z mixes the old state and candidate. This guide uses:\n\nThus z near 1 favors the new candidate under this convention. Some sources exchange z and 1 − z; always follow the equation printed in the question.\n\n| Model | Carried state | Main gates | Typical parameter implication for the same dimensions |\n| Vanilla RNN | Hidden state | None of these memory gates | Fewest affine transformations |\n| LSTM | Cell and hidden states | Forget, input and output gates | Four affine transformations including candidate |\n| GRU | Hidden state | Reset and update gates | Three affine transformations including candidate |\n\nGated models improve control over information flow but do not guarantee perfect indefinite memory. Their benefit must be evaluated for the sequence task.\n"},{"title":"Sequence mappings and applications","blocks":["Definition. A sequence-model configuration specifies how many inputs and outputs are processed and how their time indices relate. One-to-many, many-to-one and many-to-many configurations correspond to different prediction tasks.
","| Mapping | Meaning | Example and output |
|---|---|---|
| One to many | One input conditions several outputs | Image captioning: image to a word sequence |
| Many to one | Sequence summarized as one output | Review sentiment: words to positive neutral or negative |
| Many to many aligned | Output associated with each input time | Video frame labeling or per-word tagging |
| Many to many unaligned | Output sequence length may differ | Encoder-decoder translation |
Music generation can use earlier notes to predict the next note repeatedly. Next-word prediction uses the prefix to produce a probability distribution over a vocabulary. A word with the largest predicted probability is a simple choice, but the precise prediction cannot be calculated without the learned model's weights.
","Sentence explanation: In “The dogs that bark near my house are loud,” the model should retain the plural subject across the intervening words when choosing “are”. Gates can retain subject information and admit later context. This is an interpretation of what a trained model may learn, not a claim that an individual gate is guaranteed to represent grammar.
","Sentiment model answer: tokenize the review, map tokens to numerical embeddings, process them with an RNN/LSTM/GRU, use a final state or pooled sequence representation, then a dense layer and softmax over three sentiment classes. Train with categorical cross-entropy on labeled reviews; evaluate on held-out data. A bidirectional model is not needed for this answer and is not explicitly listed in the current syllabus.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s17","search":"Sequence mappings and applications **Definition.** A sequence-model configuration specifies how many inputs and outputs are processed and how their time indices relate. One-to-many, many-to-one and many-to-many configurations correspond to different prediction tasks.\n\n### Explanation and working\n\n| Mapping | Meaning | Example and output |\n| One to many | One input conditions several outputs | Image captioning: image to a word sequence |\n| Many to one | Sequence summarized as one output | Review sentiment: words to positive neutral or negative |\n| Many to many aligned | Output associated with each input time | Video frame labeling or per-word tagging |\n| Many to many unaligned | Output sequence length may differ | Encoder-decoder translation |\nMusic generation can use earlier notes to predict the next note repeatedly. Next-word prediction uses the prefix to produce a probability distribution over a vocabulary. A word with the largest predicted probability is a simple choice, but the precise prediction cannot be calculated without the learned model's weights.\n**Sentence explanation: ** In “The dogs that bark near my house are loud,” the model should retain the plural subject across the intervening words when choosing “are”. Gates can retain subject information and admit later context. This is an interpretation of what a trained model may learn, not a claim that an individual gate is guaranteed to represent grammar.\n**Sentiment model answer: ** tokenize the review, map tokens to numerical embeddings, process them with an RNN/LSTM/GRU, use a final state or pooled sequence representation, then a dense layer and softmax over three sentiment classes. Train with categorical cross-entropy on labeled reviews; evaluate on held-out data. A bidirectional model is not needed for this answer and is not explicitly listed in the current syllabus."},{"title":"Recurrent networks: the complete relationship","blocks":["A vanilla RNN repeatedly combines the new input and previous hidden state with shared weights. Backpropagation through time accumulates contributions from every use of those shared weights. LSTM adds an explicit cell state with forget, input and output gates. GRU mixes an old hidden state and a candidate using reset and update gates.
","For every recurrent calculation, compute the gate or hidden preactivation first, apply its stated activation, then combine the resulting values. Elementwise gate multiplication is different from matrix multiplication. The stated GRU update convention determines whether z retains the old state or selects the candidate.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s18","search":"Recurrent networks: the complete relationship A vanilla RNN repeatedly combines the new input and previous hidden state with shared weights. Backpropagation through time accumulates contributions from every use of those shared weights. LSTM adds an explicit cell state with forget, input and output gates. GRU mixes an old hidden state and a candidate using reset and update gates.\n\n\n\nFor every recurrent calculation, compute the gate or hidden preactivation first, apply its stated activation, then combine the resulting values. Elementwise gate multiplication is different from matrix multiplication. The stated GRU update convention determines whether z retains the old state or selects the candidate.\n"},{"title":"Gate logic without memorizing a story","blocks":["In an LSTM, let the previous cell be 2, forget gate 0.5, input gate 0.25, candidate −0.4 and output gate 1. Calculate the retained part, admitted part, new cell and hidden output.
","Step 1 retained content: 0.5 × 2 = 1.
","Step 2 admitted content: 0.25 × (−0.4) = −0.1. A candidate can be negative even though the gate itself is between zero and one.
","Step 3 cell: 1 + (−0.1) = 0.9.
","Step 4 hidden: output gate × tanh(cell) = 1 × tanh(0.9) ≈ 0.716298. The hidden output is not simply 0.9. Explain this distinction once before attempting another LSTM numerical.
"],"question":[],"answer":[],"kind":"read","id":"tee-c9-s19","search":"Gate logic without memorizing a story In an LSTM, let the previous cell be 2, forget gate 0.5, input gate 0.25, candidate −0.4 and output gate 1. Calculate the retained part, admitted part, new cell and hidden output.\n\n\n\n**Step 1 retained content:** 0.5 × 2 = 1.\n\n**Step 2 admitted content:** 0.25 × (−0.4) = −0.1. A candidate can be negative even though the gate itself is between zero and one.\n**Step 3 cell:** 1 + (−0.1) = 0.9.\n\n**Step 4 hidden:** output gate × tanh(cell) = 1 × tanh(0.9) ≈ 0.716298. The hidden output is not simply 0.9. Explain this distinction once before attempting another LSTM numerical.\n\n"}]},{"n":10,"title":"Current MT1 and the two teacher backprop examples","sections":[{"title":"Start here","blocks":["Sources S03, S01 and the three Teams photographs supplied later. MT1 establishes prerequisites; it does not prove that earlier concepts cannot reappear in MT2.
","Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible.
"],"kind":"read","id":"tee-c10-intro"},{"title":"MT1A Q1a","question":["Question (2 marks). Define deep learning and explain its relationship with AI and machine learning.
"],"answer":["Artificial intelligence is the broad goal of building systems that perform tasks associated with intelligence. Machine learning is a subset that learns patterns from data. Deep learning is a subset of machine learning using neural networks with multiple processing layers to learn representations. Thus DL ⊂ ML ⊂ AI. Example: a CNN learning image features is deep learning; a decision tree trained on tabular data is machine learning without being deep learning.
"],"kind":"practice","id":"tee-c10-s0","search":"MT1A Q1a **Question (2 marks).** Define deep learning and explain its relationship with AI and machine learning.\n\n**Complete worked answer.**\nArtificial intelligence is the broad goal of building systems that perform tasks associated with intelligence. Machine learning is a subset that learns patterns from data. Deep learning is a subset of machine learning using neural networks with multiple processing layers to learn representations. Thus DL ⊂ ML ⊂ AI. Example: a CNN learning image features is deep learning; a decision tree trained on tabular data is machine learning without being deep learning.\n"},{"title":"MT1A Q1b","question":["Question (2 marks). List three applications of deep learning and explain any one.
"],"answer":["Three applications are image recognition, speech recognition and language translation. In image recognition, a CNN applies filters to learn local edges and textures, combines them into more complex features, and uses an output layer to assign class probabilities. It learns these features from labeled examples instead of requiring every image feature to be manually designed.
"],"kind":"practice","id":"tee-c10-s1","search":"MT1A Q1b **Question (2 marks).** List three applications of deep learning and explain any one.\n\n**Complete worked answer.**\nThree applications are image recognition, speech recognition and language translation. In image recognition, a CNN applies filters to learn local edges and textures, combines them into more complex features, and uses an output layer to assign class probabilities. It learns these features from labeled examples instead of requiring every image feature to be manually designed.\n"},{"title":"MT1A Q2","question":["Question (3 marks). A sigmoid neuron has inputs [2,1,3], weights [0.4,−0.2,0.5] and bias 0.1. Compute its output.
"],"answer":["Inputs x₁ = 2,x₂ = 1,x₃ = 3.
","Matching weights w₁ = 0.4,w₂ = −0.2,w₃ = 0.5.
","Bias b = 0.1; activation = sigmoid.
","Calculate net input z and activated output a.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid output. The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.
","z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.
","Yaad rakhne ka cue: Minus-z ki e power, plus one, phir one divided by whole denominator.
","Step 1: multiply input 1 by weight 1.
","Step 2: multiply input 2 by weight 2.
","The minus sign belongs to the weight, so the contribution stays negative.
","Step 3: multiply input 3 by weight 3.
","Step 4: add the three contributions.
","Step 5: add the bias once.
","This is the preactivation. It is not yet the requested sigmoid output.
","Step 6: negate z for the exponential.
","Step 7: calculate the complete denominator.
","Step 8: divide one by that denominator.
","Final answer: sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025.
"],"kind":"practice","id":"tee-c10-s2","search":"MT1A Q2 **Question (3 marks).** A sigmoid neuron has inputs [2,1,3], weights [0.4,−0.2,0.5] and bias 0.1. Compute its output.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInputs x₁ = 2,x₂ = 1,x₃ = 3.\n\nMatching weights w₁ = 0.4,w₂ = −0.2,w₃ = 0.5.\n\nBias b = 0.1; activation = sigmoid.\n\n### To find\n\nCalculate net input z and activated output a.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal.\n\nz = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base.\n\n\n\n**Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator.\n\n### Solution — step by step\n\n\n\n\n\n\n**Step 1: multiply input 1 by weight 1.**\n\n\n**Step 2: multiply input 2 by weight 2.**\n\nThe minus sign belongs to the weight, so the contribution stays negative.\n\n**Step 3: multiply input 3 by weight 3.**\n\n\n**Step 4: add the three contributions.**\n\n\n**Step 5: add the bias once.**\n\nThis is the preactivation. It is not yet the requested sigmoid output.\n\n**Step 6: negate z for the exponential.**\n\n\n\n**Step 7: calculate the complete denominator.**\n\n\n**Step 8: divide one by that denominator.**\n\n**Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025.\n"},{"title":"MT1A Q3","question":["Question (3 marks). For z = [−1,0,1], calculate sigmoid, softmax and Leaky ReLU with α = 0.01 (an explicit assumption because the paper does not give the slope).
"],"answer":["Scores z = [−1,0,1].
","Calculate sigmoid, softmax and Leaky ReLU.
","The paper omits the Leaky slope: α = 0.01 is the stated practice assumption.
","Calculate every component under each of the three activations.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.
","zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.
","Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Sigmoid is applied separately: σ(−1) = = 0.268941421; σ(0) = 1/2 = 0.5; σ(1) = = 0.731058579.
Step 1 stabilize: subtract the largest logit 1 from every value: [-2, -1, 0]. This does not change softmax.
","Step 2 exponentiate: exp values = [0.135335283, 0.367879441, 1].
","Step 3 denominator: sum = 1.503214724.
","Step 4 divide each value by this same sum: probabilities = [0.090030573, 0.244728471, 0.665240956]. Check: all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.
","Leaky ReLU multiplies only negative values by α: [0.01 × (−1),0,1] = [−0.01,0,1]. Softmax normalizes the entire vector; independent sigmoid values need not sum to 1.
"],"kind":"practice","id":"tee-c10-s3","search":"MT1A Q3 **Question (3 marks).** For z = [−1,0,1], calculate sigmoid, softmax and Leaky ReLU with α = 0.01 (an explicit assumption because the paper does not give the slope).\n\n**Complete worked answer.**\n\n\n\n### Given\n\nScores z = [−1,0,1].\n\nCalculate sigmoid, softmax and Leaky ReLU.\n\nThe paper omits the Leaky slope: α = 0.01 is the stated practice assumption.\n\n### To find\n\nCalculate every component under each of the three activations.\n\n### Formula — which one and why\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.\n\nzⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\n\n\nSigmoid is applied separately: σ(−1) = \\(\\frac{1}{1+e^{1}}\\) = 0.268941421; σ(0) = 1/2 = 0.5; σ(1) = \\(\\frac{1}{1+e^{-1}}\\) = 0.731058579.\n**Step 1 stabilize: ** subtract the largest logit 1 from every value: [-2, -1, 0]. This does not change softmax.\n\n**Step 2 exponentiate: ** exp values = [0.135335283, 0.367879441, 1].\n\n**Step 3 denominator: ** sum = 1.503214724.\n\n**Step 4 divide each value by this same sum: ** probabilities = **[0.090030573, 0.244728471, 0.665240956]**. **Check: ** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.\nLeaky ReLU multiplies only negative values by α: [0.01 × (−1),0,1] = **[−0.01,0,1]**. Softmax normalizes the entire vector; independent sigmoid values need not sum to 1.\n"},{"title":"MT1A Q4","question":["Question (3 marks). Input x = [2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1]; W₂ is shown below; b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.
","Input x = [2].
","W₁ is printed with the question.
","b₁ = [0.1].
","W₂ is printed with the question.
","b₂ = [0.2].
","Hidden and output activations are ReLU.
","Target y = [1].
","Find all preactivations, hidden activations and final outputs. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","Output neuron 1:
","Step 2 loss convention. The question specifies half squared loss. No learning rate is supplied, so report gradients, not invented weight updates.
","Prediction minus target = [0.08]. Squared errors = [0.0064]. Half-sum squared loss = 0.0032; mean squared error = 0.0064. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","The ReLU derivative is 1 here because this preactivation is positive.
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Requested answers: dL/dw₂ = 0.088 and dL/db₂ = 0.08. Hidden gradients above explain the dependency and are additional practice.
"],"kind":"practice","id":"tee-c10-s4","search":"MT1A Q4 **Question (3 marks).** Input x = [2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1]; W₂ is shown below; b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput x = [2].\n\nW₁ is printed with the question.\n\nb₁ = [0.1].\n\nW₂ is printed with the question.\n\nb₂ = [0.2].\n\nHidden and output activations are ReLU.\n\nTarget y = [1].\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** The question specifies half squared loss. No learning rate is supplied, so report gradients, not invented weight updates.\nPrediction minus target = [0.08]. Squared errors = [0.0064]. Half-sum squared loss = 0.0032; mean squared error = 0.0064. These are different unless the number of output components makes the scale coincide.\n**Output delta 1:**\n\nThe ReLU derivative is 1 here because this preactivation is positive.\n**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.\nh1 → o1: gradient = (0.08) × 1.1 = 0.088.\n**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.\n**Hidden 1 downstream contribution:**\n\n\n**Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output.\nx1 → h1: gradient = (0.064) × (2) = 0.128.\n**Requested answers: ** dL/dw₂ = **0.088** and dL/db₂ = **0.08**. Hidden gradients above explain the dependency and are additional practice.\n"},{"title":"MT1B Q1a","question":["Question (2 marks). State two motivations for using deep learning.
"],"answer":["First, deep networks can learn useful representations directly from complex data such as images and speech, reducing dependence on manually engineered features. Second, multiple nonlinear layers can model complicated patterns and relationships when suitable data, computation and training are available. Example: a CNN can learn edges, parts and object features at successive levels.
"],"kind":"practice","id":"tee-c10-s5","search":"MT1B Q1a **Question (2 marks).** State two motivations for using deep learning.\n\n**Complete worked answer.**\nFirst, deep networks can learn useful representations directly from complex data such as images and speech, reducing dependence on manually engineered features. Second, multiple nonlinear layers can model complicated patterns and relationships when suitable data, computation and training are available. Example: a CNN can learn edges, parts and object features at successive levels.\n"},{"title":"MT1B Q1b","question":["Question (2 marks). Differentiate machine learning and deep learning with respect to feature extraction and data requirements.
"],"answer":["Traditional machine-learning pipelines often rely on manually selected or engineered features; deep networks often learn feature representations jointly with the task. Training a large deep network from scratch commonly needs more data than simpler models. However, pretrained deep features can be useful even with a small target dataset. These are practical tendencies, not absolute definitions.
"],"kind":"practice","id":"tee-c10-s6","search":"MT1B Q1b **Question (2 marks).** Differentiate machine learning and deep learning with respect to feature extraction and data requirements.\n\n**Complete worked answer.**\nTraditional machine-learning pipelines often rely on manually selected or engineered features; deep networks often learn feature representations jointly with the task. Training a large deep network from scratch commonly needs more data than simpler models. However, pretrained deep features can be useful even with a small target dataset. These are practical tendencies, not absolute definitions.\n"},{"title":"MT1B Q2","question":["Question (3 marks). Show how a perceptron with w₁ = w₂ = 0.6 and bias −0.8 implements AND.
"],"answer":["Binary inputs 00,01,10,11.
","w₁ = w₂ = 0.6; b = −0.8; output 1 when net ≥ 0.
","Calculate all four net values and outputs to show AND.
","Threshold logic. Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.
","step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.
","Yaad rakhne ka cue: Har input row ka sum nikalo, phir threshold lagao.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","Step 4: add the bias once.
","Step 5: apply the stated threshold.
","z = -0.8; output = 0 because output is 1 exactly when z ≥ 0.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","Step 4: add the bias once.
","Step 5: apply the stated threshold.
","z = -0.2; output = 0 because output is 1 exactly when z ≥ 0.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","Step 4: add the bias once.
","Step 5: apply the stated threshold.
","z = -0.2; output = 0 because output is 1 exactly when z ≥ 0.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","Step 4: add the bias once.
","Step 5: apply the stated threshold.
","z = 0.4; output = 1 because output is 1 exactly when z ≥ 0.
","Final answer: outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND.
"],"kind":"practice","id":"tee-c10-s7","search":"MT1B Q2 **Question (3 marks).** Show how a perceptron with w₁ = w₂ = 0.6 and bias −0.8 implements AND.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nBinary inputs 00,01,10,11.\n\nw₁ = w₂ = 0.6; b = −0.8; output 1 when net ≥ 0.\n\n### To find\n\nCalculate all four net values and outputs to show AND.\n\n### Formula — which one and why\n\n**Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.\n\nstep(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao.\n\n### Solution — step by step\n\n\n\n\n### Input (0, 0)\n\n**Step 1: multiply input 1 by its weight.**\n\n\n\n\n\n\n\n**Step 2: multiply input 2 by its weight.**\n\n\n\n\n\n\n\n**Step 3: add the two products.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply the stated threshold.**\n\nz = -0.8; output = 0 because output is 1 exactly when z ≥ 0.\n\n\n### Input (0, 1)\n\n**Step 1: multiply input 1 by its weight.**\n\n\n\n\n\n\n\n**Step 2: multiply input 2 by its weight.**\n\n\n\n\n\n\n\n**Step 3: add the two products.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply the stated threshold.**\n\nz = -0.2; output = 0 because output is 1 exactly when z ≥ 0.\n\n\n### Input (1, 0)\n\n**Step 1: multiply input 1 by its weight.**\n\n\n\n\n\n\n\n**Step 2: multiply input 2 by its weight.**\n\n\n\n\n\n\n\n**Step 3: add the two products.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply the stated threshold.**\n\nz = -0.2; output = 0 because output is 1 exactly when z ≥ 0.\n\n\n### Input (1, 1)\n\n**Step 1: multiply input 1 by its weight.**\n\n\n\n\n\n\n\n**Step 2: multiply input 2 by its weight.**\n\n\n\n\n\n\n\n**Step 3: add the two products.**\n\n\n\n\n\n\n\n**Step 4: add the bias once.**\n\n\n\n\n\n\n\n**Step 5: apply the stated threshold.**\n\nz = 0.4; output = 1 because output is 1 exactly when z ≥ 0.\n\n**Final answer:** outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND.\n\n\n"},{"title":"MT1B Q3","question":["Question (3 marks). Calculate sigmoid, ReLU and tanh for z = [−2,0,2].
"],"answer":["Input scores = [−2,0,2].
","Required activations: sigmoid, ReLU and tanh.
","Calculate all three activation outputs for each score.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Tanh and its slope. Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².
","z = value entering tanh; a = its output. A prime means derivative, or local rate of change.
","Yaad rakhne ka cue: Tanh signed output deta hai; slope = 1 minus output ka square.
","Step 1: negate z for sigmoid.
","Step 2: evaluate the exponential.
","Step 3: add one to form the denominator.
","Step 4: take the reciprocal for sigmoid.
","Step 5: compare z with zero for ReLU.
","Step 6: calculate the tanh numerator.
","Step 7: calculate the tanh denominator.
","Step 8: divide for tanh.
","Step 1: negate z for sigmoid.
","Step 2: evaluate the exponential.
","Step 3: add one to form the denominator.
","Step 4: take the reciprocal for sigmoid.
","Step 5: compare z with zero for ReLU.
","Step 6: calculate the tanh numerator.
","Step 7: calculate the tanh denominator.
","Step 8: divide for tanh.
","Step 1: negate z for sigmoid.
","Step 2: evaluate the exponential.
","Step 3: add one to form the denominator.
","Step 4: take the reciprocal for sigmoid.
","Step 5: compare z with zero for ReLU.
","Step 6: calculate the tanh numerator.
","Step 7: calculate the tanh denominator.
","Step 8: divide for tanh.
","Final answer: sigmoid = [0.119202922, 0.5, 0.880797078]; ReLU = [0, 0, 2]; tanh = [−0.964027580, 0, 0.964027580].
"],"kind":"practice","id":"tee-c10-s8","search":"MT1B Q3 **Question (3 marks).** Calculate sigmoid, ReLU and tanh for z = [−2,0,2].\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput scores = [−2,0,2].\n\nRequired activations: sigmoid, ReLU and tanh.\n\n### To find\n\nCalculate all three activation outputs for each score.\n\n### Formula — which one and why\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n**Tanh and its slope.** Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².\n\nz = value entering tanh; a = its output. A prime means derivative, or local rate of change.\n\n\n\n\n\n**Yaad rakhne ka cue:** Tanh signed output deta hai; slope = 1 minus output ka square.\n\n### Solution — step by step\n\n\n\n\n\n\n### Preactivation z = -2\n\n**Step 1: negate z for sigmoid.**\n\n\n\n\n\n\n\n**Step 2: evaluate the exponential.**\n\n\n\n\n\n\n\n**Step 3: add one to form the denominator.**\n\n\n\n\n\n\n\n**Step 4: take the reciprocal for sigmoid.**\n\n\n\n\n\n\n\n**Step 5: compare z with zero for ReLU.**\n\n\n\n\n\n\n\n**Step 6: calculate the tanh numerator.**\n\n\n\n\n\n\n\n**Step 7: calculate the tanh denominator.**\n\n\n\n\n\n\n\n**Step 8: divide for tanh.**\n\n\n\n\n\n\n\n\n### Preactivation z = 0\n\n**Step 1: negate z for sigmoid.**\n\n\n\n\n\n\n\n**Step 2: evaluate the exponential.**\n\n\n\n\n\n\n\n**Step 3: add one to form the denominator.**\n\n\n\n\n\n\n\n**Step 4: take the reciprocal for sigmoid.**\n\n\n\n\n\n\n\n**Step 5: compare z with zero for ReLU.**\n\n\n\n\n\n\n\n**Step 6: calculate the tanh numerator.**\n\n\n\n\n\n\n\n**Step 7: calculate the tanh denominator.**\n\n\n\n\n\n\n\n**Step 8: divide for tanh.**\n\n\n\n\n\n\n\n\n### Preactivation z = 2\n\n**Step 1: negate z for sigmoid.**\n\n\n\n\n\n\n\n**Step 2: evaluate the exponential.**\n\n\n\n\n\n\n\n**Step 3: add one to form the denominator.**\n\n\n\n\n\n\n\n**Step 4: take the reciprocal for sigmoid.**\n\n\n\n\n\n\n\n**Step 5: compare z with zero for ReLU.**\n\n\n\n\n\n\n\n**Step 6: calculate the tanh numerator.**\n\n\n\n\n\n\n\n**Step 7: calculate the tanh denominator.**\n\n\n\n\n\n\n\n**Step 8: divide for tanh.**\n\n\n\n\n\n\n\n**Final answer:** sigmoid = [0.119202922, 0.5, 0.880797078]; ReLU = [0, 0, 2]; tanh = [−0.964027580, 0, 0.964027580].\n"},{"title":"MT1B Q4","question":["Question (3 marks). Input x = [3]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are sigmoid. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.
","Input x = [3].
","W₁ is printed with the question.
","b₁ = [0.2].
","W₂ is printed with the question.
","b₂ = [0.1].
","Hidden and output activations are sigmoid.
","Target y = [1].
","Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","Output neuron 1:
","Step 2 loss convention. Use the half squared loss stated in this question. No learning rate is supplied.
","Prediction minus target = [-0.358631142]. Squared errors = [0.128616296]. Half-sum squared loss = 0.064308148; mean squared error = 0.128616296. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Requested answers: dL/dw₂ = −0.066172539; dL/db₂ = −0.082490487. Negative gradients mean ordinary gradient descent would increase those parameters.
"],"kind":"practice","id":"tee-c10-s9","search":"MT1B Q4 **Question (3 marks).** Input x = [3]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are sigmoid. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput x = [3].\n\nW₁ is printed with the question.\n\nb₁ = [0.2].\n\nW₂ is printed with the question.\n\nb₂ = [0.1].\n\nHidden and output activations are sigmoid.\n\nTarget y = [1].\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** Use the half squared loss stated in this question. No learning rate is supplied.\nPrediction minus target = [-0.358631142]. Squared errors = [0.128616296]. Half-sum squared loss = 0.064308148; mean squared error = 0.128616296. These are different unless the number of output components makes the scale coincide.\n**Output delta 1:**\n\nThe derivative is o(1−o).\n**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.\nh1 → o1: gradient = (-0.082490487) × 0.802183889 = -0.066172539.\n**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.\n**Hidden 1 downstream contribution:**\n\n\n**Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output.\nx1 → h1: gradient = (-0.007853997) × (3) = -0.02356199.\n**Requested answers: ** dL/dw₂ = **−0.066172539**; dL/db₂ = **−0.082490487**. Negative gradients mean ordinary gradient descent would increase those parameters.\n"},{"title":"Teacher vectorized ReLU example","blocks":["Question. Input x = [2, 3]. Each row below lists the incoming weights of one receiving neuron. W₁ = ; b₁ = [0, 0]; W₂ = ; b₂ = [0]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.05. Use half squared loss and perform forward and backward passes.
Input x = [2, 3].
","W₁ = .
b₁ = [0, 0].
","W₂ = .
b₂ = [0].
","Hidden and output activations are ReLU.
","Target y = [1].
","Learning rate η = 0.05.
","Use half squared loss and perform forward and backward passes.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","Hidden neuron 2:
","Output neuron 1:
","Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [-0.809]. Squared errors = [0.654481]. Half-sum squared loss = 0.3272405; mean squared error = 0.654481. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","The ReLU derivative is 1 here because this preactivation is positive.
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.0343825. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","h2 → o1:
","Subtract η × gradient = −0.019416. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = −0.011326. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h1:
","Subtract η × gradient = −0.016989. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x1 → h2:
","Subtract η × gradient = −0.012135. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h2:
","Subtract η × gradient = −0.0182025. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
","Correction to the handwritten working: the hidden deltas are negative (−0.11326 and −0.12135). A positive value at that step is a sign error. Keep output error as prediction minus target throughout.
"],"question":[],"answer":[],"kind":"read","id":"tee-c10-s10","search":"Teacher vectorized ReLU example \n**Question.** Input x = [2, 3]. Each row below lists the incoming weights of one receiving neuron. W₁ = ; b₁ = [0, 0]; W₂ = ; b₂ = [0]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.05. Use half squared loss and perform forward and backward passes.\n\n\n\n### Given\n\nInput x = [2, 3].\n\nW₁ = .\n\nb₁ = [0, 0].\n\nW₂ = .\n\nb₂ = [0].\n\nHidden and output activations are ReLU.\n\nTarget y = [1].\n\nLearning rate η = 0.05.\n\nUse half squared loss and perform forward and backward passes.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.\nPrediction minus target = [-0.809]. Squared errors = [0.654481]. Half-sum squared loss = 0.3272405; mean squared error = 0.654481. These are different unless the number of output components makes the scale coincide.\n**Output delta 1:**\n\nThe ReLU derivative is 1 here because this preactivation is positive.\n**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.\nh1 → o1: gradient = (-0.809) × 0.85 = -0.68765. New weight = 0.14−0.05 × (-0.68765) = **0.1743825**.\nh2 → o1: gradient = (-0.809) × 0.48 = -0.38832. New weight = 0.15−0.05 × (-0.38832) = **0.169416**.\n**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.\n*"},{"title":"Teams photos full sigmoid example","blocks":["The three photographs show this 2 → 2 → 2 network and its chain rule. The photo label for w₃ is read as 0.25, consistent with the visible output/error values and the matching worked example in S05. Learning rate is not legible/specified in these photos; η = 0.5 below is an explicit practice assumption.
","Question. Input x = [0.05, 0.1]. Each row below lists the incoming weights of one receiving neuron. W₁ = ; b₁ = [0.35, 0.35]; W₂ = ; b₂ = [0.6, 0.6]. Hidden and output activations are sigmoid. Target y = [0.01, 0.99]. Learning rate η = 0.5. Compute all outputs, total half squared loss, all eight weight gradients and one simultaneous weight update. Map weights in order: w₁ = x₁ → h₁, w₂ = x₂ → h₁, w₃ = x₁ → h₂, w₄ = x₂ → h₂; w₅ = h₁ → o₁, w₆ = h₂ → o₁, w₇ = h₁ → o₂, w₈ = h₂ → o₂.
Input x = [0.05, 0.1].
","W₁ = .
b₁ = [0.35, 0.35].
","W₂ = .
b₂ = [0.6, 0.6].
","Hidden and output activations are sigmoid.
","Target y = [0.01, 0.99].
","Learning rate η = 0.5.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","Hidden neuron 2:
","Output neuron 1:
","Output neuron 2:
","Step 2 loss convention. The photos explicitly use ½(t₁−o₁)² + ½(t₂−o₂)². With exactly two output components, this equals their mean squared error, 0.298371109. The unhalved sum is 0.596742218.
","Prediction minus target = [0.74136507, -0.217071535]. Squared errors = [0.549622166, 0.047120051]. Half-sum squared loss = 0.298371109; mean squared error = 0.298371109. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","The derivative is o(1−o).
","Output delta 2:
","The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = 0.04108352. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","h2 → o1:
","Subtract η × gradient = 0.041333814. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","h1 → o2:
","Subtract η × gradient = −0.01130127. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","h2 → o2:
","Subtract η × gradient = −0.011370121. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Contribution from output 2.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Contribution from output 2.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = 0.000219284. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h1:
","Subtract η × gradient = 0.000438568. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x1 → h2:
","Subtract η × gradient = 0.000248856. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h2:
","Subtract η × gradient = 0.000497712. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
","Why the photos add two terms for h₁: dL/dh₁ = δo₁w₅ + δo₂w₇ = 0.138498562 × 0.4 + (−0.038098237) × 0.5 = 0.055399425−0.019049118 = 0.036350306. Now multiply by h₁(1−h₁) = 0.241300709 to obtain δh₁ = 0.008771355; multiply by x₁ = 0.05 to obtain dL/dw₁ = 0.000438568. Adding the two output paths is essential.
","Memory sequence: error → output slope → outgoing weight → add paths → hidden slope → input. If you can explain why each factor appears, changed numbers or extra output neurons are much less confusing.
","Bias convention: the photo draws one bias value per layer. If these are literally tied scalars rather than equal initial per-neuron biases, sum the relevant deltas: hidden shared b becomes 0.35−0.5(0.008771355+0.009954255) = 0.340637195; output shared b becomes 0.6−0.5(0.138498562−0.038098237) = 0.549799837. If only weights are requested, do not invent a bias update requirement.
"],"question":[],"answer":[],"kind":"read","id":"tee-c10-s11","search":"Teams photos full sigmoid example The three photographs show this 2 → 2 → 2 network and its chain rule. The photo label for w₃ is read as 0.25, consistent with the visible output/error values and the matching worked example in S05. Learning rate is not legible/specified in these photos; η = 0.5 below is an explicit practice assumption.\n**Question.** Input x = [0.05, 0.1]. Each row below lists the incoming weights of one receiving neuron. W₁ = ; b₁ = [0.35, 0.35]; W₂ = ; b₂ = [0.6, 0.6]. Hidden and output activations are sigmoid. Target y = [0.01, 0.99]. Learning rate η = 0.5. Compute all outputs, total half squared loss, all eight weight gradients and one simultaneous weight update. Map weights in order: w₁ = x₁ → h₁, w₂ = x₂ → h₁, w₃ = x₁ → h₂, w₄ = x₂ → h₂; w₅ = h₁ → o₁, w₆ = h₂ → o₁, w₇ = h₁ → o₂, w₈ = h₂ → o₂.\n\n\n\n### Given\n\nInput x = [0.05, 0.1].\n\nW₁ = .\n\nb₁ = [0.35, 0.35].\n\nW₂ = .\n\nb₂ = [0.6, 0.6].\n\nHidden and output activations are sigmoid.\n\nTarget y = [0.01, 0.99].\n\nLearning rate η = 0.5.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Output neuron 2: **\n\n\n**Step 2 loss convention.** The photos explicitly use "}]},{"n":11,"title":"Final examination December 2025 fully solved","sections":[{"title":"Start here","blocks":["F25 = S12, 3 December 2025, with S13 as a comparison answer key. S18 is another copy of this same paper. All questions are included; its historical choice rule does not remove questions from this practice bank.
","Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible.
"],"kind":"read","id":"tee-c11-intro"},{"title":"F25 Q1a","question":["Question (5 marks). Give four ML versus DL differences with examples. Choose and justify ML or DL for (1) 2,000 short product reviews, (2) labeled financial fraud records, (3) captions for 100,000 labeled natural-scene images.
"],"answer":["Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.
","| Comparison | Traditional ML approaches | Deep learning approaches |
|---|---|---|
| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |
| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |
| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |
| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |
These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.
","Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.
","Choices: (1) Start with traditional ML using text features because the labeled dataset is small; pretrained text models are a viable extension. (2) Start with a tabular ML classifier such as a tree ensemble; labeled structured features do not automatically require DL. (3) DL is appropriate for learning visual features and generating a word sequence; a CNN encoder with a sequence decoder is a syllabus-aligned design. Compare held-out metrics rather than claiming one method must always win.
"],"kind":"practice","id":"tee-c11-s0","search":"F25 Q1a **Question (5 marks).** Give four ML versus DL differences with examples. Choose and justify ML or DL for (1) 2,000 short product reviews, (2) labeled financial fraud records, (3) captions for 100,000 labeled natural-scene images.\n\n**Complete worked answer.**\nDeep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.\n| Comparison | Traditional ML approaches | Deep learning approaches |\n| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |\n| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |\n| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |\n| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |\nThese are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.\nDrivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.\n**Choices: ** (1) Start with traditional ML using text features because the labeled dataset is small; pretrained text models are a viable extension. (2) Start with a tabular ML classifier such as a tree ensemble; labeled structured features do not automatically require DL. (3) DL is appropriate for learning visual features and generating a word sequence; a CNN encoder with a sequence decoder is a syllabus-aligned design. Compare held-out metrics rather than claiming one method must always win.\n"},{"title":"F25 Q1b","question":["Question (5 marks). For automated leaf-disease image classification, discuss factors affecting loss selection. For classes Normal, Mild, Severe with target [0,1,0] and predicted probabilities [0.1,0.8,0.1], compute and interpret categorical cross-entropy.
"],"answer":["Classes = Normal,Mild,Severe; target = [0,1,0].
","Predicted probabilities = [0.1,0.8,0.1]; use natural logarithms.
","Calculate categorical cross-entropy and interpret the result.
","Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.
","yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.
","Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.
","Select a loss according to the task and output meaning: real-valued regression often uses MSE/MAE, binary classification uses binary cross-entropy, and one-of-K classification uses categorical cross-entropy. Consider class imbalance, noisy labels/outliers, whether outputs are calibrated probabilities, and differentiability for optimization.
","Step 1: expand the categorical cross-entropy sum.
","Step 2: keep the term for the true class.
","Step 3: calculate its natural logarithm.
","Step 4: negate the logarithm.
","![\"L\\approx0.223143551\"](\"assets/equations/178c3708a55e1ab7.svg\")
Final answer: loss ≈ 0.223144 nats. The true class received probability 0.8, so the loss is relatively small; probability 1 would give loss zero.
"],"kind":"practice","id":"tee-c11-s1","search":"F25 Q1b **Question (5 marks).** For automated leaf-disease image classification, discuss factors affecting loss selection. For classes Normal, Mild, Severe with target [0,1,0] and predicted probabilities [0.1,0.8,0.1], compute and interpret categorical cross-entropy.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nClasses = Normal,Mild,Severe; target = [0,1,0].\n\nPredicted probabilities = [0.1,0.8,0.1]; use natural logarithms.\n\n### To find\n\nCalculate categorical cross-entropy and interpret the result.\n\n### Formula — which one and why\n\n**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.\n\nyⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.\n\n\n\n\n\n**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.\n\n### Solution — step by step\n\n\n\n\n\n\nSelect a loss according to the task and output meaning: real-valued regression often uses MSE/MAE, binary classification uses binary cross-entropy, and one-of-K classification uses categorical cross-entropy. Consider class imbalance, noisy labels/outliers, whether outputs are calibrated probabilities, and differentiability for optimization.\n**Step 1: expand the categorical cross-entropy sum.**\n\n\n\n\n\n\n\n**Step 2: keep the term for the true class.**\n\n\n\n\n\n\n\n**Step 3: calculate its natural logarithm.**\n\n\n\n\n\n\n\n**Step 4: negate the logarithm.**\n\n\n\n\n\n\n\n**Final answer:** loss ≈ 0.223144 nats. The true class received probability 0.8, so the loss is relatively small; probability 1 would give loss zero.\n"},{"title":"F25 Q1c","question":["Question (5 marks). Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization.
"],"answer":["Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
"],"kind":"practice","id":"tee-c11-s2","search":"F25 Q1c **Question (5 marks).** Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization.\n\n**Complete worked answer.**\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\n"},{"title":"F25 Q1d","question":["Question (5 marks). Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
"],"kind":"practice","id":"tee-c11-s3","search":"F25 Q1d **Question (5 marks).** Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate.\n\n**Complete worked answer.**\nA recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.\n\n\nAt each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.\nFor next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.\nA basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.\n"},{"title":"F25 Q2a","question":["Question (10 marks). A 2 → 2 → 1 sigmoid network has all four input-to-hidden weights 0.5, both hidden-to-output weights 0.25, hidden and output biases 0, and target 1. Perform forward and backward passes to update the x₁ → h₁ weight, h₁ → output weight and output bias at η = 0.5. The paper does not state the numerical inputs or loss.
","Architecture 2 → 2 → 1; every neuron uses sigmoid.
","All four input-to-hidden weights = 0.5; both hidden-to-output weights = 0.25.
","All biases = 0; target = 1; η = 0.5.
","The paper omits inputs and loss: this worked solution explicitly assumes x = [1,1] and half squared loss.
","Calculate the forward pass, then gradients and updates for x₁→h₁, h₁→output and the output bias.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Assumptions: use x = [1,1] and half squared loss. The paper permits suitable assumptions and its key also introduces assumed inputs. There is no unique numerical result without them.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/ebc8dda9737caeb3.svg\")
![\"h_{1}=\\frac{1}{1+e^{-(1)}}=0.731058579\"](\"assets/equations/9d9a8c14314f06ad.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/9d50f49ceec1f464.svg\")
![\"h_{2}=\\frac{1}{1+e^{-(1)}}=0.731058579\"](\"assets/equations/5f1426f29e3d87e6.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/c44792db3abbdcd0.svg\")
![\"o_{1}=\\frac{1}{1+e^{-(0.365529289)}}=0.590378257\"](\"assets/equations/07b12d14b1f2a0ff.svg\")
Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [-0.409621743]. Squared errors = [0.167789972]. Half-sum squared loss = 0.083894986; mean squared error = 0.167789972. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/b7b0310d2c3be2ab.svg\")
The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.036209168. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=(-0.099059551)\\times(0.731058579)\\approx](\"assets/equations/30f7bf8aa04d2bf2.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,1,\\mathrm{new}}=0.25-0.5(-0.072418335)\\approx](\"assets/equations/82203dea1f21db8c.svg\")
h2 → o1:
","Subtract η × gradient = −0.036209168. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,2}^{(2)}=(-0.099059551)\\times(0.731058579)\\approx](\"assets/equations/fd02f5bab6bb789a.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,2,\\mathrm{new}}=0.25-0.5(-0.072418335)\\approx](\"assets/equations/66e7552de1ba0f15.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.25)\\times(-0.099059551)\\approx](\"assets/equations/486b61a3d9f85a8a.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/d708f3df447c37c4.svg\")
![\"\\delta_{h_1}](\"assets/equations/faa954804a7ac56f.svg\")
Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\delta_{h_2}](\"assets/equations/c9f11eb5cc7cb0d3.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = −0.002434536. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=(-0.004869072)\\times(1)\\approx](\"assets/equations/9759c3655229ab0e.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,1,\\mathrm{new}}=0.5-0.5(-0.004869072)\\approx](\"assets/equations/c3d90c909dab8209.svg\")
x2 → h1:
","Subtract η × gradient = −0.002434536. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,2}^{(1)}=(-0.004869072)\\times(1)\\approx](\"assets/equations/3a5cd9a960e8e0d8.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,2,\\mathrm{new}}=0.5-0.5(-0.004869072)\\approx](\"assets/equations/057d087cc10bb2a8.svg\")
x1 → h2:
","Subtract η × gradient = −0.002434536. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,1}^{(1)}=(-0.004869072)\\times(1)\\approx](\"assets/equations/cd47a26ddf1d4480.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,1,\\mathrm{new}}=0.5-0.5(-0.004869072)\\approx](\"assets/equations/5614079c42cc22b4.svg\")
x2 → h2:
","Subtract η × gradient = −0.002434536. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,2}^{(1)}=(-0.004869072)\\times(1)\\approx](\"assets/equations/d63a8ff35a6906b2.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,2,\\mathrm{new}}=0.5-0.5(-0.004869072)\\approx](\"assets/equations/bfabbdcd726cbd9c.svg\")
Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","![\"b_{h_1,\\mathrm{new}}=0-0.5(-0.004869072)=0.002434536\"](\"assets/equations/c94badc259c5a62b.svg\")
![\"b_{h_2,\\mathrm{new}}=0-0.5(-0.004869072)=0.002434536\"](\"assets/equations/a2b6bc6b29544f77.svg\")
![\"b_{o_1,\\mathrm{new}}=0-0.5(-0.099059551)=0.049529775\"](\"assets/equations/55595ef27eaeb48f.svg\")
These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
","Requested values: input weight 0.502434536, output weight 0.286209167, output bias 0.049529775. Other supplied inputs produce different correct conditional results.
"],"kind":"practice","id":"tee-c11-s4","search":"F25 Q2a **Question (10 marks).** A 2 → 2 → 1 sigmoid network has all four input-to-hidden weights 0.5, both hidden-to-output weights 0.25, hidden and output biases 0, and target 1. Perform forward and backward passes to update the x₁ → h₁ weight, h₁ → output weight and output bias at η = 0.5. The paper does not state the numerical inputs or loss.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nArchitecture 2 → 2 → 1; every neuron uses sigmoid.\n\nAll four input-to-hidden weights = 0.5; both hidden-to-output weights = 0.25.\n\nAll biases = 0; target = 1; η = 0.5.\n\nThe paper omits inputs and loss: this worked solution explicitly assumes x = [1,1] and half squared loss.\n\n### To find\n\nCalculate the forward pass, then gradients and updates for x₁→h₁, h₁→output and the output bias.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Assumptions: ** use x = [1,1] and half squared loss. The paper permits suitable assumptions and its key also introduces assumed inputs. There is no unique numerical result without them.\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.\nPrediction minus target = [-0.409621743]. Squared errors = [0.167789972]. Half-sum squared loss = 0.083894986; mean squared error = 0.167789972. These are different unless the number of output components makes the scale"},{"title":"F25 Q2b","question":["Question (10 marks). Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction.
"],"answer":["A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.
","The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.
","Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].
","An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
","AlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs.
"],"kind":"practice","id":"tee-c11-s5","search":"F25 Q2b **Question (10 marks).** Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction.\n\n**Complete worked answer.**\nA residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.\n\n\nThe identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.\nElementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.\nAlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs.\n"},{"title":"F25 Q3a","question":["Question (10 marks). Train one epoch of a single perceptron with η = 1, initial weights and bias 0, threshold 0, on [−1,1,−1] → 0, [−1,1,1] → 1, [1,−1,1] → 1, [−1,1,1] → 1. Then map biological-neuron components to artificial-neuron components.
"],"answer":["Initial weights and bias = 0; η = 1; threshold = 0.
","Samples in order: [−1,1,−1]→0; [−1,1,1]→1; [1,−1,1]→1; [−1,1,1]→1.
","Use output 1 when net > 0 and 0 otherwise, matching the supplied key; one epoch.
","Process each sample in order, carrying updated weights forward, and report final weights/bias.
","Perceptron learning rule. Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters.
","t = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question.
","![\"w_i^{\\mathrm{new}}=w_i+\\eta(t-o)x_i\"](\"assets/equations/1fccc40e029a1d4e.svg\")
![\"b^{\\mathrm{new}}=b+\\eta(t-o)\"](\"assets/equations/58dd8fcb5fcd562f.svg\")
Yaad rakhne ka cue: Prediction nikalo; target−prediction se correction nikalo.
","State the threshold rule. Use binary output 1 if z>0 and 0 otherwise, matching the associated key. Update Δw = η(t−o)x and Δb = η(t−o), with η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly.
","Step 1: copy this sample and target.
","x = [-1, 1, -1], target = 0.
","Step 2: calculate the current net.
","z = (0 × -1) + (0 × 1) + (0 × -1) + (0) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = 0
","Step 5: calculate all weight changes.
","Δw = 0 × [-1, 1, -1] = [0, 0, 0]
","Step 6: calculate the bias change.
","Δb = 0.
","Step 7: carry the updated parameters forward.
","New w = [0, 0, 0], new b = 0.
","Step 1: copy this sample and target.
","x = [-1, 1, 1], target = 1.
","Step 2: calculate the current net.
","z = (0 × -1) + (0 × 1) + (0 × 1) + (0) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = 1
","Step 5: calculate all weight changes.
","Δw = 1 × [-1, 1, 1] = [-1, 1, 1]
","Step 6: calculate the bias change.
","Δb = 1.
","Step 7: carry the updated parameters forward.
","New w = [-1, 1, 1], new b = 1.
","Step 1: copy this sample and target.
","x = [1, -1, 1], target = 1.
","Step 2: calculate the current net.
","z = (-1 × 1) + (1 × -1) + (1 × 1) + (1) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = 1
","Step 5: calculate all weight changes.
","Δw = 1 × [1, -1, 1] = [1, -1, 1]
","Step 6: calculate the bias change.
","Δb = 1.
","Step 7: carry the updated parameters forward.
","New w = [0, 0, 2], new b = 2.
","Step 1: copy this sample and target.
","x = [-1, 1, 1], target = 1.
","Step 2: calculate the current net.
","z = (0 × -1) + (0 × 1) + (2 × 1) + (2) = 4
","Step 3: apply the threshold.
","output = 1.
","Step 4: calculate the update multiplier.
","Update multiplier = 0
","Step 5: calculate all weight changes.
","Δw = 0 × [-1, 1, 1] = [0, 0, 0]
","Step 6: calculate the bias change.
","Δb = 0.
","Step 7: carry the updated parameters forward.
","New w = [0, 0, 2], new b = 2.
","Final after exactly the requested epochs: w = [0, 0, 2], b = 2. A finite number of requested epochs does not itself prove convergence or correct classification of every sample.
","Biological dendrites receive signals, synapses influence connection strength, the cell body integrates information, and an axon carries an output signal. In the artificial analogy, inputs represent received signals, weights represent connection strengths, the weighted sum plus bias performs aggregation, and an activation function produces an output. The bias shifts the firing threshold. This is a simplified computational analogy, not a complete biological model.
","Question (10 marks). Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning.
"],"answer":["Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.
","1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.
","2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.
","3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.
","4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.
","5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.
","6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.
","Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.
"],"kind":"practice","id":"tee-c11-s7","search":"F25 Q3b **Question (10 marks).** Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning.\n\n**Complete worked answer.**\nTransfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.\n1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.\n2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.\n3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.\n4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.\n5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.\n6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.\nFeature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.\n"},{"title":"F25 Q4a","question":["Question (10 marks). For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters.
"],"answer":["N = 1,000; mini-batch size = 100.
","J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9.
","Use signed velocity.
","Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations.
","Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.
","N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.
","Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","Step 1: count full-batch updates.
","All 1,000 samples contribute to one update: 1 per epoch.
","Step 2: count stochastic updates.
","One update per sample: 1,000 per epoch.
","Step 3: count mini-batches.
","Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ.
","Step 1: evaluate the gradient at the current parameter.
","Step 2: retain the previous displacement.
","Step 3: scale the current gradient.
","Step 4: combine to obtain the new signed velocity.
","Step 5: add new velocity to the actual old parameter.
","Step 1: evaluate the gradient at the current parameter.
","Step 2: retain the previous displacement.
","Step 3: scale the current gradient.
","Step 4: combine to obtain the new signed velocity.
","Step 5: add new velocity to the actual old parameter.
","Hyperparameters: the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update.
"],"kind":"practice","id":"tee-c11-s8","search":"F25 Q4a **Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nN = 1,000; mini-batch size = 100.\n\nJ(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9.\n\nUse signed velocity.\n\n### To find\n\nCount updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations.\n\n### Formula — which one and why\n\n**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.\n\nN = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n\n**Step 1: count full-batch updates.**\n\nAll 1,000 samples contribute to one update: 1 per epoch.\n\n**Step 2: count stochastic updates.**\n\nOne update per sample: 1,000 per epoch.\n\n**Step 3: count mini-batches.**\n\n\n\n\n\n\n\n\nUse signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ.\n### Momentum iteration 1\n\n**Step 1: evaluate the gradient at the current parameter.**\n\n\n\n\n\n\n\n**Step 2: retain the previous displacement.**\n\n\n\n\n\n\n\n**Step 3: scale the current gradient.**\n\n\n\n\n\n\n\n**Step 4: combine to obtain the new signed velocity.**\n\n\n\n\n\n\n\n**Step 5: add new velocity to the actual old parameter.**\n\n\n\n\n\n\n\n\n### Momentum iteration 2\n\n**Step 1: evaluate the gradient at the current parameter.**\n\n\n\n\n\n\n\n**Step 2: retain the previous displacement.**\n\n\n\n\n\n\n\n**Step 3: scale the current gradient.**\n\n\n\n\n\n\n\n**Step 4: combine to obtain the new signed velocity.**\n\n\n\n\n\n\n\n**Step 5: add new velocity to the actual old parameter.**\n\n\n\n\n\n\n\n\n**Hyperparameters: ** the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update.\n"},{"title":"F25 Q4b","question":["Question (10 marks). Define vanishing gradients in RNNs and explain why they occur. Use “The dogs barked at the thief and … were appreciated” to discuss long context. Explain LSTM forgetting/retention, then compute the retained contribution for f = [0.1,0.9], Cprevious = [2,1].
"],"answer":["Forget-gate outputs f = [0.1,0.9]; old cell = [2,1].
","No input gate or candidate is supplied.
","Explain forgetting/retention and calculate the retained old-cell part only.
","LSTM retained old memory. Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell.
","f = activated forget gate; C_old = previous cell state; ⊙ means multiply matching entries.
","Yaad rakhne ka cue: Forget gate × old cell = bacha hua purana hissa.
","Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.
","1. Initialize the hidden state and run the forward recurrence from the first input to the last.
","2. Compute the specified output loss, at the final step or across several steps.
","3. Start from the last loss and propagate derivatives backwards through output and hidden operations.
","4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.
","5. Sum all uses of each shared weight before applying the optimizer update.
","![\"\\delta_t=(W_{hy}^{T}\\delta_t^{out}+W_{hh}^{T}\\delta_{t+1})\\odot(1-h_t^2),\\quad](\"assets/equations/fdf1ee48fdcfd79f.svg\")
The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.
","The plural subject “dogs” must remain represented across intervening words. A vanilla RNN may fail to learn that dependency when early gradients are weak; it is not guaranteed to fail on every such sentence. In an LSTM, forget components close to 1 retain relevant subject information along the cell path.
","Step 1: retain the first old-cell component.
","Step 2: retain the second old-cell component.
","Step 3: assemble the retained old-cell contribution.
","![\"f\\odot](\"assets/equations/813a6e1d025fa14c.svg\")
Final answer: [0.2,0.9] is only retained old content. The input gate and candidate are missing, so the complete updated cell cannot be calculated.
"],"kind":"practice","id":"tee-c11-s9","search":"F25 Q4b **Question (10 marks).** Define vanishing gradients in RNNs and explain why they occur. Use “The dogs barked at the thief and … were appreciated” to discuss long context. Explain LSTM forgetting/retention, then compute the retained contribution for f = [0.1,0.9], Cprevious = [2,1].\n\n**Complete worked answer.**\n\n\n\n### Given\n\nForget-gate outputs f = [0.1,0.9]; old cell = [2,1].\n\nNo input gate or candidate is supplied.\n\n### To find\n\nExplain forgetting/retention and calculate the retained old-cell part only.\n\n### Formula — which one and why\n\n**LSTM retained old memory.** Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell.\n\nf = activated forget gate; C_old = previous cell state; ⊙ means multiply matching entries.\n\n\n\n**Yaad rakhne ka cue:** Forget gate × old cell = bacha hua purana hissa.\n\n### Solution — step by step\n\n\n\n\nBackpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.\n1. Initialize the hidden state and run the forward recurrence from the first input to the last.\n2. Compute the specified output loss, at the final step or across several steps.\n3. Start from the last loss and propagate derivatives backwards through output and hidden operations.\n4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.\n5. Sum all uses of each shared weight before applying the optimizer update.\n\nThe tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.\n\nThe plural subject “dogs” must remain represented across intervening words. A vanilla RNN may fail to learn that dependency when early gradients are weak; it is not guaranteed to fail on every such sentence. In an LSTM, forget components close to 1 retain relevant subject information along the cell path.\n**Step 1: retain the first old-cell component.**\n\n\n\n\n\n\n\n**Step 2: retain the second old-cell component.**\n\n\n\n\n\n\n\n**Step 3: assemble the retained old-cell contribution.**\n\n\n\n\n\n\n\n**Final answer:** [0.2,0.9] is only retained old content. The input gate and candidate are missing, so the complete updated cell cannot be calculated.\n"},{"title":"F25 Q5a","question":["Question (10 marks). State the purpose of each LSTM gate. For “Artificial Intelligence has revolutionized every domain. The work force will have to update themselves to remain relevant,” explain a context change. Calculate C and h given i = 0.7, f = 0.4, o = 0.8, candidate = 0.9, Cprevious = 0.5. Why is LSTM more effective than a simple RNN for language modeling?
"],"answer":["Activated gate values: i = 0.7,f = 0.4,o = 0.8.
","Candidate = 0.9; old cell = 0.5.
","Explain the context change in the question; calculate the new cell and hidden state.
","LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.
","f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.
","Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.
","An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.
","![\"\\widetilde](\"assets/equations/31dbb5df72a8ba34.svg\")
The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.
","In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.
","The second sentence shifts attention toward the workforce while retaining useful prior AI context. Input gates can admit the new subject; forget gates can reduce obsolete details.
","Step 1: retain old memory.
","![\"fC_{old}=0.4(0.5)=0.2\"](\"assets/equations/8f3bf24a2e1d90a0.svg\")
Step 2: admit candidate content.
","![\"i\\widetilde](\"assets/equations/7a48bf51c7d9b9c7.svg\")
Step 3: add the two contributions.
","![\"C=0.2+0.63=0.83\"](\"assets/equations/b89883a392a4e3ec.svg\")
Step 4: apply tanh to the updated cell.
","![\"\\tanh(0.83)\\approx0.680476006\"](\"assets/equations/6aee7e26a8e96885.svg\")
Step 5: multiply by the output gate.
","![\"h=0.8(0.680476006)\\approx0.544380805\"](\"assets/equations/c85eca49bc37b1e7.svg\")
Final answer: C = 0.83 and h ≈ 0.544380805. The supplied gates are already activated; do not apply sigmoid again.
"],"kind":"practice","id":"tee-c11-s10","search":"F25 Q5a **Question (10 marks).** State the purpose of each LSTM gate. For “Artificial Intelligence has revolutionized every domain. The work force will have to update themselves to remain relevant,” explain a context change. Calculate C and h given i = 0.7, f = 0.4, o = 0.8, candidate = 0.9, Cprevious = 0.5. Why is LSTM more effective than a simple RNN for language modeling?\n\n**Complete worked answer.**\n\n\n\n### Given\n\nActivated gate values: i = 0.7,f = 0.4,o = 0.8.\n\nCandidate = 0.9; old cell = 0.5.\n\n### To find\n\nExplain the context change in the question; calculate the new cell and hidden state.\n\n### Formula — which one and why\n\n**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.\n\nf,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.\n\n\n\n\n\n**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.\n\n### Solution — step by step\n\n\n\n\nAn LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.\n\n\n\n\nThe forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.\nIn language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.\nThe second sentence shifts attention toward the workforce while retaining useful prior AI context. Input gates can admit the new subject; forget gates can reduce obsolete details.\n**Step 1: retain old memory.**\n\n\n\n\n\n\n\n**Step 2: admit candidate content.**\n\n\n\n\n\n\n\n**Step 3: add the two contributions.**\n\n\n\n\n\n\n\n**Step 4: apply tanh to the updated cell.**\n\n\n\n\n\n\n\n**Step 5: multiply by the output gate.**\n\n\n\n\n\n\n\n**Final answer:** C = 0.83 and h ≈ 0.544380805. The supplied gates are already activated; do not apply sigmoid again.\n"},{"title":"F25 Q5b","question":["Question (10 marks). Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD.
"],"answer":["AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:
","Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.
","Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.
","Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.
"],"kind":"practice","id":"tee-c11-s11","search":"F25 Q5b **Question (10 marks).** Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD.\n\n**Complete worked answer.**\nAdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:\n\nFrequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.\nExample with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.\nAdvantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.\n"},{"title":"F25 Q6a","question":["Question (10 marks). Calculate every activation shape and trainable-parameter count for: Input 128 × 128 × 3. → Conv K = 3, filters = 16, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 3, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 240 units. → Dense 120 units. → Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2.
","| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Stride 2; P = 0 (no padding).
Input 128 × 128 × 3.
","Conv K = 3, filters = 16, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 3, filters = 32, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Dense 240 units.
","Dense 120 units.
","Dense 5 units.
","Also convolve the input A with the kernel K shown below, valid padding and stride 2.
","Find every layer shape, its parameter count and the network total. Find every convolution output cell.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 128 × 128 × 3. Use 16 filters of spatial size 3 × 3, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 126 × 126 × 16.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 126 × 126 × 16. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 63 × 63 × 16.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 63 × 63 × 16. Use 32 filters of spatial size 3 × 3, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 61 × 61 × 32.
","3. Count trainable values. Each filter receives all 16 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 61 × 61 × 32. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 30 × 30 × 32.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The division gives 29.5; take its floor before adding 1. A partial window outside the unpadded input is not included.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 30 × 30 × 32. Rearrange all activation values into one vector.
","Output: a vector with 28,800 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 28,800 features. Output: 240 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 240 features. Output: 120 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 120 features. Output: 5 neurons. Every output neuron connects to every input feature and has one bias.
","| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | | 126 x 126 x 16 | 432 | 16 | 448 |
| 2 pool | | 63 x 63 x 16 | 0 | 0 | 0 |
| 3 conv | | 61 x 61 x 32 | 4,608 | 32 | 4,640 |
| 4 pool | | 30 x 30 x 32 | 0 | 0 | 0 |
| 5 flat | 30 × 30 × 32 | 28800 | 0 | 0 | 0 |
| 6 dense | 28800 → 240 | 240 | 6,912,000 | 240 | 6,912,240 |
| 7 dense | 240 → 120 | 120 | 28,800 | 120 | 28,920 |
| 8 dense | 120 → 5 | 5 | 600 | 5 | 605 |
Parameter substitutions, in layer order:
","Layer 1: (3 x 3 x 3 + 1) x 16 = 448.
","Layer 3: (3 x 3 x 16 + 1) x 32 = 4,640.
","Layer 6: (28800 + 1) x 240 = 6,912,240.
","Layer 7: (240 + 1) x 120 = 28,920.
","Layer 8: (120 + 1) x 5 = 605.
","Total: 448 + 4,640 + 6,912,240 + 28,920 + 605 = 6,946,853 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","Separate convolution calculation:
","Step 1 shape: = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 1–3. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 3 | 0 | −2 |
| 4 | 0 | −4 |
| 3 | 0 | −1 |
Each cell is input × matching kernel coefficient.
1 + 0 + 2 = 3
| 3 | 5 |
| 3 | −7 |
Output row 1, column 1.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 3–5. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 2 | 0 | −1 |
| 4 | 0 | 0 |
| 1 | 0 | −1 |
Each cell is input × matching kernel coefficient.
1 + 4 + 0 = 5
| 3 | 5 |
| 3 | −7 |
Output row 1, column 2.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 1–3. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 3 | 0 | −1 |
| 0 | 0 | 0 |
| 1 | 0 | 0 |
Each cell is input × matching kernel coefficient.
2 + 0 + 1 = 3
| 3 | 5 |
| 3 | −7 |
Output row 2, column 1.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 3–5. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 | −1 |
| 0 | 0 | −4 |
| 0 | 0 | −3 |
Each cell is input × matching kernel coefficient.
0 + (−4) + (−3) = −7
| 3 | 5 |
| 3 | −7 |
Output row 2, column 2.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Final output: . Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
Question (10 marks). Explain activations and the role of nonlinearity. Compare sigmoid, tanh, ReLU and Leaky ReLU including advantages and disadvantages.
"],"answer":["An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.
","Sigmoid
","Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.
","Hyperbolic tangent
","The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.
","ReLU
","The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.
","Leaky ReLU
","Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.
","Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.
"],"kind":"practice","id":"tee-c11-s13","search":"F25 Q6b **Question (10 marks).** Explain activations and the role of nonlinearity. Compare sigmoid, tanh, ReLU and Leaky ReLU including advantages and disadvantages.\n\n**Complete worked answer.**\nAn activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.\n**Sigmoid**\n\nHere z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.\n**Hyperbolic tangent**\n\nThe numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.\n**ReLU**\n\nThe output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.\n**Leaky ReLU**\n\nAlpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.\nSigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.\n"},{"title":"F25 Q7a","question":["Question (10 marks). Explain feedforward information flow, weights, biases and activations, then calculate the output. Input x = [1, 0]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU.
","Input x = [1, 0].
","W₁ is printed with the question.
","b₁ = [0.1, 0.2].
","W₂ is printed with the question.
","b₂ = [0.1].
","Hidden and output activations are ReLU.
","Find all preactivations, hidden activations and the final output. Explain the role of weights, biases and activations in this forward pass.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","A feedforward network transforms inputs layer by layer with no recurrent feedback. Weights scale contributions; bias shifts the preactivation; activation supplies the nonlinear transformation.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/cd4f127dab262e74.svg\")
![\"h_{1}](\"assets/equations/2b99f7dd17297569.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/6bf10ef9c5abf8e9.svg\")
![\"h_{2}](\"assets/equations/ac1a985150f1b35b.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/725a43999780ebd1.svg\")
![\"o_{1}](\"assets/equations/d22e4c171ba2abbb.svg\")
Matrix-orientation note: this solution uses column input and rows as receiving neurons, giving 0.55. The historical key uses the transpose interpretation for W₁ and obtains 0.61. If the problem does not define its orientation, state your convention; do not silently swap it during calculation. Under the alternative: hidden values = [0.3,0.6], output = 0.5 × 0.3+0.6 × 0.6+0.1 = 0.61.
"],"kind":"practice","id":"tee-c11-s14","search":"F25 Q7a **Question (10 marks).** Explain feedforward information flow, weights, biases and activations, then calculate the output. Input x = [1, 0]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput x = [1, 0].\n\nW₁ is printed with the question.\n\nb₁ = [0.1, 0.2].\n\nW₂ is printed with the question.\n\nb₂ = [0.1].\n\nHidden and output activations are ReLU.\n\n### To find\n\nFind all preactivations, hidden activations and the final output. Explain the role of weights, biases and activations in this forward pass.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\nA feedforward network transforms inputs layer by layer with no recurrent feedback. Weights scale contributions; bias shifts the preactivation; activation supplies the nonlinear transformation.\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Matrix-orientation note: ** this solution uses column input and rows as receiving neurons, giving **0.55**. The historical key uses the transpose interpretation for W₁ and obtains 0.61. If the problem does not define its orientation, state your convention; do not silently swap it during calculation. Under the alternative: hidden values = [0.3,0.6], output = 0.5 × 0.3+0.6 × 0.6+0.1 = 0.61.\n"},{"title":"F25 Q7b","question":["Question (10 marks). Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages.
"],"answer":["A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification.
","Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities.
","Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation.
","For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions.
"],"kind":"practice","id":"tee-c11-s15","search":"F25 Q7b **Question (10 marks).** Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages.\n\n**Complete worked answer.**\nA convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification.\n\nConvolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities.\nCompared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation.\nFor medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions.\n\n\n"}]},{"n":12,"title":"Re examination February 2026 fully solved","sections":[{"title":"Start here","blocks":["R26 = S09, 10 February 2026. S08 is the associated answer key; S10/S11 are duplicate-content copies in another program folder. One subpart asks a topic not explicitly listed in the current syllabus and is flagged below.
","Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible.
"],"kind":"read","id":"tee-c12-intro"},{"title":"R26 Q1a","question":["Question (5 marks). Describe two situations favoring DL over traditional ML. Choose ML or DL for house prices from 1,000 numerical records, real-time facial-emotion video recognition, and translation from millions of sentence pairs.
"],"answer":["Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.
","| Comparison | Traditional ML approaches | Deep learning approaches |
|---|---|---|
| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |
| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |
| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |
| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |
These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.
","Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.
","Choices: house prices: start with tabular ML because the features are structured and data are limited. Video emotion: DL can learn visual spatial features and temporal patterns. Large parallel-text translation: DL can learn representations and sequence dependencies. Two situations particularly suited to learned deep representations are complex image recognition and large-scale speech/language tasks; compare validation results instead of assuming an unconditional advantage.
"],"kind":"practice","id":"tee-c12-s0","search":"R26 Q1a **Question (5 marks).** Describe two situations favoring DL over traditional ML. Choose ML or DL for house prices from 1,000 numerical records, real-time facial-emotion video recognition, and translation from millions of sentence pairs.\n\n**Complete worked answer.**\nDeep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.\n| Comparison | Traditional ML approaches | Deep learning approaches |\n| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |\n| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |\n| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |\n| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |\nThese are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.\nDrivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.\n**Choices: ** house prices: start with tabular ML because the features are structured and data are limited. Video emotion: DL can learn visual spatial features and temporal patterns. Large parallel-text translation: DL can learn representations and sequence dependencies. Two situations particularly suited to learned deep representations are complex image recognition and large-scale speech/language tasks; compare validation results instead of assuming an unconditional advantage.\n"},{"title":"R26 Q1b","question":["Question (5 marks). Explain how activations affect gradients, convergence and performance. State preferred contexts for sigmoid, tanh, ReLU and Leaky ReLU.
"],"answer":["An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.
","Sigmoid
","Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.
","Hyperbolic tangent
","The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.
","ReLU
","The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.
","Leaky ReLU
","Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.
","Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.
"],"kind":"practice","id":"tee-c12-s1","search":"R26 Q1b **Question (5 marks).** Explain how activations affect gradients, convergence and performance. State preferred contexts for sigmoid, tanh, ReLU and Leaky ReLU.\n\n**Complete worked answer.**\nAn activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.\n**Sigmoid**\n\nHere z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.\n**Hyperbolic tangent**\n\nThe numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.\n**ReLU**\n\nThe output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.\n**Leaky ReLU**\n\nAlpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.\nSigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.\n"},{"title":"R26 Q1c","question":["Question (5 marks). Explain L1 and L2 regularization. Apply 50% dropout to outputs [0.8,−0.4,0.6,0.2] using an explicitly assumed mask.
"],"answer":["Activations = [0.8,−0.4,0.6,0.2]; dropout rate = 50%.
","The worked example assumes mask [1,0,1,0] and uses inverted dropout.
","No numerical weights or λ are supplied for the regularization explanation.
","Explain L1/L2 and calculate post-dropout activations under the stated mask.
","L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.
","λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.
","Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.
","L2 penalty with a factor of one-half. This answer uses λ/2 times the sum of squared weights. Differentiating a square supplies a factor of 2, which cancels the one-half, so the added gradient is λw. Use this same convention in the explanation below.
","λ = penalty strength; J_data = original data loss; wᵢ = weight i.
","Yaad rakhne ka cue: L2 mein half diya hai: square ka 2 cancel, gradient mein lambda × weight.
","Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.
","aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.
","Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.
","Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
","Step 1: convert dropout probability to keep probability.
","Step 2: state a possible mask.
","Choose [1,0,1,0]; 1 means keep and 0 means drop. The question allows an assumed mask; others give different realizations.
","Step 3: multiply each activation by its mask entry.
","Step 4: divide the kept values by q for inverted dropout.
","Step 5: assemble the training output.
","Final answer: simple masking gives [0.8,0,0.6,0]; inverted training dropout gives [1.6,0,1.2,0]. At inference inverted dropout is disabled.
"],"kind":"practice","id":"tee-c12-s2","search":"R26 Q1c **Question (5 marks).** Explain L1 and L2 regularization. Apply 50% dropout to outputs [0.8,−0.4,0.6,0.2] using an explicitly assumed mask.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nActivations = [0.8,−0.4,0.6,0.2]; dropout rate = 50%.\n\nThe worked example assumes mask [1,0,1,0] and uses inverted dropout.\n\nNo numerical weights or λ are supplied for the regularization explanation.\n\n### To find\n\nExplain L1/L2 and calculate post-dropout activations under the stated mask.\n\n### Formula — which one and why\n\n**L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.\n\nλ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.\n\n\n\n\n\n**Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai.\n\n**L2 penalty with a factor of one-half.** This answer uses λ/2 times the sum of squared weights. Differentiating a square supplies a factor of 2, which cancels the one-half, so the added gradient is λw. Use this same convention in the explanation below.\n\nλ = penalty strength; J_data = original data loss; wᵢ = weight i.\n\n\n\n\n\n**Yaad rakhne ka cue:** L2 mein half diya hai: square ka 2 cancel, gradient mein lambda × weight.\n\n**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.\n\naᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.\n\n\n\n\n\n**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.\n\n### Solution — step by step\n\n\n\n\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\n### Numerical with one explicitly chosen mask\n\n**Step 1: convert dropout probability to keep probability.**\n\n\n\n\n\n\n\n**Step 2: state a possible mask.**\n\nChoose [1,0,1,0]; 1 means keep and 0 means drop. The question allows an assumed mask; others give different realizations.\n\n**Step 3: multiply each activation by its mask entry.**\n\n\n\n\n\n\n\n**Step 4: divide the kept values by q for inverted dropout.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 5: assemble the training output.**\n\n\n\n\n\n\n\n**Final answer:** simpl"},{"title":"R26 Q1d","question":["Question (5 marks). Explain an RNN for positive, negative or neutral e-commerce review sentiment. Why are recurrence and word order relevant?
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","For this task select a many-to-one output: tokenize and embed the full review, update the recurrent state across its words, and classify the final sequence representation with a three-unit softmax. Train using one-hot sentiment labels and categorical cross-entropy. “Not worth buying” and “worth buying” illustrate why negation and order matter.
"],"kind":"practice","id":"tee-c12-s3","search":"R26 Q1d **Question (5 marks).** Explain an RNN for positive, negative or neutral e-commerce review sentiment. Why are recurrence and word order relevant?\n\n**Complete worked answer.**\nA recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.\n\n\nAt each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.\nFor next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.\nA basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.\nFor this task select a many-to-one output: tokenize and embed the full review, update the recurrent state across its words, and classify the final sequence representation with a three-unit softmax. Train using one-hot sentiment labels and categorical cross-entropy. “Not worth buying” and “worth buying” illustrate why negation and order matter.\n"},{"title":"R26 Q2a","question":["Question (10 marks). Use the sigmoid network Input x = [0.25, 0.7]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [1, 1]; W₂ is shown below; b₂ = [1, 1]. Hidden and output activations are sigmoid. Target y = [0.7, 0.3]. Learning rate η = 1. Here w₁ = x₁ → h₁, w₅ = h₁ → o₁ and w₇ = h₁ → o₂. Run forward and backward passes and update w₁, w₅, w₇. The paper gives no explicit loss formula.
","Input x = [0.25,0.7]; W₁ and W₂ are printed with the question.
","Hidden biases = [1,1]; output biases = [1,1]; all activations sigmoid.
","Target = [0.7,0.3]; η = 1.
","The paper omits the loss; this solution explicitly assumes half summed squared loss.
","Calculate forward values and update only w₁ = x₁→h₁, w₅ = h₁→o₁ and w₇ = h₁→o₂.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/c1e1a5f6adaaf15c.svg\")
![\"h_{1}=\\frac{1}{1+e^{-(1.155)}}=0.760423002\"](\"assets/equations/46c27da6983edf63.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/474f99259d04d172.svg\")
![\"h_{2}=\\frac{1}{1+e^{-(1.28)}}=0.782449776\"](\"assets/equations/c014b8bfa585b6d4.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/ffac566fd3026595.svg\")
![\"o_{1}=\\frac{1}{1+e^{-(1.263830247)}}=0.779684759\"](\"assets/equations/ee7303ec679c99fc.svg\")
Output neuron 2:
","![\"z_{o_2}](\"assets/equations/32b3910a4ac203c7.svg\")
![\"o_{2}=\\frac{1}{1+e^{-(1.255565213)}}=0.778261739\"](\"assets/equations/86e75776a23fe7d0.svg\")
Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [0.079684759, 0.478261739]. Squared errors = [0.006349661, 0.228734291]. Half-sum squared loss = 0.117541976; mean squared error = 0.117541976. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/8de071d3c6435068.svg\")
The derivative is o(1−o).
","Output delta 2:
","![\"\\delta_{o_2}](\"assets/equations/60c2dd77da13f6a8.svg\")
The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = 0.010408643. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=(0.013687964)\\times(0.760423002)\\approx](\"assets/equations/084609daa967b1e5.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,1,\\mathrm{new}}=0.1-1(0.010408643)\\approx](\"assets/equations/642d828dd8261846.svg\")
h2 → o1:
","Subtract η × gradient = 0.010710144. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,2}^{(2)}=(0.013687964)\\times(0.782449776)\\approx](\"assets/equations/64e131d9d2fc4b37.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,2,\\mathrm{new}}=0.24-1(0.010710144)\\approx](\"assets/equations/7790608d36a6c436.svg\")
h1 → o2:
","Subtract η × gradient = 0.062760617. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{2,1}^{(2)}=(0.082533822)\\times(0.760423002)\\approx](\"assets/equations/7a7372df9b0b6f47.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{2,1,\\mathrm{new}}=0.12-1(0.062760617)\\approx](\"assets/equations/3c6ebfdbda545915.svg\")
h2 → o2:
","Subtract η × gradient = 0.06457857. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{2,2}^{(2)}=(0.082533822)\\times(0.782449776)\\approx](\"assets/equations/2264f32e74104528.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{2,2,\\mathrm{new}}=0.21-1(0.06457857)\\approx](\"assets/equations/334d93272bd11b36.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.1)\\times(0.013687964)\\approx](\"assets/equations/02eca34bc6cd23b0.svg\")
Contribution from output 2.
","![\"C_2=(0.12)\\times(0.082533822)\\approx](\"assets/equations/aad6a8cccacf85e0.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1+C_2\\approx](\"assets/equations/53b10cae0b234be6.svg\")
![\"\\delta_{h_1}](\"assets/equations/8f0662ebcf4e93d4.svg\")
Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.24)\\times(0.013687964)\\approx](\"assets/equations/276b2a8c520bda99.svg\")
Contribution from output 2.
","![\"C_2=(0.21)\\times(0.082533822)\\approx](\"assets/equations/c5de7487aca6930f.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1+C_2\\approx](\"assets/equations/22192ace89a11d90.svg\")
![\"\\delta_{h_2}](\"assets/equations/60a157af571eaf0a.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = 0.000513422. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=(0.002053687)\\times(0.25)\\approx](\"assets/equations/3c850e9fc715149c.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,1,\\mathrm{new}}=0.2-1(0.000513422)\\approx](\"assets/equations/b64d0783616b88f9.svg\")
x2 → h1:
","Subtract η × gradient = 0.001437581. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,2}^{(1)}=(0.002053687)\\times(0.7)\\approx](\"assets/equations/acf691feadca0305.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,2,\\mathrm{new}}=0.15-1(0.001437581)\\approx](\"assets/equations/dcaf6fc9df70c8e9.svg\")
x1 → h2:
","Subtract η × gradient = 0.000877376. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,1}^{(1)}=(0.003509506)\\times(0.25)\\approx](\"assets/equations/7f5370d39d2a07d5.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,1,\\mathrm{new}}=0.28-1(0.000877376)\\approx](\"assets/equations/a8f3aa782e00c9f2.svg\")
x2 → h2:
","Subtract η × gradient = 0.002456654. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,2}^{(1)}=(0.003509506)\\times(0.7)\\approx](\"assets/equations/a4233a0dbb4825a8.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,2,\\mathrm{new}}=0.3-1(0.002456654)\\approx](\"assets/equations/5c8769134fc1a5f1.svg\")
Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","![\"b_{h_1,\\mathrm{new}}=1-1(0.002053687)=0.997946313\"](\"assets/equations/08fecfb5b65de142.svg\")
![\"b_{h_2,\\mathrm{new}}=1-1(0.003509506)=0.996490494\"](\"assets/equations/e12e4a571f964fad.svg\")
![\"b_{o_1,\\mathrm{new}}=1-1(0.013687964)=0.986312036\"](\"assets/equations/001388bc6f2ce55e.svg\")
![\"b_{o_2,\\mathrm{new}}=1-1(0.082533822)=0.917466178\"](\"assets/equations/7e14e96554640753.svg\")
These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
","Requested updates: w₁ = 0.199486578, w₅ = 0.089591357, w₇ = 0.057239383. For two outputs, this chosen half-sum equals their MSE; an unaveraged unhalved sum would double the gradients.
"],"kind":"practice","id":"tee-c12-s4","search":"R26 Q2a **Question (10 marks).** Use the sigmoid network Input x = [0.25, 0.7]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [1, 1]; W₂ is shown below; b₂ = [1, 1]. Hidden and output activations are sigmoid. Target y = [0.7, 0.3]. Learning rate η = 1. Here w₁ = x₁ → h₁, w₅ = h₁ → o₁ and w₇ = h₁ → o₂. Run forward and backward passes and update w₁, w₅, w₇. The paper gives no explicit loss formula.\n\n\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput x = [0.25,0.7]; W₁ and W₂ are printed with the question.\n\nHidden biases = [1,1]; output biases = [1,1]; all activations sigmoid.\n\nTarget = [0.7,0.3]; η = 1.\n\nThe paper omits the loss; this solution explicitly assumes half summed squared loss.\n\n### To find\n\nCalculate forward values and update only w₁ = x₁→h₁, w₅ = h₁→o₁ and w₇ = h₁→o₂.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Output neuron 2: **\n\n\n**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.\nPrediction minus target = [0.079684759, 0.478261739]. Squared errors = [0.006349661, 0.228734291]. Half-sum squared loss = 0.117541976; mean squared error = 0.117541976. These are different unless the number of output components makes the scale coincide.\n**Output delta 1:**\n\nThe derivative is o(1−o).\n**Output delta 2:**\n\nThe d"},{"title":"R26 Q2b","question":["Question (10 marks). Explain AlexNet regularization and training improvements; explain Inception multiscale features and 1 × 1 convolutions; recommend an architecture for large-scale image classification with justification.
"],"answer":["AlexNet uses a sequence of convolutional layers followed by fully connected classification layers. Its influential design used ReLU activations, max pooling, dropout in the fully connected part, data augmentation and GPU training. ReLU's positive derivative supports learning compared with heavily saturated sigmoid hidden units. Augmentation supplies varied training views; dropout reduces co-adaptation; GPU computation makes a large network practical to train.
","The common 227-pixel classroom layout is: Conv 11 stride 4 with 96 filters; pool 3 stride 2; Conv 5 padding 2 with 256 filters; pool 3 stride 2; three Conv 3 padding 1 layers with 384,384,256 filters; pool 3 stride 2; dense 4096; dense 4096; output 1000. Nonlinearities are applied between learned layers. Exact padding, grouping and input conventions must follow the question: the original model used channel groups in some layers, so parameter counts differ from a fully connected-channel simplification.
","AlexNet follows a mainly sequential feature-extraction path. An Inception module processes several spatial scales in parallel and concatenates them, often using 1 × 1 bottlenecks to reduce cost. Neither architecture is universally best; the task, resource budget and validation performance determine the choice.
","An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
","Recommendation: an Inception-family model is a defensible choice when varied object scales and computation efficiency matter, because parallel scales with bottlenecks provide diverse features at controlled cost. Compare held-out accuracy, latency and memory with available alternatives. This is a justified choice, not a claim that Inception always beats AlexNet on every dataset.
"],"kind":"practice","id":"tee-c12-s5","search":"R26 Q2b **Question (10 marks).** Explain AlexNet regularization and training improvements; explain Inception multiscale features and 1 × 1 convolutions; recommend an architecture for large-scale image classification with justification.\n\n**Complete worked answer.**\nAlexNet uses a sequence of convolutional layers followed by fully connected classification layers. Its influential design used ReLU activations, max pooling, dropout in the fully connected part, data augmentation and GPU training. ReLU's positive derivative supports learning compared with heavily saturated sigmoid hidden units. Augmentation supplies varied training views; dropout reduces co-adaptation; GPU computation makes a large network practical to train.\n\nThe common 227-pixel classroom layout is: Conv 11 stride 4 with 96 filters; pool 3 stride 2; Conv 5 padding 2 with 256 filters; pool 3 stride 2; three Conv 3 padding 1 layers with 384,384,256 filters; pool 3 stride 2; dense 4096; dense 4096; output 1000. Nonlinearities are applied between learned layers. Exact padding, grouping and input conventions must follow the question: the original model used channel groups in some layers, so parameter counts differ from a fully connected-channel simplification.\nAlexNet follows a mainly sequential feature-extraction path. An Inception module processes several spatial scales in parallel and concatenates them, often using 1 × 1 bottlenecks to reduce cost. Neither architecture is universally best; the task, resource budget and validation performance determine the choice.\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.\n**Recommendation: ** an Inception-family model is a defensible choice when varied object scales and computation efficiency matter, because parallel scales with bottlenecks provide diverse features at controlled cost. Compare held-out accuracy, latency and memory with available alternatives. This is a justified choice, not a claim that Inception always beats AlexNet on every dataset.\n"},{"title":"R26 Q3a","question":["Question (10 marks). Train a perceptron for one epoch, η = 1, initial weights and bias 0, threshold 0. Samples: [−1,1,−1] → −1; [1,1,1] → 1; [−1,−1,1] → −1; [−1,1,1] → −1. Discuss limitations with examples.
"],"answer":["Train a perceptron for one epoch, η = 1, initial weights and bias 0, threshold 0.
","Samples: [−1,1,−1] → −1.
","[1,1,1] → 1.
","[−1,−1,1] → −1.
","[−1,1,1] → −1.
","Process samples in order for the stated epochs and report the final weights and bias.
","Target-based perceptron correction. This worked convention changes parameters only on a wrong prediction. Use the target itself as the signed multiplier; it differs from the binary error rule.
","t is the bipolar target; η is the learning rate. Output is +1 above zero, −1 below zero and 0 at exactly zero.
","![\"\\Delta](\"assets/equations/aae3c82602ee4336.svg\")
Yaad rakhne ka cue: Is convention mein galat hua toh target × input se correction.
","State the threshold rule. Use output +1 if z>0, −1 if z<0, and 0 at z = 0. On a mismatch use Δw = η tx and Δb = ηt; on a match make no update. η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly.
","Step 1: copy this sample and target.
","x = [-1, 1, -1], target = -1.
","Step 2: calculate the current net.
","z = (0 × -1) + (0 × 1) + (0 × -1) + (0) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = -1
","Step 5: calculate all weight changes.
","Δw = -1 × [-1, 1, -1] = [1, -1, 1]
","Step 6: calculate the bias change.
","Δb = -1.
","Step 7: carry the updated parameters forward.
","New w = [1, -1, 1], new b = -1.
","Step 1: copy this sample and target.
","x = [1, 1, 1], target = 1.
","Step 2: calculate the current net.
","z = (1 × 1) + (-1 × 1) + (1 × 1) + (-1) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = 1
","Step 5: calculate all weight changes.
","Δw = 1 × [1, 1, 1] = [1, 1, 1]
","Step 6: calculate the bias change.
","Δb = 1.
","Step 7: carry the updated parameters forward.
","New w = [2, 0, 2], new b = 0.
","Step 1: copy this sample and target.
","x = [-1, -1, 1], target = -1.
","Step 2: calculate the current net.
","z = (2 × -1) + (0 × -1) + (2 × 1) + (0) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = -1
","Step 5: calculate all weight changes.
","Δw = -1 × [-1, -1, 1] = [1, 1, -1]
","Step 6: calculate the bias change.
","Δb = -1.
","Step 7: carry the updated parameters forward.
","New w = [3, 1, 1], new b = -1.
","Step 1: copy this sample and target.
","x = [-1, 1, 1], target = -1.
","Step 2: calculate the current net.
","z = (3 × -1) + (1 × 1) + (1 × 1) + (-1) = -2
","Step 3: apply the threshold.
","output = -1.
","Step 4: calculate the update multiplier.
","Update multiplier = 0
","Step 5: calculate all weight changes.
","Δw = 0 × [-1, 1, 1] = [0, 0, 0]
","Step 6: calculate the bias change.
","Δb = 0.
","Step 7: carry the updated parameters forward.
","New w = [3, 1, 1], new b = -1.
","Final after exactly the requested epochs: w = [3, 1, 1], b = -1. A finite number of requested epochs does not itself prove convergence or correct classification of every sample.
","A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator.
","For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1.
","For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR.
","A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map.
","Key correction: recomputing each net with the latest weights gives net values 0,0,0,−2 and final w = [3,1,1], b = −1 under the stated bipolar rule. The supplied answer-key table contains inconsistent intermediate arithmetic.
"],"kind":"practice","id":"tee-c12-s6","search":"R26 Q3a **Question (10 marks).** Train a perceptron for one epoch, η = 1, initial weights and bias 0, threshold 0. Samples: [−1,1,−1] → −1; [1,1,1] → 1; [−1,−1,1] → −1; [−1,1,1] → −1. Discuss limitations with examples.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nTrain a perceptron for one epoch, η = 1, initial weights and bias 0, threshold 0.\n\nSamples: [−1,1,−1] → −1.\n\n[1,1,1] → 1.\n\n[−1,−1,1] → −1.\n\n[−1,1,1] → −1.\n\n### To find\n\nProcess samples in order for the stated epochs and report the final weights and bias.\n\n### Formula — which one and why\n\n**Target-based perceptron correction.** This worked convention changes parameters only on a wrong prediction. Use the target itself as the signed multiplier; it differs from the binary error rule.\n\nt is the bipolar target; η is the learning rate. Output is +1 above zero, −1 below zero and 0 at exactly zero.\n\n\n\n**Yaad rakhne ka cue:** Is convention mein galat hua toh target × input se correction.\n\n### Solution — step by step\n\n\n\n\n**State the threshold rule.** Use output +1 if z>0, −1 if z<0, and 0 at z = 0. On a mismatch use Δw = η tx and Δb = ηt; on a match make no update. η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly.\n### Epoch 1, sample 1\n\n**Step 1: copy this sample and target.**\n\nx = [-1, 1, -1], target = -1.\n\n**Step 2: calculate the current net.**\n\nz = (0 × -1) + (0 × 1) + (0 × -1) + (0) = 0\n\n**Step 3: apply the threshold.**\n\noutput = 0.\n\n**Step 4: calculate the update multiplier.**\n\nUpdate multiplier = -1\n\n**Step 5: calculate all weight changes.**\n\nΔw = -1 × [-1, 1, -1] = [1, -1, 1]\n\n**Step 6: calculate the bias change.**\n\nΔb = -1.\n\n**Step 7: carry the updated parameters forward.**\n\nNew w = [1, -1, 1], new b = -1.\n\n\n### Epoch 1, sample 2\n\n**Step 1: copy this sample and target.**\n\nx = [1, 1, 1], target = 1.\n\n**Step 2: calculate the current net.**\n\nz = (1 × 1) + (-1 × 1) + (1 × 1) + (-1) = 0\n\n**Step 3: apply the threshold.**\n\noutput = 0.\n\n**Step 4: calculate the update multiplier.**\n\nUpdate multiplier = 1\n\n**Step 5: calculate all weight changes.**\n\nΔw = 1 × [1, 1, 1] = [1, 1, 1]\n\n**Step 6: calculate the bias change.**\n\nΔb = 1.\n\n**Step 7: carry the updated parameters forward.**\n\nNew w = [2, 0, 2], new b = 0.\n\n\n### Epoch 1, sample 3\n\n**Step 1: copy this sample and target.**\n\nx = [-1, -1, 1], target = -1.\n\n**Step 2: calculate the current net.**\n\nz = (2 × -1) + (0 × -1) + (2 × 1) + (0) = 0\n\n**Step 3: apply the threshold.**\n\noutput = 0.\n\n**Step 4: calculate the update multiplier.**\n\nUpdate multiplier = -1\n\n**Step 5: calculate all weight changes.**\n\nΔw = -1 × [-1, -1, 1] = [1, 1, -1]\n\n**Step 6: calculate the bias change.**\n\nΔb = -1.\n\n**Step 7: carry the updated parameters forward.**\n\nNew w = [3, 1, 1], new b = -1.\n\n\n### Epoch 1, sample 4\n\n**Step 1: copy this sample and target.**\n\nx = [-1, 1, 1], target = -1.\n\n**Step 2: calculate the current net.**\n\nz = (3 × -1) + (1 × 1) + (1 × 1) + (-1) = -2\n\n**Step 3: apply the threshold.**\n\noutput = -1.\n\n**Step 4: calculate the update multiplier.**\n\nUpdate multiplier = 0\n\n**Step 5: calculate all weight changes.**\n\nΔw = 0 × [-1, 1, 1] = [0, 0, 0]\n\n**Step 6: calculate the bias change.**\n\nΔb = 0.\n\n**Step 7: carry the updated parameters forward.**\n\nNew w = [3, 1, 1], new b = -1.\n\n\n**Final after exactly the requested epochs: ** w = **[3, 1, 1]**, b = **-1**. A finite number of requested epochs does not itself prove convergence or correct classification of every sample.\nA single perceptron forms a weighted sum and a"},{"title":"R26 Q3b","question":["Question (10 marks). How should CNN filter counts be selected? Explain effects on feature extraction, dimensionality, computation/memory, overfitting and hierarchical learning.
"],"answer":["Choose filter counts as architecture hyperparameters using task complexity, available data, validation performance and resource limits. There is no universal correct count.
","1. More filters can represent a wider variety of learned features at a layer; redundant filters do not automatically improve accuracy.
","2. Cout filters produce Cout output channels; spatial height/width still follow kernel, padding and stride.
","3. A K × K layer uses K²CinCout weights plus Cout biases. More output channels increase current activations and the input-channel cost of the next convolution.
","4. Extra capacity can overfit limited data, so use validation, augmentation, transfer learning or regularization. Too few channels may underfit.
","5. Later layers often use more channels after spatial downsampling, allowing complex feature combinations at a smaller spatial size. This is a common design choice, not a compulsory rule.
","Example: a 3 × 3 layer with 16 input channels has (9 × 16+1) × 32 = 4,640 parameters for 32 filters, versus 9,280 for 64 filters. Output-channel count doubles while the same spatial-size formula applies.
"],"kind":"practice","id":"tee-c12-s7","search":"R26 Q3b **Question (10 marks).** How should CNN filter counts be selected? Explain effects on feature extraction, dimensionality, computation/memory, overfitting and hierarchical learning.\n\n**Complete worked answer.**\nChoose filter counts as architecture hyperparameters using task complexity, available data, validation performance and resource limits. There is no universal correct count.\n1. More filters can represent a wider variety of learned features at a layer; redundant filters do not automatically improve accuracy.\n2. Cout filters produce Cout output channels; spatial height/width still follow kernel, padding and stride.\n3. A K × K layer uses K²CinCout weights plus Cout biases. More output channels increase current activations and the input-channel cost of the next convolution.\n4. Extra capacity can overfit limited data, so use validation, augmentation, transfer learning or regularization. Too few channels may underfit.\n5. Later layers often use more channels after spatial downsampling, allowing complex feature combinations at a smaller spatial size. This is a common design choice, not a compulsory rule.\nExample: a 3 × 3 layer with 16 input channels has (9 × 16+1) × 32 = 4,640 parameters for 32 filters, versus 9,280 for 64 filters. Output-channel count doubles while the same spatial-size formula applies.\n"},{"title":"R26 Q4a","question":["Question (10 marks). Compare batch GD and SGD for loss oscillations with a graph, memory and large datasets. Discuss learning rate extremes. For J(θ) = 0.7 θ³, θ₀ = 1, η = 0.1, perform two iterations.
"],"answer":["J(θ) = 0.7θ³; θ₀ = 1; η = 0.1.
","Calculate the derivative and perform two updates.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.
","Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.
","Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.
","Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.
","Step 1: differentiate the objective before substituting.
","Step 2: evaluate gradient at the original parameter.
","Step 3: scale the first gradient.
","Step 4: subtract to obtain the first update.
","Step 5: square the changed parameter.
","Step 6: recalculate the gradient.
","Step 7: scale the second gradient.
","Step 8: subtract from the first updated parameter.
","Final answer: θ₁ = 0.79; θ₂ = 0.658939. No individual samples are supplied, so these are the stated scalar gradient updates. The cubic is unbounded below; two steps are not a convergence proof.
"],"kind":"practice","id":"tee-c12-s8","search":"R26 Q4a **Question (10 marks).** Compare batch GD and SGD for loss oscillations with a graph, memory and large datasets. Discuss learning rate extremes. For J(θ) = 0.7 θ³, θ₀ = 1, η = 0.1, perform two iterations.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nJ(θ) = 0.7θ³; θ₀ = 1; η = 0.1.\n\n### To find\n\nCalculate the derivative and perform two updates.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\nThe learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.\n\nBatch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.\nMomentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.\nDiagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.\n**Step 1: differentiate the objective before substituting.**\n\n\n\n\n\n\n\n\n\n\n\n**Step 2: evaluate gradient at the original parameter.**\n\n\n\n\n\n\n\n**Step 3: scale the first gradient.**\n\n\n\n\n\n\n\n**Step 4: subtract to obtain the first update.**\n\n\n\n\n\n\n\n**Step 5: square the changed parameter.**\n\n\n\n\n\n\n\n**Step 6: recalculate the gradient.**\n\n\n\n\n\n\n\n**Step 7: scale the second gradient.**\n\n\n\n\n\n\n\n**Step 8: subtract from the first updated parameter.**\n\n\n\n\n\n\n\n**Final answer:** θ₁ = 0.79; θ₂ = 0.658939. No individual samples are supplied, so these are the stated scalar gradient updates. The cubic is unbounded below; two steps are not a convergence proof.\n"},{"title":"R26 Q4b","question":["Question. Compare LSTM and GRU gates and architecture. Explain the potential computational efficiency of GRU and how its gates can reduce the effect of irrelevant tokens.
","Syllabus-aligned extract: source R26 Q4b, parts i–iii. Marks for these retained parts are not separately stated.
"],"answer":["A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:
","![\"z_t=\\sigma(W_zx_t+U_zh_{t-1}+b_z),\\quad](\"assets/equations/552178cdb39642a9.svg\")
![\"\\widetilde](\"assets/equations/24dba2ef5a8c9844.svg\")
The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.
","Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.
"],"kind":"practice","id":"tee-c12-s9","search":"R26 Q4b **Question.** Compare LSTM and GRU gates and architecture. Explain the potential computational efficiency of GRU and how its gates can reduce the effect of irrelevant tokens.\n\n**Syllabus-aligned extract:** source R26 Q4b, parts i–iii. Marks for these retained parts are not separately stated.\n\n**Complete worked answer.**\nA GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:\n\n\n\nThe reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.\nCompared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.\n"},{"title":"R26 Q5a","question":["Question (10 marks). Explain feedforward limitations for sequential context and how recurrence helps. Calculate h using tanh for Wxh is shown below, Whh is shown below, x = [0,1], hprevious = [0,1], zero bias. Explain next-word probabilities.
","Wxh and Whh are printed in the question; use their receiving-neuron rows.
","Current input = [0,1]; previous hidden state = [0,1]; bias = 0.
","Activation = tanh.
","Calculate both entries of the new hidden state. Output-layer weights are not supplied.
","Recurrent forward pass. At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.
","x_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.
","Yaad rakhne ka cue: Naya input + pichhla activated state; phir activation.
","A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","Step 1: multiply the input vector into row 1.
","Step 2: multiply the input vector into row 2.
","Step 3: multiply previous state into recurrent row 1.
","Step 4: multiply previous state into recurrent row 2.
","Step 5: combine contributions for hidden unit 1.
","Step 6: combine contributions for hidden unit 2.
","Step 7: apply tanh separately to each complete sum.
","Final answer: h ≈ [0.604367777,0.800499022]. The absent vocabulary output weights prevent calculating a numerical next-word probability list.
"],"kind":"practice","id":"tee-c12-s10","search":"R26 Q5a **Question (10 marks).** Explain feedforward limitations for sequential context and how recurrence helps. Calculate h using tanh for Wxh is shown below, Whh is shown below, x = [0,1], hprevious = [0,1], zero bias. Explain next-word probabilities.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nWxh and Whh are printed in the question; use their receiving-neuron rows.\n\nCurrent input = [0,1]; previous hidden state = [0,1]; bias = 0.\n\nActivation = tanh.\n\n### To find\n\nCalculate both entries of the new hidden state. Output-layer weights are not supplied.\n\n### Formula — which one and why\n\n**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.\n\nx_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.\n\n### Solution — step by step\n\n\n\n\nA recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.\n\n\nAt each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.\nFor next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.\nA basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.\n**Step 1: multiply the input vector into row 1.**\n\n\n\n\n\n\n\n**Step 2: multiply the input vector into row 2.**\n\n\n\n\n\n\n\n**Step 3: multiply previous state into recurrent row 1.**\n\n\n\n\n\n\n\n**Step 4: multiply previous state into recurrent row 2.**\n\n\n\n\n\n\n\n**Step 5: combine contributions for hidden unit 1.**\n\n\n\n\n\n\n\n**Step 6: combine contributions for hidden unit 2.**\n\n\n\n\n\n\n\n**Step 7: apply tanh separately to each complete sum.**\n\n\n\n\n\n\n\n\n\n\n\n**Final answer:** h ≈ [0.604367777,0.800499022]. The absent vocabulary output weights prevent calculating a numerical next-word probability list.\n"},{"title":"R26 Q5b","question":["Question (10 marks). Explain RMSProp, its update rule and accumulated squared gradients. Give advantages and disadvantages relative to SGD.
"],"answer":["RMSProp keeps an exponentially weighted average of squared gradients, instead of AdaGrad's ever-growing sum. For each parameter, initialize s₀ = 0 and use:
","ρ controls how strongly earlier squared gradients are retained. A large average magnitude produces a larger denominator and a smaller step. A small average magnitude produces a relatively larger step. Squaring happens before averaging; averaging signed gradients instead would allow cancellation and would not be RMSProp.
","Example: θ₀ = 1, η = 0.1, ρ = 0.9, s₀ = 0, g₁ = 2. Then s₁ = 0.9 × 0+0.1 × 4 = 0.4. θ₁ = 1−0.2/√0.4 = 0.683772234 when ε is negligible. If g₂ = 1, s₂ = 0.9 × 0.4+0.1 × 1 = 0.46 and θ₂ = 0.683772234−0.1/√0.46 = 0.536330278.
","Benefits include adapting to different coordinate scales and avoiding AdaGrad's permanent accumulation of all past squared gradients. Costs include extra state memory and tuning η, ρ and ε; noisy or badly chosen settings can still produce poor training. Unlike standard SGD it rescales each coordinate; unlike Adam it does not include a separate first-moment estimate in this basic form.
"],"kind":"practice","id":"tee-c12-s11","search":"R26 Q5b **Question (10 marks).** Explain RMSProp, its update rule and accumulated squared gradients. Give advantages and disadvantages relative to SGD.\n\n**Complete worked answer.**\nRMSProp keeps an exponentially weighted average of squared gradients, instead of AdaGrad's ever-growing sum. For each parameter, initialize s₀ = 0 and use:\n\nρ controls how strongly earlier squared gradients are retained. A large average magnitude produces a larger denominator and a smaller step. A small average magnitude produces a relatively larger step. Squaring happens before averaging; averaging signed gradients instead would allow cancellation and would not be RMSProp.\nExample: θ₀ = 1, η = 0.1, ρ = 0.9, s₀ = 0, g₁ = 2. Then s₁ = 0.9 × 0+0.1 × 4 = 0.4. θ₁ = 1−0.2/√0.4 = 0.683772234 when ε is negligible. If g₂ = 1, s₂ = 0.9 × 0.4+0.1 × 1 = 0.46 and θ₂ = 0.683772234−0.1/√0.46 = 0.536330278.\nBenefits include adapting to different coordinate scales and avoiding AdaGrad's permanent accumulation of all past squared gradients. Costs include extra state memory and tuning η, ρ and ε; noisy or badly chosen settings can still produce poor training. Unlike standard SGD it rescales each coordinate; unlike Adam it does not include a separate first-moment estimate in this basic form.\n"},{"title":"R26 Q6a","question":["Question (10 marks). For the binary CNN below, calculate output dimensions and trainable parameters after every layer. Input 256 × 256 × 3. → Conv K = 5, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 128, stride = 1, padding = 1. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 84 units. → Dense 32 units. → Dense 1 units. The printed output size is unspecified; assume one sigmoid output. Unspecified convolution padding is taken as valid.
"],"answer":["Input 256 × 256 × 3.
","Conv K = 5, filters = 32, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 5, filters = 64, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 5, filters = 128, stride = 1, padding = 1.
","Pool K = 2, stride = 2, padding = 0.
","Dense 84 units.
","Dense 32 units.
","Dense 1 units.
","assume one sigmoid output.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 256 × 256 × 3. Use 32 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 252 × 252 × 32.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 252 × 252 × 32. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 126 × 126 × 32.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 126 × 126 × 32. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 122 × 122 × 64.
","3. Count trainable values. Each filter receives all 32 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 122 × 122 × 64. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 61 × 61 × 64.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 61 × 61 × 64. Use 128 filters of spatial size 5 × 5, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 59 × 59 × 128.
","3. Count trainable values. Each filter receives all 64 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 59 × 59 × 128. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 29 × 29 × 128.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The division gives 28.5; take its floor before adding 1. A partial window outside the unpadded input is not included.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 29 × 29 × 128. Rearrange all activation values into one vector.
","Output: a vector with 107,648 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 107,648 features. Output: 84 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 84 features. Output: 32 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 32 features. Output: 1 neurons. Every output neuron connects to every input feature and has one bias.
","| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | | 252 x 252 x 32 | 2,400 | 32 | 2,432 |
| 2 pool | | 126 x 126 x 32 | 0 | 0 | 0 |
| 3 conv | | 122 x 122 x 64 | 51,200 | 64 | 51,264 |
| 4 pool | | 61 x 61 x 64 | 0 | 0 | 0 |
| 5 conv | | 59 x 59 x 128 | 204,800 | 128 | 204,928 |
| 6 pool | | 29 x 29 x 128 | 0 | 0 | 0 |
| 7 flat | 29 × 29 × 128 | 107648 | 0 | 0 | 0 |
| 8 dense | 107648 → 84 | 84 | 9,042,432 | 84 | 9,042,516 |
| 9 dense | 84 → 32 | 32 | 2,688 | 32 | 2,720 |
| 10 dense | 32 → 1 | 1 | 32 | 1 | 33 |
Parameter substitutions, in layer order:
","Layer 1: (5 x 5 x 3 + 1) x 32 = 2,432.
","Layer 3: (5 x 5 x 32 + 1) x 64 = 51,264.
","Layer 5: (5 x 5 x 64 + 1) x 128 = 204,928.
","Layer 8: (107648 + 1) x 84 = 9,042,516.
","Layer 9: (84 + 1) x 32 = 2,720.
","Layer 10: (32 + 1) x 1 = 33.
","Total: 2,432 + 51,264 + 204,928 + 9,042,516 + 2,720 + 33 = 9,303,893 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","Correction: Conv 3 has (5 × 5 × 64+1) × 128 = 204,928 parameters, not 73,856 as printed in the key. Correct total is 9,303,893. If binary classification is instead implemented with two softmax outputs, the last dense layer has(32+1) × 2 = 66 parameters and the total becomes 9,303,926; explicitly state the output convention.
"],"kind":"practice","id":"tee-c12-s12","search":"R26 Q6a **Question (10 marks).** For the binary CNN below, calculate output dimensions and trainable parameters after every layer. Input 256 × 256 × 3. → Conv K = 5, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 128, stride = 1, padding = 1. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 84 units. → Dense 32 units. → Dense 1 units. The printed output size is unspecified; assume one sigmoid output. Unspecified convolution padding is taken as valid.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput 256 × 256 × 3.\n\nConv K = 5, filters = 32, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nConv K = 5, filters = 64, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nConv K = 5, filters = 128, stride = 1, padding = 1.\n\nPool K = 2, stride = 2, padding = 0.\n\nDense 84 units.\n\nDense 32 units.\n\nDense 1 units.\n\nassume one sigmoid output.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{256+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 252 x 252 x 32 | 2,400 | 32 | 2,432 |\n| 2 pool | \\(\\left\\lfloor\\frac{252-2}{2}\\right\\rfloor+1\\) | 126 x 126 x 32 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{126+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 122 x 122 x 64 | 51,200 | 64 | 51,264 |\n| 4 pool | \\(\\left\\lfloor\\frac{122-2}{2}\\right\\rfloor+1\\) | 61 x 61 x 64 | 0 | 0 | 0 |\n| 5 conv | \\(\\left\\lfloor\\frac{61+2 \\times 1-5}{1}\\right\\rfloor+1\\) | 59 x 59 x 128 | 204,800 | 128 | 204,928 |\n| 6 pool | \\(\\left\\lfloor\\frac{59-2}{2}\\right\\rfloor+1\\) | 29 x 29 x 128 | 0 | 0 | 0 |\n| 7 flat | 29 × 29 × 128 | 107648 | 0 | 0 | 0 |\n| 8 dense | 107648 → 84 | 84 | 9,042,432 | 84 | 9,042,516 |\n| 9 dense | 84 → 32 | 32 | 2,688 | 32 | 2,720 |\n| 10 dense | 32 → 1 | 1 | 32 | 1 | 33 |\n**Parameter substitutions, in layer order: **\nLayer 1: (5 x 5 x 3 + 1) x 32 = 2,432.\nLayer 3: (5 x 5 x 32 + 1) x 64 = 51,264.\nLayer 5: (5 x 5 x 64 + 1) x 128 = 204,928.\nLayer 8: (107648 + 1) x 84 = 9,042,516.\nLayer 9: (84 + 1) x 32 = 2,720.\nLayer 10: (32 + 1) x 1 = 33.\n**Total: ** 2,432 + 51,264 + 204,928 + 9,042,516 + 2,720 + 33 = **9,303,893 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.\n**Correction: ** Conv 3 has (5 × 5 × 64+1) × 128 = **204,928** parameters, not 73,856 as printed in the key. Correct total is "},{"title":"R26 Q6b","question":["Question (10 marks). For a 10-layer network using sigmoid: compute f(6), compute its derivative at 6, explain its effect on backpropagation, and calculate softmax for [2.3,4.5,3.2].
"],"answer":["Sigmoid is evaluated at z = 6 in a ten-layer network.
","Softmax scores = [2.3,4.5,3.2].
","Calculate sigmoid and its derivative at 6, explain the gradient effect and calculate all three softmax probabilities.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.
","zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.
","Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.
","f(6) = ![\"\\frac{1}{1+e^{-6}}\"](\"assets/equations/ac8062a4bfd609d2.svg\")
= 0.997527377. Derivative = f(6)[1−f(6)] = 0.997527377 × 0.002472623 = 0.002466509. This is a saturated sigmoid with a very small slope. Repeatedly multiplying small activation derivatives and weight factors can weaken earlier-layer gradients; the exact gradient also depends on weights and all preactivations, so it is not simply this derivative raised to 10 unless those conditions are assumed.
Step 1 stabilize: subtract the largest logit 4.5 from every value: [-2.2, 0, -1.3]. This does not change softmax.
","Step 2 exponentiate: exp values = [0.110803158, 1, 0.272531793].
","Step 3 denominator: sum = 1.383334951.
","Step 4 divide each value by this same sum: probabilities = [0.080098575, 0.722890721, 0.197010704]. Check: all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.
"],"kind":"practice","id":"tee-c12-s13","search":"R26 Q6b **Question (10 marks).** For a 10-layer network using sigmoid: compute f(6), compute its derivative at 6, explain its effect on backpropagation, and calculate softmax for [2.3,4.5,3.2].\n\n**Complete worked answer.**\n\n\n\n### Given\n\nSigmoid is evaluated at z = 6 in a ten-layer network.\n\nSoftmax scores = [2.3,4.5,3.2].\n\n### To find\n\nCalculate sigmoid and its derivative at 6, explain the gradient effect and calculate all three softmax probabilities.\n\n### Formula — which one and why\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.\n\nzⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.\n\n### Solution — step by step\n\n\n\n\n\n\nf(6) = \\(\\frac{1}{1+e^{-6}}\\) = **0.997527377**. Derivative = f(6)[1−f(6)] = 0.997527377 × 0.002472623 = **0.002466509**. This is a saturated sigmoid with a very small slope. Repeatedly multiplying small activation derivatives and weight factors can weaken earlier-layer gradients; the exact gradient also depends on weights and all preactivations, so it is not simply this derivative raised to 10 unless those conditions are assumed.\n**Step 1 stabilize: ** subtract the largest logit 4.5 from every value: [-2.2, 0, -1.3]. This does not change softmax.\n\n**Step 2 exponentiate: ** exp values = [0.110803158, 1, 0.272531793].\n\n**Step 3 denominator: ** sum = 1.383334951.\n\n**Step 4 divide each value by this same sum: ** probabilities = **[0.080098575, 0.722890721, 0.197010704]**. **Check: ** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.\n"},{"title":"R26 Q7a","question":["Question (10 marks). Compare single-layer and multilayer feedforward networks and explain hidden nonlinearity. Compute the output for Input x = [0.5, 0.2, 0.1]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.5]. Hidden and output activations are sigmoid.
","Input x = [0.5, 0.2, 0.1].
","W₁ is printed with the question.
","b₁ = [0.1, 0.2].
","W₂ is printed with the question.
","b₂ = [0.5].
","Hidden and output activations are sigmoid.
","Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator.
","For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1.
","For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR.
","A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/eecbe3a4008b0943.svg\")
![\"h_{1}=\\frac{1}{1+e^{-(0.25)}}=0.562176501\"](\"assets/equations/00585fcf1de8913d.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/b2e10076eac3269f.svg\")
![\"h_{2}=\\frac{1}{1+e^{-(0.52)}}=0.627147766\"](\"assets/equations/e33e2e29f7e69ace.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/8b8732deca8578d9.svg\")
![\"o_{1}=\\frac{1}{1+e^{-(1.40173889)}}=0.802459679\"](\"assets/equations/728269faf184ac73.svg\")
Question (10 marks). Explain transfer learning with CNNs and how to adapt a pretrained model to a new image classification task, including an example model.
"],"answer":["Transfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.
","1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together.
","2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.
","3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.
","4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.
","5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.
","6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds.
","Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.
","Concrete example: replace the 1000-class head of a pretrained ResNet with a 10-class vegetable head. Train the new head with the base frozen, then optionally fine-tune late residual blocks at a smaller rate. Input preprocessing must match the pretrained model. The 10 output probabilities sum to 1 for mutually exclusive classes.
"],"kind":"practice","id":"tee-c12-s15","search":"R26 Q7b **Question (10 marks).** Explain transfer learning with CNNs and how to adapt a pretrained model to a new image classification task, including an example model.\n\n**Complete worked answer.**\nTransfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.\n1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together.\n2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.\n3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.\n4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.\n5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.\n6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds.\nFeature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.\n**Concrete example: ** replace the 1000-class head of a pretrained ResNet with a 10-class vegetable head. Train the new head with the base frozen, then optionally fine-tune late residual blocks at a smaller rate. Input preprocessing must match the pretrained model. The 10 output probabilities sum to 1 for mutually exclusive classes.\n\n\n"}]},{"n":13,"title":"Final examination December 2023 fully solved","sections":[{"title":"Start here","blocks":["F23 = S17, 2 December 2023. Scanned formulas and diagrams were checked visually rather than trusting OCR alone.
","Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible.
"],"kind":"read","id":"tee-c13-intro"},{"title":"F23 Q1a","question":["Question (5 marks). State the features and limitations of deep learning.
"],"answer":["Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.
","| Comparison | Traditional ML approaches | Deep learning approaches |
|---|---|---|
| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |
| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |
| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |
| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |
These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.
","Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.
"],"kind":"practice","id":"tee-c13-s0","search":"F23 Q1a **Question (5 marks).** State the features and limitations of deep learning.\n\n**Complete worked answer.**\nDeep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.\n| Comparison | Traditional ML approaches | Deep learning approaches |\n| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |\n| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |\n| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |\n| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |\nThese are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.\nDrivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.\n"},{"title":"F23 Q1b","question":["Question (5 marks). Identify the hyperparameters to tune when training an ANN and explain their significance.
"],"answer":["Hyperparameters are chosen outside the ordinary weight-gradient updates. Learning rate controls step size; number of layers and units controls capacity; activation determines nonlinear behavior; batch size controls the data used per update and memory demand; epochs control training duration. Regularization strength, dropout rate and early-stopping patience affect generalization. Optimizer-specific choices include momentum β or Adam β₁, β₂,ε. Tune using validation data while keeping the final test set separate. Weights and biases are learned parameters; they are not hyperparameters merely because their initial values must be selected.
"],"kind":"practice","id":"tee-c13-s1","search":"F23 Q1b **Question (5 marks).** Identify the hyperparameters to tune when training an ANN and explain their significance.\n\n**Complete worked answer.**\nHyperparameters are chosen outside the ordinary weight-gradient updates. Learning rate controls step size; number of layers and units controls capacity; activation determines nonlinear behavior; batch size controls the data used per update and memory demand; epochs control training duration. Regularization strength, dropout rate and early-stopping patience affect generalization. Optimizer-specific choices include momentum β or Adam β₁, β₂,ε. Tune using validation data while keeping the final test set separate. Weights and biases are learned parameters; they are not hyperparameters merely because their initial values must be selected.\n"},{"title":"F23 Q1c","question":["Question (5 marks). Draw a time-unrolled RNN and briefly explain training by backpropagation through time.
"],"answer":["Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.
","1. Initialize the hidden state and run the forward recurrence from the first input to the last.
","2. Compute the specified output loss, at the final step or across several steps.
","3. Start from the last loss and propagate derivatives backwards through output and hidden operations.
","4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.
","5. Sum all uses of each shared weight before applying the optimizer update.
","The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.
","Question (5 marks). Compute the convolution using the input A and kernel K shown below using padding 0 and stride 1.
","| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Stride 1; P = 0 (no padding).
Compute the convolution using the input A and kernel K shown below using padding 0 and stride 1.
","Find every convolution output cell.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Step 1 shape: = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 1–3. The window moves 1 cell per output step.
| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
Only the outlined cells belong to this output.
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Same row and column positions; no matrix multiplication.
| 10 | 9 | 5 |
| 5 | 14 | 6 |
| 3 | 3 | 2 |
Each cell is input × matching kernel coefficient.
24 + 25 + 8 = 57
| 57 | 59 |
| 40 | 50 |
Output row 1, column 1.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 2–4. The window moves 1 cell per output step.
| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
Only the outlined cells belong to this output.
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Same row and column positions; no matrix multiplication.
| 9 | 5 | 6 |
| 7 | 12 | 7 |
| 3 | 2 | 8 |
Each cell is input × matching kernel coefficient.
20 + 26 + 13 = 59
| 57 | 59 |
| 40 | 50 |
Output row 1, column 2.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–4, columns 1–3. The window moves 1 cell per output step.
| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
Only the outlined cells belong to this output.
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Same row and column positions; no matrix multiplication.
| 5 | 7 | 6 |
| 3 | 6 | 2 |
| 3 | 2 | 6 |
Each cell is input × matching kernel coefficient.
18 + 11 + 11 = 40
| 57 | 59 |
| 40 | 50 |
Output row 2, column 1.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–4, columns 2–4. The window moves 1 cell per output step.
| 10 | 9 | 5 | 6 |
| 5 | 7 | 6 | 7 |
| 3 | 3 | 2 | 8 |
| 3 | 2 | 6 | 7 |
Only the outlined cells belong to this output.
| 1 | 1 | 1 |
| 1 | 2 | 1 |
| 1 | 1 | 1 |
Same row and column positions; no matrix multiplication.
| 7 | 6 | 7 |
| 3 | 4 | 8 |
| 2 | 6 | 7 |
Each cell is input × matching kernel coefficient.
20 + 15 + 15 = 50
| 57 | 59 |
| 40 | 50 |
Output row 2, column 2.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Final output: . Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
Question (10 marks). A feedforward network has inputs 1,2; hidden nodes 3,4; outputs 5,6. Weights: w13 = −2, w23 = 3, w14 = 4, w24 = −1, w35 = 1, w45 = −1, w36 = −1, w46 = 1; all biases 0. All nodes 3–6 use step(z) = 1 if z≥0 else 0. Find(y5, y6) for inputs(0,0), (1,0), (0,1), (1,1).
"],"answer":["A feedforward network has inputs 1,2.
","hidden nodes 3,4.
","outputs 5,6.
","Weights: w13 = −2, w23 = 3, w14 = 4, w24 = −1, w35 = 1, w45 = −1, w36 = −1, w46 = 1.
","all biases 0.
","All nodes 3–6 use step(z) = 1 if z≥0 else 0.
","Find(y5, y6) for inputs(0,0), (1,0), (0,1), (1,1).
","Write every row of the truth table and the requested network output.
","Threshold logic. Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.
","step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.
","Yaad rakhne ka cue: Har input row ka sum nikalo, phir threshold lagao.
","Use exactly the printed step: output 1 if z≥0, otherwise 0. In particular, step(0) = 1. The equations are z₃ = −2x₁+3x₂, z₄ = 4x₁−x₂, z₅ = h₃−h₄, z₆ = −h₃+h₄.
","Step 1: Input(0,0). z₃ = −2 × 0+3 × 0 = 0; h₃ = 1. z₄ = 4 × 0−0 = 0; h₄ = 1. z₅ = 1−1 = 0; y₅ = 1. z₆ = −1+1 = 0; y₆ = 1. Final pair = (1,1).
","Step 2: Input(1,0). z₃ = −2 × 1+3 × 0 = -2; h₃ = 0. z₄ = 4 × 1−0 = 4; h₄ = 1. z₅ = 0−1 = -1; y₅ = 0. z₆ = −0+1 = 1; y₆ = 1. Final pair = (0,1).
","Step 3: Input(0,1). z₃ = −2 × 0+3 × 1 = 3; h₃ = 1. z₄ = 4 × 0−1 = -1; h₄ = 0. z₅ = 1−0 = 1; y₅ = 1. z₆ = −1+0 = -1; y₆ = 0. Final pair = (1,0).
","Step 4: Input(1,1). z₃ = −2 × 1+3 × 1 = 1; h₃ = 1. z₄ = 4 × 1−1 = 3; h₄ = 1. z₅ = 1−1 = 0; y₅ = 1. z₆ = −1+1 = 0; y₆ = 1. Final pair = (1,1).
"],"kind":"practice","id":"tee-c13-s4","search":"F23 Q2a **Question (10 marks).** A feedforward network has inputs 1,2; hidden nodes 3,4; outputs 5,6. Weights: w13 = −2, w23 = 3, w14 = 4, w24 = −1, w35 = 1, w45 = −1, w36 = −1, w46 = 1; all biases 0. All nodes 3–6 use step(z) = 1 if z≥0 else 0. Find(y5, y6) for inputs(0,0), (1,0), (0,1), (1,1).\n\n**Complete worked answer.**\n\n\n\n### Given\n\nA feedforward network has inputs 1,2.\n\nhidden nodes 3,4.\n\noutputs 5,6.\n\nWeights: w13 = −2, w23 = 3, w14 = 4, w24 = −1, w35 = 1, w45 = −1, w36 = −1, w46 = 1.\n\nall biases 0.\n\nAll nodes 3–6 use step(z) = 1 if z≥0 else 0.\n\nFind(y5, y6) for inputs(0,0), (1,0), (0,1), (1,1).\n\n### To find\n\nWrite every row of the truth table and the requested network output.\n\n### Formula — which one and why\n\n**Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.\n\nstep(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao.\n\n### Solution — step by step\n\n\n\n\n\n\nUse exactly the printed step: output 1 if z≥0, otherwise 0. In particular, step(0) = 1. The equations are z₃ = −2x₁+3x₂, z₄ = 4x₁−x₂, z₅ = h₃−h₄, z₆ = −h₃+h₄.\n**Step 1: Input(0,0).** z₃ = −2 × 0+3 × 0 = 0; h₃ = 1. z₄ = 4 × 0−0 = 0; h₄ = 1. z₅ = 1−1 = 0; y₅ = 1. z₆ = −1+1 = 0; y₆ = 1. Final pair = **(1,1)**.\n**Step 2: Input(1,0).** z₃ = −2 × 1+3 × 0 = -2; h₃ = 0. z₄ = 4 × 1−0 = 4; h₄ = 1. z₅ = 0−1 = -1; y₅ = 0. z₆ = −0+1 = 1; y₆ = 1. Final pair = **(0,1)**.\n**Step 3: Input(0,1).** z₃ = −2 × 0+3 × 1 = 3; h₃ = 1. z₄ = 4 × 0−1 = -1; h₄ = 0. z₅ = 1−0 = 1; y₅ = 1. z₆ = −1+0 = -1; y₆ = 0. Final pair = **(1,0)**.\n**Step 4: Input(1,1).** z₃ = −2 × 1+3 × 1 = 1; h₃ = 1. z₄ = 4 × 1−1 = 3; h₄ = 1. z₅ = 1−1 = 0; y₅ = 1. z₆ = −1+1 = 0; y₆ = 1. Final pair = **(1,1)**.\n"},{"title":"F23 Q2b","question":["Question (10 marks). (i) Justify the need for activations and explain any two. (ii) Choose ML or DL for facial recognition using 227 × 227 × 1 images and state the advantages.
"],"answer":["An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.
","Sigmoid
","Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.
","Hyperbolic tangent
","The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.
","ReLU
","The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.
","Leaky ReLU
","Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.
","Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.
","Facial recognition: a CNN is a suitable DL choice because the input contains spatial visual patterns. It learns local and hierarchical features with shared filters, reducing dependence on manually selected face descriptors. One channel means grayscale. With limited target data, pretrained features and carefully chosen augmentation are preferable starting points; the image dimensions alone do not guarantee enough data for training from scratch.
"],"kind":"practice","id":"tee-c13-s5","search":"F23 Q2b **Question (10 marks).** (i) Justify the need for activations and explain any two. (ii) Choose ML or DL for facial recognition using 227 × 227 × 1 images and state the advantages.\n\n**Complete worked answer.**\nAn activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.\n**Sigmoid**\n\nHere z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.\n**Hyperbolic tangent**\n\nThe numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.\n**ReLU**\n\nThe output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.\n**Leaky ReLU**\n\nAlpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.\nSigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.\n**Facial recognition: ** a CNN is a suitable DL choice because the input contains spatial visual patterns. It learns local and hierarchical features with shared filters, reducing dependence on manually selected face descriptors. One channel means grayscale. With limited target data, pretrained features and carefully chosen augmentation are preferable starting points; the image dimensions alone do not guarantee enough data for training from scratch.\n"},{"title":"F23 Q3a","question":["Question (10 marks). For x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers. Weights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1. Update w46 and w56 using backpropagation.
","For x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers.
","Weights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1.
","Update w46 and w56 using backpropagation.
","Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/743c8cd16a50ddc1.svg\")
![\"h_{1}=\\frac{1}{1+e^{-(-0.7)}}=0.331812228\"](\"assets/equations/a03d842e36f6059c.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/16f057ecfbad99c5.svg\")
![\"h_{2}=\\frac{1}{1+e^{-(0.1)}}=0.524979187\"](\"assets/equations/7b09cff87f366114.svg\")
Output neuron 1:
","![\"z_{o_1}](\"assets/equations/c16074919b105b06.svg\")
![\"o_{1}=\\frac{1}{1+e^{-(-0.104539506)}}=0.473888899\"](\"assets/equations/942e59b171acdc83.svg\")
Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [-0.526111101]. Squared errors = [0.276792891]. Half-sum squared loss = 0.138396445; mean squared error = 0.276792891. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/fa2e026fc3449bb0.svg\")
The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.039171154. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=(-0.131169078)\\times(0.331812228)\\approx](\"assets/equations/4a6b58e5f34c0b65.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,1,\\mathrm{new}}=-0.3-0.9(-0.043523504)\\approx](\"assets/equations/08c4c4ecc81b818b.svg\")
h2 → o1:
","Subtract η × gradient = −0.061974932. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,2}^{(2)}=(-0.131169078)\\times(0.524979187)\\approx](\"assets/equations/a93072cdabc372b8.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,2,\\mathrm{new}}=-0.2-0.9(-0.068861036)\\approx](\"assets/equations/2f20144355441ad1.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(-0.3)\\times(-0.131169078)\\approx](\"assets/equations/0153f0b9bcb4caa4.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/c677dcae787d564f.svg\")
![\"\\delta_{h_1}](\"assets/equations/890b9a76a0888667.svg\")
Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(-0.2)\\times(-0.131169078)\\approx](\"assets/equations/b7f5cea4501981c5.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/cea4b9d0baabf5b5.svg\")
![\"\\delta_{h_2}](\"assets/equations/fc3edfca69eaeef5.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = 0.007852106. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=(0.008724562)\\times(1)\\approx](\"assets/equations/548224ebc74d73ba.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,1,\\mathrm{new}}=0.2-0.9(0.008724562)\\approx](\"assets/equations/0b50d4b76d5185b7.svg\")
x2 → h1:
","Subtract η × gradient = 0. This weight stays unchanged.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,2}^{(1)}=(0.008724562)\\times(0)\\approx](\"assets/equations/5c9199d39ef465bf.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,2,\\mathrm{new}}=0.4-0.9(0)\\approx](\"assets/equations/10496bc73c8c3b85.svg\")
x3 → h1:
","Subtract η × gradient = 0.007852106. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,3}^{(1)}=\\frac{\\partial](\"assets/equations/75cb25a10b4d1d19.svg\")
![\"g_{1,3}^{(1)}=(0.008724562)\\times(1)\\approx](\"assets/equations/27c65cc6a73f6ef7.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,3,\\mathrm{new}}=-0.5-0.9(0.008724562)\\approx](\"assets/equations/b5e18accd983b7d3.svg\")
x1 → h2:
","Subtract η × gradient = 0.005887876. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,1}^{(1)}=(0.006542085)\\times(1)\\approx](\"assets/equations/e657c5ac09391365.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,1,\\mathrm{new}}=-0.3-0.9(0.006542085)\\approx](\"assets/equations/5c15a33a77f594c0.svg\")
x2 → h2:
","Subtract η × gradient = 0. This weight stays unchanged.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,2}^{(1)}=(0.006542085)\\times(0)\\approx](\"assets/equations/e95a4cf560314569.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,2,\\mathrm{new}}=0.1-0.9(0)\\approx](\"assets/equations/c531dfb193e72d45.svg\")
x3 → h2:
","Subtract η × gradient = 0.005887876. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,3}^{(1)}=\\frac{\\partial](\"assets/equations/48d08524866113b9.svg\")
![\"g_{2,3}^{(1)}=(0.006542085)\\times(1)\\approx](\"assets/equations/c2728e830590acf7.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,3,\\mathrm{new}}=0.2-0.9(0.006542085)\\approx](\"assets/equations/d92699928835e1d2.svg\")
Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","![\"b_{h_1,\\mathrm{new}}=-0.4-0.9(0.008724562)=-0.407852106\"](\"assets/equations/96118c608a159aef.svg\")
![\"b_{h_2,\\mathrm{new}}=0.2-0.9(0.006542085)=0.194112124\"](\"assets/equations/85c1ef6621c6cf83.svg\")
![\"b_{o_1,\\mathrm{new}}=0.1-0.9(-0.131169078)=0.218052170\"](\"assets/equations/172a74d19c4bbc33.svg\")
These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
","Required: w46 = −0.260828846, w56 = −0.138025068 under half squared loss. If a one-output unhalved MSE is explicitly required, double the shown gradients before updating; the paper itself does not state that scaling.
"],"kind":"practice","id":"tee-c13-s6","search":"F23 Q3a **Question (10 marks).** For x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers. Weights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1. Update w46 and w56 using backpropagation.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nFor x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers.\n\nWeights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1.\n\nUpdate w46 and w56 using backpropagation.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.\nPrediction minus target = [-0.526111101]. Squared errors = [0.276792891]. Half-sum squared loss = 0.138396445; mean squared error = 0.276792891. These are different unless the number of output components makes the scale coincide.\n**Output delta 1:**\n\nThe derivative is o(1−o).\n**Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta.\nh1 → o1: gradient = (-0.131169078) × 0.331812228 = -0.043523504. New weight = -0.3−0.9 × (-0.043523504) = **-0.260828846**.\nh2 → o1: gradient = (-0.131169078) × 0.524979187 = -0.068861036. New weight = -0.2−0.9 × (-0.068861036) = **-0.138025068**.\n**Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.\n**Hidden 1 downstream contribution:**\n\n\n**Hidden 2 downstream contribution:**\n\n\n**Step 5 input weights.** M"},{"title":"F23 Q3b","question":["Question (10 marks). Explain key advantages of Adam for ANN training and how it differs from SGD.
"],"answer":["Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.
","Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.
","For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.
","Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.
"],"kind":"practice","id":"tee-c13-s7","search":"F23 Q3b **Question (10 marks).** Explain key advantages of Adam for ANN training and how it differs from SGD.\n\n**Complete worked answer.**\nAdam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.\n\n\nUse update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.\nFor g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.\nCompared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.\n"},{"title":"F23 Q4a","question":["Question (10 marks). Determine feature-map dimensions and parameters at every layer of the pictured AlexNet-style CNN. The normalized architecture is Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units.
"],"answer":["The normalized architecture is Input 227 × 227 × 3.
","Conv K = 11, filters = 96, stride = 4, padding = 0.
","Pool K = 3, stride = 2, padding = 0.
","Conv K = 5, filters = 256, stride = 1, padding = 2.
","Pool K = 3, stride = 2, padding = 0.
","Conv K = 3, filters = 384, stride = 1, padding = 1.
","Conv K = 3, filters = 384, stride = 1, padding = 1.
","Conv K = 3, filters = 256, stride = 1, padding = 1.
","Pool K = 3, stride = 2, padding = 0.
","Dense 4096 units.
","Dense 4096 units.
","Dense 1000 units.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 227 × 227 × 3. Use 96 filters of spatial size 11 × 11, stride 4, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 55 × 55 × 96.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 55 × 55 × 96. Use a 3 × 3 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 27 × 27 × 96.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 27 × 27 × 96. Use 256 filters of spatial size 5 × 5, stride 1, and padding 2 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 27 × 27 × 256.
","3. Count trainable values. Each filter receives all 96 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 27 × 27 × 256. Use a 3 × 3 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 13 × 13 × 256.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 256. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.
","3. Count trainable values. Each filter receives all 256 input channels. Its weights are reused at every spatial position.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 384. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.
","3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 384. Use 256 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 256.
","3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 13 × 13 × 256. Use a 3 × 3 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 6 × 6 × 256.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 6 × 6 × 256. Rearrange all activation values into one vector.
","Output: a vector with 9,216 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 9,216 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 4,096 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 4,096 features. Output: 1,000 neurons. Every output neuron connects to every input feature and has one bias.
","| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | | 55 x 55 x 96 | 34,848 | 96 | 34,944 |
| 2 pool | | 27 x 27 x 96 | 0 | 0 | 0 |
| 3 conv | | 27 x 27 x 256 | 614,400 | 256 | 614,656 |
| 4 pool | | 13 x 13 x 256 | 0 | 0 | 0 |
| 5 conv | | 13 x 13 x 384 | 884,736 | 384 | 885,120 |
| 6 conv | | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 |
| 7 conv | | 13 x 13 x 256 | 884,736 | 256 | 884,992 |
| 8 pool | | 6 x 6 x 256 | 0 | 0 | 0 |
| 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 |
| 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 |
| 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 |
| 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 |
Parameter substitutions, in layer order:
","Layer 1: (11 x 11 x 3 + 1) x 96 = 34,944.
","Layer 3: (5 x 5 x 96 + 1) x 256 = 614,656.
","Layer 5: (3 x 3 x 256 + 1) x 384 = 885,120.
","Layer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488.
","Layer 7: (3 x 3 x 384 + 1) x 256 = 884,992.
","Layer 10: (9216 + 1) x 4096 = 37,752,832.
","Layer 11: (4096 + 1) x 4096 = 16,781,312.
","Layer 12: (4096 + 1) x 1000 = 4,097,000.
","Total: 34,944 + 614,656 + 885,120 + 1,327,488 + 884,992 + 37,752,832 + 16,781,312 + 4,097,000 = 62,378,344 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","The count treats every listed convolution as connected to all input channels. Original grouped AlexNet implementations have different counts; do not insert grouping that is not shown in the question.
"],"kind":"practice","id":"tee-c13-s8","search":"F23 Q4a **Question (10 marks).** Determine feature-map dimensions and parameters at every layer of the pictured AlexNet-style CNN. The normalized architecture is Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nThe normalized architecture is Input 227 × 227 × 3.\n\nConv K = 11, filters = 96, stride = 4, padding = 0.\n\nPool K = 3, stride = 2, padding = 0.\n\nConv K = 5, filters = 256, stride = 1, padding = 2.\n\nPool K = 3, stride = 2, padding = 0.\n\nConv K = 3, filters = 384, stride = 1, padding = 1.\n\nConv K = 3, filters = 384, stride = 1, padding = 1.\n\nConv K = 3, filters = 256, stride = 1, padding = 1.\n\nPool K = 3, stride = 2, padding = 0.\n\nDense 4096 units.\n\nDense 4096 units.\n\nDense 1000 units.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{227+2 \\times 0-11}{4}\\right\\rfloor+1\\) | 55 x 55 x 96 | 34,848 | 96 | 34,944 |\n| 2 pool | \\(\\left\\lfloor\\frac{55-3}{2}\\right\\rfloor+1\\) | 27 x 27 x 96 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{27+2 \\times 2-5}{1}\\right\\rfloor+1\\) | 27 x 27 x 256 | 614,400 | 256 | 614,656 |\n| 4 pool | \\(\\left\\lfloor\\frac{27-3}{2}\\right\\rfloor+1\\) | 13 x 13 x 256 | 0 | 0 | 0 |\n| 5 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 384 | 884,736 | 384 | 885,120 |\n| 6 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 |\n| 7 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 256 | 884,736 | 256 | 884,992 |\n| 8 pool | \\(\\left\\lfloor\\frac{13-3}{2}\\right\\rfloor+1\\) | 6 x 6 x 256 | 0 | 0 | 0 |\n| 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 |\n| 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 |\n| 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 |\n| 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 |\n**Parameter substitutions, in layer order: **\nLayer 1: (11 x 11 x 3 + 1) x 96 = 34,944.\nLayer 3: (5 x 5 x 96 + 1) x 256 = 614,656.\nLayer 5: (3 x 3 x 256 + 1) x 384 = 885,120.\nLayer 6: (3 x"},{"title":"F23 Q4b","question":["Question (10 marks). Design an LSTM autocomplete model for “Maharashtra is a state in the western part of India. The capital of Maharashtra is ___.” Draw the model and write conditional probabilities at each time step.
"],"answer":["An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.
","The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.
","In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.
","Architecture: token → embedding → LSTM state → vocabulary softmax → next-token distribution. During training use the actual prefix and its next-word labels with cross-entropy. At inference, feed the available prefix and choose/sample the next token. “Mumbai” is the intended factual completion in this example; a model only predicts it if learned probabilities support it.
","One explicit tokenization ignoring punctuation:
","1. P(Maharashtra | START).
","2. P(is | Maharashtra).
","3. P(a | Maharashtra is).
","4. P(state | Maharashtra is a).
","5. P(in | Maharashtra is a state).
","6. P(the | Maharashtra is a state in).
","7. P(western | Maharashtra is a state in the).
","8. P(part | Maharashtra is a state in the western).
","9. P(of | Maharashtra is a state in the western part).
","10. P(India | Maharashtra is a state in the western part of).
","11. P(The | Maharashtra is a state in the western part of India).
","12. P(capital | Maharashtra is a state in the western part of India The).
","13. P(of | Maharashtra is a state in the western part of India The capital).
","14. P(Maharashtra | Maharashtra is a state in the western part of India The capital of).
","15. P(is | Maharashtra is a state in the western part of India The capital of Maharashtra).
","16. P(Mumbai | Maharashtra is a state in the western part of India The capital of Maharashtra is).
","The joint sequence probability is the product of these conditionals. Numerical probability values cannot be computed from the sentence alone because weights are not supplied.
","Question (10 marks). For Inception V 1, justify different kernel sizes in one block, explain 1 × 1 convolutions and sketch one block.
"],"answer":["An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
"],"kind":"practice","id":"tee-c13-s10","search":"F23 Q5a **Question (10 marks).** For Inception V 1, justify different kernel sizes in one block, explain 1 × 1 convolutions and sketch one block.\n\n**Complete worked answer.**\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.\n"},{"title":"F23 Q5b","question":["Question (10 marks). Explain why a single perceptron cannot classify nonlinearly separable data. Implement XOR and XNOR using suitable perceptrons.
"],"answer":["Required logic functions = XOR and XNOR on two binary inputs.
","Weights and biases are design choices, not given constants.
","Explain the failure of a single linear separator, choose suitable hidden/output perceptrons and show all four truth-table rows.
","Threshold logic. Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.
","step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.
","Yaad rakhne ka cue: Har input row ka sum nikalo, phir threshold lagao.
","A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator.
","For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1.
","For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR.
","A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map.
","Step 1: calculate hidden OR preactivation.
","![\"z_1=0+0-0.5=-0.5\"](\"assets/equations/aac6904086092425.svg\")
Step 2: apply its threshold.
","![\"h_1=0\"](\"assets/equations/738e2d766424863e.svg\")
Step 3: calculate hidden AND preactivation.
","![\"z_2=0+0-1.5=-1.5\"](\"assets/equations/ad20a476f045fd25.svg\")
Step 4: apply its threshold.
","![\"h_2=0\"](\"assets/equations/1de867d02667e910.svg\")
Step 5: calculate XOR output preactivation.
","![\"z_{XOR}=0-2(0)-0.5=-0.5\"](\"assets/equations/65c6f2e86202561b.svg\")
Step 6: apply the XOR threshold.
","![\"y_{XOR}=0\"](\"assets/equations/c3624761b5a00605.svg\")
Step 7: calculate XNOR output preactivation.
","![\"z_{XNOR}=-0+2(0)+0.5=0.5\"](\"assets/equations/458da5139ca95df2.svg\")
Step 8: apply the XNOR threshold.
","![\"y_{XNOR}=1\"](\"assets/equations/016b62d9d7436c82.svg\")
Step 1: calculate hidden OR preactivation.
","![\"z_1=0+1-0.5=0.5\"](\"assets/equations/90f88a492c8d4d66.svg\")
Step 2: apply its threshold.
","![\"h_1=1\"](\"assets/equations/6025713e0c4b56ee.svg\")
Step 3: calculate hidden AND preactivation.
","![\"z_2=0+1-1.5=-0.5\"](\"assets/equations/60ef65c2ea0ecf5f.svg\")
Step 4: apply its threshold.
","Step 5: calculate XOR output preactivation.
","![\"z_{XOR}=1-2(0)-0.5=0.5\"](\"assets/equations/6ee6edf14f171730.svg\")
Step 6: apply the XOR threshold.
","![\"y_{XOR}=1\"](\"assets/equations/43f7e4fce3fb5771.svg\")
Step 7: calculate XNOR output preactivation.
","![\"z_{XNOR}=-1+2(0)+0.5=-0.5\"](\"assets/equations/fbcbb7c54a55ba6f.svg\")
Step 8: apply the XNOR threshold.
","![\"y_{XNOR}=0\"](\"assets/equations/51f578d6708881af.svg\")
Step 1: calculate hidden OR preactivation.
","![\"z_1=1+0-0.5=0.5\"](\"assets/equations/318ee2a6640d10f0.svg\")
Step 2: apply its threshold.
","Step 3: calculate hidden AND preactivation.
","![\"z_2=1+0-1.5=-0.5\"](\"assets/equations/6d1cd15a27d9572e.svg\")
Step 4: apply its threshold.
","Step 5: calculate XOR output preactivation.
","Step 6: apply the XOR threshold.
","Step 7: calculate XNOR output preactivation.
","Step 8: apply the XNOR threshold.
","Step 1: calculate hidden OR preactivation.
","![\"z_1=1+1-0.5=1.5\"](\"assets/equations/285947e47dfebe51.svg\")
Step 2: apply its threshold.
","Step 3: calculate hidden AND preactivation.
","![\"z_2=1+1-1.5=0.5\"](\"assets/equations/86c5a45ae836809c.svg\")
Step 4: apply its threshold.
","![\"h_2=1\"](\"assets/equations/397f5515899ffae5.svg\")
Step 5: calculate XOR output preactivation.
","![\"z_{XOR}=1-2(1)-0.5=-1.5\"](\"assets/equations/dd1914e888d3b3fb.svg\")
Step 6: apply the XOR threshold.
","Step 7: calculate XNOR output preactivation.
","![\"z_{XNOR}=-1+2(1)+0.5=1.5\"](\"assets/equations/7ce164dbb1fd1549.svg\")
Step 8: apply the XNOR threshold.
","Final answer: XOR = [0,1,1,0]; XNOR = [1,0,0,1] for 00,01,10,11. The hidden nonlinear units make this possible.
"],"kind":"practice","id":"tee-c13-s11","search":"F23 Q5b **Question (10 marks).** Explain why a single perceptron cannot classify nonlinearly separable data. Implement XOR and XNOR using suitable perceptrons.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nRequired logic functions = XOR and XNOR on two binary inputs.\n\nWeights and biases are design choices, not given constants.\n\n### To find\n\nExplain the failure of a single linear separator, choose suitable hidden/output perceptrons and show all four truth-table rows.\n\n### Formula — which one and why\n\n**Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.\n\nstep(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao.\n\n### Solution — step by step\n\n\n\n\nA single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator.\n\nFor binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1.\nFor XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR.\nA multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map.\n\n\n### Verify input (0,0)\n\n**Step 1: calculate hidden OR preactivation.**\n\n\n\n\n\n\n\n**Step 2: apply its threshold.**\n\n\n\n\n\n\n\n**Step 3: calculate hidden AND preactivation.**\n\n\n\n\n\n\n\n**Step 4: apply its threshold.**\n\n\n\n\n\n\n\n**Step 5: calculate XOR output preactivation.**\n\n\n\n\n\n\n\n**Step 6: apply the XOR threshold.**\n\n\n\n\n\n\n\n**Step 7: calculate XNOR output preactivation.**\n\n\n\n\n\n\n\n**Step 8: apply the XNOR threshold.**\n\n\n\n\n\n\n\n\n### Verify input (0,1)\n\n**Step 1: calculate hidden OR preactivation.**\n\n\n\n\n\n\n\n**Step 2: apply its threshold.**\n\n\n\n\n\n\n\n**Step 3: calculate hidden AND preactivation.**\n\n\n\n\n\n\n\n**Step 4: apply its threshold.**\n\n\n\n\n\n\n\n**Step 5: calculate XOR output preactivation.**\n\n\n\n\n\n\n\n**Step 6: apply the XOR threshold.**\n\n\n\n\n\n\n\n**Step 7: calculate XNOR output preactivation.**\n\n\n\n\n\n\n\n**Step 8: apply the XNOR threshold.**\n\n\n\n\n\n\n\n\n### Verify input (1,0)\n\n**Step 1: calculate hidden OR preactivation.**\n\n\n\n\n\n\n\n**Step 2: apply its threshold.**\n\n\n\n\n\n\n\n**Step 3: calculate hidden AND preactivation.**\n\n\n\n\n\n\n\n**Step 4: apply its threshold.**\n\n\n\n\n\n\n\n**Step 5: calculate XOR output preactivation.**\n\n\n\n\n\n\n\n**Step 6: apply the XOR threshold.**\n\n\n\n\n\n\n\n**Step 7: calculate XNOR output preactivation.**\n\n\n\n\n\n\n\n**Step 8: apply the XNOR threshold.**\n\n\n\n\n\n\n\n\n### Verify input (1,1)\n\n**Step 1: calculate hidden OR preactivation.**\n\n\n\n\n\n\n\n**Step 2: apply its threshold.**\n\n\n\n\n\n\n\n**Step 3: calculate hidden AND preactivation.**\n\n\n\n\n\n\n\n**Step 4: apply its threshold.**\n\n\n\n\n\n\n\n**Step 5: calculate XOR output preactivation.**\n\n\n\n\n\n\n\n**Step 6: apply the XOR threshold.**\n\n\n\n\n\n\n\n**Step 7: calculate XNOR output preactivation.**\n\n\n\n\n\n\n\n**Step 8: apply the XNOR "},{"title":"F23 Q6a","question":["Question (10 marks). For the pictured LSTM time step, explain the forget, “update” and output gates, develop “update and reset” equations, and discuss long sequence lengths. The source mixes LSTM and GRU terminology.
"],"answer":["Terminology correction: the pictured LSTM uses forget, input and output gates plus a candidate. Its input/candidate path updates the cell. A reset gate belongs to GRU, so no genuine LSTM reset-gate equation exists. The complete consistent LSTM answer is:
","An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.
","The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.
","In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.
","For the explicitly named GRU terms in the question:
","A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:
","The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.
","Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.
","In an exam, state the mismatch in one sentence, label each architecture clearly and then give its correct equations. Do not memorize a fictitious reset gate for LSTM.
"],"kind":"practice","id":"tee-c13-s12","search":"F23 Q6a **Question (10 marks).** For the pictured LSTM time step, explain the forget, “update” and output gates, develop “update and reset” equations, and discuss long sequence lengths. The source mixes LSTM and GRU terminology.\n\n**Complete worked answer.**\n**Terminology correction: ** the pictured LSTM uses forget, input and output gates plus a candidate. Its input/candidate path updates the cell. A reset gate belongs to GRU, so no genuine LSTM reset-gate equation exists. The complete consistent LSTM answer is:\nAn LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.\n\n\n\n\nThe forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.\nIn language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.\n**For the explicitly named GRU terms in the question: **\nA GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:\n\n\n\nThe reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.\nCompared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.\nIn an exam, state the mismatch in one sentence, label each architecture clearly and then give its correct equations. Do not memorize a fictitious reset gate for LSTM.\n"},{"title":"F23 Q6b","question":["Question (10 marks). Explain how ResNet addresses vanishing gradients. Compare a block output with and without a residual connection.
"],"answer":["A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.
","The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.
","Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].
"],"kind":"practice","id":"tee-c13-s13","search":"F23 Q6b **Question (10 marks).** Explain how ResNet addresses vanishing gradients. Compare a block output with and without a residual connection.\n\n**Complete worked answer.**\nA residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.\n\n\nThe identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.\nElementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].\n"},{"title":"F23 Q7a","question":["Question (10 marks). Design and sketch a sentiment-analysis model whose input is a small paragraph. Explain how its output is generated.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","Concrete many-to-one design: tokenize paragraph → embeddings → LSTM sequence processor → final hidden state → dense 3 → softmax for positive/neutral/negative. If the task specifies only two labels, use one sigmoid or two softmax outputs instead. Train with the matching cross-entropy. Keep token order, handle padding/masks consistently and evaluate held-out reviews.
","Question (10 marks). Explain the bias-variance tradeoff. Fill effects on bias and variance for weight regularization, more hidden units, dropout and more training data from the same distribution.
"],"answer":["Bias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data.
","Increasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws.
","Underfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate.
","Training and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting.
","| Change | Typical bias | Typical variance |
|---|---|---|
| Stronger weight regularization | Increases | Decreases |
| More hidden units | Decreases | Increases |
| Dropout during training | Increases | Decreases |
| More same-distribution training data | Approximately no change | Decreases |
These are the conventional exam trends, assuming other conditions are comparable. They are not guarantees for every training run.
"],"kind":"practice","id":"tee-c13-s15","search":"F23 Q7b **Question (10 marks).** Explain the bias-variance tradeoff. Fill effects on bias and variance for weight regularization, more hidden units, dropout and more training data from the same distribution.\n\n**Complete worked answer.**\nBias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data.\n\nIncreasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws.\nUnderfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate.\nTraining and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting.\n| Change | Typical bias | Typical variance |\n| Stronger weight regularization | Increases | Decreases |\n| More hidden units | Decreases | Increases |\n| Dropout during training | Increases | Decreases |\n| More same-distribution training data | Approximately no change | Decreases |\nThese are the conventional exam trends, assuming other conditions are comparable. They are not guarantees for every training run.\n\n\n"}]},{"n":14,"title":"Re examination February 2024 fully solved","sections":[{"title":"Start here","blocks":["R24 = S14, 13 February 2024. Its source matrix uses an input-to-hidden indexing convention, explicitly handled below.
","Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible.
"],"kind":"read","id":"tee-c14-intro"},{"title":"R24 Q1a","question":["Question (5 marks). What factors drive widespread adoption of deep learning?
"],"answer":["Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.
","| Comparison | Traditional ML approaches | Deep learning approaches |
|---|---|---|
| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |
| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |
| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |
| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |
These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.
","Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.
"],"kind":"practice","id":"tee-c14-s0","search":"R24 Q1a **Question (5 marks).** What factors drive widespread adoption of deep learning?\n\n**Complete worked answer.**\nDeep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.\n| Comparison | Traditional ML approaches | Deep learning approaches |\n| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |\n| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |\n| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |\n| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |\nThese are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.\nDrivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.\n"},{"title":"R24 Q1b","question":["Question (5 marks). Draw an animal-classification network with 10 inputs, three hidden layers of 5 neurons each and 4 output classes. Every hidden/output neuron has a bias. Count all trainable parameters.
"],"answer":["10 input features → 5 hidden → 5 hidden → 5 hidden → 4 output classes.
","One bias per hidden/output neuron.
","Draw the layer structure and count weights and biases.
","Dense-layer parameters. Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.
","n_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.
","Yaad rakhne ka cue: Har output ke liye saare inputs plus ek bias.
","Architecture: 10 → 5 → 5 → 5 → 4. Each non-input neuron has one bias.
","Step 1: count connections.
","![\"10(5)=50\"](\"assets/equations/6bd3c4c12c1c454b.svg\")
Step 2: count receiving-neuron biases.
","There are 5 receiving neurons, so 5 biases.
","Step 3: add weights and biases.
","![\"50+5=55\"](\"assets/equations/cacbd65e2c06f4f4.svg\")
Step 1: count connections.
","Step 2: count receiving-neuron biases.
","There are 5 receiving neurons, so 5 biases.
","Step 3: add weights and biases.
","Step 1: count connections.
","Step 2: count receiving-neuron biases.
","There are 5 receiving neurons, so 5 biases.
","Step 3: add weights and biases.
","Step 1: count connections.
","Step 2: count receiving-neuron biases.
","There are 4 receiving neurons, so 4 biases.
","Step 3: add weights and biases.
","Total = 139. Input values and activations are not trainable parameters.
","Use hidden nonlinearities such as ReLU and a 4-class softmax. Learned parameters are the connections and biases; the number of layers and units are design choices.
"],"kind":"practice","id":"tee-c14-s1","search":"R24 Q1b **Question (5 marks).** Draw an animal-classification network with 10 inputs, three hidden layers of 5 neurons each and 4 output classes. Every hidden/output neuron has a bias. Count all trainable parameters.\n\n**Complete worked answer.**\n\n\n\n### Given\n\n10 input features → 5 hidden → 5 hidden → 5 hidden → 4 output classes.\n\nOne bias per hidden/output neuron.\n\n### To find\n\nDraw the layer structure and count weights and biases.\n\n### Formula — which one and why\n\n**Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.\n\nn_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.\n\n\n\n**Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias.\n\n### Solution — step by step\n\n\n\n\n\n\nArchitecture: 10 → 5 → 5 → 5 → 4. Each non-input neuron has one bias.\n\n### Layer 10 → 5\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 5 receiving neurons, so 5 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 5 → 5\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 5 receiving neurons, so 5 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 5 → 5\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 5 receiving neurons, so 5 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 5 → 4\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 4 receiving neurons, so 4 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n**Total = 139**. Input values and activations are not trainable parameters.\nUse hidden nonlinearities such as ReLU and a 4-class softmax. Learned parameters are the connections and biases; the number of layers and units are design choices.\n"},{"title":"R24 Q1c","question":["Question (5 marks). A deep network strongly overfits its training set and performs poorly on unseen data. Explain remedies.
"],"answer":["Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
","Use a representative validation set to choose the remedy. Also inspect leakage, label quality and distribution mismatch. If the final test set was repeatedly used to tune changes, reserve a fresh untouched set for final reporting.
"],"kind":"practice","id":"tee-c14-s2","search":"R24 Q1c **Question (5 marks).** A deep network strongly overfits its training set and performs poorly on unseen data. Explain remedies.\n\n**Complete worked answer.**\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\nUse a representative validation set to choose the remedy. Also inspect leakage, label quality and distribution mismatch. If the final test set was repeatedly used to tune changes, reserve a fresh untouched set for final reporting.\n"},{"title":"R24 Q1d","question":["Question (5 marks). For feature map Matrix 1 shown below, find max and average pooling with 2 × 2 windows, stride 2.
","For feature map Matrix 1 shown below, find max and average pooling with 2 × 2 windows, stride 2.
","Find the requested pooled output(s).
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","Shape: = 2 on each axis; channels stay unchanged.
Step 1: select the entries.
","[21, 11, 12, 9]
","Step 2: add all selected entries.
","21 + 11 + 12 + 9 = 53
","Step 3: divide by the 4 entries in this window.
","53/4 = 13.25. Maximum = 21
","Step 1: select the entries.
","[15, 10, 10, 11]
","Step 2: add all selected entries.
","15 + 10 + 10 + 11 = 46
","Step 3: divide by the 4 entries in this window.
","46/4 = 11.5. Maximum = 15
","Step 1: select the entries.
","[11, 13, 7, 4]
","Step 2: add all selected entries.
","11 + 13 + 7 + 4 = 35
","Step 3: divide by the 4 entries in this window.
","35/4 = 8.75. Maximum = 13
","Step 1: select the entries.
","[18, 12, 15, 12]
","Step 2: add all selected entries.
","18 + 12 + 15 + 12 = 57
","Step 3: divide by the 4 entries in this window.
","57/4 = 14.25. Maximum = 18
","Average output: . Max output: . Pooling normally has zero trainable parameters and reduces spatial dimensions.
Question (10 marks). A 4 → 4 → 1 sigmoid network has input [.4,.3,.5,−.4]. The printed W₁ is shown below, W₂ = [.5,.7,−.6,.2]; hidden bias 0.6, output bias 0.2, target 0.9, η = .5. Compute mean squared error and update the connection h₁ → output. In the graph θ ij connects input i to hidden j.
","A 4 → 4 → 1 sigmoid network has input [.4,.3,.5,−.4].
","The printed W₁ is printed with the question, W₂ = [.5,.7,−.6,.2].
","hidden bias 0.6, output bias 0.2, target 0.9, η = .5.
","Find all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Mean squared loss. Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean.
","n = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target.
","![\"L=\\frac1n\\sum_j(o_j-y_j)^2\"](\"assets/equations/f5544f3d0af79c26.svg\")
![\"\\frac{\\partial](\"assets/equations/3b0ae494a9901408.svg\")
Yaad rakhne ka cue: MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Orientation: the source rows list sending inputs, so use each COLUMN to form a receiving hidden neuron. In our usual column-input convention this means using the transpose of printed W₁. The loss is one-output MSE = (o−0.9)², so its derivative includes 2.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","![\"z_{h_1}](\"assets/equations/ee9cf86ec848950c.svg\")
![\"h_{1}=\\frac{1}{1+e^{-(0.9)}}=0.710949503\"](\"assets/equations/f76f739a80a9361b.svg\")
Hidden neuron 2:
","![\"z_{h_2}](\"assets/equations/13efea50d945325a.svg\")
![\"h_{2}=\\frac{1}{1+e^{-(0.7)}}=0.668187772\"](\"assets/equations/d9473d1a5522dcc7.svg\")
Hidden neuron 3:
","![\"z_{h_3}](\"assets/equations/cbf6b0463b666856.svg\")
![\"h_{3}=\\frac{1}{1+e^{-(0.56)}}=0.63645254\"](\"assets/equations/0b006fd2ae332459.svg\")
Hidden neuron 4:
","![\"z_{h_4}](\"assets/equations/7e4d728bbba36941.svg\")
![\"h_{4}=\\frac{1}{1+e^{-(0.6)}}=0.645656306\"](\"assets/equations/3e375a5e30dabd6f.svg\")
Output neuron 1:
","![\"\\begin{aligned}z_{o_1}={}&0.5(0.710949503)+0.7(0.668187772)\\\\&-0.6(0.63645254)+0.2(0.645656306)+0.2\\\\={}&0.770465929\\end{aligned}\"](\"assets/equations/b82d291c09c8255f.svg\")
![\"o_{1}=\\frac{1}{1+e^{-(0.770465929)}}=0.683621675\"](\"assets/equations/5ce3c8d47df25f40.svg\")
Step 2 loss convention. Use the explicitly requested one-output MSE, not half squared loss.
","Prediction minus target = [-0.216378325]. Squared errors = [0.04681958]. Half-sum squared loss = 0.02340979; mean squared error = 0.04681958. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","![\"\\delta_{o_1}](\"assets/equations/7665056831c42066.svg\")
The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.033271705. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,1}^{(2)}=(-0.093597941)\\times(0.710949503)\\approx](\"assets/equations/a4c5e0af98c4e800.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,1,\\mathrm{new}}=0.5-0.5(-0.06654341)\\approx](\"assets/equations/d5193daabf661dea.svg\")
h2 → o1:
","Subtract η × gradient = −0.0312705. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,2}^{(2)}=(-0.093597941)\\times(0.668187772)\\approx](\"assets/equations/fc6c904cdf89e2ca.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,2,\\mathrm{new}}=0.7-0.5(-0.062541)\\approx](\"assets/equations/a56876d47ec41774.svg\")
h3 → o1:
","Subtract η × gradient = −0.029785324. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,3}^{(2)}=\\frac{\\partial](\"assets/equations/85e8a967a0a12f92.svg\")
![\"g_{1,3}^{(2)}=(-0.093597941)\\times(0.63645254)\\approx](\"assets/equations/bc44ca9541b76f51.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,3,\\mathrm{new}}=-0.6-0.5(-0.059570648)\\approx](\"assets/equations/30d92749f17fd5d6.svg\")
h4 → o1:
","Subtract η × gradient = −0.03021605. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","![\"g_{1,4}^{(2)}=\\frac{\\partial](\"assets/equations/62648b24044cc2ea.svg\")
![\"g_{1,4}^{(2)}=(-0.093597941)\\times(0.645656306)\\approx](\"assets/equations/95d6468f8fc70889.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(2)}_{1,4,\\mathrm{new}}=0.2-0.5(-0.060432101)\\approx](\"assets/equations/09adce4276bdd712.svg\")
Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.5)\\times(-0.093597941)\\approx](\"assets/equations/e1dda3f8681ba484.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/f1f4d5705234aba2.svg\")
![\"\\delta_{h_1}](\"assets/equations/81cfb49d2e12d4ca.svg\")
Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.7)\\times(-0.093597941)\\approx](\"assets/equations/4d3e0ecf8502df13.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/893a1456a543073d.svg\")
![\"\\delta_{h_2}](\"assets/equations/26b1106bf05e76a6.svg\")
Hidden 3 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(-0.6)\\times(-0.093597941)\\approx](\"assets/equations/44316b3d49dde0a3.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/b369b4600b5c9b60.svg\")
![\"\\delta_{h_3}](\"assets/equations/43d6f061bf15172b.svg\")
Hidden 4 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","![\"C_1=(0.2)\\times(-0.093597941)\\approx](\"assets/equations/0303b36ec5b8423b.svg\")
Accumulate the contributions, then apply the hidden activation derivative below.
","![\"\\text{Sum}=C_1\\approx](\"assets/equations/4daca8885b8234c0.svg\")
![\"\\delta_{h_4}](\"assets/equations/b8fde02c1bff3e4f.svg\")
Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = −0.00192344. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,1}^{(1)}=(-0.009617203)\\times(0.4)\\approx](\"assets/equations/1ec98954f2916f01.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,1,\\mathrm{new}}=0.8-0.5(-0.003846881)\\approx](\"assets/equations/c75e592a0b5226d2.svg\")
x2 → h1:
","Subtract η × gradient = −0.00144258. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,2}^{(1)}=(-0.009617203)\\times(0.3)\\approx](\"assets/equations/3ad09496c853dfe2.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,2,\\mathrm{new}}=-0.5-0.5(-0.002885161)\\approx](\"assets/equations/8044a6006ab0ca77.svg\")
x3 → h1:
","Subtract η × gradient = −0.0024043. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,3}^{(1)}=(-0.009617203)\\times(0.5)\\approx](\"assets/equations/053d52dc796832a9.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,3,\\mathrm{new}}=0.1-0.5(-0.004808601)\\approx](\"assets/equations/86d9f58aa097fd71.svg\")
x4 → h1:
","Subtract η × gradient = 0.00192344. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{1,4}^{(1)}=\\frac{\\partial](\"assets/equations/51881657a5cb91ef.svg\")
![\"g_{1,4}^{(1)}=(-0.009617203)\\times(-0.4)\\approx](\"assets/equations/a4af21673941e601.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{1,4,\\mathrm{new}}=-0.2-0.5(0.003846881)\\approx](\"assets/equations/bdc8b7088a5ab57a.svg\")
x1 → h2:
","Subtract η × gradient = −0.002905262. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,1}^{(1)}=(-0.014526308)\\times(0.4)\\approx](\"assets/equations/afb5bb784d05f461.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,1,\\mathrm{new}}=0.2-0.5(-0.005810523)\\approx](\"assets/equations/d7bc835c6d4c639d.svg\")
x2 → h2:
","Subtract η × gradient = −0.002178946. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,2}^{(1)}=(-0.014526308)\\times(0.3)\\approx](\"assets/equations/7674dd635ddd9ccc.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,2,\\mathrm{new}}=0.4-0.5(-0.004357892)\\approx](\"assets/equations/3acb31488ce3b963.svg\")
x3 → h2:
","Subtract η × gradient = −0.003631577. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,3}^{(1)}=(-0.014526308)\\times(0.5)\\approx](\"assets/equations/5d328fe04a37cfc6.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,3,\\mathrm{new}}=0.2-0.5(-0.007263154)\\approx](\"assets/equations/5eeed9f16670c55b.svg\")
x4 → h2:
","Subtract η × gradient = 0.002905262. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{2,4}^{(1)}=\\frac{\\partial](\"assets/equations/c111f40d015517f1.svg\")
![\"g_{2,4}^{(1)}=(-0.014526308)\\times(-0.4)\\approx](\"assets/equations/e120af498da6af3d.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{2,4,\\mathrm{new}}=0.5-0.5(0.005810523)\\approx](\"assets/equations/901765ceedbe9376.svg\")
x1 → h3:
","Subtract η × gradient = 0.002598811. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{3,1}^{(1)}=\\frac{\\partial](\"assets/equations/c647740724db2342.svg\")
![\"g_{3,1}^{(1)}=(0.012994055)\\times(0.4)\\approx](\"assets/equations/618f330eb0013565.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{3,1,\\mathrm{new}}=0.3-0.5(0.005197622)\\approx](\"assets/equations/f7218a8971ce6230.svg\")
x2 → h3:
","Subtract η × gradient = 0.001949108. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{3,2}^{(1)}=\\frac{\\partial](\"assets/equations/11a3a8fd1c29457a.svg\")
![\"g_{3,2}^{(1)}=(0.012994055)\\times(0.3)\\approx](\"assets/equations/96944db9c899fc7e.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{3,2,\\mathrm{new}}=0.3-0.5(0.003898216)\\approx](\"assets/equations/87125a5db71e09ef.svg\")
x3 → h3:
","Subtract η × gradient = 0.003248514. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{3,3}^{(1)}=\\frac{\\partial](\"assets/equations/ec47dc9f5403d781.svg\")
![\"g_{3,3}^{(1)}=(0.012994055)\\times(0.5)\\approx](\"assets/equations/9d6717c29bdbe2b4.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{3,3,\\mathrm{new}}=-0.1-0.5(0.006497027)\\approx](\"assets/equations/80c1a9c61f216679.svg\")
x4 → h3:
","Subtract η × gradient = −0.002598811. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{3,4}^{(1)}=\\frac{\\partial](\"assets/equations/4791ee899924c031.svg\")
![\"g_{3,4}^{(1)}=(0.012994055)\\times(-0.4)\\approx](\"assets/equations/406ff2da076d59bc.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{3,4,\\mathrm{new}}=0.5-0.5(-0.005197622)\\approx](\"assets/equations/85589f5b6f5c0202.svg\")
x1 → h4:
","Subtract η × gradient = −0.00085655. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{4,1}^{(1)}=\\frac{\\partial](\"assets/equations/e5b6b4e6329ece21.svg\")
![\"g_{4,1}^{(1)}=(-0.004282747)\\times(0.4)\\approx](\"assets/equations/313f27074af89747.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{4,1,\\mathrm{new}}=0.4-0.5(-0.001713099)\\approx](\"assets/equations/15f2ab91f5697a46.svg\")
x2 → h4:
","Subtract η × gradient = −0.000642412. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{4,2}^{(1)}=\\frac{\\partial](\"assets/equations/4e7cc212944346db.svg\")
![\"g_{4,2}^{(1)}=(-0.004282747)\\times(0.3)\\approx](\"assets/equations/b3de47e4aca426e5.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{4,2,\\mathrm{new}}=0.2-0.5(-0.001284824)\\approx](\"assets/equations/6e488fc21bed8f11.svg\")
x3 → h4:
","Subtract η × gradient = −0.001070686. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{4,3}^{(1)}=\\frac{\\partial](\"assets/equations/f10479c14422cd43.svg\")
![\"g_{4,3}^{(1)}=(-0.004282747)\\times(0.5)\\approx](\"assets/equations/2192ca5fcdf60ef8.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{4,3,\\mathrm{new}}=0.2-0.5(-0.002141373)\\approx](\"assets/equations/5bb6f3c8fa38e377.svg\")
x4 → h4:
","Subtract η × gradient = 0.00085655. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","![\"g_{4,4}^{(1)}=\\frac{\\partial](\"assets/equations/54a191df99677201.svg\")
![\"g_{4,4}^{(1)}=(-0.004282747)\\times(-0.4)\\approx](\"assets/equations/f326d58085d05315.svg\")
Apply the learning rate to this gradient and subtract the result from the old weight.
","![\"W^{(1)}_{4,4,\\mathrm{new}}=0.8-0.5(0.001713099)\\approx](\"assets/equations/b007fcf05f829e6c.svg\")
Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","![\"b_{h_1,\\mathrm{new}}=0.6-0.5(-0.009617203)=0.604808601\"](\"assets/equations/5d88aea5b1073764.svg\")
![\"b_{h_2,\\mathrm{new}}=0.6-0.5(-0.014526308)=0.607263154\"](\"assets/equations/758b03c4be06d181.svg\")
![\"b_{h_3,\\mathrm{new}}=0.6-0.5(0.012994055)=0.593502972\"](\"assets/equations/01910665e2cb2391.svg\")
![\"b_{h_4,\\mathrm{new}}=0.6-0.5(-0.004282747)=0.602141373\"](\"assets/equations/f3f5c6e854f64acf.svg\")
![\"b_{o_1,\\mathrm{new}}=0.2-0.5(-0.093597941)=0.246798971\"](\"assets/equations/75d8de25e1127fac.svg\")
These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
","Required result: output 0.683621675, MSE 0.046819580, h₁ → output new weight 0.533271705. If instead each printed row is assumed to receive inputs, the conditional alternative is output 0.681090691, MSE 0.047921285 and updated weight 0.534478309. Explain the arrow convention rather than mixing these results.
"],"kind":"practice","id":"tee-c14-s4","search":"R24 Q2a **Question (10 marks).** A 4 → 4 → 1 sigmoid network has input [.4,.3,.5,−.4]. The printed W₁ is shown below, W₂ = [.5,.7,−.6,.2]; hidden bias 0.6, output bias 0.2, target 0.9, η = .5. Compute mean squared error and update the connection h₁ → output. In the graph θ ij connects input i to hidden j.\n\n\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nA 4 → 4 → 1 sigmoid network has input [.4,.3,.5,−.4].\n\nThe printed W₁ is printed with the question, W₂ = [.5,.7,−.6,.2].\n\nhidden bias 0.6, output bias 0.2, target 0.9, η = .5.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Mean squared loss.** Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean.\n\nn = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target.\n\n\n\n\n\n**Yaad rakhne ka cue:** MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Orientation: ** the source rows list sending inputs, so use each COLUMN to form a receiving hidden neuron. In our usual column-input convention this means using the transpose of printed W₁. The loss is one-output MSE = (o−0.9)², so its derivative includes 2.\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Hidden neuron 3: **\n\n\n**Hidden neuron 4: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** Use the explicitly requested one-output MSE, not half squared loss.\nPrediction minus target = [-0.216378325]. Squared errors = [0.04681958]. Half-sum squared loss = 0.02340979; mean squared error = 0.04681958. These are different unless the number of output components makes the scale coincide.\n**Output delta 1:**\n\nThe derivative is o(1−o).\n**Step 3 output weights.** Gradient = receiving delta × send"},{"title":"R24 Q2b","question":["Question. Compare standard momentum and Nesterov accelerated gradient using their gradient-evaluation points and update equations. Explain what may happen when momentum is extremely close to 1, for example 0.9999.
","Syllabus-aligned extract: source R24 Q2b. The current-syllabus optimizer comparison and high-momentum discussion are retained; marks for this extract are not separately stated.
"],"answer":["Use signed velocity. Standard momentum evaluates g at current θ: vnew = βv−ηg(θ), θ new = θ+vnew. Nesterov evaluates at a look-ahead point: θ look = θ+βv; vnew = βv−ηg(θ look); θ new = θ+vnew. Thus the gradient can correct a direction after anticipating part of the momentum move. Nesterov often reduces overshoot in suitable settings, but it is not universally faster for every neural network.
","With β = 0.9999, old velocity decays very slowly. The method can respond sluggishly to direction changes, overshoot or oscillate, particularly with a large learning rate. An approximate memory scale 1/(1−β) = 10,000 steps illustrates the long persistence; it is not an exact convergence time.
"],"kind":"practice","id":"tee-c14-s5","search":"R24 Q2b **Question.** Compare standard momentum and Nesterov accelerated gradient using their gradient-evaluation points and update equations. Explain what may happen when momentum is extremely close to 1, for example 0.9999.\n\n**Syllabus-aligned extract:** source R24 Q2b. The current-syllabus optimizer comparison and high-momentum discussion are retained; marks for this extract are not separately stated.\n\n**Complete worked answer.**\nUse signed velocity. Standard momentum evaluates g at current θ: vnew = βv−ηg(θ), θ new = θ+vnew. Nesterov evaluates at a look-ahead point: θ look = θ+βv; vnew = βv−ηg(θ look); θ new = θ+vnew. Thus the gradient can correct a direction after anticipating part of the momentum move. Nesterov often reduces overshoot in suitable settings, but it is not universally faster for every neural network.\nWith β = 0.9999, old velocity decays very slowly. The method can respond sluggishly to direction changes, overshoot or oscillate, particularly with a large learning rate. An approximate memory scale 1/(1−β) = 10,000 steps illustrates the long persistence; it is not an exact convergence time.\n"},{"title":"R24 Q3a","question":["Question (10 marks). Differentiate sigmoid and ReLU(4). Explain why sigmoid/tanh are often avoided in deep hidden layers(3). Compute softmax of [2.1,4.8,3.5](3).
"],"answer":["Softmax scores = [2.1,4.8,3.5].
","The numerals (4),(3),(3) in the prompt are marks allocations, not activation inputs.
","Differentiate sigmoid/ReLU, explain hidden-layer saturation and calculate softmax.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.
","zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.
","Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.
","Sigmoid
","Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.
","Hyperbolic tangent
","The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.
","ReLU
","The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.
","Leaky ReLU
","Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.
","Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.
","Step 1 stabilize: subtract the largest logit 4.8 from every value: [-2.7, 0, -1.3]. This does not change softmax.
","Step 2 exponentiate: exp values = [0.067205513, 1, 0.272531793].
","Step 3 denominator: sum = 1.339737306.
","Step 4 divide each value by this same sum: probabilities = [0.050163202, 0.746414984, 0.203421814]. Check: all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.
"],"kind":"practice","id":"tee-c14-s6","search":"R24 Q3a **Question (10 marks).** Differentiate sigmoid and ReLU(4). Explain why sigmoid/tanh are often avoided in deep hidden layers(3). Compute softmax of [2.1,4.8,3.5](3).\n\n**Complete worked answer.**\n\n\n\n### Given\n\nSoftmax scores = [2.1,4.8,3.5].\n\nThe numerals (4),(3),(3) in the prompt are marks allocations, not activation inputs.\n\n### To find\n\nDifferentiate sigmoid/ReLU, explain hidden-layer saturation and calculate softmax.\n\n### Formula — which one and why\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.\n\nzⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\nAn activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.\n**Sigmoid**\n\nHere z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.\n**Hyperbolic tangent**\n\nThe numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.\n**ReLU**\n\nThe output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.\n**Leaky ReLU**\n\nAlpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.\nSigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.\n**Step 1 stabilize: ** subtract the largest logit 4.8 from every value: [-2.7, 0, -1"},{"title":"R24 Q3b","question":["Question (10 marks). Explain why regularization matters, list techniques and elaborate one.
"],"answer":["Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
","Elaborating L2: add(λ/2)Σw² to the data loss. For weight w = .5, data gradient.2, λ = .1, η = .1, total gradient = .2+.1 × .5 = .25, so wnew = .5−.1 × .25 = .475. Relative to the data-only update.48, the penalty pulls the weight further toward 0. Too large λ can remove useful signal, so tune on validation data.
"],"kind":"practice","id":"tee-c14-s7","search":"R24 Q3b **Question (10 marks).** Explain why regularization matters, list techniques and elaborate one.\n\n**Complete worked answer.**\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\n**Elaborating L2: ** add(λ/2)Σw² to the data loss. For weight w = .5, data gradient.2, λ = .1, η = .1, total gradient = .2+.1 × .5 = .25, so wnew = .5−.1 × .25 = .475. Relative to the data-only update.48, the penalty pulls the weight further toward 0. Too large λ can remove useful signal, so tune on validation data.\n"},{"title":"R24 Q4a","question":["Question (10 marks). Train a perceptron for 2 epochs with η = 1, initial weights and bias 0, threshold 0. Samples in order: [1,1,1,1] → 1; [−1,1,−1,−1] → 1; [1,1,1,−1] → −1; [1,−1,−1,1] → −1. Bias input is 1.
"],"answer":["Train a perceptron for 2 epochs with η = 1, initial weights and bias 0, threshold 0.
","Samples in order: [1,1,1,1] → 1.
","[−1,1,−1,−1] → 1.
","[1,1,1,−1] → −1.
","[1,−1,−1,1] → −1.
","Bias input is 1.
","Process samples in order for the stated epochs and report the final weights and bias.
","Target-based perceptron correction. This worked convention changes parameters only on a wrong prediction. Use the target itself as the signed multiplier; it differs from the binary error rule.
","t is the bipolar target; η is the learning rate. Output is +1 above zero, −1 below zero and 0 at exactly zero.
","Yaad rakhne ka cue: Is convention mein galat hua toh target × input se correction.
","State the threshold rule. Use output +1 if z>0, −1 if z<0, and 0 at z = 0. On a mismatch use Δw = η tx and Δb = ηt; on a match make no update. η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly.
","Step 1: copy this sample and target.
","x = [1, 1, 1, 1], target = 1.
","Step 2: calculate the current net.
","z = (0 × 1) + (0 × 1) + (0 × 1) + (0 × 1) + (0) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = 1
","Step 5: calculate all weight changes.
","Δw = 1 × [1, 1, 1, 1] = [1, 1, 1, 1]
","Step 6: calculate the bias change.
","Δb = 1.
","Step 7: carry the updated parameters forward.
","New w = [1, 1, 1, 1], new b = 1.
","Step 1: copy this sample and target.
","x = [-1, 1, -1, -1], target = 1.
","Step 2: calculate the current net.
","z = (1 × -1) + (1 × 1) + (1 × -1) + (1 × -1) + (1) = -1
","Step 3: apply the threshold.
","output = -1.
","Step 4: calculate the update multiplier.
","Update multiplier = 1
","Step 5: calculate all weight changes.
","Δw = 1 × [-1, 1, -1, -1] = [-1, 1, -1, -1]
","Step 6: calculate the bias change.
","Δb = 1.
","Step 7: carry the updated parameters forward.
","New w = [0, 2, 0, 0], new b = 2.
","Step 1: copy this sample and target.
","x = [1, 1, 1, -1], target = -1.
","Step 2: calculate the current net.
","z = (0 × 1) + (2 × 1) + (0 × 1) + (0 × -1) + (2) = 4
","Step 3: apply the threshold.
","output = 1.
","Step 4: calculate the update multiplier.
","Update multiplier = -1
","Step 5: calculate all weight changes.
","Δw = -1 × [1, 1, 1, -1] = [-1, -1, -1, 1]
","Step 6: calculate the bias change.
","Δb = -1.
","Step 7: carry the updated parameters forward.
","New w = [-1, 1, -1, 1], new b = 1.
","Step 1: copy this sample and target.
","x = [1, -1, -1, 1], target = -1.
","Step 2: calculate the current net.
","z = (-1 × 1) + (1 × -1) + (-1 × -1) + (1 × 1) + (1) = 1
","Step 3: apply the threshold.
","output = 1.
","Step 4: calculate the update multiplier.
","Update multiplier = -1
","Step 5: calculate all weight changes.
","Δw = -1 × [1, -1, -1, 1] = [-1, 1, 1, -1]
","Step 6: calculate the bias change.
","Δb = -1.
","Step 7: carry the updated parameters forward.
","New w = [-2, 2, 0, 0], new b = 0.
","Step 1: copy this sample and target.
","x = [1, 1, 1, 1], target = 1.
","Step 2: calculate the current net.
","z = (-2 × 1) + (2 × 1) + (0 × 1) + (0 × 1) + (0) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = 1
","Step 5: calculate all weight changes.
","Δw = 1 × [1, 1, 1, 1] = [1, 1, 1, 1]
","Step 6: calculate the bias change.
","Δb = 1.
","Step 7: carry the updated parameters forward.
","New w = [-1, 3, 1, 1], new b = 1.
","Step 1: copy this sample and target.
","x = [-1, 1, -1, -1], target = 1.
","Step 2: calculate the current net.
","z = (-1 × -1) + (3 × 1) + (1 × -1) + (1 × -1) + (1) = 3
","Step 3: apply the threshold.
","output = 1.
","Step 4: calculate the update multiplier.
","Update multiplier = 0
","Step 5: calculate all weight changes.
","Δw = 0 × [-1, 1, -1, -1] = [0, 0, 0, 0]
","Step 6: calculate the bias change.
","Δb = 0.
","Step 7: carry the updated parameters forward.
","New w = [-1, 3, 1, 1], new b = 1.
","Step 1: copy this sample and target.
","x = [1, 1, 1, -1], target = -1.
","Step 2: calculate the current net.
","z = (-1 × 1) + (3 × 1) + (1 × 1) + (1 × -1) + (1) = 3
","Step 3: apply the threshold.
","output = 1.
","Step 4: calculate the update multiplier.
","Update multiplier = -1
","Step 5: calculate all weight changes.
","Δw = -1 × [1, 1, 1, -1] = [-1, -1, -1, 1]
","Step 6: calculate the bias change.
","Δb = -1.
","Step 7: carry the updated parameters forward.
","New w = [-2, 2, 0, 2], new b = 0.
","Step 1: copy this sample and target.
","x = [1, -1, -1, 1], target = -1.
","Step 2: calculate the current net.
","z = (-2 × 1) + (2 × -1) + (0 × -1) + (2 × 1) + (0) = -2
","Step 3: apply the threshold.
","output = -1.
","Step 4: calculate the update multiplier.
","Update multiplier = 0
","Step 5: calculate all weight changes.
","Δw = 0 × [1, -1, -1, 1] = [0, 0, 0, 0]
","Step 6: calculate the bias change.
","Δb = 0.
","Step 7: carry the updated parameters forward.
","New w = [-2, 2, 0, 2], new b = 0.
","Final after exactly the requested epochs: w = [-2, 2, 0, 2], b = 0. A finite number of requested epochs does not itself prove convergence or correct classification of every sample.
"],"kind":"practice","id":"tee-c14-s8","search":"R24 Q4a **Question (10 marks).** Train a perceptron for 2 epochs with η = 1, initial weights and bias 0, threshold 0. Samples in order: [1,1,1,1] → 1; [−1,1,−1,−1] → 1; [1,1,1,−1] → −1; [1,−1,−1,1] → −1. Bias input is 1.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nTrain a perceptron for 2 epochs with η = 1, initial weights and bias 0, threshold 0.\n\nSamples in order: [1,1,1,1] → 1.\n\n[−1,1,−1,−1] → 1.\n\n[1,1,1,−1] → −1.\n\n[1,−1,−1,1] → −1.\n\nBias input is 1.\n\n### To find\n\nProcess samples in order for the stated epochs and report the final weights and bias.\n\n### Formula — which one and why\n\n**Target-based perceptron correction.** This worked convention changes parameters only on a wrong prediction. Use the target itself as the signed multiplier; it differs from the binary error rule.\n\nt is the bipolar target; η is the learning rate. Output is +1 above zero, −1 below zero and 0 at exactly zero.\n\n\n\n**Yaad rakhne ka cue:** Is convention mein galat hua toh target × input se correction.\n\n### Solution — step by step\n\n\n\n\n**State the threshold rule.** Use output +1 if z>0, −1 if z<0, and 0 at z = 0. On a mismatch use Δw = η tx and Δb = ηt; on a match make no update. η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly.\n### Epoch 1, sample 1\n\n**Step 1: copy this sample and target.**\n\nx = [1, 1, 1, 1], target = 1.\n\n**Step 2: calculate the current net.**\n\nz = (0 × 1) + (0 × 1) + (0 × 1) + (0 × 1) + (0) = 0\n\n**Step 3: apply the threshold.**\n\noutput = 0.\n\n**Step 4: calculate the update multiplier.**\n\nUpdate multiplier = 1\n\n**Step 5: calculate all weight changes.**\n\nΔw = 1 × [1, 1, 1, 1] = [1, 1, 1, 1]\n\n**Step 6: calculate the bias change.**\n\nΔb = 1.\n\n**Step 7: carry the updated parameters forward.**\n\nNew w = [1, 1, 1, 1], new b = 1.\n\n\n### Epoch 1, sample 2\n\n**Step 1: copy this sample and target.**\n\nx = [-1, 1, -1, -1], target = 1.\n\n**Step 2: calculate the current net.**\n\nz = (1 × -1) + (1 × 1) + (1 × -1) + (1 × -1) + (1) = -1\n\n**Step 3: apply the threshold.**\n\noutput = -1.\n\n**Step 4: calculate the update multiplier.**\n\nUpdate multiplier = 1\n\n**Step 5: calculate all weight changes.**\n\nΔw = 1 × [-1, 1, -1, -1] = [-1, 1, -1, -1]\n\n**Step 6: calculate the bias change.**\n\nΔb = 1.\n\n**Step 7: carry the updated parameters forward.**\n\nNew w = [0, 2, 0, 0], new b = 2.\n\n\n### Epoch 1, sample 3\n\n**Step 1: copy this sample and target.**\n\nx = [1, 1, 1, -1], target = -1.\n\n**Step 2: calculate the current net.**\n\nz = (0 × 1) + (2 × 1) + (0 × 1) + (0 × -1) + (2) = 4\n\n**Step 3: apply the threshold.**\n\noutput = 1.\n\n**Step 4: calculate the update multiplier.**\n\nUpdate multiplier = -1\n\n**Step 5: calculate all weight changes.**\n\nΔw = -1 × [1, 1, 1, -1] = [-1, -1, -1, 1]\n\n**Step 6: calculate the bias change.**\n\nΔb = -1.\n\n**Step 7: carry the updated parameters forward.**\n\nNew w = [-1, 1, -1, 1], new b = 1.\n\n\n### Epoch 1, sample 4\n\n**Step 1: copy this sample and target.**\n\nx = [1, -1, -1, 1], target = -1.\n\n**Step 2: calculate the current net.**\n\nz = (-1 × 1) + (1 × -1) + (-1 × -1) + (1 × 1) + (1) = 1\n\n**Step 3: apply the threshold.**\n\noutput = 1.\n\n**Step 4: calculate the update multiplier.**\n\nUpdate multiplier = -1\n\n**Step 5: calculate all weight changes.**\n\nΔw = -1 × [1, -1, -1, 1] = [-1, 1, 1, -1]\n\n**Step 6: calculate the bias change.**\n\nΔb = -1.\n\n**Step 7: carry the updated parameters forward.**\n\nNew w = [-2, 2, 0, 0], new b = 0.\n\n\n### Epoch 2, sample 1\n\n**Step 1: copy this sample and target.**\n\nx = [1, 1, 1, 1], target = 1.\n\n**Step "},{"title":"R24 Q4b","question":["Question (10 marks). Calculate weights, biases and feature-map sizes of the CNN: Input 227 × 227 × 3. → Conv K = 9, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 120 units. → Dense 84 units. → Dense 10 units.
"],"answer":["Input 227 × 227 × 3.
","Conv K = 9, filters = 32, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 5, filters = 64, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 5, filters = 64, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Dense 120 units.
","Dense 84 units.
","Dense 10 units.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 227 × 227 × 3. Use 32 filters of spatial size 9 × 9, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 219 × 219 × 32.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 219 × 219 × 32. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 109 × 109 × 32.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The division gives 108.5; take its floor before adding 1. A partial window outside the unpadded input is not included.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 109 × 109 × 32. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 105 × 105 × 64.
","3. Count trainable values. Each filter receives all 32 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 105 × 105 × 64. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 52 × 52 × 64.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The division gives 51.5; take its floor before adding 1. A partial window outside the unpadded input is not included.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 52 × 52 × 64. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 48 × 48 × 64.
","3. Count trainable values. Each filter receives all 64 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 48 × 48 × 64. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 24 × 24 × 64.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 24 × 24 × 64. Rearrange all activation values into one vector.
","Output: a vector with 36,864 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 36,864 features. Output: 120 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 120 features. Output: 84 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 84 features. Output: 10 neurons. Every output neuron connects to every input feature and has one bias.
","| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | | 219 x 219 x 32 | 7,776 | 32 | 7,808 |
| 2 pool | | 109 x 109 x 32 | 0 | 0 | 0 |
| 3 conv | | 105 x 105 x 64 | 51,200 | 64 | 51,264 |
| 4 pool | | 52 x 52 x 64 | 0 | 0 | 0 |
| 5 conv | | 48 x 48 x 64 | 102,400 | 64 | 102,464 |
| 6 pool | | 24 x 24 x 64 | 0 | 0 | 0 |
| 7 flat | 24 × 24 × 64 | 36864 | 0 | 0 | 0 |
| 8 dense | 36864 → 120 | 120 | 4,423,680 | 120 | 4,423,800 |
| 9 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 |
| 10 dense | 84 → 10 | 10 | 840 | 10 | 850 |
Parameter substitutions, in layer order:
","Layer 1: (9 x 9 x 3 + 1) x 32 = 7,808.
","Layer 3: (5 x 5 x 32 + 1) x 64 = 51,264.
","Layer 5: (5 x 5 x 64 + 1) x 64 = 102,464.
","Layer 8: (36864 + 1) x 120 = 4,423,800.
","Layer 9: (120 + 1) x 84 = 10,164.
","Layer 10: (84 + 1) x 10 = 850.
","Total: 7,808 + 51,264 + 102,464 + 4,423,800 + 10,164 + 850 = 4,596,350 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
"],"kind":"practice","id":"tee-c14-s9","search":"R24 Q4b **Question (10 marks).** Calculate weights, biases and feature-map sizes of the CNN: Input 227 × 227 × 3. → Conv K = 9, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 120 units. → Dense 84 units. → Dense 10 units.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput 227 × 227 × 3.\n\nConv K = 9, filters = 32, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nConv K = 5, filters = 64, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nConv K = 5, filters = 64, stride = 1, padding = 0.\n\nPool K = 2, stride = 2, padding = 0.\n\nDense 120 units.\n\nDense 84 units.\n\nDense 10 units.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{227+2 \\times 0-9}{1}\\right\\rfloor+1\\) | 219 x 219 x 32 | 7,776 | 32 | 7,808 |\n| 2 pool | \\(\\left\\lfloor\\frac{219-2}{2}\\right\\rfloor+1\\) | 109 x 109 x 32 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{109+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 105 x 105 x 64 | 51,200 | 64 | 51,264 |\n| 4 pool | \\(\\left\\lfloor\\frac{105-2}{2}\\right\\rfloor+1\\) | 52 x 52 x 64 | 0 | 0 | 0 |\n| 5 conv | \\(\\left\\lfloor\\frac{52+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 48 x 48 x 64 | 102,400 | 64 | 102,464 |\n| 6 pool | \\(\\left\\lfloor\\frac{48-2}{2}\\right\\rfloor+1\\) | 24 x 24 x 64 | 0 | 0 | 0 |\n| 7 flat | 24 × 24 × 64 | 36864 | 0 | 0 | 0 |\n| 8 dense | 36864 → 120 | 120 | 4,423,680 | 120 | 4,423,800 |\n| 9 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 |\n| 10 dense | 84 → 10 | 10 | 840 | 10 | 850 |\n**Parameter substitutions, in layer order: **\nLayer 1: (9 x 9 x 3 + 1) x 32 = 7,808.\nLayer 3: (5 x 5 x 32 + 1) x 64 = 51,264.\nLayer 5: (5 x 5 x 64 + 1) x 64 = 102,464.\nLayer 8: (36864 + 1) x 120 = 4,423,800.\nLayer 9: (120 + 1) x 84 = 10,164.\nLayer 10: (84 + 1) x 10 = 850.\n**Total: ** 7,808 + 51,264 + 102,464 + 4,423,800 + 10,164 + 850 = **4,596,350 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.\n"},{"title":"R24 Q5a","question":["Question (10 marks). A GoogLeNet classifier trains slowly and vanishing gradients are suspected. Suggest an architecture change and explain its important features.
"],"answer":["A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.
","The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.
","Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].
","A ResNet is a suitable architecture to investigate because skip additions provide shorter gradient paths. First verify the diagnosis with gradients and training curves; slow training alone does not establish vanishing gradients. Match projection shortcuts when spatial size/channels change and compare validation performance after the redesign.
"],"kind":"practice","id":"tee-c14-s10","search":"R24 Q5a **Question (10 marks).** A GoogLeNet classifier trains slowly and vanishing gradients are suspected. Suggest an architecture change and explain its important features.\n\n**Complete worked answer.**\nA residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.\n\n\nThe identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.\nElementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].\nA ResNet is a suitable architecture to investigate because skip additions provide shorter gradient paths. First verify the diagnosis with gradients and training curves; slow training alone does not establish vanishing gradients. Match projection shortcuts when spatial size/channels change and compare validation performance after the redesign.\n"},{"title":"R24 Q5b","question":["Question (10 marks). (i) Justify DL for chest-cancer diagnosis from X-ray images(5). (ii) List RNN advantages and disadvantages(5).
"],"answer":["A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification.
","Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities.
","Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation.
","For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions.
","RNN advantages: ordered context, shared weights over time, variable sequence lengths and flexible input/output mappings. Limitations: sequential computation reduces time-step parallelism; vanilla cells have vanishing/exploding gradient issues and limited learned long context; many training choices still require validation. Use CNNs for spatial image patterns and recurrent models when temporal/ordered structure is relevant.
"],"kind":"practice","id":"tee-c14-s11","search":"R24 Q5b **Question (10 marks).** (i) Justify DL for chest-cancer diagnosis from X-ray images(5). (ii) List RNN advantages and disadvantages(5).\n\n**Complete worked answer.**\nA convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification.\n\nConvolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities.\nCompared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation.\nFor medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions.\n**RNN advantages: ** ordered context, shared weights over time, variable sequence lengths and flexible input/output mappings. **Limitations: ** sequential computation reduces time-step parallelism; vanilla cells have vanishing/exploding gradient issues and limited learned long context; many training choices still require validation. Use CNNs for spatial image patterns and recurrent models when temporal/ordered structure is relevant.\n"},{"title":"R24 Q6a","question":["Question (10 marks). For a GRU time step, explain the significance of update and reset gates and develop their equations.
"],"answer":["A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:
","The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.
","Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.
"],"kind":"practice","id":"tee-c14-s12","search":"R24 Q6a **Question (10 marks).** For a GRU time step, explain the significance of update and reset gates and develop their equations.\n\n**Complete worked answer.**\nA GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:\n\n\n\nThe reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.\nCompared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.\n"},{"title":"R24 Q6b","question":["Question (10 marks). A supermarket vegetable classifier has 10 classes but few examples per class and limited training resources. Explain different ways to use a pretrained model.
"],"answer":["Transfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.
","1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together.
","2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.
","3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.
","4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.
","5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.
","6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds.
","Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.
","For vegetables, replace the source head with 10 softmax outputs. Strategy 1: freeze the base and train only this head. Strategy 2: initialize from the same pretrained model and fine-tune selected later blocks at a small rate after head training. The first is cheaper and less prone to fitting all parameters from limited examples; the second can adapt features more closely to vegetables when validation supports it.
"],"kind":"practice","id":"tee-c14-s13","search":"R24 Q6b **Question (10 marks).** A supermarket vegetable classifier has 10 classes but few examples per class and limited training resources. Explain different ways to use a pretrained model.\n\n**Complete worked answer.**\nTransfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.\n1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together.\n2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.\n3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.\n4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.\n5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.\n6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds.\nFeature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.\nFor vegetables, replace the source head with 10 softmax outputs. Strategy 1: freeze the base and train only this head. Strategy 2: initialize from the same pretrained model and fine-tune selected later blocks at a small rate after head training. The first is cheaper and less prone to fitting all parameters from limited examples; the second can adapt features more closely to vegetables when validation supports it.\n"},{"title":"R24 Q7a","question":["Question (10 marks). Draw and explain LSTM architecture.
"],"answer":["An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.
","The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.
","In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.
"],"kind":"practice","id":"tee-c14-s14","search":"R24 Q7a **Question (10 marks).** Draw and explain LSTM architecture.\n\n**Complete worked answer.**\nAn LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.\n\n\n\n\nThe forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.\nIn language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.\n"},{"title":"R24 Q7b","question":["Question (10 marks). Explain different RNN input/output sequence types with architecture sketches.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","One-to-many: one image representation generates a caption. Many-to-one: a review produces one sentiment. Aligned many-to-many: each video frame receives a label. Unaligned many-to-many: an encoder reads a source sentence and a decoder generates a translation of possibly different length. Specify which outputs carry losses during training; variable lengths do not require new weights for each position.
"],"kind":"practice","id":"tee-c14-s15","search":"R24 Q7b **Question (10 marks).** Explain different RNN input/output sequence types with architecture sketches.\n\n**Complete worked answer.**\nA recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.\n\n\nAt each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.\nFor next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.\nA basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.\n\nOne-to-many: one image representation generates a caption. Many-to-one: a review produces one sentiment. Aligned many-to-many: each video frame receives a label. Unaligned many-to-many: an encoder reads a source sentence and a decoder generates a translation of possibly different length. Specify which outputs carry losses during training; variable lengths do not require new weights for each position.\n\n\n"}]},{"n":15,"title":"Final examination November 2022 fully solved","sections":[{"title":"Start here","blocks":["F22 = S16, 19 November 2022. The older LeNet calculation is retained as standard CNN arithmetic and because the current course policy includes a LeNet lab. Historical diagram ambiguities are stated rather than silently guessed.
","Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible.
"],"kind":"read","id":"tee-c15-intro"},{"title":"F22 Q1a","question":["Question (4 marks). A normalized 5 × 5 image has every row [0.5,0.5,0.3,0.3,0.3]. Convolve it with a 3 × 3 filter whose every row is [1,0,−1], stride 2, no padding. Compute the output and identify the feature.
"],"answer":["A normalized 5 × 5 image has every row [0.5,0.5,0.3,0.3,0.3].
","Convolve it with a 3 × 3 filter whose every row is [1,0,−1], stride 2, no padding.
","Find every convolution output cell.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Step 1 shape: = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 1–3. The window moves 2 cells per output step.
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0.5 | 0 | −0.3 |
| 0.5 | 0 | −0.3 |
| 0.5 | 0 | −0.3 |
Each cell is input × matching kernel coefficient.
0.2 + 0.2 + 0.2 = 0.6
| 0.6 | 0 |
| 0.6 | 0 |
Output row 1, column 1.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 3–5. The window moves 2 cells per output step.
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0.3 | 0 | −0.3 |
| 0.3 | 0 | −0.3 |
| 0.3 | 0 | −0.3 |
Each cell is input × matching kernel coefficient.
0 + 0 + 0 = 0
| 0.6 | 0 |
| 0.6 | 0 |
Output row 1, column 2.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 1–3. The window moves 2 cells per output step.
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0.5 | 0 | −0.3 |
| 0.5 | 0 | −0.3 |
| 0.5 | 0 | −0.3 |
Each cell is input × matching kernel coefficient.
0.2 + 0.2 + 0.2 = 0.6
| 0.6 | 0 |
| 0.6 | 0 |
Output row 2, column 1.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 3–5. The window moves 2 cells per output step.
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
| 0.5 | 0.5 | 0.3 | 0.3 | 0.3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0.3 | 0 | −0.3 |
| 0.3 | 0 | −0.3 |
| 0.3 | 0 | −0.3 |
Each cell is input × matching kernel coefficient.
0 + 0 + 0 = 0
| 0.6 | 0 |
| 0.6 | 0 |
Output row 2, column 2.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Final output: . Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
The filter compares left and right neighborhoods and responds to a vertical brightness boundary. Inputs are already normalized. No additional output-normalization rule is supplied, so the determined convolution is . If an examiner explicitly requests min-max normalization of this result, divide by 0.6 to obtain ; that is a separate stated operation.
Question (4 marks). Weights are 0.55 and 0.65; their gradients are+0.3 and−0.25. With η = 0.1, compute gradient-descent changes and new weights.
"],"answer":["Old weights = [0.55,0.65].
","Corresponding supplied gradients = [0.3,−0.25]; η = 0.1.
","Calculate the signed weight changes and both new weights.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","Step 1: calculate the change in weight 1.
","Step 2: add that change to old weight 1.
","Step 3: calculate the change in weight 2.
","Step 4: add that change to old weight 2.
","Final answer: changes = [−0.03, +0.025]; new weights = [0.52, 0.675]. A negative gradient increases its weight under descent.
"],"kind":"practice","id":"tee-c15-s1","search":"F22 Q1b **Question (4 marks).** Weights are 0.55 and 0.65; their gradients are+0.3 and−0.25. With η = 0.1, compute gradient-descent changes and new weights.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nOld weights = [0.55,0.65].\n\nCorresponding supplied gradients = [0.3,−0.25]; η = 0.1.\n\n### To find\n\nCalculate the signed weight changes and both new weights.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n\n**Step 1: calculate the change in weight 1.**\n\n\n\n\n\n\n\n**Step 2: add that change to old weight 1.**\n\n\n\n\n\n\n\n**Step 3: calculate the change in weight 2.**\n\n\n\n\n\n\n\n**Step 4: add that change to old weight 2.**\n\n\n\n\n\n\n\n**Final answer:** changes = [−0.03, +0.025]; new weights = [0.52, 0.675]. A negative gradient increases its weight under descent.\n"},{"title":"F22 Q1c","question":["Question (4 marks). Compare ML and DL algorithms.
"],"answer":["Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.
","| Comparison | Traditional ML approaches | Deep learning approaches |
|---|---|---|
| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |
| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |
| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |
| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |
These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.
","Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.
"],"kind":"practice","id":"tee-c15-s2","search":"F22 Q1c **Question (4 marks).** Compare ML and DL algorithms.\n\n**Complete worked answer.**\nDeep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.\n| Comparison | Traditional ML approaches | Deep learning approaches |\n| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |\n| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |\n| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |\n| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |\nThese are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.\nDrivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.\n"},{"title":"F22 Q1d","question":["Question (4 marks). Compare sigmoid and tanh, emphasizing tanh’s advantage.
"],"answer":["An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.
","Sigmoid
","Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.
","Hyperbolic tangent
","The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.
","ReLU
","The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.
","Leaky ReLU
","Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.
","Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.
","The requested advantage is that tanh is zero-centered, so hidden activations can be positive or negative. Sigmoid activations lie above 0. Both functions saturate and can produce small derivatives; zero-centering does not eliminate the vanishing-gradient problem.
"],"kind":"practice","id":"tee-c15-s3","search":"F22 Q1d **Question (4 marks).** Compare sigmoid and tanh, emphasizing tanh’s advantage.\n\n**Complete worked answer.**\nAn activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.\n**Sigmoid**\n\nHere z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.\n**Hyperbolic tangent**\n\nThe numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.\n**ReLU**\n\nThe output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.\n**Leaky ReLU**\n\nAlpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.\nSigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.\nThe requested advantage is that tanh is zero-centered, so hidden activations can be positive or negative. Sigmoid activations lie above 0. Both functions saturate and can produce small derivatives; zero-centering does not eliminate the vanishing-gradient problem.\n"},{"title":"F22 Q1e","question":["Question (4 marks). Suggest and sketch an RNN architecture for image captioning.
"],"answer":["Use a CNN encoder to map the image to a visual representation and a recurrent decoder (RNN/LSTM/GRU) to generate words one at a time. Condition the decoder’s initial state or its inputs on image features. At each step, vocabulary softmax predicts the next word based on the image and earlier words. Train using correct next-word targets and cross-entropy; stop generation at an end token. This is a one-to-many mapping from one image to a caption sequence.
","Question (10 marks). Explain GRU advantages over a normal RNN. For “The student who did not submit the assignment was bound to loose marks,” show how memory and update gates can retain the subject before predicting “was”.
"],"answer":["A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:
","The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.
","Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.
","Important data limit: no numerical weights or token vectors are supplied, so actual gate outputs cannot be uniquely calculated. The following is an explicitly idealized one-component illustration: candidate subject feature = 1 at “student”; use hnew = (1−z) hold+z candidate. Exact 0/1 gates are limiting idealizations of sigmoid values.
","| Time and token | Illustrative z | Singular-subject memory after step |
|---|---|---|
| 1 The | 0 while retaining this illustrative feature | 0 before subject is read |
| 2 student | 1 when admitting subject feature | 1 after subject is read |
| 3 who | 0 while retaining this illustrative feature | 1 after subject is read |
| 4 did | 0 while retaining this illustrative feature | 1 after subject is read |
| 5 not | 0 while retaining this illustrative feature | 1 after subject is read |
| 6 submit | 0 while retaining this illustrative feature | 1 after subject is read |
| 7 the | 0 while retaining this illustrative feature | 1 after subject is read |
| 8 assignment | 0 while retaining this illustrative feature | 1 after subject is read |
At “student”: hnew = (1−1) × 0+1 × 1 = 1. At each intervening token in this illustrative component: hnew = (1−0) × 1+0 × candidate = 1. The retained singular subject helps the output layer choose “was”. Other hidden components may still update to represent intervening words; this table is not a computed trace from an actual trained network.
"],"kind":"practice","id":"tee-c15-s5","search":"F22 Q2a **Question (10 marks).** Explain GRU advantages over a normal RNN. For “The student who did not submit the assignment was bound to loose marks,” show how memory and update gates can retain the subject before predicting “was”.\n\n**Complete worked answer.**\nA GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate:\n\n\n\nThe reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate.\nCompared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN.\n**Important data limit: ** no numerical weights or token vectors are supplied, so actual gate outputs cannot be uniquely calculated. The following is an explicitly idealized one-component illustration: candidate subject feature = 1 at “student”; use hnew = (1−z) hold+z candidate. Exact 0/1 gates are limiting idealizations of sigmoid values.\n| Time and token | Illustrative z | Singular-subject memory after step |\n| 1 The | 0 while retaining this illustrative feature | 0 before subject is read |\n| 2 student | 1 when admitting subject feature | 1 after subject is read |\n| 3 who | 0 while retaining this illustrative feature | 1 after subject is read |\n| 4 did | 0 while retaining this illustrative feature | 1 after subject is read |\n| 5 not | 0 while retaining this illustrative feature | 1 after subject is read |\n| 6 submit | 0 while retaining this illustrative feature | 1 after subject is read |\n| 7 the | 0 while retaining this illustrative feature | 1 after subject is read |\n| 8 assignment | 0 while retaining this illustrative feature | 1 after subject is read |\n\nAt “student”: hnew = (1−1) × 0+1 × 1 = 1. At each intervening token in this illustrative component: hnew = (1−0) × 1+0 × candidate = 1. The retained singular subject helps the output layer choose “was”. Other hidden components may still update to represent intervening words; this table is not a computed trace from an actual trained network.\n"},{"title":"F22 Q2b","question":["Question (10 marks). For a three-stage sigmoid network, use samples [0.3,0.2,0.4] → 0 and [0.5,0.6,0.8] → 1. First-stage feature multipliers are [0.3,0.4,0.6]. Next receiving rows are [0.5,0.2,0.3], [0.3,0.3,0.3], and output weights [0.3,0.3]. Find both predictions, average MSE and output-weight updates after the first backward pass with η = 0.2. Assume zero biases.
"],"answer":["For a three-stage sigmoid network, use samples [0.3,0.2,0.4] → 0 and [0.5,0.6,0.8] → 1.
","First-stage feature multipliers are [0.3,0.4,0.6].
","Next receiving rows are [0.5,0.2,0.3], [0.3,0.3,0.3], and output weights [0.3,0.3].
","Find both predictions, average MSE and output-weight updates after the first backward pass with η = 0.2.
","Assume zero biases.
","Find all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Mean squared loss. Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean.
","n = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target.
","Yaad rakhne ka cue: MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Interpretation. Follow the supplied diagram: each raw feature is multiplied by 0.3, 0.4 or 0.6, then passed through sigmoid before the next two-neuron layer. Unspecified biases are zero. MSE is averaged over the two samples. The phrase “first backward pass” does not specify batch versus single-sample updating, so both interpretations are identified below.
","For sample 1, the preactivations are [0.3 × 0.3, 0.2 × 0.4, 0.4 × 0.6] = [0.09, 0.08, 0.24]. Sigmoid gives [0.522484825, 0.519989340, 0.559713649].
","For sample 2, they are [0.5 × 0.3, 0.6 × 0.4, 0.8 × 0.6] = [0.15, 0.24, 0.48]. Sigmoid gives [0.537429845, 0.559713649, 0.617747875].
","For each first-stage output a, calculate z₁ = 0.5a₁ + 0.2a₂ + 0.3a₃ and z₂ = 0.3a₁ + 0.3a₂ + 0.3a₃, then apply sigmoid.
","| Sample | Hidden preactivation 1 | Hidden preactivation 2 | Hidden output 1 | Hidden output 2 |
|---|---|---|---|---|
| 1 | 0.533154375 | 0.480656344 | 0.630218519 | 0.617902849 |
| 2 | 0.565982015 | 0.514467411 | 0.637835530 | 0.625853155 |
For example, sample 1's first hidden sum is 0.5(0.522484825) + 0.2(0.519989340) + 0.3(0.559713649) = 0.533154375.
","The output preactivation is 0.3h₁ + 0.3h₂; apply sigmoid to the complete sum.
","| Sample | Output preactivation | Prediction | Target | Squared error |
|---|---|---|---|---|
| 1 | 0.374436410 | 0.592530535 | 0 | 0.351092435 |
| 2 | 0.379106605 | 0.593657609 | 1 | 0.165114139 |
MSE = (0.351092435 + 0.165114139)/2 = 0.258103287.
","For one sample, output delta = 2(o − y)o(1 − o); each output-weight gradient is this delta multiplied by its sending hidden activation. Calculate both samples at the original weights before taking their mean.
","| Sample | Output delta | Gradient for h₁ → output | Gradient for h₂ → output |
|---|---|---|---|
| 1 | 0.286118893 | 0.180317425 | 0.176793679 |
| 2 | −0.196042530 | −0.125042891 | −0.122693836 |
The mean gradients are [(0.180317425 − 0.125042891)/2, (0.176793679 − 0.122693836)/2] ≈ [0.027637267, 0.027049922].
","First weight: 0.3 − 0.2(0.027637267) = 0.294472547.
","Second weight: 0.3 − 0.2(0.027049922) = 0.294590016.
","Final answer under batch MSE: predictions [0.592530535, 0.593657609], average MSE 0.258103287, updated output weights [0.294472547, 0.294590016]. Averaging the sample gradients is required because the stated objective is their mean loss.
","If “first backward pass” means an update after sample 1 alone, use its gradients without averaging: 0.3 − 0.2(0.180317425) = 0.263936515 and 0.3 − 0.2(0.176793679) = 0.264641264. This is a different update schedule. State the interpretation and do not combine a batch loss with an unexplained single-sample gradient.
"],"kind":"practice","id":"tee-c15-s6","search":"F22 Q2b **Question (10 marks).** For a three-stage sigmoid network, use samples [0.3,0.2,0.4] → 0 and [0.5,0.6,0.8] → 1. First-stage feature multipliers are [0.3,0.4,0.6]. Next receiving rows are [0.5,0.2,0.3], [0.3,0.3,0.3], and output weights [0.3,0.3]. Find both predictions, average MSE and output-weight updates after the first backward pass with η = 0.2. Assume zero biases.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nFor a three-stage sigmoid network, use samples [0.3,0.2,0.4] → 0 and [0.5,0.6,0.8] → 1.\n\nFirst-stage feature multipliers are [0.3,0.4,0.6].\n\nNext receiving rows are [0.5,0.2,0.3], [0.3,0.3,0.3], and output weights [0.3,0.3].\n\nFind both predictions, average MSE and output-weight updates after the first backward pass with η = 0.2.\n\nAssume zero biases.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Mean squared loss.** Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean.\n\nn = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target.\n\n\n\n\n\n**Yaad rakhne ka cue:** MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n**Interpretation.** Follow the supplied diagram: each raw feature is multiplied by 0.3, 0.4 or 0.6, then passed through sigmoid before the next two-neuron layer. Unspecified biases are zero. MSE is averaged over the two samples. The phrase “first backward pass” does not specify batch versus single-sample updating, so both interpretations are identified below.\n\n### Step 1: apply the first-stage multipliers and sigmoid\nFor sample 1, the preactivations are [0.3 × 0.3, 0.2 × 0.4, 0.4 × 0.6] = [0.09, 0.08, 0.24]. Sigmoid gives [0.522484825, 0.519989340, 0.559713649].\n\nFor sample 2, they are [0.5 × 0.3, 0.6 × 0.4, 0.8 × 0.6] = [0.15, 0.24, 0.48]. Sigmoid gives [0.537429845, 0.559713649, 0.617747875].\n\n### Step 2: evaluate the two hidden neurons\nFor each first-stage output a, c"},{"title":"F22 Q3a","question":["Question (10 marks). Explain bias, variance, underfitting and overfitting. A four-feature product-texture classifier has four grades including reject. Choose output activation/loss. Diagnose training error 15% and validation error 18% and suggest remedies.
"],"answer":["Bias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data.
","Increasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws.
","Underfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate.
","Training and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting.
","Classifier: four mutually exclusive grades require 4 output scores, softmax probabilities and categorical cross-entropy. A reject class is still a labeled class in this formulation.
","Diagnosis: a modest 3 percentage-point gap with both errors appreciable suggests underfitting/high bias if a much better attainable baseline exists. Confirm learning curves and label quality. Improve relevant features or model capacity, train sufficiently, tune the optimizer and reduce excessive regularization. The percentages alone do not prove the cause.
"],"kind":"practice","id":"tee-c15-s7","search":"F22 Q3a **Question (10 marks).** Explain bias, variance, underfitting and overfitting. A four-feature product-texture classifier has four grades including reject. Choose output activation/loss. Diagnose training error 15% and validation error 18% and suggest remedies.\n\n**Complete worked answer.**\nBias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data.\n\nIncreasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws.\nUnderfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate.\nTraining and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting.\n**Classifier: ** four mutually exclusive grades require 4 output scores, softmax probabilities and categorical cross-entropy. A reject class is still a labeled class in this formulation.\n**Diagnosis: ** a modest 3 percentage-point gap with both errors appreciable suggests underfitting/high bias if a much better attainable baseline exists. Confirm learning curves and label quality. Improve relevant features or model capacity, train sufficiently, tune the optimizer and reduce excessive regularization. The percentages alone do not prove the cause.\n"},{"title":"F22 Q3b","question":["Question (10 marks). Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2; convolution 5 × 5: 16 filters, pooling 2; dense 120, dense 84, softmax 10. Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps.
"],"answer":["Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2.
","convolution 5 × 5: 16 filters, pooling 2.
","dense 120, dense 84, softmax 10.
","Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 32 × 32 × 1. Use 6 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 28 × 28 × 6.
","3. Count trainable values. Each filter receives all 1 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 28 × 28 × 6. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 14 × 14 × 6.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 14 × 14 × 6. Use 16 filters of spatial size 5 × 5, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 10 × 10 × 16.
","3. Count trainable values. Each filter receives all 6 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 10 × 10 × 16. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 5 × 5 × 16.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 5 × 5 × 16. Rearrange all activation values into one vector.
","Output: a vector with 400 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 400 features. Output: 120 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 120 features. Output: 84 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 84 features. Output: 10 neurons. Every output neuron connects to every input feature and has one bias.
","| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | | 28 x 28 x 6 | 150 | 6 | 156 |
| 2 pool | | 14 x 14 x 6 | 0 | 0 | 0 |
| 3 conv | | 10 x 10 x 16 | 2,400 | 16 | 2,416 |
| 4 pool | | 5 x 5 x 16 | 0 | 0 | 0 |
| 5 flat | 5 × 5 × 16 | 400 | 0 | 0 | 0 |
| 6 dense | 400 → 120 | 120 | 48,000 | 120 | 48,120 |
| 7 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 |
| 8 dense | 84 → 10 | 10 | 840 | 10 | 850 |
Parameter substitutions, in layer order:
","Layer 1: (5 x 5 x 1 + 1) x 6 = 156.
","Layer 3: (5 x 5 x 6 + 1) x 16 = 2,416.
","Layer 6: (400 + 1) x 120 = 48,120.
","Layer 7: (120 + 1) x 84 = 10,164.
","Layer 8: (84 + 1) x 10 = 850.
","Total: 156 + 2,416 + 48,120 + 10,164 + 850 = 61,706 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","This is the standard classroom full-channel interpretation with nontrainable pooling and a dense 10-class softmax. The historical original LeNet used partial channel connections and other details, so its published parameter count need not equal this diagram-based count. If an input label is read as 28, the stated later 28 map after valid 5 × 5 convolution is inconsistent; record the 32 input assumption rather than silently ignoring the mismatch.
"],"kind":"practice","id":"tee-c15-s8","search":"F22 Q3b **Question (10 marks).** Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2; convolution 5 × 5: 16 filters, pooling 2; dense 120, dense 84, softmax 10. Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nCount trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2.\n\nconvolution 5 × 5: 16 filters, pooling 2.\n\ndense 120, dense 84, softmax 10.\n\nUse the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{32+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 28 x 28 x 6 | 150 | 6 | 156 |\n| 2 pool | \\(\\left\\lfloor\\frac{28-2}{2}\\right\\rfloor+1\\) | 14 x 14 x 6 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{14+2 \\times 0-5}{1}\\right\\rfloor+1\\) | 10 x 10 x 16 | 2,400 | 16 | 2,416 |\n| 4 pool | \\(\\left\\lfloor\\frac{10-2}{2}\\right\\rfloor+1\\) | 5 x 5 x 16 | 0 | 0 | 0 |\n| 5 flat | 5 × 5 × 16 | 400 | 0 | 0 | 0 |\n| 6 dense | 400 → 120 | 120 | 48,000 | 120 | 48,120 |\n| 7 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 |\n| 8 dense | 84 → 10 | 10 | 840 | 10 | 850 |\n**Parameter substitutions, in layer order: **\nLayer 1: (5 x 5 x 1 + 1) x 6 = 156.\nLayer 3: (5 x 5 x 6 + 1) x 16 = 2,416.\nLayer 6: (400 + 1) x 120 = 48,120.\nLayer 7: (120 + 1) x 84 = 10,164.\nLayer 8: (84 + 1) x 10 = 850.\n**Total: ** 156 + 2,416 + 48,120 + 10,164 + 850 = **61,706 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.\nThis is the standard classroom full-channel interpretation with nontrainable pooling and a dense 10-class softmax. The historical original LeNet used partial channel connections and other details, so its published parameter count need not equal this diagram-based count. If an input label is read as 28, the stated later 28 map after valid 5 × 5 convolution is inconsistent; record the 32 input assumption rather than silently ignoring the mismatch.\n"},{"title":"F22 Q4a","question":["Question (10 marks). An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved. Compute operations and explain 1 × 1 convolution.
"],"answer":["An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved.
","Find direct and bottleneck computation counts and the requested saving.
","Multiply–accumulate count. Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.
","H_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.
","Yaad rakhne ka cue: Har output cell ka kaam × kitne output cells.
","Step 2: direct path.
","output stays 28 × 28 with 32 channels.
","Step 3: direct path.
","MACs = 28 × 28 × 192 × 32 × 5 × 5 = 120,422,400.
","Step 4: bottleneck path with 16 reduction channels.
","1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs.
","Step 5: bottleneck path with 16 reduction channels.
","The 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs.
","Step 6: bottleneck path with 16 reduction channels.
","Total = 12,443,648.
","Step 7: saving.
","120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = 89.666667%. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = 777,728r
","Step 8: saving.
","there is no unique numeric bottleneck cost until r is chosen.
","Step 9: calculate this quantity.
","An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","Step 10: calculate this quantity.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
"],"kind":"practice","id":"tee-c15-s9","search":"F22 Q4a **Question (10 marks).** An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved. Compute operations and explain 1 × 1 convolution.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nAn Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved.\n\n### To find\n\nFind direct and bottleneck computation counts and the requested saving.\n\n### Formula — which one and why\n\n**Multiply–accumulate count.** Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.\n\nH_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells.\n\n### Solution — step by step\n\n\n\n\n\n\n\n\n\n\n### Direct path\n\n**Step 2: direct path.**\n\noutput stays 28 × 28 with 32 channels.\n\n**Step 3: direct path.**\n\nMACs = 28 × 28 × 192 × 32 × 5 × 5 = **120,422,400**.\n\n### Bottleneck path with 16 reduction channels\n\n**Step 4: bottleneck path with 16 reduction channels.**\n\n1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs.\n\n**Step 5: bottleneck path with 16 reduction channels.**\n\nThe 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs.\n\n**Step 6: bottleneck path with 16 reduction channels.**\n\nTotal = **12,443,648**.\n\n### Saving\n\n**Step 7: saving.**\n\n120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = **89.666667%**. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = **777,728r**\n\n**Step 8: saving.**\n\nthere is no unique numeric bottleneck cost until r is chosen.\n\n**Step 9: calculate this quantity.**\n\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\n\n\n**Step 10: calculate this quantity.**\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\n\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.\n"},{"title":"F22 Q4b","question":["Question (10 marks). Draw a 6-input, two-hidden-layer network with 5 neurons each and 4 classes, trained by AdaGrad. Identify learned parameters and tunable hyperparameters; no input-layer weights or biases.
"],"answer":["6 inputs → hidden 5 → hidden 5 → 4 output classes.
","AdaGrad is the specified training algorithm; one bias per receiving neuron.
","Draw the architecture and count its learned weights/biases. The optimizer name does not change the count.
","Dense-layer parameters. Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.
","n_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.
","Yaad rakhne ka cue: Har output ke liye saare inputs plus ek bias.
","Architecture: 6 → 5 → 5 → 4. Each non-input neuron has one bias.
","Step 1: count connections.
","Step 2: count receiving-neuron biases.
","There are 5 receiving neurons, so 5 biases.
","Step 3: add weights and biases.
","Step 1: count connections.
","Step 2: count receiving-neuron biases.
","There are 5 receiving neurons, so 5 biases.
","Step 3: add weights and biases.
","Step 1: count connections.
","Step 2: count receiving-neuron biases.
","There are 4 receiving neurons, so 4 biases.
","Step 3: add weights and biases.
","Total = 89. Input values and activations are not trainable parameters.
","The learned parameters are input-to-hidden, hidden-to-hidden and hidden-to-output weights plus biases at receiving neurons. AdaGrad accumulators are optimizer state, not additional predictive weights. Hyperparameters include hidden widths/depth, learning rate η,ε, batch size, epochs, activations and regularization settings. Use 4-class softmax with categorical cross-entropy.
","AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:
","Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.
","Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.
","Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.
"],"kind":"practice","id":"tee-c15-s10","search":"F22 Q4b **Question (10 marks).** Draw a 6-input, two-hidden-layer network with 5 neurons each and 4 classes, trained by AdaGrad. Identify learned parameters and tunable hyperparameters; no input-layer weights or biases.\n\n**Complete worked answer.**\n\n\n\n### Given\n\n6 inputs → hidden 5 → hidden 5 → 4 output classes.\n\nAdaGrad is the specified training algorithm; one bias per receiving neuron.\n\n### To find\n\nDraw the architecture and count its learned weights/biases. The optimizer name does not change the count.\n\n### Formula — which one and why\n\n**Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.\n\nn_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.\n\n\n\n**Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias.\n\n### Solution — step by step\n\n\n\n\nArchitecture: 6 → 5 → 5 → 4. Each non-input neuron has one bias.\n\n### Layer 6 → 5\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 5 receiving neurons, so 5 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 5 → 5\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 5 receiving neurons, so 5 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 5 → 4\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 4 receiving neurons, so 4 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n**Total = 89**. Input values and activations are not trainable parameters.\nThe learned parameters are input-to-hidden, hidden-to-hidden and hidden-to-output weights plus biases at receiving neurons. AdaGrad accumulators are optimizer state, not additional predictive weights. Hyperparameters include hidden widths/depth, learning rate η,ε, batch size, epochs, activations and regularization settings. Use 4-class softmax with categorical cross-entropy.\nAdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:\n\nFrequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.\nExample with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.\nAdvantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.\n"},{"title":"F22 Q5a","question":["Question (10 marks). Develop an RNN with hidden states and explain gradient updates using backpropagation.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.
","1. Initialize the hidden state and run the forward recurrence from the first input to the last.
","2. Compute the specified output loss, at the final step or across several steps.
","3. Start from the last loss and propagate derivatives backwards through output and hidden operations.
","4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.
","5. Sum all uses of each shared weight before applying the optimizer update.
","The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.
","Question (10 marks). Explain DL optimization challenges and the advantage of momentum over plain mini-batch gradient descent.
"],"answer":["The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.
","Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.
","Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.
","Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.
","Momentum uses vnew = βv−ηg and θ new = θ+vnew; consistent gradients accumulate direction, whereas some alternating components cancel. Plain mini-batch descent uses only the current batch gradient. Both can be combined: momentum with mini-batch gradients is a standard coherent choice.
"],"kind":"practice","id":"tee-c15-s12","search":"F22 Q5b **Question (10 marks).** Explain DL optimization challenges and the advantage of momentum over plain mini-batch gradient descent.\n\n**Complete worked answer.**\nThe learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.\n\nBatch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.\nMomentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.\nDiagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.\nMomentum uses vnew = βv−ηg and θ new = θ+vnew; consistent gradients accumulate direction, whereas some alternating components cancel. Plain mini-batch descent uses only the current batch gradient. Both can be combined: momentum with mini-batch gradients is a standard coherent choice.\n"},{"title":"F22 Q6a","question":["Question (10 marks). A CNN image classifier performs poorly because of vanishing gradients. Explain remedies.
"],"answer":["A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.
","The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.
","Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].
","Also inspect saturated sigmoid/tanh hidden units, initialization and gradient magnitudes. Appropriate ReLU-family activations and initialization can help preserve usable gradients. Residual blocks require shape-compatible additions or projections. Verify that the diagnosis is correct; additional training time alone cannot repair gradients that are consistently negligible.
"],"kind":"practice","id":"tee-c15-s13","search":"F22 Q6a **Question (10 marks).** A CNN image classifier performs poorly because of vanishing gradients. Explain remedies.\n\n**Complete worked answer.**\nA residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.\n\n\nThe identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.\nElementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].\nAlso inspect saturated sigmoid/tanh hidden units, initialization and gradient magnitudes. Appropriate ReLU-family activations and initialization can help preserve usable gradients. Residual blocks require shape-compatible additions or projections. Verify that the diagnosis is correct; additional training time alone cannot repair gradients that are consistently negligible.\n"},{"title":"F22 Q6b","question":["Question (10 marks). Design a clothes classifier for 250,000 six-dimensional samples,6 classes and 2 hidden layers. Give train/validation/test split, activations/loss, actions for 15% training 20% validation error, interpret improvement to 1%/1.5%, then address 15% test error.
"],"answer":["One explicit design: 6 → 16 → 8 → 6, ReLU hidden layers,6-unit softmax output, categorical cross-entropy. The hidden widths are chosen assumptions, not supplied numerical requirements.
","Architecture: 6 → 16 → 8 → 6. Each non-input neuron has one bias.
","Step 1: count connections.
","Step 2: count receiving-neuron biases.
","There are 16 receiving neurons, so 16 biases.
","Step 3: add weights and biases.
","Step 1: count connections.
","Step 2: count receiving-neuron biases.
","There are 8 receiving neurons, so 8 biases.
","Step 3: add weights and biases.
","Step 1: count connections.
","Step 2: count receiving-neuron biases.
","There are 6 receiving neurons, so 6 biases.
","Step 3: add weights and biases.
","Total = 302. Input values and activations are not trainable parameters.
","Split: a stratified 70/15/15 split gives 175,000 training,37,500 validation,37,500 test samples. Fit preprocessing on training data only and keep related duplicates within a single split.
","15%/20%: inspect learning curves and baseline difficulty. Substantial training error suggests capacity/optimization or data-quality issues; the 5 point gap also suggests a generalization issue. Improve training fit, tune regularization and collect useful representative data instead of blindly increasing capacity.
","1%/1.5%: good comparable-set results with a small gap are encouraging; verify that validation has not leaked into training.
","15% test: inspect distribution shift, preprocessing differences, label quality, leakage or repeated validation overfitting. Compare per-class errors and collection conditions. Address the actual cause and collect representative data if necessary. If test results guide retuning, use a fresh untouched final test set.
"],"kind":"practice","id":"tee-c15-s14","search":"F22 Q6b **Question (10 marks).** Design a clothes classifier for 250,000 six-dimensional samples,6 classes and 2 hidden layers. Give train/validation/test split, activations/loss, actions for 15% training 20% validation error, interpret improvement to 1%/1.5%, then address 15% test error.\n\n**Complete worked answer.**\n**One explicit design: **6 → 16 → 8 → 6, ReLU hidden layers,6-unit softmax output, categorical cross-entropy. The hidden widths are chosen assumptions, not supplied numerical requirements.\nArchitecture: 6 → 16 → 8 → 6. Each non-input neuron has one bias.\n### Layer 6 → 16\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 16 receiving neurons, so 16 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 16 → 8\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 8 receiving neurons, so 8 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 8 → 6\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 6 receiving neurons, so 6 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n**Total = 302**. Input values and activations are not trainable parameters.\n**Split: ** a stratified 70/15/15 split gives 175,000 training,37,500 validation,37,500 test samples. Fit preprocessing on training data only and keep related duplicates within a single split.\n**15%/20%: ** inspect learning curves and baseline difficulty. Substantial training error suggests capacity/optimization or data-quality issues; the 5 point gap also suggests a generalization issue. Improve training fit, tune regularization and collect useful representative data instead of blindly increasing capacity.\n**1%/1.5%: ** good comparable-set results with a small gap are encouraging; verify that validation has not leaked into training.\n**15% test: ** inspect distribution shift, preprocessing differences, label quality, leakage or repeated validation overfitting. Compare per-class errors and collection conditions. Address the actual cause and collect representative data if necessary. If test results guide retuning, use a fresh untouched final test set.\n"},{"title":"F22 Q7a","question":["Question (10 marks). Explain with an example how LSTM handles long sequences in language modeling.
"],"answer":["An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.
","The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.
","In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.
","Example: “The student, after several difficult assignments and a long discussion, was relieved.” Useful singular-subject information can be retained in selected cell components while intervening words update other components. At the final step a vocabulary softmax predicts the next token using h. Gates close to 1 on the retained path preserve context; numerical gate values require learned weights and cannot be inferred uniquely from the sentence.
"],"kind":"practice","id":"tee-c15-s15","search":"F22 Q7a **Question (10 marks).** Explain with an example how LSTM handles long sequences in language modeling.\n\n**Complete worked answer.**\nAn LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.\n\n\n\n\nThe forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.\nIn language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.\nExample: “The student, after several difficult assignments and a long discussion, was relieved.” Useful singular-subject information can be retained in selected cell components while intervening words update other components. At the final step a vocabulary softmax predicts the next token using h. Gates close to 1 on the retained path preserve context; numerical gate values require learned weights and cannot be inferred uniquely from the sentence.\n"},{"title":"F22 Q7b","question":["Question (10 marks). Justify that single-layer perceptrons cannot classify nonlinearly separable data using an example, and suggest a solution.
"],"answer":["A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator.
","For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1.
","For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR.
","A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map.
"],"kind":"practice","id":"tee-c15-s16","search":"F22 Q7b **Question (10 marks).** Justify that single-layer perceptrons cannot classify nonlinearly separable data using an example, and suggest a solution.\n\n**Complete worked answer.**\nA single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator.\n\nFor binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1.\nFor XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR.\nA multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map.\n\n\n"}]},{"n":16,"title":"Re examination January 2023 fully solved","sections":[{"title":"Start here","blocks":["R23 = S15, 28 January 2023. Its AlexNet and convolution questions are retained by current Unit 4; sequence questions map to current Unit 5.
","Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible.
"],"kind":"read","id":"tee-c16-intro"},{"title":"R23 Q1a","question":["Question (4 marks). Suggest and sketch an RNN architecture to synthesize a musical tune.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","Represent notes as discrete tokens or suitable pitch-duration vectors. Train a recurrent decoder to predict the next note from preceding notes. A vocabulary softmax handles discrete note tokens; repeatedly feed generated notes to continue the tune until a stop condition. This is autoregressive sequence generation, typically a many-to-many next-token training pattern.
","Question (4 marks). Apply 3 × 3 average pooling with stride 2 to Matrix 1 shown below. Give the output and justify pooling.
","Apply 3 × 3 average pooling with stride 2 to Matrix 1 shown below.
","Find the requested pooled output(s).
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","Shape: = 2 on each axis; channels stay unchanged.
Step 1: select the entries.
","[1, 2, 3, 5, 3, 4, 7, 4, 8]
","Step 2: add all selected entries.
","1 + 2 + 3 + 5 + 3 + 4 + 7 + 4 + 8 = 37
","Step 3: divide by the 9 entries in this window.
","37/9 = 4.111111111
","Step 1: select the entries.
","[3, 2, 5, 4, 6, 3, 8, 5, 9]
","Step 2: add all selected entries.
","3 + 2 + 5 + 4 + 6 + 3 + 8 + 5 + 9 = 45
","Step 3: divide by the 9 entries in this window.
","45/9 = 5
","Step 1: select the entries.
","[7, 4, 8, 8, 5, 10, 9, 10, 3]
","Step 2: add all selected entries.
","7 + 4 + 8 + 8 + 5 + 10 + 9 + 10 + 3 = 64
","Step 3: divide by the 9 entries in this window.
","64/9 = 7.111111111
","Step 1: select the entries.
","[8, 5, 9, 10, 4, 2, 3, 4, 3]
","Step 2: add all selected entries.
","8 + 5 + 9 + 10 + 4 + 2 + 3 + 4 + 3 = 48
","Step 3: divide by the 9 entries in this window.
","48/9 = 5.333333333
","Average output: . Pooling normally has zero trainable parameters and reduces spatial dimensions.
Question (4 marks). State DL advantages and limitations.
"],"answer":["Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.
","| Comparison | Traditional ML approaches | Deep learning approaches |
|---|---|---|
| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |
| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |
| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |
| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |
These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.
","Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.
"],"kind":"practice","id":"tee-c16-s2","search":"R23 Q1c **Question (4 marks).** State DL advantages and limitations.\n\n**Complete worked answer.**\nDeep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.\n| Comparison | Traditional ML approaches | Deep learning approaches |\n| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |\n| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |\n| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |\n| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |\nThese are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.\nDrivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.\n"},{"title":"R23 Q1d","question":["Question (4 marks). Evaluate how mini-batch size affects optimization and parallel processing.
"],"answer":["A mini-batch forms one gradient from B examples. Smaller B usually gives noisier gradients, lower activation memory and more updates per epoch. Larger B averages more examples, often uses vectorized hardware more efficiently up to resource limits, but needs more memory and gives fewer updates per epoch. N examples produce ceil(N/B) updates if the last partial batch is kept. Too-large B does not guarantee faster wall-clock progress or better generalization. Tune batch size with learning rate and resource constraints. Computation within a batch can be parallelized; successive optimizer steps remain dependent on the updated parameters.
"],"kind":"practice","id":"tee-c16-s3","search":"R23 Q1d **Question (4 marks).** Evaluate how mini-batch size affects optimization and parallel processing.\n\n**Complete worked answer.**\nA mini-batch forms one gradient from B examples. Smaller B usually gives noisier gradients, lower activation memory and more updates per epoch. Larger B averages more examples, often uses vectorized hardware more efficiently up to resource limits, but needs more memory and gives fewer updates per epoch. N examples produce ceil(N/B) updates if the last partial batch is kept. Too-large B does not guarantee faster wall-clock progress or better generalization. Tune batch size with learning rate and resource constraints. Computation within a batch can be parallelized; successive optimizer steps remain dependent on the updated parameters.\n"},{"title":"R23 Q1e","question":["Question (4 marks). Design a perceptron network for inputs 00 → 0,10 → 0,01 → 0,11 → 1; choose suitable weights.
"],"answer":["Truth table: 00→0,10→0,01→0,11→1.
","Weights and bias are to be designed.
","Choose a perceptron that implements AND and show its four outputs.
","Threshold logic. Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.
","step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.
","Yaad rakhne ka cue: Har input row ka sum nikalo, phir threshold lagao.
","Choose weights w₁ = w₂ = 1 and bias −1.5. Use output 1 for z ≥ 0.
","Step 1: multiply input 1 by its weight.
","![\"1(0)=0\"](\"assets/equations/3798c4898b2e9ea7.svg\")
Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","Step 4: add the bias once.
","![\"z=0+(-1.5)=-1.5\"](\"assets/equations/e7029152ef49c01f.svg\")
Step 5: apply the stated threshold.
","z = -1.5; output = 0 because output is 1 exactly when z ≥ 0.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","![\"1(1)=1\"](\"assets/equations/e75a800ad85088f5.svg\")
Step 3: add the two products.
","Step 4: add the bias once.
","![\"z=1+(-1.5)=-0.5\"](\"assets/equations/740ecc8b0234d38c.svg\")
Step 5: apply the stated threshold.
","z = -0.5; output = 0 because output is 1 exactly when z ≥ 0.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","Step 4: add the bias once.
","Step 5: apply the stated threshold.
","z = -0.5; output = 0 because output is 1 exactly when z ≥ 0.
","Step 1: multiply input 1 by its weight.
","Step 2: multiply input 2 by its weight.
","Step 3: add the two products.
","Step 4: add the bias once.
","![\"z=2+(-1.5)=0.5\"](\"assets/equations/174899e28e234350.svg\")
Step 5: apply the stated threshold.
","z = 0.5; output = 1 because output is 1 exactly when z ≥ 0.
","Final answer: outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND.
","A hidden layer is unnecessary for this linearly separable truth table.
","Question (10 marks). A CNN with convolution, pooling, dense and output layers receives 127 × 127 images for 5 classes and trains with Adam. Identify learned parameters and tunable hyperparameters.
"],"answer":["Convolutional kernel entries and filter biases are learned parameters. Dense weight matrices and biases are learned parameters. Pooling, flattening and ordinary activations have no learned weights. The output has 5 class scores and usually softmax with categorical cross-entropy. Adam’s m, v are running optimizer states, not extra predictive parameters.
","Hyperparameters include kernel sizes, filter counts, strides, padding, pooling configuration, dense widths, activations, learning rate, Adam β₁/β₂/ε, batch size, epochs and regularization. Architecture choices control representation capacity and resource demand; optimizer settings control update behavior. No filter counts or dense widths are specified, so a unique numerical parameter total cannot be calculated from 127 × 127 and 5 classes alone.
","Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.
","Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.
","For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.
","Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.
"],"kind":"practice","id":"tee-c16-s5","search":"R23 Q2a **Question (10 marks).** A CNN with convolution, pooling, dense and output layers receives 127 × 127 images for 5 classes and trains with Adam. Identify learned parameters and tunable hyperparameters.\n\n**Complete worked answer.**\nConvolutional kernel entries and filter biases are learned parameters. Dense weight matrices and biases are learned parameters. Pooling, flattening and ordinary activations have no learned weights. The output has 5 class scores and usually softmax with categorical cross-entropy. Adam’s m, v are running optimizer states, not extra predictive parameters.\nHyperparameters include kernel sizes, filter counts, strides, padding, pooling configuration, dense widths, activations, learning rate, Adam β₁/β₂/ε, batch size, epochs and regularization. Architecture choices control representation capacity and resource demand; optimizer settings control update behavior. No filter counts or dense widths are specified, so a unique numerical parameter total cannot be calculated from 127 × 127 and 5 classes alone.\nAdam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization.\n\n\nUse update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g.\nFor g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9.\nCompared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum.\n"},{"title":"R23 Q2b","question":["Question (10 marks). Justify ReLU over sigmoid/tanh for deep hidden layers. For predicting brand ratings on a scale of 10, choose output activation and loss. Diagnose 2% training versus 18% validation error and suggest remedies.
"],"answer":["An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.
","Sigmoid
","Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.
","Hyperbolic tangent
","The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.
","ReLU
","The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.
","Leaky ReLU
","Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.
","Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.
","Rating regression: for a continuous score, one linear output with MSE is a standard baseline. If strict 0–10 bounds are required, use 10 × sigmoid(z) and the matching regression loss, noting saturation near the endpoints. If labels are categorical rating bins, a categorical formulation would be different; state the continuous-score assumption.
","2% versus 18%: if errors use the same comparable metric and distribution, this large generalization gap strongly suggests overfitting. Use appropriate regularization, early stopping, more representative data and possibly a smaller model; inspect leakage and preprocessing too.
","Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
"],"kind":"practice","id":"tee-c16-s6","search":"R23 Q2b **Question (10 marks).** Justify ReLU over sigmoid/tanh for deep hidden layers. For predicting brand ratings on a scale of 10, choose output activation and loss. Diagnose 2% training versus 18% validation error and suggest remedies.\n\n**Complete worked answer.**\nAn activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.\n**Sigmoid**\n\nHere z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.\n**Hyperbolic tangent**\n\nThe numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.\n**ReLU**\n\nThe output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.\n**Leaky ReLU**\n\nAlpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.\nSigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.\n**Rating regression: ** for a continuous score, one linear output with MSE is a standard baseline. If strict 0–10 bounds are required, use 10 × sigmoid(z) and the matching regression loss, noting saturation near the endpoints. If labels are categorical rating bins, a categorical formulation would be different; state the continuous-score assumption.\n**2% versus 18%: ** if errors use the same comparable metric and distribution, this large generalization gap strongly suggests overfitting. Use appropriate regularization, early stopping, more representative data and possibly a smaller model; inspect leakage and preprocessing too.\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of u"},{"title":"R23 Q3a","question":["Question (10 marks). Build and train an RNN language model for “Mumbai is the financial capital of Maharashtra.” Draw the architecture and write next-token conditional probabilities at each step.
"],"answer":["A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
","Tokenize, build a vocabulary, encode tokens, pair each prefix with its next token, unroll the recurrence, apply vocabulary softmax, sum/average cross-entropy, then use BPTT and an optimizer to update shared weights. Include start/end tokens consistently.
","Explicit next-token conditionals ignoring punctuation:
","1. P(Mumbai | START).
","2. P(is | Mumbai).
","3. P(the | Mumbai is).
","4. P(financial | Mumbai is the).
","5. P(capital | Mumbai is the financial).
","6. P(of | Mumbai is the financial capital).
","7. P(Maharashtra | Mumbai is the financial capital of).
","8. P(END | Mumbai is the financial capital of Maharashtra).
","The sequence likelihood is their product; the negative log-likelihood is the sum of negative log conditionals. No numeric probabilities are determined without weights.
","Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.
","1. Initialize the hidden state and run the forward recurrence from the first input to the last.
","2. Compute the specified output loss, at the final step or across several steps.
","3. Start from the last loss and propagate derivatives backwards through output and hidden operations.
","4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.
","5. Sum all uses of each shared weight before applying the optimizer update.
","The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.
","Question (10 marks). For input [0.5,0.7,0.8], first-stage multipliers [0.3,0.4,0.6], next receiving rows [0.5,0.2,0.3], [0.3,0.3,0.3], output rows [0.3,0.2], [0.2,0.3], use ReLU in the first two stages and sigmoid at outputs. Targets [0,1], zero biases, η = 0.1. Find predictions, MSE and all four hidden-to-output weight updates.
"],"answer":["For input [0.5,0.7,0.8], first-stage multipliers [0.3,0.4,0.6], next receiving rows [0.5,0.2,0.3], [0.3,0.3,0.3], output rows [0.3,0.2], [0.2,0.3], use ReLU in the first two stages and sigmoid at outputs.
","Targets [0,1], zero biases, η = 0.1.
","Find all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Mean squared loss. Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean.
","n = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target.
","Yaad rakhne ka cue: MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","The raw feature products are [0.5 × 0.3, 0.7 × 0.4, 0.8 × 0.6] = [0.15, 0.28, 0.48]. All are positive, so ReLU leaves them unchanged. Biases are zero.
","h₁ = max(0, 0.5(0.15) + 0.2(0.28) + 0.3(0.48)) = max(0, 0.075 + 0.056 + 0.144) = 0.275.
","h₂ = max(0, 0.3(0.15) + 0.3(0.28) + 0.3(0.48)) = max(0, 0.045 + 0.084 + 0.144) = 0.273.
","z₁ = 0.3(0.275) + 0.2(0.273) = 0.1371; o₁ = σ(z₁) = 0.534221413.
","z₂ = 0.2(0.275) + 0.3(0.273) = 0.1369; o₂ = σ(z₂) = 0.534171647.
","MSE = [(o₁ − 0)² + (o₂ − 1)²]/2 = 0.251194286. There are exactly two output components, so the square derivative's factor 2 cancels the division by 2.
","δ₁ = (o₁ − 0)o₁(1 − o₁) = 0.132929724.
","δ₂ = (o₂ − 1)o₂(1 − o₂) = −0.115913140.
","A weight gradient equals its receiving output delta multiplied by its sending hidden activation. Use the same old forward pass for all four connections.
","| Connection | Gradient calculation | Gradient |
|---|---|---|
| h₁ → o₁ | 0.132929724 × 0.275 | 0.036555674 |
| h₂ → o₁ | 0.132929724 × 0.273 | 0.036289815 |
| h₁ → o₂ | −0.115913140 × 0.275 | −0.031876113 |
| h₂ → o₂ | −0.115913140 × 0.273 | −0.031644287 |
| Connection | Update at η = 0.1 | New weight |
|---|---|---|
| h₁ → o₁ | 0.3 − 0.1(0.036555674) | 0.296344433 |
| h₂ → o₁ | 0.2 − 0.1(0.036289815) | 0.196371019 |
| h₁ → o₂ | 0.2 − 0.1(−0.031876113) | 0.203187611 |
| h₂ → o₂ | 0.3 − 0.1(−0.031644287) | 0.303164429 |
Final answer: predictions [0.534221413, 0.534171647], MSE 0.251194286, and output-weight rows [0.296344433, 0.196371019], [0.203187611, 0.303164429]. The positive gradients decrease weights feeding output 1, while the negative gradients increase weights feeding output 2.
"],"kind":"practice","id":"tee-c16-s8","search":"R23 Q3b **Question (10 marks).** For input [0.5,0.7,0.8], first-stage multipliers [0.3,0.4,0.6], next receiving rows [0.5,0.2,0.3], [0.3,0.3,0.3], output rows [0.3,0.2], [0.2,0.3], use ReLU in the first two stages and sigmoid at outputs. Targets [0,1], zero biases, η = 0.1. Find predictions, MSE and all four hidden-to-output weight updates.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nFor input [0.5,0.7,0.8], first-stage multipliers [0.3,0.4,0.6], next receiving rows [0.5,0.2,0.3], [0.3,0.3,0.3], output rows [0.3,0.2], [0.2,0.3], use ReLU in the first two stages and sigmoid at outputs.\n\nTargets [0,1], zero biases, η = 0.1.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Mean squared loss.** Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean.\n\nn = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target.\n\n\n\n\n\n**Yaad rakhne ka cue:** MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n### Step 1: evaluate the first ReLU stage\nThe raw feature products are [0.5 × 0.3, 0.7 × 0.4, 0.8 × 0.6] = **[0.15, 0.28, 0.48]**. All are positive, so ReLU leaves them unchanged. Biases are zero.\n\n### Step 2: evaluate the second ReLU stage\nh₁ = max(0, 0.5(0.15) + 0.2(0.28) + 0.3(0.48)) = max(0, 0.075 + 0.056 + 0.144) = **0.275**.\n\nh₂ = max(0, 0.3(0.15) + 0.3(0.28) + 0.3(0.48)) = max(0, 0.045 + 0.084 + 0.144) = **0.273**.\n\n### Step 3: evaluate both sigmoid outputs\nz₁ = 0.3(0.275) + 0.2(0.273) = 0.1371; o₁ = σ(z₁) = **0.534221413**.\n\nz₂ = 0.2(0.275) + 0.3(0.273) = 0.1369; o₂ = σ(z₂) = **0.534171647**.\n\n### Step 4: calculate MSE and the output deltas\nMSE = [(o₁ − 0)² + (o₂ − 1)²]/2 = **0.251194286**. There are exactly two output components, so the square derivative's factor 2 cancels the division by 2.\n\nδ₁ = (o₁ − 0)o₁(1 − o₁) = **0.132929724**.\n\nδ₂ = (o₂ − 1)o₂(1 − o₂) = **−0.115913140**.\n\n### Step 5: calculate each"},{"title":"R23 Q4a","question":["Question (10 marks). Design employee face-recognition attendance using a CNN with too few training images. Explain solutions.
"],"answer":["Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.
","1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.
","2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.
","3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.
","4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.
","5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.
","6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.
","Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.
"],"kind":"practice","id":"tee-c16-s9","search":"R23 Q4a **Question (10 marks).** Design employee face-recognition attendance using a CNN with too few training images. Explain solutions.\n\n**Complete worked answer.**\nTransfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.\n1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.\n2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.\n3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.\n4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.\n5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.\n6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.\nFeature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.\n"},{"title":"R23 Q4b","question":["Question (10 marks). Estimate parameters of the AlexNet diagram. Use Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units.
"],"answer":["Use Input 227 × 227 × 3.
","Conv K = 11, filters = 96, stride = 4, padding = 0.
","Pool K = 3, stride = 2, padding = 0.
","Conv K = 5, filters = 256, stride = 1, padding = 2.
","Pool K = 3, stride = 2, padding = 0.
","Conv K = 3, filters = 384, stride = 1, padding = 1.
","Conv K = 3, filters = 384, stride = 1, padding = 1.
","Conv K = 3, filters = 256, stride = 1, padding = 1.
","Pool K = 3, stride = 2, padding = 0.
","Dense 4096 units.
","Dense 4096 units.
","Dense 1000 units.
","Find every layer shape, its parameter count and the network total.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 227 × 227 × 3. Use 96 filters of spatial size 11 × 11, stride 4, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 55 × 55 × 96.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 55 × 55 × 96. Use a 3 × 3 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 27 × 27 × 96.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 27 × 27 × 96. Use 256 filters of spatial size 5 × 5, stride 1, and padding 2 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 27 × 27 × 256.
","3. Count trainable values. Each filter receives all 96 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 27 × 27 × 256. Use a 3 × 3 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 13 × 13 × 256.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 256. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.
","3. Count trainable values. Each filter receives all 256 input channels. Its weights are reused at every spatial position.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 384. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 384.
","3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 13 × 13 × 384. Use 256 filters of spatial size 3 × 3, stride 1, and padding 1 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 13 × 13 × 256.
","3. Count trainable values. Each filter receives all 384 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 13 × 13 × 256. Use a 3 × 3 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 6 × 6 × 256.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 6 × 6 × 256. Rearrange all activation values into one vector.
","Output: a vector with 9,216 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 9,216 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 4,096 features. Output: 4,096 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 4,096 features. Output: 1,000 neurons. Every output neuron connects to every input feature and has one bias.
","| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | | 55 x 55 x 96 | 34,848 | 96 | 34,944 |
| 2 pool | | 27 x 27 x 96 | 0 | 0 | 0 |
| 3 conv | | 27 x 27 x 256 | 614,400 | 256 | 614,656 |
| 4 pool | | 13 x 13 x 256 | 0 | 0 | 0 |
| 5 conv | | 13 x 13 x 384 | 884,736 | 384 | 885,120 |
| 6 conv | | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 |
| 7 conv | | 13 x 13 x 256 | 884,736 | 256 | 884,992 |
| 8 pool | | 6 x 6 x 256 | 0 | 0 | 0 |
| 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 |
| 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 |
| 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 |
| 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 |
Parameter substitutions, in layer order:
","Layer 1: (11 x 11 x 3 + 1) x 96 = 34,944.
","Layer 3: (5 x 5 x 96 + 1) x 256 = 614,656.
","Layer 5: (3 x 3 x 256 + 1) x 384 = 885,120.
","Layer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488.
","Layer 7: (3 x 3 x 384 + 1) x 256 = 884,992.
","Layer 10: (9216 + 1) x 4096 = 37,752,832.
","Layer 11: (4096 + 1) x 4096 = 16,781,312.
","Layer 12: (4096 + 1) x 1000 = 4,097,000.
","Total: 34,944 + 614,656 + 885,120 + 1,327,488 + 884,992 + 37,752,832 + 16,781,312 + 4,097,000 = 62,378,344 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","The diagram labels its final pooled map 6 × 6 × 256; a 3 × 3, stride 2 pool from 13 × 13 produces 6. If a small printed stride label is read as 3, it conflicts with the shown 6 × 6 map. The answer uses the consistent classic stride 2 interpretation. Original grouped AlexNet counts are a different architecture assumption.
"],"kind":"practice","id":"tee-c16-s10","search":"R23 Q4b **Question (10 marks).** Estimate parameters of the AlexNet diagram. Use Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nUse Input 227 × 227 × 3.\n\nConv K = 11, filters = 96, stride = 4, padding = 0.\n\nPool K = 3, stride = 2, padding = 0.\n\nConv K = 5, filters = 256, stride = 1, padding = 2.\n\nPool K = 3, stride = 2, padding = 0.\n\nConv K = 3, filters = 384, stride = 1, padding = 1.\n\nConv K = 3, filters = 384, stride = 1, padding = 1.\n\nConv K = 3, filters = 256, stride = 1, padding = 1.\n\nPool K = 3, stride = 2, padding = 0.\n\nDense 4096 units.\n\nDense 4096 units.\n\nDense 1000 units.\n\n### To find\n\nFind every layer shape, its parameter count and the network total.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n\nUse output size \\(\\left\\lfloor\\frac{N+2P-K}{S}\\right\\rfloor+1\\), Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.\n| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |\n| 1 conv | \\(\\left\\lfloor\\frac{227+2 \\times 0-11}{4}\\right\\rfloor+1\\) | 55 x 55 x 96 | 34,848 | 96 | 34,944 |\n| 2 pool | \\(\\left\\lfloor\\frac{55-3}{2}\\right\\rfloor+1\\) | 27 x 27 x 96 | 0 | 0 | 0 |\n| 3 conv | \\(\\left\\lfloor\\frac{27+2 \\times 2-5}{1}\\right\\rfloor+1\\) | 27 x 27 x 256 | 614,400 | 256 | 614,656 |\n| 4 pool | \\(\\left\\lfloor\\frac{27-3}{2}\\right\\rfloor+1\\) | 13 x 13 x 256 | 0 | 0 | 0 |\n| 5 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 384 | 884,736 | 384 | 885,120 |\n| 6 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 |\n| 7 conv | \\(\\left\\lfloor\\frac{13+2 \\times 1-3}{1}\\right\\rfloor+1\\) | 13 x 13 x 256 | 884,736 | 256 | 884,992 |\n| 8 pool | \\(\\left\\lfloor\\frac{13-3}{2}\\right\\rfloor+1\\) | 6 x 6 x 256 | 0 | 0 | 0 |\n| 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 |\n| 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 |\n| 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 |\n| 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 |\n**Parameter substitutions, in layer order: **\nLayer 1: (11 x 11 x 3 + 1) x 96 = 34,944.\nLayer 3: (5 x 5 x 96 + 1) x 256 = 614,656.\nLayer 5: (3 x 3 x 256 + 1) x 384 = 885,120.\nLayer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488.\nLayer 7: (3 x 3 x 384 + 1) x 256 = 884,992.\nLayer 10: (9216 + 1) x 4096 = 3"},{"title":"R23 Q5a","question":["Question (10 marks). An RNN does not improve because vanishing gradients prevent useful parameter updates. Suggest a solution and explain required architectural changes.
"],"answer":["Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.
","1. Initialize the hidden state and run the forward recurrence from the first input to the last.
","2. Compute the specified output loss, at the final step or across several steps.
","3. Start from the last loss and propagate derivatives backwards through output and hidden operations.
","4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.
","5. Sum all uses of each shared weight before applying the optimizer update.
","The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.
","Replace the vanilla cell with an LSTM or GRU while preserving a suitable sequence output. An LSTM introduces cell state and learned forget, input, output gates; a GRU introduces reset/update gates and a gated hidden-state mixture. These paths make retaining long context easier. Tune initialization and learning rate and inspect gradients.
","An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.
","The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.
","In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.
"],"kind":"practice","id":"tee-c16-s11","search":"R23 Q5a **Question (10 marks).** An RNN does not improve because vanishing gradients prevent useful parameter updates. Suggest a solution and explain required architectural changes.\n\n**Complete worked answer.**\nBackpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.\n1. Initialize the hidden state and run the forward recurrence from the first input to the last.\n2. Compute the specified output loss, at the final step or across several steps.\n3. Start from the last loss and propagate derivatives backwards through output and hidden operations.\n4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.\n5. Sum all uses of each shared weight before applying the optimizer update.\n\nThe tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.\n\nReplace the vanilla cell with an LSTM or GRU while preserving a suitable sequence output. An LSTM introduces cell state and learned forget, input, output gates; a GRU introduces reset/update gates and a gated hidden-state mixture. These paths make retaining long context easier. Tune initialization and learning rate and inspect gradients.\nAn LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.\n\n\n\n\nThe forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.\nIn language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.\n"},{"title":"R23 Q5b","question":["Question (10 marks). Justify multiple ANN layers. Design a fraudulent/non-fraudulent transaction classifier for 10,000 samples with 4 features; choose layers/units and count parameters.
"],"answer":["10,000 samples; 4 features per sample; two target classes: fraudulent/non-fraudulent.
","Hidden layer sizes are design choices, not fixed in the question.
","Explain multiple layers, propose an appropriate classifier and count parameters for that explicit design.
","Dense-layer parameters. Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.
","n_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.
","Yaad rakhne ka cue: Har output ke liye saare inputs plus ek bias.
","A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator.
","For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1.
","For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR.
","A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map.
","Chosen design: 4 → 8 → 4 → 1 with ReLU hidden layers, one sigmoid output and binary cross-entropy. Hidden widths are explicit design assumptions.
","Architecture: 4 → 8 → 4 → 1. Each non-input neuron has one bias.
","Step 1: count connections.
","![\"4(8)=32\"](\"assets/equations/69e92d1e9a3f1ea7.svg\")
Step 2: count receiving-neuron biases.
","There are 8 receiving neurons, so 8 biases.
","Step 3: add weights and biases.
","![\"32+8=40\"](\"assets/equations/b223db7a6cdbf384.svg\")
Step 1: count connections.
","![\"8(4)=32\"](\"assets/equations/e2ee029f7f3db879.svg\")
Step 2: count receiving-neuron biases.
","There are 4 receiving neurons, so 4 biases.
","Step 3: add weights and biases.
","![\"32+4=36\"](\"assets/equations/c052c15ffd718757.svg\")
Step 1: count connections.
","![\"4(1)=4\"](\"assets/equations/6d1e399cd2e46091.svg\")
Step 2: count receiving-neuron biases.
","There are 1 receiving neurons, so 1 biases.
","Step 3: add weights and biases.
","Total = 81. Input values and activations are not trainable parameters.
","Use stratified train/validation/test partitions; handle class imbalance using appropriate training weights and evaluate precision/recall or a suitable decision threshold alongside accuracy. More layers are useful only if they learn needed nonlinear structure; the problem statement does not prescribe one uniquely correct architecture.
","Step 1: calculate hidden OR preactivation.
","Step 2: apply its threshold.
","Step 3: calculate hidden AND preactivation.
","Step 4: apply its threshold.
","Step 5: calculate XOR output preactivation.
","Step 6: apply the XOR threshold.
","Step 7: calculate XNOR output preactivation.
","Step 8: apply the XNOR threshold.
","Step 1: calculate hidden OR preactivation.
","Step 2: apply its threshold.
","Step 3: calculate hidden AND preactivation.
","Step 4: apply its threshold.
","Step 5: calculate XOR output preactivation.
","Step 6: apply the XOR threshold.
","Step 7: calculate XNOR output preactivation.
","Step 8: apply the XNOR threshold.
","Step 1: calculate hidden OR preactivation.
","Step 2: apply its threshold.
","Step 3: calculate hidden AND preactivation.
","Step 4: apply its threshold.
","Step 5: calculate XOR output preactivation.
","Step 6: apply the XOR threshold.
","Step 7: calculate XNOR output preactivation.
","Step 8: apply the XNOR threshold.
","Step 1: calculate hidden OR preactivation.
","Step 2: apply its threshold.
","Step 3: calculate hidden AND preactivation.
","Step 4: apply its threshold.
","Step 5: calculate XOR output preactivation.
","Step 6: apply the XOR threshold.
","Step 7: calculate XNOR output preactivation.
","Step 8: apply the XNOR threshold.
","Final answer: XOR = [0,1,1,0]; XNOR = [1,0,0,1] for 00,01,10,11. The hidden nonlinear units make this possible.
"],"kind":"practice","id":"tee-c16-s12","search":"R23 Q5b **Question (10 marks).** Justify multiple ANN layers. Design a fraudulent/non-fraudulent transaction classifier for 10,000 samples with 4 features; choose layers/units and count parameters.\n\n**Complete worked answer.**\n\n\n\n### Given\n\n10,000 samples; 4 features per sample; two target classes: fraudulent/non-fraudulent.\n\nHidden layer sizes are design choices, not fixed in the question.\n\n### To find\n\nExplain multiple layers, propose an appropriate classifier and count parameters for that explicit design.\n\n### Formula — which one and why\n\n**Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers.\n\nn_in = incoming features; n_out = receiving neurons. The input layer itself has no learned receiving weights or biases.\n\n\n\n**Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias.\n\n### Solution — step by step\n\n\n\n\n\n\nA single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator.\n\nFor binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1.\nFor XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR.\nA multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map.\n**Chosen design: **4 → 8 → 4 → 1 with ReLU hidden layers, one sigmoid output and binary cross-entropy. Hidden widths are explicit design assumptions.\nArchitecture: 4 → 8 → 4 → 1. Each non-input neuron has one bias.\n### Layer 4 → 8\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 8 receiving neurons, so 8 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 8 → 4\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 4 receiving neurons, so 4 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n### Layer 4 → 1\n\n**Step 1: count connections.**\n\n\n\n\n\n\n\n**Step 2: count receiving-neuron biases.**\n\nThere are 1 receiving neurons, so 1 biases.\n\n**Step 3: add weights and biases.**\n\n\n\n\n\n\n\n\n**Total = 81**. Input values and activations are not trainable parameters.\nUse stratified train/validation/test partitions; handle class imbalance using appropriate training weights and evaluate precision/recall or a suitable decision threshold alongside accuracy. More layers are useful only if they learn needed nonlinear structure; the problem statement does not prescribe one uniquely correct architecture.\n\n\n### Verify input (0,0)\n\n**Step 1: calculate hidden OR preactivation.**\n\n\n\n\n\n\n\n**Step 2: apply its threshold.**\n\n\n\n\n\n\n\n**Step 3: calculate hidden AND preactivation.**\n\n\n\n\n\n\n\n**Step 4: apply its threshold.**\n\n\n\n\n\n\n\n**Step 5: calculate XOR output preactivation.**\n\n\n\n\n\n\n\n**Step 6: apply the XOR threshold.**\n\n\n\n\n\n\n\n**Step 7: calculate XNOR output preactivation.**\n\n\n\n\n\n\n\n**Step 8: apply the XNOR threshold.**\n\n\n\n\n\n\n\n\n### Verify input (0"},{"title":"R23 Q6a","question":["Question (10 marks). Suggest and justify remedies when (1) loss does not minimize after many epochs, (2) gradients are noisy and loss oscillates, (3) loss falls then rises because updates miss a minimum.
"],"answer":["The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.
","Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.
","Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.
","Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.
","Scenario 1: check correctness and gradient magnitudes, then tune learning rate, activation, initialization or optimizer. Scenario 2: consider an appropriate larger mini-batch or momentum; distinguish ordinary stochastic noise from divergence. Scenario 3: reduce learning rate or use a schedule; momentum/Nesterov can help but excessive momentum can worsen overshoot. Judge training and validation curves separately.
"],"kind":"practice","id":"tee-c16-s13","search":"R23 Q6a **Question (10 marks).** Suggest and justify remedies when (1) loss does not minimize after many epochs, (2) gradients are noisy and loss oscillates, (3) loss falls then rises because updates miss a minimum.\n\n**Complete worked answer.**\nThe learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable.\n\nBatch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory.\nMomentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories.\nDiagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD.\n**Scenario 1: ** check correctness and gradient magnitudes, then tune learning rate, activation, initialization or optimizer. **Scenario 2: ** consider an appropriate larger mini-batch or momentum; distinguish ordinary stochastic noise from divergence. **Scenario 3: ** reduce learning rate or use a schedule; momentum/Nesterov can help but excessive momentum can worsen overshoot. Judge training and validation curves separately.\n"},{"title":"R23 Q6b","question":["Question (10 marks). A 28 × 28 × 192 input is convolved with 32 same-padded 5 × 5 filters to produce 28 × 28 × 32. Count computations. What happens if a 1 × 1 convolution precedes it? The reduction-channel count is not supplied.
"],"answer":["Input = 28 × 28 × 192; 32 same-padded 5 × 5 filters.
","A preceding 1 × 1 convolution would reduce to C channels.
","C is not supplied, so first give the symbolic bottleneck count; 16 channels is only an illustrative assumption.
","Calculate direct cost and express bottleneck cost in C; explain when the bottleneck saves computation.
","Multiply–accumulate count. Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.
","H_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.
","Yaad rakhne ka cue: Har output cell ka kaam × kitne output cells.
","Step 2: direct path.
","output stays 28 × 28 with 32 channels.
","Step 3: direct path.
","MACs = 28 × 28 × 192 × 32 × 5 × 5 = 120,422,400.
","Step 4: bottleneck path with 16 reduction channels.
","1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs.
","Step 5: bottleneck path with 16 reduction channels.
","The 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs.
","Step 6: bottleneck path with 16 reduction channels.
","Total = 12,443,648.
","Step 7: saving.
","120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = 89.666667%. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = 777,728r
","Step 8: saving.
","there is no unique numeric bottleneck cost until r is chosen.
","The 16-channel bottleneck is an explicitly chosen illustrative value, consistent with a related paper; the symbolic 777,728 r answer is required when r has not been specified.
","Step 9: calculate this quantity.
","An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","Step 10: calculate this quantity.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
"],"kind":"practice","id":"tee-c16-s14","search":"R23 Q6b **Question (10 marks).** A 28 × 28 × 192 input is convolved with 32 same-padded 5 × 5 filters to produce 28 × 28 × 32. Count computations. What happens if a 1 × 1 convolution precedes it? The reduction-channel count is not supplied.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput = 28 × 28 × 192; 32 same-padded 5 × 5 filters.\n\nA preceding 1 × 1 convolution would reduce to C channels.\n\nC is not supplied, so first give the symbolic bottleneck count; 16 channels is only an illustrative assumption.\n\n### To find\n\nCalculate direct cost and express bottleneck cost in C; explain when the bottleneck saves computation.\n\n### Formula — which one and why\n\n**Multiply–accumulate count.** Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.\n\nH_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells.\n\n### Solution — step by step\n\n\n\n\n\n\n\n\n\n\n### Direct path\n\n**Step 2: direct path.**\n\noutput stays 28 × 28 with 32 channels.\n\n**Step 3: direct path.**\n\nMACs = 28 × 28 × 192 × 32 × 5 × 5 = **120,422,400**.\n\n### Bottleneck path with 16 reduction channels\n\n**Step 4: bottleneck path with 16 reduction channels.**\n\n1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs.\n\n**Step 5: bottleneck path with 16 reduction channels.**\n\nThe 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs.\n\n**Step 6: bottleneck path with 16 reduction channels.**\n\nTotal = **12,443,648**.\n\n### Saving\n\n**Step 7: saving.**\n\n120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = **89.666667%**. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = **777,728r**\n\n**Step 8: saving.**\n\nthere is no unique numeric bottleneck cost until r is chosen.\n\nThe 16-channel bottleneck is an explicitly chosen illustrative value, consistent with a related paper; the symbolic 777,728 r answer is required when r has not been specified.\n\n**Step 9: calculate this quantity.**\n\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\n\n\n**Step 10: calculate this quantity.**\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\n\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kern"},{"title":"R23 Q7a","question":["Question (10 marks). Process with LSTM: “XYZ was very brave soldier. He was a part of the Kargil war and laid down his life fighting against the enemy. The president of India honored patriotic ___.” Explain the need for its gates.
"],"answer":["An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.
","The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.
","In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.
","The model can retain the earlier entity XYZ and the soldier context while updating intervening events. At the final prefix a vocabulary softmax predicts a suitable next token; “soldier” is a plausible completion. The exact output is not uniquely determined by the text or architecture alone. The forget gate retains useful entity content, the input gate adds the war/honor context, and the output gate exposes relevant state for prediction. Do not invent numerical gate values when no weights are provided.
"],"kind":"practice","id":"tee-c16-s15","search":"R23 Q7a **Question (10 marks).** Process with LSTM: “XYZ was very brave soldier. He was a part of the Kargil war and laid down his life fighting against the enemy. The president of India honored patriotic ___.” Explain the need for its gates.\n\n**Complete worked answer.**\nAn LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.\n\n\n\n\nThe forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.\nIn language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.\nThe model can retain the earlier entity XYZ and the soldier context while updating intervening events. At the final prefix a vocabulary softmax predicts a suitable next token; “soldier” is a plausible completion. The exact output is not uniquely determined by the text or architecture alone. The forget gate retains useful entity content, the input gate adds the war/honor context, and the output gate exposes relevant state for prediction. Do not invent numerical gate values when no weights are provided.\n"},{"title":"R23 Q7b","question":["Question (10 marks). Training performance is good but unseen-data performance degrades. Identify the issue and explain ways to improve it.
"],"answer":["Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
","A comparable-distribution train/unseen gap suggests overfitting. Also check whether the new data have shifted, preprocessing differs or labels are inconsistent. Select regularization using validation performance, not final-test tuning. Improvement must be demonstrated on representative held-out examples.
"],"kind":"practice","id":"tee-c16-s16","search":"R23 Q7b **Question (10 marks).** Training performance is good but unseen-data performance degrades. Identify the issue and explain ways to improve it.\n\n**Complete worked answer.**\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\nA comparable-distribution train/unseen gap suggests overfitting. Also check whether the new data have shifted, preprocessing differs or labels are inconsistent. Select regularization using validation performance, not final-test tuning. Improvement must be demonstrated on representative held-out examples.\n\n\n"}]},{"n":17,"title":"Numerical practice from direct to mixed","sections":[{"title":"Start here","blocks":["These are authored learning exercises, not additional PYQs. Their purpose is to practise the methods before attempting the full examination questions in the past exam questions under Solved Questions. Any practice marks describe an exercise's suggested size, not marks or frequency observed in a past paper. For TEE priorities, start with the December 2025 and February 2026 paper question forms; use extra variants to repair a method that is still unclear.
","These 52 original drills supplement every solved PYQ. They are not predicted exam questions. Use a blank page: copy the question, cover the answer, choose the formula, solve, then compare. After an error, redo the same type with the answer covered later that day and once the next morning. Reading or copying a solution once is not the same as being able to retrieve its method.
","For TEE, complete every family, including drills 42–45 and 50 on gates and BPTT. A ten-minute block can contain one long numerical or three short ones. Repeat a solved family later with its answer covered before counting it as reliable.
"],"kind":"read","id":"tee-c17-intro"},{"title":"How to solve a mixed numerical without guessing","blocks":["A mixed question is a chain of familiar operations. First circle the requested final quantity. Underline the given activation, loss, padding, stride, threshold, learning rate and any fixed parameters. Draw a small box for each stage. Write the output of one box as the input to the next. Never skip a stage because its title looks unfamiliar.
","If the question says output or prediction: do only the forward chain, unless training is separately requested. For a neuron, multiply and add before activating. For a CNN, determine spatial shapes before flattening. For an RNN, carry the previous state into the next time step.
","If it says train or update: finish the entire old forward pass, calculate the stated loss derivative, move backward, store every gradient, then update together. If there are several output branches, add their effects at the shared hidden unit. For a batch, follow the stated sum or mean loss before updating.
","If it mixes dropout and a loss: apply the given training mask and scaling first; then predict from the resulting activations. The gradient to a dropped path is zero for that mask. If it also specifies L2, add its gradient to the data gradient before the optimizer step. If it also specifies momentum or Adam, carry their stored states as well as the parameter.
","If it mixes convolution, pooling and a dense classifier: output shape → next layer input shape → flatten length → dense scores → activation → loss. Count learned parameters separately from output values. Pooling and flattening have zero learned parameters in these questions.
","If it changes a familiar problem: an extra input adds another product; an extra output adds another backward path; a new channel adds another kernel sum; a new time step adds another shared-state calculation. The method stays the same. The drills below deliberately change signs, targets, loss scales and layer combinations.
","Your completion rule: solve a representative problem today without the answer, repeat it after several hours, then solve a changed-number or mixed version tomorrow. Mark the exact line where your reasoning failed. Correcting that line is what builds reliable recall; reading an open solution does not test it.
"],"question":[],"answer":[],"kind":"read","id":"tee-c17-s0","search":"How to solve a mixed numerical without guessing A mixed question is a chain of familiar operations. First circle the requested final quantity. Underline the given activation, loss, padding, stride, threshold, learning rate and any fixed parameters. Draw a small box for each stage. Write the output of one box as the input to the next. Never skip a stage because its title looks unfamiliar.\n**If the question says output or prediction:** do only the forward chain, unless training is separately requested. For a neuron, multiply and add before activating. For a CNN, determine spatial shapes before flattening. For an RNN, carry the previous state into the next time step.\n**If it says train or update:** finish the entire old forward pass, calculate the stated loss derivative, move backward, store every gradient, then update together. If there are several output branches, add their effects at the shared hidden unit. For a batch, follow the stated sum or mean loss before updating.\n**If it mixes dropout and a loss:** apply the given training mask and scaling first; then predict from the resulting activations. The gradient to a dropped path is zero for that mask. If it also specifies L2, add its gradient to the data gradient before the optimizer step. If it also specifies momentum or Adam, carry their stored states as well as the parameter.\n**If it mixes convolution, pooling and a dense classifier:** output shape → next layer input shape → flatten length → dense scores → activation → loss. Count learned parameters separately from output values. Pooling and flattening have zero learned parameters in these questions.\n**If it changes a familiar problem:** an extra input adds another product; an extra output adds another backward path; a new channel adds another kernel sum; a new time step adds another shared-state calculation. The method stays the same. The drills below deliberately change signs, targets, loss scales and layer combinations.\n**Your completion rule:** solve a representative problem today without the answer, repeat it after several hours, then solve a changed-number or mixed version tomorrow. Mark the exact line where your reasoning failed. Correcting that line is what builds reliable recall; reading an open solution does not test it.\n"},{"title":"Drill 1 Signs and brackets","question":["Question. Evaluate 2−0.1 × (−3), then (2−0.1) × (−3). Explain why they differ.
"],"answer":["Expressions: 2−0.1×(−3) and (2−0.1)×(−3).
","Evaluate both and explain the effect of grouping.
","Order of operations. Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.
","A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.
","Yaad rakhne ka cue: Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.
","A minus sign can mean subtraction or a negative number. Brackets tell us which part belongs together. These two expressions have different brackets, so they describe different calculations.
","Do brackets first, then multiplication, then addition or subtraction. Subtracting a negative number means adding its positive size.
","Step 1: First expression: multiply before subtracting.
","First multiply: a positive number times a negative number is negative.
","Put this complete result back into the expression. Keep its negative sign inside brackets.
","Step 2: Second expression: finish the bracket first.
","Now multiply the whole bracket result by negative 3.
","Final answer: first expression = 2.3; second expression = −5.7. In a weight update, multiply learning rate by gradient first, then subtract that product from the old weight.
"],"kind":"practice","id":"tee-c17-s1","search":"Drill 1 Signs and brackets **Question.** Evaluate 2−0.1 × (−3), then (2−0.1) × (−3). Explain why they differ.\n\n**Full solution.**\n\n\n\n### Given\n\nExpressions: 2−0.1×(−3) and (2−0.1)×(−3).\n\n### To find\n\nEvaluate both and explain the effect of grouping.\n\n### Formula — which one and why\n\n**Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction.\n\nA minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition.\n\n\n\n**Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the calculation\nA minus sign can mean subtraction or a negative number. Brackets tell us which part belongs together. These two expressions have different brackets, so they describe different calculations.\n### Rule used\nDo brackets first, then multiplication, then addition or subtraction. Subtracting a negative number means adding its positive size.\n**Step 1: First expression: multiply before subtracting.**\n\nFirst multiply: a positive number times a negative number is negative.\n\nPut this complete result back into the expression. Keep its negative sign inside brackets.\n\n**Step 2: Second expression: finish the bracket first.**\n\nNow multiply the whole bracket result by negative 3.\n\n**Final answer:** first expression = 2.3; second expression = −5.7. In a weight update, multiply learning rate by gradient first, then subtract that product from the old weight.\n"},{"title":"Drill 2 Half sum versus mean","question":["Question. Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE.
"],"answer":["Predictions = [0.2, 0.7, 0.4].
","Targets = [0, 1, 1]; number of outputs n = 3.
","Find SSE, half SSE, MSE and the prediction gradients requested in the question.
","Squared-error objectives. Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.
","ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.
","Yaad rakhne ka cue: Pehle error square; phir question ke hisaab se sum, half ya mean.
","There are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i.
","SSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively.
","| Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual |
|---|---|---|---|---|
| 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 |
| 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 |
| 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 |
Square each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel.
","For half SSE, the denominator is 2:
","For MSE, the denominator is the number of pairs, n = 3:
","Final answer: SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question.
"],"kind":"practice","id":"tee-c17-s2","search":"Drill 2 Half sum versus mean \n**Question.** Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE.\n\n**Full solution.**\n\n\n\n### Given\n\nPredictions = [0.2, 0.7, 0.4].\n\nTargets = [0, 1, 1]; number of outputs n = 3.\n\n### To find\n\nFind SSE, half SSE, MSE and the prediction gradients requested in the question.\n\n### Formula — which one and why\n\n**Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions.\n\nŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean.\n\n### Solution — step by step\n\n\n\n\n\n### Given values and loss definitions\nThere are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i.\n\n\n\nSSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively.\n\n### Step 1: calculate each residual and its square\n| Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual |\n| 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 |\n| 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 |\n| 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 |\n\n### Step 2: sum the squared residuals\n\nSquare each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel.\n\n### Step 3: apply each requested normalization\nFor half SSE, the denominator is 2:\n\nFor MSE, the denominator is the number of pairs, n = 3:\n\n\n**Final answer:** SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question.\n\n"},{"title":"Drill 3 Natural log and probability","question":["Question. Compute −ln(0.9),−ln(0.1), and .
Probabilities: 0.9 and 0.1.
","Exponential expression: exp(−ln 2).
","Find −ln(0.9), −ln(0.1) and exp(−ln 2).
","Natural logarithms. The natural logarithm reverses the exponential. A negative log turns a small correct-class probability into a larger positive loss.
","ln uses base e ≈ 2.71828. exp(a) means e raised to the power a. Keep the minus outside ln when the question says −ln(p).
","Yaad rakhne ka cue: exp banata hai power; ln us power ko wapas nikaalta hai.
","The natural logarithm ln tells us the power of e needed to obtain a number. It reverses the exponential operation. A calculator evaluates ln; it is not ordinary division.
","When these are correct-class probabilities, 0.1 represents a worse prediction than 0.9, so its negative-log loss is larger.
","Final answers: 0.105360516, 2.302585093 and 0.5. Use the ln key; base-10 log gives different first and second answers.
"],"kind":"practice","id":"tee-c17-s3","search":"Drill 3 Natural log and probability **Question.** Compute −ln(0.9),−ln(0.1), and \\(e^{-\\ln 2}\\).\n\n**Full solution.**\n\n\n\n### Given\n\nProbabilities: 0.9 and 0.1.\n\nExponential expression: exp(−ln 2).\n\n### To find\n\nFind −ln(0.9), −ln(0.1) and exp(−ln 2).\n\n### Formula — which one and why\n\n**Natural logarithms.** The natural logarithm reverses the exponential. A negative log turns a small correct-class probability into a larger positive loss.\n\nln uses base e ≈ 2.71828. exp(a) means e raised to the power a. Keep the minus outside ln when the question says −ln(p).\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** exp banata hai power; ln us power ko wapas nikaalta hai.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe natural logarithm ln tells us the power of e needed to obtain a number. It reverses the exponential operation. A calculator evaluates ln; it is not ordinary division.\n\n\n### Step 1: calculate the first logarithm, then its minus sign\n\n\n### Step 2: calculate the second logarithm\n\n\nWhen these are correct-class probabilities, 0.1 represents a worse prediction than 0.9, so its negative-log loss is larger.\n### Step 3: simplify the exponential using the inverse rule\n\n**Final answers:** 0.105360516, 2.302585093 and 0.5. Use the ln key; base-10 log gives different first and second answers.\n"},{"title":"Drill 4 Dot product and bias","question":["Question. Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU.
"],"answer":["Inputs x = [2, −1, 0.5].
","Weights w = [0.3, 0.4, −0.2]; bias b = 0.1.
","Activation: ReLU.
","Find the weighted sum z and ReLU output.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero.
","Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output.
","Final answer: weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive.
"],"kind":"practice","id":"tee-c17-s4","search":"Drill 4 Dot product and bias **Question.** Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU.\n\n**Full solution.**\n\n\n\n### Given\n\nInputs x = [2, −1, 0.5].\n\nWeights w = [0.3, 0.4, −0.2]; bias b = 0.1.\n\nActivation: ReLU.\n\n### To find\n\nFind the weighted sum z and ReLU output.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nA dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero.\n\nHere x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output.\n### Step 1: multiply the matching pairs\n\n\n\n### Step 2: add all products and add bias once\n\n\n### Step 3: apply the activation\n\n**Final answer:** weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive.\n"},{"title":"Drill 5 Matrix rows receive inputs","question":["Question. For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU.
","W = .
Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ.
","Activation: ReLU on each result.
","Find Wx+b and the activated output.
","Row–column multiplication. Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.
","Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.
","Yaad rakhne ka cue: Ek row × poora input column = ek receiving neuron ka answer.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row.
","The shape is 2 rows by 1 column: one output for each receiving row.
"],"kind":"practice","id":"tee-c17-s5","search":"Drill 5 Matrix rows receive inputs **Question.** For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU.\n\n\n\n**Full solution.**\n\n\n\n### Given\n\nW = .\n\nInput column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ.\n\nActivation: ReLU on each result.\n\n### To find\n\nFind Wx+b and the activated output.\n\n### Formula — which one and why\n\n**Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias.\n\nWⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns.\n\n\n\n**Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formula\nEach row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row.\n\n\n### Step 1: calculate receiving neuron 1 from row 1\n\n\n### Step 2: calculate receiving neuron 2 from row 2\n\n\n### Step 3: apply ReLU to each entry\n\n\nThe shape is 2 rows by 1 column: one output for each receiving row.\n"},{"title":"Drill 6 Chain rule on a tiny loss","question":["Question. Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1.
"],"answer":["Model: o = wx+b; loss L = ½(o−y)².
","x = 3; y = 1; w = 0.2; b = 0.1.
","Learning rate η = 0.1.
","Find ∂L/∂w, ∂L/∂b and the updated weight and bias.
","Chain rule for a linear neuron. The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.
","z = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.
","Yaad rakhne ka cue: Ek path mein multiply; alag paths mil rahe hon toh add.
","The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects.
","The learning rate η says how much of the gradient to subtract.
","Final answer: weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass.
"],"kind":"practice","id":"tee-c17-s6","search":"Drill 6 Chain rule on a tiny loss **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1.\n\n**Full solution.**\n\n\n\n### Given\n\nModel: o = wx+b; loss L = ½(o−y)².\n\nx = 3; y = 1; w = 0.2; b = 0.1.\n\nLearning rate η = 0.1.\n\n### To find\n\nFind ∂L/∂w, ∂L/∂b and the updated weight and bias.\n\n### Formula — which one and why\n\n**Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x.\n\nz = prediction; y = target; e = z−y; g_w and g_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate.\n\n\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects.\n\n\n\nThe learning rate η says how much of the gradient to subtract.\n\n### Step 1: obtain the current prediction and error\n\n\n\n### Step 2: calculate both gradients from this same prediction\n\n\n### Step 3: update the weight\n\n### Step 4: update the bias\n\n**Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass.\n"},{"title":"Drill 7 Sigmoid with binary cross entropy","question":["Question. A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z.
"],"answer":["Logit z = 1; binary target y = 1.
","Output: sigmoid; loss: binary cross-entropy.
","Find probability, BCE and the loss derivative with respect to z.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Binary cross-entropy. For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.
","y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.
","Yaad rakhne ka cue: Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).
","A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome.
","For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form:
","The second term has multiplier zero, so it contributes zero.
","Final answer: probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice.
"],"kind":"practice","id":"tee-c17-s7","search":"Drill 7 Sigmoid with binary cross entropy **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z.\n\n**Full solution.**\n\n\n\n### Given\n\nLogit z = 1; binary target y = 1.\n\nOutput: sigmoid; loss: binary cross-entropy.\n\n### To find\n\nFind probability, BCE and the loss derivative with respect to z.\n\n### Formula — which one and why\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.\n\ny is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.\n\n\n\n\n\n**Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nA logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome.\n\n\nFor sigmoid combined with this loss, the gradient with respect to the logit has the simplified form:\n\n### Step 1: convert logit 1 into probability\n\n\n\n### Step 2: substitute the target in the full loss formula\n\nThe second term has multiplier zero, so it contributes zero.\n\n### Step 3: calculate the combined gradient\n\n**Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice.\n"},{"title":"Drill 8 Activation contrast","question":["Question. For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0.
"],"answer":["Inputs z = [−3,0,2].
","Leaky-ReLU negative slope α = 0.1.
","For the sigmoid derivative, use z = 0.
","Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","ReLU compares the input with zero and returns the larger value.
","Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.
","ReLU output = [0, 0, 2].
","The zero and positive entries remain 0 and 2. Leaky ReLU output = [−0.3, 0, 2].
","The sigmoid value is 0.5 but its slope is 0.25. They are different quantities.
"],"kind":"practice","id":"tee-c17-s8","search":"Drill 8 Activation contrast **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0.\n\n**Full solution.**\n\n\n\n### Given\n\nInputs z = [−3,0,2].\n\nLeaky-ReLU negative slope α = 0.1.\n\nFor the sigmoid derivative, use z = 0.\n\n### To find\n\nFind ReLU and Leaky-ReLU outputs and the sigmoid slope at zero.\n\n### Formula — which one and why\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n### Solution — step by step\n\n\n\n\n\n### Formulas and their meanings\nReLU compares the input with zero and returns the larger value.\n\nLeaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative.\n\n### Step 1: apply ReLU separately to all three inputs\n\n**ReLU output = [0, 0, 2].**\n### Step 2: use α = 0.1 only on the negative input\n\nThe zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].**\n### Step 3: calculate sigmoid before its derivative\n\n\n\nThe sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities.\n"},{"title":"Drill 9 Softmax and true class loss","question":["Question. Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy.
"],"answer":["Logits = [1,2,0].
","One-hot target = [0,1,0]: class 2 is correct.
","Find all three softmax probabilities and CCE.
","Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.
","zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.
","Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.
","Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.
","yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.
","Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.
","The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities.
","Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials.
","The largest of [1,2,0] is 2.
","The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability.
","Final answer: probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606.
"],"kind":"practice","id":"tee-c17-s9","search":"Drill 9 Softmax and true class loss **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy.\n\n**Full solution.**\n\n\n\n### Given\n\nLogits = [1,2,0].\n\nOne-hot target = [0,1,0]: class 2 is correct.\n\n### To find\n\nFind all three softmax probabilities and CCE.\n\n### Formula — which one and why\n\n**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.\n\nzⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.\n\n**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.\n\nyⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.\n\n\n\n\n\n**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities.\n\nSubtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials.\n\n### Step 1: subtract the largest score\nThe largest of [1,2,0] is 2.\n\n### Step 2: calculate the three exponentials\n\n### Step 3: add them to get one common denominator\n\n### Step 4: divide each exponential by this denominator\n\n\n\nThe probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability.\n### Step 5: use the probability of the correct class in CCE\n\n\n**Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606.\n"},{"title":"Drill 10 Sigmoid neuron gradient","question":["Question. x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update.
"],"answer":["x = 2; w = 0.5; b = 0; target y = 1.
","Activation: sigmoid; loss: half squared error.
","Learning rate η = 0.1.
","Find the output, delta, weight/bias gradients and one update.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly.
","The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z.
","The weight receives x = 2. The bias effectively receives the constant input 1.
","Final answer: new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here.
"],"kind":"practice","id":"tee-c17-s10","search":"Drill 10 Sigmoid neuron gradient **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update.\n\n**Full solution.**\n\n\n\n### Given\n\nx = 2; w = 0.5; b = 0; target y = 1.\n\nActivation: sigmoid; loss: half squared error.\n\nLearning rate η = 0.1.\n\n### To find\n\nFind the output, delta, weight/bias gradients and one update.\n\n### Formula — which one and why\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nHere the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly.\n\nThe symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z.\n\n\n\n### Step 1: weighted sum and sigmoid\n\n\n### Step 2: error and activation slope\n\n\n### Step 3: multiply the linked effects to obtain delta\n\n### Step 4: multiply delta by the input to each parameter\nThe weight receives x = 2. The bias effectively receives the constant input 1.\n\n\n### Step 5: subtract learning rate times each gradient\n\n\n**Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here.\n"},{"title":"Drill 11 One perceptron learning update","question":["Question. Binary perceptron, w = [0,0], b = 0, x = [1,1], target 1, η = 0.5. Use output 1 if z>0 else 0. Perform one update.
"],"answer":["w = [0,0]; b = 0; x = [1,1]; target t = 1.
","Learning rate η = 0.5.
","Output 1 only when z > 0; otherwise output 0.
","Perform one update and report new weights and bias.
","Perceptron learning rule. Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters.
","t = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question.
","Yaad rakhne ka cue: Prediction nikalo; target−prediction se correction nikalo.
","A perceptron first computes a weighted sum z and then applies the exact threshold rule in the question. This question uses strictly greater than zero, so z = 0 produces output 0.
","![\"z=w_1x_1+w_2x_2+b,\\qquad](\"assets/equations/10b03fc6ae774e47.svg\")
The perceptron learning rule uses target minus output, written e below. This is a discrete perceptron rule, not the sigmoid backpropagation derivative.
","![\"e=t-o,\\qquad](\"assets/equations/cd56f1cf8ec70869.svg\")
![\"z=0\\times1+0\\times1+0=0,\\qquad](\"assets/equations/6eff381fa8ae5b02.svg\")
![\"e=1-0=1\"](\"assets/equations/7a3b896434f77ddd.svg\")
![\"w_{1,\\mathrm{new}}=0+0.5\\times1\\times1=0.5\"](\"assets/equations/61526c5e9b2167a4.svg\")
![\"w_{2,\\mathrm{new}}=0+0.5\\times1\\times1=0.5\"](\"assets/equations/1e2f50640ebb5c1e.svg\")
![\"b_{\\mathrm{new}}=0+0.5\\times1=0.5\"](\"assets/equations/ef3a575874682e63.svg\")
![\"z_{\\mathrm{new}}=0.5\\times1+0.5\\times1+0.5=1.5\"](\"assets/equations/9b6e487add8bda9e.svg\")
Since 1.5 > 0, the new output is 1. Final parameters: w = [0.5, 0.5], b = 0.5.
"],"kind":"practice","id":"tee-c17-s11","search":"Drill 11 One perceptron learning update **Question.** Binary perceptron, w = [0,0], b = 0, x = [1,1], target 1, η = 0.5. Use output 1 if z>0 else 0. Perform one update.\n\n**Full solution.**\n\n\n\n### Given\n\nw = [0,0]; b = 0; x = [1,1]; target t = 1.\n\nLearning rate η = 0.5.\n\nOutput 1 only when z > 0; otherwise output 0.\n\n### To find\n\nPerform one update and report new weights and bias.\n\n### Formula — which one and why\n\n**Perceptron learning rule.** Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters.\n\nt = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Prediction nikalo; target−prediction se correction nikalo.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formula\nA perceptron first computes a weighted sum z and then applies the exact threshold rule in the question. This question uses **strictly greater than zero**, so z = 0 produces output 0.\n\nThe perceptron learning rule uses target minus output, written e below. This is a discrete perceptron rule, not the sigmoid backpropagation derivative.\n\n### Step 1: obtain the old output\n\n### Step 2: compare it with the target\n\n### Step 3: update each weight separately\n\n\n\n### Step 4: interpret the updated prediction\n\nSince 1.5 > 0, the new output is 1. **Final parameters: w = [0.5, 0.5], b = 0.5.**\n"},{"title":"Drill 12 XOR using two hidden units","question":["Question. Use h1 = step(x1+x2−0.5), h2 = step(x1+x2−1.5), output = step(h1−2h2−0.5), step(z) = 1 if z≥0. Find all four outputs.
"],"answer":["h₁ = step(x₁+x₂−0.5); h₂ = step(x₁+x₂−1.5).
","Output = step(h₁−2h₂−0.5).
","step(z) = 1 when z ≥ 0, otherwise 0.
","Calculate hidden values and output for 00, 01, 10 and 11.
","Threshold logic. Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.
","step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.
","Yaad rakhne ka cue: Har input row ka sum nikalo, phir threshold lagao.
","XOR should output 1 when exactly one input is 1, and 0 when the inputs match. The first hidden unit detects “at least one input is 1”; the second detects “both inputs are 1”. The output combines these detections.
","![\"h_1=\\operatorname{step}(x_1+x_2-0.5)\"](\"assets/equations/bbffb2524956d0e5.svg\")
![\"h_2=\\operatorname{step}(x_1+x_2-1.5)\"](\"assets/equations/31fad4d0ea8f08ac.svg\")
![\"o=\\operatorname{step}(h_1-2h_2-0.5)\"](\"assets/equations/0af913266b36a0b9.svg\")
For this question, step returns 1 at nonnegative input and 0 at negative input.
","Step 1: Input pair 0, 0.
","![\"z_{h_1}=0+0-0.5=-0.5\\Rightarrow](\"assets/equations/6669d48b7635634e.svg\")
![\"z_{h_2}=0+0-1.5=-1.5\\Rightarrow](\"assets/equations/dcc9f2d8a503f369.svg\")
![\"z_o=0-2(0)-0.5=-0.5\\Rightarrow](\"assets/equations/96a5d17e0b8dd052.svg\")
Step 2: Input pair 0, 1.
","![\"z_{h_1}=0+1-0.5=0.5\\Rightarrow](\"assets/equations/f9475f1991e07af3.svg\")
![\"z_{h_2}=0+1-1.5=-0.5\\Rightarrow](\"assets/equations/45e0fb48b7159e24.svg\")
![\"z_o=1-2(0)-0.5=0.5\\Rightarrow](\"assets/equations/0ba860b3391cf5c7.svg\")
Step 3: Input pair 1, 0.
","![\"z_{h_1}=1+0-0.5=0.5\\Rightarrow](\"assets/equations/986114a7a93ab89f.svg\")
![\"z_{h_2}=1+0-1.5=-0.5\\Rightarrow](\"assets/equations/b9f2126aba5c12cc.svg\")
Step 4: Input pair 1, 1.
","![\"z_{h_1}=1+1-0.5=1.5\\Rightarrow](\"assets/equations/c6382db6dca562bf.svg\")
![\"z_{h_2}=1+1-1.5=0.5\\Rightarrow](\"assets/equations/208b1fc802d6494c.svg\")
![\"z_o=1-2(1)-0.5=-1.5\\Rightarrow](\"assets/equations/f0db03c68af4ca9e.svg\")
Final truth table: 00 → 0; 01 → 1; 10 → 1; 11 → 0.
"],"kind":"practice","id":"tee-c17-s12","search":"Drill 12 XOR using two hidden units **Question.** Use h1 = step(x1+x2−0.5), h2 = step(x1+x2−1.5), output = step(h1−2h2−0.5), step(z) = 1 if z≥0. Find all four outputs.\n\n**Full solution.**\n\n\n\n### Given\n\nh₁ = step(x₁+x₂−0.5); h₂ = step(x₁+x₂−1.5).\n\nOutput = step(h₁−2h₂−0.5).\n\nstep(z) = 1 when z ≥ 0, otherwise 0.\n\n### To find\n\nCalculate hidden values and output for 00, 01, 10 and 11.\n\n### Formula — which one and why\n\n**Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them.\n\nstep(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the given network\nXOR should output 1 when exactly one input is 1, and 0 when the inputs match. The first hidden unit detects “at least one input is 1”; the second detects “both inputs are 1”. The output combines these detections.\n\n\n\nFor this question, step returns 1 at nonnegative input and 0 at negative input.\n**Step 1: Input pair 0, 0.**\n\n\n\n**Step 2: Input pair 0, 1.**\n\n\n\n**Step 3: Input pair 1, 0.**\n\n\n\n**Step 4: Input pair 1, 1.**\n\n\n\n**Final truth table:** 00 → 0; 01 → 1; 10 → 1; 11 → 0.\n"},{"title":"Drill 13 Full ReLU backpropagation","question":["Question. Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases.
","Input x = [1, 2].
","W₁ is printed with the question.
","b₁ = [0.1, 0.2].
","W₂ is printed with the question.
","b₂ = [0.1].
","Hidden and output activations are ReLU.
","Target y = [1].
","Learning rate η = 0.1.
","Use half squared loss and update all weights and biases.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.
","Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.
","The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.
","For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","Hidden neuron 2:
","Output neuron 1:
","Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","The ReLU derivative is 1 here because this preactivation is positive.
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.0275. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","h2 → o1:
","Subtract η × gradient = −0.0165. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = −0.022. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h1:
","Subtract η × gradient = −0.044. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x1 → h2:
","Subtract η × gradient = −0.0275. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h2:
","Subtract η × gradient = −0.055. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
"],"kind":"practice","id":"tee-c17-s13","search":"Drill 13 Full ReLU backpropagation **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases.\n\n\n\n\n\n**Full solution.**\n\n\n\n### Given\n\nInput x = [1, 2].\n\nW₁ is printed with the question.\n\nb₁ = [0.1, 0.2].\n\nW₂ is printed with the question.\n\nb₂ = [0.1].\n\nHidden and output activations are ReLU.\n\nTarget y = [1].\n\nLearning rate η = 0.1.\n\nUse half squared loss and update all weights and biases.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used before substituting numbers\nFor every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.\n\nDelta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.\n\n\nThe gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.\n\n\nFor sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.\nPrediction minus target = [-0.55]."},{"title":"Drill 14 Two output sigmoid backpropagation","question":["Question. Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters.
","Input x = [0.2, 0.4].
","W₁ is printed with the question.
","b₁ = [0.1, 0.1].
","W₂ is printed with the question.
","b₂ = [0, 0.1].
","Hidden and output activations are sigmoid.
","Target y = [1, 0].
","Learning rate η = 0.2.
","Use half summed squared loss and update all parameters.
","Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.
","Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.
","The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.
","For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","Hidden neuron 2:
","Output neuron 1:
","Output neuron 2:
","Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","The derivative is o(1−o).
","Output delta 2:
","The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.013354367. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","h2 → o1:
","Subtract η × gradient = −0.013234017. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","h1 → o2:
","Subtract η × gradient = 0.01604788. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","h2 → o2:
","Subtract η × gradient = 0.015903256. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Contribution from output 2.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Contribution from output 2.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = 0.000192985. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h1:
","Subtract η × gradient = 0.000385971. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x1 → h2:
","Subtract η × gradient = 0.000699494. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h2:
","Subtract η × gradient = 0.001398988. This weight decreases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
"],"kind":"practice","id":"tee-c17-s14","search":"Drill 14 Two output sigmoid backpropagation **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters.\n\n\n\n\n\n**Full solution.**\n\n\n\n### Given\n\nInput x = [0.2, 0.4].\n\nW₁ is printed with the question.\n\nb₁ = [0.1, 0.1].\n\nW₂ is printed with the question.\n\nb₂ = [0, 0.1].\n\nHidden and output activations are sigmoid.\n\nTarget y = [1, 0].\n\nLearning rate η = 0.2.\n\nUse half summed squared loss and update all parameters.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used before substituting numbers\nFor every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons.\n\nDelta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path.\n\n\nThe gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1.\n\n\nFor sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated.\n\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each "},{"title":"Drill 15 A negative ReLU blocks this path","question":["Question. x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2.
"],"answer":["x = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2.
","Hidden activation: ReLU; output activation: linear.
","Target y = 1; loss: half squared error.
","Find the gradients of w₁ and w₂ and explain any zero gradients.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","The hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output.
","Because the raw hidden score is negative, ReLU's derivative at this point is 0.
","This is zero because the input to that connection, h, is zero.
","Final answer: both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero.
"],"kind":"practice","id":"tee-c17-s15","search":"Drill 15 A negative ReLU blocks this path **Question.** x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2.\n\n**Full solution.**\n\n\n\n### Given\n\nx = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2.\n\nHidden activation: ReLU; output activation: linear.\n\nTarget y = 1; loss: half squared error.\n\n### To find\n\nFind the gradients of w₁ and w₂ and explain any zero gradients.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used\nThe hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output.\n\n\n\n### Step 1: hidden weighted sum and activation\n\n\nBecause the raw hidden score is negative, ReLU's derivative at this point is 0.\n### Step 2: output prediction and output delta\n\n\n### Step 3: output weight gradient\n\nThis is zero because the input to that connection, h, is zero.\n### Step 4: hidden delta and input weight gradient\n\n\n**Final answer:** both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero.\n"},{"title":"Drill 16 Batch and sequential SGD","question":["Question. ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch.
"],"answer":["Model ŷ = wx; initial w = 0; η = 0.1.
","Samples in order: (x,y) = (1,1), then (2,2).
","Batch objective: mean half squared loss.
","Compare the weight after one batch update and one sequential SGD epoch.
","Batch mean versus sequential SGD. A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.
","B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.
","Yaad rakhne ka cue: Batch: same old weight par average; SGD: har sample ke baad weight change.
","For one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the same old weight. Sequential SGD updates after each sample and uses that new weight for the next sample.
","Step 1: Part A: one batch update from w = 0.
","For sample (1,1), prediction = 0 × 1 = 0, so:
","For sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so:
","Average these two gradients, then update once.
","Step 2: Part B: one SGD epoch, restarting at w = 0.
","The first sample has gradient −1, just as above.
","For the second sample use this new weight, so prediction is:
","Final answer: batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates.
"],"kind":"practice","id":"tee-c17-s16","search":"Drill 16 Batch and sequential SGD **Question.** ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch.\n\n**Full solution.**\n\n\n\n### Given\n\nModel ŷ = wx; initial w = 0; η = 0.1.\n\nSamples in order: (x,y) = (1,1), then (2,2).\n\nBatch objective: mean half squared loss.\n\n### To find\n\nCompare the weight after one batch update and one sequential SGD epoch.\n\n### Formula — which one and why\n\n**Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter.\n\nB = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nFor one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the **same old weight**. Sequential SGD updates after each sample and uses that new weight for the next sample.\n\n\n**Step 1: Part A: one batch update from w = 0.**\nFor sample (1,1), prediction = 0 × 1 = 0, so:\n\nFor sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so:\n\nAverage these two gradients, then update once.\n\n\n**Step 2: Part B: one SGD epoch, restarting at w = 0.**\nThe first sample has gradient −1, just as above.\n\nFor the second sample use this new weight, so prediction is:\n\n\n\n**Final answer:** batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates.\n"},{"title":"Drill 17 Incomplete mini batch","question":["Question. N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it.
"],"answer":["N = 103 samples; B = 20 samples per mini-batch.
","E = 3 epochs.
","Compare both remainder policies: keep or drop the final 3 examples.
","Find total updates for full-batch GD, SGD, and both mini-batch policies.
","Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.
","N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.
","Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.
","An epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs.
","Ceiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained.
","All 103 samples form one batch per epoch.
","Each sample gets its own update.
","The batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give:
","Only the five complete batches are used each epoch.
","Final answer: 3, 309, 18 and 15 updates respectively.
"],"kind":"practice","id":"tee-c17-s17","search":"Drill 17 Incomplete mini batch **Question.** N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it.\n\n**Full solution.**\n\n\n\n### Given\n\nN = 103 samples; B = 20 samples per mini-batch.\n\nE = 3 epochs.\n\nCompare both remainder policies: keep or drop the final 3 examples.\n\n### To find\n\nFind total updates for full-batch GD, SGD, and both mini-batch policies.\n\n### Formula — which one and why\n\n**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.\n\nN = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nAn epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs.\n\n\nCeiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained.\n### Step 1: batch GD\nAll 103 samples form one batch per epoch.\n\n### Step 2: SGD\nEach sample gets its own update.\n\n### Step 3: mini-batches with the remainder\n\nThe batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give:\n\n### Step 4: mini-batches without the remainder\nOnly the five complete batches are used each epoch.\n\n**Final answer:** 3, 309, 18 and 15 updates respectively.\n"},{"title":"Drill 18 GD variation 1","question":["Question. J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates.
"],"answer":["Objective: J(θ) = ½θ².
","Initial θ₀ = 1; η = 0.1; two iterations.
","Find every stored state needed by the method and the parameter after each of two updates.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.
","Step 1: Update 1.
","Step 2: Update 2.
","Final parameter after two updates: 0.81. The second gradient is evaluated again using the parameter obtained after the first update.
"],"kind":"practice","id":"tee-c17-s18","search":"Drill 18 GD variation 1 **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates. \n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½θ².\n\nInitial θ₀ = 1; η = 0.1; two iterations.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.\n### Update formulas\n\n**Step 1: Update 1.**\n\n\n**Step 2: Update 2.**\n\n\n**Final parameter after two updates: 0.81.** The second gradient is evaluated again using the parameter obtained after the first update.\n"},{"title":"Drill 19 GD variation 2","question":["Question. J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates.
"],"answer":["Objective: J(θ) = ½(θ−1)².
","Initial θ₀ = −0.5; η = 0.1; two iterations.
","Find every stored state needed by the method and the parameter after each of two updates.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.
","Step 1: Update 1.
","Step 2: Update 2.
","Final parameter after two updates: -0.215. The second gradient is evaluated again using the parameter obtained after the first update.
"],"kind":"practice","id":"tee-c17-s19","search":"Drill 19 GD variation 2 **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates. \n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½(θ−1)².\n\nInitial θ₀ = −0.5; η = 0.1; two iterations.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.\n### Update formulas\n\n**Step 1: Update 1.**\n\n\n**Step 2: Update 2.**\n\n\n**Final parameter after two updates: -0.215.** The second gradient is evaluated again using the parameter obtained after the first update.\n"},{"title":"Drill 20 Momentum variation 1","question":["Question. J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.
"],"answer":["Objective: J(θ) = ½θ².
","Initial θ₀ = 1; η = 0.1; two iterations.
","β = 0.9; initial signed velocity v₀ = 0.
","Find every stored state needed by the method and the parameter after each of two updates.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.
","v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.
","Step 1: Update 1.
","Step 2: Update 2.
","Final parameter after two updates: 0.72. The second gradient is evaluated again using the parameter obtained after the first update.
"],"kind":"practice","id":"tee-c17-s20","search":"Drill 20 Momentum variation 1 **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.\n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½θ².\n\nInitial θ₀ = 1; η = 0.1; two iterations.\n\nβ = 0.9; initial signed velocity v₀ = 0.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.\n### Update formulas\nv stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.\n\n\n**Step 1: Update 1.**\n\n\n\n**Step 2: Update 2.**\n\n\n\n**Final parameter after two updates: 0.72.** The second gradient is evaluated again using the parameter obtained after the first update.\n"},{"title":"Drill 21 Momentum variation 2","question":["Question. J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.
"],"answer":["Objective: J(θ) = ½(θ−1)².
","Initial θ₀ = −0.5; η = 0.1; two iterations.
","β = 0.9; initial signed velocity v₀ = 0.
","Find every stored state needed by the method and the parameter after each of two updates.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.
","v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.
","Step 1: Update 1.
","Step 2: Update 2.
","Final parameter after two updates: -0.08. The second gradient is evaluated again using the parameter obtained after the first update.
"],"kind":"practice","id":"tee-c17-s21","search":"Drill 21 Momentum variation 2 **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity.\n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½(θ−1)².\n\nInitial θ₀ = −0.5; η = 0.1; two iterations.\n\nβ = 0.9; initial signed velocity v₀ = 0.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.\n### Update formulas\nv stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.\n\n\n**Step 1: Update 1.**\n\n\n\n**Step 2: Update 2.**\n\n\n\n**Final parameter after two updates: -0.08.** The second gradient is evaluated again using the parameter obtained after the first update.\n"},{"title":"Drill 22 Nesterov variation 1","question":["Question. J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.
"],"answer":["Objective: J(θ) = ½θ².
","Initial θ₀ = 1; η = 0.1; two iterations.
","β = 0.9; initial signed velocity v₀ = 0.
","Find every stored state needed by the method and the parameter after each of two updates.
","Nesterov lookahead. First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.
","θ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.
","Yaad rakhne ka cue: Pehle look ahead; gradient wahan; update original position se.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.
","v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.
","Step 1: Update 1.
","Step 2: Update 2.
","Final parameter after two updates: 0.729. The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.
"],"kind":"practice","id":"tee-c17-s22","search":"Drill 22 Nesterov variation 1 **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.\n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½θ².\n\nInitial θ₀ = 1; η = 0.1; two iterations.\n\nβ = 0.9; initial signed velocity v₀ = 0.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.\n\nθ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0.\n### Update formulas\nv stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.\n\n\n**Step 1: Update 1.**\n\n\n\n\n**Step 2: Update 2.**\n\n\n\n\n**Final parameter after two updates: 0.729.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.\n"},{"title":"Drill 23 Nesterov variation 2","question":["Question. J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.
"],"answer":["Objective: J(θ) = ½(θ−1)².
","Initial θ₀ = −0.5; η = 0.1; two iterations.
","β = 0.9; initial signed velocity v₀ = 0.
","Find every stored state needed by the method and the parameter after each of two updates.
","Nesterov lookahead. First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.
","θ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.
","Yaad rakhne ka cue: Pehle look ahead; gradient wahan; update original position se.
","θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.
","The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.
","v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.
","Step 1: Update 1.
","Step 2: Update 2.
","Final parameter after two updates: -0.0935. The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.
"],"kind":"practice","id":"tee-c17-s23","search":"Drill 23 Nesterov variation 2 **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity.\n\n**Full solution.**\n\n\n\n### Given\n\nObjective: J(θ) = ½(θ−1)².\n\nInitial θ₀ = −0.5; η = 0.1; two iterations.\n\nβ = 0.9; initial signed velocity v₀ = 0.\n\n### To find\n\nFind every stored state needed by the method and the parameter after each of two updates.\n\n### Formula — which one and why\n\n**Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter.\n\nθ_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of the symbols\nθ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value.\n### Differentiate the objective first\n\nThe power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1.\n### Update formulas\nv stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0.\n\n\n**Step 1: Update 1.**\n\n\n\n\n**Step 2: Update 2.**\n\n\n\n\n**Final parameter after two updates: -0.0935.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead.\n"},{"title":"Drill 24 AdaGrad variation 1","question":["Question. θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation.
"],"answer":["Initial θ₀ = 1; η = 0.1.
","Supplied gradients: [2,1].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","Find every stored state needed by the method and the parameter after each of two updates.
","AdaGrad. Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.
","G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.
","Yaad rakhne ka cue: Square karo, purane sum mein jodo, root se scale karo.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient 2.
","Add the new squared gradient to all previous squares.
","Step 2: Iteration 2 with supplied gradient 1.
","Add the new squared gradient to all previous squares.
","Question. θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation.
"],"answer":["Initial θ₀ = −0.5; η = 0.1.
","Supplied gradients: [−1,2].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","Find every stored state needed by the method and the parameter after each of two updates.
","AdaGrad. Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.
","G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.
","Yaad rakhne ka cue: Square karo, purane sum mein jodo, root se scale karo.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient -1.
","Add the new squared gradient to all previous squares.
","Step 2: Iteration 2 with supplied gradient 2.
","Add the new squared gradient to all previous squares.
","Question. θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9.
"],"answer":["Initial θ₀ = 1; η = 0.1.
","Supplied gradients: [2,1].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","ρ = 0.9.
","Find every stored state needed by the method and the parameter after each of two updates.
","RMSProp. Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.
","s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.
","Yaad rakhne ka cue: Saare purane squares nahi—unki fading average.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient 2.
","Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.
","Step 2: Iteration 2 with supplied gradient 1.
","Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.
","Question. θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9.
"],"answer":["Initial θ₀ = −0.5; η = 0.1.
","Supplied gradients: [−1,2].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","ρ = 0.9.
","Find every stored state needed by the method and the parameter after each of two updates.
","RMSProp. Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.
","s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.
","Yaad rakhne ka cue: Saare purane squares nahi—unki fading average.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient -1.
","Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.
","Step 2: Iteration 2 with supplied gradient 2.
","Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.
","Question. θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction.
"],"answer":["Initial θ₀ = 1; η = 0.1.
","Supplied gradients: [2,1].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","β₁ = 0.9; β₂ = 0.999; use bias correction.
","Find every stored state needed by the method and the parameter after each of two updates.
","Adam with bias correction. Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update.
","m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence.
","Yaad rakhne ka cue: Direction memory + square memory; hats lagao, phir update.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient 2.
","First calculate the signed-gradient mean m and squared-gradient mean v.
","Correct each zero-initialized mean using this iteration number in the power.
","Divide the corrected direction by the corrected root, then subtract the scaled step.
","Step 2: Iteration 2 with supplied gradient 1.
","First calculate the signed-gradient mean m and squared-gradient mean v.
","Correct each zero-initialized mean using this iteration number in the power.
","Divide the corrected direction by the corrected root, then subtract the scaled step.
","Question. θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction.
"],"answer":["Initial θ₀ = −0.5; η = 0.1.
","Supplied gradients: [−1,2].
","Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.
","β₁ = 0.9; β₂ = 0.999; use bias correction.
","Find every stored state needed by the method and the parameter after each of two updates.
","Adam with bias correction. Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update.
","m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence.
","Yaad rakhne ka cue: Direction memory + square memory; hats lagao, phir update.
","θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.
","m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999.
","Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.
","Step 1: Iteration 1 with supplied gradient -1.
","First calculate the signed-gradient mean m and squared-gradient mean v.
","Correct each zero-initialized mean using this iteration number in the power.
","Divide the corrected direction by the corrected root, then subtract the scaled step.
","Step 2: Iteration 2 with supplied gradient 2.
","First calculate the signed-gradient mean m and squared-gradient mean v.
","Correct each zero-initialized mean using this iteration number in the power.
","Divide the corrected direction by the corrected root, then subtract the scaled step.
","Question. Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw².
"],"answer":["Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1.
","For L2, the question includes the factor ½.
","Find each penalty and the total L1/L2 objective.
","L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.
","λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.
","Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.
","L2 penalty with a factor of one-half. Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.
","λ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.
","Yaad rakhne ka cue: Half wala L2 derivative mein lambda × weight deta hai.
","The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty.
","Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations.
","Step 1: Part A: L1 objective.
","Step 2: Part B: L2 objective with the given half.
","Final answer: L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values.
"],"kind":"practice","id":"tee-c17-s30","search":"Drill 30 Regularized objective **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw².\n\n**Full solution.**\n\n\n\n### Given\n\nData loss = 0.4; weights = [−2,1,0.5]; λ = 0.1.\n\nFor L2, the question includes the factor ½.\n\n### To find\n\nFind each penalty and the total L1/L2 objective.\n\n### Formula — which one and why\n\n**L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.\n\nλ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.\n\n\n\n\n\n**Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai.\n\n**L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.\n\nλ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty.\n\n\nAbsolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations.\n**Step 1: Part A: L1 objective.**\n\n\n\n\n**Step 2: Part B: L2 objective with the given half.**\n\n\n\n\n\n**Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values.\n"},{"title":"Drill 31 L1 update on positive and negative weights","question":["Question. w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero.
"],"answer":["Old weights = [−0.5,0.3].
","Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2.
","Find total gradients and the two updated weights.
","L1 penalty and gradient. Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.
","λ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.
","Yaad rakhne ka cue: L1 absolute values leta hai; gradient mein sign aata hai.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight.
","Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed.
","Step 1: Weight 1: old value −0.5.
","Step 2: Weight 2: old value 0.3.
","Final answer: [−0.52, 0.3]. L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero.
"],"kind":"practice","id":"tee-c17-s31","search":"Drill 31 L1 update on positive and negative weights **Question.** w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero.\n\n**Full solution.**\n\n\n\n### Given\n\nOld weights = [−0.5,0.3].\n\nData gradients = [0.2,−0.1]; λ = 0.1; η = 0.2.\n\n### To find\n\nFind total gradients and the two updated weights.\n\n### Formula — which one and why\n\n**L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign.\n\nλ (lambda) = penalty strength; J_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed.\n\n\n\n\n\n**Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai.\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formula\nThe supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight.\n\n\nBoth weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed.\n**Step 1: Weight 1: old value −0.5.**\n\n\n\n**Step 2: Weight 2: old value 0.3.**\n\n\n\n**Final answer: [−0.52, 0.3].** L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero.\n"},{"title":"Drill 32 L2 update","question":["Question. w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw².
"],"answer":["Old weights = [0.5,−0.2]; data gradients = [0.1,0.3].
","λ = 0.2; η = 0.1; penalty = (λ/2)Σw².
","Find penalty gradients, total gradients and updated weights.
","L2 penalty with a factor of one-half. Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.
","λ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.
","Yaad rakhne ka cue: Half wala L2 derivative mein lambda × weight deta hai.
","The derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating.
","Step 1: Weight 1.
","Step 2: Weight 2.
","Final answer: [0.48, −0.226]. The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative.
"],"kind":"practice","id":"tee-c17-s32","search":"Drill 32 L2 update **Question.** w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw².\n\n**Full solution.**\n\n\n\n### Given\n\nOld weights = [0.5,−0.2]; data gradients = [0.1,0.3].\n\nλ = 0.2; η = 0.1; penalty = (λ/2)Σw².\n\n### To find\n\nFind penalty gradients, total gradients and updated weights.\n\n### Formula — which one and why\n\n**L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.\n\nλ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai.\n\n### Solution — step by step\n\n\n\n\n\n### Formula and its reason\nThe derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating.\n\n\n**Step 1: Weight 1.**\n\n\n\n**Step 2: Weight 2.**\n\n\n\n**Final answer: [0.48, −0.226].** The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative.\n"},{"title":"Drill 33 Inverted dropout","question":["Question. Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference.
"],"answer":["Activations = [2,−1,0,3].
","Keep probability q = 0.75; keep mask = [1,0,1,1].
","Find training outputs and state the inference rule.
","Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.
","aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.
","Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.
","The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved.
","The multiplication is entry by entry; there is no dot-product sum.
","Training output = [2.666666667, 0, 0, 4]. A kept activation that was already zero stays zero.
","Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training.
"],"kind":"practice","id":"tee-c17-s33","search":"Drill 33 Inverted dropout **Question.** Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference.\n\n**Full solution.**\n\n\n\n### Given\n\nActivations = [2,−1,0,3].\n\nKeep probability q = 0.75; keep mask = [1,0,1,1].\n\n### To find\n\nFind training outputs and state the inference rule.\n\n### Formula — which one and why\n\n**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.\n\naᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.\n\n\n\n\n\n**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formula\nThe mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved.\n\nThe multiplication is entry by entry; there is no dot-product sum.\n### Step 1: apply the mask\n\n### Step 2: divide each result by 0.75\n\n\n\n**Training output = [2.666666667, 0, 0, 4].** A kept activation that was already zero stays zero.\n### During inference\nDisable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training.\n"},{"title":"Drill 34 Convolution with a negative coefficient","question":["Question. Use the input A and kernel K shown below, stride 1, padding 0. Compute all outputs.
","| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
| 1 | −1 |
| 0 | 1 |
Stride 1; P = 0 (no padding).
Input A and kernel K are printed with the question.
","Stride S = 1; padding P = 0; bias = 0.
","Use the kernel without flipping it.
","Calculate all nine output cells.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation.
","Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability.
","N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Step 1 shape: = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 1 | −2 |
| 0 | 1 |
Each cell is input × matching kernel coefficient.
(−1) + 1 = 0
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 1, column 1.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 2 | 0 |
| 0 | 3 |
Each cell is input × matching kernel coefficient.
2 + 3 = 5
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 1, column 2.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 0 | −1 |
| 0 | 2 |
Each cell is input × matching kernel coefficient.
(−1) + 2 = 1
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 1, column 3.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 0 | −1 |
| 0 | 1 |
Each cell is input × matching kernel coefficient.
(−1) + 1 = 0
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 2, column 1.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 1 | −3 |
| 0 | 0 |
Each cell is input × matching kernel coefficient.
(−2) + 0 = −2
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 2, column 2.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 3 | −2 |
| 0 | 1 |
Each cell is input × matching kernel coefficient.
1 + 1 = 2
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 2, column 3.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 2 | −1 |
| 0 | 0 |
Each cell is input × matching kernel coefficient.
1 + 0 = 1
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 3, column 1.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 |
| 0 | 2 |
Each cell is input × matching kernel coefficient.
1 + 2 = 3
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 3, column 2.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | −1 |
| 0 | 1 |
Same row and column positions; no matrix multiplication.
| 0 | −1 |
| 0 | 3 |
Each cell is input × matching kernel coefficient.
(−1) + 3 = 2
| 0 | 5 | 1 |
| 0 | −2 | 2 |
| 1 | 3 | 2 |
Output row 3, column 3.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Final output: . Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
Question. Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total.
"],"answer":["Input shape = 31 × 31 × 3.
","Filter size = 5 × 5; number of filters = 8.
","Stride S = 2; padding P = 1 per side; one bias per filter.
","Find output shape, weights, biases and total parameters.
","Convolution/pooling spatial size. Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.
","N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.
","Yaad rakhne ka cue: Input + dono sides ki padding − kernel; stride se divide; floor + 1.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","The input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels.
","Padding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33.
","The height uses the same calculation. Output shape = 15 × 15 × 8.
","Each filter covers 5 × 5 positions in each of 3 channels.
","Final answer: shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights.
"],"kind":"practice","id":"tee-c17-s35","search":"Drill 35 Padding stride and channels **Question.** Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total.\n\n**Full solution.**\n\n\n\n### Given\n\nInput shape = 31 × 31 × 3.\n\nFilter size = 5 × 5; number of filters = 8.\n\nStride S = 2; padding P = 1 per side; one bias per filter.\n\n### To find\n\nFind output shape, weights, biases and total parameters.\n\n### Formula — which one and why\n\n**Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.\n\nN = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.\n\n\n\n**Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning of each given value\nThe input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels.\n### Formulas used\n\n\n\n### Step 1: find the output height and width\nPadding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33.\n\n\nThe height uses the same calculation. **Output shape = 15 × 15 × 8.**\n### Step 2: count weights in one filter\nEach filter covers 5 × 5 positions in each of 3 channels.\n\n### Step 3: count all filters and their biases\n\n\n**Final answer:** shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights.\n"},{"title":"Drill 36 Max and average pooling","question":["Question. Pool the input A shown below with 2 × 2 windows and stride 2. Find max and average outputs.
","Input A is printed with the question.
","Window size = 2 × 2; stride = 2; no padding.
","Find the complete max-pooling and average-pooling outputs.
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer.
","Use the input size at the pooling layer. The output keeps the same channel count.
","Shape: = 2 on each axis; channels stay unchanged.
Step 1: select the entries.
","[1, 4, 0, 2]
","Step 2: add all selected entries.
","1 + 4 + 0 + 2 = 7
","Step 3: divide by the 4 entries in this window.
","7/4 = 1.75. Maximum = 4
","Step 1: select the entries.
","[2, 3, 5, 1]
","Step 2: add all selected entries.
","2 + 3 + 5 + 1 = 11
","Step 3: divide by the 4 entries in this window.
","11/4 = 2.75. Maximum = 5
","Step 1: select the entries.
","[6, 3, 1, 7]
","Step 2: add all selected entries.
","6 + 3 + 1 + 7 = 17
","Step 3: divide by the 4 entries in this window.
","17/4 = 4.25. Maximum = 7
","Step 1: select the entries.
","[2, 0, 4, 8]
","Step 2: add all selected entries.
","2 + 0 + 4 + 8 = 14
","Step 3: divide by the 4 entries in this window.
","14/4 = 3.5. Maximum = 8
","Average output: . Max output: . Pooling normally has zero trainable parameters and reduces spatial dimensions.
Question. Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters.
"],"answer":["Input = 10 × 10 × 1.
","Conv: 3 × 3, 4 filters, stride 1, valid padding.
","Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs.
","Include one bias per filter/dense neuron.
","Find all output shapes, layer parameter counts and their sum.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","“Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters.
","Four filters produce four channels: 8 × 8 × 4.
","Pooling now receives size 8, not the original size 10.
","Channels remain 4: 4 × 4 × 4. Pooling parameters = 0.
","Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0.
","Final answer: shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters.
"],"kind":"practice","id":"tee-c17-s37","search":"Drill 37 Complete small CNN **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters.\n\n**Full solution.**\n\n\n\n### Given\n\nInput = 10 × 10 × 1.\n\nConv: 3 × 3, 4 filters, stride 1, valid padding.\n\nPool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs.\n\nInclude one bias per filter/dense neuron.\n\n### To find\n\nFind all output shapes, layer parameter counts and their sum.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas and assumptions\n“Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters.\n\n\n\n### Step 1: convolution shape\n\nFour filters produce four channels: **8 × 8 × 4**.\n### Step 2: convolution parameters\n\n\n### Step 3: pooling shape\nPooling now receives size 8, not the original size 10.\n\nChannels remain 4: **4 × 4 × 4**. Pooling parameters = 0.\n### Step 4: flatten\n\nFlatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0.\n### Step 5: dense layer with 3 outputs\n\n\n### Step 6: add learned parameters\n\n**Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters.\n"},{"title":"Drill 38 Reverse a spatial formula","question":["Question. A stride-1, valid 3 × 3 convolution outputs 18 × 18 × 12. What input spatial size is needed? Is input channel count determined?
"],"answer":["Output shape = 18 × 18 × 12.
","Kernel = 3 × 3; stride = 1; valid padding P = 0.
","Input channel count is not supplied.
","Solve for input height/width; state whether input channels and parameter count are uniquely determined.
","Convolution/pooling spatial size. Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.
","N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.
","Yaad rakhne ka cue: Input + dono sides ki padding − kernel; stride se divide; floor + 1.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","For stride 1 and valid padding, the output width is input width minus kernel width plus one.
","Add 2 to both sides. Doing the same operation to both sides preserves the equality.
","The 12 output channels tell us there are 12 filters. They do not say how many channels the input has. An RGB input and a many-channel intermediate map could both feed 12 filters.
","With one bias per filter, the parameter formula would be:
","Final answer: input spatial shape 20 × 20; input channel count is undetermined from these givens.
"],"kind":"practice","id":"tee-c17-s38","search":"Drill 38 Reverse a spatial formula **Question.** A stride-1, valid 3 × 3 convolution outputs 18 × 18 × 12. What input spatial size is needed? Is input channel count determined?\n\n**Full solution.**\n\n\n\n### Given\n\nOutput shape = 18 × 18 × 12.\n\nKernel = 3 × 3; stride = 1; valid padding P = 0.\n\nInput channel count is not supplied.\n\n### To find\n\nSolve for input height/width; state whether input channels and parameter count are uniquely determined.\n\n### Formula — which one and why\n\n**Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.\n\nN = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.\n\n\n\n**Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n\n### Formula used\nFor stride 1 and valid padding, the output width is input width minus kernel width plus one.\n\n### Step 1: substitute the known output and kernel sizes\n\n### Step 2: isolate the unknown input size\nAdd 2 to both sides. Doing the same operation to both sides preserves the equality.\n\n\n### Step 3: interpret the channel information\nThe 12 output channels tell us there are 12 filters. They do not say how many channels the input has. An RGB input and a many-channel intermediate map could both feed 12 filters.\nWith one bias per filter, the parameter formula would be:\n\n**Final answer:** input spatial shape 20 × 20; input channel count is undetermined from these givens.\n"},{"title":"Drill 39 Bottleneck cost with new values","question":["Question. A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first.
"],"answer":["Input = 16 × 16 × 64.
","Direct branch: 32 same-padded 3 × 3 filters.
","Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer.
","Spatial size remains 16 × 16; exclude bias additions.
","Find both MAC counts, absolute saving and percentage saving.
","Multiply–accumulate count. Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.
","H_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.
","Yaad rakhne ka cue: Har output cell ka kaam × kitne output cells.
","A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer.
","Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs.
","Final answer: direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32.
"],"kind":"practice","id":"tee-c17-s39","search":"Drill 39 Bottleneck cost with new values **Question.** A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first.\n\n**Full solution.**\n\n\n\n### Given\n\nInput = 16 × 16 × 64.\n\nDirect branch: 32 same-padded 3 × 3 filters.\n\nAlternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer.\n\nSpatial size remains 16 × 16; exclude bias additions.\n\n### To find\n\nFind both MAC counts, absolute saving and percentage saving.\n\n### Formula — which one and why\n\n**Multiply–accumulate count.** Each output position and output channel uses K²C_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution.\n\nH_out,W_out = output map size; C_in,C_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded.\n\n\n\n\n\n**Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formula\nA MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer.\n\n### Step 1: direct 3 × 3 convolution\nEach output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs.\n\n### Step 2: 1 × 1 reduction from 64 to 8 channels\n\n### Step 3: 3 × 3 convolution now receives only 8 channels\n\n### Step 4: add the two bottleneck stages\n\n### Step 5: calculate the saving\n\n\n**Final answer:** direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32.\n"},{"title":"Drill 40 Residual addition","question":["Question. x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block.
"],"answer":["x = [2,−3,1]; F(x) = [−1,1,0.5].
","Find x+F(x), apply ReLU, and compare with ReLU(F(x)).
","Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.
","x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.
","Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join.
","The residual output before ReLU is [1, −2, 1.5].
","Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is [0, 1, 0.5].
","Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation.
"],"kind":"practice","id":"tee-c17-s40","search":"Drill 40 Residual addition **Question.** x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block.\n\n**Full solution.**\n\n\n\n### Given\n\nx = [2,−3,1]; F(x) = [−1,1,0.5].\n\n### To find\n\nFind x+F(x), apply ReLU, and compare with ReLU(F(x)).\n\n### Formula — which one and why\n\n**Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.\n\nx = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.\n\n\n\n**Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join.\n\n### Step 1: add each pair\n\n\n\nThe residual output before ReLU is **[1, −2, 1.5]**.\n### Step 2: apply ReLU\n\n### Step 3: compare with the plain branch\nWithout a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is **[0, 1, 0.5]**.\nResidual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation.\n"},{"title":"Drill 41 Two RNN steps","question":["Question. h₀ = 0, Wxh = 0.5, Whh = 0.2, b = 0, x₁ = 1, x₂ = 2, tanh activation. Find h₁, h₂.
"],"answer":["Initial h₀ = 0; input weight Wxh = 0.5; recurrent weight Whh = 0.2; b = 0.
","Inputs: x₁ = 1, x₂ = 2; activation: tanh.
","Calculate h₁, then use it to calculate h₂.
","Recurrent forward pass. At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.
","x_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.
","Yaad rakhne ka cue: Naya input + pichhla activated state; phir activation.
","At time t, the RNN combines the current input xₜ and the previous hidden state hₜ₋₁. The same two weights are reused at both times.
","The raw sum a is not the hidden state. Tanh converts that sum into the state that is carried forward.
","Final answer: h₁ ≈ 0.462117157; h₂ ≈ 0.797760954. Pass h₁ to the next step, not the unactivated value 0.5.
"],"kind":"practice","id":"tee-c17-s41","search":"Drill 41 Two RNN steps **Question.** h₀ = 0, Wxh = 0.5, Whh = 0.2, b = 0, x₁ = 1, x₂ = 2, tanh activation. Find h₁, h₂.\n\n**Full solution.**\n\n\n\n### Given\n\nInitial h₀ = 0; input weight Wxh = 0.5; recurrent weight Whh = 0.2; b = 0.\n\nInputs: x₁ = 1, x₂ = 2; activation: tanh.\n\n### To find\n\nCalculate h₁, then use it to calculate h₂.\n\n### Formula — which one and why\n\n**Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step.\n\nx_t = current input; h_(t−1) = previous hidden state; W_xh = input weights; W_hh = recurrent weights; b = bias.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nAt time t, the RNN combines the current input xₜ and the previous hidden state hₜ₋₁. The same two weights are reused at both times.\n\nThe raw sum a is not the hidden state. Tanh converts that sum into the state that is carried forward.\n\n### Step 1: calculate the first raw sum\n\n### Step 2: activate it to obtain the first state\n\n\n### Step 3: use that state in the second raw sum\n\n\n### Step 4: activate the second sum\n\n**Final answer:** h₁ ≈ 0.462117157; h₂ ≈ 0.797760954. Pass h₁ to the next step, not the unactivated value 0.5.\n"},{"title":"Drill 42 LSTM with a negative candidate","question":["Question. f = 0.8, i = 0.2, o = 0.6, candidate = −0.5, Cprevious = 0.3. Find C, h.
"],"answer":["Activated gates: f = 0.8, i = 0.2, o = 0.6.
","Candidate = −0.5; old cell C = 0.3.
","Find the new cell and hidden state.
","LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.
","f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.
","Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.
","An LSTM keeps a cell memory C and a visible hidden state h. The forget gate f controls old memory, the input gate i controls new candidate content, and the output gate o controls the visible state. These gate values are already supplied.
","![\"C=fC_{\\mathrm{previous}}+i\\widetilde](\"assets/equations/7d40da391fed6757.svg\")
The candidate, written C with a tilde, may be negative. A negative candidate subtracts content when admitted.
","![\"fC_{\\mathrm{previous}}=0.8\\times0.3=0.24\"](\"assets/equations/692dcfa59ab03f38.svg\")
![\"i\\widetilde](\"assets/equations/b92f4b1761bc1ff0.svg\")
![\"C=0.24+(-0.1)=0.14\"](\"assets/equations/752dd644f97def6b.svg\")
![\"\\tanh(0.14)=\\frac{e^{0.28}-1}{e^{0.28}+1}=0.139092448\"](\"assets/equations/0cf22c6bf34159ce.svg\")
![\"h=0.6\\times0.139092448=0.083455469\"](\"assets/equations/cc330c5bb2af2ffc.svg\")
Final answer: cell memory 0.14; hidden state approximately 0.083455469. C and h are different quantities.
"],"kind":"practice","id":"tee-c17-s42","search":"Drill 42 LSTM with a negative candidate **Question.** f = 0.8, i = 0.2, o = 0.6, candidate = −0.5, Cprevious = 0.3. Find C, h.\n\n**Full solution.**\n\n\n\n### Given\n\nActivated gates: f = 0.8, i = 0.2, o = 0.6.\n\nCandidate = −0.5; old cell C = 0.3.\n\n### To find\n\nFind the new cell and hidden state.\n\n### Formula — which one and why\n\n**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.\n\nf,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.\n\n\n\n\n\n**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nAn LSTM keeps a cell memory C and a visible hidden state h. The forget gate f controls old memory, the input gate i controls new candidate content, and the output gate o controls the visible state. These gate values are already supplied.\n\nThe candidate, written C with a tilde, may be negative. A negative candidate subtracts content when admitted.\n### Step 1: retained old memory\n\n### Step 2: admitted new content\n\n### Step 3: add them to obtain the new cell memory\n\n### Step 4: apply tanh to the cell memory\n\n### Step 5: apply the output gate\n\n**Final answer:** cell memory 0.14; hidden state approximately 0.083455469. C and h are different quantities.\n"},{"title":"Drill 43 Calculate gates before the cell","question":["Question. Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h.
"],"answer":["Raw gate scores: zf = zi = zo = 0.
","Raw candidate score = 0.5; old cell C = 0.2.
","Activate gates/candidate, then calculate the new cell and hidden state.
","Activate raw gate scores first. A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.
","z_f,z_i,z_o = gate preactivations; z_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.
","Yaad rakhne ka cue: Raw score pehle activate; activated gate ko dobara sigmoid nahi.
","LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.
","f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.
","Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.
","The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh.
","![\"f=\\sigma(z_f),\\quad](\"assets/equations/48a155da6647af6e.svg\")
![\"\\sigma(z)=\\frac1{1+e^{-z}},\\quad](\"assets/equations/2240d8201e156ff4.svg\")
![\"\\sigma(0)=\\frac1{1+e^0}=\\frac12=0.5\"](\"assets/equations/171834ffe9579c30.svg\")
Thus f = i = o = 0.5. A zero score is not a zero gate.
","![\"\\widetilde](\"assets/equations/4b709e4b1fd2b287.svg\")
![\"fC_{\\mathrm{previous}}=0.5\\times0.2=0.1\"](\"assets/equations/cc79f89f1b075375.svg\")
![\"i\\widetilde](\"assets/equations/17c2ce9389d35386.svg\")
![\"C=0.1+0.231058579=0.331058579\"](\"assets/equations/b3473959824ebef9.svg\")
![\"\\tanh(C)=\\tanh(0.331058579)\\approx0.319471636\"](\"assets/equations/1b0373022abd784e.svg\")
![\"h=0.5\\times0.319471636=0.159735818\"](\"assets/equations/1edcaccecc620bf5.svg\")
Final answer: gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818.
"],"kind":"practice","id":"tee-c17-s43","search":"Drill 43 Calculate gates before the cell **Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h.\n\n**Full solution.**\n\n\n\n### Given\n\nRaw gate scores: zf = zi = zo = 0.\n\nRaw candidate score = 0.5; old cell C = 0.2.\n\n### To find\n\nActivate gates/candidate, then calculate the new cell and hidden state.\n\n### Formula — which one and why\n\n**Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations.\n\nz_f,z_i,z_o = gate preactivations; z_c = candidate preactivation. An already activated gate must not be passed through sigmoid again.\n\n\n\n\n\n**Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi.\n\n**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.\n\nf,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.\n\n\n\n\n\n**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh.\n\n\n### Step 1: turn each zero score into a gate\n\nThus f = i = o = 0.5. A zero score is not a zero gate.\n### Step 2: calculate candidate content\n\n### Step 3: combine retained memory and new content\n\n\n\n### Step 4: compute the visible hidden state\n\n\n**Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818.\n"},{"title":"Drill 44 GRU reset then mix","question":["Question. hprevious = 0.4, r = 0.5, z = 0.25, x = 1, W = 0.2, U = 0.5, b = 0. Use candidate = tanh(Wx+U(r × hprevious)) and h = (1−z) hprevious+z candidate.
"],"answer":["Old hidden state = 0.4; reset r = 0.5; update z = 0.25.
","x = 1; W = 0.2; U = 0.5; b = 0.
","Use h = (1−z)h_old + z×candidate.
","Find reset contribution, candidate and new hidden state.
","GRU reset and state mixture. The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate.
","r = reset gate; z = update gate; h_old = previous state; h_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state.
","Yaad rakhne ka cue: Reset candidate ko control; update gate old-new mixture ko control.
","The reset gate r scales the old state while making the candidate. The update gate z controls the amount of candidate used in the final mixture under this question's convention. Some books reverse the roles of z and 1 − z, so copy the supplied formula.
","![\"\\widetilde](\"assets/equations/a7608696c8363c58.svg\")
![\"h=(1-z)h_{\\mathrm{previous}}+z\\widetilde](\"assets/equations/7c9d53d54875e282.svg\")
![\"rh_{\\mathrm{previous}}=0.5\\times0.4=0.2\"](\"assets/equations/90ee4b754abc5317.svg\")
![\"Wx=0.2\\times1=0.2\"](\"assets/equations/545ce783984e0414.svg\")
![\"U(rh_{\\mathrm{previous}})=0.5\\times0.2=0.1\"](\"assets/equations/afd69509f26a8283.svg\")
![\"a=0.2+0.1+0=0.3\"](\"assets/equations/41862a6072aeaa37.svg\")
![\"\\widetilde](\"assets/equations/6ce7040fd0d1872b.svg\")
![\"1-z=1-0.25=0.75\"](\"assets/equations/854154216ffe0396.svg\")
![\"h=0.75(0.4)+0.25(0.291312612)\"](\"assets/equations/82108e6e910af0fd.svg\")
![\"h=0.3+0.072828153=0.372828153\"](\"assets/equations/4fbe2a7f0d77a4fd.svg\")
Final answer: h ≈ 0.372828153. The final old-state term uses 0.4, not the reset value 0.2.
"],"kind":"practice","id":"tee-c17-s44","search":"Drill 44 GRU reset then mix **Question.** hprevious = 0.4, r = 0.5, z = 0.25, x = 1, W = 0.2, U = 0.5, b = 0. Use candidate = tanh(Wx+U(r × hprevious)) and h = (1−z) hprevious+z candidate.\n\n**Full solution.**\n\n\n\n### Given\n\nOld hidden state = 0.4; reset r = 0.5; update z = 0.25.\n\nx = 1; W = 0.2; U = 0.5; b = 0.\n\nUse h = (1−z)h_old + z×candidate.\n\n### To find\n\nFind reset contribution, candidate and new hidden state.\n\n### Formula — which one and why\n\n**GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate.\n\nr = reset gate; z = update gate; h_old = previous state; h_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state.\n\n\n\n\n\n**Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and given convention\nThe reset gate r scales the old state while making the candidate. The update gate z controls the amount of candidate used in the final mixture **under this question's convention**. Some books reverse the roles of z and 1 − z, so copy the supplied formula.\n\n\n### Step 1: reset the old state for candidate construction\n\n### Step 2: form the candidate's raw score\n\n\n\n### Step 3: apply tanh\n\n### Step 4: mix the original old state with the candidate\n\n\n\n**Final answer: h ≈ 0.372828153.** The final old-state term uses 0.4, not the reset value 0.2.\n"},{"title":"Drill 45 BPTT shared weights","question":["Question. Linear recurrence hₜ = uxₜ+vhₜ₋₁+b, output h₂, target 1, half squared loss. u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1, x₂ = 2, η = 0.1. Find both states and one update.
"],"answer":["Linear recurrence hₜ = uxₜ+vhₜ₋₁+b.
","u = 0.2; v = 0.5; b = 0; h₀ = 0.
","x₁ = 1; x₂ = 2; target for h₂ = 1.
","Half squared loss; learning rate η = 0.1.
","Find both states, all three shared-parameter gradients and one simultaneous update.
","Backpropagation through time. Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once.
","For h_t=ux_t+vh_(t−1)+b with loss only at the last state: δ_t = loss derivative at that state. The linear recurrence has activation slope 1.
","Yaad rakhne ka cue: Time ke har use ka gradient jodo; shared weight ko ek baar update karo.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","The same u, v and b appear at both times. u multiplies the input, v multiplies the previous state, and b is a shared bias. The loss is only at the final state h₂. Because the recurrence is linear, its local activation derivative is 1.
","![\"h_t=ux_t+vh_{t-1}+b,\\qquad](\"assets/equations/fd56e04419aef1fc.svg\")
![\"\\delta_2=h_2-y,\\qquad](\"assets/equations/49cb0da4f4956b69.svg\")
Each shared parameter receives a contribution from both of its uses.
","![\"h_2=0.2(2)+0.5(0.2)+0=0.4+0.1=0.5\"](\"assets/equations/0cd838106937b9a7.svg\")
![\"\\delta_2=0.5-1=-0.5\"](\"assets/equations/50ef5b3a75d08159.svg\")
![\"\\delta_1=0.5(-0.5)=-0.25\"](\"assets/equations/98a077f6210fe712.svg\")
![\"\\frac{\\partial](\"assets/equations/af14a42daa22ee7b.svg\")
![\"\\frac{\\partial](\"assets/equations/dadd294afa7bf158.svg\")
![\"\\frac{\\partial](\"assets/equations/3e07d50d2b197b5f.svg\")
Final answer: states 0.2 and 0.5; new parameters u = 0.325, v = 0.51, b = 0.075.
"],"kind":"practice","id":"tee-c17-s45","search":"Drill 45 BPTT shared weights **Question.** Linear recurrence hₜ = uxₜ+vhₜ₋₁+b, output h₂, target 1, half squared loss. u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1, x₂ = 2, η = 0.1. Find both states and one update.\n\n**Full solution.**\n\n\n\n### Given\n\nLinear recurrence hₜ = uxₜ+vhₜ₋₁+b.\n\nu = 0.2; v = 0.5; b = 0; h₀ = 0.\n\nx₁ = 1; x₂ = 2; target for h₂ = 1.\n\nHalf squared loss; learning rate η = 0.1.\n\n### To find\n\nFind both states, all three shared-parameter gradients and one simultaneous update.\n\n### Formula — which one and why\n\n**Backpropagation through time.** Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once.\n\nFor h_t=ux_t+vh_(t−1)+b with loss only at the last state: δ_t = loss derivative at that state. The linear recurrence has activation slope 1.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Time ke har use ka gradient jodo; shared weight ko ek baar update karo.\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThe same u, v and b appear at both times. u multiplies the input, v multiplies the previous state, and b is a shared bias. The loss is only at the final state h₂. Because the recurrence is linear, its local activation derivative is 1.\n\n\nEach shared parameter receives a contribution from both of its uses.\n\n\n### Step 1: forward through time 1\n\n### Step 2: forward through time 2\n\n\n### Step 3: carry the gradient backwards\n\n\n### Step 4: add contributions for each shared parameter\n\n\n\n### Step 5: update once after summing\n\n\n\n**Final answer:** states 0.2 and 0.5; new parameters u = 0.325, v = 0.51, b = 0.075.\n"},{"title":"Drill 46 Mixed convolution pool dense and loss","question":["Question. A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE.
","| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
| 1 | 0 |
| 0 | −1 |
Stride 1; P = 0 (no padding).
Input A and kernel Matrix 2 are printed with the question.
","Conv: stride 1, valid padding; follow with ReLU.
","Max pool: 2 × 2, stride 1; flatten row by row.
","Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output.
","Binary target = 1.
","Find convolution, activated map, pooled map, flattened values, prediction and BCE.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Pooling. Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.
","V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.
","Yaad rakhne ka cue: Max mein biggest; average mein total ÷ window ka count.
","Weighted sum and activation. Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.
","xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.
","Yaad rakhne ka cue: Input × weight, sabka sum, bias once, phir activation.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Binary cross-entropy. For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.
","y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.
","Yaad rakhne ka cue: Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).
","A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation.
","Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability.
","N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one.
","Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer.
","Use the input size at the pooling layer. The output keeps the same channel count.
","Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p).
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Step 1 shape: = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 |
| 0 | −1 |
Each cell is input × matching kernel coefficient.
1 + (−1) = 0
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 1, column 1.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 2 | 0 |
| 0 | −3 |
Each cell is input × matching kernel coefficient.
2 + (−3) = −1
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 1, column 2.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–2, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0 | 0 |
| 0 | −2 |
Each cell is input × matching kernel coefficient.
0 + (−2) = −2
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 1, column 3.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0 | 0 |
| 0 | −1 |
Each cell is input × matching kernel coefficient.
0 + (−1) = −1
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 2, column 1.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 |
| 0 | 0 |
Each cell is input × matching kernel coefficient.
1 + 0 = 1
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 2, column 2.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 2–3, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 3 | 0 |
| 0 | −1 |
Each cell is input × matching kernel coefficient.
3 + (−1) = 2
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 2, column 3.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 1–2. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 2 | 0 |
| 0 | 0 |
Each cell is input × matching kernel coefficient.
2 + 0 = 2
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 3, column 1.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 2–3. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 |
| 0 | −2 |
Each cell is input × matching kernel coefficient.
1 + (−2) = −1
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 3, column 2.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–4, columns 3–4. The window moves 1 cell per output step.
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 |
| 0 | −1 |
Same row and column positions; no matrix multiplication.
| 0 | 0 |
| 0 | −3 |
Each cell is input × matching kernel coefficient.
0 + (−3) = −3
| 0 | −1 | −2 |
| −1 | 1 | 2 |
| 2 | −1 | −3 |
Output row 3, column 3.
Window row 1.
","Window row 2.
","Add the rows to obtain one output cell.
","Final output: . Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
ReLU turns negative entries into 0: .
Shape: = 2 on each axis; channels stay unchanged.
Step 1: select the entries.
","[0, 0, 0, 1]
","Step 2: add all selected entries.
","0 + 0 + 0 + 1 = 1
","Step 3: divide by the 4 entries in this window.
","1/4 = 0.25. Maximum = 1
","Step 1: select the entries.
","[0, 0, 1, 2]
","Step 2: add all selected entries.
","0 + 0 + 1 + 2 = 3
","Step 3: divide by the 4 entries in this window.
","3/4 = 0.75. Maximum = 2
","Step 1: select the entries.
","[0, 1, 2, 0]
","Step 2: add all selected entries.
","0 + 1 + 2 + 0 = 3
","Step 3: divide by the 4 entries in this window.
","3/4 = 0.75. Maximum = 2
","Step 1: select the entries.
","[1, 2, 0, 0]
","Step 2: add all selected entries.
","1 + 2 + 0 + 0 = 3
","Step 3: divide by the 4 entries in this window.
","3/4 = 0.75. Maximum = 2
","Average output: . Max output: . Pooling normally has zero trainable parameters and reduces spatial dimensions.
Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = 0.139386758. Average pool values were computed for contrast; the question requires max pool.
"],"kind":"practice","id":"tee-c17-s46","search":"Drill 46 Mixed convolution pool dense and loss **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE.\n\n\n\n\n**Full solution.**\n\n\n\n### Given\n\nInput A and kernel Matrix 2 are printed with the question.\n\nConv: stride 1, valid padding; follow with ReLU.\n\nMax pool: 2 × 2, stride 1; flatten row by row.\n\nDense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output.\n\nBinary target = 1.\n\n### To find\n\nFind convolution, activated map, pooled map, flattened values, prediction and BCE.\n\n### Formula — which one and why\n\n**Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.\n\nX = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.\n\n\n\n**Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar.\n\n**Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately.\n\nV = current pooling window; n = number of entries in that window. Pooling learns no weights or biases.\n\n\n\n\n\n**Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count.\n\n**Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation.\n\nxᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function.\n\n\n\n\n\n**Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y.\n\ny is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative.\n\n\n\n\n\n**Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p).\n\n### Solution — step by step\n\n\n\n\n\n### Formulas and the operation we are performing\nA filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation.\n\nHere i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability.\n\nN is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one.\n### Formulas and meaning\nEach 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the g"},{"title":"Drill 47 Mixed backpropagation and L2","question":["Question. Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight.
"],"answer":["Prediction ŷ = wx; x = 2; w = 0.4; target y = 1.
","Data loss = ½(ŷ−y)²; penalty = (λ/2)w².
","λ = 0.1; η = 0.2.
","Find total objective, its weight gradient and the new weight.
","L2 penalty with a factor of one-half. Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.
","λ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.
","Yaad rakhne ka cue: Half wala L2 derivative mein lambda × weight deta hai.
","Gradient descent. Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.
","θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.
","Yaad rakhne ka cue: Naya = purana − learning rate × current gradient.
","This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once.
","Final answer: total loss 0.028; gradient −0.36; new weight 0.472.
"],"kind":"practice","id":"tee-c17-s47","search":"Drill 47 Mixed backpropagation and L2 **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight.\n\n**Full solution.**\n\n\n\n### Given\n\nPrediction ŷ = wx; x = 2; w = 0.4; target y = 1.\n\nData loss = ½(ŷ−y)²; penalty = (λ/2)w².\n\nλ = 0.1; η = 0.2.\n\n### To find\n\nFind total objective, its weight gradient and the new weight.\n\n### Formula — which one and why\n\n**L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw.\n\nλ = penalty strength; w = old weight; J_data and g_data = data loss and its gradient.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai.\n\n**Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update.\n\nθ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number.\n\n\n\n\n\n**Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nThis objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once.\n\n\n### Step 1: prediction and error\n\n### Step 2: data loss\n\n### Step 3: regularization penalty and total objective\n\n\n### Step 4: calculate the two gradient contributions\n\n\n\n### Step 5: update\n\n**Final answer:** total loss 0.028; gradient −0.36; new weight 0.472.\n"},{"title":"Drill 48 Mixed dropout and a momentum update","question":["Question. Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1.
"],"answer":["Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5.
","Linear-output weights = [0.2,0.3]; bias = 0; target = 1.
","Half squared loss; update only the output weights.
","Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1.
","Find dropped/scaled activations, prediction, gradients, new velocities and new output weights.
","Inverted dropout. Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.
","aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.
","Yaad rakhne ka cue: Mask se keep/drop; kept value ko keep probability se divide.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients.
","The signed-velocity momentum convention is:
","Final answer: [0.41, 0.21]. The second weight moves even with zero current gradient because its old momentum is nonzero.
"],"kind":"practice","id":"tee-c17-s48","search":"Drill 48 Mixed dropout and a momentum update **Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1.\n\n**Full solution.**\n\n\n\n### Given\n\nHidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5.\n\nLinear-output weights = [0.2,0.3]; bias = 0; target = 1.\n\nHalf squared loss; update only the output weights.\n\nOld signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1.\n\n### To find\n\nFind dropped/scaled activations, prediction, gradients, new velocities and new output weights.\n\n### Formula — which one and why\n\n**Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks.\n\naᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference.\n\n\n\n\n\n**Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide.\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used\nDropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients.\n\nThe signed-velocity momentum convention is:\n\n### Step 1: apply inverted dropout\n\n### Step 2: prediction and error\n\n\n### Step 3: current weight gradients\n\n### Step 4: remembered movement plus new correction\n\n\n### Step 5: add velocities to old weights\n\n\n**Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero.\n"},{"title":"Drill 49 Mixed CNN dimensions softmax and loss","question":["Question. Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE.
"],"answer":["Input = 8 × 8 × 3.
","Conv: kernel 3, stride 1, padding 1, 2 filters.
","Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs.
","Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]).
","Find shapes, parameter counts, probabilities and CCE.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Stable softmax. Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.
","zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.
","Yaad rakhne ka cue: Largest score ghatao, exp nikalo, total se sabko divide karo.
","Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.
","yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.
","Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.
","Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride.
","The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class.
","Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = 155.
","Step 1 stabilize: subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax.
","Step 2 exponentiate: exp values = [0.367879441, 1, 0.135335283].
","Step 3 denominator: sum = 1.503214724.
","Step 4 divide each value by this same sum: probabilities = [0.244728471, 0.665240956, 0.090030573]. Check: all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.
","Correct class probability = 0.665240956; CCE = −ln(0.665240956) = 0.407605964.
"],"kind":"practice","id":"tee-c17-s49","search":"Drill 49 Mixed CNN dimensions softmax and loss **Question.** Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE.\n\n**Full solution.**\n\n\n\n### Given\n\nInput = 8 × 8 × 3.\n\nConv: kernel 3, stride 1, padding 1, 2 filters.\n\nPool: kernel 2, stride 2; flatten; dense layer with 3 outputs.\n\nLogits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]).\n\n### To find\n\nFind shapes, parameter counts, probabilities and CCE.\n\n### Formula — which one and why\n\n**Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.\n\nH,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count.\n\n**Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged.\n\nzⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1.\n\n\n\n\n\n**Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo.\n\n**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.\n\nyⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.\n\n\n\n\n\n**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.\n\n### Solution — step by step\n\n\n\n\n\n### Formulas used\nSeparate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride.\n\n\n\nThe +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class.\n\n\nConv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = **155**.\n**Step 1 stabilize: ** subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax.\n\n**Step 2 exponentiate: ** exp values = [0.367879441, 1, 0.135335283].\n\n**Step 3 denominator: ** sum = 1.503214724.\n\n**Step 4 divide each value by this same sum: ** probabilities = **[0.244728471, 0.665240956, 0.090030573]**. **Check: ** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability.\nCorrect class probability = 0.665240956; CCE = −ln(0.665240956) = **0.407605964**.\n"},{"title":"Drill 50 Three step LSTM retention and replacement","question":["Question. C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step.
"],"answer":["Initial C₀ = 0; output gate o = 0.7 at all three steps.
","Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6).
","Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order.
","LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.
","f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.
","Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.
","At each time step use the previous cell state C, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step.
","Step 1: Time 1: admit new content.
","![\"f_1C_0=1(0)=0,\\qquad](\"assets/equations/12440f871df30d1a.svg\")
![\"C_1=0+0.4=0.4\"](\"assets/equations/7d6404b116e9cbf8.svg\")
![\"h_1=0.7\\tanh(0.4)=0.7(0.379948962)=0.265964274\"](\"assets/equations/bc61a66b7daaff60.svg\")
Step 2: Time 2: retain memory and admit no new content.
","![\"f_2C_1=0.9(0.4)=0.36,\\qquad](\"assets/equations/21e9e3dc7975b5fa.svg\")
![\"C_2=0.36+0=0.36\"](\"assets/equations/e78e9762a4457abe.svg\")
![\"h_2=0.7\\tanh(0.36)=0.7(0.345214034)=0.241649824\"](\"assets/equations/36c9d733b0dbfd49.svg\")
Step 3: Time 3: retain a small part and add negative content.
","![\"f_3C_2=0.2(0.36)=0.072,\\qquad](\"assets/equations/39fc048515ad2602.svg\")
![\"C_3=0.072-0.3=-0.228\"](\"assets/equations/3e11138e5892ca48.svg\")
![\"h_3=0.7\\tanh(-0.228)=0.7(-0.224129674)=-0.156890772\"](\"assets/equations/84a567774fa02539.svg\")
Final cell states: 0.4, 0.36, −0.228. Final hidden states: approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given.
"],"kind":"practice","id":"tee-c17-s50","search":"Drill 50 Three step LSTM retention and replacement **Question.** C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step.\n\n**Full solution.**\n\n\n\n### Given\n\nInitial C₀ = 0; output gate o = 0.7 at all three steps.\n\nSupplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6).\n\n### To find\n\nCalculate C₁,h₁,C₂,h₂,C₃,h₃ in order.\n\n### Formula — which one and why\n\n**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.\n\nf,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.\n\n\n\n\n\n**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nAt each time step use the previous **cell state C**, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step.\n\n**Step 1: Time 1: admit new content.**\n\n\n\n**Step 2: Time 2: retain memory and admit no new content.**\n\n\n\n**Step 3: Time 3: retain a small part and add negative content.**\n\n\n\n**Final cell states:** 0.4, 0.36, −0.228. **Final hidden states:** approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given.\n"},{"title":"Drill 51 Inception concatenation and parameter count","question":["Question. Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters.
"],"answer":["Input = 10 × 10 × 8.
","Branch A: 1 × 1 convolution, 4 filters.
","Branch B: 3 × 3 same convolution, 6 filters.
","Branch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters.
","Find each branch shape and parameters, then the concatenated shape and total parameters.
","Inception channel concatenation. Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches.
","C₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions.
","Yaad rakhne ka cue: Channels jodo; height-width same rakho.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","Each parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values.
","Assume one bias per filter. Pooling has no learned parameters.
","Step 1: Branch A.
","Its 4 filters produce shape 10 × 10 × 4.
","Step 2: Branch B.
","Its 6 filters produce shape 10 × 10 × 6.
","Step 3: Branch C.
","Pooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2.
","Step 4: Join the shapes and add the parameters.
","Final answer: combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches.
"],"kind":"practice","id":"tee-c17-s51","search":"Drill 51 Inception concatenation and parameter count **Question.** Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters.\n\n**Full solution.**\n\n\n\n### Given\n\nInput = 10 × 10 × 8.\n\nBranch A: 1 × 1 convolution, 4 filters.\n\nBranch B: 3 × 3 same convolution, 6 filters.\n\nBranch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters.\n\n### To find\n\nFind each branch shape and parameters, then the concatenated shape and total parameters.\n\n### Formula — which one and why\n\n**Inception channel concatenation.** Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches.\n\nC₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions.\n\n\n\n\n\n**Yaad rakhne ka cue:** Channels jodo; height-width same rakho.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nEach parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values.\n\n\nAssume one bias per filter. Pooling has no learned parameters.\n**Step 1: Branch A.**\nIts 4 filters produce shape 10 × 10 × 4.\n\n**Step 2: Branch B.**\nIts 6 filters produce shape 10 × 10 × 6.\n\n**Step 3: Branch C.**\nPooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2.\n\n**Step 4: Join the shapes and add the parameters.**\n\n\n**Final answer:** combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches.\n"},{"title":"Drill 52 Projection shortcut","question":["Question. A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters.
"],"answer":["Residual input = 16 × 16 × 8; main-path output = 8 × 8 × 16.
","Shortcut kernel = 1 × 1; stride 2; padding 0.
","Use one bias per projection filter.
","Choose the number of filters and find projection output shape and parameters.
","Convolution/pooling spatial size. Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.
","N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.
","Yaad rakhne ka cue: Input + dono sides ki padding − kernel; stride se divide; floor + 1.
","Convolution parameters. Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.
","K_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.
","Yaad rakhne ka cue: Filter area × input channels × filters; phir ek bias per filter.
","Residual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16.
","One filter creates one channel, so 16 filters give the required 16 output channels.
","The floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1.
","Final answer: projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid.
"],"kind":"practice","id":"tee-c17-s52","search":"Drill 52 Projection shortcut **Question.** A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters.\n\n**Full solution.**\n\n\n\n### Given\n\nResidual input = 16 × 16 × 8; main-path output = 8 × 8 × 16.\n\nShortcut kernel = 1 × 1; stride 2; padding 0.\n\nUse one bias per projection filter.\n\n### To find\n\nChoose the number of filters and find projection output shape and parameters.\n\n### Formula — which one and why\n\n**Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1.\n\nN = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here.\n\n\n\n**Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1.\n\n**Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters.\n\nK_h,K_w = filter height/width; C_in = input channels; C_out = number of filters and output channels. Spatial output positions reuse these weights.\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter.\n\n### Solution — step by step\n\n\n\n\n\n### Meaning and formulas\nResidual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16.\n\n\n### Step 1: choose 16 filters\nOne filter creates one channel, so 16 filters give the required 16 output channels.\n### Step 2: use the specified 1 × 1 kernel and stride 2\n\n\nThe floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1.\n### Step 3: count weights and biases\n\n\n**Final answer:** projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid.\n"},{"title":"Revising solved examples","blocks":["Revisit a direct example and then its changed-number version. Write the given values, the formula and the substitutions on separate lines. If an answer differs, compare from the first line until the difference appears. Use the mixed questions after the individual methods make sense.
"],"question":[],"answer":[],"kind":"read","id":"tee-c17-s53","search":"Revising solved examples Revisit a direct example and then its changed-number version. Write the given values, the formula and the substitutions on separate lines. If an answer differs, compare from the first line until the difference appears. Use the mixed questions after the individual methods make sense.\n\n"}]},{"n":18,"title":"Complete TEE paper rehearsal","sections":[{"title":"Start here","blocks":["Three hours; 100 marks. Answer Q1 and any four of Q2–Q7. Every question below is immediately followed by its full solution. This is the supplied December 2025 historical paper used for a timed rehearsal.
","F25 = S12, 3 December 2025, with S13 as a comparison answer key. S18 is another copy of this same paper. All questions are included; its historical choice rule does not remove questions from this practice bank.
","Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible.
"],"kind":"read","id":"tee-c18-intro"},{"title":"Rehearsal Q1a","blocks":["Question (5 marks). Give four ML versus DL differences with examples. Choose and justify ML or DL for (1) 2,000 short product reviews, (2) labeled financial fraud records, (3) captions for 100,000 labeled natural-scene images.
","Complete worked answer.
","Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.
","| Comparison | Traditional ML approaches | Deep learning approaches |
|---|---|---|
| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |
| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |
| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |
| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |
These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.
","Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.
","Choices: (1) Start with traditional ML using text features because the labeled dataset is small; pretrained text models are a viable extension. (2) Start with a tabular ML classifier such as a tree ensemble; labeled structured features do not automatically require DL. (3) DL is appropriate for learning visual features and generating a word sequence; a CNN encoder with a sequence decoder is a syllabus-aligned design. Compare held-out metrics rather than claiming one method must always win.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s0","search":"Rehearsal Q1a **Question (5 marks).** Give four ML versus DL differences with examples. Choose and justify ML or DL for (1) 2,000 short product reviews, (2) labeled financial fraud records, (3) captions for 100,000 labeled natural-scene images.\n\n**Complete worked answer.**\nDeep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition.\n| Comparison | Traditional ML approaches | Deep learning approaches |\n| Features | Often depend on manually designed features | Often learn features jointly with the prediction task |\n| Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data |\n| Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory |\n| Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain |\nThese are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance.\nDrivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis.\n**Choices: ** (1) Start with traditional ML using text features because the labeled dataset is small; pretrained text models are a viable extension. (2) Start with a tabular ML classifier such as a tree ensemble; labeled structured features do not automatically require DL. (3) DL is appropriate for learning visual features and generating a word sequence; a CNN encoder with a sequence decoder is a syllabus-aligned design. Compare held-out metrics rather than claiming one method must always win.\n"},{"title":"Rehearsal Q1b","blocks":["Question (5 marks). For automated leaf-disease image classification, discuss factors affecting loss selection. For classes Normal, Mild, Severe with target [0,1,0] and predicted probabilities [0.1,0.8,0.1], compute and interpret categorical cross-entropy.
","Complete worked answer.
","Classes = Normal,Mild,Severe; target = [0,1,0].
","Predicted probabilities = [0.1,0.8,0.1]; use natural logarithms.
","Calculate categorical cross-entropy and interpret the result.
","Categorical cross-entropy. For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.
","yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.
","Yaad rakhne ka cue: One-hot mein sirf correct class ki probability ka minus ln.
","Select a loss according to the task and output meaning: real-valued regression often uses MSE/MAE, binary classification uses binary cross-entropy, and one-of-K classification uses categorical cross-entropy. Consider class imbalance, noisy labels/outliers, whether outputs are calibrated probabilities, and differentiability for optimization.
","Step 1: expand the categorical cross-entropy sum.
","Step 2: keep the term for the true class.
","Step 3: calculate its natural logarithm.
","Step 4: negate the logarithm.
","Final answer: loss ≈ 0.223144 nats. The true class received probability 0.8, so the loss is relatively small; probability 1 would give loss zero.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s1","search":"Rehearsal Q1b **Question (5 marks).** For automated leaf-disease image classification, discuss factors affecting loss selection. For classes Normal, Mild, Severe with target [0,1,0] and predicted probabilities [0.1,0.8,0.1], compute and interpret categorical cross-entropy.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nClasses = Normal,Mild,Severe; target = [0,1,0].\n\nPredicted probabilities = [0.1,0.8,0.1]; use natural logarithms.\n\n### To find\n\nCalculate categorical cross-entropy and interpret the result.\n\n### Formula — which one and why\n\n**Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class.\n\nyⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified.\n\n\n\n\n\n**Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln.\n\n### Solution — step by step\n\n\n\n\n\n\nSelect a loss according to the task and output meaning: real-valued regression often uses MSE/MAE, binary classification uses binary cross-entropy, and one-of-K classification uses categorical cross-entropy. Consider class imbalance, noisy labels/outliers, whether outputs are calibrated probabilities, and differentiability for optimization.\n**Step 1: expand the categorical cross-entropy sum.**\n\n\n\n\n\n\n\n**Step 2: keep the term for the true class.**\n\n\n\n\n\n\n\n**Step 3: calculate its natural logarithm.**\n\n\n\n\n\n\n\n**Step 4: negate the logarithm.**\n\n\n\n\n\n\n\n**Final answer:** loss ≈ 0.223144 nats. The true class received probability 0.8, so the loss is relatively small; probability 1 would give loss zero.\n"},{"title":"Rehearsal Q1c","blocks":["Question (5 marks). Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization.
","Complete worked answer.
","Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.
","L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.
","Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s2","search":"Rehearsal Q1c **Question (5 marks).** Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization.\n\n**Complete worked answer.**\nOverfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data.\n\nL1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw.\nDropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen.\n"},{"title":"Rehearsal Q1d","blocks":["Question (5 marks). Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate.
","Complete worked answer.
","A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.
","At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.
","For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.
","A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s3","search":"Rehearsal Q1d **Question (5 marks).** Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate.\n\n**Complete worked answer.**\nA recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps.\n\n\nAt each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed.\nFor next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”.\nA basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture.\n"},{"title":"Rehearsal Q2a","blocks":["Question (10 marks). A 2 → 2 → 1 sigmoid network has all four input-to-hidden weights 0.5, both hidden-to-output weights 0.25, hidden and output biases 0, and target 1. Perform forward and backward passes to update the x₁ → h₁ weight, h₁ → output weight and output bias at η = 0.5. The paper does not state the numerical inputs or loss.
","Complete worked answer.
","Architecture 2 → 2 → 1; every neuron uses sigmoid.
","All four input-to-hidden weights = 0.5; both hidden-to-output weights = 0.25.
","All biases = 0; target = 1; η = 0.5.
","The paper omits inputs and loss: this worked solution explicitly assumes x = [1,1] and half squared loss.
","Calculate the forward pass, then gradients and updates for x₁→h₁, h₁→output and the output bias.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","Half summed squared loss. This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.
","oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.
","Yaad rakhne ka cue: Half aur square ka 2 cancel: output derivative = prediction − target.
","Sigmoid and its slope. Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).
","z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.
","Yaad rakhne ka cue: Sigmoid output p; uska slope p × (1−p).
","Backpropagation. The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.
","δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.
","Yaad rakhne ka cue: Receiving delta × sending value = us connection ka gradient.
","Assumptions: use x = [1,1] and half squared loss. The paper permits suitable assumptions and its key also introduces assumed inputs. There is no unique numerical result without them.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","Hidden neuron 2:
","Output neuron 1:
","Step 2 loss convention. This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.
","Prediction minus target = [-0.409621743]. Squared errors = [0.167789972]. Half-sum squared loss = 0.083894986; mean squared error = 0.167789972. These are different unless the number of output components makes the scale coincide.
","Output delta 1:
","The derivative is o(1−o).
","Step 3 output weights. Gradient = receiving delta × sending activation. An output bias gradient equals its delta.
","h1 → o1:
","Subtract η × gradient = −0.036209168. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","h2 → o1:
","Subtract η × gradient = −0.036209168. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies its activation.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 4 hidden deltas. A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative.
","Hidden 1 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Hidden 2 downstream contribution:
","Calculate each outgoing path using the old outgoing weight and that output's delta.
","Multiply within each path. Add the resulting contributions at the shared hidden neuron.
Contribution from output 1.
","Accumulate the contributions, then apply the hidden activation derivative below.
","Step 5 input weights. Multiply the hidden delta by the corresponding raw input, not by the hidden output.
","x1 → h1:
","Subtract η × gradient = −0.002434536. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h1:
","Subtract η × gradient = −0.002434536. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x1 → h2:
","Subtract η × gradient = −0.002434536. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","x2 → h2:
","Subtract η × gradient = −0.002434536. This weight increases.
The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights.
The receiving neuron supplies the delta; the sending neuron supplies the input value.
","Apply the learning rate to this gradient and subtract the result from the old weight.
","Step 6 bias updates. A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates.
","These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it.
","Requested values: input weight 0.502434536, output weight 0.286209167, output bias 0.049529775. Other supplied inputs produce different correct conditional results.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s4","search":"Rehearsal Q2a **Question (10 marks).** A 2 → 2 → 1 sigmoid network has all four input-to-hidden weights 0.5, both hidden-to-output weights 0.25, hidden and output biases 0, and target 1. Perform forward and backward passes to update the x₁ → h₁ weight, h₁ → output weight and output bias at η = 0.5. The paper does not state the numerical inputs or loss.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nArchitecture 2 → 2 → 1; every neuron uses sigmoid.\n\nAll four input-to-hidden weights = 0.5; both hidden-to-output weights = 0.25.\n\nAll biases = 0; target = 1; η = 0.5.\n\nThe paper omits inputs and loss: this worked solution explicitly assumes x = [1,1] and half squared loss.\n\n### To find\n\nCalculate the forward pass, then gradients and updates for x₁→h₁, h₁→output and the output bias.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative.\n\noⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target.\n\n**Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output).\n\nz = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula.\n\n\n\n\n\n**Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p).\n\n**Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path.\n\nδ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question.\n\n\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient.\n\n### Solution — step by step\n\n\n\n\n**Assumptions: ** use x = [1,1] and half squared loss. The paper permits suitable assumptions and its key also introduces assumed inputs. There is no unique numerical result without them.\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption.\nPrediction minus target = [-0.409621743]. Squared errors = [0.167789972]. Half-sum squared loss = 0.083894986; mean squared error = 0.167789972. These are different unless the number of output components makes the"},{"title":"Rehearsal Q2b","blocks":["Question (10 marks). Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction.
","Complete worked answer.
","A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.
","The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.
","Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].
","An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.
","A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.
","For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.
","AlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s5","search":"Rehearsal Q2b **Question (10 marks).** Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction.\n\n**Complete worked answer.**\nA residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward.\n\n\nThe identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network.\nElementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2].\nAn Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis.\n\nA 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations.\nFor H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module.\nAlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs.\n"},{"title":"Rehearsal Q3a","blocks":["Question (10 marks). Train one epoch of a single perceptron with η = 1, initial weights and bias 0, threshold 0, on [−1,1,−1] → 0, [−1,1,1] → 1, [1,−1,1] → 1, [−1,1,1] → 1. Then map biological-neuron components to artificial-neuron components.
","Complete worked answer.
","Initial weights and bias = 0; η = 1; threshold = 0.
","Samples in order: [−1,1,−1]→0; [−1,1,1]→1; [1,−1,1]→1; [−1,1,1]→1.
","Use output 1 when net > 0 and 0 otherwise, matching the supplied key; one epoch.
","Process each sample in order, carrying updated weights forward, and report final weights/bias.
","Perceptron learning rule. Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters.
","t = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question.
","Yaad rakhne ka cue: Prediction nikalo; target−prediction se correction nikalo.
","State the threshold rule. Use binary output 1 if z>0 and 0 otherwise, matching the associated key. Update Δw = η(t−o)x and Δb = η(t−o), with η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly.
","Step 1: copy this sample and target.
","x = [-1, 1, -1], target = 0.
","Step 2: calculate the current net.
","z = (0 × -1) + (0 × 1) + (0 × -1) + (0) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = 0
","Step 5: calculate all weight changes.
","Δw = 0 × [-1, 1, -1] = [0, 0, 0]
","Step 6: calculate the bias change.
","Δb = 0.
","Step 7: carry the updated parameters forward.
","New w = [0, 0, 0], new b = 0.
","Step 1: copy this sample and target.
","x = [-1, 1, 1], target = 1.
","Step 2: calculate the current net.
","z = (0 × -1) + (0 × 1) + (0 × 1) + (0) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = 1
","Step 5: calculate all weight changes.
","Δw = 1 × [-1, 1, 1] = [-1, 1, 1]
","Step 6: calculate the bias change.
","Δb = 1.
","Step 7: carry the updated parameters forward.
","New w = [-1, 1, 1], new b = 1.
","Step 1: copy this sample and target.
","x = [1, -1, 1], target = 1.
","Step 2: calculate the current net.
","z = (-1 × 1) + (1 × -1) + (1 × 1) + (1) = 0
","Step 3: apply the threshold.
","output = 0.
","Step 4: calculate the update multiplier.
","Update multiplier = 1
","Step 5: calculate all weight changes.
","Δw = 1 × [1, -1, 1] = [1, -1, 1]
","Step 6: calculate the bias change.
","Δb = 1.
","Step 7: carry the updated parameters forward.
","New w = [0, 0, 2], new b = 2.
","Step 1: copy this sample and target.
","x = [-1, 1, 1], target = 1.
","Step 2: calculate the current net.
","z = (0 × -1) + (0 × 1) + (2 × 1) + (2) = 4
","Step 3: apply the threshold.
","output = 1.
","Step 4: calculate the update multiplier.
","Update multiplier = 0
","Step 5: calculate all weight changes.
","Δw = 0 × [-1, 1, 1] = [0, 0, 0]
","Step 6: calculate the bias change.
","Δb = 0.
","Step 7: carry the updated parameters forward.
","New w = [0, 0, 2], new b = 2.
","Final after exactly the requested epochs: w = [0, 0, 2], b = 2. A finite number of requested epochs does not itself prove convergence or correct classification of every sample.
","Biological dendrites receive signals, synapses influence connection strength, the cell body integrates information, and an axon carries an output signal. In the artificial analogy, inputs represent received signals, weights represent connection strengths, the weighted sum plus bias performs aggregation, and an activation function produces an output. The bias shifts the firing threshold. This is a simplified computational analogy, not a complete biological model.
","Question (10 marks). Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning.
","Complete worked answer.
","Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.
","1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.
","2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.
","3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.
","4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.
","5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.
","6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.
","Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s7","search":"Rehearsal Q3b **Question (10 marks).** Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning.\n\n**Complete worked answer.**\nTransfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset.\n1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split.\n2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes.\n3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy.\n4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage.\n5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization.\n6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior.\nFeature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition.\n"},{"title":"Rehearsal Q4a","blocks":["Question (10 marks). For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters.
","Complete worked answer.
","N = 1,000; mini-batch size = 100.
","J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9.
","Use signed velocity.
","Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations.
","Number of updates. An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.
","N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.
","Yaad rakhne ka cue: Kitne batches process hue, utne updates hue.
","Momentum with signed velocity. Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.
","v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.
","Yaad rakhne ka cue: Pehle velocity banao; phir old parameter mein velocity jodo.
","Step 1: count full-batch updates.
","All 1,000 samples contribute to one update: 1 per epoch.
","Step 2: count stochastic updates.
","One update per sample: 1,000 per epoch.
","Step 3: count mini-batches.
","Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ.
","Step 1: evaluate the gradient at the current parameter.
","Step 2: retain the previous displacement.
","Step 3: scale the current gradient.
","Step 4: combine to obtain the new signed velocity.
","Step 5: add new velocity to the actual old parameter.
","Step 1: evaluate the gradient at the current parameter.
","Step 2: retain the previous displacement.
","Step 3: scale the current gradient.
","Step 4: combine to obtain the new signed velocity.
","Step 5: add new velocity to the actual old parameter.
","Hyperparameters: the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s8","search":"Rehearsal Q4a **Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters.\n\n**Complete worked answer.**\n\n\n\n### Given\n\nN = 1,000; mini-batch size = 100.\n\nJ(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9.\n\nUse signed velocity.\n\n### To find\n\nCount updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations.\n\n### Formula — which one and why\n\n**Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy.\n\nN = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue.\n\n**Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter.\n\nv = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update.\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo.\n\n### Solution — step by step\n\n\n\n\n\n\n**Step 1: count full-batch updates.**\n\nAll 1,000 samples contribute to one update: 1 per epoch.\n\n**Step 2: count stochastic updates.**\n\nOne update per sample: 1,000 per epoch.\n\n**Step 3: count mini-batches.**\n\n\n\n\n\n\n\n\nUse signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ.\n### Momentum iteration 1\n\n**Step 1: evaluate the gradient at the current parameter.**\n\n\n\n\n\n\n\n**Step 2: retain the previous displacement.**\n\n\n\n\n\n\n\n**Step 3: scale the current gradient.**\n\n\n\n\n\n\n\n**Step 4: combine to obtain the new signed velocity.**\n\n\n\n\n\n\n\n**Step 5: add new velocity to the actual old parameter.**\n\n\n\n\n\n\n\n\n### Momentum iteration 2\n\n**Step 1: evaluate the gradient at the current parameter.**\n\n\n\n\n\n\n\n**Step 2: retain the previous displacement.**\n\n\n\n\n\n\n\n**Step 3: scale the current gradient.**\n\n\n\n\n\n\n\n**Step 4: combine to obtain the new signed velocity.**\n\n\n\n\n\n\n\n**Step 5: add new velocity to the actual old parameter.**\n\n\n\n\n\n\n\n\n**Hyperparameters: ** the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update.\n"},{"title":"Rehearsal Q4b","blocks":["Question (10 marks). Define vanishing gradients in RNNs and explain why they occur. Use “The dogs barked at the thief and … were appreciated” to discuss long context. Explain LSTM forgetting/retention, then compute the retained contribution for f = [0.1,0.9], Cprevious = [2,1].
","Complete worked answer.
","Forget-gate outputs f = [0.1,0.9]; old cell = [2,1].
","No input gate or candidate is supplied.
","Explain forgetting/retention and calculate the retained old-cell part only.
","LSTM retained old memory. Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell.
","f = activated forget gate; C_old = previous cell state; ⊙ means multiply matching entries.
","Yaad rakhne ka cue: Forget gate × old cell = bacha hua purana hissa.
","Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.
","1. Initialize the hidden state and run the forward recurrence from the first input to the last.
","2. Compute the specified output loss, at the final step or across several steps.
","3. Start from the last loss and propagate derivatives backwards through output and hidden operations.
","4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.
","5. Sum all uses of each shared weight before applying the optimizer update.
","The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.
","The plural subject “dogs” must remain represented across intervening words. A vanilla RNN may fail to learn that dependency when early gradients are weak; it is not guaranteed to fail on every such sentence. In an LSTM, forget components close to 1 retain relevant subject information along the cell path.
","Step 1: retain the first old-cell component.
","Step 2: retain the second old-cell component.
","Step 3: assemble the retained old-cell contribution.
","Final answer: [0.2,0.9] is only retained old content. The input gate and candidate are missing, so the complete updated cell cannot be calculated.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s9","search":"Rehearsal Q4b **Question (10 marks).** Define vanishing gradients in RNNs and explain why they occur. Use “The dogs barked at the thief and … were appreciated” to discuss long context. Explain LSTM forgetting/retention, then compute the retained contribution for f = [0.1,0.9], Cprevious = [2,1].\n\n**Complete worked answer.**\n\n\n\n### Given\n\nForget-gate outputs f = [0.1,0.9]; old cell = [2,1].\n\nNo input gate or candidate is supplied.\n\n### To find\n\nExplain forgetting/retention and calculate the retained old-cell part only.\n\n### Formula — which one and why\n\n**LSTM retained old memory.** Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell.\n\nf = activated forget gate; C_old = previous cell state; ⊙ means multiply matching entries.\n\n\n\n**Yaad rakhne ka cue:** Forget gate × old cell = bacha hua purana hissa.\n\n### Solution — step by step\n\n\n\n\nBackpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph.\n1. Initialize the hidden state and run the forward recurrence from the first input to the last.\n2. Compute the specified output loss, at the final step or across several steps.\n3. Start from the last loss and propagate derivatives backwards through output and hidden operations.\n4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states.\n5. Sum all uses of each shared weight before applying the optimizer update.\n\nThe tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step.\n\nThe plural subject “dogs” must remain represented across intervening words. A vanilla RNN may fail to learn that dependency when early gradients are weak; it is not guaranteed to fail on every such sentence. In an LSTM, forget components close to 1 retain relevant subject information along the cell path.\n**Step 1: retain the first old-cell component.**\n\n\n\n\n\n\n\n**Step 2: retain the second old-cell component.**\n\n\n\n\n\n\n\n**Step 3: assemble the retained old-cell contribution.**\n\n\n\n\n\n\n\n**Final answer:** [0.2,0.9] is only retained old content. The input gate and candidate are missing, so the complete updated cell cannot be calculated.\n"},{"title":"Rehearsal Q5a","blocks":["Question (10 marks). State the purpose of each LSTM gate. For “Artificial Intelligence has revolutionized every domain. The work force will have to update themselves to remain relevant,” explain a context change. Calculate C and h given i = 0.7, f = 0.4, o = 0.8, candidate = 0.9, Cprevious = 0.5. Why is LSTM more effective than a simple RNN for language modeling?
","Complete worked answer.
","Activated gate values: i = 0.7,f = 0.4,o = 0.8.
","Candidate = 0.9; old cell = 0.5.
","Explain the context change in the question; calculate the new cell and hidden state.
","LSTM cell and hidden state. Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.
","f,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.
","Yaad rakhne ka cue: Purana rakho + naya lo; phir tanh aur output gate.
","An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.
","The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.
","In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.
","The second sentence shifts attention toward the workforce while retaining useful prior AI context. Input gates can admit the new subject; forget gates can reduce obsolete details.
","Step 1: retain old memory.
","Step 2: admit candidate content.
","Step 3: add the two contributions.
","Step 4: apply tanh to the updated cell.
","Step 5: multiply by the output gate.
","Final answer: C = 0.83 and h ≈ 0.544380805. The supplied gates are already activated; do not apply sigmoid again.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s10","search":"Rehearsal Q5a **Question (10 marks).** State the purpose of each LSTM gate. For “Artificial Intelligence has revolutionized every domain. The work force will have to update themselves to remain relevant,” explain a context change. Calculate C and h given i = 0.7, f = 0.4, o = 0.8, candidate = 0.9, Cprevious = 0.5. Why is LSTM more effective than a simple RNN for language modeling?\n\n**Complete worked answer.**\n\n\n\n### Given\n\nActivated gate values: i = 0.7,f = 0.4,o = 0.8.\n\nCandidate = 0.9; old cell = 0.5.\n\n### To find\n\nExplain the context change in the question; calculate the new cell and hidden state.\n\n### Formula — which one and why\n\n**LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate.\n\nf,i,o = activated forget/input/output gates; C_old = previous cell; C_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication.\n\n\n\n\n\n**Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate.\n\n### Solution — step by step\n\n\n\n\nAn LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content.\n\n\n\n\nThe forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components.\nIn language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory.\nThe second sentence shifts attention toward the workforce while retaining useful prior AI context. Input gates can admit the new subject; forget gates can reduce obsolete details.\n**Step 1: retain old memory.**\n\n\n\n\n\n\n\n**Step 2: admit candidate content.**\n\n\n\n\n\n\n\n**Step 3: add the two contributions.**\n\n\n\n\n\n\n\n**Step 4: apply tanh to the updated cell.**\n\n\n\n\n\n\n\n**Step 5: multiply by the output gate.**\n\n\n\n\n\n\n\n**Final answer:** C = 0.83 and h ≈ 0.544380805. The supplied gates are already activated; do not apply sigmoid again.\n"},{"title":"Rehearsal Q5b","blocks":["Question (10 marks). Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD.
","Complete worked answer.
","AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:
","Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.
","Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.
","Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s11","search":"Rehearsal Q5b **Question (10 marks).** Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD.\n\n**Complete worked answer.**\nAdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise:\n\nFrequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change.\nExample with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864.\nAdvantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average.\n"},{"title":"Rehearsal Q6a","blocks":["Question (10 marks). Calculate every activation shape and trainable-parameter count for: Input 128 × 128 × 3. → Conv K = 3, filters = 16, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 3, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 240 units. → Dense 120 units. → Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2.
","| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Stride 2; P = 0 (no padding).
Complete worked answer.
","Input 128 × 128 × 3.
","Conv K = 3, filters = 16, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Conv K = 3, filters = 32, stride = 1, padding = 0.
","Pool K = 2, stride = 2, padding = 0.
","Dense 240 units.
","Dense 120 units.
","Dense 5 units.
","Also convolve the input A with the kernel K shown below, valid padding and stride 2.
","Find every layer shape, its parameter count and the network total. Find every convolution output cell.
","Complete CNN calculation. Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs.
","H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise.
","Yaad rakhne ka cue: Ek layer ka output, next layer ka input; shape aur parameters alag count.
","Convolution values. At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat.
","X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention.
","Yaad rakhne ka cue: Matching cells multiply; sab products add; bias sirf ek baar.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","Use output size , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters.
Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 128 × 128 × 3. Use 16 filters of spatial size 3 × 3, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 126 × 126 × 16.
","3. Count trainable values. Each filter receives all 3 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 126 × 126 × 16. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 63 × 63 × 16.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","Spatial positions share the filter weights. Output depth equals the number of filters.
Input: 63 × 63 × 16. Use 32 filters of spatial size 3 × 3, stride 1, and padding 0 on each side.
","1. Calculate height and width.
","2. Determine output channels. Each filter produces one feature map. Therefore the output is 61 × 61 × 32.
","3. Count trainable values. Each filter receives all 16 input channels. Its weights are reused at every spatial position.
","Spatial dimensions change; the channel count stays the same. Pooling learns no weights.
Input: 61 × 61 × 32. Use a 2 × 2 window, stride 2, and zero padding.
","Pooling acts independently on each channel. The output is 30 × 30 × 32.
","Trainable parameters: 0. Max/average pooling has no learned filter weights or biases.
","The division gives 29.5; take its floor before adding 1. A partial window outside the unpadded input is not included.
","The same activation values are rearranged into one vector. No parameter is learned.
Input: 30 × 30 × 32. Rearrange all activation values into one vector.
","Output: a vector with 28,800 entries. Flattening changes the arrangement, so it adds 0 trainable parameters.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 28,800 features. Output: 240 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 240 features. Output: 120 neurons. Every output neuron connects to every input feature and has one bias.
","Every output neuron receives every input feature, with one bias per output neuron.
Input: 120 features. Output: 5 neurons. Every output neuron connects to every input feature and has one bias.
","| Layer | Spatial or flatten calculation | Output | Weights | Biases | Total |
|---|---|---|---|---|---|
| 1 conv | | 126 x 126 x 16 | 432 | 16 | 448 |
| 2 pool | | 63 x 63 x 16 | 0 | 0 | 0 |
| 3 conv | | 61 x 61 x 32 | 4,608 | 32 | 4,640 |
| 4 pool | | 30 x 30 x 32 | 0 | 0 | 0 |
| 5 flat | 30 × 30 × 32 | 28800 | 0 | 0 | 0 |
| 6 dense | 28800 → 240 | 240 | 6,912,000 | 240 | 6,912,240 |
| 7 dense | 240 → 120 | 120 | 28,800 | 120 | 28,920 |
| 8 dense | 120 → 5 | 5 | 600 | 5 | 605 |
Parameter substitutions, in layer order:
","Layer 1: (3 x 3 x 3 + 1) x 16 = 448.
","Layer 3: (3 x 3 x 16 + 1) x 32 = 4,640.
","Layer 6: (28800 + 1) x 240 = 6,912,240.
","Layer 7: (240 + 1) x 120 = 28,920.
","Layer 8: (120 + 1) x 5 = 605.
","Total: 448 + 4,640 + 6,912,240 + 28,920 + 605 = 6,946,853 trainable parameters. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions.
","Separate convolution calculation:
","Step 1 shape: = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel.
Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 1–3. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 3 | 0 | −2 |
| 4 | 0 | −4 |
| 3 | 0 | −1 |
Each cell is input × matching kernel coefficient.
1 + 0 + 2 = 3
| 3 | 5 |
| 3 | −7 |
Output row 1, column 1.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 1–3, columns 3–5. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 2 | 0 | −1 |
| 4 | 0 | 0 |
| 1 | 0 | −1 |
Each cell is input × matching kernel coefficient.
1 + 4 + 0 = 5
| 3 | 5 |
| 3 | −7 |
Output row 1, column 2.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 1–3. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 3 | 0 | −1 |
| 0 | 0 | 0 |
| 1 | 0 | 0 |
Each cell is input × matching kernel coefficient.
2 + 0 + 1 = 3
| 3 | 5 |
| 3 | −7 |
Output row 2, column 1.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Use this position's input window. Multiply corresponding entries row by row; then add all row totals.
","Input rows 3–5, columns 3–5. The window moves 2 cells per output step.
| 3 | 1 | 2 | 2 | 1 |
| 2 | 1 | 2 | 1 | 0 |
| 3 | 2 | 1 | 0 | 1 |
| 0 | 3 | 0 | 1 | 2 |
| 1 | 1 | 0 | 2 | 3 |
Only the outlined cells belong to this output.
| 1 | 0 | −1 |
| 2 | 0 | −2 |
| 1 | 0 | −1 |
Same row and column positions; no matrix multiplication.
| 1 | 0 | −1 |
| 0 | 0 | −4 |
| 0 | 0 | −3 |
Each cell is input × matching kernel coefficient.
0 + (−4) + (−3) = −7
| 3 | 5 |
| 3 | −7 |
Output row 2, column 2.
Window row 1.
","Window row 2.
","Window row 3.
","Add the rows to obtain one output cell.
","Final output: . Each output came from its own complete window; stride changes the next window position, not the multiplication rule.
Question (10 marks). Explain activations and the role of nonlinearity. Compare sigmoid, tanh, ReLU and Leaky ReLU including advantages and disadvantages.
","Complete worked answer.
","An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.
","Sigmoid
","Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.
","Hyperbolic tangent
","The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.
","ReLU
","The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.
","Leaky ReLU
","Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.
","Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s13","search":"Rehearsal Q6b **Question (10 marks).** Explain activations and the role of nonlinearity. Compare sigmoid, tanh, ReLU and Leaky ReLU including advantages and disadvantages.\n\n**Complete worked answer.**\nAn activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR.\n**Sigmoid**\n\nHere z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25.\n**Hyperbolic tangent**\n\nThe numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate.\n**ReLU**\n\nThe output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive.\n**Leaky ReLU**\n\nAlpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent.\nSigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable.\n"},{"title":"Rehearsal Q7a","blocks":["Question (10 marks). Explain feedforward information flow, weights, biases and activations, then calculate the output. Input x = [1, 0]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU.
","Complete worked answer.
","Input x = [1, 0].
","W₁ is printed with the question.
","b₁ = [0.1, 0.2].
","W₂ is printed with the question.
","b₂ = [0.1].
","Hidden and output activations are ReLU.
","Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.
","Forward propagation. Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.
","x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.
","Yaad rakhne ka cue: Pehli layer ka activated output hi agli layer ka input hai.
","ReLU and Leaky ReLU. ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.
","z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.
","Yaad rakhne ka cue: Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.
","A feedforward network transforms inputs layer by layer with no recurrent feedback. Weights scale contributions; bias shifts the preactivation; activation supplies the nonlinear transformation.
","Step 1 map the arrows. Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.
","Hidden neuron 1:
","Hidden neuron 2:
","Output neuron 1:
","Matrix-orientation note: this solution uses column input and rows as receiving neurons, giving 0.55. The historical key uses the transpose interpretation for W₁ and obtains 0.61. If the problem does not define its orientation, state your convention; do not silently swap it during calculation. Under the alternative: hidden values = [0.3,0.6], output = 0.5 × 0.3+0.6 × 0.6+0.1 = 0.61.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s14","search":"Rehearsal Q7a **Question (10 marks).** Explain feedforward information flow, weights, biases and activations, then calculate the output. Input x = [1, 0]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU.\n\n\n\n**Complete worked answer.**\n\n\n\n### Given\n\nInput x = [1, 0].\n\nW₁ is printed with the question.\n\nb₁ = [0.1, 0.2].\n\nW₂ is printed with the question.\n\nb₂ = [0.1].\n\nHidden and output activations are ReLU.\n\n### To find\n\nFind all preactivations, hidden activations and final outputs. Find the requested activation values and slopes.\n\n### Formula — which one and why\n\n**Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases.\n\nx = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations.\n\n\n\n\n\n\n\n\n\n**Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai.\n\n**ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope.\n\nz = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention.\n\n\n\n\n\n**Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do.\n\n### Solution — step by step\n\n\n\n\nA feedforward network transforms inputs layer by layer with no recurrent feedback. Weights scale contributions; bias shifts the preactivation; activation supplies the nonlinear transformation.\n**Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass.\n**Hidden neuron 1: **\n\n\n**Hidden neuron 2: **\n\n\n**Output neuron 1: **\n\n\n**Matrix-orientation note: ** this solution uses column input and rows as receiving neurons, giving **0.55**. The historical key uses the transpose interpretation for W₁ and obtains 0.61. If the problem does not define its orientation, state your convention; do not silently swap it during calculation. Under the alternative: hidden values = [0.3,0.6], output = 0.5 × 0.3+0.6 × 0.6+0.1 = 0.61.\n"},{"title":"Rehearsal Q7b","blocks":["Question (10 marks). Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages.
","Complete worked answer.
","A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification.
","Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities.
","Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation.
","For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions.
"],"question":[],"answer":[],"kind":"read","id":"tee-c18-s15","search":"Rehearsal Q7b **Question (10 marks).** Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages.\n\n**Complete worked answer.**\nA convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification.\n\nConvolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities.\nCompared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation.\nFor medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions.\n\n"}]},{"n":19,"title":"Formula reference and exam revision","sections":[{"title":"The one page method map","blocks":["| Question clue | First action | Core formula or check |
|---|---|---|
| Neuron output | Multiply each input by its matching weight | , then activate |
| Network prediction | Finish one whole layer before the next | |
| Backpropagation | State loss then prediction-minus-target | delta times sending activation |
| Optimizer update | Identify current gradient and stored state | old parameter minus update |
| CNN dimensions | Write N K S P and channels | |
| CNN parameters | Count one filter then all filters | |
| Pooling | Draw every valid window | Maximum, or mean of the window |
| Dense parameters | Count connections and receiving biases | |
| RNN hidden state | Add new-input and old-state transforms | ![\"\\tanh(W_{xh}x+W_{hh}h_{t-1}+b)\"](\"assets/equations/99df7aa8ced09e2c.svg\") |
| LSTM | Retain old add new then expose | Retain old cell, add admitted candidate, then apply output gate |
| GRU | Check the stated z convention | ![\"h=(1-z)h_{t-1}+z\\widetilde](\"assets/equations/9abbc158fb98ec74.svg\") |
These are the formulas used by this course. Each entry names its symbols and the type of calculation it serves. Numerical practice in the teaching chapters shows the longer working. Follow any different convention explicitly stated in the examination.
","Symbols: n is the count; mu is the mean; v is the population variance; s is the standard deviation.
","Use: Use when calculating a population mean/variance or standardization. Sample-variance estimates may use n−1 when explicitly requested.
","Substitution example: For [2,4]: mean = 3; deviations = −1,1; squares = 1,1; variance = 1; standard deviation = 1.
","Symbols: Use the mean and standard deviation estimated from training data.
","Use: Use to express a value relative to the training distribution. The standard deviation must be nonzero.
","Substitution example: For x = 7, mean = 5, standard deviation = 2: (7−5)/2 = 1.
","Symbols: c is a constant, x is the changing variable and n is the power.
","Use: Use to differentiate the loss supplied in a gradient-descent question.
","Substitution example: For 0.7x³: derivative = 0.7 × 3 × x² = 2.1x².
","Symbols: z is the intermediate value linking w to L.
","Use: Multiply derivatives along a path; add contributions from distinct paths that meet.
","Substitution example: For z = 3w+1 and L = z² at w = 1: z = 4; derivative = (2×4)×3 = 24.
","Symbols: x is the input; w is its matching weight; b is one bias; z is the weighted sum; a is the activated output.
","Use: Multiply each matching pair, add the products, add bias once, then activate.
","Substitution example: x = [2,1], w = [0.5,−0.2], b = 0.1: z = 1−0.2+0.1 = 0.9.
","Symbols: i selects a row of A; j selects a column of B; k runs along the matching entries.
","Use: Use row-times-column sums. A with shape m×n and B with n×p produces m×p.
","Substitution example: Row [1,2] times column [3,4] gives 1×3+2×4 = 11.
","Symbols: l labels the layer; the previous layer activations become inputs to the next layer.
","Use: For receiving-neuron rows, compute each row dot product and add that receiving neuron’s bias. Finish the whole layer before moving on.
","Symbols: e is approximately 2.71828; sigma is the output.
","Use: Use for a sigmoid forward pass and for its local backpropagation derivative.
","Substitution example: At z = 0: output = 1/(1+1) = 0.5; derivative = 0.5×0.5 = 0.25.
","Symbols: Tanh gives an output between −1 and 1.
","Use: Use for a stated tanh activation, commonly an RNN hidden state or recurrent candidate.
","Substitution example: At z = 0: numerator = 0, denominator = 2; output = 0; derivative = 1.
","Symbols: max means choose the larger value.
","Use: Use for ReLU. At zero the derivative is mathematically undefined; use the convention stated by the question or implementation.
","Substitution example: For −2, output = 0 and derivative = 0. For 3, output = 3 and derivative = 1.
","Symbols: a is the small negative-side slope, commonly 0.01 unless specified.
","Use: For negative z multiply by a; the output remains negative.
","Substitution example: For z = −2 and a = 0.01: output = −0.02; derivative on this side is 0.01.
","Symbols: z values are class scores; p values are probabilities.
","Use: Subtract the same maximum from every score, exponentiate, add, divide each by that total.
","Substitution example: For [0,0]: exponentials = [1,1]; total = 2; probabilities = [0.5,0.5].
","Symbols: o is prediction; y is target; n is the number of averaged terms.
","Use: Read whether the question uses a half sum, a mean, or an unhalved sum. The scaling changes gradients.
","Substitution example: For errors [−1,2]: half sum = (1+4)/2 = 2.5; MSE = 2.5; MAE = (1+2)/2 = 1.5. The first two coincide only because n = 2.
","Symbols: y is 0 or 1; p is the predicted probability for class 1; ln is the natural logarithm.
","Use: Use for a sigmoid binary prediction. If y = 1 use −ln(p); if y = 0 use −ln(1−p).
","Substitution example: For y = 1 and p = 0.8: loss = 0.223144.
","For a soft target distribution, retain every nonzero target-weighted term. Natural logs give nats; base-2 logs give bits.
","Symbols: The final equality assumes one-hot labels, with exactly one y equal to 1.
","Use: Use for mutually exclusive multiclass labels. Select the probability of the true class.
","Substitution example: True class 2 and probabilities [0.1,0.8,0.1]: loss = −ln(0.8) = 0.223144.
","Symbols: Delta is the loss derivative with respect to the preactivation z.
","Use: Use with half squared loss. For unhalved one-output MSE multiply by 2; for an MSE averaged over m outputs multiply the half-sum delta by 2/m.
","Substitution example: Sigmoid output 0.5 and target 1: delta = (−0.5)×0.25 = −0.125.
","Symbols: p and y can be vectors for softmax.
","Use: This is the combined derivative with respect to the logits. Do not multiply by the activation derivative again.
","Substitution example: Binary p = 0.8, y = 1: delta = −0.2.
","Symbols: a_i is the activation sending a signal along the weight; delta_j belongs to the receiving neuron.
","Use: Use OLD downstream weights throughout one backward pass. Add all outgoing-path contributions.
","Substitution example: If receiving delta = −0.2 and sending activation = 3: weight gradient = −0.6; bias gradient = −0.2.
","Symbols: theta is the parameter; eta is learning rate; g is the current gradient.
","Use: Recalculate g at the current parameter unless the question supplies gradients directly.
","Substitution example: theta = 1, eta = 0.1, g = −2 gives 1−0.1(−2) = 1.2.
","Symbols: B is batch size; N is sample count; U is updates per epoch when the final incomplete batch is retained.
","Use: SGD uses B = 1. Full batch uses B = N. Average only when the stated objective is a mean.
","Substitution example: N = 105, B = 20: six updates, with five samples in the last batch.
","Symbols: v stores signed movement; beta retains part of old movement.
","Use: Use this convention consistently. Other textbooks may define an unsigned gradient accumulator and subtract it instead.
","Substitution example: beta = 0.8, old v = −0.2, eta = 0.1, g = 1.8 gives v = −0.34.
","Symbols: The look-ahead point is temporary; the update is added to the original theta.
","Use: Evaluate the gradient at the look-ahead point, not at the old theta.
","Substitution example: theta = 1.8, beta = 0.8 and v = −0.2 gives look-ahead = 1.64.
","Symbols: G is accumulated squared gradient; epsilon prevents division by zero.
","Use: Accumulate every squared gradient, separately for each parameter. Use the epsilon placement stated by the problem.
","Substitution example: Old G = 4 and g = −3 gives G = 13.
","Symbols: rho is the decay; s is a decaying mean of squared gradients.
","Use: Calculate the new square memory first, then update the parameter.
","Substitution example: rho = 0.9, old s = 4, g = −3 gives s = 3.6+0.9 = 4.5.
","Symbols: m remembers signed gradients; v remembers squared gradients.
","Use: Start with the initial states specified. Common defaults are beta1 = 0.9 and beta2 = 0.999.
","Symbols: t counts updates starting at 1, not 0.
","Use: Correct both zero-initialized memories before the update; do not multiply by the gradient a second time.
","Substitution example: At t = 1, m = 0.2 and beta1 = 0.9: corrected m = 0.2/0.1 = 2.
","Symbols: lambda is regularization strength; sign(w) is +1 for positive w and −1 for negative w.
","Use: At zero, L1 has a subgradient rather than a unique derivative; use the specified convention. Include only parameters named in the penalty.
","Substitution example: w = −2, lambda = 0.1, data gradient = 0 gives penalty gradient = −0.1.
","Symbols: The half cancels the 2 produced by differentiating a square.
","Use: If the objective uses lambda times the sum of squares WITHOUT a half, use 2 lambda w instead.
","Substitution example: w = 2, lambda = 0.1, data gradient = 0.3 gives total gradient = 0.5.
","Symbols: p is drop probability; q is keep probability; mask m contains zeros and ones; the circled dot means elementwise multiplication.
","Use: During training, zero dropped entries and divide kept entries by q. Disable dropout at inference.
","Substitution example: a = 3, q = 0.5, mask = 1 gives 3/0.5 = 6.
","Symbols: N is input height or width; P is padding on each side; K is kernel size; S is stride; floor rounds down.
","Use: Apply separately to height and width. Output channels equal the number of filters. This formula assumes dilation 1.
","Substitution example: N = 7, P = 1, K = 3, S = 2 gives floor(6/2)+1 = 4.
","Symbols: c runs over input channels; u,v run over kernel positions; o selects one filter.
","Use: This is the usual deep-learning cross-correlation operation. Use zeros outside the padded image. Multiply matching cells, add all channels, then add the filter bias once.
","Min pooling keeps the smallest entry in each window. For a four-entry window:
","Symbols: W is one pooling window; its size is the number of included cells.
","Use: Move the window by the stated stride; process channels separately. For padding, follow the stated average-pooling denominator convention.
","Substitution example: For [1,4,2,3]: maximum = 4; average = 10/4 = 2.5.
","Symbols: Kernel height and width times input channels gives weights per filter; +1 is one bias; Cout is filter count.
","Use: Use for a fully connected, ungrouped convolution with bias. Output height and width do not multiply the parameter count.
","Substitution example: 3×3 kernel, 3 input channels, 2 filters: (27+1)×2 = 56.
","Symbols: H,W,C are activation dimensions; Nin and Nout are dense-layer input and output counts.
","Use: Flatten has no trainable parameters. Dense biases belong to receiving neurons.
","Substitution example: 2×2×2 flattens to 8; dense 3 then has (8+1)×3 = 27 parameters.
","Symbols: Each output value uses kernel area times input-channel count multiplications.
","Use: Used for Inception bottleneck comparisons. This counts one multiply-accumulate as one MAC and ignores bias additions. Some FLOP conventions count multiply and add separately.
","Symbols: b labels branches with matching spatial dimensions.
","Use: Inception concatenates channels; residual addition requires matching full shapes. Use a projection when needed.
","Substitution example: Channels 16+32+8+8 give 64. Adding two 14×14×64 tensors keeps 14×14×64.
","Symbols: x_t is current input; h_previous is previous state; the same matrices are reused at every time step.
","Use: Calculate the two matrix-vector products, add bias, then tanh. Apply the stated output activation to o_t when required.
","![\"\\widehat](\"assets/equations/2fd235815e859770.svg\")
Symbols: The step threshold and label encoding must match the question.
","Use: Use for a stated perceptron learning rule; this is not the sigmoid derivative rule.
","![\"\\delta_t=\\left[W_{hy}^T(p_t-y_t)+W_{hh}^T\\delta_{t+1}\\right]\\odot(1-h_t^2)\"](\"assets/equations/77be07478b9cbea0.svg\")
Symbols: Here p_t is the softmax probability vector. The output term assumes softmax plus cross entropy at each stated output step; omit it at steps with no direct loss.
","Use: Backpropagate from the last time step. Shared-weight gradients sum contributions across time.
","For gradient norm clipping, τ > 0 is the maximum norm. Leave a zero gradient unchanged. This is a remedy for exploding gradients; it is distinct from the shared-weight gradient formula below.
","![\"\\frac{\\partial](\"assets/equations/9aa073a86bb67d16.svg\")
Symbols: These are outer products, not scalar dot products.
","Use: Sum each occurrence of a shared weight. Apply the learning-rate update after accumulating the required contributions.
","Symbols: f is forget gate; i is input gate; [h,x] means concatenation.
","Use: Compute each gate from its own weights and bias. Gate values lie between 0 and 1.
","![\"\\widetilde](\"assets/equations/ffe858586bf2d707.svg\")
Symbols: The candidate can be negative; the gates are sigmoid outputs.
","Use: Do not confuse the candidate value with its input gate.
","![\"C_t=f_t\\odot](\"assets/equations/e295aa726de62b85.svg\")
Symbols: C is the cell state; h is the exposed hidden state.
","Use: Calculate old retained part, new admitted part, sum them, apply tanh, then multiply by output gate.
","Symbols: z is update gate; r is reset gate.
","Use: Use the exact gate and reset placement convention in the question.
","![\"\\widetilde](\"assets/equations/3206de1fb9907f41.svg\")
Symbols: In THIS convention, z weights the old state. Some sources reverse the two coefficients.
","Use: Read the stated update equation before substituting. For vector states, all gate products here are elementwise.
","Use during training to normalize intermediate activations before a learned scale and shift. m is the number of values; μ_B and σ²_B are the batch mean and population variance; γ and β are learned. Epsilon is inside the square root. Ordinary inference uses stored running statistics. The linked full lesson calculates every stage for [2,4].
","Take a separate average for each full spatial feature map. H and W count rows and columns; c selects one of C channels; a is an activation value. Output has C values. The linked lesson computes two channel averages and the following dense-layer parameters.
","N is the total number of evaluated examples. C_k counts those whose true class is among the k highest-scoring predictions. Count failures, divide by total, then convert to percent. The worked lesson contrasts top-1 and top-5 using the same 100 images.
","The reported model name is “Casio FX ES 990”; the exact model is not confirmed. Use your calculator’s actual function labels. On common ES/ES Plus models, exp(x) is usually the shifted ln key, while the EXP/ × 10ˣ key enters scientific notation and is NOT the same as exp(x). Check the exact key labeling on your device.
","For sigmoid at 2.2, type 1÷(1+exp(−2.2)). Keep the negative sign inside the exponential. Expected 0.900249511. For a negative gradient update, type old−rate × (negative gradient); use parentheses around the negative value. For cross-entropy, use −ln(probability), not log 10. For tanh, use the hyperbolic function or its exponential fraction if that function is difficult to find.
","Retain 6–9 digits in intermediate results or use stored values; round final answers to 4–6 decimal places unless the paper says otherwise. Work line by line on paper even if the calculator can evaluate the whole expression. Check exp(0) = 1, ln(1) = 0, sigmoid(0) = 0.5, tanh(0) = 0 before relying on the chosen key sequence.
"],"question":[],"answer":[],"kind":"read","id":"tee-c19-s2","search":"Calculator sequence without model guesswork The reported model name is “Casio FX ES 990”; the exact model is not confirmed. Use your calculator’s actual function labels. On common ES/ES Plus models, exp(x) is usually the shifted ln key, while the EXP/ × 10ˣ key enters scientific notation and is NOT the same as exp(x). Check the exact key labeling on your device.\nFor sigmoid at 2.2, type 1÷(1+exp(−2.2)). Keep the negative sign inside the exponential. Expected 0.900249511. For a negative gradient update, type old−rate × (negative gradient); use parentheses around the negative value. For cross-entropy, use −ln(probability), not log 10. For tanh, use the hyperbolic function or its exponential fraction if that function is difficult to find.\nRetain 6–9 digits in intermediate results or use stored values; round final answers to 4–6 decimal places unless the paper says otherwise. Work line by line on paper even if the calculator can evaluate the whole expression. Check exp(0) = 1, ln(1) = 0, sigmoid(0) = 0.5, tanh(0) = 0 before relying on the chosen key sequence."},{"title":"What to write if the exam page feels blank","blocks":["Pause briefly, put both feet down and take one slow breath. Then write only the GIVEN values and what is ASKED. Classify the question using the method map. Write its general formula before inserting numbers. One correct first line is a smaller task than “solve the whole question”.
","For CNN, draw the shape chain. For back prop, draw arrows and label the loss. For an optimizer, draw columns for step, gradient, state, new parameter. For LSTM, write old part, new part, cell, hidden. If you remain stuck after a short attempt, leave visible working space and move to a question you can start. Return after collecting those marks.
","The goal is to reduce the next action to something concrete and familiar. Practice it during mocks so it is available under time pressure.
"],"question":[],"answer":[],"kind":"read","id":"tee-c19-s3","search":"What to write if the exam page feels blank Pause briefly, put both feet down and take one slow breath. Then write only the GIVEN values and what is ASKED. Classify the question using the method map. Write its general formula before inserting numbers. One correct first line is a smaller task than “solve the whole question”.\nFor CNN, draw the shape chain. For back prop, draw arrows and label the loss. For an optimizer, draw columns for step, gradient, state, new parameter. For LSTM, write old part, new part, cell, hidden. If you remain stuck after a short attempt, leave visible working space and move to a question you can start. Return after collecting those marks.\nThe goal is to reduce the next action to something concrete and familiar. Practice it during mocks so it is available under time pressure."},{"title":"Answer lengths and partial credit habits","blocks":["For 2 marks, give a precise definition plus the key distinction/example. For 5 marks, give the definition, a labeled diagram or formula, working/principle and a relevant implication. For 10 marks, answer every subpart: architecture or setup, main equations, stepwise explanation, required numerical working, and limitations or justification where asked. Do not pad a numerical with unrelated theory.
","Always label assumptions when inputs, loss, threshold or matrix orientation are missing. Box the requested final result, include shapes or units, and retain working. An unsupported final number makes it harder to award method credit. Never say an optimizer guarantees global convergence or an architecture always outperforms another unless a specific theorem and its conditions actually apply.
"],"question":[],"answer":[],"kind":"read","id":"tee-c19-s4","search":"Answer lengths and partial credit habits For 2 marks, give a precise definition plus the key distinction/example. For 5 marks, give the definition, a labeled diagram or formula, working/principle and a relevant implication. For 10 marks, answer every subpart: architecture or setup, main equations, stepwise explanation, required numerical working, and limitations or justification where asked. Do not pad a numerical with unrelated theory.\nAlways label assumptions when inputs, loss, threshold or matrix orientation are missing. Box the requested final result, include shapes or units, and retain working. An unsupported final number makes it harder to award method credit. Never say an optimizer guarantees global convergence or an architecture always outperforms another unless a specific theorem and its conditions actually apply."},{"title":"Methods to revise before the examination","blocks":["This is a revision digest after studying the lessons. Full solutions remain beside each topic and under Solved Questions. If a step here is unclear, return to that worked example; the final day is for recovering the method, not memorizing unexplained final numbers.
","Machine learning estimates a model from data. Deep learning uses multiple trainable layers to learn successive representations. Inputs describe a sample; targets are the required answers; predictions are the model's answers. Weights and biases are learned parameters. Learning rate, batch size and architecture are chosen hyperparameters.
","Training changes parameters using training examples. Validation selects settings/stopping time. Testing evaluates the final choice on held-out data. Overfitting means strong training performance with weak generalization; underfitting means the model has not captured enough of the relationship.
","For one neuron, multiply each input by its own weight, add all products, add the bias once and apply the activation. A layer repeats this for several receiving neurons. Each weight-matrix row belongs to one receiving neuron. Transpose swaps rows and columns; it does not change signs.
","Memory cue: multiply → sum → bias → activate. In a full network, finish one complete layer before the next.
","Sigmoid maps to (0,1) and is used for binary-class probabilities. Tanh maps to (−1,1), centred on zero. Both can saturate, producing small derivatives. ReLU retains positive values and returns zero on the negative side; its negative-side gradient is zero. Leaky ReLU retains a specified small negative-side slope. Softmax normalizes a complete list of logits into mutually exclusive class probabilities.
","For squared errors, SSE sums squares, half SSE divides that sum by 2, and MSE divides it by the number of scalar predictions. Their derivatives differ. For one-hot categorical cross-entropy, only the correct-class probability contributes. For binary cross-entropy, p means probability of class 1, not automatically the true class.
","Memory cue: loss ka denominator dekho before differentiating. Sigmoid output is not its derivative.
","Write the given loss and activation first. Compute all forward sums/activations with the original parameters. Differentiate loss with respect to each output, multiply by the output activation derivative, and call the result the output delta.
","For a hidden neuron, add the outgoing weighted deltas first and then multiply by the hidden activation derivative. Each connection gradient is receiving delta × sending activation. The bias gradient is the receiving delta. Use old weights for all gradients before making the simultaneous update.
","Memory cue: path mein multiply; paths milne par add; update sab gradients ke baad. For sigmoid+BCE or softmax+CCE, the logit delta simplifies to prediction−target for the stated single-example sum convention.
","Full-batch GD uses the entire dataset for one update; SGD uses one sample; mini-batch uses a subset. An epoch is one pass through the data. With a final incomplete batch included, updates per epoch are ceiling(N/B).
","| Method | State and calculation order | Detail that changes the answer |
|---|---|---|
| GD | Current gradient → subtract ηg | Recompute at the new parameter unless gradients are explicitly supplied |
| Momentum | Retain βv → subtract ηg → add new velocity to parameter | Signed velocity and gradient-average conventions must not be mixed |
| Nesterov | Compute lookahead θ+βv → gradient there → new velocity → update actual θ | Do not add new velocity to the lookahead point |
| AdaGrad | Add g² to all-past sum G → divide gradient by √G+ε | G never decreases |
| RMSProp | Decay old squared state → add recent g² → normalize | Use the specified decay factor |
| Adam | Update signed mean m and squared mean v → correct both → normalize | Carry raw states forward; correction powers use iteration t |
Here v means signed velocity in momentum and squared-gradient state in Adam; the method defines the symbol. Memory cue: AdaGrad sab squares; RMSProp recent squares; Adam direction + squares + correction.
","L1 adds λΣ|w| and a nonzero-weight derivative λsign(w), encouraging sparsity. L2 with the half convention adds (λ/2)Σw² and derivative λw. Without that half, the derivative is 2λw. Regularization gradients add to data gradients before updating the weight.
","Inverted dropout uses keep probability q: retain a sampled activation with scale 1/q, or replace it by zero. Inference uses the full activation without that training-time rescaling. Drop probability p gives q = 1−p. Early stopping tracks validation performance and restores the best checkpoint under the stated patience rule.
","Memory cue: dropout probability aur keep probability alag; L2 mein half hai ya nahi, pehle dekho.
","Use the deep-learning cross-correlation convention unless the question explicitly requires kernel flipping. At each output position, multiply matching input-window and kernel entries, sum over spatial entries and all input channels, and add that filter's bias once. A new filter creates a new output channel.
","Apply the shape formula separately to height and width. Pooling changes spatial dimensions but preserves channels; ordinary pooling and flattening have zero learned parameters. Maximum pooling chooses the largest window value; average pooling adds its values and divides by the number of entries. A filter's spatial reuse changes computation, not its parameter count.
","For a complete architecture, make one row per layer: input shape → operation → output shape → weights → biases → total. Flatten uses the size after the final convolution/pool. Memory cue: shape count aur parameter count ko kabhi mix mat karo.
","CNNs use local connectivity and weight sharing, reducing parameters compared with unrestricted dense image mappings. Convolution learns features; activations introduce nonlinearity; pooling summarizes local responses; dense or global-pooling/output stages make the final task prediction.
","AlexNet: stacked convolutions with ReLU, pooling and dense classification; dropout in dense layers and data augmentation help control overfitting. Inception: parallel spatial scales, 1 × 1 reductions and channel concatenation. ResNet: learns a residual F(x) and adds a shortcut x; use projection when shapes differ. Concatenation adds channels; residual addition requires equal shapes and retains their count.
","Transfer learning reuses a pretrained feature extractor, replaces the task head, trains the head and optionally fine-tunes selected pretrained layers at a suitable smaller learning rate. Explain classification, detection and segmentation as different outputs, with an appropriate concrete application.
","An RNN combines the current input with the previous activated hidden state using weights shared across time. Unroll the sequence and calculate one step at a time.
","![\"h_t=\\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b)\"](\"assets/equations/9dd7210eacaa003c.svg\")
For TEE, revise every recurrent-model item below. The faculty MT2 update received on 12 October focuses MT2 on Units 3 and 4; these Unit 5 RNN topics belong to the complete TEE preparation route.
","BPTT applies the chain rule backward through the unrolled network. A shared weight receives contributions from all relevant time steps; sum them before the update. Repeated small derivative products can vanish; repeated large products can explode. Gradient clipping limits large updates; gated memory helps preserve useful long-range dependencies.
","LSTM has a cell memory C and hidden state h. Sigmoid gates control retain/write/read proportions; the candidate uses tanh. Calculate activated gates first when raw gate scores are supplied.
","![\"h_t=o_t\\odot\\tanh(C_t)\"](\"assets/equations/5a01a031c432f962.svg\")
GRU uses reset and update gates with one hidden state. In the candidate-weighting convention used here:
","![\"\\widetilde](\"assets/equations/37663e895cb447b3.svg\")
If the question instead uses z to retain the old state, follow its reversed mixing convention. Memory cue: LSTM = old memory retain + new content write, then output gate. GRU = reset for candidate, update for final mixture.
","For every numerical: full question → Given → To find → governing formula and why → separate numbered steps → final answer. Carry extra calculator precision; round at the end. Label each layer, iteration and connection so the examiner can follow the calculation.
","For a definition question, state the technical meaning and the requested distinction/example. For a longer mechanism question, include a labelled diagram or equations, explain their sequence, then give the relevant benefits, limitations or comparison. Marks printed beside a historical question are its actual marks; individual steps here are not assigned invented official marks.
","For a 180-minute TEE, a workable allocation is 10 minutes to read/select, about 30 minutes per attempted 20-mark main question, and 20 minutes to complete or review. Adjust to the paper. If stuck, write the givens and governing formula first, complete the solvable parts, then return to the unfinished calculation.
"],"question":[],"answer":[],"kind":"read","id":"tee-c19-s5","search":"Methods to revise before the examination \n### Last-day revision: read in this order\nThis is a revision digest after studying the lessons. Full solutions remain beside each topic and under Solved Questions. If a step here is unclear, return to that worked example; the final day is for recovering the method, not memorizing unexplained final numbers.\n\n### 1. Learning and the neuron\nMachine learning estimates a model from data. Deep learning uses multiple trainable layers to learn successive representations. Inputs describe a sample; targets are the required answers; predictions are the model's answers. Weights and biases are learned parameters. Learning rate, batch size and architecture are chosen hyperparameters.\n\nTraining changes parameters using training examples. Validation selects settings/stopping time. Testing evaluates the final choice on held-out data. Overfitting means strong training performance with weak generalization; underfitting means the model has not captured enough of the relationship.\n\nFor one neuron, multiply each input by its own weight, add all products, add the bias once and apply the activation. A layer repeats this for several receiving neurons. Each weight-matrix row belongs to one receiving neuron. Transpose swaps rows and columns; it does not change signs.\n\n**Memory cue:** multiply → sum → bias → activate. In a full network, finish one complete layer before the next.\n\n### 2. Activation and loss choices\nSigmoid maps to (0,1) and is used for binary-class probabilities. Tanh maps to (−1,1), centred on zero. Both can saturate, producing small derivatives. ReLU retains positive values and returns zero on the negative side; its negative-side gradient is zero. Leaky ReLU retains a specified small negative-side slope. Softmax normalizes a complete list of logits into mutually exclusive class probabilities.\n\n\n\nFor squared errors, SSE sums squares, half SSE divides that sum by 2, and MSE divides it by the number of scalar predictions. Their derivatives differ. For one-hot categorical cross-entropy, only the correct-class probability contributes. For binary cross-entropy, p means probability of class 1, not automatically the true class.\n\n\n\n\n**Memory cue:** loss ka denominator dekho before differentiating. Sigmoid output is not its derivative.\n\n### 3. Backpropagation: the repeatable method\nWrite the given loss and activation first. Compute all forward sums/activations with the original parameters. Differentiate loss with respect to each output, multiply by the output activation derivative, and call the result the output delta.\n\nFor a hidden neuron, add the outgoing weighted deltas first and then multiply by the hidden activation derivative. Each connection gradient is receiving delta × sending activation. The bias gradient is the receiving delta. Use old weights for all gradients before making the simultaneous update.\n\n\n\n**Memory cue:** path mein multiply; paths milne par add; update sab gradients ke baad. For sigmoid+BCE or softmax+CCE, the logit delta simplifies to prediction−target for the stated single-example sum convention.\n\n### 4. Optimizers: identify the stored state\nFull-batch GD uses the entire dataset for one update; SGD uses one sample; mini-batch uses a subset. An epoch is one pass through the data. With a final incomplete batch included, updates per epoch are ceiling(N/B).\n\n| Method | State and calculation order | Detail that changes the answer |\n| GD | Current gradient → subtract ηg | Recompute at the new pa"}]},{"n":20,"title":"Course vocabulary in plain words","sections":[{"title":"Learning and the numbers we adjust","blocks":["| Term | Meaning | Tiny example |
|---|---|---|
| Input / feature | A number given to the model | Image pixel intensity |
| Sample | One example processed by the model | One labeled image |
| Dataset | A collection of examples | 1,000 training images |
| Target / label | Correct answer supplied for training | Class 2 is correct |
| Prediction | Model's current answer | Class probabilities |
| Weight | Learned multiplier on a connection | 0.4 multiplying an input |
| Bias | Learned value added to a weighted sum | z = wx + b |
| Parameter | A value learned during training | Weight or bias |
| Hyperparameter | A setting chosen for training | Learning rate or batch size |
| Activation | Transformation of a neuron's raw score | ReLU or sigmoid |
| Loss | Numerical measure of prediction error | Mean squared error |
| Gradient | Local rate of loss change for parameters | Positive g means locally uphill |
| Optimizer | Rule using gradients to update parameters | Momentum or Adam |
| Epoch | One pass through the training dataset | All 100 samples processed |
| Iteration | One parameter update in these exercises | One mini-batch update |
| Inference | Applying the trained model | Predicting a new image's class |
| Term | Meaning | What to keep separate |
|---|---|---|
| Tensor | Array of numbers with one or more axes | Its shape and its values |
| Channel | One feature map within an image-like tensor | Channels versus spatial size |
| Kernel / filter | Learned weights sliding over a map | Kernel size versus filter count |
| Stride | Step between adjacent window positions | Stride versus kernel width |
| Padding | Extra border cells around an input | Per-side P versus total 2P |
| Feature map | Responses from a filter across positions | Activations versus parameters |
| Pooling | Window-based reduction of feature maps | Maximum versus average |
| Flatten | Rearrange H × W × C entries as a vector | Rearrangement has no parameters |
| Logit | Raw score before an output activation | Score versus probability |
| Hidden state | Recurrent features carried through time | Previous state versus current input |
| Shared weights | One learned rule reused at many positions/times | Reuse versus extra parameters |
“Calculate” asks for arithmetic and a result. Write the formula and substitutions before the final number.
","“Explain” asks how and why a method works. Give its definition, describe the steps, and connect each step to its purpose.
","“Compare” asks for matched differences. Discuss the same attributes for both methods, such as stored state, update rule, benefit and limitation.
","“Draw” asks for a labeled structure. Label inputs, layers or time steps, arrows, activations and the output. A diagram and explanation should describe the same computation.
","“Design” asks for a justified arrangement. State the task, input and output, chosen layers, activation/loss pair and training procedure. State any assumption needed because the question leaves a value unspecified.
"],"question":[],"answer":[],"kind":"read","id":"tee-c20-s2","search":"Reading a question precisely “Calculate” asks for arithmetic and a result. Write the formula and substitutions before the final number.\n“Explain” asks how and why a method works. Give its definition, describe the steps, and connect each step to its purpose.\n“Compare” asks for matched differences. Discuss the same attributes for both methods, such as stored state, update rule, benefit and limitation.\n“Draw” asks for a labeled structure. Label inputs, layers or time steps, arrows, activations and the output. A diagram and explanation should describe the same computation.\n“Design” asks for a justified arrangement. State the task, input and output, chosen layers, activation/loss pair and training procedure. State any assumption needed because the question leaves a value unspecified."}]},{"n":21,"title":"February 2025 re-exam · complete solved paper","sourceKind":"actual-paper","sections":[{"id":"tee-c21-s0","title":"R25 Q1a · ML versus DL and architecture choice","kind":"practice","question":["Question. Compare ML and DL. Choose architectures for five-class diabetic-retinopathy retinal images, alphabet-image recognition, and weather prediction from N days of history. (5 marks)
"],"answer":["Definition. Deep learning is a subfield of machine learning that learns representations through multiple trainable processing layers. Each layer transforms the preceding representation, allowing the complete model to learn a composition of functions from data.
Artificial intelligence studies systems that perform tasks such as perception, reasoning and decision-making. Machine learning develops models whose parameters or behavior are estimated from data. Deep learning uses layered neural models within machine learning.
Representation learning. Hand-designing every useful image or speech feature is difficult. A deep model can learn intermediate representations together with the final prediction objective. In a CNN, early layers may respond to local edges, while later layers combine information over larger regions. These patterns depend on training and are not guaranteed to have a particular human meaning.
Nonlinear relationships. Real tasks often depend on combinations of input variables. Composing affine transformations with nonlinear activations can represent interactions that a single linear decision boundary cannot. Depth supplies repeated transformations; increasing only the number of neurons in one layer changes width instead.
End-to-end optimization. The final task loss supplies gradients to the preceding trainable layers. The feature extractor and prediction head can therefore be adjusted for the same task. Preprocessing and human decisions about architecture, data and evaluation still remain necessary.
| Factor | How it contributes |
|---|---|
| Larger datasets | Provide more examples of the variations a complex model must learn |
| GPUs and parallel computation | Accelerate the large matrix and convolution operations used in training |
| Improved training methods | Backpropagation, suitable activations, initialization and optimizers make fitting deep models more practical |
| Pretrained models | Reuse learned features so a new task may require less labeled data and training effort |
An application answer should identify the input, the prediction and the role of the learned representation.
| Domain | Example input | Required output | Role of deep learning |
|---|---|---|---|
| Computer vision | Photograph or video frame | Image class, object locations or pixel labels | Learn visual features with a CNN and an appropriate task head |
| Language | Review, sentence or conversation | Sentiment label, translated words or a response | Learn representations of text and its context |
| Speech | Audio sequence | Words or a speaker label | Model patterns in acoustic features over time |
| Healthcare | Medical image or patient measurements | A task-specific class or numerical estimate | Learn associations from labeled examples; assess results on suitable held-out data |
| Autonomous systems and robotics | Camera and other sensor measurements | Perception estimates used by a larger control system | Recognize objects and relevant environmental features |
| Financial services | Transaction features or a time-ordered record | Fraud score, risk estimate or forecast | Model patterns and interactions among observations |
| Aspect | Traditional machine learning often uses | Deep learning often uses |
|---|---|---|
| Representation | Selected or engineered input features | Features learned jointly across layers |
| Useful architectures | Linear models, decision trees, SVMs | MLPs, CNNs, RNNs and gated recurrent models |
| Training from scratch | Can be effective with modest structured datasets | Large models often need substantial data and computation |
| Interpretation | Some models have directly inspectable rules or coefficients | Learned internal representations can be harder to interpret |
Feature learning, flexible nonlinear modeling and reuse of pretrained representations are major advantages. Limitations include computation and memory costs, overfitting with insufficient representative data, difficulty interpreting predictions, and reduced reliability when deployment data differs from training data. High training accuracy does not establish generalization.
These are tendencies, not absolute divisions. Traditional ML can also learn representations; deep models can process tabular data. A pretrained deep model may be useful even with a small target dataset. Select the model according to the task, available data, resources and validation results.
Question. Explain the motivation for deep learning and describe applications in at least three domains.
Answer. Deep learning is motivated by the need to learn useful hierarchical representations and nonlinear relationships directly from complex data. Multiple trainable layers compose feature transformations, and the task loss can train those transformations jointly with the predictor. Larger datasets, GPU computation and improved optimization methods have made this practical. In computer vision, CNNs learn visual features for image classification and object localization. In language processing, neural sequence models use word representations and context for sentiment analysis or translation. In speech processing, neural models map acoustic patterns to words. The benefits must be balanced against data requirements, computation, overfitting and interpretability limitations.
| Task | Suitable architecture | Why / output |
|---|---|---|
| Retinal images,5classes | CNN | Spatial image features;5-way softmax for exclusive classes |
| Alphabet images | CNN | Edges/shapes compose into characters; class-count softmax |
| Weather from a temporal history | RNN/LSTM or GRU | Ordered history; regression output for numeric weather quantities, suitable classification head for categorical weather |
Architecture choice is justified by data structure. A fully connected model can technically take flattened images, but ignores spatial locality; sequential models preserve temporal order.
"],"search":"ML versus DL and architecture choice Compare ML and DL. Choose architectures for five-class diabetic-retinopathy retinal images, alphabet-image recognition, and weather prediction from N days of history.","units":[1,4,5],"marks":5,"sourcePages":[1],"sourceKind":"actual-paper"},{"id":"tee-c21-s1","title":"R25 Q1b · Training and validation errors","kind":"practice","question":["Question. Choose best and worst models from training/validation errors: M1=0.9/1.2; M2=0.3/0.6; M3=1.5/0.5; M4=1.2/1.1. Explain the reasoning and remedies. (5 marks)
"],"answer":["| Model | Train | Validation |
|---|---|---|
| 1 | 0.9 | 1.2 |
| 2 | 0.3 | 0.6 |
| 3 | 1.5 | 0.5 |
| 4 | 1.2 | 1.1 |
For the same validation set and same loss definition, M3 is best (0.5) and M1 is worst (1.2). Choose by validation performance, not the smallest training loss. M2 has the smallest training error but a larger validation error than M3.
M3’s validation loss below training loss is unusual but not logically impossible: training-only dropout/augmentation, different regularization inclusion or an easier validation sample can cause this. First verify comparable evaluation conditions and leakage-free splits. Error values alone do not prove overfitting or underfitting without baselines and learning curves.
For poor generalization, inspect data quality/splits and learning curves; if overfitting is established, use regularization, dropout, more suitable data or early stopping; if underfitting is established, improve model capacity/features or training. Do not prescribe a larger network solely from these four numbers.
"],"search":"Training and validation errors Choose best and worst models from training/validation errors: M1=0.9/1.2; M2=0.3/0.6; M3=1.5/0.5; M4=1.2/1.1. Explain the reasoning and remedies.","units":[3],"marks":5,"sourcePages":[1],"sourceKind":"actual-paper"},{"id":"tee-c21-s2","title":"R25 Q1c · CNN hierarchy and stride-two convolution","kind":"practice","question":["Question. Explain why CNNs use a hierarchy of convolutional layers. Apply the printed3×3kernel to the5×5input with stride2. (5 marks)
"],"answer":["Early layers learn local edge/texture patterns; deeper layers combine them into parts and class-relevant configurations, with a growing receptive field. Nonlinearities make stacked layers more expressive than one linear convolution.
Input:
![\"\\begin{bmatrix}2&3&8&2&3\\\\2&2&4&4&6\\\\1&2&1&2&9\\\\7&5&3&3&2\\\\2&4&3&5&4\\end{bmatrix}\"](\"assets/equations/recovered-26a644e9131b783f.svg\")
Kernel:
![\"\\begin{bmatrix}1&0&1\\\\2&4&0\\\\6&4&2\\end{bmatrix}\"](\"assets/equations/recovered-dfae566e77e4351c.svg\")
Stride=2, zero padding=0, no bias supplied.
Valid cross-correlation output (CNN convention, no kernel flip).
![\"O=\\left\\lfloor\\frac{N+2P-F}{S}\\right\\rfloor+1\"](\"assets/equations/recovered-cbf0b1f38ebaaccb.svg\")
Multiply corresponding cells and sum. Here O=⌊(5−3)/2⌋+1=2 in each direction.
![\"\\begin{bmatrix}2&3&8\\\\2&2&4\\\\1&2&1\\end{bmatrix}\"](\"assets/equations/recovered-217f7d6e6c6e566c.svg\")
2×1 + 3×0 + 8×1 + 2×2 + 2×4 + 4×0 + 1×6 + 2×4 + 1×2 = 2 + 0 + 8 + 4 + 8 + 0 + 6 + 8 + 2 = 38
![\"\\begin{bmatrix}8&2&3\\\\4&4&6\\\\1&2&9\\end{bmatrix}\"](\"assets/equations/recovered-fd5db35a9f88c331.svg\")
8×1 + 2×0 + 3×1 + 4×2 + 4×4 + 6×0 + 1×6 + 2×4 + 9×2 = 8 + 0 + 3 + 8 + 16 + 0 + 6 + 8 + 18 = 67
![\"\\begin{bmatrix}1&2&1\\\\7&5&3\\\\2&4&3\\end{bmatrix}\"](\"assets/equations/recovered-c03180114d7dc748.svg\")
1×1 + 2×0 + 1×1 + 7×2 + 5×4 + 3×0 + 2×6 + 4×4 + 3×2 = 1 + 0 + 1 + 14 + 20 + 0 + 12 + 16 + 6 = 70
![\"\\begin{bmatrix}1&2&9\\\\3&3&2\\\\3&5&4\\end{bmatrix}\"](\"assets/equations/recovered-99c4e3c248d2285d.svg\")
1×1 + 2×0 + 9×1 + 3×2 + 3×4 + 2×0 + 3×6 + 5×4 + 4×2 = 1 + 0 + 9 + 6 + 12 + 0 + 18 + 20 + 8 = 74
![\"\\begin{bmatrix}38&67\\\\70&74\\end{bmatrix}\"](\"assets/equations/recovered-58e9e7ec888c133f.svg\")
Source / convention note. Padding is not specified; valid/no-padding is declared. CNN cross-correlation convention is used without kernel reversal.
"],"search":"CNN hierarchy and stride-two convolution Explain why CNNs use a hierarchy of convolutional layers. Apply the printed3×3kernel to the5×5input with stride2.","units":[4],"marks":5,"sourcePages":[1],"sourceKind":"actual-paper"},{"id":"tee-c21-s3","title":"R25 Q1d · BPTT and recurrent strengths","kind":"practice","question":["Question. Compare BPTT with feedforward backpropagation and state the strengths of RNNs. (5 marks)
"],"answer":["Definition. Backpropagation through time (BPTT) differentiates the loss of an RNN by applying backpropagation to its temporally unrolled computational graph and summing contributions to shared parameters.
Unroll the recurrent network for the sequence length. Run the forward calculation in time order. Calculate the specified loss. Differentiate backwards through the unrolled graph. Add contributions to each shared weight from every time it was used. Update each shared weight once for the selected sequence or batch. This is backpropagation through time, or BPTT.
For tanh, let δₜ mean the gradient with respect to the hidden preactivation. The following form allows losses at several time steps; a missing output loss contributes zero.
The symbol ⊙ means pairwise multiplication. The transpose T turns a column into a row. δnext carries the effect of future time steps. The summation is essential because these are shared parameters.
Question. In a scalar tanh RNN, h_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.
Activated h_t = 0.5; q_t = 0.2; δ_(t+1) = 0.3; recurrent weight u = 0.4.
Current preactivation delta δ_t.
Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.
q_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.
Yaad rakhne ka cue: Future aur current dono add; tanh slope uske baad.
Step 1: calculate the future path contribution.
Step 2: add the current output contribution.
Step 3: calculate the current tanh slope.
Step 4: multiply by that slope.
Final answer: δ_t = 0.24.
Question. In a scalar tanh RNN, h_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.
Activated h_t = -0.2; q_t = 0.1; δ_(t+1) = 0; recurrent weight u = 0.5.
Current preactivation delta δ_t.
Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.
q_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.
Yaad rakhne ka cue: Future aur current dono add; tanh slope uske baad.
Step 1: calculate the future path contribution.
Step 2: add the current output contribution.
Step 3: calculate the current tanh slope.
Step 4: multiply by that slope.
Final answer: δ_t = 0.096.
| Feedforward backprop | BPTT |
|---|---|
| Propagate through layers in one acyclic graph | Unroll recurrence through time, then propagate through the resulting acyclic graph |
| Layer-specific parameters usually distinct | Recurrent parameters shared across time; sum gradient contributions over all uses |
| No persistent temporal state by default | Hidden state connects earlier and later steps |
| Depth-related gradient issues | Long time paths add vanishing/exploding difficulty; truncation limits path length |
RNN strengths: variable-length sequences, shared parameters and order-dependent context; applications include language, speech and time series. Memory is not perfect and vanilla RNNs struggle with long dependencies.
"],"search":"BPTT and recurrent strengths Compare BPTT with feedforward backpropagation and state the strengths of RNNs.","units":[5],"marks":5,"sourcePages":[1,2],"sourceKind":"actual-paper"},{"id":"tee-c21-s4","title":"R25 Q2a · Activation, ReLU and numeric outputs","kind":"practice","question":["Question. Explain why activation functions are needed and sketch ReLU. Is sigmoid appropriate for an unscaled house-price output? Compute softmax([1.2,3.4,0.9]) and sigmoid(2.5). (10 marks)
"],"answer":["An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary.
For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second:
This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent.
| Activation | Output range | Useful property | Main limitation |
|---|---|---|---|
| Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small |
| Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients |
| ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient |
Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score.
Exam conclusion: select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction.
![\"\\mathrm{ReLU}(z)=\\max(0,z)\"](\"assets/equations/recovered-1c6ec1b1ec321f01.svg\")
Unscaled house prices are not restricted to(0,1), so a sigmoid cannot directly represent them. Use a linear regression output (or a justified positive-valued transform). A sigmoid could be used only with a deliberately bounded/normalized target and inverse transformation.
Logits=[1.2,3.4,0.9]; separate sigmoid input=2.5.
Normalized class probabilities and sigmoid value.
![\"p_i=\\frac{e^{z_i-m}}{\\sum_je^{z_j-m}},\\quad](\"assets/equations/recovered-051b60342e33ce99.svg\")
[−2.2,0,−2.5]. This common shift leaves softmax unchanged.
[0.11080315836233387, 1.0, 0.0820849986238988]; sum=1.192888156986.
| Logit | Exponential / sum | Probability |
|---|---|---|
| 1.2 | 0.110803158362/1.192888156986 | 0.092886460 |
| 3.4 | 1.000000000000/1.192888156986 | 0.838301558 |
| 0.9 | 0.082084998624/1.192888156986 | 0.068811982 |
e^(−2.5)=0.082084998624;1+e^(−2.5)=1.082084998624; reciprocal=0.924141820.
Softmax=[0.09288646, 0.838301558, 0.068811982], sum=1.000000000; sigmoid(2.5)=0.924141820.
"],"search":"Activation, ReLU and numeric outputs Explain why activation functions are needed and sketch ReLU. Is sigmoid appropriate for an unscaled house-price output? Compute softmax([1.2,3.4,0.9]) and sigmoid(2.5).","units":[2],"marks":10,"sourcePages":[2],"sourceKind":"actual-paper"},{"id":"tee-c21-s5","title":"R25 Q2b · Vanishing/exploding gradients and GRU","kind":"practice","question":["Question. Explain vanishing/exploding gradients in RNNs and the gates of a GRU. (10 marks)
"],"answer":["Definition. Vanishing and exploding gradients describe the attenuation or amplification of backward derivatives through repeated Jacobian products. In recurrent networks, these products extend across time and affect long-range credit assignment.
In BPTT, the gradient repeatedly multiplies recurrent weights and activation derivatives. Products of many small factors can approach zero; early information then receives very little learning signal. Large repeated factors can make the gradient explode. These are about training signals, not an instruction that every hidden-state value must become zero or infinity.
Numerical intuition: 0.5¹⁰ = 0.0009765625, whereas 2¹⁰ = 1,024. Sigmoid's derivative is at most 0.25 and becomes even smaller in saturation. Tanh also saturates for large magnitude inputs.
Useful remedies include LSTM or GRU gates and cell paths, suitable initialization and learning rate, and gradient clipping for explosion. Clipping does not by itself solve vanishing gradients. LSTM makes long-range learning easier; it does not guarantee perfect unlimited memory.
For a vanilla tanh RNN, each hidden-state transition contributes a Jacobian. The derivative through several transitions contains a product of these Jacobians:
The product is ordered from the latest transition toward the earliest. Small activation derivatives and recurrent transformations can attenuate the signal; repeated amplification can make it grow. A one-dimensional product illustrates the issue, but a vector RNN depends on the full matrix products, not one weight alone.
Norm clipping rescales an oversized gradient while retaining its direction. It limits the update size caused by an exploding gradient.
If g is the zero vector, leave it unchanged. The threshold τ is positive. This is global norm clipping for the vector being considered; elementwise clipping is a different operation.
Question. A gradient is g = [6, 8]ᵀ. Clip its Euclidean norm to at most τ = 5, then update w = [1, 1]ᵀ with learning rate η = 0.1.
Gradient g = [6,8]ᵀ; norm threshold τ = 5.
Old weights w = [1,1]ᵀ; η = 0.1.
Find the clipped gradient and updated weights.
Gradient norm clipping. If the vector norm exceeds the allowed threshold, multiply every component by the same scale. This changes magnitude while preserving direction.
g = gradient vector; τ (tau) = maximum norm; ||g||₂ = square root of the sum of squared components; η = learning rate.
Yaad rakhne ka cue: Bada gradient? Sab components ko ek hi factor se chhota karo.
Step 1: calculate the original norm.
Step 2: find the scale. Since 10 > 5, scale the vector by 5/10 = 0.5.
Step 3: verify the clipped norm.
Step 4: update each parameter with the clipped gradient.
Final answer: clipped gradient [3, 4]ᵀ and new weights [0.7, 0.6]ᵀ. Clipping does not increase a vanishing gradient and does not replace the need for an appropriate recurrent architecture.
Question. Clip gradient g = [3, 4] to maximum L2 norm 10. Use global norm clipping.
g = [3, 4]; threshold τ = 10.
The norm, scaling factor and clipped gradient.
Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold.
τ is the maximum permitted norm, not a per-entry bound.
Yaad rakhne ka cue: Norm bada ho tab sab entries ko same ratio se chhota karo.
Step 1: calculate the squared norm.
Step 2: take the square root.
Step 3: choose the shared scale.
Step 4: scale every entry.
Final answer: Clipped gradient = [3, 4].
Question. Clip gradient g = [0, 12] to maximum L2 norm 3. Use global norm clipping.
g = [0, 12]; threshold τ = 3.
The norm, scaling factor and clipped gradient.
Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold.
τ is the maximum permitted norm, not a per-entry bound.
Yaad rakhne ka cue: Norm bada ho tab sab entries ko same ratio se chhota karo.
Step 1: calculate the squared norm.
Step 2: take the square root.
Step 3: choose the shared scale.
Step 4: scale every entry.
Final answer: Clipped gradient = [0.0, 3.0].
Definition. A gated recurrent unit (GRU) is a recurrent architecture that uses reset and update gates to control its hidden-state transition. Unlike a standard LSTM, it does not carry a separate cell state alongside the hidden state.
The reset gate r controls how the previous hidden state enters candidate formation. The update gate z controls the mixture of the old state and the candidate. Both are calculated from the current input and previous hidden state:
The candidate equation used in these notes applies reset before the recurrent candidate transformation:
These notes define z as the fraction of new candidate admitted:
Thus z near 0 favors retention of the old hidden state, while z near 1 favors the candidate. The reset gate affects the candidate calculation; it does not directly multiply the retained old-state term in the final mixture.
Question. A scalar GRU has hₜ₋₁ = 0.4, reset gate rₜ = 0.5, update gate zₜ = 0.25, xₜ = 1, Wₕ = 0.2, Uₕ = 0.5 and bₕ = 0. Calculate the candidate and new hidden state using hₜ = (1−zₜ)hₜ₋₁ + zₜh̃ₜ.
Old hidden state = 0.4; r = 0.5; z = 0.25.
x = 1; W = 0.2; U = 0.5; b = 0.
Here z weights the candidate in the final mixture.
Find candidate and new hidden state.
GRU reset and state mixture. The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate.
r = reset gate; z = update gate; h_old = previous state; h_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state.
Yaad rakhne ka cue: Reset candidate ko control; update gate old-new mixture ko control.
Step 1: apply the reset gate to the previous hidden state.
The gates are already supplied as outputs, so no further sigmoid calculation is needed.
Step 2: calculate the candidate preactivation.
Step 3: apply tanh to obtain the candidate.
Step 4: calculate the two mixture contributions.
Step 5: add the contributions.
Final answer: candidate ≈ 0.291313 and new hidden state ≈ 0.372828. The new scalar state lies between the old state 0.4 and the candidate 0.291313, as expected for this convex mixture.
Some texts define z as the fraction of the old state retained:
Under that explicitly different convention, the same numerical value z = 0.25 gives:
This is not the answer to the worked question above, which states the first convention. To represent an identical mixture while switching conventions, the gate values must be complemented as well. Always identify the final mixture equation before substituting values.
A basic GRU has three affine transformations: reset, update and candidate. A standard LSTM has four: forget, input, output and candidate. With equal input and hidden dimensions and the same bias convention, GRU therefore has fewer recurrent parameters. This is a parameter-count comparison, not a guarantee of higher accuracy or faster execution for every task.
Question. Use h_new = z h_old+(1−z) candidate. Given h_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.
h_old = 0.4; z = 0; r = 0.5; x = 1.
W = 0.2; U = 0.5; b = 0; z is the retain-old gate.
Candidate and final hidden state.
The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.
The question explicitly makes z multiply the old state. r and z do different jobs.
Yaad rakhne ka cue: Reset sirf candidate mein; update mix mein original old state.
Step 1: reset the old state for candidate formation.
Step 2: calculate the candidate raw sum.
Step 3: apply tanh.
Step 4: retain the old-state fraction.
Step 5: calculate the candidate fraction.
Step 6: add the two fractions.
Final answer: Candidate ≈ 0.291312612; new state ≈ 0.291312612.
Question. Use h_new = z h_old+(1−z) candidate. Given h_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.
h_old = -0.2; z = 0.8; r = 0.5; x = 1.
W = 0.2; U = 0.5; b = 0; z is the retain-old gate.
Candidate and final hidden state.
The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.
The question explicitly makes z multiply the old state. r and z do different jobs.
Yaad rakhne ka cue: Reset sirf candidate mein; update mix mein original old state.
Step 1: reset the old state for candidate formation.
Step 2: calculate the candidate raw sum.
Step 3: apply tanh.
Step 4: retain the old-state fraction.
Step 5: calculate the candidate fraction.
Step 6: add the two fractions.
Final answer: Candidate ≈ 0.148885034; new state ≈ -0.130222993.
"],"search":"Vanishing/exploding gradients and GRU Explain vanishing/exploding gradients in RNNs and the gates of a GRU.","units":[5],"marks":10,"sourcePages":[2],"sourceKind":"actual-paper"},{"id":"tee-c21-s6","title":"R25 Q3a · 4–3–2 forward pass and dropout","kind":"practice","question":["Question. For the printed network, inputs=[0.1,0.2,0.3,0.4], targets=[0.01,0.99], w1…w18=[0.10,0.15,…,0.95] and both layer biases=0.60. Find total forward error, then drop hidden neuron H12 and compare errors. (10 marks)
"],"answer":["Input→hidden rows (destinations H11,H12,H13):
![\"\\begin{bmatrix}0.1&0.25&0.4&0.55\\\\0.15&0.3&0.45&0.6\\\\0.2&0.35&0.5&0.65\\end{bmatrix}\"](\"assets/equations/recovered-4cd053daebb13329.svg\")
Hidden→output rows (destinations O1,O2):
![\"\\begin{bmatrix}0.7&0.8&0.9\\\\0.75&0.85&0.95\\end{bmatrix}\"](\"assets/equations/recovered-327612e026ce8f89.svg\")
Original and dropped forward outputs and loss.
The paper omits activation and error convention. Declare sigmoid at both layers and half-SSE. For the given single dropped unit use an ordinary binary mask, no additional inverted-dropout scaling because no keep probability is supplied.
![\"h_j=\\sigma(\\sum_iW_{ji}x_i+0.6),\\quad](\"assets/equations/recovered-356678101dc6b13b.svg\")
| Neuron | Products + bias | Net | Sigmoid |
|---|---|---|---|
| 1 | 0.1×0.1 + 0.2×0.25 + 0.3×0.4 + 0.4×0.55 +0.6 | 1.000000000 | 0.731058578630 |
| 2 | 0.1×0.15 + 0.2×0.3 + 0.3×0.45 + 0.4×0.6 +0.6 | 1.050000000 | 0.740774899182 |
| 3 | 0.1×0.2 + 0.2×0.35 + 0.3×0.5 + 0.4×0.65 +0.6 | 1.100000000 | 0.750260105595 |
| Output | Full substitution | Net | Output |
|---|---|---|---|
| 1 | 0.731058578630×0.7 + 0.740774899182×0.8 + 0.750260105595×0.9 +0.6 | 2.379595019422 | 0.915258028946 |
| 2 | 0.731058578630×0.75 + 0.740774899182×0.85 + 0.750260105595×0.95 +0.6 | 2.490699698593 | 0.923487256983 |
| Output | Prediction−target | Square | Half square |
|---|---|---|---|
| 1 | 0.9152580289459775−0.01=0.905258028946 | 0.819492098971 | 0.409746049486 |
| 2 | 0.9234872569833742−0.99=-0.066512743017 | 0.004423944984 | 0.002211972492 |
Total=0.411958021977.
Masked hidden activations=[0.7310585786300049, 0, 0.7502601055951177]. Other hidden activations and all weights are unchanged.
| Output | Net with H12 term zero | Sigmoid |
|---|---|---|
| 1 | 0.731058578630×0.7+0×0.8+0.750260105595×0.9+0.6=1.786975100077 | 0.856556014402 |
| 2 | 0.731058578630×0.75+0×0.85+0.750260105595×0.95+0.6=1.861041034288 | 0.865418242930 |
Half squares: 0.5×(0.856556014402−0.01)² + 0.5×(0.865418242930−0.99)² = 0.366088849857. Difference dropped−original=-0.045869172120.
Original E=0.411958021977; dropped E=0.366088849857. A lower single-example masked loss is not evidence that dropout always lowers training or test loss.
Source / convention note. Activation/loss/dropout scaling missing from original; result is conditional on explicit standard classroom assumptions.
"],"search":"4–3–2 forward pass and dropout For the printed network, inputs=[0.1,0.2,0.3,0.4], targets=[0.01,0.99], w1…w18=[0.10,0.15,…,0.95] and both layer biases=0.60. Find total forward error, then drop hidden neuron H12 and compare errors.","units":[2,3],"marks":10,"sourcePages":[3],"sourceKind":"actual-paper"},{"id":"tee-c21-s7","title":"R25 Q3b · Four CNN design explanations","kind":"practice","question":["Question. Explain(i)parameter sharing and sparse connections,(ii)5×5versus3×3kernel trade-offs,(iii)ReLU’s effect on learning,(iv)whether CNNs can work without pooling. (10 marks)
"],"answer":["| Part | Explanation |
|---|---|
| i | Local receptive fields create sparse connectivity. The same filter weights are reused across spatial positions, reducing parameters and recognizing a feature in different positions. This gives translation equivariance under appropriate boundary/stride conditions, not automatic complete invariance. |
| ii | At fixed channel counts,5×5uses25weights per channel-pair,3×3uses9. Two stride1 3×3layers yield an effective5×5receptive field with an extra nonlinearity and18C²weights when all channel widths equalC, versus25C². Intermediate channels, biases and compute/memory can change the comparison. |
| iii | ReLU passes positive values/gradient1 and zeros negative values/gradient0. It introduces nonlinearity and avoids positive-side sigmoid saturation, but can produce inactive/dead units. It does not eliminate all gradient problems. |
| iv | Yes. Strided convolutions can downsample; global pooling or other aggregation may form the head. Removing pooling without another reduction can increase feature-map sizes/compute. Pooling is a design choice, not a mathematical necessity. |
Question. For input64×64×3: conv3×3,32filters,P1,S1; maxpool2×2,S2; conv3×3,64,P1,S1; pool2,S2; conv3×3,128,P1,S1; pool2,S2; FC256; softmax43. Find each shape and all weights/biases. (10 marks)
"],"answer":["Input=[64, 64, 3]
| Layer | Kernel | Stride | Padding | Filters / units |
|---|---|---|---|---|
| Conv | 3 | 1 | 1 | 32 |
| Pool | 2 | 2 | 0 | 0 |
| Conv | 3 | 1 | 1 | 64 |
| Pool | 2 | 2 | 0 | 0 |
| Conv | 3 | 1 | 1 | 128 |
| Pool | 2 | 2 | 0 | 0 |
| Dense | 0 | 0 | 0 | 256 |
| Dense | 0 | 0 | 0 | 43 |
Each output shape and all trainable weights plus biases.
![\"P_{conv}=(F_hF_wC_{in}+1)C_{out},\\quad](\"assets/equations/recovered-69756abf2018c02c.svg\")
Each filter/unit has one bias. Pooling has zero learned parameters. Flatten preserves the number of values, then a dense layer connects every input value to every output unit.
| Layer | Input | Full substitution and reason | Output | Parameters |
|---|---|---|---|---|
| Conv | 64×64×3 | ⌊(64+2×1−3)/1⌋+1=64; parameters=(3×3×3+1)×32=896 | 64×64×32 | 896 |
| Pool | 64×64×32 | ⌊(64−2)/2⌋+1=32; channels stay 32; no trainable weights | 32×32×32 | 0 |
| Conv | 32×32×32 | ⌊(32+2×1−3)/1⌋+1=32; parameters=(3×3×32+1)×64=18496 | 32×32×64 | 18496 |
| Pool | 32×32×64 | ⌊(32−2)/2⌋+1=16; channels stay 64; no trainable weights | 16×16×64 | 0 |
| Conv | 16×16×64 | ⌊(16+2×1−3)/1⌋+1=16; parameters=(3×3×64+1)×128=73856 | 16×16×128 | 73856 |
| Pool | 16×16×128 | ⌊(16−2)/2⌋+1=8; channels stay 128; no trainable weights | 8×8×128 | 0 |
| Dense | 8×8×128 | Flatten/input length=8×8×128=8192; parameters=(8192+1)×256=2097408 | 256 | 2097408 |
| Dense | 1×1×256 | Flatten/input length=1×1×256=256; parameters=(256+1)×43=11051 | 43 | 11051 |
896 + 18496 + 73856 + 2097408 + 11051 = 2201707
Total parameters=2201707. Shape and parameter count are separate results.
"],"search":"64×64 CNN shape and parameter table For input64×64×3: conv3×3,32filters,P1,S1; maxpool2×2,S2; conv3×3,64,P1,S1; pool2,S2; conv3×3,128,P1,S1; pool2,S2; FC256; softmax43. Find each shape and all weights/biases.","units":[4],"marks":10,"sourcePages":[4],"sourceKind":"actual-paper"},{"id":"tee-c21-s9","title":"R25 Q4b · Bipolar AND perceptron","kind":"practice","question":["Question. Implement AND with bipolar inputs/targets and learning rate1. (10 marks)
"],"answer":["Truth table: (−1,−1)→−1;(−1,+1)→−1;(+1,−1)→−1;(+1,+1)→+1. η=1.
A trained separating perceptron.
Initial weights/bias and zero-threshold tie rule are unspecified. Choose w=(0,0),b=0; output+1ifnet≥0,else−1. Use error update w←w+η(t−y)x,b←b+η(t−y). A sign-only misclassification update differs by a factor2 but can also learn the gate.
![\"z=w^Tx+b,\\quad](\"assets/equations/recovered-31e9ee08177b99d9.svg\")
| Epoch | x | Target | Old parameters | Net | Output | t−y | New parameters |
|---|---|---|---|---|---|---|---|
| 1 | [-1, -1] | -1 | [0, 0]; b=0 | 0 | 1 | -2 | [2, 2]; b=-2 |
| 1 | [-1, 1] | -1 | [2, 2]; b=-2 | -2 | -1 | 0 | [2, 2]; b=-2 |
| 1 | [1, -1] | -1 | [2, 2]; b=-2 | -2 | -1 | 0 | [2, 2]; b=-2 |
| 1 | [1, 1] | 1 | [2, 2]; b=-2 | 2 | 1 | 0 | [2, 2]; b=-2 |
| 2 | [-1, -1] | -1 | [2, 2]; b=-2 | -6 | -1 | 0 | [2, 2]; b=-2 |
| 2 | [-1, 1] | -1 | [2, 2]; b=-2 | -2 | -1 | 0 | [2, 2]; b=-2 |
| 2 | [1, -1] | -1 | [2, 2]; b=-2 | -2 | -1 | 0 | [2, 2]; b=-2 |
| 2 | [1, 1] | 1 | [2, 2]; b=-2 | 2 | 1 | 0 | [2, 2]; b=-2 |
| x | Net | Output | Target |
|---|---|---|---|
| [-1, -1] | -6 | -1 | -1 |
| [-1, 1] | -2 | -1 | -1 |
| [1, -1] | -2 | -1 | -1 |
| [1, 1] | 2 | 1 | 1 |
One valid learned solution: w=[2, 2], b=-2. Last full epoch makes no changes; all four rows are correct.
Source / convention note. Initialization, order and threshold convention explicitly supplied as assumptions; multiple equivalent separating weights exist.
"],"search":"Bipolar AND perceptron Implement AND with bipolar inputs/targets and learning rate1.","units":[2],"marks":10,"sourcePages":[4],"sourceKind":"actual-paper"},{"id":"tee-c21-s10","title":"R25 Q5a · ResNet shortcut and Inception1×1","kind":"practice","question":["Question. Describe skip connections and their benefit in ResNet. Explain a1×1convolution’s role in Inception with an example. (10 marks)
"],"answer":["Definition. A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x.
If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch.
For the identity-shortcut formulation, the derivative includes a direct identity contribution.
This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients.
If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride.
An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition.
Question. Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum.
x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3].
Find the three components of x+F(x).
Residual addition. Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested.
x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition.
Yaad rakhne ka cue: Same position ke numbers add; shape same rehni chahiye.
Step 1: add the first matching pair.
Step 2: add the second matching pair.
Step 3: add the third matching pair.
Step 4: assemble in the original order.
The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum.
Definition. A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts.
Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.
Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size.
Direct MACs, total two-stage bottleneck MACs and percentage saved.
Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added.
One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs.
Step 1: direct layer. Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs.
Step 2: 1 × 1 reduction. There are 16 receiving filters, each spanning all 192 original channels.
Step 3: reduced-depth spatial layer. Its 32 filters now span 16 channels.
Step 4: add the two bottleneck costs.
Step 5: calculate the saving.
Final answer: direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption.
Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.
Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1.
Direct MACs, bottleneck MACs and saving.
Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers.
Step 1: direct spatial convolution.
Step 2: 1 × 1 reduction.
Step 3: reduced-depth 3 × 3 convolution.
Step 4: add the stages and subtract from the direct cost.
Final answer: direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values.
"],"search":"ResNet shortcut and Inception1×1 Describe skip connections and their benefit in ResNet. Explain a1×1convolution’s role in Inception with an example.","units":[4],"marks":10,"sourcePages":[4],"sourceKind":"actual-paper"},{"id":"tee-c21-s11","title":"R25 Q5b · ReLU RNN state with declared orientation","kind":"practice","question":["Question. Sketch a one-hidden-layer RNN and state its hidden-state formula. Compute hₜ for Wxh=[[0.2,0.4],[0.1,0.3]], Whh=[[0.5,0.6],[0.3,0.4]], b=[0.1,0.2], x=[1,0], previoush=[0,1], ReLU. (10 marks)
"],"answer":["Input-to-hidden weight matrix Wₓₕ: transforms the two current input values into contributions to the two hidden units.
![\"W_{xh}=\\begin{bmatrix}0.2&0.4\\\\0.1&0.3\\end{bmatrix}\"](\"assets/equations/recovered-a3abd76a293a016b.svg\")
Hidden-to-hidden weight matrix Wₕₕ: transforms the two previous hidden activations into recurrent contributions.
![\"W_{hh}=\\begin{bmatrix}0.5&0.6\\\\0.3&0.4\\end{bmatrix}\"](\"assets/equations/recovered-c38f9bda7b0fbd25.svg\")
| Quantity | Given value | Meaning |
|---|---|---|
| xₜ | [1, 0] | Current input row: two input values |
| hₜ₋₁ | [0, 1] | Previous hidden-state row: two stored activations |
| b | [0.1, 0.2] | One bias for each hidden unit |
| Activation | ReLU(z) = max(0, z) | Apply separately to each hidden weighted sum |
Sketch the one-hidden-layer RNN, state its recurrence, and calculate the two components of hₜ.
The printed vectors are horizontal and the weight-index convention is not specified. Adopt row vectors multiplied on the right:
![\"h_t=\\operatorname{ReLU}(x_tW_{xh}+h_{t-1}W_{hh}+b)\"](\"assets/equations/recovered-88904d5970c9803f.svg\")
[1,0]Wxh=[1×0.2+0×0.1,1×0.4+0×0.3]=[0.2,0.4].
[0,1]Whh=[0×0.5+1×0.3,0×0.6+1×0.4]=[0.3,0.4].
[0.2+0.3+0.1,0.4+0.4+0.2]=[0.6,1.0].
[max(0,0.6),max(0,1.0)]=[0.6,1.0].
Row-vector convention: hₜ=[0.6,1.0]. If the supplied matrices instead map column inputs by Wxh x + Whh h +b, the untransposed arrays give[0.9,0.7]. That is a different orientation; explicitly state the formula. Transposing the matrices when switching convention preserves the same network.
Source / convention note. Square matrices and printed horizontal vectors leave orientation ambiguous; both possible readings are explained, main answer follows row-vector form.
"],"search":"ReLU RNN state with declared orientation Sketch a one-hidden-layer RNN and state its hidden-state formula. Compute hₜ for Wxh=[[0.2,0.4],[0.1,0.3]], Whh=[[0.5,0.6],[0.3,0.4]], b=[0.1,0.2], x=[1,0], previoush=[0,1], ReLU.","units":[5],"marks":10,"sourcePages":[4],"sourceKind":"actual-paper"},{"id":"tee-c21-s12","title":"R25 Q6a · 3–2–1 forward pass and update w7,w8","kind":"practice","question":["Question. For inputs[1,0,1], weights w1…w8=[0.2,−0.3,0.4,0.1,−0.5,0.2,−0.3,−0.2], biases[−0.4,0.2,0.1], target1 and sigmoid at all neurons: find prediction,error,and update w7,w8 with η=0.9. (10 marks)
"],"answer":["Diagram edge mapping: x1→h1=w1,x1→h2=w2;x2→h1=w3,x2→h2=w4;x3→h1=w5,x3→h2=w6;h1→out=w7,h2→out=w8.
![\"\\begin{bmatrix}0.2&0.4&-0.5\\\\-0.3&0.1&0.2\\end{bmatrix}\"](\"assets/equations/recovered-af190f1cb4fd05fd.svg\")
Prediction, explicit error convention, two simultaneous output-weight updates.
![\"E=\\frac12(o-t)^2,\\quad\\delta_o=(o-t)o(1-o),\\quad\\frac{\\partial](\"assets/equations/recovered-bca4073f798d2cce.svg\")
The paper says error without naming its scaling. Use half squared error and also report residual/SSE, so the gradient convention is clear.
| Unit | Products +bias | Net | Activation |
|---|---|---|---|
| 1 | 1×(0.2) + 0×(0.4) + 1×(-0.5) +(-0.4) | -0.7 | 0.331812227832 |
| 2 | 1×(-0.3) + 0×(0.1) + 1×(0.2) +(0.2) | 0.10000000000000003 | 0.524979187479 |
z=0.331812227832×(−0.3)+0.524979187479×(−0.2)+0.1=-0.104539505845; o=1/(1+e^(−z))=0.473888898824.
Residual o−t=-0.526111101176; SSE=0.276792890781; halfSSE=0.138396445390.
δ=(0.473888898824−1)×0.473888898824×(1−0.473888898824)=-0.131169078214.
| Weight | δ × its source activation | Gradient |
|---|---|---|
| w7 | -0.131169078214×0.331812227832 | -0.043523504065 |
| w8 | -0.131169078214×0.524979187479 | -0.068861036103 |
| Weight | Old − learning rate × gradient | New |
|---|---|---|
| w7 | -0.3−0.9×(-0.043523504065) | -0.260828846342 |
| w8 | -0.2−0.9×(-0.068861036103) | -0.138025067507 |
w7′=-0.260828846342; w8′=-0.138025067507. Only the requested two weights are updated; use the same old forward pass for both gradients. With unhalved SSE the gradients double, so numerical updates would differ.
Source / convention note. Original does not specify loss scaling; half-SSE is declared and alternatives distinguished.
"],"search":"3–2–1 forward pass and update w7,w8 For inputs[1,0,1], weights w1…w8=[0.2,−0.3,0.4,0.1,−0.5,0.2,−0.3,−0.2], biases[−0.4,0.2,0.1], target1 and sigmoid at all neurons: find prediction,error,and update w7,w8 with η=0.9.","units":[2],"marks":10,"sourcePages":[5],"sourceKind":"actual-paper"},{"id":"tee-c21-s13","title":"R25 Q6b · AdaGrad stagnation versus RMSProp","kind":"practice","question":["Question. A healthcare model trained with AdaGrad becomes stagnant after many epochs. Discuss AdaGrad and RMSProp in detail and their fundamental differences. (10 marks)
"],"answer":["Definition. AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator.
For one parameter θ, let gₜ be its current gradient and s₀ = 0:
s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign.
Question. Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero.
θ₀ = 1; initial state(s) = 0; η = 0.1.
Supplied gradients g₁ = 2, g₂ = 1.
ε = 0 for this hand calculation.
Calculate all required states and the parameter after each of two updates.
AdaGrad. Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude.
G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so.
Yaad rakhne ka cue: Square karo, purane sum mein jodo, root se scale karo.
The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter.
Iteration 1 — step 1: update the accumulator.
Step 2: calculate the effective learning rate.
Step 3: apply the signed gradient.
Iteration 2 — step 1: retain s₁ and add the new squared gradient.
Step 2: calculate the new coefficient.
Step 3: update from θ₁.
Final answer: θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5.
Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes.
ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical.
Definition. RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator.
Using decay β and zero initial state:
β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update.
If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever.
Question. θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation.
θ₀ = 1; initial state(s) = 0; η = 0.1.
Supplied gradients g₁ = 2, g₂ = 1.
ε = 0 for this hand calculation.
β = 0.9.
Calculate all required states and the parameter after each of two updates.
RMSProp. Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.
s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.
Yaad rakhne ka cue: Saare purane squares nahi—unki fading average.
Iteration 1 — step 1: calculate the squared-gradient average.
Step 2: calculate the denominator and signed correction.
Step 3: update the parameter.
Iteration 2 — step 1: use s₁, not the original zero state.
Step 2: calculate the denominator and signed correction.
Step 3: update from θ₁, retaining calculator precision.
Final answer: θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46.
AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum.
AdaGrad’s nondecreasing squared-gradient sum can shrink its effective step excessively. RMSProp’s exponential average discounts older squared gradients, so it avoids that permanent accumulation. Both adapt rates coordinatewise and use ε for numerical stability. Switching is a plausible remedy, not a guarantee: inspect learning rate, gradient scale, data and convergence before attributing every plateau to the optimizer.
"],"search":"AdaGrad stagnation versus RMSProp A healthcare model trained with AdaGrad becomes stagnant after many epochs. Discuss AdaGrad and RMSProp in detail and their fundamental differences.","units":[3],"marks":10,"sourcePages":[5],"sourceKind":"actual-paper"},{"id":"tee-c21-s14","title":"R25 Q7a · Long-text sentiment architecture","kind":"practice","question":["Question. Choose a deep architecture for long social-media/customer-review sequences classified positive,negative or neutral. Explain its operation with equations. (10 marks)
"],"answer":["An LSTM is a suitable syllabus-aligned choice for long dependencies. Tokenize and embed the text, process the sequence through LSTM states, form a sequence representation and use a three-class softmax. Padding/masking must prevent padded tokens from being treated as content.
Definition. Long short-term memory (LSTM) is a gated recurrent architecture with a cell state and hidden state. Forget, input and output gates regulate state retention, incorporation of candidate information and exposure of the updated cell.
Long short-term memory, or LSTM, keeps a cell state Cₜ and a hidden state hₜ. A sigmoid gate produces values from 0 to 1: approximately 0 blocks a component, approximately 1 passes it. Intermediate values pass a fraction. Gates are learned from the previous hidden state and current input, not manually assigned by the user during normal inference.
The brackets [h, x] concatenate two vectors: [a, b] joined with [c, d] gives [a, b, c, d]. Concatenation is not addition. f is the forget gate: how much old cell content remains. i is the input gate: how much new candidate enters. The candidate is proposed content. o is the output gate: how much of the activated cell becomes visible as the hidden state.
![\"p=\\operatorname{softmax}(Wh+b),\\qquad](\"assets/equations/recovered-ce62d9e3abab42e3.svg\")
Train on labelled examples with cross-entropy and BPTT; evaluate on held-out reviews with class-wise precision/recall/F1. LSTM gates reduce long-path gradient decay but do not guarantee perfect memory. The three outputs are mutually exclusive classes for this question.
"],"search":"Long-text sentiment architecture Choose a deep architecture for long social-media/customer-review sequences classified positive,negative or neutral. Explain its operation with equations.","units":[5],"marks":10,"sourcePages":[5,6],"sourceKind":"actual-paper"},{"id":"tee-c21-s15","title":"R25 Q7b · Transfer learning","kind":"practice","question":["Question. Explain transfer learning and its benefits with suitable examples. (5 marks)
"],"answer":["Definition. Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers.
Transfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort.
Full scenario. Build a face-attendance model with few images per student. Explain a training plan.
Step 1. Define and split the data. Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization.
Step 2. Choose a suitable pretrained backbone. For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available.
Step 3. Replace the classifier. Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy.
Step 4. Freeze and train. Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring.
Step 5. Fine-tune if justified. Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint.
Step 6. Evaluate. Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition.
Feature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario.
Early image layers commonly learn reusable edges and textures. Later layers become more specialized to source-task categories. “Earlier” means closer to the input, regardless of whether a drawing places those layers at the top or bottom.
| Target dataset | Sensible starting choice | Reason |
|---|---|---|
| Small and similar | Freeze the backbone; train a new head first | Useful features already exist, while few examples make overfitting likely |
| Large and similar | Train a new head, then fine-tune more layers if validation improves | More data can support more trainable parameters |
| Small and different | Try a more relevant pretrained model or fixed earlier/intermediate features; tune cautiously | Later source features may be unsuitable, but small data still makes extensive tuning risky |
| Large and different | Compare broad fine-tuning with training from scratch | Sufficient data permits stronger adaptation; the value of source features must be tested |
These are starting strategies, not universal rules. Fine-tuning an early layer means updating its pretrained weights, not automatically learning it “from scratch”. With very little data, simply unfreezing early layers because the task differs can overfit. Choose changes using validation performance and apply the required preprocessing consistently.
Example. With 100 labelled microscopic images, an ImageNet model may provide reusable low-level patterns, but its animal/object classifier is inappropriate. Replace the head, test fixed representations and monitor validation before increasing the number of trainable layers. With a large representative image dataset, broader fine-tuning becomes a more reasonable candidate.
Yaad rakho: data kam toh kam weights hilao; task alag toh features ki suitability pehle dekho.
"],"search":"Transfer learning Explain transfer learning and its benefits with suitable examples.","units":[4],"marks":5,"sourcePages":[6],"sourceKind":"actual-paper"},{"id":"tee-c21-s16","title":"R25 Q7c · Learning rate and two losses","kind":"practice","question":["Question. Explain the importance of learning rate. Give and explain one classification loss and one regression loss. (5 marks)
"],"answer":["Definition. The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss.
A very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate.
| Observation | Possible reason | Sensible first action |
|---|---|---|
| Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation |
| Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate |
| Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum |
| Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping |
| All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights |
Convergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum.
![\"L_{class}=-\\sum_ky_k\\log](\"assets/equations/recovered-bb4226c392bbf68f.svg\")
Categorical cross-entropy penalizes low assigned probability to the true class; for a one-hot target and true-class probability0.8, L=−ln0.8≈0.223144. MSE measures average squared numeric prediction error; errors1and−2give(1²+(-2)²)/2=2.5. These are illustrative examples; their output interpretation and loss scaling must match the model.
"],"search":"Learning rate and two losses Explain the importance of learning rate. Give and explain one classification loss and one regression loss.","units":[2,3],"marks":5,"sourcePages":[6],"sourceKind":"actual-paper"}]},{"n":22,"title":"Faculty Assignment 2 · worked practice","sourceKind":"faculty-assignment","sections":[{"id":"tee-c22-s0","title":"Faculty Assignment2 Q1 · One momentum update with an omitted coefficient","kind":"practice","question":["Question. For J(w)=½(w−3)²,w₀=5,learning rate0.1,perform one SGD-with-momentum iteration. (Faculty assignment: no marks printed)
"],"answer":["w₀=5,η=0.1. βandinitialvelocityare not supplied.
One update conditional on zero initial velocity.
![\"g=w-3,\\quad](\"assets/equations/recovered-d398e0121f17d787.svg\")
Choose displacement convention and v₀=0; then the first step is independent of β.
g₀=5−3=2.
v₁=β×0+0.1×2=0.2.
w₁=5−0.2=4.8.
w₁=4.8 if v₀=0. For a nonzero initial velocity, w₁=4.8−βv₀; no unique numerical value follows without those givens.
Source / convention note. Authentic faculty Assignment2; no marks printed. β/v₀ omission explicitly identified.
"],"search":"One momentum update with an omitted coefficient For J(w)=½(w−3)²,w₀=5,learning rate0.1,perform one SGD-with-momentum iteration.","units":[3],"marks":null,"sourcePages":[],"sourceKind":"faculty-assignment"},{"id":"tee-c22-s1","title":"Faculty Assignment2 Q2 · Specified momentum update","kind":"practice","question":["Question. For J(w)=½(w+2)²,η=0.2,β=0.8,w₀=4,v₀=0,perform one momentum step. (Faculty assignment: no marks printed)
"],"answer":["All givens above.
v₁andw₁.
![\"g=w+2,\\quad](\"assets/equations/recovered-5b00da4482787868.svg\")
g₀=4+2=6.
v₁=0.8×0+0.2×6=1.2.
w₁=4−1.2=2.8.
v₁=1.2,w₁=2.8. A signed-velocity convention givesv₁=−1.2andw₁=w₀+v₁; the final weight is the same.
"],"search":"Specified momentum update For J(w)=½(w+2)²,η=0.2,β=0.8,w₀=4,v₀=0,perform one momentum step.","units":[3],"marks":null,"sourcePages":[],"sourceKind":"faculty-assignment"},{"id":"tee-c22-s2","title":"Faculty Assignment2 Q3 · 28×28 CNN","kind":"practice","question":["Question. Input28×28×1; conv5,6valid; pool2stride2; conv5,16valid; pool2stride2; dense120,dense84,dense10. Find layer dimensions and parameters. (Faculty assignment: no marks printed)
"],"answer":["Input=[28, 28, 1]
| Layer | Kernel | Stride | Padding | Filters / units |
|---|---|---|---|---|
| Conv | 5 | 1 | 0 | 6 |
| Pool | 2 | 2 | 0 | 0 |
| Conv | 5 | 1 | 0 | 16 |
| Pool | 2 | 2 | 0 | 0 |
| Dense | 0 | 0 | 0 | 120 |
| Dense | 0 | 0 | 0 | 84 |
| Dense | 0 | 0 | 0 | 10 |
Each output shape and all trainable weights plus biases.
Each filter/unit has one bias. Pooling has zero learned parameters. Flatten preserves the number of values, then a dense layer connects every input value to every output unit.
| Layer | Input | Full substitution and reason | Output | Parameters |
|---|---|---|---|---|
| Conv | 28×28×1 | ⌊(28+2×0−5)/1⌋+1=24; parameters=(5×5×1+1)×6=156 | 24×24×6 | 156 |
| Pool | 24×24×6 | ⌊(24−2)/2⌋+1=12; channels stay 6; no trainable weights | 12×12×6 | 0 |
| Conv | 12×12×6 | ⌊(12+2×0−5)/1⌋+1=8; parameters=(5×5×6+1)×16=2416 | 8×8×16 | 2416 |
| Pool | 8×8×16 | ⌊(8−2)/2⌋+1=4; channels stay 16; no trainable weights | 4×4×16 | 0 |
| Dense | 4×4×16 | Flatten/input length=4×4×16=256; parameters=(256+1)×120=30840 | 120 | 30840 |
| Dense | 1×1×120 | Flatten/input length=1×1×120=120; parameters=(120+1)×84=10164 | 84 | 10164 |
| Dense | 1×1×84 | Flatten/input length=1×1×84=84; parameters=(84+1)×10=850 | 10 | 850 |
156 + 2416 + 30840 + 10164 + 850 = 44426
Total parameters=44426. Shape and parameter count are separate results.
"],"search":"28×28 CNN Input28×28×1; conv5,6valid; pool2stride2; conv5,16valid; pool2stride2; dense120,dense84,dense10. Find layer dimensions and parameters.","units":[4],"marks":null,"sourcePages":[],"sourceKind":"faculty-assignment"},{"id":"tee-c22-s3","title":"Faculty Assignment2 Q4 · Batch linear regression update","kind":"practice","question":["Question. x=[1,2],y=[3,5],w=b=0,η=0.1;J=(1/(2m))Σ(wxᵢ+b−yᵢ)²,m=2. Compute both gradients and update. (Faculty assignment: no marks printed)
"],"answer":["Two examples, shared w,b, m=2.
One simultaneous batch update.
![\"\\frac{\\partial](\"assets/equations/recovered-4886822a6f85a414.svg\")
| x | y | wx+b | e | e×x |
|---|---|---|---|---|
| 1 | 3 | 0 | -3 | -3 |
| 2 | 5 | 0 | -5 | -10 |
dw=(−3−10)/2=−6.5;db=(−3−5)/2=−4.
w′=0−0.1×(−6.5)=0.65;b′=0−0.1×(−4)=0.4.
w′=0.65,b′=0.4. New predictions1.05and1.7; do not updatew before computing theoldbgradient.
"],"search":"Batch linear regression update x=[1,2],y=[3,5],w=b=0,η=0.1;J=(1/(2m))Σ(wxᵢ+b−yᵢ)²,m=2. Compute both gradients and update.","units":[2,3],"marks":null,"sourcePages":[],"sourceKind":"faculty-assignment"},{"id":"tee-c22-s4","title":"Faculty Assignment2 Q5 · 32×32 three-convolution CNN","kind":"practice","question":["Question. Input32×32×1; conv3,8same; pool2stride2; conv5,16valid; pool2stride2; conv3,32same; dense64; output5. Find shapes and parameters. (Faculty assignment: no marks printed)
"],"answer":["Input=[32, 32, 1]
| Layer | Kernel | Stride | Padding | Filters / units |
|---|---|---|---|---|
| Conv | 3 | 1 | 1 | 8 |
| Pool | 2 | 2 | 0 | 0 |
| Conv | 5 | 1 | 0 | 16 |
| Pool | 2 | 2 | 0 | 0 |
| Conv | 3 | 1 | 1 | 32 |
| Dense | 0 | 0 | 0 | 64 |
| Dense | 0 | 0 | 0 | 5 |
Each output shape and all trainable weights plus biases.
Each filter/unit has one bias. Pooling has zero learned parameters. Flatten preserves the number of values, then a dense layer connects every input value to every output unit.
| Layer | Input | Full substitution and reason | Output | Parameters |
|---|---|---|---|---|
| Conv | 32×32×1 | ⌊(32+2×1−3)/1⌋+1=32; parameters=(3×3×1+1)×8=80 | 32×32×8 | 80 |
| Pool | 32×32×8 | ⌊(32−2)/2⌋+1=16; channels stay 8; no trainable weights | 16×16×8 | 0 |
| Conv | 16×16×8 | ⌊(16+2×0−5)/1⌋+1=12; parameters=(5×5×8+1)×16=3216 | 12×12×16 | 3216 |
| Pool | 12×12×16 | ⌊(12−2)/2⌋+1=6; channels stay 16; no trainable weights | 6×6×16 | 0 |
| Conv | 6×6×16 | ⌊(6+2×1−3)/1⌋+1=6; parameters=(3×3×16+1)×32=4640 | 6×6×32 | 4640 |
| Dense | 6×6×32 | Flatten/input length=6×6×32=1152; parameters=(1152+1)×64=73792 | 64 | 73792 |
| Dense | 1×1×64 | Flatten/input length=1×1×64=64; parameters=(64+1)×5=325 | 5 | 325 |
80 + 3216 + 4640 + 73792 + 325 = 82053
Total parameters=82053. Shape and parameter count are separate results.
"],"search":"32×32 three-convolution CNN Input32×32×1; conv3,8same; pool2stride2; conv5,16valid; pool2stride2; conv3,32same; dense64; output5. Find shapes and parameters.","units":[4],"marks":null,"sourcePages":[],"sourceKind":"faculty-assignment"},{"id":"tee-c22-s5","title":"Faculty Assignment2 Q6 · 64×64 two-convolution CNN","kind":"practice","question":["Question. Input64×64×3; conv5,16valid; pool2stride2; conv3,32same; pool2stride2; dense128; output10. Find shapes and parameters. (Faculty assignment: no marks printed)
"],"answer":["Input=[64, 64, 3]
| Layer | Kernel | Stride | Padding | Filters / units |
|---|---|---|---|---|
| Conv | 5 | 1 | 0 | 16 |
| Pool | 2 | 2 | 0 | 0 |
| Conv | 3 | 1 | 1 | 32 |
| Pool | 2 | 2 | 0 | 0 |
| Dense | 0 | 0 | 0 | 128 |
| Dense | 0 | 0 | 0 | 10 |
Each output shape and all trainable weights plus biases.
Each filter/unit has one bias. Pooling has zero learned parameters. Flatten preserves the number of values, then a dense layer connects every input value to every output unit.
| Layer | Input | Full substitution and reason | Output | Parameters |
|---|---|---|---|---|
| Conv | 64×64×3 | ⌊(64+2×0−5)/1⌋+1=60; parameters=(5×5×3+1)×16=1216 | 60×60×16 | 1216 |
| Pool | 60×60×16 | ⌊(60−2)/2⌋+1=30; channels stay 16; no trainable weights | 30×30×16 | 0 |
| Conv | 30×30×16 | ⌊(30+2×1−3)/1⌋+1=30; parameters=(3×3×16+1)×32=4640 | 30×30×32 | 4640 |
| Pool | 30×30×32 | ⌊(30−2)/2⌋+1=15; channels stay 32; no trainable weights | 15×15×32 | 0 |
| Dense | 15×15×32 | Flatten/input length=15×15×32=7200; parameters=(7200+1)×128=921728 | 128 | 921728 |
| Dense | 1×1×128 | Flatten/input length=1×1×128=128; parameters=(128+1)×10=1290 | 10 | 1290 |
1216 + 4640 + 921728 + 1290 = 928874
Total parameters=928874. Shape and parameter count are separate results.
"],"search":"64×64 two-convolution CNN Input64×64×3; conv5,16valid; pool2stride2; conv3,32same; pool2stride2; dense128; output10. Find shapes and parameters.","units":[4],"marks":null,"sourcePages":[],"sourceKind":"faculty-assignment"}]}]},"meta":{"mt2":{"questions":[{"paper":"MT1A","q":"Q2","marks":3,"unit":"2","page":1,"status":"Retain"},{"paper":"MT1A","q":"Q3","marks":3,"unit":"2","page":1,"status":"Retain"},{"paper":"MT1A","q":"Q4","marks":3,"unit":"2","page":1,"status":"Retain"},{"paper":"MT1B","q":"Q3","marks":3,"unit":"2","page":2,"status":"Retain"},{"paper":"MT1B","q":"Q4","marks":3,"unit":"2","page":2,"status":"Retain"},{"paper":"F25","q":"Q1c","marks":5,"unit":"3","page":1,"status":"Retain"},{"paper":"F25","q":"Q1d","marks":5,"unit":"5","page":"1–2","status":"Retain"},{"paper":"F25","q":"Q2b","marks":10,"unit":"4","page":2,"status":"Retain"},{"paper":"F25","q":"Q3b","marks":10,"unit":"4","page":2,"status":"Retain"},{"paper":"F25","q":"Q4a","marks":10,"unit":"3","page":"2–3","status":"Retain"},{"paper":"F25","q":"Q5b","marks":10,"unit":"3","page":3,"status":"Retain"},{"paper":"F25","q":"Q6a","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"F25","q":"Q7b","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"R26","q":"Q1c","marks":5,"unit":"3","page":1,"status":"Retain"},{"paper":"R26","q":"Q1d","marks":5,"unit":"5","page":1,"status":"Retain"},{"paper":"R26","q":"Q2b","marks":10,"unit":"4","page":2,"status":"Retain"},{"paper":"R26","q":"Q3b","marks":10,"unit":"4","page":2,"status":"Retain"},{"paper":"R26","q":"Q4a","marks":10,"unit":"3","page":2,"status":"Retain"},{"paper":"R26","q":"Q5a","marks":10,"unit":"5","page":3,"status":"Retain"},{"paper":"R26","q":"Q5b","marks":10,"unit":"3","page":3,"status":"Retain"},{"paper":"R26","q":"Q6a","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"R26","q":"Q7b","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"F23","q":"Q1b","marks":5,"unit":"2–3","page":1,"status":"Retain"},{"paper":"F23","q":"Q1d","marks":5,"unit":"4","page":1,"status":"Retain"},{"paper":"F23","q":"Q3b","marks":10,"unit":"3","page":3,"status":"Retain"},{"paper":"F23","q":"Q4a","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"F23","q":"Q5a","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"F23","q":"Q6b","marks":10,"unit":"4","page":5,"status":"Retain"},{"paper":"F23","q":"Q7a","marks":10,"unit":"5","page":5,"status":"Retain"},{"paper":"F23","q":"Q7b","marks":10,"unit":"3","page":5,"status":"Retain"},{"paper":"R24","q":"Q1c","marks":5,"unit":"3","page":1,"status":"Retain"},{"paper":"R24","q":"Q1d","marks":5,"unit":"4","page":2,"status":"Retain"},{"paper":"R24","q":"Q2b","marks":10,"unit":"3","page":3,"status":"Retain comparison and ii; exclude formal rate bounds"},{"paper":"R24","q":"Q3b","marks":10,"unit":"3","page":3,"status":"Retain"},{"paper":"R24","q":"Q4b","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"R24","q":"Q5a","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"R24","q":"Q6b","marks":10,"unit":"4","page":5,"status":"Retain"},{"paper":"R24","q":"Q7b","marks":10,"unit":"5","page":5,"status":"Retain"},{"paper":"F22","q":"Q1a","marks":4,"unit":"4","page":1,"status":"Retain"},{"paper":"F22","q":"Q1b","marks":4,"unit":"3","page":1,"status":"Retain"},{"paper":"F22","q":"Q1e","marks":4,"unit":"5","page":2,"status":"Retain"},{"paper":"F22","q":"Q3a","marks":10,"unit":"2–3","page":3,"status":"Retain"},{"paper":"F22","q":"Q3b","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"F22","q":"Q4a","marks":10,"unit":"4","page":"3–4","status":"Retain"},{"paper":"F22","q":"Q4b","marks":10,"unit":"2–3","page":4,"status":"Retain"},{"paper":"F22","q":"Q5b","marks":10,"unit":"3","page":4,"status":"Retain"},{"paper":"F22","q":"Q6a","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"F22","q":"Q6b","marks":10,"unit":"2–3","page":"4–5","status":"Retain"},{"paper":"R23","q":"Q1a","marks":4,"unit":"5","page":1,"status":"Retain"},{"paper":"R23","q":"Q1b","marks":4,"unit":"4","page":1,"status":"Retain"},{"paper":"R23","q":"Q1d","marks":4,"unit":"3","page":1,"status":"Retain"},{"paper":"R23","q":"Q2a","marks":10,"unit":"2–4","page":2,"status":"Retain"},{"paper":"R23","q":"Q2b","marks":10,"unit":"2–3","page":2,"status":"Retain"},{"paper":"R23","q":"Q4a","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"R23","q":"Q4b","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"R23","q":"Q6a","marks":10,"unit":"3","page":4,"status":"Retain"},{"paper":"R23","q":"Q6b","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"R23","q":"Q7b","marks":10,"unit":"3","page":4,"status":"Retain"}],"drill_mapping":{"1":1,"2":2,"3":3,"4":4,"5":5,"6":6,"7":7,"8":8,"9":9,"10":10,"13":11,"14":12,"15":13,"16":14,"17":15,"18":16,"19":17,"20":18,"21":19,"22":20,"23":21,"24":22,"25":23,"26":24,"27":25,"28":26,"29":27,"30":28,"31":29,"32":30,"33":31,"34":32,"35":33,"36":34,"37":35,"38":36,"39":37,"40":38,"41":39,"46":40,"47":41,"48":42,"49":43,"51":44,"52":45},"drills":45,"source_pages":7},"tee":{"questions":[{"paper":"MT1A","q":"Q1a","marks":2,"unit":"1","page":1,"status":"Retain"},{"paper":"MT1A","q":"Q1b","marks":2,"unit":"1","page":1,"status":"Retain"},{"paper":"MT1A","q":"Q2","marks":3,"unit":"2","page":1,"status":"Retain"},{"paper":"MT1A","q":"Q3","marks":3,"unit":"2","page":1,"status":"Retain"},{"paper":"MT1A","q":"Q4","marks":3,"unit":"2","page":1,"status":"Retain"},{"paper":"MT1B","q":"Q1a","marks":2,"unit":"1","page":2,"status":"Retain"},{"paper":"MT1B","q":"Q1b","marks":2,"unit":"1","page":2,"status":"Retain"},{"paper":"MT1B","q":"Q2","marks":3,"unit":"2","page":2,"status":"Retain"},{"paper":"MT1B","q":"Q3","marks":3,"unit":"2","page":2,"status":"Retain"},{"paper":"MT1B","q":"Q4","marks":3,"unit":"2","page":2,"status":"Retain"},{"paper":"F25","q":"Q1a","marks":5,"unit":"1","page":1,"status":"Retain"},{"paper":"F25","q":"Q1b","marks":5,"unit":"2","page":1,"status":"Retain"},{"paper":"F25","q":"Q1c","marks":5,"unit":"3","page":1,"status":"Retain"},{"paper":"F25","q":"Q1d","marks":5,"unit":"5","page":"1–2","status":"Retain"},{"paper":"F25","q":"Q2a","marks":10,"unit":"2","page":2,"status":"Retain"},{"paper":"F25","q":"Q2b","marks":10,"unit":"4","page":2,"status":"Retain"},{"paper":"F25","q":"Q3a","marks":10,"unit":"2","page":2,"status":"Retain"},{"paper":"F25","q":"Q3b","marks":10,"unit":"4","page":2,"status":"Retain"},{"paper":"F25","q":"Q4a","marks":10,"unit":"3","page":"2–3","status":"Retain"},{"paper":"F25","q":"Q4b","marks":10,"unit":"5","page":3,"status":"Retain"},{"paper":"F25","q":"Q5a","marks":10,"unit":"5","page":3,"status":"Retain"},{"paper":"F25","q":"Q5b","marks":10,"unit":"3","page":3,"status":"Retain"},{"paper":"F25","q":"Q6a","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"F25","q":"Q6b","marks":10,"unit":"2","page":3,"status":"Retain"},{"paper":"F25","q":"Q7a","marks":10,"unit":"2","page":"3–4","status":"Retain"},{"paper":"F25","q":"Q7b","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"R26","q":"Q1a","marks":5,"unit":"1","page":1,"status":"Retain"},{"paper":"R26","q":"Q1b","marks":5,"unit":"2","page":1,"status":"Retain"},{"paper":"R26","q":"Q1c","marks":5,"unit":"3","page":1,"status":"Retain"},{"paper":"R26","q":"Q1d","marks":5,"unit":"5","page":1,"status":"Retain"},{"paper":"R26","q":"Q2a","marks":10,"unit":"2","page":2,"status":"Retain"},{"paper":"R26","q":"Q2b","marks":10,"unit":"4","page":2,"status":"Retain"},{"paper":"R26","q":"Q3a","marks":10,"unit":"2","page":2,"status":"Retain"},{"paper":"R26","q":"Q3b","marks":10,"unit":"4","page":2,"status":"Retain"},{"paper":"R26","q":"Q4a","marks":10,"unit":"3","page":2,"status":"Retain"},{"paper":"R26","q":"Q4b","marks":10,"unit":"5","page":"2–3","status":"Retain i–iii; exclude iv beyond explicit syllabus"},{"paper":"R26","q":"Q5a","marks":10,"unit":"5","page":3,"status":"Retain"},{"paper":"R26","q":"Q5b","marks":10,"unit":"3","page":3,"status":"Retain"},{"paper":"R26","q":"Q6a","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"R26","q":"Q6b","marks":10,"unit":"2","page":4,"status":"Retain"},{"paper":"R26","q":"Q7a","marks":10,"unit":"2","page":4,"status":"Retain"},{"paper":"R26","q":"Q7b","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"F23","q":"Q1a","marks":5,"unit":"1","page":1,"status":"Retain"},{"paper":"F23","q":"Q1b","marks":5,"unit":"2–3","page":1,"status":"Retain"},{"paper":"F23","q":"Q1c","marks":5,"unit":"5","page":1,"status":"Retain"},{"paper":"F23","q":"Q1d","marks":5,"unit":"4","page":1,"status":"Retain"},{"paper":"F23","q":"Q2a","marks":10,"unit":"2","page":2,"status":"Retain"},{"paper":"F23","q":"Q2b","marks":10,"unit":"1–2","page":3,"status":"Retain"},{"paper":"F23","q":"Q3a","marks":10,"unit":"2","page":3,"status":"Retain"},{"paper":"F23","q":"Q3b","marks":10,"unit":"3","page":3,"status":"Retain"},{"paper":"F23","q":"Q4a","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"F23","q":"Q4b","marks":10,"unit":"5","page":4,"status":"Retain"},{"paper":"F23","q":"Q5a","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"F23","q":"Q5b","marks":10,"unit":"2","page":4,"status":"Retain"},{"paper":"F23","q":"Q6a","marks":10,"unit":"5","page":5,"status":"Retain"},{"paper":"F23","q":"Q6b","marks":10,"unit":"4","page":5,"status":"Retain"},{"paper":"F23","q":"Q7a","marks":10,"unit":"5","page":5,"status":"Retain"},{"paper":"F23","q":"Q7b","marks":10,"unit":"3","page":5,"status":"Retain"},{"paper":"R24","q":"Q1a","marks":5,"unit":"1","page":1,"status":"Retain"},{"paper":"R24","q":"Q1b","marks":5,"unit":"2","page":1,"status":"Retain"},{"paper":"R24","q":"Q1c","marks":5,"unit":"3","page":1,"status":"Retain"},{"paper":"R24","q":"Q1d","marks":5,"unit":"4","page":2,"status":"Retain"},{"paper":"R24","q":"Q2a","marks":10,"unit":"2","page":"2–3","status":"Retain"},{"paper":"R24","q":"Q2b","marks":10,"unit":"3","page":3,"status":"Retain comparison and ii; exclude formal rate bounds"},{"paper":"R24","q":"Q3a","marks":10,"unit":"2","page":3,"status":"Retain"},{"paper":"R24","q":"Q3b","marks":10,"unit":"3","page":3,"status":"Retain"},{"paper":"R24","q":"Q4a","marks":10,"unit":"2","page":3,"status":"Retain"},{"paper":"R24","q":"Q4b","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"R24","q":"Q5a","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"R24","q":"Q5b","marks":10,"unit":"1,4,5","page":4,"status":"Retain"},{"paper":"R24","q":"Q6a","marks":10,"unit":"5","page":"4–5","status":"Retain"},{"paper":"R24","q":"Q6b","marks":10,"unit":"4","page":5,"status":"Retain"},{"paper":"R24","q":"Q7a","marks":10,"unit":"5","page":5,"status":"Retain"},{"paper":"R24","q":"Q7b","marks":10,"unit":"5","page":5,"status":"Retain"},{"paper":"F22","q":"Q1a","marks":4,"unit":"4","page":1,"status":"Retain"},{"paper":"F22","q":"Q1b","marks":4,"unit":"3","page":1,"status":"Retain"},{"paper":"F22","q":"Q1c","marks":4,"unit":"1","page":2,"status":"Retain"},{"paper":"F22","q":"Q1d","marks":4,"unit":"2","page":2,"status":"Retain"},{"paper":"F22","q":"Q1e","marks":4,"unit":"5","page":2,"status":"Retain"},{"paper":"F22","q":"Q2a","marks":10,"unit":"5","page":2,"status":"Retain"},{"paper":"F22","q":"Q2b","marks":10,"unit":"2","page":2,"status":"Retain"},{"paper":"F22","q":"Q3a","marks":10,"unit":"2–3","page":3,"status":"Retain"},{"paper":"F22","q":"Q3b","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"F22","q":"Q4a","marks":10,"unit":"4","page":"3–4","status":"Retain"},{"paper":"F22","q":"Q4b","marks":10,"unit":"2–3","page":4,"status":"Retain"},{"paper":"F22","q":"Q5a","marks":10,"unit":"5","page":4,"status":"Retain"},{"paper":"F22","q":"Q5b","marks":10,"unit":"3","page":4,"status":"Retain"},{"paper":"F22","q":"Q6a","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"F22","q":"Q6b","marks":10,"unit":"2–3","page":"4–5","status":"Retain"},{"paper":"F22","q":"Q7a","marks":10,"unit":"5","page":5,"status":"Retain"},{"paper":"F22","q":"Q7b","marks":10,"unit":"2","page":5,"status":"Retain"},{"paper":"R23","q":"Q1a","marks":4,"unit":"5","page":1,"status":"Retain"},{"paper":"R23","q":"Q1b","marks":4,"unit":"4","page":1,"status":"Retain"},{"paper":"R23","q":"Q1c","marks":4,"unit":"1","page":1,"status":"Retain"},{"paper":"R23","q":"Q1d","marks":4,"unit":"3","page":1,"status":"Retain"},{"paper":"R23","q":"Q1e","marks":4,"unit":"2","page":2,"status":"Retain"},{"paper":"R23","q":"Q2a","marks":10,"unit":"2–4","page":2,"status":"Retain"},{"paper":"R23","q":"Q2b","marks":10,"unit":"2–3","page":2,"status":"Retain"},{"paper":"R23","q":"Q3a","marks":10,"unit":"5","page":2,"status":"Retain"},{"paper":"R23","q":"Q3b","marks":10,"unit":"2","page":3,"status":"Retain"},{"paper":"R23","q":"Q4a","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"R23","q":"Q4b","marks":10,"unit":"4","page":3,"status":"Retain"},{"paper":"R23","q":"Q5a","marks":10,"unit":"5","page":4,"status":"Retain"},{"paper":"R23","q":"Q5b","marks":10,"unit":"2","page":4,"status":"Retain"},{"paper":"R23","q":"Q6a","marks":10,"unit":"3","page":4,"status":"Retain"},{"paper":"R23","q":"Q6b","marks":10,"unit":"4","page":4,"status":"Retain"},{"paper":"R23","q":"Q7a","marks":10,"unit":"5","page":4,"status":"Retain"},{"paper":"R23","q":"Q7b","marks":10,"unit":"3","page":4,"status":"Retain"}],"drills":52,"source_pages":34}},"titles":{"Variables functions and symbols":"Notation and numerical conventions","Matrix multiplication: every row has a job":"Linear algebra revision: vectors, matrices and affine layers","Derivatives without a calculus course":"Differentiation revision: derivatives, partial derivatives and gradients","The chain rule is multiplying linked effects":"Chain rule revision and a complete loss derivative","Probability mean and variance in plain language":"Probability and statistics revision","Exponentials and natural logarithms":"Exponential, logarithmic and related calculus identities","What does learning from data mean":"Machine learning, deep learning and the training objective","What one artificial neuron does":"Artificial neuron: mathematical model and working","What an optimizer is trying to do":"Optimization: objective, gradient and descent direction","A moving average remembers the recent past":"Exponentially weighted moving averages","Recognizing overfitting and underfitting":"Generalization, underfitting and overfitting","AlexNet and the reason for its components":"AlexNet: architecture, contributions and limitations","Inception examines several spatial scales":"Inception: parallel multi-scale feature extraction","ResNet adds a shortcut":"Residual networks: identity mapping and projection shortcuts","What makes a sequence different":"Sequence modeling and recurrent neural networks","The ordinary gradient descent update":"Gradient descent: algorithm and numerical","Batch stochastic and mini batch gradient descent":"Batch GD, stochastic GD and mini-batch GD","A tiny dataset shows why the methods differ":"Batch update versus sequential SGD: worked comparison","Learning rate and common training problems":"Learning rate, convergence and optimization difficulties","Momentum remembers movement":"Momentum: velocity, update rule and working","Nesterov looks before correcting":"Nesterov accelerated gradient (NAG)","AdaGrad adds every squared gradient":"AdaGrad: adaptive per-parameter learning rates","RMSProp remembers recent squared gradients":"RMSProp: decaying squared-gradient average","Adam remembers both direction and squared size":"Adam: moment estimates and parameter updates","Adam bias correction: why the denominator is one minus beta to a power":"Adam bias correction: derivation","One comparison to rebuild all optimizer formulas":"Optimizer comparison and update conventions","L1 and L2 add a cost for large weights":"L1 and L2 regularization: objectives and derivatives","Dropout removes activations during training":"Dropout: stochastic regularization","Early stopping uses the best validation point":"Early stopping and checkpoint selection","Choosing a remedy":"Selecting a remedy from training behavior","Inverted dropout during training and inference":"Inverted dropout: expectation and inference","Reading image shapes before doing CNN arithmetic":"CNN tensor notation and layer dimensions","A computer sees an image as numbers":"Image tensors and learned feature maps","The convolution calculation":"Convolution: operation and complete numerical","Stride padding and floor":"Stride, padding and output-size equations","Convolution over several channels":"Convolution across input channels","One filter over multiple channels: all contributions go into one cell":"Multi-channel convolution: worked numerical","Choosing more or fewer filters":"Filter count and output-channel depth","Pooling needs comparisons or averages":"Max pooling and average pooling","Parameters are different from activation values":"Trainable parameters versus activations","One complete CNN pipeline":"Complete CNN: output shapes and parameter counts","Why one by one convolution saves work":"1 × 1 convolutions and bottleneck calculations","Transfer learning from a small dataset":"Transfer learning: feature extraction and fine-tuning","The vanilla RNN equations from scratch":"Vanilla RNN: state and output equations","Your first complete training step":"Training a linear neuron: two complete updates","Activation functions and how to calculate them":"Activation functions: definitions, derivatives and limitations","Choosing a loss":"Loss functions: regression and classification","What you should understand before optimization":"Training procedure: forward pass, backward pass and update","LSTM a memory path with three gates":"LSTM: architecture and gate equations","GRU two gates and one state":"GRU: architecture and update conventions","Mean, variance and standard deviation from ordinary arithmetic":"Mean, variance and standard deviation","Standardization: what the answer 1 actually means":"Feature standardization: definition and calculation","Loss: how one wrong prediction becomes a number":"Squared-error objectives and their derivatives","Classification regression and learning types":"Learning tasks and supervision types","Training validation and test data":"Training, validation and test sets","Parameters and hyperparameters":"Parameters, hyperparameters and optimizer states","Complete sigmoid training example":"Sigmoid neuron: complete forward and backward pass","Backpropagation with one hidden neuron":"Backpropagation: chain rule through a hidden layer","More outputs and vectorized backpropagation":"Multi-output and vectorized backpropagation","Forward propagation through multiple neurons":"Forward propagation through a multilayer network","BPTT with every arithmetic step visible":"BPTT: complete two-step gradient calculation","Reading an LSTM equation one operation at a time":"LSTM state update: mathematical interpretation"},"chapterMeta":{"2":{"title":"Mathematics revision","label":"REFERENCE","description":"Notation, linear algebra, differentiation, probability and logarithms. Use this short refresher when required; elementary arithmetic is assumed."},"4":{"title":"Neural-network foundations","label":"UNITS 1–2 · PREREQUISITES","description":"Technical definitions, neuron models, activations, losses, forward propagation and backpropagation. These foundations support the MT2 calculations."},"5":{"title":"Optimization algorithms","label":"UNIT 3","description":"Gradient descent, batch variants, momentum, Nesterov, AdaGrad, RMSProp and Adam. Definitions and derivations lead into complete worked updates."},"6":{"title":"Regularization and generalization","label":"UNIT 3","description":"Overfitting, L1/L2 penalties, dropout and early stopping: mechanisms, equations, comparisons and numerical applications."},"7":{"title":"Convolutional neural networks","label":"UNIT 4","description":"Convolution, pooling, output dimensions and parameter counts, including full network calculations."},"8":{"title":"CNN architectures and transfer learning","label":"UNIT 4","description":"AlexNet, Inception, ResNet, bottlenecks and transfer learning. Study their structure, design purpose and calculations."},"9":{"title":"Recurrent neural networks","label":"UNIT 5","description":"State equations, parameter sharing and sequence mappings. The MT2 route covers introductory RNNs; TEE adds recurrent training and gated architectures."},"19":{"title":"Formula reference and exam revision","label":"REVISION","description":"Core formulas, their conditions and the complete worked applications."},"21":{"title":"February 2025 re-exam · complete solved paper","label":"Actual historical paper","description":"22 February 2025 re-examination; full original marks and explicit source conventions."},"22":{"title":"Faculty Assignment 2 · worked practice","label":"Faculty practice, not a PYQ","description":"Assignment 2 from faculty Teams; not counted as an examination paper."}},"edition":"faculty-mt2-pattern-2026-10-12","mt2Exam":{"received":"2026-10-12","source":"Faculty message relayed directly by the user in this chat","message":"There can be numericals from unit 3\nAnd 2 numericals from unit 4\n2mk can be a theory numerical combo or only theory\n3mk is 2 numericals and theory or 1 numerical and theory question with all theory of unit 3 and 4","clarification":"Either 2 numerical and theory or 1 theory 1 numerical combo with 1 numerical and 1 theory","threeMarkCombinations":[["numerical","numerical","theory"],["theory + numerical","numerical","theory"]],"focusUnits":[3,4],"marks":10,"durationMinutes":60,"readingSummary":"Faculty update, 12 October: prepare Unit 3 and Unit 4 theory and numericals. Give extra practice to two Unit 4 numerical questions. ANN and maths remain prerequisites; RNN is available in TEE.","mockInstructions":"60 minutes · 10 marks. Q1(a,b): 2 marks each, compulsory; attempt any two of Q2–Q4: 3 marks each. This uses the shared M2 choice pattern already supplied. The latest faculty message guides the Unit 3–4 theory/numerical mix; it does not fix individual question numbers or numerical topics.","mockDescriptions":{"A":"Q1 theory · two Unit 4 numericals + Unit 3 theory among the 3-mark choices","B":"Q1 theory/numerical combination · Unit 3 theory + numerical mixed question, Unit 4 numerical, Unit 4 theory"},"qualifications":["Both Unit 3 and Unit 4 theory must be prepared; numerical-heavy guidance is not permission to skip theory.","Two Unit 4 numericals are highlighted in the message, but their exact location among compulsory and optional parts is not given.","The user's clarification identifies N + N + T or mixed theory/numerical + N + T among the three 3-mark choices. It does not specify their order or exact topics.","RNN is not named in this update, so it is removed from the focused MT2 path and retained in TEE."],"excludedSectionIds":["mt2-c11-s1","mt2-c12-s1","mt2-c12-s5","mt2-c13-s6","mt2-c14-s7","mt2-c15-s2","mt2-c16-s0","mt2-c17-s39","mt2-c9-intro","mt2-c9-s0","mt2-c9-s1","mt2-c9-s2","mt2-c9-s3","mt2-c9-s4","mt2-c9-s5"]},"examSchedule":{"subject":"Deep Learning","date":"2026-10-13","day":"Tuesday","start":"15:00","end":"16:00","timezone":"Asia/Calcutta","duration_minutes":60,"marks":10,"marks_confirmed":true,"choice_pattern_confirmed":false,"exact_portion_confirmed":false,"source":"User-supplied Class Test II timetable; S02 course policy section 4"},"courses":{"mt2":[{"id":"mt2-c2-s5","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"mt2-c17-s1","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"mt2-c2-s5"},{"id":"mt2-c3-s1","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"mt2-c17-s4","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"mt2-c3-s1"},{"id":"mt2-c17-s5","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"mt2-c3-s1"},{"id":"mt2-c3-s3","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"mt2-c3-s6","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"mt2-c17-s6","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"mt2-c3-s6"},{"id":"mt2-c3-s9","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"mt2-c2-s6","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"mt2-c17-s3","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"mt2-c2-s6"},{"id":"mt2-c4-s0","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c4-s2","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c4-s3","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c4-s4","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c4-s5","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c4-s7","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c17-s8","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"mt2-c4-s7"},{"id":"mt2-c4-s9","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c17-s2","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"mt2-c4-s9"},{"id":"mt2-c17-s7","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"mt2-c4-s9"},{"id":"mt2-c17-s9","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"mt2-c4-s9"},{"id":"mt2-c4-s17","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c4-s10","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c4-s1","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c4-s16","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c17-s10","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"mt2-c4-s16"},{"id":"mt2-c4-s11","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c4-s12","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c17-s11","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"mt2-c4-s12"},{"id":"mt2-c17-s12","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"mt2-c4-s12"},{"id":"mt2-c17-s13","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"mt2-c4-s12"},{"id":"mt2-c4-s14","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"mt2-c10-s5","chapter":4,"group":"Full teacher backprop examples","role":"Worked example"},{"id":"mt2-c10-s6","chapter":4,"group":"Full teacher backprop examples","role":"Worked example"},{"id":"mt2-c10-s0","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"mt2-c10-s1","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"mt2-c10-s2","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"mt2-c10-s3","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"mt2-c10-s4","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"mt2-c5-s2","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c5-s3","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c17-s16","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s3"},{"id":"mt2-c17-s17","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s3"},{"id":"mt2-c5-s4","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c17-s15","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s4"},{"id":"mt2-c5-s6","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c17-s14","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s6"},{"id":"mt2-c5-s7","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c5-s8","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c5-s9","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c17-s18","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s9"},{"id":"mt2-c17-s19","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s9"},{"id":"mt2-c5-s10","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c17-s20","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s10"},{"id":"mt2-c17-s21","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s10"},{"id":"mt2-c5-s12","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c17-s22","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s12"},{"id":"mt2-c17-s23","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s12"},{"id":"mt2-c5-s13","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c17-s24","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s13"},{"id":"mt2-c17-s25","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s13"},{"id":"mt2-c5-s14","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c5-s15","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c17-s26","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s15"},{"id":"mt2-c17-s27","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"mt2-c5-s15"},{"id":"mt2-c5-s17","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c5-s18","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"mt2-c11-s4","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c11-s5","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c12-s4","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c12-s6","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c13-s0","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c13-s2","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c14-s0","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c14-s2","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c15-s1","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c15-s6","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c15-s7","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c16-s2","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c16-s7","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c16-s9","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"mt2-c6-s0","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"mt2-c6-s1","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"mt2-c17-s28","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"mt2-c6-s1"},{"id":"mt2-c17-s29","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"mt2-c6-s1"},{"id":"mt2-c17-s30","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"mt2-c6-s1"},{"id":"mt2-c17-s41","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"mt2-c6-s1"},{"id":"mt2-c6-s3","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"mt2-c6-s4","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"mt2-c6-s6","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"mt2-c6-s7","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"mt2-c17-s31","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"mt2-c6-s7"},{"id":"mt2-c17-s42","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"mt2-c6-s7"},{"id":"mt2-c11-s0","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"mt2-c12-s0","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"mt2-c13-s7","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"mt2-c14-s3","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"mt2-c15-s3","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"mt2-c15-s9","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"mt2-c16-s4","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"mt2-c7-s0","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c7-s2","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c17-s32","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"mt2-c7-s2"},{"id":"mt2-c7-s3","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c17-s33","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"mt2-c7-s3"},{"id":"mt2-c17-s36","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"mt2-c7-s3"},{"id":"mt2-c7-s5","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c7-s7","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c17-s34","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"mt2-c7-s7"},{"id":"mt2-c7-s8","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c7-s9","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c17-s35","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"mt2-c7-s9"},{"id":"mt2-c17-s40","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"mt2-c7-s9"},{"id":"mt2-c17-s43","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"mt2-c7-s9"},{"id":"mt2-c7-s10","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c7-s14","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c7-s15","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c7-s16","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c7-s17","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c7-s18","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"mt2-c11-s6","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c11-s7","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c12-s3","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c12-s7","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c13-s1","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c14-s1","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c14-s4","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c14-s5","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c15-s0","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c15-s4","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c15-s8","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c16-s1","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c16-s3","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c16-s5","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c16-s8","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"mt2-c8-s0","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s1","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s2","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c17-s37","chapter":8,"group":"CNN architectures and transfer learning","role":"Solved numerical","conceptId":"mt2-c8-s2"},{"id":"mt2-c8-s3","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c17-s38","chapter":8,"group":"CNN architectures and transfer learning","role":"Solved numerical","conceptId":"mt2-c8-s3"},{"id":"mt2-c17-s45","chapter":8,"group":"CNN architectures and transfer learning","role":"Solved numerical","conceptId":"mt2-c8-s3"},{"id":"mt2-c8-s4","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s7","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c17-s44","chapter":8,"group":"CNN architectures and transfer learning","role":"Solved numerical","conceptId":"mt2-c8-s7"},{"id":"mt2-c8-s9","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s10","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s11","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s12","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s13","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s14","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s15","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s16","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s17","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c8-s18","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"mt2-c11-s2","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c11-s3","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c12-s2","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c12-s8","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c13-s3","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c13-s4","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c13-s5","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c14-s6","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c15-s5","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c16-s6","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"mt2-c19-s0","chapter":19,"group":"Final revision","role":"Review"},{"id":"mt2-c19-s1","chapter":19,"group":"Final revision","role":"Review"},{"id":"mt2-c19-s2","chapter":19,"group":"Final revision","role":"Review"},{"id":"mt2-c19-s3","chapter":19,"group":"Final revision","role":"Review"},{"id":"mt2-c19-s4","chapter":19,"group":"Final revision","role":"Review"},{"id":"mt2-c19-s5","chapter":19,"group":"Final revision","role":"Review"}],"tee":[{"id":"tee-c2-s5","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"tee-c17-s1","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"tee-c2-s5"},{"id":"tee-c3-s1","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"tee-c17-s4","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"tee-c3-s1"},{"id":"tee-c17-s5","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"tee-c3-s1"},{"id":"tee-c3-s3","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"tee-c3-s6","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"tee-c17-s6","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"tee-c3-s6"},{"id":"tee-c3-s9","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"tee-c2-s6","chapter":2,"group":"The arithmetic and calculator you need","role":"Maths revision"},{"id":"tee-c17-s3","chapter":2,"group":"The arithmetic and calculator you need","role":"Solved numerical","conceptId":"tee-c2-s6"},{"id":"tee-c4-s0","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s5","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s2","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s3","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s4","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s6","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s21","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s8","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c17-s11","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s8"},{"id":"tee-c4-s9","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c17-s12","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s9"},{"id":"tee-c4-s10","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c17-s8","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s10"},{"id":"tee-c4-s12","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c17-s2","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s12"},{"id":"tee-c17-s7","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s12"},{"id":"tee-c17-s9","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s12"},{"id":"tee-c4-s20","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s13","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s1","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s19","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c17-s10","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s19"},{"id":"tee-c4-s14","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c4-s15","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c17-s13","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s15"},{"id":"tee-c17-s14","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s15"},{"id":"tee-c17-s15","chapter":4,"group":"Machine learning and neural networks from zero","role":"Solved numerical","conceptId":"tee-c4-s15"},{"id":"tee-c4-s17","chapter":4,"group":"Machine learning and neural networks from zero","role":"Theory and working"},{"id":"tee-c10-s10","chapter":4,"group":"Full teacher backprop examples","role":"Worked example"},{"id":"tee-c10-s11","chapter":4,"group":"Full teacher backprop examples","role":"Worked example"},{"id":"tee-c10-s0","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c10-s1","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c10-s2","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c10-s3","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c10-s4","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c10-s5","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c10-s6","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c10-s7","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c10-s8","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c10-s9","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c11-s0","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c11-s1","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c11-s4","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c11-s6","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c11-s13","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c11-s14","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c12-s0","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c12-s1","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c12-s4","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c12-s6","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c12-s13","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c12-s14","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c13-s0","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c13-s4","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c13-s5","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c13-s6","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c13-s11","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c14-s0","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c14-s1","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c14-s4","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c14-s6","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c14-s8","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c15-s2","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c15-s3","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c15-s6","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c15-s16","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c16-s2","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c16-s4","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c16-s8","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c16-s12","chapter":4,"group":"Machine learning and neural networks from zero","role":"Exam question + answer"},{"id":"tee-c5-s2","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c5-s3","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c17-s18","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s3"},{"id":"tee-c17-s19","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s3"},{"id":"tee-c5-s4","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c17-s17","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s4"},{"id":"tee-c5-s6","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c17-s16","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s6"},{"id":"tee-c5-s7","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c5-s8","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c5-s9","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c17-s20","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s9"},{"id":"tee-c17-s21","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s9"},{"id":"tee-c5-s10","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c17-s22","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s10"},{"id":"tee-c17-s23","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s10"},{"id":"tee-c5-s12","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c17-s24","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s12"},{"id":"tee-c17-s25","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s12"},{"id":"tee-c5-s13","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c17-s26","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s13"},{"id":"tee-c17-s27","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s13"},{"id":"tee-c5-s14","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c5-s15","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c17-s28","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s15"},{"id":"tee-c17-s29","chapter":5,"group":"Optimization one update at a time","role":"Solved numerical","conceptId":"tee-c5-s15"},{"id":"tee-c5-s17","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c5-s18","chapter":5,"group":"Optimization one update at a time","role":"Theory and working"},{"id":"tee-c11-s8","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c11-s11","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c12-s8","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c12-s11","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c13-s1","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c13-s7","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c14-s2","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c14-s5","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c15-s1","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c15-s10","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c15-s12","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c16-s3","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c16-s13","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c16-s16","chapter":5,"group":"Optimization one update at a time","role":"Exam question + answer"},{"id":"tee-c6-s0","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"tee-c6-s1","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"tee-c17-s30","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"tee-c6-s1"},{"id":"tee-c17-s31","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"tee-c6-s1"},{"id":"tee-c17-s32","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"tee-c6-s1"},{"id":"tee-c17-s47","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"tee-c6-s1"},{"id":"tee-c6-s3","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"tee-c6-s4","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"tee-c6-s6","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"tee-c6-s7","chapter":6,"group":"Generalization and regularization","role":"Theory and working"},{"id":"tee-c17-s33","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"tee-c6-s7"},{"id":"tee-c17-s48","chapter":6,"group":"Generalization and regularization","role":"Solved numerical","conceptId":"tee-c6-s7"},{"id":"tee-c11-s2","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"tee-c12-s2","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"tee-c13-s15","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"tee-c14-s7","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"tee-c15-s7","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"tee-c15-s14","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"tee-c16-s6","chapter":6,"group":"Generalization and regularization","role":"Exam question + answer"},{"id":"tee-c7-s0","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c7-s2","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c17-s34","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"tee-c7-s2"},{"id":"tee-c7-s3","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c17-s35","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"tee-c7-s3"},{"id":"tee-c17-s38","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"tee-c7-s3"},{"id":"tee-c7-s5","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c7-s7","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c17-s36","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"tee-c7-s7"},{"id":"tee-c7-s8","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c7-s9","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c17-s37","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"tee-c7-s9"},{"id":"tee-c17-s46","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"tee-c7-s9"},{"id":"tee-c17-s49","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Solved numerical","conceptId":"tee-c7-s9"},{"id":"tee-c7-s10","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c7-s14","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c7-s15","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c7-s16","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c7-s17","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c7-s18","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Theory and working"},{"id":"tee-c11-s12","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c11-s15","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c12-s7","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c12-s12","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c13-s3","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c14-s3","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c14-s9","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c14-s10","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c15-s0","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c15-s8","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c15-s13","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c16-s1","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c16-s5","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c16-s9","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c16-s14","chapter":7,"group":"CNN calculations from a grid of numbers","role":"Exam question + answer"},{"id":"tee-c8-s0","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s1","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s2","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c17-s39","chapter":8,"group":"CNN architectures and transfer learning","role":"Solved numerical","conceptId":"tee-c8-s2"},{"id":"tee-c8-s3","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c17-s40","chapter":8,"group":"CNN architectures and transfer learning","role":"Solved numerical","conceptId":"tee-c8-s3"},{"id":"tee-c17-s52","chapter":8,"group":"CNN architectures and transfer learning","role":"Solved numerical","conceptId":"tee-c8-s3"},{"id":"tee-c8-s4","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s7","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c17-s51","chapter":8,"group":"CNN architectures and transfer learning","role":"Solved numerical","conceptId":"tee-c8-s7"},{"id":"tee-c8-s9","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s10","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s11","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s12","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s13","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s14","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s15","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s16","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s17","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c8-s18","chapter":8,"group":"CNN architectures and transfer learning","role":"Theory and working"},{"id":"tee-c11-s5","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c11-s7","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c12-s5","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c12-s15","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c13-s8","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c13-s10","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c13-s13","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c14-s13","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c15-s9","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c16-s10","chapter":8,"group":"CNN architectures and transfer learning","role":"Exam question + answer"},{"id":"tee-c9-s0","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s1","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s3","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s4","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s5","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c17-s41","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Solved numerical","conceptId":"tee-c9-s5"},{"id":"tee-c9-s6","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s7","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c17-s45","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Solved numerical","conceptId":"tee-c9-s7"},{"id":"tee-c9-s8","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s10","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s11","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s12","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c17-s42","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Solved numerical","conceptId":"tee-c9-s12"},{"id":"tee-c9-s13","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c17-s43","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Solved numerical","conceptId":"tee-c9-s13"},{"id":"tee-c17-s50","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Solved numerical","conceptId":"tee-c9-s13"},{"id":"tee-c9-s14","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c17-s44","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Solved numerical","conceptId":"tee-c9-s14"},{"id":"tee-c9-s15","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s17","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s18","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c9-s19","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Theory and working"},{"id":"tee-c11-s3","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c11-s9","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c11-s10","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c12-s3","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c12-s9","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c12-s10","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c13-s2","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c13-s9","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c13-s12","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c13-s14","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c14-s11","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c14-s12","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c14-s14","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c14-s15","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c15-s4","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c15-s5","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c15-s11","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c15-s15","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c16-s0","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c16-s7","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c16-s11","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c16-s15","chapter":9,"group":"Sequences RNN LSTM and GRU","role":"Exam question + answer"},{"id":"tee-c19-s0","chapter":19,"group":"Final revision","role":"Review"},{"id":"tee-c19-s1","chapter":19,"group":"Final revision","role":"Review"},{"id":"tee-c19-s2","chapter":19,"group":"Final revision","role":"Review"},{"id":"tee-c19-s3","chapter":19,"group":"Final revision","role":"Review"},{"id":"tee-c19-s4","chapter":19,"group":"Final revision","role":"Review"},{"id":"tee-c19-s5","chapter":19,"group":"Final revision","role":"Review"}]},"redirects":{"mt2":{"mt2-c2-s0":{"id":"mt2-c2-s5","chapter":2},"mt2-c2-s1":{"id":"mt2-c2-s5","chapter":2},"mt2-c2-s2":{"id":"mt2-c2-s5","chapter":2},"mt2-c2-s3":{"id":"mt2-c2-s5","chapter":2},"mt2-c2-s4":{"id":"mt2-c2-s5","chapter":2},"mt2-c2-s7":{"id":"mt2-c2-s6","chapter":2},"mt2-c2-s8":{"id":"mt2-c2-s5","chapter":2},"mt2-c2-s9":{"id":"mt2-c4-s9","chapter":4},"mt2-c3-s0":{"id":"mt2-c2-s5","chapter":2},"mt2-c3-s2":{"id":"mt2-c2-s5","chapter":2},"mt2-c3-s4":{"id":"mt2-c2-s5","chapter":2},"mt2-c3-s5":{"id":"mt2-c2-s5","chapter":2},"mt2-c3-s7":{"id":"mt2-c2-s5","chapter":2},"mt2-c3-s8":{"id":"mt2-c2-s5","chapter":2},"mt2-c3-s10":{"id":"mt2-c2-s5","chapter":2},"mt2-c3-s11":{"id":"mt2-c2-s5","chapter":2},"mt2-c3-s12":{"id":"mt2-c3-s6","chapter":2},"mt2-c4-s8":{"id":"mt2-c4-s7","chapter":4},"mt2-c4-s13":{"id":"mt2-c4-s12","chapter":4},"mt2-c5-s5":{"id":"mt2-c5-s4","chapter":5},"mt2-c5-s11":{"id":"mt2-c5-s10","chapter":5},"mt2-c5-s16":{"id":"mt2-c5-s15","chapter":5},"mt2-c6-s2":{"id":"mt2-c6-s1","chapter":6},"mt2-c6-s8":{"id":"mt2-c6-s7","chapter":6},"mt2-c7-s13":{"id":"mt2-c7-s9","chapter":7},"mt2-c9-s4":{"id":"tee-c9-s17","chapter":9,"exam":"tee"},"mt2-c14-s7":{"id":"tee-c14-s15","chapter":9,"exam":"tee"},"mt2-c16-s0":{"id":"tee-c16-s0","chapter":9,"exam":"tee"},"mt2-c9-s0":{"id":"tee-c9-s1","chapter":9,"exam":"tee"},"mt2-c15-s2":{"id":"tee-c15-s4","chapter":9,"exam":"tee"},"mt2-c9-s1":{"id":"tee-c9-s1","chapter":9,"exam":"tee"},"mt2-c11-s1":{"id":"tee-c11-s3","chapter":9,"exam":"tee"},"mt2-c9-intro":{"id":"tee-c9-s0","chapter":9,"exam":"tee"},"mt2-c13-s6":{"id":"tee-c13-s14","chapter":9,"exam":"tee"},"mt2-c17-s39":{"id":"tee-c17-s41","chapter":9,"exam":"tee"},"mt2-c12-s5":{"id":"tee-c12-s10","chapter":9,"exam":"tee"},"mt2-c9-s3":{"id":"tee-c9-s5","chapter":9,"exam":"tee"},"mt2-c9-s5":{"id":"tee-c9-s1","chapter":9,"exam":"tee"},"mt2-c12-s1":{"id":"tee-c12-s3","chapter":9,"exam":"tee"},"mt2-c9-s2":{"id":"tee-c9-s4","chapter":9,"exam":"tee"}},"tee":{"tee-c2-s0":{"id":"tee-c2-s5","chapter":2},"tee-c2-s1":{"id":"tee-c2-s5","chapter":2},"tee-c2-s2":{"id":"tee-c2-s5","chapter":2},"tee-c2-s3":{"id":"tee-c2-s5","chapter":2},"tee-c2-s4":{"id":"tee-c2-s5","chapter":2},"tee-c2-s7":{"id":"tee-c2-s6","chapter":2},"tee-c2-s8":{"id":"tee-c2-s5","chapter":2},"tee-c2-s9":{"id":"tee-c4-s12","chapter":4},"tee-c3-s0":{"id":"tee-c2-s5","chapter":2},"tee-c3-s2":{"id":"tee-c2-s5","chapter":2},"tee-c3-s4":{"id":"tee-c2-s5","chapter":2},"tee-c3-s5":{"id":"tee-c2-s5","chapter":2},"tee-c3-s7":{"id":"tee-c2-s5","chapter":2},"tee-c3-s8":{"id":"tee-c2-s5","chapter":2},"tee-c3-s10":{"id":"tee-c2-s5","chapter":2},"tee-c3-s11":{"id":"tee-c2-s5","chapter":2},"tee-c3-s12":{"id":"tee-c3-s6","chapter":2},"tee-c4-s11":{"id":"tee-c4-s10","chapter":4},"tee-c4-s16":{"id":"tee-c4-s15","chapter":4},"tee-c5-s5":{"id":"tee-c5-s4","chapter":5},"tee-c5-s11":{"id":"tee-c5-s10","chapter":5},"tee-c5-s16":{"id":"tee-c5-s15","chapter":5},"tee-c6-s2":{"id":"tee-c6-s1","chapter":6},"tee-c6-s8":{"id":"tee-c6-s7","chapter":6},"tee-c7-s13":{"id":"tee-c7-s9","chapter":7},"tee-c9-s9":{"id":"tee-c9-s7","chapter":9}}},"formulaGuides":{"mt2":[{"n":1,"title":"Mean, variance and standard deviation","intro":["The mean is a measure of location; variance and standard deviation quantify dispersion. Distinguish population and sample conventions before calculating these statistics.
"],"formulas":["![\"\\mu=\\frac{1}{n}\\sum_i](\"assets/equations/515c6d49d5a0ee05.svg\")
![\"v=\\frac{1}{n}\\sum_i(x_i-\\mu)^2\"](\"assets/equations/5b9364e6a03439b4.svg\")
![\"s=\\sqrt{v}\"](\"assets/equations/c9fd8a46ad5f2e60.svg\")
Question. For the population [1, 2, 3], find mean, population variance and standard deviation.
","Values = [1, 2, 3]; N = 3. Use population division by N.
","Mean μ, variance σ² and standard deviation σ.
","The question specifies a population, so use N rather than the N−1 sample-variance divisor.
","μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.
","Yaad rakhne ka cue: Mean, distance, square, mean again, then root.
","Step 1: add all values.
","Step 2: divide by the number of values.
","Step 3: find and square deviation 1.
","Step 4: find and square deviation 2.
","Step 5: find and square deviation 3.
","Step 6: average the squared deviations.
","Step 7: take the nonnegative square root.
","Final answer: μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581.
","Question. For the population [2, 2, 2], find mean, population variance and standard deviation.
","Values = [2, 2, 2]; N = 3. Use population division by N.
","Mean μ, variance σ² and standard deviation σ.
","The question specifies a population, so use N rather than the N−1 sample-variance divisor.
","μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.
","Yaad rakhne ka cue: Mean, distance, square, mean again, then root.
","Step 1: add all values.
","Step 2: divide by the number of values.
","Step 3: find and square deviation 1.
","Step 4: find and square deviation 2.
","Step 5: find and square deviation 3.
","Step 6: average the squared deviations.
","Step 7: take the nonnegative square root.
","Final answer: μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation.
"]},{"n":2,"title":"Standardization","intro":["Standardization has a meaning before it has a formula: measure how far an input is from its training mean, then express that distance in standard-deviation units. The worked question below calculates the mean and standard deviation first, so neither number appears without explanation.
"],"formulas":["Derivatives describe local rates of change. Partial derivatives hold other independent variables fixed; their collection forms the gradient used for optimization.
"],"formulas":["![\"\\frac{d}{dx}(cx^n)=cnx^{n-1}\"](\"assets/equations/b5c78e62c5b2a236.svg\")
![\"\\frac{d}{dx}c=0\"](\"assets/equations/d569e096ba9d6dba.svg\")
![\"(f+g)\\prime=f\\prime+g\\prime\"](\"assets/equations/032a4e4c1f8b2817.svg\")
The chain rule differentiates a composite function. Multiply local derivatives along a dependency path and add the contributions of different paths to the same variable.
"],"formulas":["A neuron starts with matched input–weight pairs. Multiply inside each pair, add the products, add the bias once, and then apply the activation. The summation sign is simply a shorter way to write those additions.
"],"formulas":["![\"z=\\sum_i](\"assets/equations/051f16f4bf1744dd.svg\")
Matrix multiplication implements affine transformations in a dense layer. Dimensions determine which products are valid; each output component is the dot product of a weight row and the input vector, plus its bias.
"],"formulas":["A layer does the neuron calculation for several receiving neurons. Finish all its outputs before using them as the next layer’s inputs. The bracketed superscript is a layer label, not a power.
"],"formulas":["![\"a^{(l)}=f(z^{(l)})\"](\"assets/equations/85d817a60006abc5.svg\")
Sigmoid takes a raw score and changes it to a number between 0 and 1. Its derivative is a different number: it tells backpropagation how sensitive that output is to its raw score. The complete example below calculates both and shows where each is used.
"],"formulas":["![\"\\sigma(z)=\\frac{1}{1+e^{-z}}\"](\"assets/equations/c3083d9958c3932d.svg\")
![\"\\sigma\\prime(z)=\\sigma(z)[1-\\sigma(z)]\"](\"assets/equations/90cd2771a42dc9ef.svg\")
Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.
","z = -2; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","Step 2: calculate the exponential.
","Step 3: add one in the denominator.
","Step 4: divide one by the complete denominator.
","Step 5: use the activated output in the slope.
","Final answer: a ≈ 0.119202922. Derivative ≈ 0.104993585.
","Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.
","z = 0; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","Step 2: calculate the exponential.
","Step 3: add one in the denominator.
","Step 4: divide one by the complete denominator.
","Step 5: use the activated output in the slope.
","Final answer: a ≈ 0.500000000. Derivative ≈ 0.250000000.
","Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.
","z = 2; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","Step 2: calculate the exponential.
","Step 3: add one in the denominator.
","Step 4: divide one by the complete denominator.
","Step 5: use the activated output in the slope.
","Final answer: a ≈ 0.880797078. Derivative ≈ 0.104993585.
"]},{"n":9,"title":"Tanh and its derivative","intro":["Tanh can produce a negative or positive output. In the formula, calculate the exponential first, then form the top and bottom of the fraction separately. The derivative uses the activated answer, not the raw input.
"],"formulas":["![\"\\tanh(z)=\\frac{e^{2z}-1}{e^{2z}+1}\"](\"assets/equations/a8ee72519b086209.svg\")
Question. Calculate tanh(0.5) and its derivative with respect to the input.
","z = 0.5.
","Find tanh(z) and its derivative.
","Tanh and its slope. Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².
","z = value entering tanh; a = its output. A prime means derivative, or local rate of change.
","Yaad rakhne ka cue: Tanh signed output deta hai; slope = 1 minus output ka square.
","Step 1: identify the input. The input z is 0.5. In this version of the tanh formula, the exponential uses 2z.
","![\"2z=2(0.5)=1,\\qquad](\"assets/equations/a761ae42b593cfbc.svg\")
Step 2: calculate the numerator. Subtract 1 from that exponential.
","![\"e^{2z}-1=2.718281828-1=1.718281828\"](\"assets/equations/dc5e5624414a3d95.svg\")
Step 3: calculate the denominator. Add 1 to the same exponential.
","![\"e^{2z}+1=2.718281828+1=3.718281828\"](\"assets/equations/343ff32a5ddf3a45.svg\")
Step 4: divide the top by the bottom.
","![\"a=\\tanh(0.5)=\\frac{1.718281828}{3.718281828}\\approx0.462117157\"](\"assets/equations/ebe33ccb05db4c46.svg\")
Step 5: calculate the local derivative. Square the output a and subtract it from 1. Keep the calculator's unrounded result.
","![\"a^2\\approx0.213552267,\\qquad1-a^2\\approx0.786447733\"](\"assets/equations/de782b0d8cb8f7bb.svg\")
Final answer: tanh output ≈ 0.462117157; local derivative ≈ 0.786447733. The output becomes the next layer's input; the derivative is used when carrying a gradient backward.
","Question. For preactivation z = -2, calculate the tanh output and state its derivative.
","z = -2; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","Step 4: divide.
","Step 5: square the activated output for its slope.
","Final answer: a ≈ -0.964027580. Derivative ≈ 0.070650825.
","Question. For preactivation z = 0, calculate the tanh output and state its derivative.
","z = 0; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","Step 4: divide.
","Step 5: square the activated output for its slope.
","Final answer: a ≈ 0.000000000. Derivative ≈ 1.000000000.
","Question. For preactivation z = 2, calculate the tanh output and state its derivative.
","z = 2; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","Step 4: divide.
","Step 5: square the activated output for its slope.
","Final answer: a ≈ 0.964027580. Derivative ≈ 0.070650825.
"]},{"n":10,"title":"ReLU and its derivative","intro":["ReLU compares its input with zero and keeps the larger value. Thus a negative input becomes zero and a positive input passes unchanged. Its slope is zero on the negative side and one on the positive side.
"],"formulas":["![\"f(z)=\\max(0,z)\"](\"assets/equations/1ad762a28c8a1a87.svg\")
![\"f\\prime(z)=\\begin{cases}1&z>0\\\\0&z<0\\end{cases}\"](\"assets/equations/458083af5d5ba2ef.svg\")
Question. For preactivation z = -2, calculate the ReLU output and state its derivative.
","z = -2; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","-2 is negative, so use the negative branch.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 0.000000000. Derivative ≈ 0.000000000.
","Question. For preactivation z = 0, calculate the ReLU output and state its derivative.
","z = 0; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","Both branches meet at zero. The value is zero, but the left and right slopes differ.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.
","Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.
","Question. For preactivation z = 2, calculate the ReLU output and state its derivative.
","z = 2; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","2 is positive, so retain it.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.
"]},{"n":11,"title":"Leaky ReLU","intro":["Leaky ReLU changes only the negative side: multiply a negative input by the small slope α. That negative sign remains. The value α is supplied by the question or chosen as a model setting; it is not the sigmoid function.
"],"formulas":["Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = -2; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","-2 is negative, so use the negative branch.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ -0.200000000. Derivative ≈ 0.100000000.
","Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = 0; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","Both branches meet at zero. The value is zero, but the left and right slopes differ.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.
","Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.
","Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = 2; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","2 is positive, so retain it.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.
"]},{"n":12,"title":"Stable softmax","intro":["Softmax turns several competing class scores into probabilities that add to one. Exponentiate each score, add those exponential values, and divide each one by their common total. Subtracting the same maximum score first gives the same probabilities with safer arithmetic.
"],"formulas":["Question. Find stable softmax probabilities for logits [0, 0, 0].
","Logits = [0, 0, 0]; there are three mutually exclusive classes.
","All three class probabilities.
","Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.
","A logit is a raw class score. c is the largest score; pᵢ is probability for class i.
","Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.
","Step 1: subtract the largest score from every entry.
","Step 2: exponentiate shifted entry 1.
","Step 3: exponentiate shifted entry 2.
","Step 4: exponentiate shifted entry 3.
","Step 5: add all exponentials into one denominator.
","Step 6: divide entry 1 by the same denominator.
","Step 7: divide entry 2 by the same denominator.
","Step 8: divide entry 3 by the same denominator.
","Final answer: Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding.
","Question. Find stable softmax probabilities for logits [1, 2, 3].
","Logits = [1, 2, 3]; there are three mutually exclusive classes.
","All three class probabilities.
","Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.
","A logit is a raw class score. c is the largest score; pᵢ is probability for class i.
","Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.
","Step 1: subtract the largest score from every entry.
","Step 2: exponentiate shifted entry 1.
","Step 3: exponentiate shifted entry 2.
","Step 4: exponentiate shifted entry 3.
","Step 5: add all exponentials into one denominator.
","Step 6: divide entry 1 by the same denominator.
","Step 7: divide entry 2 by the same denominator.
","Step 8: divide entry 3 by the same denominator.
","Final answer: Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding.
"]},{"n":13,"title":"Squared and absolute losses","intro":["SSE, half SSE and MSE start with the same errors but apply different final divisions. First understand what an error and its square mean. The full question below keeps all three answers separate.
"],"formulas":["Question. Targets are [2, 4] and predictions are [1, 6]. Find mean absolute error.
","Targets = [2,4]; predictions = [1,6]; n = 2.
","Find mean absolute error.
","Mean absolute error. Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes.
","n = number of predictions. |e| means the nonnegative magnitude of error e.
","Yaad rakhne ka cue: MAE mein magnitude; MSE mein square.
","The signed errors are 1 − 2 = −1 and 6 − 4 = 2. Absolute value keeps the size of a number without its sign, so their absolute errors are 1 and 2. Add them to obtain 3. There are two prediction–target pairs, so divide by 2.
","![\"\\mathrm{MAE}=\\frac{|1-2|+|6-4|}{2}=\\frac{1+2}{2}=1.5\"](\"assets/equations/506b26dfbefb2185.svg\")
Final answer: MAE = 1.5. Do not square the errors for MAE.
"]},{"n":14,"title":"Binary cross entropy","intro":["Binary cross-entropy measures how much probability was assigned to the correct binary answer. A target of 1 keeps the −ln(p) term; a target of 0 keeps the −ln(1−p) term. The factors y and 1−y select the appropriate term.
"],"formulas":["Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.
","Target y = 1; p = P(y=1) = 0.8; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","Final answer: Loss ≈ 0.223143551 nats.
","Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.
","Target y = 0; p = P(y=1) = 0.8; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","Final answer: Loss ≈ 1.609437912 nats.
","Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.
","Target y = 1; p = P(y=1) = 0.5; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","Final answer: Loss ≈ 0.693147181 nats.
"]},{"n":15,"title":"Categorical cross entropy","intro":["For one-hot multiclass targets, only one target entry is 1. The other entries are zero, so the loss becomes minus the logarithm of the true class probability. The worked question first calculates every softmax probability, then selects the correct one.
"],"formulas":["![\"\\log_2q=\\frac{\\ln](\"assets/equations/945fabaced090f8f.svg\")
Delta is a short name for the loss derivative with respect to a neuron’s raw input z. For half squared loss, multiply prediction minus target by the activation’s derivative. The result is then multiplied by an incoming activation to obtain a weight gradient.
"],"formulas":["Some activation–loss pairs simplify together. Sigmoid with binary cross-entropy, and softmax with one-hot categorical cross-entropy, give prediction minus target at the raw-score level. This result already includes the activation’s effect.
"],"formulas":["Question. Softmax probabilities are [0.1, 0.8, 0.1] and the one-hot target is [0, 1, 0]. For categorical cross-entropy on this one example, find the derivatives with respect to all three logits.
","Softmax probabilities = [0.1,0.8,0.1]; target = [0,1,0].
","One-example categorical cross-entropy.
","Find all three logit derivatives.
","Softmax with categorical cross-entropy. For one example with one-hot target, differentiating softmax together with CCE simplifies the logit derivative to predicted probability minus target.
","zᵢ = raw score before softmax; pᵢ = class probability; yᵢ = matching target entry.
","![\"\\frac{\\partial](\"assets/equations/2f0494c85fef2342.svg\")
Yaad rakhne ka cue: Softmax + CCE together: p minus y. Extra sigmoid slope nahi.
","A logit is the raw score before softmax. Subtract each target entry from its matching probability entry.
","![\"\\frac{\\partial](\"assets/equations/b1d662617a540534.svg\")
![\"\\frac{\\partial](\"assets/equations/ed27397a5b5cfeec.svg\")
![\"\\frac{\\partial](\"assets/equations/64941f713e55940c.svg\")
Final answer: [0.1, −0.2, 0.1]. Do not multiply these by a separate sigmoid derivative or by p(1−p). A mean over several examples introduces the corresponding averaging factor.
","Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.
","p = 0.8; y = 1; single-example BCE; p = sigmoid(z).
","dL/dz, not dL/dp.
","Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.
","p is an activated probability; z is its raw score; the requested derivative is with respect to z.
","Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.
","Step 1: calculate the sigmoid slope.
","Step 2: differentiate BCE with respect to p.
","Step 3: multiply the two local derivatives.
","Final answer: dL/dz = -0.2. No further sigmoid derivative is needed.
","Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.
","p = 0.8; y = 0; single-example BCE; p = sigmoid(z).
","dL/dz, not dL/dp.
","Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.
","p is an activated probability; z is its raw score; the requested derivative is with respect to z.
","Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.
","Step 1: calculate the sigmoid slope.
","Step 2: differentiate BCE with respect to p.
","Step 3: multiply the two local derivatives.
","Final answer: dL/dz = 0.8. No further sigmoid derivative is needed.
"]},{"n":18,"title":"Hidden delta, weight gradient and bias gradient","intro":["A hidden neuron can influence several outputs. Add all those backward contributions before multiplying by its own activation derivative. For each connection, use the receiving neuron’s delta and the sending neuron’s activation. Keep the old weights unchanged until all gradients have been found.
"],"formulas":["![\"\\delta_j^h=f\\prime(z_j^h)\\sum_k](\"assets/equations/7a85cefe361f399b.svg\")
A gradient tells us the local direction in which loss rises. Gradient descent subtracts a small multiple of that gradient. Learning rate η controls the size of this correction. After the first update, calculate the next gradient at the new parameter.
"],"formulas":["![\"\\theta_{new}=\\theta-\\eta](\"assets/equations/84b9c6c86f29aa06.svg\")
![\"g=\\frac{\\partial](\"assets/equations/ac6a050d5a887603.svg\")
A batch is a group of training examples used together for one update. An epoch is one pass through the dataset. Averaging gradients and counting updates are different questions, so both receive complete worked examples below.
"],"formulas":["![\"g_{batch}=\\frac1B\\sum_{i=1}^{B}g_i\"](\"assets/equations/af742484e9a4fffc.svg\")
![\"U_{epoch}=\\left\\lceil\\frac](\"assets/equations/24d8d40909055c78.svg\")
Momentum carries some of the previous movement into the next update. Here v is signed movement, so the new parameter is the old parameter plus v. A negative v moves the parameter down. Use the same convention throughout the calculation.
"],"formulas":["Nesterov first uses the old movement to choose a temporary look-ahead position. It measures the gradient there, calculates the new movement, then adds that movement to the original parameter. The temporary look-ahead position does not replace the original parameter in the final addition.
"],"formulas":["![\"\\theta_{look}=\\theta+\\beta](\"assets/equations/47358875d94f2558.svg\")
![\"v_{new}=\\beta](\"assets/equations/a9d756d38e5b7933.svg\")
![\"\\theta_{new}=\\theta+v_{new}\"](\"assets/equations/73afc93b70062b2d.svg\")
AdaGrad stores a running sum of squared gradients for each parameter. Squaring removes the sign only in this stored total. The unsquared gradient in the numerator still decides the direction of the update.
"],"formulas":["RMSProp remembers a weighted average of squared gradients. With decay 0.9, keep 90% of the old stored value and add 10% of the new squared gradient. Calculate this new stored value before taking its square root and updating the parameter.
"],"formulas":["Adam keeps two separate running averages. The first remembers signed gradients and can be negative. The second remembers squared gradients and is nonnegative. These stored averages are not weights, and they are not extra gradients to multiply together.
"],"formulas":["Adam starts its two averages at zero. Early averages therefore contain less total weight on observed gradients. Bias correction compensates for that initialization. The power in βᵗ uses the current update number, and both averages must be corrected before the parameter update.
"],"formulas":["![\"\\widehat](\"assets/equations/a9f81435f55f92a6.svg\")
![\"\\theta_t=\\theta_{t-1}-\\eta\\frac{\\widehat](\"assets/equations/8fbedcf992dd719a.svg\")
L1 adds a cost based on the absolute sizes of weights. Its derivative is positive for a positive weight and negative for a negative weight. The optimizer adds this penalty gradient to the data gradient before applying the usual update.
"],"formulas":["![\"J=L+\\lambda\\sum_j|w_j|\"](\"assets/equations/14b76928315b8569.svg\")
![\"w_{new}=w-\\eta\\left[g_{data}+\\lambda\\operatorname{sign}(w)\\right]\"](\"assets/equations/e25593d3f219dc71.svg\")
L2 adds a cost based on squared weights. Differentiating a square produces a factor of 2. If the objective includes a half, that half cancels the 2; if it does not, the 2 remains. Read the objective before choosing the gradient.
"],"formulas":["![\"J=L+\\frac\\lambda2\\sum_jw_j^2\"](\"assets/equations/119bfafa31acc3ba.svg\")
![\"w_{new}=w-\\eta(g_{data}+\\lambda](\"assets/equations/349319f842d12440.svg\")
Inverted dropout uses a mask: zero means drop this activation, one means keep it. During training, divide each retained activation by the keep probability. During inference, dropout is disabled and this training-time division is not repeated.
"],"formulas":["![\"q=1-p\"](\"assets/equations/536be85758e6d02d.svg\")
![\"a_{train}=\\frac{m\\odot](\"assets/equations/504d04d32df8a91c.svg\")
![\"a_{test}=a\"](\"assets/equations/e1251aa7fa418d32.svg\")
This formula counts how many legal filter positions fit along one image dimension. Padding adds space at both ends, the kernel occupies K positions, and stride controls how far the next window moves. The floor keeps only complete valid positions.
"],"formulas":["One convolution output is a sum of matched input–kernel products. Move the window and repeat that exact rule for the next output. With several input channels, add contributions from every channel and add the filter bias once.
"],"formulas":["Pooling reduces each window to one value. Max pooling keeps the largest entry; average pooling adds all included entries and divides by their count. Pooling is performed independently in each channel.
"],"formulas":["![\"y_{max}=\\max_{(i,j)\\in](\"assets/equations/79973536a2057db8.svg\")
![\"y_{avg}=\\frac1{|W|}\\sum_{(i,j)\\in](\"assets/equations/bd2034a9c3d7a387.svg\")
A filter has one learned weight for each spatial position in each input channel. Count the weights in one filter, add its one bias, then multiply by the number of filters. Sliding a filter to many positions reuses these weights; it does not create new parameters.
"],"formulas":["Flattening rearranges all activation values into a list. It learns no new numbers. A dense layer then connects every value in that list to every receiving neuron; each receiving neuron also has one bias.
"],"formulas":["![\"N_{flat}=HWC\"](\"assets/equations/31ebc79afc790262.svg\")
![\"P_{dense}=(N_{in}+1)N_{out}\"](\"assets/equations/c28314915c655018.svg\")
A parameter count counts learned numbers. A multiply-accumulate count counts how many times input–weight products are used. One filter is reused at every output position, so computation includes output height and width even though its parameter count does not.
"],"formulas":["Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.
","Spatial output = 8 × 8 in both stages.
","Input channels = 16; output filters = 8; main kernel = 3 × 3.
","Bottleneck uses 4 filters of 1 × 1.
","Direct MACs, two-stage bottleneck MACs and savings.
","Each output scalar requires one multiply-accumulate per incoming kernel entry. A bottleneck reduces the input depth of the expensive spatial convolution.
","One MAC is one multiply-accumulate pair. If counting multiplication and addition separately, the operation count is approximately twice the MAC count.
","![\"\\mathrm{MACs}=H_{\\mathrm{out}}W_{\\mathrm{out}}C_{\\mathrm{out}}K_hK_wC_{\\mathrm{in}}\"](\"assets/equations/40f8ac20e40ebe3e.svg\")
Yaad rakhne ka cue: Spatial kernel se pehle channels kam; dono stages ka cost add.
","Step 1: count output positions per channel.
","![\"8(8)=64\"](\"assets/equations/0917529398667f13.svg\")
Step 2: count direct work per output scalar.
","![\"3(3)(16)=144\"](\"assets/equations/7b0c2720e21a1506.svg\")
Step 3: count the full direct layer.
","![\"64(8)(144)=73728\"](\"assets/equations/531722215214eb67.svg\")
Step 4: count the 1 × 1 reduction.
","![\"64(4)(1)(1)(16)=4096\"](\"assets/equations/a619ccae82b20a14.svg\")
Step 5: count the reduced-depth 3 × 3 layer.
","![\"64(8)(3)(3)(4)=18432\"](\"assets/equations/f478849ec0c64569.svg\")
Step 6: sum both bottleneck stages.
","Step 7: subtract to obtain savings.
","Final answer: Direct = 73,728 MACs; bottleneck = 22,528 MACs; savings = 51,200 MACs.
"]},{"n":36,"title":"Concatenation and residual addition","intro":["Concatenation and addition combine branches differently. Concatenation places channel groups next to each other and increases the channel count. Residual addition adds matching entries, so the two full shapes must be compatible.
"],"formulas":["![\"C_{concat}=\\sum_b](\"assets/equations/963754238ceb3b1d.svg\")
![\"y_{res}=F(x)+x\"](\"assets/equations/d362a7b729c4f87f.svg\")
Batch normalization first uses batch statistics, then applies a learned scale and shift. Epsilon belongs inside the square root. Inference ordinarily uses stored running statistics.
"],"formulas":["Global average pooling takes one mean per complete spatial channel. The worked example compares the resulting dense classifier with a classifier after flattening.
"],"formulas":["Top-k error counts examples whose correct class is absent from the highest-scoring k predictions. Success counts must not be added twice.
"],"formulas":["The mean is a measure of location; variance and standard deviation quantify dispersion. Distinguish population and sample conventions before calculating these statistics.
"],"formulas":["Question. For the population [1, 2, 3], find mean, population variance and standard deviation.
","Values = [1, 2, 3]; N = 3. Use population division by N.
","Mean μ, variance σ² and standard deviation σ.
","The question specifies a population, so use N rather than the N−1 sample-variance divisor.
","μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.
","Yaad rakhne ka cue: Mean, distance, square, mean again, then root.
","Step 1: add all values.
","Step 2: divide by the number of values.
","Step 3: find and square deviation 1.
","Step 4: find and square deviation 2.
","Step 5: find and square deviation 3.
","Step 6: average the squared deviations.
","Step 7: take the nonnegative square root.
","Final answer: μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581.
","Question. For the population [2, 2, 2], find mean, population variance and standard deviation.
","Values = [2, 2, 2]; N = 3. Use population division by N.
","Mean μ, variance σ² and standard deviation σ.
","The question specifies a population, so use N rather than the N−1 sample-variance divisor.
","μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units.
","Yaad rakhne ka cue: Mean, distance, square, mean again, then root.
","Step 1: add all values.
","Step 2: divide by the number of values.
","Step 3: find and square deviation 1.
","Step 4: find and square deviation 2.
","Step 5: find and square deviation 3.
","Step 6: average the squared deviations.
","Step 7: take the nonnegative square root.
","Final answer: μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation.
"]},{"n":2,"title":"Standardization","intro":["Standardization has a meaning before it has a formula: measure how far an input is from its training mean, then express that distance in standard-deviation units. The worked question below calculates the mean and standard deviation first, so neither number appears without explanation.
"],"formulas":["Derivatives describe local rates of change. Partial derivatives hold other independent variables fixed; their collection forms the gradient used for optimization.
"],"formulas":["The chain rule differentiates a composite function. Multiply local derivatives along a dependency path and add the contributions of different paths to the same variable.
"],"formulas":["A neuron starts with matched input–weight pairs. Multiply inside each pair, add the products, add the bias once, and then apply the activation. The summation sign is simply a shorter way to write those additions.
"],"formulas":["Matrix multiplication implements affine transformations in a dense layer. Dimensions determine which products are valid; each output component is the dot product of a weight row and the input vector, plus its bias.
"],"formulas":["A layer does the neuron calculation for several receiving neurons. Finish all its outputs before using them as the next layer’s inputs. The bracketed superscript is a layer label, not a power.
"],"formulas":["Sigmoid takes a raw score and changes it to a number between 0 and 1. Its derivative is a different number: it tells backpropagation how sensitive that output is to its raw score. The complete example below calculates both and shows where each is used.
"],"formulas":["Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.
","z = -2; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","Step 2: calculate the exponential.
","Step 3: add one in the denominator.
","Step 4: divide one by the complete denominator.
","Step 5: use the activated output in the slope.
","Final answer: a ≈ 0.119202922. Derivative ≈ 0.104993585.
","Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.
","z = 0; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","Step 2: calculate the exponential.
","Step 3: add one in the denominator.
","Step 4: divide one by the complete denominator.
","Step 5: use the activated output in the slope.
","Final answer: a ≈ 0.500000000. Derivative ≈ 0.250000000.
","Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.
","z = 2; activation = sigmoid.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: One over one plus e ki minus-z power; slope = output × (1−output).
","Step 1: negate the whole input.
","Step 2: calculate the exponential.
","Step 3: add one in the denominator.
","Step 4: divide one by the complete denominator.
","Step 5: use the activated output in the slope.
","Final answer: a ≈ 0.880797078. Derivative ≈ 0.104993585.
"]},{"n":9,"title":"Tanh and its derivative","intro":["Tanh can produce a negative or positive output. In the formula, calculate the exponential first, then form the top and bottom of the fraction separately. The derivative uses the activated answer, not the raw input.
"],"formulas":["Question. Calculate tanh(0.5) and its derivative with respect to the input.
","z = 0.5.
","Find tanh(z) and its derivative.
","Tanh and its slope. Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a².
","z = value entering tanh; a = its output. A prime means derivative, or local rate of change.
","Yaad rakhne ka cue: Tanh signed output deta hai; slope = 1 minus output ka square.
","Step 1: identify the input. The input z is 0.5. In this version of the tanh formula, the exponential uses 2z.
","Step 2: calculate the numerator. Subtract 1 from that exponential.
","Step 3: calculate the denominator. Add 1 to the same exponential.
","Step 4: divide the top by the bottom.
","Step 5: calculate the local derivative. Square the output a and subtract it from 1. Keep the calculator's unrounded result.
","Final answer: tanh output ≈ 0.462117157; local derivative ≈ 0.786447733. The output becomes the next layer's input; the derivative is used when carrying a gradient backward.
","Question. For preactivation z = -2, calculate the tanh output and state its derivative.
","z = -2; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","Step 4: divide.
","Step 5: square the activated output for its slope.
","Final answer: a ≈ -0.964027580. Derivative ≈ 0.070650825.
","Question. For preactivation z = 0, calculate the tanh output and state its derivative.
","z = 0; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","Step 4: divide.
","Step 5: square the activated output for its slope.
","Final answer: a ≈ 0.000000000. Derivative ≈ 1.000000000.
","Question. For preactivation z = 2, calculate the tanh output and state its derivative.
","z = 2; activation = tanh.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Tanh ka answer −1 se 1; slope one minus output squared.
","Step 1: double z before exponentiation.
","Step 2: calculate the exponential.
","Step 3: calculate numerator and denominator separately.
","Step 4: divide.
","Step 5: square the activated output for its slope.
","Final answer: a ≈ 0.964027580. Derivative ≈ 0.070650825.
"]},{"n":10,"title":"ReLU and its derivative","intro":["ReLU compares its input with zero and keeps the larger value. Thus a negative input becomes zero and a positive input passes unchanged. Its slope is zero on the negative side and one on the positive side.
"],"formulas":["Question. For preactivation z = -2, calculate the ReLU output and state its derivative.
","z = -2; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","-2 is negative, so use the negative branch.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 0.000000000. Derivative ≈ 0.000000000.
","Question. For preactivation z = 0, calculate the ReLU output and state its derivative.
","z = 0; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","Both branches meet at zero. The value is zero, but the left and right slopes differ.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.
","Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.
","Question. For preactivation z = 2, calculate the ReLU output and state its derivative.
","z = 2; activation = ReLU.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","2 is positive, so retain it.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.
"]},{"n":11,"title":"Leaky ReLU","intro":["Leaky ReLU changes only the negative side: multiply a negative input by the small slope α. That negative sign remains. The value α is supplied by the question or chosen as a model setting; it is not the sigmoid function.
"],"formulas":["Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = -2; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","-2 is negative, so use the negative branch.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ -0.200000000. Derivative ≈ 0.100000000.
","Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = 0; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","Both branches meet at zero. The value is zero, but the left and right slopes differ.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required.
","Final answer: a ≈ 0.000000000. Ordinary derivative is undefined at zero.
","Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.
","z = 2; activation = Leaky ReLU.
","α = 0.1.
","The output a and the derivative with respect to z.
","Use the activation named in the question. The derivative is a second calculation required for a backward pass.
","z is the raw sum; a is the activated output; f′(z) is the local slope.
","Yaad rakhne ka cue: Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho.
","Step 1: select the branch from the sign of z.
","2 is positive, so retain it.
","Step 2: calculate the output.
","Step 3: state the slope on that branch.
","Final answer: a ≈ 2.000000000. Derivative ≈ 1.000000000.
"]},{"n":12,"title":"Stable softmax","intro":["Softmax turns several competing class scores into probabilities that add to one. Exponentiate each score, add those exponential values, and divide each one by their common total. Subtracting the same maximum score first gives the same probabilities with safer arithmetic.
"],"formulas":["Question. Find stable softmax probabilities for logits [0, 0, 0].
","Logits = [0, 0, 0]; there are three mutually exclusive classes.
","All three class probabilities.
","Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.
","A logit is a raw class score. c is the largest score; pᵢ is probability for class i.
","Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.
","Step 1: subtract the largest score from every entry.
","Step 2: exponentiate shifted entry 1.
","Step 3: exponentiate shifted entry 2.
","Step 4: exponentiate shifted entry 3.
","Step 5: add all exponentials into one denominator.
","Step 6: divide entry 1 by the same denominator.
","Step 7: divide entry 2 by the same denominator.
","Step 8: divide entry 3 by the same denominator.
","Final answer: Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding.
","Question. Find stable softmax probabilities for logits [1, 2, 3].
","Logits = [1, 2, 3]; there are three mutually exclusive classes.
","All three class probabilities.
","Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum.
","A logit is a raw class score. c is the largest score; pᵢ is probability for class i.
","Yaad rakhne ka cue: Max minus, e-power, same denominator se sab divide.
","Step 1: subtract the largest score from every entry.
","Step 2: exponentiate shifted entry 1.
","Step 3: exponentiate shifted entry 2.
","Step 4: exponentiate shifted entry 3.
","Step 5: add all exponentials into one denominator.
","Step 6: divide entry 1 by the same denominator.
","Step 7: divide entry 2 by the same denominator.
","Step 8: divide entry 3 by the same denominator.
","Final answer: Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding.
"]},{"n":13,"title":"Squared and absolute losses","intro":["SSE, half SSE and MSE start with the same errors but apply different final divisions. First understand what an error and its square mean. The full question below keeps all three answers separate.
"],"formulas":["Question. Targets are [2, 4] and predictions are [1, 6]. Find mean absolute error.
","Targets = [2,4]; predictions = [1,6]; n = 2.
","Find mean absolute error.
","Mean absolute error. Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes.
","n = number of predictions. |e| means the nonnegative magnitude of error e.
","Yaad rakhne ka cue: MAE mein magnitude; MSE mein square.
","The signed errors are 1 − 2 = −1 and 6 − 4 = 2. Absolute value keeps the size of a number without its sign, so their absolute errors are 1 and 2. Add them to obtain 3. There are two prediction–target pairs, so divide by 2.
","Final answer: MAE = 1.5. Do not square the errors for MAE.
"]},{"n":14,"title":"Binary cross entropy","intro":["Binary cross-entropy measures how much probability was assigned to the correct binary answer. A target of 1 keeps the −ln(p) term; a target of 0 keeps the −ln(1−p) term. The factors y and 1−y select the appropriate term.
"],"formulas":["Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.
","Target y = 1; p = P(y=1) = 0.8; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","Final answer: Loss ≈ 0.223143551 nats.
","Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.
","Target y = 0; p = P(y=1) = 0.8; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","Final answer: Loss ≈ 1.609437912 nats.
","Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.
","Target y = 1; p = P(y=1) = 0.5; use natural log ln.
","The binary cross-entropy loss for this one example.
","The target selects the probability assigned to the correct class. Take its negative natural logarithm.
","p is always the probability of class 1. When y = 0, the correct-class probability is 1−p.
","Yaad rakhne ka cue: True class ki probability lo, uska minus ln lo.
","Step 1: identify the correct-class probability.
","Step 2: evaluate its natural logarithm.
","Step 3: apply the minus sign outside the logarithm.
","Final answer: Loss ≈ 0.693147181 nats.
"]},{"n":15,"title":"Categorical cross entropy","intro":["For one-hot multiclass targets, only one target entry is 1. The other entries are zero, so the loss becomes minus the logarithm of the true class probability. The worked question first calculates every softmax probability, then selects the correct one.
"],"formulas":["Delta is a short name for the loss derivative with respect to a neuron’s raw input z. For half squared loss, multiply prediction minus target by the activation’s derivative. The result is then multiplied by an incoming activation to obtain a weight gradient.
"],"formulas":["Some activation–loss pairs simplify together. Sigmoid with binary cross-entropy, and softmax with one-hot categorical cross-entropy, give prediction minus target at the raw-score level. This result already includes the activation’s effect.
"],"formulas":["Question. Softmax probabilities are [0.1, 0.8, 0.1] and the one-hot target is [0, 1, 0]. For categorical cross-entropy on this one example, find the derivatives with respect to all three logits.
","Softmax probabilities = [0.1,0.8,0.1]; target = [0,1,0].
","One-example categorical cross-entropy.
","Find all three logit derivatives.
","Softmax with categorical cross-entropy. For one example with one-hot target, differentiating softmax together with CCE simplifies the logit derivative to predicted probability minus target.
","zᵢ = raw score before softmax; pᵢ = class probability; yᵢ = matching target entry.
","Yaad rakhne ka cue: Softmax + CCE together: p minus y. Extra sigmoid slope nahi.
","A logit is the raw score before softmax. Subtract each target entry from its matching probability entry.
","Final answer: [0.1, −0.2, 0.1]. Do not multiply these by a separate sigmoid derivative or by p(1−p). A mean over several examples introduces the corresponding averaging factor.
","Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.
","p = 0.8; y = 1; single-example BCE; p = sigmoid(z).
","dL/dz, not dL/dp.
","Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.
","p is an activated probability; z is its raw score; the requested derivative is with respect to z.
","Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.
","Step 1: calculate the sigmoid slope.
","Step 2: differentiate BCE with respect to p.
","Step 3: multiply the two local derivatives.
","Final answer: dL/dz = -0.2. No further sigmoid derivative is needed.
","Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.
","p = 0.8; y = 0; single-example BCE; p = sigmoid(z).
","dL/dz, not dL/dp.
","Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y.
","p is an activated probability; z is its raw score; the requested derivative is with respect to z.
","Yaad rakhne ka cue: BCE + sigmoid together ka shortcut p−y; chain already included.
","Step 1: calculate the sigmoid slope.
","Step 2: differentiate BCE with respect to p.
","Step 3: multiply the two local derivatives.
","Final answer: dL/dz = 0.8. No further sigmoid derivative is needed.
"]},{"n":18,"title":"Hidden delta, weight gradient and bias gradient","intro":["A hidden neuron can influence several outputs. Add all those backward contributions before multiplying by its own activation derivative. For each connection, use the receiving neuron’s delta and the sending neuron’s activation. Keep the old weights unchanged until all gradients have been found.
"],"formulas":["A gradient tells us the local direction in which loss rises. Gradient descent subtracts a small multiple of that gradient. Learning rate η controls the size of this correction. After the first update, calculate the next gradient at the new parameter.
"],"formulas":["A batch is a group of training examples used together for one update. An epoch is one pass through the dataset. Averaging gradients and counting updates are different questions, so both receive complete worked examples below.
"],"formulas":["Momentum carries some of the previous movement into the next update. Here v is signed movement, so the new parameter is the old parameter plus v. A negative v moves the parameter down. Use the same convention throughout the calculation.
"],"formulas":["Nesterov first uses the old movement to choose a temporary look-ahead position. It measures the gradient there, calculates the new movement, then adds that movement to the original parameter. The temporary look-ahead position does not replace the original parameter in the final addition.
"],"formulas":["AdaGrad stores a running sum of squared gradients for each parameter. Squaring removes the sign only in this stored total. The unsquared gradient in the numerator still decides the direction of the update.
"],"formulas":["RMSProp remembers a weighted average of squared gradients. With decay 0.9, keep 90% of the old stored value and add 10% of the new squared gradient. Calculate this new stored value before taking its square root and updating the parameter.
"],"formulas":["Adam keeps two separate running averages. The first remembers signed gradients and can be negative. The second remembers squared gradients and is nonnegative. These stored averages are not weights, and they are not extra gradients to multiply together.
"],"formulas":["Adam starts its two averages at zero. Early averages therefore contain less total weight on observed gradients. Bias correction compensates for that initialization. The power in βᵗ uses the current update number, and both averages must be corrected before the parameter update.
"],"formulas":["L1 adds a cost based on the absolute sizes of weights. Its derivative is positive for a positive weight and negative for a negative weight. The optimizer adds this penalty gradient to the data gradient before applying the usual update.
"],"formulas":["L2 adds a cost based on squared weights. Differentiating a square produces a factor of 2. If the objective includes a half, that half cancels the 2; if it does not, the 2 remains. Read the objective before choosing the gradient.
"],"formulas":["Inverted dropout uses a mask: zero means drop this activation, one means keep it. During training, divide each retained activation by the keep probability. During inference, dropout is disabled and this training-time division is not repeated.
"],"formulas":["This formula counts how many legal filter positions fit along one image dimension. Padding adds space at both ends, the kernel occupies K positions, and stride controls how far the next window moves. The floor keeps only complete valid positions.
"],"formulas":["One convolution output is a sum of matched input–kernel products. Move the window and repeat that exact rule for the next output. With several input channels, add contributions from every channel and add the filter bias once.
"],"formulas":["Pooling reduces each window to one value. Max pooling keeps the largest entry; average pooling adds all included entries and divides by their count. Pooling is performed independently in each channel.
"],"formulas":["A filter has one learned weight for each spatial position in each input channel. Count the weights in one filter, add its one bias, then multiply by the number of filters. Sliding a filter to many positions reuses these weights; it does not create new parameters.
"],"formulas":["Flattening rearranges all activation values into a list. It learns no new numbers. A dense layer then connects every value in that list to every receiving neuron; each receiving neuron also has one bias.
"],"formulas":["A parameter count counts learned numbers. A multiply-accumulate count counts how many times input–weight products are used. One filter is reused at every output position, so computation includes output height and width even though its parameter count does not.
"],"formulas":["Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.
","Spatial output = 8 × 8 in both stages.
","Input channels = 16; output filters = 8; main kernel = 3 × 3.
","Bottleneck uses 4 filters of 1 × 1.
","Direct MACs, two-stage bottleneck MACs and savings.
","Each output scalar requires one multiply-accumulate per incoming kernel entry. A bottleneck reduces the input depth of the expensive spatial convolution.
","One MAC is one multiply-accumulate pair. If counting multiplication and addition separately, the operation count is approximately twice the MAC count.
","Yaad rakhne ka cue: Spatial kernel se pehle channels kam; dono stages ka cost add.
","Step 1: count output positions per channel.
","Step 2: count direct work per output scalar.
","Step 3: count the full direct layer.
","Step 4: count the 1 × 1 reduction.
","Step 5: count the reduced-depth 3 × 3 layer.
","Step 6: sum both bottleneck stages.
","Step 7: subtract to obtain savings.
","Final answer: Direct = 73,728 MACs; bottleneck = 22,528 MACs; savings = 51,200 MACs.
"]},{"n":36,"title":"Concatenation and residual addition","intro":["Concatenation and addition combine branches differently. Concatenation places channel groups next to each other and increases the channel count. Residual addition adds matching entries, so the two full shapes must be compatible.
"],"formulas":["A recurrent state uses two sources: the current input and the previous hidden state. Transform each with its own weight, add the contributions and bias, and apply tanh. The activated state—not its raw sum—is carried into the next time step.
"],"formulas":["![\"o_t=W_{hy}h_t+b_y\"](\"assets/equations/d195662850af1e7e.svg\")
The perceptron uses a threshold to choose a class and an error-based learning rule to change weights. Its threshold convention must be stated, especially at zero. This discrete learning rule is different from sigmoid backpropagation.
"],"formulas":["![\"\\widehat](\"assets/equations/9c6e513ba240f8dc.svg\")
![\"w_{new}=w+\\eta(y-\\widehat](\"assets/equations/82b6f20b5710456c.svg\")
![\"b_{new}=b+\\eta(y-\\widehat](\"assets/equations/a75f55d01930a8a4.svg\")
At an RNN hidden state, a loss gradient can arrive from the current output and from the next time step. Add these two contributions, then multiply by the local activation derivative. The displayed output term p−y assumes softmax with categorical cross-entropy.
"],"formulas":["Question. In a scalar tanh RNN, the current hidden state is hₜ = 0.6. The current output path contributes 0.1 to the derivative with respect to hₜ. The next hidden preactivation has delta 0.4 and recurrent weight 0.5. Find the current hidden preactivation delta.
","Current hₜ = 0.6; local output-path derivative = 0.1.
","Next-step delta = 0.4; recurrent weight = 0.5; current activation = tanh.
","Find the current hidden preactivation delta.
","Current and future recurrent paths. Add the local-output contribution and the weighted next-step delta, then multiply by the current tanh slope.
","hₜ = current activated state; qₜ = current output-path derivative; u = recurrent weight; δₜ₊₁ = next-step preactivation delta.
","Yaad rakhne ka cue: Current path plus future path, phir current activation slope.
","The next-step contribution must travel through the recurrent weight. Multiply that weight by the supplied next delta.
","![\"\\text{future](\"assets/equations/76c508722fc577c7.svg\")
Add the current output-path contribution. These are two routes to the same hidden state, so addition is required.
","![\"\\frac{\\partial](\"assets/equations/6685e4ec204aeb2d.svg\")
The current activation is tanh. Its derivative uses the current activated value hₜ.
","![\"1-h_t^2=1-(0.6)^2=1-0.36=0.64\"](\"assets/equations/dca805bc28733c27.svg\")
Multiply the combined gradient by this local derivative.
","![\"\\delta_t=0.3(0.64)=0.192\"](\"assets/equations/4f408ded05ecb892.svg\")
Final answer: δₜ = 0.192. This is the derivative with respect to the current hidden raw score. At the last step, there is no future-state contribution unless the question connects another computation there.
","Question. In a scalar tanh RNN, h_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.
","Activated h_t = 0.5; q_t = 0.2; δ_(t+1) = 0.3; recurrent weight u = 0.4.
","Current preactivation delta δ_t.
","Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.
","q_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.
","Yaad rakhne ka cue: Future aur current dono add; tanh slope uske baad.
","Step 1: calculate the future path contribution.
","Step 2: add the current output contribution.
","Step 3: calculate the current tanh slope.
","Step 4: multiply by that slope.
","Final answer: δ_t = 0.24.
","Question. In a scalar tanh RNN, h_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.
","Activated h_t = -0.2; q_t = 0.1; δ_(t+1) = 0; recurrent weight u = 0.5.
","Current preactivation delta δ_t.
","Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh.
","q_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation.
","Yaad rakhne ka cue: Future aur current dono add; tanh slope uske baad.
","Step 1: calculate the future path contribution.
","Step 2: add the current output contribution.
","Step 3: calculate the current tanh slope.
","Step 4: multiply by that slope.
","Final answer: δ_t = 0.096.
"]},{"n":40,"title":"BPTT recurrent-weight gradient","intro":["The same recurrent weight is reused at several times. Each use contributes to its gradient, so sum all required time-step contributions before updating it. For vectors, delta is a column and the previous state is transposed into a row: their outer product has the same shape as the weight matrix.
"],"formulas":["![\"\\frac{\\partial](\"assets/equations/0db4b816627efa5e.svg\")
![\"\\frac{\\partial](\"assets/equations/4953486e2063f6f0.svg\")
An LSTM gate is a multiplier between zero and one. The forget gate chooses the fraction of old cell content retained; the input gate chooses the fraction of candidate content admitted. Each gate is calculated from its own weighted sum and sigmoid.
"],"formulas":["![\"f_t=\\sigma(W_f[h_{t-1},x_t]+b_f)\"](\"assets/equations/42d259e3cca6ee0a.svg\")
![\"i_t=\\sigma(W_i[h_{t-1},x_t]+b_i)\"](\"assets/equations/48984d6eb2de43b5.svg\")
The candidate is proposed new cell content, so it may be negative. The output gate is a positive sigmoid multiplier that controls how much of the activated cell is exposed. Candidate content and gate values therefore have different roles.
"],"formulas":["![\"\\widetilde](\"assets/equations/6c797c7201f58bbb.svg\")
![\"o_t=\\sigma(W_o[h_{t-1},x_t]+b_o)\"](\"assets/equations/e82eb2b55af9249e.svg\")
First calculate the retained old contribution. Separately calculate the admitted new contribution. Add them to obtain the cell state. Only afterward apply tanh and the output gate to obtain the hidden state.
"],"formulas":["A GRU has a reset gate and an update gate. Reset changes how earlier state enters the candidate calculation; update controls the final old/new mixture. The numerical below states its convention explicitly: z is the fraction of the new candidate admitted.
"],"formulas":["Some GRU questions use the opposite naming convention: z is the fraction of the OLD state retained. The formula below uses that convention. Do not combine its formula with an explanation that says z retains the new candidate.
"],"formulas":["![\"h_t=z_t\\odot](\"assets/equations/fa76004105af78bd.svg\")
Question. Use the retain-old GRU convention. Given hprevious = 0.4, reset gate r = 0.5, update gate z = 0.25, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and the next hidden state.
","h_old = 0.4; r = 0.5; z = 0.25; x = 1.
","W = 0.2; U = 0.5; b = 0; z retains the old state.
","Find the candidate and new hidden state.
","GRU with a retain-old update gate. Form the candidate using reset memory; then z retains the old state and 1−z admits the candidate. Follow this question-specific convention.
","r = reset gate; z = retain-old gate; h_old = previous state; h̃ = candidate.
","Yaad rakhne ka cue: Question mein z kisko multiply karta hai wahi meaning; yahan old ko retain.
","Step 1: reset the old state for candidate formation.
","![\"r](\"assets/equations/b5a08a736e0f6884.svg\")
Step 2: calculate the candidate's raw input. Its new-input contribution is 0.2 × 1; its reset-memory contribution is 0.5 × 0.2.
","![\"a=0.2(1)+0.5(0.2)+0=0.2+0.1=0.3\"](\"assets/equations/ed7061874c7df25f.svg\")
Step 3: apply tanh to this raw input.
","Step 4: calculate how much old state is retained. In this question, z = 0.25 multiplies the original old state 0.4.
","![\"z](\"assets/equations/62b46633101b4f4e.svg\")
Step 5: calculate how much candidate is admitted. The other fraction is 1 − z = 0.75.
","![\"](\"assets/equations/f5dda0f46575aa22.svg\")
Step 6: add the two final contributions.
","![\"h=0.1+0.218484459=0.318484459\"](\"assets/equations/b283dd6b6dd4cf79.svg\")
Final answer: h ≈ 0.318484459. The candidate was 0.291312612; the final hidden state is a mixture of that candidate and the original old state. Under the admit-new convention with the same numeric z, the result would instead be 0.372828153.
","Question. Use h_new = z h_old+(1−z) candidate. Given h_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.
","h_old = 0.4; z = 0; r = 0.5; x = 1.
","W = 0.2; U = 0.5; b = 0; z is the retain-old gate.
","Candidate and final hidden state.
","The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.
","The question explicitly makes z multiply the old state. r and z do different jobs.
","Yaad rakhne ka cue: Reset sirf candidate mein; update mix mein original old state.
","Step 1: reset the old state for candidate formation.
","Step 2: calculate the candidate raw sum.
","Step 3: apply tanh.
","Step 4: retain the old-state fraction.
","Step 5: calculate the candidate fraction.
","Step 6: add the two fractions.
","Final answer: Candidate ≈ 0.291312612; new state ≈ 0.291312612.
","Question. Use h_new = z h_old+(1−z) candidate. Given h_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.
","h_old = -0.2; z = 0.8; r = 0.5; x = 1.
","W = 0.2; U = 0.5; b = 0; z is the retain-old gate.
","Candidate and final hidden state.
","The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate.
","The question explicitly makes z multiply the old state. r and z do different jobs.
","Yaad rakhne ka cue: Reset sirf candidate mein; update mix mein original old state.
","Step 1: reset the old state for candidate formation.
","Step 2: calculate the candidate raw sum.
","Step 3: apply tanh.
","Step 4: retain the old-state fraction.
","Step 5: calculate the candidate fraction.
","Step 6: add the two fractions.
","Final answer: Candidate ≈ 0.148885034; new state ≈ -0.130222993.
"]},{"n":46,"title":"Batch normalization and learned scale/shift","intro":["Batch normalization first uses batch statistics, then applies a learned scale and shift. Epsilon belongs inside the square root. Inference ordinarily uses stored running statistics.
"],"formulas":["Global average pooling takes one mean per complete spatial channel. The worked example compares the resulting dense classifier with a classifier after flattening.
"],"formulas":["Top-k error counts examples whose correct class is absent from the highest-scoring k predictions. Success counts must not be added twice.
"],"formulas":["Q1a compulsory (2 marks). Explain the key difference between the AdaGrad and RMSProp accumulators. Why can AdaGrad's effective learning rate become very small during prolonged training?
"],"answer":["Let gₜ be the current gradient and Gₜ the stored sum of squared gradients. AdaGrad updates that sum as follows:
","Every added square is nonnegative, so G cannot decrease. All past squared gradients remain in the sum.
","Let sₜ be the stored squared-gradient average and ρ the fraction of the previous average retained.
","RMSProp gradually reduces the influence of older squared gradients. Its accumulator can rise or fall, depending on the new squared gradient.
","AdaGrad's effective learning rate is:
","Here η is the base learning rate and ε is a small positive number that stabilizes division. When G becomes large, its square root becomes large, so dividing by it produces a smaller effective learning rate. RMSProp's decaying average avoids this particular permanent-accumulation problem.
","Final answer: AdaGrad uses a nondecreasing sum of all squared gradients; RMSProp uses a decaying average. AdaGrad's growing denominator can make future updates very small.
","Yaad rakho: AdaGrad sab squares jodta rehta hai; RMSProp purane squares ka asar dheere kam karta hai.
"]},{"title":"Q1b","question":["Q1b compulsory (2 marks). Explain local connectivity and parameter sharing in a CNN. How do they reduce the number of learned parameters compared with a fully connected mapping of an image?
"],"answer":["Each convolutional output depends on a small spatial neighbourhood of the input, called its receptive field, rather than every pixel in the entire image. The filter spans the input channels within that neighbourhood. This limits the connections used to calculate each output.
","The same filter weights and bias are reused at every spatial position where that filter is applied. A CNN therefore learns one set of parameters for that filter, rather than a separate set for each image location.
","An edge-detecting filter can identify a vertical boundary near the left of an image and then slide to look for a similar boundary near the right. Both locations use the same learned filter. The network does not need to learn an independent edge detector for every position.
","A fully connected layer generally learns a separate weight for every input-to-output connection. Convolution reduces this requirement in two ways: local connectivity restricts each output's input region, while parameter sharing reuses one filter across locations. More output positions create more activations and computation, but do not create new parameters for that shared filter.
","Final answer: local connectivity uses a limited input neighbourhood; parameter sharing reuses the filter's learned values across spatial positions. Together they can require far fewer parameters than a fully connected image mapping.
","Yaad rakho: Filter ke numbers ek baar seekhte hain; image par baar-baar wahi use hote hain.
"]},{"title":"Q2","question":["Q2 optional (3 marks). An input of size 7 × 7 × 3 passes through a convolutional layer with two 3 × 3 filters, stride 2 and padding 1 on each side; then a 2 × 2 pooling layer with stride 2 and no padding; then flattening; then a dense layer with three output units. Find the output shape after every layer and the total trainable parameters. Include one bias per convolutional filter and per dense unit.
"],"answer":["- Input: height 7, width 7, channels 3.
","- Convolution: kernel 3 × 3, filters 2, stride 2, padding 1 per side.
","- Pooling: window 2 × 2, stride 2, padding 0.
","- Flattening followed by 3 dense output units.
","- Include all specified biases. Pooling and flattening have no learned parameters.
","The convolution output, pooling output, flatten length, dense output length and total trainable parameter count.
","For either spatial axis, use the output-size formula. N is that axis's input size, K the window size, S its stride and P padding per side. The floor means discard any fractional part because a full window must fit.
","Each convolution filter spans all input channels; each dense unit connects to every flattened input. Therefore:
","Flattening simply counts all values in the preceding feature maps:
","The numerator is 7 + 2 − 3 = 6. Divide by stride 2 to get 3, then add 1 to get 4 positions. The same calculation applies to height and width. Two filters create two output channels.
","Convolution output: 4 × 4 × 2.
","The pooling input size is now 4, not the original 7.
","Pooling operates separately on each channel, so the two channels remain two.
","Pooling output: 2 × 2 × 2. Pooling parameters: 0.
","There are eight values to send to the dense layer. Flattening rearranges them into a list and learns no parameters.
","Flatten output: 8 values. Flatten parameters: 0.
","Each of the three dense units receives eight inputs and has one bias.
","Dense output: 3 values.
","Final answer: 4 × 4 × 2 → 2 × 2 × 2 → 8 → 3. Total: 83 trainable parameters.
","Yaad rakho: Pehle shape; phir weights. Ek layer ka output hi next layer ka input hai.
"]},{"title":"Q3","question":["Q3 optional (3 marks). Apply the printed kernel K to input I using the deep-learning cross-correlation convention, with stride 1, no padding and zero bias. Do not flip K. Apply ReLU to the resulting map, followed by 2 × 2 max pooling with stride 2 and no padding. Show every convolution entry and find the final output.
","- Input I is the printed 3 × 3 matrix; kernel K is the printed 2 × 2 matrix.
","- Convolution stride 1, padding 0, bias 0; use K in its printed orientation.
","- Activation: ReLU.
","- Pooling: max, 2 × 2 window, stride 2, padding 0.
","All four convolution values, the ReLU output matrix and the final pooled output.
","For each input window, multiply entries at matching positions, then add all products and the bias once.
","X is the input, S is stride, u and v select a kernel entry, and b is bias. This formula uses zero-based array indices; the written solution labels positions from 1 for readability.
","ReLU is required because the question requests activation after convolution. It replaces a negative value by zero.
","Max pooling selects the largest value in its current window V.
","Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs.
","For each axis: (3 − 2)/1 + 1 = 2. Therefore calculate a 2 × 2 output matrix.
","The window is .
Multiply matching cells: 1 × 1 = 1; 2 × (−1) = −2; 0 × 0 = 0; 1 × 1 = 1.
","Add the products and bias: 1 − 2 + 0 + 1 + 0 = 0.
","Move the window one column to the right. The window is .
Multiply matching cells: 2 × 1 = 2; 1 × (−1) = −1; 1 × 0 = 0; 3 × 1 = 3.
","Add: 2 − 1 + 0 + 3 + 0 = 4.
","The window is .
Multiply matching cells: 0 × 1 = 0; 1 × (−1) = −1; 2 × 0 = 0; 1 × 1 = 1.
","Add: 0 − 1 + 0 + 1 + 0 = 0.
","The window is .
Multiply matching cells: 1 × 1 = 1; 3 × (−1) = −3; 1 × 0 = 0; 0 × 1 = 0.
","Add: 1 − 3 + 0 + 0 + 0 = −2.
","Convolution map: .
- max(0,0) = 0.
","- max(0,4) = 4.
","- max(0,0) = 0.
","- max(0,−2) = 0.
","ReLU map: .
The 2 × 2 pooling window covers all four activated entries. Compare 0, 4, 0 and 0. Their maximum is 4.
","Final answer: convolution → ReLU → pooled output , a 1 × 1 map.
Yaad rakho: Multiply-add se convolution; negative hatao ReLU; biggest uthao max pool.
"]},{"title":"Q4","question":["Q4 optional (3 marks). What is overfitting? Explain how dropout and early stopping help improve generalization. State which dataset should select the stopping point.
"],"answer":["Overfitting occurs when a model learns training-specific details or noise that do not generalize to new data. It commonly shows low training error together with substantially higher validation error. Generalization means performing well on new examples drawn from the task's data distribution.
","During training, dropout randomly sets selected activations to zero. The active subnetwork changes across training passes, discouraging reliance on a fixed combination of hidden units. Under inverted dropout, retained activations are divided by the keep probability to preserve their expected scale. During ordinary inference, the masking is disabled and all units are used without that training-time rescaling.
","Dropout is a regularizer; it can reduce overfitting, but excessive dropout can cause underfitting.
","Train on the training set and monitor performance on a separate validation set. Save the model whenever validation loss improves. Stop after the chosen number of consecutive non-improving evaluations, called patience, then restore the saved best model.
","For example, validation losses 0.8, 0.6, 0.5, 0.53 and 0.55 have their minimum at epoch 3. With patience 2, stop after epoch 5 and restore epoch 3. The stopping epoch and saved-model epoch are different.
","Final answer: dropout restricts dependence on particular training-time activations; early stopping limits training after validation performance stops improving. Choose the stopping point using validation data and reserve the test set for final evaluation.
","Yaad rakho: Training se seekho, validation se best model chuno, test se final result batao.
"]}],"B":[{"title":"Q1a","question":["Q1a compulsory (2 marks). Explain the purpose of scaling retained activations in inverted dropout. For activations [1,2,3,4], keep probability 0.5 and mask [1,0,1,0], calculate the activations used during this training pass. A mask value of 1 means keep and 0 means drop.
"],"answer":["Randomly dropping units would reduce the average activation. Inverted dropout divides every retained activation by the keep probability. Averaged over the random masks, this preserves the original activation's expected value. A dropped activation is still zero.
","- Original activations a = [1,2,3,4].
","- Keep probability q = 0.5, meaning a unit has a 50% chance of being kept.
","- This pass's supplied mask m = [1,0,1,0].
","The four post-dropout activations used in this training pass.
","Multiply the original activation aᵢ by its mask mᵢ, then divide by keep probability q.
","Mask 1 keeps the value. Multiply 1 × 1 = 1; divide 1 by 0.5 to get 2.
","Mask 0 drops the value. Multiply 0 × 2 = 0; divide 0 by 0.5 to get 0.
","Mask 1 keeps the value. Multiply 1 × 3 = 3; divide 3 by 0.5 to get 6.
","Mask 0 drops the value. Multiply 0 × 4 = 0; divide 0 by 0.5 to get 0.
","Final answer: [2,0,6,0]. This is the training-pass result. In ordinary inference with inverted dropout, disable the mask and its scaling.
","Yaad rakho: Mask lagao; bache hue values ko keep probability se divide karo.
"]},{"title":"Q1b","question":["Q1b compulsory (2 marks). Explain a ResNet shortcut. Why must the two tensors being added have matching shapes, and what can be done if their shapes differ?
"],"answer":["A residual block combines a learned transformation F(x) with a shortcut carrying the input x:
","F(x) is the output of the learned residual branch. Adding x provides a direct path for information and gradients, helping optimize deep networks. If the branch produces zero, the addition stage returns x. A post-addition activation, if specified by the architecture, is applied afterward.
","The addition is entry by entry. Each element in F(x) needs a corresponding element in x, so height, width and channel count must match. For example, a 14 × 14 × 32 tensor cannot be directly added to a 14 × 14 × 64 tensor in a standard residual shortcut.
","A learned projection, commonly a 1 × 1 convolution with an appropriate number of filters and stride, maps the shortcut to the required shape. Then add the projected shortcut and the residual branch. This is addition, not channel concatenation.
","Final answer: a shortcut adds input information to the learned residual branch. Use an identity shortcut when shapes match and a suitable projection when they do not.
","Yaad rakho: Same-size entries jodte hain; mismatch ho toh shortcut ka size pehle match karo.
"]},{"title":"Q2","question":["Q2 optional (3 marks). (a) Explain why Adam uses bias correction for its initially zero moment estimates. (1 mark) (b) Adam starts with θ₀ = 1 and m₀ = v₀ = 0. Use learning rate η = 0.1, β₁ = 0.9 and β₂ = 0.99. The supplied gradients are g₁ = 2 and g₂ = −1. Perform two bias-corrected updates. Show the raw first and second moments, corrected moments and updated parameter after each iteration. Set ε = 0 only for this nonzero-denominator hand calculation. (2 marks)
"],"answer":["Adam initializes both moving averages to zero. At early iterations, too little gradient history has accumulated, so the raw estimates are biased toward zero. Dividing mₜ by 1 − β₁ᵗ and vₜ by 1 − β₂ᵗ corrects this initialization bias before the parameter update. The power uses the current iteration number t. As t increases, these denominators approach 1.
","- Starting parameter θ₀ = 1.
","- Starting first moment m₀ = 0 and second moment v₀ = 0.
","- Learning rate η = 0.1.
","- Decay factors β₁ = 0.9 and β₂ = 0.99.
","- Supplied gradients, in order: 2, then −1.
","- ε = 0 for this exercise; both denominators will be nonzero.
","For iterations 1 and 2, calculate m, v, their bias-corrected values and the new parameter θ.
","Adam keeps two moving averages: m stores signed gradients and v stores squared gradients. Zero initialization makes these averages initially too small, so divide by the correction factors for the current iteration t.
","The hat marks a corrected value. Carry the raw m and v into the next moving-average recurrence, not the corrected values. Keep full calculator precision until the final answer.
","The current gradient is 2. Retain 90% of the old first moment and add 10% of this gradient.
","Square the gradient first: 2² = 4. Retain 99% of the old second moment and add 1% of that square.
","Use t = 1. The correction denominators are 1 − 0.9 = 0.1 and 1 − 0.99 = 0.01.
","The square root of the corrected second moment is √4 = 2. The step is 0.1 × 2 ÷ 2 = 0.1.
","Use supplied gradient −1, previous raw m = 0.2 and previous raw v = 0.04.
","The two contributions are 0.18 and −0.1; their sum is 0.08.
","The square of −1 is +1.
","The two contributions are 0.0396 and 0.01; their sum is 0.0496.
","Use t = 2. The correction denominators are 1 − 0.9² = 0.19 and 1 − 0.99² = 0.0199.
","Start from the updated parameter 0.9. Take the square root of the corrected second moment, divide the corrected first moment by it, multiply by 0.1 and subtract.
","The displayed decimals are rounded; evaluate using the unrounded fractions when possible.
","Final answer: θ₁ = 0.9 and θ₂ ≈ 0.873330057. The second supplied gradient is negative, but the first-moment estimate is still positive because it retains part of the previous positive gradient.
","Yaad rakho: Do raw memories banao → current t se hats nikalo → update; next round mein raw memories le jao.
"]},{"title":"Q3","question":["Q3 optional (3 marks). A 16 × 16 × 64 feature map is processed by 32 filters of size 3 × 3 using stride 1 and same padding. Compare the direct convolution's MAC count with a bottleneck that first applies eight 1 × 1 filters, then the same 32-filter 3 × 3 convolution. Both stages keep spatial size 16 × 16. Find both costs, the absolute saving and percentage saving. Count one multiply–accumulate as one MAC and omit bias additions.
"],"answer":["- Input map: 16 × 16 with 64 channels.
","- Direct layer: 32 filters, each 3 × 3, receiving 64 channels.
","- Bottleneck stage 1: eight 1 × 1 filters, receiving 64 channels.
","- Bottleneck stage 2: 32 filters, each 3 × 3, now receiving only 8 channels.
","- Every layer has output height and width 16; ignore bias additions.
","Direct MACs, both bottleneck-stage MACs, their total, absolute saving and saving percentage.
","Each output entry uses K²Cᵢₙ weight–input pairs. There are HₒᵤₜWₒᵤₜCₒᵤₜ output entries. Multiply these counts:
","The bottleneck has two layers, so calculate each layer's work and add them. Saving is direct work minus reduced work, expressed as a fraction of direct work.
","Each output entry uses 3 × 3 × 64 = 576 weight–input pairs.
","There are 16 × 16 × 32 = 8,192 output entries.
","Each reduction output uses 1 × 1 × 64 = 64 weight–input pairs.
","There are 16 × 16 × 8 = 2,048 reduction outputs.
","The new map is 16 × 16 × 8. This changes the next layer's input channels to 8.
","Each output now uses 3 × 3 × 8 = 72 weight–input pairs, instead of 576.
","The number of outputs remains 16 × 16 × 32 = 8,192.
","Include both stages; do not compare the direct layer against only the second stage.
","Divide the saving by the original direct cost, then multiply by 100.
","Final answer: direct 4,718,592 MACs; bottleneck 720,896 MACs; saving 3,997,696 MACs, approximately 84.72%. Both routes end in a 16 × 16 × 32 output, though they are different learned transformations.
","Yaad rakho: 1 × 1 se channels kam; expensive 3 × 3 ka kaam kam. Dono stages ka cost jodna hai.
"]},{"title":"Q4","question":["Q4 optional (3 marks). Explain transfer learning for image classification with limited labelled data. Distinguish feature extraction from fine-tuning and describe a suitable training procedure.
"],"answer":["Transfer learning reuses knowledge learned on a source task to help solve a target task. For image classification, a pretrained CNN can provide visual features that a new classifier uses for the target classes. This can reduce the amount of target data and training needed, provided the source features are useful for the target task.
","| Method | Which parameters learn? | Main effect |
|---|---|---|
| Feature extraction | Keep pretrained backbone weights fixed; train a new classification head | Learn to classify using fixed pretrained features |
| Fine-tuning | Update selected pretrained layers as well as the new head | Adapt representations to the target task |
Freezing a layer stops its parameter updates; it does not remove its stored weights. Fine-tuning typically exposes more trainable parameters and therefore needs particular care with small datasets.
","1. Create separate training, validation and test splits for the target problem.
","2. Choose a suitable pretrained CNN and apply its required input size and preprocessing.
","3. Replace the original classifier with a new head matching the target classes.
","4. Freeze the backbone and train the new head on the training set.
","5. If validation results justify it, unfreeze selected later layers and continue with an appropriately smaller learning rate.
","6. Select the best checkpoint using validation performance; evaluate the final model on the held-out test set.
","For a small dataset containing cats and dogs, reuse a pretrained image backbone and replace its original classifier with a binary-classification head. First learn the new head with the backbone frozen. Later, selected high-level layers may be fine-tuned if this improves validation performance.
","Final answer: feature extraction learns a new classifier over fixed features; fine-tuning also adapts selected pretrained features. The correct choice depends on target data quantity, similarity to the source data and validation results.
","Yaad rakho: Freeze mein purane features fixed; fine-tune mein selected purane features bhi seekhte hain.
"]}]}}