# Methods to revise before the examination

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/19/mt2-c19-s5

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c19-s5/

### Last-day revision: read in this order

This is a revision digest after studying the lessons. Full solutions remain beside each topic and under Solved Questions. If a step here is unclear, return to that worked example; the final day is for recovering the method, not memorizing unexplained final numbers.

### 1. Learning and the neuron

Machine learning estimates a model from data. Deep learning uses multiple trainable layers to learn successive representations. Inputs describe a sample; targets are the required answers; predictions are the model's answers. Weights and biases are learned parameters. Learning rate, batch size and architecture are chosen hyperparameters.

Training changes parameters using training examples. Validation selects settings/stopping time. Testing evaluates the final choice on held-out data. Overfitting means strong training performance with weak generalization; underfitting means the model has not captured enough of the relationship.

For one neuron, multiply each input by its own weight, add all products, add the bias once and apply the activation. A layer repeats this for several receiving neurons. Each weight-matrix row belongs to one receiving neuron. Transpose swaps rows and columns; it does not change signs.



$$
z=\sum_iw_ix_i+b,\qquad a=f(z)
$$



**Memory cue:** multiply → sum → bias → activate. In a full network, finish one complete layer before the next.

### 2. Activation and loss choices

Sigmoid maps to (0,1) and is used for binary-class probabilities. Tanh maps to (−1,1), centred on zero. Both can saturate, producing small derivatives. ReLU retains positive values and returns zero on the negative side; its negative-side gradient is zero. Leaky ReLU retains a specified small negative-side slope. Softmax normalizes a complete list of logits into mutually exclusive class probabilities.



$$
\sigma(z)=\frac1{1+e^{-z}},\qquad\sigma\prime(z)=\sigma(z)[1-\sigma(z)]
$$





$$
\tanh\prime(z)=1-\tanh^2(z)
$$





$$
p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j
$$



For squared errors, SSE sums squares, half SSE divides that sum by 2, and MSE divides it by the number of scalar predictions. Their derivatives differ. For one-hot categorical cross-entropy, only the correct-class probability contributes. For binary cross-entropy, p means probability of class 1, not automatically the true class.



$$
L_{\mathrm{half}}=\frac12\sum_i(\hat y_i-y_i)^2
$$





$$
\mathrm{MSE}=\frac1n\sum_i(\hat y_i-y_i)^2
$$





$$
L_{\mathrm{BCE}}=-[y\ln p+(1-y)\ln(1-p)]
$$





$$
L_{\mathrm{CCE}}=-\ln p_{\mathrm{correct}}
$$



**Memory cue:** loss ka denominator dekho before differentiating. Sigmoid output is not its derivative.

### 3. Backpropagation: the repeatable method

Write the given loss and activation first. Compute all forward sums/activations with the original parameters. Differentiate loss with respect to each output, multiply by the output activation derivative, and call the result the output delta.



$$
\delta_o=\frac{\partial L}{\partial o}f_o\prime(z_o)
$$



For a hidden neuron, add the outgoing weighted deltas first and then multiply by the hidden activation derivative. Each connection gradient is receiving delta × sending activation. The bias gradient is the receiving delta. Use old weights for all gradients before making the simultaneous update.



$$
\delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j}
$$





$$
\frac{\partial L}{\partial W_{ji}}=\delta_ja_i,\qquad\frac{\partial L}{\partial b_j}=\delta_j
$$





$$
\theta_{\mathrm{new}}=\theta-\eta\frac{\partial L}{\partial\theta}
$$



**Memory cue:** path mein multiply; paths milne par add; update sab gradients ke baad. For sigmoid+BCE or softmax+CCE, the logit delta simplifies to prediction−target for the stated single-example sum convention.

### 4. Optimizers: identify the stored state

Full-batch GD uses the entire dataset for one update; SGD uses one sample; mini-batch uses a subset. An epoch is one pass through the data. With a final incomplete batch included, updates per epoch are ceiling(N/B).

| Method | State and calculation order | Detail that changes the answer |
| --- | --- | --- |
| GD | Current gradient → subtract ηg | Recompute at the new parameter unless gradients are explicitly supplied |
| Momentum | Retain βv → subtract ηg → add new velocity to parameter | Signed velocity and gradient-average conventions must not be mixed |
| Nesterov | Compute lookahead θ+βv → gradient there → new velocity → update actual θ | Do not add new velocity to the lookahead point |
| AdaGrad | Add g² to all-past sum G → divide gradient by √G+ε | G never decreases |
| RMSProp | Decay old squared state → add recent g² → normalize | Use the specified decay factor |
| Adam | Update signed mean m and squared mean v → correct both → normalize | Carry raw states forward; correction powers use iteration t |



$$
v_t=\beta v_{t-1}-\eta g_t,\qquad\theta_t=\theta_{t-1}+v_t
$$





$$
G_t=G_{t-1}+g_t^2
$$





$$
s_t=\rho s_{t-1}+(1-\rho)g_t^2
$$





$$
m_t=\beta_1m_{t-1}+(1-\beta_1)g_t
$$





$$
v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2
$$





$$
\hat m_t=\frac{m_t}{1-\beta_1^t},\qquad\hat v_t=\frac{v_t}{1-\beta_2^t}
$$





$$
\theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon}
$$



Here v means signed velocity in momentum and squared-gradient state in Adam; the method defines the symbol. **Memory cue:** AdaGrad sab squares; RMSProp recent squares; Adam direction + squares + correction.

### 5. Regularization and generalization

L1 adds λΣ|w| and a nonzero-weight derivative λsign(w), encouraging sparsity. L2 with the half convention adds (λ/2)Σw² and derivative λw. Without that half, the derivative is 2λw. Regularization gradients add to data gradients before updating the weight.

Inverted dropout uses keep probability q: retain a sampled activation with scale 1/q, or replace it by zero. Inference uses the full activation without that training-time rescaling. Drop probability p gives q = 1−p. Early stopping tracks validation performance and restores the best checkpoint under the stated patience rule.



$$
a'_i=\frac{m_i a_i}{q},\qquad m_i\in\{0,1\}
$$



**Memory cue:** dropout probability aur keep probability alag; L2 mein half hai ya nahi, pehle dekho.

### 6. CNN calculation sequence

Use the deep-learning cross-correlation convention unless the question explicitly requires kernel flipping. At each output position, multiply matching input-window and kernel entries, sum over spatial entries and all input channels, and add that filter's bias once. A new filter creates a new output channel.



$$
N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1
$$





$$
P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}}
$$





$$
n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}}
$$



Apply the shape formula separately to height and width. Pooling changes spatial dimensions but preserves channels; ordinary pooling and flattening have zero learned parameters. Maximum pooling chooses the largest window value; average pooling adds its values and divides by the number of entries. A filter's spatial reuse changes computation, not its parameter count.

For a complete architecture, make one row per layer: input shape → operation → output shape → weights → biases → total. Flatten uses the size after the final convolution/pool. **Memory cue:** shape count aur parameter count ko kabhi mix mat karo.

### 7. CNN theory and architectures

CNNs use local connectivity and weight sharing, reducing parameters compared with unrestricted dense image mappings. Convolution learns features; activations introduce nonlinearity; pooling summarizes local responses; dense or global-pooling/output stages make the final task prediction.

AlexNet: stacked convolutions with ReLU, pooling and dense classification; dropout in dense layers and data augmentation help control overfitting. Inception: parallel spatial scales, 1 × 1 reductions and channel concatenation. ResNet: learns a residual F(x) and adds a shortcut x; use projection when shapes differ. Concatenation adds channels; residual addition requires equal shapes and retains their count.

Transfer learning reuses a pretrained feature extractor, replaces the task head, trains the head and optionally fine-tunes selected pretrained layers at a suitable smaller learning rate. Explain classification, detection and segmentation as different outputs, with an appropriate concrete application.

### 8. Match the announced Unit 3–4 mix

Revise all theory from optimization, regularization, CNN operations and CNN architectures. For 2 marks, answer exactly the requested definition, distinction or small calculation. For 3 marks, show the complete requested numerical method or explain the mechanism clearly. The paper may combine both.

Prepare Unit 3 numericals and especially the two Unit 4 numerical opportunities highlighted by the faculty. Their exact question numbers are not specified. RNN, LSTM and GRU remain in the TEE material; they are not part of this focused MT2 revision.

### The final answer-writing routine

For every numerical: full question → Given → To find → governing formula and why → separate numbered steps → final answer. Carry extra calculator precision; round at the end. Label each layer, iteration and connection so the examiner can follow the calculation.

For a definition question, state the technical meaning and the requested distinction/example. For a longer mechanism question, include a labelled diagram or equations, explain their sequence, then give the relevant benefits, limitations or comparison. Marks printed beside a historical question are its actual marks; individual steps here are not assigned invented official marks.

For this 60-minute MT2, a workable allocation is 5 minutes to read and select, 15 minutes for Q1(a,b), 15 minutes for each selected 3-mark question and 10 minutes to review. Adjust if one calculation needs more working. If stuck, write the givens and governing formula first, complete the solvable parts, then return to the unfinished calculation.