Deep Learning · TEE · section

Drill 26 RMSProp variation 1

Numerical practice from direct to mixed

Question

Question. θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9.

Answer — step by step

Given

Initial θ₀ = 1; η = 0.1.

Supplied gradients: [2,1].

Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation.

ρ = 0.9.

To find

Find every stored state needed by the method and the parameter after each of two updates.

Formula — which one and why

RMSProp. Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence.

s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer.

$s_t=\rho s_{t-1}+(1-\rho)g_t^2$ LaTeX: s_t=\rho s_{t-1}+(1-\rho)g_t^2

$\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon}$ LaTeX: \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon}

Yaad rakhne ka cue: Saare purane squares nahi—unki fading average.

Solution — step by step

Symbols and formulas used

θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise.

s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0.

$s_t=\rho s_{t-1}+(1-\rho)g_t^2$ LaTeX: s_t=\rho s_{t-1}+(1-\rho)g_t^2

$\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon}$ LaTeX: \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon}

Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result.

Step 1: Iteration 1 with supplied gradient 2.

Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.

$s_1=0.9(0)+0.1(2)^2=0.4$ LaTeX: s_1=0.9(0)+0.1(2)^2=0.4

Step 2: Iteration 2 with supplied gradient 1.

Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square.