# Q7(b) Bellman equation and applications of RL [10]

Subject: Intelligent Systems II

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q7b

Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q7b/

Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged)

**Explain the Bellman equation in reinforcement learning. Also give applications of reinforcement learning.** [10 marks]

### Answer

Write the answer below in this order. It fits in about two pages.

### Part A · Bellman equation (5 marks)

**1. Setting.** Reinforcement learning problems are modelled as a Markov decision process 

$$
\langle\mathcal S,\mathcal A,P,R,\gamma\rangle
$$

. The agent follows a policy 

$$
\pi(a\mid s)
$$

 and wants to maximise the return 

$$
G_t=\sum_{k\ge0}\gamma^kR_{t+k+1}
$$

.

**2. Value functions.** 

$$
v_\pi(s)=\mathbb E_\pi[G_t\mid S_t=s]
$$

 and 

$$
q_\pi(s,a)=\mathbb E_\pi[G_t\mid S_t=s,A_t=a]
$$

.

**3. Key idea.** The return is recursive: 

$$
G_t=R_{t+1}+\gamma G_{t+1}
$$

. Taking expectations gives the **Bellman expectation equation**:



$$
v_\pi(s)=\sum_a\pi(a\mid s)\sum_{s'}P(s'\mid s,a)\,[R(s,a,s')+\gamma v_\pi(s')]
$$





$$
q_\pi(s,a)=\sum_{s'}P(s'\mid s,a)\,\Big[R(s,a,s')+\gamma\sum_{a'}\pi(a'\mid s')q_\pi(s',a')\Big]
$$



**4. Optimality.** For the optimal policy the average over actions becomes a maximum (**Bellman optimality equation**):



$$
v_*(s)=\max_a\sum_{s'}P(s'\mid s,a)\,[R+\gamma v_*(s')],\qquad q_*(s,a)=\sum_{s'}P(s'\mid s,a)\,[R+\gamma\max_{a'}q_*(s',a')]
$$



**5. Meaning and use.** The value of a state equals the immediate reward plus the discounted value of the successor state. This breaks a long-horizon problem into one-step sub-problems (dynamic programming). It is the basis of policy evaluation, policy iteration, value iteration and Q-learning; the optimal policy is 

$$
\pi_*(s)=\arg\max_a q_*(s,a)
$$

.

**6. Small illustration** (adds clarity): if staying in a state gives +1 each step with 

$$
\gamma=0.9
$$

, then 

$$
v=1+0.9v\Rightarrow v=10
$$

.

### Part B · Applications of reinforcement learning (5 marks)

1. **Game playing**: AlphaGo (defeated Lee Sedol, 2016), Atari agents (DQN), and the Mario agent built in this course's lab.
2. **Robotics**: walking, grasping and balancing learned by trial and error in simulation.
3. **Autonomous driving**: lane changing, merging and parking decisions.
4. **Recommendation systems**: choosing content to maximise long-term engagement rather than single clicks.
5. **Finance**: trading and portfolio rebalancing as sequential decisions.
6. **Healthcare**: dynamic treatment regimes (dose adjustment over time).
7. **Resource and energy management**: data-centre cooling control, traffic-signal timing, inventory control.
8. **Language models**: reinforcement learning from human feedback (RLHF) for chatbots.

#### One application in detail: traffic-signal control

|  |  |
| --- | --- |
| Agent | Controller at an intersection. |
| State | Queue length and waiting time on each approach, current signal phase. |
| Actions | Keep the current phase or switch to the next green phase. |
| Reward | Negative of total waiting time (or vehicles cleared per cycle). |
| Learning | Q-learning updates 

$$
Q(s,a)
$$

 from observed traffic; ε-greedy exploration tries alternative timings. |
| Benefit | Adapts to rush hours and incidents; fixed timers cannot. |
| Challenge | Safety during exploration, so agents are trained in simulation first. |

**Yaad rakho:** Bellman ka sar: “value = reward abhi + γ × value aage”. Application likho to S, A, R batao: examiner ko yahi chahiye.