# F22 Q1e

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s2

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s2/

## Question

**Question (4 marks).** Suggest and sketch an RNN architecture for image captioning.

## Answer — step by step

Use a CNN encoder to map the image to a visual representation and a recurrent decoder (vanilla RNN) to generate words one at a time. Condition the decoder’s initial state or its inputs on image features. At each step, vocabulary softmax predicts the next word based on the image and earlier words. Train using correct next-word targets and cross-entropy; stop generation at an end token. This is a one-to-many mapping from one image to a caption sequence.

![The one-to-many pattern models image captioning](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg)

Diagram: The one-to-many pattern models image captioning. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths.

The one-to-many pattern models image captioning Tap the diagram to enlarge.