# Transfer learning from a small dataset

Subject: Deep Learning

Exam: TEE

Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s4

Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s4/

**Definition.** Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers.

### Explanation and working

Transfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort.

![Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change.](https://deep-learning-study-room.vercel.app/assets/figures/v_transfer.svg)

Diagram: Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features.

Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change. Tap the diagram to enlarge.

**Full scenario.** Build a face-attendance model with few images per student. Explain a training plan.

**Step 1. Define and split the data.** Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization.

**Step 2. Choose a suitable pretrained backbone.** For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available.

**Step 3. Replace the classifier.** Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy.

**Step 4. Freeze and train.** Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring.

**Step 5. Fine-tune if justified.** Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint.

**Step 6. Evaluate.** Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition.

Feature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario.