Deep Learning · MT2 · section

Transfer learning from a small dataset

CNN architectures and transfer learning

Definition. Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers.

Explanation and working

Transfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort.

Diagram: Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features.

Full scenario. Build a face-attendance model with few images per student. Explain a training plan.

Step 1. Define and split the data. Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization.

Step 2. Choose a suitable pretrained backbone. For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available.

Step 3. Replace the classifier. Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy.

Step 4. Freeze and train. Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring.

Step 5. Fine-tune if justified. Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint.

Step 6. Evaluate. Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition.

Feature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario.