# Parameters and hyperparameters

Subject: Deep Learning

Exam: MT2

Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s4

Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s4/

**Definition.** Model parameters are quantities estimated during training, whereas hyperparameters specify the architecture or training procedure and are selected separately. Optimizer states are algorithmic accumulators associated with the training procedure.

### Explanation and working

Weights and biases are parameters learned during training. Learning rate, batch size, number of epochs, network architecture, dropout rate and regularization strength are hyperparameters chosen or tuned using validation performance. Adam’s moving averages are optimizer state: they are updated by the algorithm but are not extra network connection weights.

![Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters.svg)

Diagram: Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately.. Labels: What is learned, and what is chosen? | What is learned, and what is chosen? | Two different kinds of quantities appear in training. | PARAMETERS | Weights and biases | Changed by training | Example: w = 0.5 → 0.6 | HYPERPARAMETERS | Learning rate, batch size | Chosen or tuned separately | Example: learning rate 0.1 | Optimizer memories are algorithm state; they are not extra network connection | weights.

Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately. Tap the diagram to enlarge.

An **epoch** is one complete pass through the training dataset. A **batch** is the group used to calculate one update. An **iteration** here means one update. If there are 100 samples and batch size 10, one epoch has 10 updates. One epoch does not generally mean one update.