Deep Learning · MT2 · section

Parameters and hyperparameters

Neural-network foundations

Definition. Model parameters are quantities estimated during training, whereas hyperparameters specify the architecture or training procedure and are selected separately. Optimizer states are algorithmic accumulators associated with the training procedure.

Explanation and working

Weights and biases are parameters learned during training. Learning rate, batch size, number of epochs, network architecture, dropout rate and regularization strength are hyperparameters chosen or tuned using validation performance. Adam’s moving averages are optimizer state: they are updated by the algorithm but are not extra network connection weights.

Diagram: Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately.. Labels: What is learned, and what is chosen? | What is learned, and what is chosen? | Two different kinds of quantities appear in training. | PARAMETERS | Weights and biases | Changed by training | Example: w = 0.5 → 0.6 | HYPERPARAMETERS | Learning rate, batch size | Chosen or tuned separately | Example: learning rate 0.1 | Optimizer memories are algorithm state; they are not extra network connection | weights.

An epoch is one complete pass through the training dataset. A batch is the group used to calculate one update. An iteration here means one update. If there are 100 samples and batch size 10, one epoch has 10 updates. One epoch does not generally mean one update.