Choosing hyperparameters

In the previous section, we discussed how test performance changes with model capacity.

Unfortunately, in the classical regime, we don’t have access to either the bias (which requires knowledge of the true underlying function) or the variance (which requires multiple independently sampled datasets to estimate).

In the modern regime, there is no way to tell how much capacity should be added before the test error stops improving.

This raises the question of exactly how we should choose model capacity in practice.

For a deep network, the model capacity depends on the numbers of hidden layers and hidden units per layer as well as other aspects of architecture that we have yet to introduce.

Furthermore, the choice of learning algorithm and any associated parameters (learning rate, etc.) also affects the test performance.

These elements are collectively termed hyperparameters.

The process of finding the best hyperparameters is termed hyperparameter search or (when focused on network structure) neural architecture search.

Hyperparameters are typically chosen empirically; we train many models with differ- ent hyperparameters on the same training set, measure their performance, and retain the best model.

However, we do not measure their performance on the test set; this would admit the possibility that these hyperparameters just happen to work well for the test set but don’t generalize to further data. Instead, we introduce a third dataset known as a validation set.

For every choice of hyperparameters, we train the associated model using the training set and evaluate performance on the validation set.

Finally, we select the model that worked best on the validation set and measure its performance on the test set.

In principle, this should give a reasonable estimate of the true performance.

The hyperparameter space is generally smaller than the parameter space but still too large to try every combination exhaustively.

Unfortunately, many hyperparameters are discrete (e.g., the number of hidden layers), and others may be conditional on one another (e.g., we only need to specify the number of hidden units in the tenth hidden layer if there are ten or more layers). Hence, we cannot rely on gradient descent methods as we did for learning the model parameters.

Hyperparameter optimization algorithms intelligently sample the space of hyperparameters, contingent on previous results.

This procedure is computationally expensive since we must train an entire model and measure the validation performance for each combination of hyperparameters.