Open In Colab

Regularization

Previously, we looked at how to measure model performance and identified that there could be a significant performance gap between the training and test data.

Possible reasons for this discrepancy include:

  1. the model describes statistical peculiarities of the training data that are not representative of the true mapping from input to output (overfitting)

  2. the model is unconstrained in areas with no training examples, leading to suboptimal predictions.

This chapter discusses regularization techniques. These are a family of methods that reduce the generalization gap between training and test performance.

Strictly speaking, regularization involves adding explicit terms to the loss function that favor certain parameter choices.

However, in machine learning, this term is commonly used to refer to any strategy that improves generalization. In this section, we consider a set of heuristic methods that improve test performance. These include early stopping, ensembling, dropout, label smoothing, and transfer learning.

The regularization methods discussed in this chapter aim to improve generalization by one of four mechanisms:

  1. Some methods aim to make the modeled function smoother. Smoother functions are less likely to overfit the training data and tend to generalize better to unseen data.

  2. Other methods increase the effective amount of data. By providing more training examples, these methods help the model learn a more accurate representation of the underlying data distribution, reducing overfitting.

  3. The third group of methods combine multiple models and hence mitigate against uncertainty in the fitting process. Ensemble methods, for example, average the predictions of several models to improve robustness and generalization.

  4. Finally, the fourth group of methods encourages the training process to converge to a wide minimum where small errors in the estimated parameters are less important. Wide minima are associated with better generalization because the model’s performance is less sensitive to slight variations in the parameters. By wide minimum, we mean a region in the parameter space where the loss function remains relatively low even if the parameters change slightly.