Consider fitting a model \(f[x,\phi]\) with parameters \(\phi\) using a training set \(\{x_i,y_i\}\) of input/output pairs. We seek the parameters \(\phi\) that minimize the loss function \(L[\phi]\) :
where the individual terms \(\ell_i[x_i,y_i]\) measure the mismatch between the network predictions \(f[x_i,\phi]\) and output targets \(y_i\) for each training pair. To bias this minimization toward certain solutions, we include an additional term:
where \(g[\phi]\) is a function that returns a scalar which takes larger values when the parameters are less preferred.
The term \(\lambda\) is a positive scalar that controls the relative contribution of the original loss function and the regularization term. The minima of the regularized loss function usually differ from those in the original, so the training procedure converges to different parameter values.
Explicit regularization. a) Loss function for Gabor model. Cyan circles represent local minima. Gray circle represents the global minimum. b) The regularization term favors parameters close to the center of the plot by adding an increasing penalty as we move away from this point. c) The final loss function is the sum of the original loss function plus the regularization term. This surface has fewer local minima, and the global minimum has moved to a different position (arrow shows change).
L2 regularization in simplified network with 14 hidden units (see fig- ure 8.4). a–f) Fitted functions as we increase the regularization coefficient λ. The black curve is the true function, the orange circles are the noisy training data, and the cyan curve is the fitted model. For small λ(panels a–b), the fitted func- tion passes exactly through the data points. For intermediate λ(panels c–d), the function is smoother and more similar to the ground truth. For large λ (panels e–f), the regularization term overpowers the likelihood term, so the fitted function is too smooth and the overall fit is worse.
L2 Regularization
This discussion has sidestepped the question of which solutions the regularization term should penalize (or equivalently that the prior should favor). Since neural networks are used in an extremely broad range of applications, these can only be very generic pref- erences. The most commonly used regularization term is the L2 norm, which penalizes the sum of the squares of the parameter values:
where j indexes the parameters. This is also referred to as Tikhonov regularization or ridge regression, or (when applied to matrices) Frobenius norm regularization.
For neural networks, L2 regularization is usually applied to the weights but not the biases and is hence referred to as a weight decay term. The effect is to encourage smaller weights, so the output function is smoother. To see this, consider that the output prediction is a weighted sum of the activations at the last hidden layer. If the weights have a smaller magnitude, the output will vary less. The same logic applies to the computation of the pre-activations at the last hidden layer and so on, progressing backward through the network. In the limit, if we forced all the weights to be zero, the network would produce a constant output determined by the final bias parameter.
Figure 9.2 shows the effect of fitting the simplified network from figure 8.4 with weight decay and different values of the regularization coefficient λ. When λ is small, it has little effect. However, as λ increases, the fit to the data becomes less accurate, and the function becomes smoother. This might improve the test performance for two reasons:
If the network is overfitting, then adding the regularization term means that the network must trade off slavish adherence to the data against the desire to be smooth. One way to think about this is that the error due to variance reduces (the model no longer needs to pass through every data point) at the cost of increased bias (the model can only describe smooth functions).
When the network is over-parameterized, some of the extra model capacity de- scribes areas with no training data. Here, the regularization term will favor func- tions that smoothly interpolate between the nearby points. This is reasonable behavior in the absence of knowledge about the true function.
# These came from writing out the expression for the sum of squares loss and taking the# derivative with respect to phi0 and phi1. It was a lot of hassle to get it right!def gabor_deriv_phi0(data_x,data_y,phi0, phi1): x =0.06* phi1 * data_x + phi0 y = data_y cos_component = np.cos(x) sin_component = np.sin(x) gauss_component = np.exp(-0.5* x *x /16) deriv = cos_component * gauss_component - sin_component * gauss_component * x /16 deriv =2* deriv * (sin_component * gauss_component - y)return np.sum(deriv)def gabor_deriv_phi1(data_x, data_y,phi0, phi1): x =0.06* phi1 * data_x + phi0 y = data_y cos_component = np.cos(x) sin_component = np.sin(x) gauss_component = np.exp(-0.5* x *x /16) deriv =0.06* data_x * cos_component * gauss_component -0.06* data_x*sin_component * gauss_component * x /16 deriv =2*deriv * (sin_component * gauss_component - y)return np.sum(deriv)def compute_gradient(data_x, data_y, phi): dl_dphi0 = gabor_deriv_phi0(data_x, data_y, phi[0],phi[1]) dl_dphi1 = gabor_deriv_phi1(data_x, data_y, phi[0],phi[1])# Return the gradientreturn np.array([[dl_dphi0],[dl_dphi1]])
Now we are ready to find the minimum. For simplicity, we’ll just use regular (non-stochastic) gradient descent with a fixed learning rate.
def gradient_descent_step(phi, data, model):# Step 1: Compute the gradient gradient = compute_gradient(data[0,:],data[1,:], phi)# Step 2: Update the parameters -- note we want to search in the negative (downhill direction) alpha =0.1 phi = phi - alpha * gradientreturn phi
# Initialize the parametersn_steps =41phi_all = np.zeros((2,n_steps+1))phi_all[0,0] =2.6phi_all[1,0] =8.5# Measure loss and draw initial modelloss = compute_loss(data[0,:], data[1,:], model, phi_all[:,0:1])draw_model(data,model,phi_all[:,0:1], "Initial parameters, Loss = %f"%(loss))for c_step inrange (n_steps):# Do gradient descent step phi_all[:,c_step+1:c_step+2] = gradient_descent_step(phi_all[:,c_step:c_step+1],data, model)# Measure loss and draw model every 8th stepif c_step %8==0: loss = compute_loss(data[0,:], data[1,:], model, phi_all[:,c_step+1:c_step+2]) draw_model(data,model,phi_all[:,c_step+1], "Iteration %d, loss = %f"%(c_step+1,loss))draw_loss_function(compute_loss, data, model, my_colormap, phi_all)
Unfortunately, when we start from this position, the solution descends to a local minimum and the final model doesn’t fit well.
But what if we had some weak knowledge that the solution was in the vicinity of \(\phi_0=0.0\), \(\phi_{1} = 12.5\) (the center of the plot)?
Let’s add a term to the loss function that penalizes solutions that deviate from this point.
where \(\lambda\) controls the relative importance of the original loss and the regularization term
# Computes the regularization termdef compute_reg_term(phi0,phi1): reg_term = phi0*phi0 + phi1*phi1return reg_term ;# Define the loss function# Note I called the weighting lambda_ to avoid confusing it with python lambda functionsdef compute_loss2(data_x, data_y, model, phi, lambda_): pred_y = model(phi, data_x) loss = np.sum((pred_y-data_y)*(pred_y-data_y))# Add the new term to the loss loss = loss + lambda_ * compute_reg_term(phi[0],phi[1])return loss
# Code to draw the regularization functiondef draw_reg_function():# Make grid of offset/frequency values to plot offsets_mesh, freqs_mesh = np.meshgrid(np.arange(-10,10.0,0.1), np.arange(2.5,22.5,0.1)) loss_mesh = np.zeros_like(freqs_mesh)# Compute loss for every set of parametersfor idslope, slope in np.ndenumerate(freqs_mesh): loss_mesh[idslope] = compute_reg_term(offsets_mesh[idslope], slope) fig,ax = plt.subplots() fig.set_size_inches(8,8) ax.contourf(offsets_mesh,freqs_mesh,loss_mesh,256,cmap=my_colormap) ax.contour(offsets_mesh,freqs_mesh,loss_mesh,20,colors=['#80808080']) ax.set_ylim([2.5,22.5]) ax.set_xlabel('Offset $\phi_{0}$'); ax.set_ylabel('Frequency, $\phi_{1}$') plt.show()# Draw the regularization function. It should look similar to figure 9.1bdraw_reg_function()
# Code to draw loss function with regularizationdef draw_loss_function_reg(data, model, lambda_, my_colormap, phi_iters =None):# Make grid of offset/frequency values to plot offsets_mesh, freqs_mesh = np.meshgrid(np.arange(-10,10.0,0.1), np.arange(2.5,22.5,0.1)) loss_mesh = np.zeros_like(freqs_mesh)# Compute loss for every set of parametersfor idslope, slope in np.ndenumerate(freqs_mesh): loss_mesh[idslope] = compute_loss2(data[0,:], data[1,:], model, np.array([[offsets_mesh[idslope]], [slope]]), lambda_) fig,ax = plt.subplots() fig.set_size_inches(8,8) ax.contourf(offsets_mesh,freqs_mesh,loss_mesh,256,cmap=my_colormap) ax.contour(offsets_mesh,freqs_mesh,loss_mesh,20,colors=['#80808080'])if phi_iters isnotNone: ax.plot(phi_iters[0,:], phi_iters[1,:],'go-') ax.set_ylim([2.5,22.5]) ax.set_xlabel('Offset $\phi_{0}$'); ax.set_ylabel('Frequency, $\phi_{1}$') plt.show()# This should look something like figure 9.1cdraw_loss_function_reg(data, model, 0.2, my_colormap)
Now we’ll compute the derivatives \(\frac{\partial L'}{\partial\phi_0}\) and \(\frac{\partial L'}{\partial\phi_1}\) of the regularized loss function:
# Finally, let's run gradient descent and draw the result# Initialize the parametersn_steps =41phi_all = np.zeros((2,n_steps+1))phi_all[0,0] =2.6phi_all[1,0] =8.5lambda_ =0.2# Measure loss and draw initial modelloss = compute_loss2(data[0,:], data[1,:], model, phi_all[:,0:1], lambda_)draw_model(data,model,phi_all[:,0:1], "Initial parameters, Loss = %f"%(loss))for c_step inrange (n_steps):# Do gradient descent step phi_all[:,c_step+1:c_step+2] = gradient_descent_step2(phi_all[:,c_step:c_step+1],lambda_, data, model)# Measure loss and draw model every 8th stepif c_step %8==0: loss = compute_loss2(data[0,:], data[1,:], model, phi_all[:,c_step+1:c_step+2], lambda_) draw_model(data,model,phi_all[:,c_step+1], "Iteration %d, loss = %f"%(c_step+1,loss))draw_loss_function_reg(data, model, lambda_, my_colormap, phi_all)
You should see that the gradient descent algorithm now finds the correct minimum. By applying a tiny bit of domain knowledge (the parameter phi0 tends to be near zero and the parameter phi1 tends to be near 12.5), we get a better solution. However, the cost is that this solution is slightly biased towards this prior knowledge.
In Pytorch, L2 regularization is implemented using the weight_decay parameter in the optimizer. For example:
For SGD, this corresponds to minimizing: \[
L_{\text{total}} = L_{\text{task}} + \frac{\lambda}{2}\sum_p \|p\|_2^2
\]
Here, weight_decay is (). Larger values penalize large weights more strongly. PyTorch SGD documentation You can also set weight_decay with Adam, where its default behavior applies an L2 penalty. AdamW instead applies decoupled weight decay, which behaves differently from adding L2 to the loss. Adam documentation To add L2 explicitly:
# Keep optimizer weight_decay=0 to avoid applying it twice.loss = criterion(model(inputs), targets)l2_penalty =sum(p.square().sum() for p in model.parameters())loss = loss + (1e-4/2) * l2_penaltyoptimizer.zero_grad()loss.backward()optimizer.step()
These examples penalize all model parameters, including biases. If you want to exclude biases from L2 regularization, you can filter them out when computing the penalty:
l2_penalty =sum(p.square().sum() for n, p in model.named_parameters() if'bias'notin n)
Early stopping
Early stopping refers to stopping the training procedure before it has fully converged.
This can reduce overfitting if the model has already captured the coarse shape of the underlying function but has not yet had time to overfit to the noise.
One way of thinking about this is that since the weights are initialized to small values they simply don’t have time to become large, so early stopping has a similar effect to explicit L2 regularization.
A different view is that early stopping reduces the effective model complexity.
Hence, we move back down the bias/variance trade-off curve from the critical region, and performance improves.
Early stopping has a single hyperparameter, the number of steps after which learning is terminated.
As usual, this is chosen empirically using a validation set.
However, for early stopping, the hyperparameter can be selected without the need to train multiple models.
The model is trained once, the performance on the validation set is monitored every T iterations, and the associated parameters are stored.
The stored parameters where the validation performance was best are selected.
Early stopping. a) Simplified shallow network model with 14 linear regions is initialized randomly (cyan curve) and trained with SGD using a batch size of five and a learning rate of 0.05. b–d) As training proceeds, the function first captures the coarse structure of the true function (black curve) before e–f) overfitting to the noisy training data (orange points). Although the training loss continues to decrease throughout this process, the learned models in panels (c) and (d) are closest to the true underlying function. They will generalize better on average to test data than those in panels (e) or (f).
Here, patience=5 allows five epochs without meaningful improvement. Your evaluation function should use model.eval() and torch.no_grad(); training should switch back to model.train().
