Shallow vs. Deep neural networks
Number of linear regresions per parameter
A shallow network with one input, one output, and \(D > 2\) hidden units can create up to \(D+ 1\) linear regions and is defined by \(3D+ 1\) parameters.
A deep network with one input, one output, and \(K\) layers of \(D>2\) hidden units can create a function with up to \((D+ 1)K\) linear regions using \(3D+ 1 + (K−1)D(D+ 1)\) parameters.
The maximum number of linear regions for neural networks increases rapidly with the network depth.
Network with \(D_i = 1\) input. Each curve represents a fixed number of hidden layers \(K\), as we vary the number of hidden units \(D\) per layer.
For a fixed parameter budget, deeper networks produce more linear regions than shallower ones. A network with \(K = 5\) layers and \(D = 10\) hidden units per layer has 471 parameters (highlighted point) and can produce 161,051 regions.
Network with \(D_i = 10\) inputs. Each subsequent point along a curve represents ten hidden units. Here, a model with \(K = 5\) layers and \(D = 50\) hidden units per layer has 10,801 parameters (highlighted point) and can create more than \(10^{40}\) linear regions.
Figure above shows how the maximum number of linear regions increases as a function of the number of parameters for networks mapping scalar input x to scalar output y.
Deep neural networks create much more complex functions for a fixed parameter budget. This effect is magnified as the number of input dimensions \(D_i\) increases, although computing the maximum number of regions is less straightforward.
Depth efficiency
Both deep and shallow networks can model arbitrary functions, but some functions can be approximated much more efficiently with deep networks.
Functions have been identified that require a shallow network with exponentially more hidden units to achieve an equivalent approximation to that of a deep network. This phenomenon is referred to as the depth efficiency of neural networks. This property is also attractive, but it’s not clear that the real-world functions that we want to approximate fall into this category.
Training and generalization
A further possible advantage of deep networks over shallow networks is their ease of fitting; it is usually easier to train moderately deep networks than to train shallow ones (see figure 20.2). It may be that overparameterized deep models (i.e., those with more parameters than training examples) have a large family of roughly equivalent solutions that are easy to find.
However, as we add more hidden layers, training becomes more difficult again.
Deep neural networks also seem to generalize to new data better than shallow ones.
In practice, the best results for most tasks have been achieved using networks with tens or hundreds of layers.
Neither of these phenomena are well understood, and we return to them in chapter 20