chapters/ch08-optimization.md
Chapter 8: Optimization for Training Deep Models
Source chapter (free, official): https://www.deeplearningbook.org/contents/optimization.html
Core Idea
Training is not pure optimization: you minimize an empirical surrogate to reduce a risk you cannot measure, on a non-convex landscape, with noisy gradients. The chapter separates the difficulties (ill-conditioning, saddles, cliffs, long-term dependencies, poor correspondence between local and global structure) from the algorithms that address them.
Frameworks Introduced
- Empirical risk minimization and its surrogates: you optimize a differentiable proxy, not the metric you report.
- Minibatch SGD: gradient noise scales roughly with 1/√batch, so bigger batches buy accuracy in the gradient at linear cost — a poor trade past a point.
- Ill-conditioning: the dominant obstacle in practice; gradient norm can grow while the loss stalls.
- Saddle points, not local minima: in high dimensions, critical points are overwhelmingly saddles. This reframes the folklore fear of local minima.
- Cliffs and exploding gradients → gradient clipping.
- Momentum and Nesterov momentum: accumulate a velocity to cross ravines.
- Initialization: Xavier/Glorot and He scaling to keep activation and gradient variance roughly constant with depth. Initialization is an algorithm, not a detail.
- Adaptive methods: AdaGrad (decaying), RMSProp, Adam (RMSProp + momentum + bias correction).
- Second-order methods: Newton, conjugate gradient, BFGS/L-BFGS — and why they rarely survive contact with minibatch noise and parameter counts.
- Batch normalization: reparameterization that stabilizes the scale of layer inputs.
- Curriculum learning / coordinate descent / Polyak averaging: the meta-strategies.
Mental Models
- Diagnose by what the gradient is doing: norm exploding = cliff, clip. Norm large but loss flat = ill-conditioning, use momentum/adaptive/normalization. Norm ~0 with high loss = dead units or saturation.
- Treat learning-rate schedule as a first-class hyperparameter — usually more important than the choice among SGD/Adam variants.
- Read batch norm as making the loss surface better conditioned; whether it does so by reducing "internal covariate shift" is contested (Santurkar et al. 2018, post-2016).
Anti-patterns
- Fearing local minima: spend the debugging effort on conditioning and learning rate.
- Tuning the optimizer before the learning rate.
- Batch norm with tiny batches — the batch statistics become noise. Use group/layer norm.
What changed after 2016
The largest delta in Part II. AdamW replaced Adam+L2. Learning-rate warmup plus cosine decay became standard for transformers. LayerNorm (and RMSNorm) displaced batch norm in sequence models; pre-norm residual placement displaced post-norm for deep stacks. Large-batch training with LARS/LAMB, and the linear-scaling-rule literature, arrived after the book. Confidence: high.
Key Takeaways
- Tune learning rate and schedule first; optimizer family second.
- Read gradient-norm behaviour before changing the architecture.
- Use He/Xavier-style initialization deliberately — bad init is a common silent failure.
Connects To
- Ch 4: conditioning, Hessians, and step size.
- Ch 10: exploding/vanishing gradients as the sequence-model version of these problems.
- scripts/training_diagnostics.py: this chapter's triage as an executable decision tree.