AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

deep-learning-book

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--deep-learning-book at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 30

chapters/ch08-optimization.md

Chapter 8: Optimization for Training Deep Models

Source chapter (free, official): https://www.deeplearningbook.org/contents/optimization.html

Core Idea

Training is not pure optimization: you minimize an empirical surrogate to reduce a risk you cannot measure, on a non-convex landscape, with noisy gradients. The chapter separates the difficulties (ill-conditioning, saddles, cliffs, long-term dependencies, poor correspondence between local and global structure) from the algorithms that address them.

Frameworks Introduced

  • Empirical risk minimization and its surrogates: you optimize a differentiable proxy, not the metric you report.
  • Minibatch SGD: gradient noise scales roughly with 1/√batch, so bigger batches buy accuracy in the gradient at linear cost — a poor trade past a point.
  • Ill-conditioning: the dominant obstacle in practice; gradient norm can grow while the loss stalls.
  • Saddle points, not local minima: in high dimensions, critical points are overwhelmingly saddles. This reframes the folklore fear of local minima.
  • Cliffs and exploding gradients → gradient clipping.
  • Momentum and Nesterov momentum: accumulate a velocity to cross ravines.
  • Initialization: Xavier/Glorot and He scaling to keep activation and gradient variance roughly constant with depth. Initialization is an algorithm, not a detail.
  • Adaptive methods: AdaGrad (decaying), RMSProp, Adam (RMSProp + momentum + bias correction).
  • Second-order methods: Newton, conjugate gradient, BFGS/L-BFGS — and why they rarely survive contact with minibatch noise and parameter counts.
  • Batch normalization: reparameterization that stabilizes the scale of layer inputs.
  • Curriculum learning / coordinate descent / Polyak averaging: the meta-strategies.

Mental Models

  • Diagnose by what the gradient is doing: norm exploding = cliff, clip. Norm large but loss flat = ill-conditioning, use momentum/adaptive/normalization. Norm ~0 with high loss = dead units or saturation.
  • Treat learning-rate schedule as a first-class hyperparameter — usually more important than the choice among SGD/Adam variants.
  • Read batch norm as making the loss surface better conditioned; whether it does so by reducing "internal covariate shift" is contested (Santurkar et al. 2018, post-2016).

Anti-patterns

  • Fearing local minima: spend the debugging effort on conditioning and learning rate.
  • Tuning the optimizer before the learning rate.
  • Batch norm with tiny batches — the batch statistics become noise. Use group/layer norm.

What changed after 2016

The largest delta in Part II. AdamW replaced Adam+L2. Learning-rate warmup plus cosine decay became standard for transformers. LayerNorm (and RMSNorm) displaced batch norm in sequence models; pre-norm residual placement displaced post-norm for deep stacks. Large-batch training with LARS/LAMB, and the linear-scaling-rule literature, arrived after the book. Confidence: high.

Key Takeaways

  1. Tune learning rate and schedule first; optimizer family second.
  2. Read gradient-norm behaviour before changing the architecture.
  3. Use He/Xavier-style initialization deliberately — bad init is a common silent failure.

Connects To

  • Ch 4: conditioning, Hessians, and step size.
  • Ch 10: exploding/vanishing gradients as the sequence-model version of these problems.
  • scripts/training_diagnostics.py: this chapter's triage as an executable decision tree.