AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

deep-learning-book

Use when studying or applying Goodfellow, Bengio and Courville's Deep Learning, or asking if its advice still holds. Indexes all 20 chapters with a 2016-to-2026 delta, plans reading paths with reading_path_planner.py and diagnoses training failures with training_diagnostics.py.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--deep-learning-book at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 30

chapters/ch06-deep-feedforward-networks.md

Chapter 6: Deep Feedforward Networks

Source chapter (free, official): https://www.deeplearningbook.org/contents/mlp.html

Core Idea

An MLP is a composed chain of parametric functions trained by gradient descent, where the choice of output unit follows from the assumed output distribution and the choice of hidden unit follows from gradient behaviour. Backpropagation is not learning; it is the chain rule scheduled well.

Key Concepts

  • Universal approximation: one sufficiently wide hidden layer can approximate any continuous function on a compact set. It says nothing about learnability, sample efficiency, or the size required — depth is how you get the same function with far fewer units.
  • Output units follow the distribution: linear + MSE → Gaussian; sigmoid + binary cross-entropy → Bernoulli; softmax + cross-entropy → categorical; mixture density → multimodal.
  • Hidden units: ReLU as the sane default; leaky/parametric ReLU, ELU, GELU/Swish (post-2016) for smoothness; sigmoid/tanh only where saturation is desired (gates).
  • Saturation: sigmoid/tanh gradients vanish in the tails, which is why they are poor hidden units and fine as gates.
  • Backpropagation: reverse-mode automatic differentiation over the computation graph. Cost is roughly one forward pass, memory is the stored activations.
  • Computation graph / autodiff: the abstraction every framework implements. Forward mode is cheap in inputs; reverse mode is cheap in outputs — losses are scalar, hence reverse.

Mental Models

  • Pick the output unit from the likelihood (Ch 3), then pick the loss as its negative log. Every "which loss?" question reduces to "which distribution?"
  • Read ReLU's advantage as gradient preservation, not nonlinearity per se: it is piecewise linear, so the gradient through an active unit is exactly 1.
  • Treat activation memory as the real cost of depth in training: the backward pass needs the forward activations, which is why checkpointing trades compute for memory.

Anti-patterns

  • Citing universal approximation to justify a shallow model — the theorem allows an exponentially wide layer, which is not an engineering plan.
  • Sigmoid hidden layers in a deep stack.
  • Pairing a sigmoid output with MSE: gradients vanish exactly where the model is most wrong. Use cross-entropy.

What changed after 2016

Smooth activations (GELU, SiLU/Swish) became the default in transformers; gated variants (GLU, SwiGLU) are now standard in large language model feedforward blocks. Residual connections (He et al. 2015) are mentioned here only in passing but became the structural default for every deep stack. Reverse-mode autodiff is unchanged. Confidence: high.

Key Takeaways

  1. Derive the loss from the output distribution rather than picking it by habit.
  2. Default to ReLU-family hidden units; reserve saturating units for gates.
  3. Budget activation memory as a first-class constraint, not an implementation detail.

Connects To

  • Ch 3: the distributions that determine output units.
  • Ch 8: why gradient preservation matters at depth.
  • scripts/model_arithmetic.py: parameter, FLOP and activation-memory accounting for a stack.