chapters/ch01-introduction.md
Chapter 1: Introduction
Source chapter (free, official): https://www.deeplearningbook.org/contents/intro.html
Core Idea
Deep learning is representation learning stacked: instead of hand-designing the features a model consumes, you let the model learn a hierarchy of representations, each layer expressed in terms of the one below it. Depth is the mechanism that makes composition cheap.
Key Concepts
- Representation learning: learning the features, not just the mapping from features to output.
- Depth as composition: a deep model reuses sub-computations, so a function needing exponentially many units at depth 1 may need polynomially many at depth k.
- The curse of dimensionality: as input dimension grows, the number of distinguishable configurations grows exponentially, so local-interpolation methods run out of neighbours.
- Prior of compositionality: deep learning bets that the data was generated by a composition of simple factors. This is a prior, not a theorem — where the bet is wrong, depth does not help.
- AI winters and rebrands: cybernetics → connectionism → deep learning; the ideas outlived three funding cycles because the constraint was data and compute, not concept.
Mental Models
- Think of each layer as re-coordinatizing the input so the next layer's job gets easier — not as "extracting meaning."
- Prefer depth over width when the target function plausibly factors; prefer width when it does not. Depth is a structural prior, and priors have to be earned.
- Treat "the model learns features automatically" as shifting where the human judgment goes (architecture, objective, data curation), not eliminating it.
Anti-patterns
- Reading this chapter as history and skipping it: the three growth curves it draws (dataset size, model size, benchmark accuracy) are the argument for why anything else works.
- Assuming depth is always better: without the compositional structure, extra depth buys optimization difficulty and nothing else.
What changed after 2016
The chapter's central claim — capability tracks dataset and model scale — held, and was made quantitative by neural scaling laws (Hestness et al. 2017; Kaplan et al. 2020; Hoffmann et al. 2022 "Chinchilla"), which the 2016 text predates. Its examples of "what deep learning can do" are now badly dated; its explanation of why depth helps is not. Confidence: high — the scaling-law literature is public and replicated.
Key Takeaways
- Before adding depth, state the compositional structure you believe the data has.
- Use the curse of dimensionality as the test for whether a nearest-neighbour-style baseline is even viable on your input.
- Read the growth curves as a resource argument: most 2016-era "deep learning doesn't work here" results were data or compute verdicts, not architecture verdicts.
Connects To
- Ch 5: the formal learning-theory machinery behind the claims made informally here.
- Ch 15: representation learning, promised here and delivered there.
- references/book_to_2026_delta.md: the scaling-law layer this chapter now needs.