references/memex-principles.md
Memex Principles
Why the LLM Wiki pattern works, and why it failed for humans until LLMs.
Vannevar Bush's Memex (1945)
In "As We May Think", Bush described a personal knowledge store where:
- Documents are curated, not just searched
- Users build associative trails — named, reusable paths through the material
- The trails are as valuable as the documents
- The system is private and personal, not a public reference
This is almost exactly the LLM Wiki pattern. The difference: Bush had no one to do the bookkeeping.
Why humans abandon wikis
The value of a wiki grows linearly with its size. The maintenance burden grows faster. At some inflection point — usually around 50-100 pages — maintenance starts to feel like chores and the wiki goes stale.
Specific tasks that die first:
- Updating cross-references when a new page is added
- Keeping summary pages current
- Noticing when new data contradicts old claims
- Consolidating pages that have drifted apart
- Filing explorations back into the knowledge base
- Keeping the index current
Humans are great at reading, curating, and thinking about what things mean. They're bad at the bookkeeping. The bookkeeping is 80% of the work.
What changed with LLMs
LLMs don't get bored. They don't forget to update a cross-reference. They can touch 15 files in one pass without losing track. They cost near-zero per maintenance operation.
This changes the economics. The wiki stays maintained because maintenance is now free (or nearly). The human's job collapses to:
- Source curation — deciding what's worth reading
- Direction — asking good questions, steering analysis
- Judgment — deciding when a contradiction matters
- Taste — knowing when the synthesis is wrong
Everything else — the 80% that killed human wikis — is delegated.
Why not just RAG?
RAG retrieves fragments at query time and synthesizes from scratch every query. It works, but:
- No accumulation. Every subtle question re-derives the same synthesis.
- Cross-references are computed on demand. If the cross-reference needs 5 sources to be visible, you'd better hope all 5 are in the retrieval window.
- Contradictions are invisible. They surface only if you explicitly ask "is there a contradiction?"
- Explorations disappear. A comparison you worked out yesterday has to be re-derived tomorrow.
The wiki fixes all of these by compiling the knowledge once. The cross-references are already there. The contradictions have been flagged. The synthesis has absorbed everything read so far.
RAG is retrieve-then-think. The wiki is think-once-retrieve-many.
When the wiki stops being enough
At ~500-1000 pages, the index approach starts to creak. Options:
- Layer on search. Add
wiki_search.py(BM25) or an external tool like qmd (hybrid BM25 + vector). Both work alongside the index. - Shard by topic. Split into multiple vaults by domain.
- Add an MCP retrieval layer. Expose the wiki as a tool so agents can query it structurally.
The wiki and RAG are not opposites. The wiki is a compiled layer above RAG. You can run RAG on top of the wiki (indexing wiki/) and you'll get the benefits of both: pre-synthesized knowledge + scalable retrieval.
The human role
A common failure mode: users delegate curation to the LLM ("just ingest all my Pocket articles"). Don't. Curation is where human judgment lives. The LLM can help you decide whether to read a source, but you pick what makes it into raw/.
If you let the LLM ingest everything, the wiki fills with low-signal summaries and the synthesis becomes meaningless. The wiki's value is a direct function of the quality of raw/.
Reading recommendations
- Vannevar Bush, "As We May Think" (Atlantic, 1945)
- Andrej Karpathy's original LLM Wiki gist (linked from SKILL.md)
- Ousterhout's A Philosophy of Software Design — for why "deep modules" (well-summarized pages) beat shallow ones
- Niklas Luhmann's Zettelkasten — an earlier manual version of the same pattern