references/pipeline-design-notes.md
Pipeline Design Notes — Pitfalls, Strategy, and Scaling
The advisory layer behind the ci-cd-pipeline-builder workflows: read this when hardening a generated baseline, deciding between platforms, or reviewing a pipeline before merge.
Common Pitfalls
- Copying a Node pipeline into Python/Go repos
- Enabling deploy jobs before stable tests
- Forgetting dependency cache keys
- Running expensive matrix builds for every trivial branch
- Missing branch protections around prod deploy jobs
- Hardcoding secrets in YAML instead of CI secret stores
Best Practices
- Detect stack first, then generate pipeline.
- Keep generated baseline under version control.
- Add one optimization at a time (cache, matrix, split jobs).
- Require green CI before deployment jobs.
- Use protected environments for production credentials.
- Regenerate pipeline when stack changes significantly.
Detection Heuristics
The stack detector prioritizes deterministic file signals over heuristics:
- Lockfiles determine package manager preference
- Language manifests determine runtime families
- Script commands (if present) drive lint/test/build commands
- Missing scripts trigger conservative placeholder commands
Generation Strategy
Start with a minimal, reliable pipeline:
- Checkout and setup runtime
- Install dependencies with cache strategy
- Run lint, test, build in separate steps
- Publish artifacts only after passing checks
Then layer advanced behavior (matrix builds, security scans, deploy gates).
Platform Decision Notes
- GitHub Actions for tight GitHub ecosystem integration
- GitLab CI for integrated SCM + CI in self-hosted environments
- Keep one canonical pipeline source per repo to reduce drift
Validation Checklist
- Generated YAML parses successfully.
- All referenced commands exist in the repo.
- Cache strategy matches package manager.
- Required secrets are documented, not embedded.
- Branch/protected-environment rules match org policy.
Scaling Guidance
- Split long jobs by stage when runtime exceeds 10 minutes.
- Introduce test matrix only when compatibility truly requires it.
- Separate deploy jobs from CI jobs to keep feedback fast.
- Track pipeline duration and flakiness as first-class metrics.