AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

basin

Build and troubleshoot Cloudflare Basin analytics workflows with Basin Pipelines, Basin Catalog, and Basin SQL. Use for streaming data into R2 Iceberg tables, managing catalogs, or querying those tables; also use for requests using the former Data Platform, Pipelines, R2 Data Catalog, or R2 SQL names.

Original source
cloudflare
License
Apache-2.0
Pinned revision
e44f580f9c4fa3bf268e284ae7706c7954ba31fd

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read cloudflare--basin at revision e44f580f9c4fa3bf268e284ae7706c7954ba31fd, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 17

references/pipelines/gotchas.md

Basin Pipelines Gotchas

Non-obvious failure modes (not well covered by docs). For current limits and error semantics, pull https://developers.cloudflare.com/basin-pipelines/platform/limits/index.md.

Events accepted but never appear (most common)

HTTP 200 / send() resolves, but no data in the sink. Causes:

  1. Schema validation failure — structured streams accept then silently drop invalid events during processing. Validate client-side (Zod) and monitor pipelinesUserErrorsAdaptiveGroups.
  2. First-flush warm-up — first data takes 3–7 minutes (warm-up + namespace/table creation) even with --roll-interval 10. Poll ≥5 min in tests.
  3. Roll interval not elapsed — default 300s.
  4. Silent sink failure — deleted bucket or expired token. Check recordsWritten > 0 but filesWritten = 0; inspect failure_reason via GET /pipelines/{id}.

Everything is immutable

Cannot modify stream schema, pipeline SQL, or sink config — delete and recreate. Use version naming (events_v1) and keep SQL in version control.

curl -X DELETE "$BASE_URL/pipelines/{id}" -H "Authorization: Bearer $API_TOKEN"
curl -X DELETE "$BASE_URL/sinks/{id}"     -H "Authorization: Bearer $API_TOKEN"
curl -X DELETE "$BASE_URL/streams/{id}"   -H "Authorization: Bearer $API_TOKEN"

Worker binding undefined (env.MY_STREAM)

  1. Use the stream ID, not pipeline ID, in wrangler.jsonc.
  2. Binding field is "stream" (June 2026); old "pipeline" still works.
  3. Redeploy after adding the binding.

REST API field names ≠ CLI flags

r2_data_catalog vs --type basin-catalog, table_name vs --table, token vs --catalog-token, and format is required in REST but implied in CLI. See configuration.md.

wrangler basin pipelines delete defaults to "no"

Non-interactive environments answer "no" automatically — use REST DELETE for CI/automation.

Behavioral Notes

  • __ingest_ts auto-added (TIMESTAMP, day-partitioned). Don't put it in your schema.
  • Sinks can't target existing tables — the sink creates its own. Use PySpark to write to existing tables.
  • JSON-only input — no Avro/Protobuf/CSV.
  • Naming: streams/sinks/pipelines use underscores; buckets use hyphens.
  • Metrics lag 5–10 min after creation.
  • Pipeline SQL is row-level only — no GROUP BY/aggregation/window functions (do aggregation in Basin SQL at query time). CTEs and UNNEST are supported.

Debug Checklist

  • [ ] Stream exists: wrangler basin pipelines streams list
  • [ ] Pipeline running (not initializing/failed): GET /pipelines/{id}, check failure_reason
  • [ ] SQL matches schema; sink token valid; bucket + catalog exist
  • [ ] Worker redeployed; binding uses stream ID under "stream"
  • [ ] Waited ≥5 min (first flush)
  • [ ] Sink metrics: filesWritten > 0; error metrics show no drops

See Also