AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

setting-up-a-data-warehouse-source

Guide the user through connecting a new data warehouse source — Postgres, MySQL, Stripe, Hubspot, MongoDB, Salesforce, BigQuery, Snowflake, and so on. Use when the user wants to "connect Stripe", "import data from Postgres", "add a new data source", "sync my warehouse tables", or wants to pick sync methods for each table. Walks through source-type discovery, credential validation, table discovery, per-table sync_type selection, and the final create call. Also covers picking a good prefix, what to do right after creation, and sending a source's tables to the user's own warehouse (Snowflake, BigQuery, Postgres, S3 and other destinations).

Original source
posthog
License
MIT
Pinned revision
cac7c80a9e5b73484d18eda1bffb47297c938e0c

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read posthog--setting-up-a-data-warehouse-source at revision cac7c80a9e5b73484d18eda1bffb47297c938e0c, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 2

SKILL.md · 24000

destination (is_posthog_warehouse in external-data-destinations-list), the source may write there by default, and attaching would stop its PostHog tables updating. If the user wants to keep them, call external-data-sources-destinations-update with that same list before step 6: it makes the warehouse an explicit link. Skip that call while any destination in external-data-destinations-list is paused, because it would delete a hidden link to that destination. Tell the user instead. When destination_ids is empty, tell the user the PostHog tables stop updating.

  • Tables with their own destination override keep it and ignore the new source link. Call external-data-schemas-destinations-retrieve for the source's tables. Those with inherits_from_source: false will not reach the new destination.
  1. Call external-data-destinations-add-sources with the source ids. A source that has already synced sends only new rows to the destination. Offer resync: true to copy the history, and say that a resync re-reads every row (including overridden tables that do not reach the new destination) and bills every destination the source writes to.

If a destination shows status: "failing" or "paused", run external-data-destinations-test-connection and relay the message of the first failed step. Redshift has no test: relay latest_error from external-data-destinations-retrieve instead. The test only reports. A paused destination stays paused until the user fixes the problem and saves it on the warehouse destinations page.

Important notes

  • Always validate creds with db-schema before create. The create endpoint will accept invalid creds and then fail asynchronously — the source appears in the list with status Error and no tables. Skipping the validation step just pushes the failure into the background.
  • Present the table list before creating. Large databases may have hundreds of tables. Don't auto-select them all — row counts and relevance matter for billing. Let the user opt in explicitly.
  • Don't invent schemas. Every entry in the schemas array must correspond to a real table from the db-schema response. You can't "also add an orders table" unless db-schema found one.
  • A new source syncs to the PostHog warehouse. That is the default and it needs no setup: a source nobody configured destinations for writes here and nowhere else. Some projects can also send a source's tables to their own database (external-data-sources-destinations-retrieve shows where a source writes, when the feature is on for the project). Setting that up is a separate step (see "Sending a source to the user's own warehouse") and never something to do as part of creating a source — adding a destination to a source that has already synced leaves it holding none of the history until the tables are resynced, and a resync bills every destination the source writes to.
  • Prefix is load-bearing. It's part of every HogQL query the user will ever write against these tables. Pick something short, descriptive, and not already taken.
  • Prefer the secure connect-link for any credentials. Use data-warehouse-source-connect-link so the user authenticates in their browser — the connect page renders the source's full connection form (OAuth and credential options alike) and stores the result without creating the source. Don't collect OAuth tokens or database passwords in chat; pass the credential_id reference to setup — source creation always happens through setup, not the UI. (An already-connected OAuth integration can also be passed directly via its id key, e.g. {"hubspot_integration_id": 123}.)
  • Webhooks are a separate step after create. Setting sync_type: "webhook" on a schema doesn't register the webhook — the create-webhook call does. Always follow create → create-webhook → webhook-info for webhook-type schemas, and never leave a webhook schema dangling without registration (it just won't receive events).
  • Webhook support is source-specific and sparse. Currently only Stripe implements WebhookSource. Don't promise webhooks for Hubspot, Salesforce, or Postgres — they'll use polling sync.
  • Row counts drive billing. Warehouse syncing is metered by rows synced. A chatty 500M-row events table synced hourly is very different from a 10k-row dimension table synced daily. Flag large tables and offer longer sync frequencies (sync_frequency: "24hour") as the default.