AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

universal-scraping-architect

Use for web scraping, crawling, document extraction or API parsing pipelines that must validate what they extract. Builds the extractor with Firecrawl or local Python and checks output with validate_extraction.py.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--universal-scraping-architect at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 5

references/firecrawl-technical-guide.md

Firecrawl Technical Guide

Technical integration patterns for the Firecrawl API within the Universal Scraping Architect (Mode 1: API-Driven). Use this when the source is a public URL, a JS-heavy SPA, requires search-first discovery, or involves bulk crawling across a domain.

Authentication (BYOK)

Firecrawl uses a Bring-Your-Own-Key model. The key is always read from the FIRECRAWL_API_KEY environment variable — never hardcoded, never committed. The free tier is sufficient for evaluation and small jobs, which keeps this skill compliant with the repo's "no paid commercial dependency" rule.

import os
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key=os.getenv("FIRECRAWL_API_KEY"))

Choosing the right operation

  • scrape — single URL → clean markdown/HTML/structured JSON. Cheapest; one credit per page. Default for "give me this page."
  • crawl — follow links across a domain up to a depth/limit. Credit cost scales with pages; always set limit and maxDepth to bound spend.
  • map — fast URL discovery (sitemap-style) without full extraction. Use it first to scope a crawl before paying for full extraction.
  • search — query-first discovery when you don't yet have URLs.

Quota & rate-limit awareness

Estimate credits before launching bulk jobs (map → count URLs → multiply). Treat HTTP 429 / quota errors as expected: back off exponentially, log the quota state, and checkpoint progress so a re-run resumes rather than restarts. Firecrawl returns structured error payloads — surface the message rather than swallowing it.

Token-budget discipline

Extracted markdown frequently exceeds an LLM context window. Estimate tokens (chars / 4 as a cheap proxy) and chunk before passing downstream, reserving output tokens for the model's response. See scripts/firecrawl_example.py.

Output formats

Request only the formats you need (["markdown"] is smallest). Markdown for prose, structured-JSON extraction for tabular/typed data, HTML only when you need the DOM.

Authoritative Sources

  1. Firecrawl API Documentation
  2. Firecrawl SDK for Python
  3. REST API Design Best Practices (Microsoft)
  4. Handling API Rate Limits (Cloudflare)
  5. JSON Schema Standard
  6. Exponential Backoff and Jitter (AWS Architecture Blog)