AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

universal-scraping-architect

Use for web scraping, crawling, document extraction or API parsing pipelines that must validate what they extract. Builds the extractor with Firecrawl or local Python and checks output with validate_extraction.py.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--universal-scraping-architect at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 5

references/local-extraction-patterns.md

Local Extraction Patterns (BeautifulSoup + pandas)

Patterns for Mode 2 (local Python) and the local half of Mode 3 (hybrid). Use these when extracting from static HTML, local files (PDF/Excel/CSV), private/sensitive data, or simple pages where Firecrawl is overkill. See scripts/local_bs4_example.py.

Fetch safely

Use requests with an identifying User-Agent, a timeout, and bounded retries with backoff. Always call raise_for_status() and handle the specific exception types (HTTPError, ConnectionError, Timeout) rather than a bare except.

Select robustly, not brittly

Anchor on stable signals — id, data-* attributes, semantic tags — not deep positional chains. Anti-pattern: div > span > ul > li:nth-child(3) breaks the moment the layout shifts. Prefer soup.find("table", {"id": "..."}) or attribute selectors, and fail loudly (with a message telling the user to update the selector) when the target isn't found.

Parse tables with pandas

pd.read_html(str(table))[0] turns an HTML <table> into a DataFrame. Scope it to the specific table element you located with BeautifulSoup rather than letting pandas guess across the whole page.

Normalize then validate

  1. Column names → snake_case (strip().lower(), collapse whitespace, drop non-word chars) so downstream code is stable.
  2. Clean → strip string cells, drop fully-empty rows, coerce dtypes.
  3. Validate before saving → assert the frame is non-empty and every required column is present. Never blindly write an empty or malformed result to disk.

Choose the output format

  • CSV — flat/tabular data.
  • JSON — nested or typed structures (validate it with scripts/validate_extraction.py).
  • Markdown — clean prose destined for an LLM.

Privacy

Local processing keeps sensitive data on-machine — this is the safe default whenever the source is private. Don't ship local files to an external API just for convenience.

Authoritative Sources

  1. Beautiful Soup Documentation
  2. pandas.read_html — API reference
  3. Requests: HTTP for Humans
  4. WHATWG HTML Living Standard — Tabular data
  5. MDN: CSS selectors
  6. pandas — Working with missing data