AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

universal-scraping-architect

Use for web scraping, crawling, document extraction or API parsing pipelines that must validate what they extract. Builds the extractor with Firecrawl or local Python and checks output with validate_extraction.py.

Original source
alirezarezvani
License
MIT
Pinned revision
e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

This skill also requires local files or dependencies. Remote reading alone may not be enough.

Ask your assistant

Use Agentic Memory skills to read alirezarezvani--universal-scraping-architect at revision e4a1dc3695908dd5a6c811cc76ab7aa4cf95f2f0, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 5

references/scraping-ethics-security.md

Scraping Ethics, Robots.txt, and Security

Guidelines for ethical, legal, and secure data collection. These rules apply across all three extraction modes and are the basis for the skill's proactive triggers.

Respect the Robots Exclusion Protocol

robots.txt (now standardized as RFC 9309) is the first thing to check. Parse it with the stdlib — no extra dependency required:

import urllib.robotparser
rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
allowed = rp.can_fetch("UniversalScrapingArchitect/1.0", target_url)

If a path is disallowed, stop and surface it to the user rather than scraping anyway.

Be a polite client

  • Send an honest, identifying User-Agent with contact info.
  • Set timeouts and a sane retry budget (exponential backoff, capped attempts).
  • Rate-limit: add delay between requests; never hammer an origin in a tight loop.
  • Prefer official APIs / sitemaps / data dumps over HTML scraping when they exist.

Legal & privacy posture

Scraping public data is generally permissible, but terms of service, copyright, and data-protection law (GDPR/CCPA) still apply. Personal data carries extra obligations. This skill's Private Data Leakage trigger fires when a user asks to send local/sensitive files to an external API — in that case prefer Mode 2 (local Python) so data never leaves the machine. None of this is legal advice; flag uncertainty and route the user to counsel for high-stakes collection.

Security of the pipeline itself

  • Secrets: API keys via environment variables only (the Hardcoded API Keys trigger). Never log the key value.
  • Untrusted input: scraped HTML/markdown is untrusted. Never eval/exec it, and validate before writing or passing downstream.
  • SSRF / scope: validate target URLs; don't let a crawl wander into internal network ranges.

Authoritative Sources

  1. The Robots Exclusion Protocol (RFC 9309)
  2. OWASP Automated Threats to Web Applications
  3. Scraping Ethics Best Practices (Ethical Web Scraping)
  4. Legal Aspects of Web Scraping (Lexology)
  5. Cloudflare Bot Management Overview
  6. Python urllib.robotparser (stdlib)