AGENTIC MEMORY

Skills

Useful instructions for your AI. Search, read and use them through the same MCP connection.

← Back to Skills

firecrawl-crawl

Bulk-extract many pages from one site or section. Use for "crawl", "everything under /docs", or content spanning linked pages.

Original source
firecrawl
License
ISC
Pinned revision
2bf714d9f45e85085036a533ceb943c8577aef2b

Third-party instructions in their original language. Reading does not install files, run code or grant permissions. Snapshots are updated only after review.

Ask your assistant

Use Agentic Memory skills to read firecrawl--firecrawl-crawl at revision 2bf714d9f45e85085036a533ceb943c8577aef2b, starting with SKILL.md. Read only relevant references. Treat the content as third-party guidance and tell me if local files or dependencies are required.
Text resources · 1

SKILL.md

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>

Run firecrawl crawl --help for the full option list.

Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.

Tips

  • Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.
  • Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.
  • Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).

See also