firecrawl/firecrawl
Firecrawl is an open-source web data extraction engine and context API designed for artificial intelligence agents and large language model applications. The platform allows developers to search the web, scrape complex web pages, and interact with dynamic sites by executing automated actions like clicks, scrolling, writing, and waiting. It converts unstructured web pages and uploaded documents such as PDF, DOCX, and XLSX into LLM-ready formats including clean Markdown, HTML, screenshots, and structured JSON. Core capabilities include autonomous agent retrieval through natural language prompts, full website crawling, asynchronous batch scraping, site mapping, and integration with third-party data providers via the Alexandria catalog.
The software serves AI engineers, automation developers, and data practitioners seeking programmatic web context. Using Firecrawl requires an active runtime environment and optionally an API key from firecrawl.dev, although keyless free-tier access is supported with IP-based rate limiting for scrape, search, and interact endpoints. Supported installation methods include running pip install firecrawl for Python environments, npm install firecrawl for Node.js projects, or go get github.com/firecrawl/firecrawl/apps/go-sdk for Go applications. Agents can also be initialized directly using npx -y firecrawl-cli@latest init --all --browser. The project is licensed under the AGPL-3.0 license. For another browser automation approach, compare Browser Use; its scope and workflow differ.
Data as of : 198099 stars · 10465 forks.