Firecrawl MCP — Web Scraping Server for AI Agents

What it is

Firecrawl MCP is the official Model Context Protocol server for Firecrawl, a web scraping and crawling platform. It exposes Firecrawl capabilities as MCP tools that AI agents can call directly. Agents can scrape single pages, crawl entire sites, extract structured data, and search the web without custom scraping code.

This server targets developers building AI agents that need real-time web data. Whether your agent researches competitors, monitors prices, or gathers training data, Firecrawl MCP handles the scraping layer.

How it saves time or tokens

Without Firecrawl MCP, an agent must parse raw HTML, handle JavaScript rendering, manage rate limits, and deal with anti-bot measures. Firecrawl abstracts all of that into clean markdown or structured JSON output. The estimated token cost is around 2,500 tokens per session, but the time saved on manual scraping setup is significant. Batch scraping handles multiple URLs in one call.

How to use

1. Get a Firecrawl API key from firecrawl.dev.
2. Configure the MCP server in your Claude Desktop or agent config.
3. The agent can now call scraping tools like scrape_url, crawl_url, search, and extract.

{
  "mcpServers": {
    "firecrawl": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": {
        "FIRECRAWL_API_KEY": "your-api-key"
      }
    }
  }
}

Example

Once configured, an agent can scrape a page:

# Agent calls the scrape_url tool
result = await mcp.call('firecrawl', 'scrape_url', {
    'url': 'https://example.com/pricing',
    'formats': ['markdown', 'structured_data']
})

# Result contains clean markdown and extracted data
print(result['markdown'])  # Clean text without HTML noise
print(result['structured_data'])  # Extracted pricing tables

Batch scraping multiple URLs:

result = await mcp.call('firecrawl', 'batch_scrape', {
    'urls': [
        'https://example.com/page1',
        'https://example.com/page2',
        'https://example.com/page3'
    ],
    'formats': ['markdown']
})

Related on TokRepo

• Web scraping tools — More scraping and data extraction tools
• MCP integrations — Agent tool integrations

Common pitfalls

• A Firecrawl API key is required. The free tier has rate limits that may not suit high-volume crawling.
• JavaScript-heavy single-page applications may require the wait_for parameter to ensure content loads before extraction.
• Batch scraping is asynchronous. For large batches, you need to poll for completion rather than expecting immediate results.
• Structured data extraction works best when you provide a schema describing what fields to extract.
• Some websites block automated access. Firecrawl handles many anti-bot measures, but heavily protected sites may still return incomplete data.
• Review the official documentation before deploying to production to ensure compatibility with your specific environment and requirements.