Open-source web crawler optimized for AI and LLM use cases. Extracts clean markdown, handles JavaScript-rendered pages, and supports structured data extraction.
MCP server that gives AI agents web crawling superpowers. Crawl4AI MCP enables Claude Code and Cursor to scrape, extract, and process web content through tool calls.
Crawl4AI 0.5 is the async Python crawler for RAG. Outputs clean markdown, no HTML cleanup. Adaptive crawling, JS rendering, AsyncWebCrawler API. 30K stars.