Open-source web crawler optimized for AI and LLM use cases. Extracts clean markdown, handles JavaScript-rendered pages, and supports structured data extraction.
这里应该同时让用户和 Agent 知道第一步该复制什么、安装什么、落到哪里。
pip install crawl4aifrom crawl4ai import AsyncWebCrawler
async with AsyncWebCrawler() as crawler:
result = await crawler.arun(url="https://example.com")
print(result.markdown) # Clean markdown contentCrawl4AI is purpose-built for feeding web content into LLMs. It crawls pages, renders JavaScript, and outputs clean markdown — perfect for RAG pipelines, research agents, and AI-powered content analysis.
Best for: RAG data ingestion, AI research agents, content extraction, web scraping for LLMs Works with: Any LLM pipeline — LangChain, LlamaIndex, custom agents
Created by unclecode. Licensed under Apache 2.0. unclecode/crawl4ai — 30K+ GitHub stars
Rust-powered Python linter and formatter. 10-100x faster than Flake8 + Black combined. 800+ lint rules, auto-fix, import sorting, and format-on-save.
Rust-powered Python package and project manager. 10-100x faster than pip. Drop-in replacement with lockfiles, virtual environments, and Python version management.
Turn any website into LLM-ready markdown. API-first web scraping with JavaScript rendering, auto-pagination, structured extraction, and batch crawling.