# Crawl4AI — LLM-Friendly Web Crawling

> Open-source web crawler optimized for AI and LLM use cases. Extracts clean markdown, handles JavaScript-rendered pages, and supports structured data extraction.

## Install

Save as a script file and run:

## Quick Use

```bash
pip install crawl4ai
```

```python
from crawl4ai import AsyncWebCrawler

async with AsyncWebCrawler() as crawler:
    result = await crawler.arun(url="https://example.com")
    print(result.markdown)  # Clean markdown content
```

---

## Intro

Crawl4AI is purpose-built for feeding web content into LLMs. It crawls pages, renders JavaScript, and outputs clean markdown — perfect for RAG pipelines, research agents, and AI-powered content analysis.

**Best for**: RAG data ingestion, AI research agents, content extraction, web scraping for LLMs
**Works with**: Any LLM pipeline — LangChain, LlamaIndex, custom agents

---

## Key Features

- **Markdown output** — Clean, LLM-ready text extraction
- **JavaScript rendering** — Handles SPAs and dynamic content
- **Structured extraction** — CSS selectors, schema-based extraction
- **Chunking strategies** — Topic-based, fixed-size, or semantic chunking
- **Media extraction** — Images, links, metadata
- **Rate limiting** — Built-in politeness and throttling
- **Async** — Fast concurrent crawling

---



### FAQ

**Q: What is Crawl4AI?**
A: Open-source web crawler optimized for AI and LLM use cases. Extracts clean markdown, handles JavaScript-rendered pages, and supports structured data extraction.

**Q: How do I install Crawl4AI?**
A: Check the Quick Use section above for step-by-step installation instructions. Most assets can be set up in under 2 minutes.

## Source & Thanks

> Created by [unclecode](https://github.com/unclecode). Licensed under Apache 2.0.
> [unclecode/crawl4ai](https://github.com/unclecode/crawl4ai) — 30K+ GitHub stars

---
Source: https://tokrepo.com/en/workflows/cb19c9d4-6c2a-4443-80eb-043a440d79eb
Author: Script Depot