Cette page est affichée en anglais. Une traduction française est en cours.
ScriptsMar 29, 2026·1 min de lecture

Crawl4AI — LLM-Friendly Web Crawling

Open-source web crawler optimized for AI and LLM use cases. Extracts clean markdown, handles JavaScript-rendered pages, and supports structured data extraction.

Script Depot
Script Depot · Community
Introduction

Crawl4AI is purpose-built for feeding web content into LLMs. It crawls pages, renders JavaScript, and outputs clean markdown — perfect for RAG pipelines, research agents, and AI-powered content analysis.

Best for: RAG data ingestion, AI research agents, content extraction, web scraping for LLMs Works with: Any LLM pipeline — LangChain, LlamaIndex, custom agents

Key Features

  • Markdown output — Clean, LLM-ready text extraction
  • JavaScript rendering — Handles SPAs and dynamic content
  • Structured extraction — CSS selectors, schema-based extraction
  • Chunking strategies — Topic-based, fixed-size, or semantic chunking
  • Media extraction — Images, links, metadata
  • Rate limiting — Built-in politeness and throttling
  • Async — Fast concurrent crawling

FAQ

Q: What is Crawl4AI? A: Open-source web crawler optimized for AI and LLM use cases. Extracts clean markdown, handles JavaScript-rendered pages, and supports structured data extraction.

Q: How do I install Crawl4AI? A: Check the Quick Use section above for step-by-step installation instructions. Most assets can be set up in under 2 minutes.

🙏

Source et remerciements

Created by unclecode. Licensed under Apache 2.0. unclecode/crawl4ai — 30K+ GitHub stars

Discussion

Connectez-vous pour rejoindre la discussion.
Aucun commentaire pour l'instant. Soyez le premier à partager votre avis.

Actifs similaires

Unkey — Open-Source API Key Management Platform

Unkey is an open-source API key management platform that lets developers create, validate, and rate-limit API keys with sub-millisecond verification and usage-based billing support.

Script Depot

Flagsmith — Open-Source Feature Flags and Remote Config

Flagsmith is a self-hosted feature flag and remote configuration service with a web dashboard and SDKs for 18+ languages, enabling teams to manage feature rollouts and A/B tests without redeploying.

Script Depot

OpenStatus — Open-Source Monitoring and Status Page Platform

OpenStatus is an open-source uptime monitoring and status page platform that checks endpoints from multiple regions, tracks latency and availability, and serves beautiful public status pages for your services.

Script Depot
◈Accueil🔍Rechercher👤Moi