Scripts2026年3月29日·1 分钟阅读

Crawl4AI — LLM-Friendly Web Crawling

Open-source web crawler optimized for AI and LLM use cases. Extracts clean markdown, handles JavaScript-rendered pages, and supports structured data extraction.

Script Depot
Script Depot · Community
介绍

Crawl4AI is purpose-built for feeding web content into LLMs. It crawls pages, renders JavaScript, and outputs clean markdown — perfect for RAG pipelines, research agents, and AI-powered content analysis.

Best for: RAG data ingestion, AI research agents, content extraction, web scraping for LLMs Works with: Any LLM pipeline — LangChain, LlamaIndex, custom agents

Key Features

  • Markdown output — Clean, LLM-ready text extraction
  • JavaScript rendering — Handles SPAs and dynamic content
  • Structured extraction — CSS selectors, schema-based extraction
  • Chunking strategies — Topic-based, fixed-size, or semantic chunking
  • Media extraction — Images, links, metadata
  • Rate limiting — Built-in politeness and throttling
  • Async — Fast concurrent crawling

FAQ

Q: What is Crawl4AI? A: Open-source web crawler optimized for AI and LLM use cases. Extracts clean markdown, handles JavaScript-rendered pages, and supports structured data extraction.

Q: How do I install Crawl4AI? A: Check the Quick Use section above for step-by-step installation instructions. Most assets can be set up in under 2 minutes.

🙏

来源与感谢

Created by unclecode. Licensed under Apache 2.0. unclecode/crawl4ai — 30K+ GitHub stars

讨论

登录后参与讨论。
还没有评论，来写第一条吧。

相关资产

NAPI-RS — Build Node.js Native Addons in Rust

Write high-performance Node.js native modules in Rust with automatic TypeScript type generation and cross-platform prebuilt binaries.

Script Depot

Mamba — Fast Cross-Platform Package Manager

A drop-in conda replacement written in C++ that resolves environments in seconds instead of minutes.

Script Depot

Plasmo — The Browser Extension Framework

Build, test, and publish browser extensions for Chrome, Firefox, and Edge using React or Vue with hot-reload and automatic manifest generation.

Script Depot
◈首页🔍搜索👤我的