LLM Scraper — Turn Any Webpage into Structured Data Using LLMs

Introduction

LLM Scraper takes a different approach to web scraping: instead of writing CSS selectors or XPath queries that break when a page layout changes, you define a Zod or JSON schema for the data you want and let an LLM extract it from the rendered page. The result is structured, typed data that adapts to layout changes automatically.

What LLM Scraper Does

• Extracts structured data from any webpage using LLM understanding
• Accepts Zod schemas to define the shape of desired output
• Works with Playwright or Puppeteer for page rendering
• Returns typed JSON that matches your schema definition
• Handles complex pages with dynamic content and varied layouts

Architecture Overview

LLM Scraper renders the target page using a headless browser (Playwright or Puppeteer), captures the page content, and sends it to an LLM along with the user-defined schema. The LLM extracts the relevant data points and returns structured JSON that conforms to the schema. The library handles retries, validation, and type coercion.

Self-Hosting & Configuration

• Install via npm as a TypeScript/JavaScript library
• Configure your LLM provider (OpenAI, Anthropic, or compatible APIs)
• Define extraction schemas using Zod for type-safe results
• Optionally configure Playwright or Puppeteer browser settings
• Set retry policies and timeout values for robustness

Key Features

• Schema-driven extraction that adapts to page layout changes
• Type-safe results with Zod schema validation
• Works with any LLM provider via configurable API endpoints
• Playwright and Puppeteer support for rendering JavaScript-heavy pages
• Minimal code required compared to traditional scraping approaches

Comparison with Similar Tools

• Firecrawl — API-based web scraping for LLMs; LLM Scraper is a library you run locally with your own browser
• Crawl4AI — LLM-friendly crawling; LLM Scraper focuses on schema-driven structured extraction from single pages
• ScrapeGraphAI — AI scraping with graph pipelines; LLM Scraper is a simpler, schema-first approach
• Cheerio/Puppeteer — traditional DOM scraping; LLM Scraper eliminates the need for selectors entirely

FAQ

Q: Does it work with any website? A: Yes, since it renders the page in a real browser and uses LLM understanding rather than selectors, it works with virtually any website.

Q: Which LLM providers are supported? A: It supports OpenAI, Anthropic, and any provider with a compatible API.

Q: How accurate is the extraction? A: Accuracy depends on the LLM model used. Modern models achieve high accuracy for well-defined schemas.

Q: Is it suitable for large-scale scraping? A: It is designed for targeted extraction tasks. For high-volume crawling, pair it with a crawling framework.

Sources

• https://github.com/mishushakov/llm-scraper