ScrapeGraphAI — AI-Powered Web Scraping

What it is

ScrapeGraphAI is a Python web scraping library that uses large language models to extract structured data from websites. Instead of writing CSS selectors or XPath queries, you describe what you want in natural language and the LLM figures out how to extract it.

ScrapeGraphAI targets developers who need data extraction from websites without building custom scrapers for each site. It works with OpenAI, Anthropic, Google, and local models via Ollama.

How it saves time or tokens

Traditional scraping requires writing and maintaining selectors that break when sites change their HTML structure. ScrapeGraphAI abstracts this away -- the LLM adapts to different page layouts without code changes. The token_estimate for this workflow is approximately 500 tokens per extraction run.

How to use

1. Install ScrapeGraphAI and Playwright:

pip install scrapegraphai
playwright install

1. Create a SmartScraperGraph with your prompt and target URL.

1. Call .run() to get structured data back as a Python dictionary.

Example

from scrapegraphai.graphs import SmartScraperGraph

graph = SmartScraperGraph(
    prompt='Extract all article titles and their authors',
    source='https://news.ycombinator.com',
    config={'llm': {'model': 'openai/gpt-4o', 'api_key': 'sk-...'}}
)

result = graph.run()
print(result)
# [{'title': '...', 'author': '...'}, ...]

Related on TokRepo

• Web Scraping Tools -- More web scraping and data extraction solutions
• AI Tools for Research -- Research automation tools powered by AI

Common pitfalls

• ScrapeGraphAI requires Playwright for dynamic JavaScript-rendered pages. Without it, only static HTML is parsed.
• LLM token costs can add up when scraping many pages. Use local models via Ollama for high-volume extraction to reduce API costs.
• The quality of extraction depends heavily on the prompt specificity. Vague prompts like 'get everything' produce inconsistent results.