GPT Crawler — Build Custom GPTs from Any Website

What it is

GPT Crawler is an open-source tool that crawls any website and converts its content into structured JSON files suitable for custom GPTs and RAG knowledge bases. Point it at documentation sites, help centers, or any public website, and it extracts clean text optimized for LLM consumption.

GPT Crawler is designed for developers and AI practitioners who want to create custom knowledge bases from existing web content without writing scrapers from scratch. It is a pure crawler with zero AI cost.

How it saves time or tokens

Building a knowledge base from a documentation site normally requires writing a custom scraper, handling pagination, cleaning HTML, and formatting output. GPT Crawler reduces this to a single command or config file. The maxTokens option lets you control output size to fit within GPT upload limits, and the CSS selector option targets specific content areas to avoid navigation and boilerplate noise.

How to use

1. Run directly with npx for quick crawls:

npx gpt-crawler --url https://docs.example.com --match 'https://docs.example.com/**'

1. For more control, clone and configure:

git clone https://github.com/BuilderIO/gpt-crawler.git
cd gpt-crawler && npm install

1. Edit the config and run:

// config.ts
export const config = {
  url: 'https://docs.example.com',
  match: 'https://docs.example.com/**',
  maxPagesToCrawl: 100,
  outputFileName: 'output.json',
  selector: '.main-content',
  maxTokens: 500000,
};

npm start

1. Upload output.json to OpenAI GPT Builder or feed it into your RAG pipeline.

Example

The output JSON structure looks like this:

[
  {
    "title": "Getting Started",
    "url": "https://docs.example.com/getting-started",
    "content": "This guide walks you through setting up..."
  },
  {
    "title": "API Reference",
    "url": "https://docs.example.com/api",
    "content": "The API accepts JSON requests on..."
  }
]

Related on TokRepo

• Web scraping tools — dedicated tools for extracting web data
• RAG resources — retrieval-augmented generation tools and frameworks

Common pitfalls

• Setting maxPagesToCrawl too high on large sites can result in multi-gigabyte output files; start with 50-100 pages and increase as needed
• Sites with client-side rendering (SPAs) may return empty content; GPT Crawler uses static fetching and does not execute JavaScript by default
• The match glob pattern must be precise; overly broad patterns will crawl unrelated sections of the site