Crawlee — Production Web Scraping for Node.js

What it is

Crawlee is a web scraping and browser automation library for Node.js built by Apify. It provides a unified interface for building production-grade crawlers using raw HTTP requests (Cheerio), headless browsers (Playwright or Puppeteer), or adaptive crawling that automatically switches between them.

Crawlee is designed for developers building data pipelines for AI and LLM systems, RAG applications, and training datasets. It handles proxy rotation, request queuing, automatic retries, and persistent storage so you can focus on data extraction logic.

How it saves time or tokens

Crawlee eliminates boilerplate code for proxy management, retry logic, and request queuing that every production crawler needs. Its adaptive crawling mode automatically picks the cheapest method (raw HTTP with Cheerio) when JavaScript rendering is not needed, falling back to Playwright only when required. This reduces compute costs and speeds up crawls. The built-in request queue with deduplication prevents wasted requests on already-visited pages, and automatic fingerprint rotation reduces blocking rates.

How to use

1. Create a new crawler project with the CLI scaffolding tool:

npx crawlee create my-crawler
cd my-crawler
npm start

1. Or add Crawlee to an existing project and write a crawler:

npm install crawlee playwright

1. Define your crawler with a request handler:

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    async requestHandler({ request, page, enqueueLinks }) {
        const title = await page.title();
        console.log(`${title} - ${request.url}`);
        await enqueueLinks({ globs: ['https://example.com/blog/**'] });
    },
});

await crawler.run(['https://example.com/blog']);

Example

Extracting structured data from product pages with Cheerio (no browser needed):

import { CheerioCrawler, Dataset } from 'crawlee';

const crawler = new CheerioCrawler({
    async requestHandler({ request, $ }) {
        const products = [];
        $('div.product-card').each((_, el) => {
            products.push({
                name: $(el).find('h2').text().trim(),
                price: $(el).find('.price').text().trim(),
                url: request.url,
            });
        });
        await Dataset.pushData(products);
    },
    maxRequestsPerCrawl: 100,
});

await crawler.run(['https://shop.example.com/products']);

Related on TokRepo

• Web scraping tools — More web scraping and data extraction tools curated on TokRepo.
• Automation tools — Browse automation frameworks for data pipelines and workflows.

Common pitfalls

• Using PlaywrightCrawler for every page wastes resources. Start with CheerioCrawler and only switch to browser-based crawling for JavaScript-heavy sites.
• Not setting maxRequestsPerCrawl can cause runaway crawls that scrape far more pages than intended. Always set a limit during development.
• Ignoring the built-in session pool leads to higher blocking rates. Enable session rotation when scraping sites with rate limits.