Scrapy — Fast High-Level Web Crawling Framework for Python

What it is

Scrapy is the most battle-tested web scraping framework for Python. It handles concurrency, retries, throttling, cookies, and export pipelines, letting you write spiders that scale from one page to millions with the same code. You define how to follow links and extract data, and Scrapy handles the rest: scheduling, deduplication, middleware, and output formatting.

Scrapy targets data engineers, researchers, and developers who need structured data from websites. It is an asynchronous framework built on Twisted, capable of handling thousands of concurrent requests while respecting rate limits and site policies.

Why it saves time or tokens

Building a web scraper from scratch requires handling HTTP connections, retries, rate limiting, cookie management, and data storage. Scrapy provides all of this as configuration. You focus exclusively on the extraction logic. When using AI assistants to build scrapers, Scrapy's well-defined Spider class and Item/Pipeline pattern produce consistent, working code because the framework constraints reduce ambiguity.

How to use

1. Install Scrapy: pip install scrapy
2. Create a project: scrapy startproject myproject
3. Create a spider: scrapy genspider example example.com and define the parse method

Example

import scrapy

class ProductSpider(scrapy.Spider):
    name = 'products'
    start_urls = ['https://example.com/products']

    def parse(self, response):
        for product in response.css('div.product'):
            yield {
                'name': product.css('h2::text').get(),
                'price': product.css('span.price::text').get(),
                'url': product.css('a::attr(href)').get(),
            }

        next_page = response.css('a.next::attr(href)').get()
        if next_page:
            yield response.follow(next_page, self.parse)

Related on TokRepo

• AI tools for web scraping — web scraping tools and frameworks on TokRepo
• AI tools for automation — data collection automation

Common pitfalls

• Scrapy runs asynchronously; using blocking libraries (requests, time.sleep) inside spiders deadlocks the event loop
• Websites change their HTML structure; selectors break silently and return empty data rather than errors, so add validation in pipelines
• Aggressive crawling gets your IP blocked; always configure DOWNLOAD_DELAY and CONCURRENT_REQUESTS_PER_DOMAIN in settings