DocETL — LLM-Powered Document Processing Pipelines

What it is

DocETL is a Python framework for building LLM-powered document processing pipelines using declarative YAML configuration. Developed at UC Berkeley, it provides operators like map, reduce, resolve, filter, and split that chain together into data processing workflows. Each operator can call LLMs for tasks like summarization, extraction, classification, and entity resolution.

DocETL targets researchers, data engineers, and developers who need to process large document collections with LLMs. Instead of writing custom Python scripts for each processing step, you define pipelines in YAML and DocETL handles execution, caching, and error recovery.

How it saves time or tokens

DocETL abstracts the boilerplate of LLM document processing. The map operator processes each document independently (parallelizable), reduce aggregates results, and resolve handles entity deduplication. Built-in caching avoids reprocessing unchanged documents. The declarative approach means you focus on what to extract rather than how to orchestrate API calls.

How to use

1. Install DocETL: pip install docetl.
2. Create a pipeline YAML file defining your datasets, operations, and output.
3. Run the pipeline: docetl run pipeline.yaml.

Example

# pipeline.yaml
datasets:
  papers:
    type: file
    path: 'papers.json'

operations:
  - name: summarize
    type: map
    prompt: |
      Summarize the following paper in 3 sentences:
      {{ input.text }}
    output:
      schema:
        summary: string

  - name: categorize
    type: map
    prompt: |
      Classify this paper into one category:
      {{ input.summary }}
    output:
      schema:
        category: string

  - name: aggregate
    type: reduce
    reduce_key: category
    prompt: |
      Summarize these papers in the {{ reduce_key }} category:
      {% for paper in inputs %}
      - {{ paper.summary }}
      {% endfor %}

pipeline:
  steps:
    - input: papers
      operations: [summarize, categorize, aggregate]
  output:
    type: file
    path: 'results.json'

Related on TokRepo

• Document AI Tools — Document processing and analysis tools
• RAG Tools — Retrieval and document understanding

Common pitfalls

• LLM costs scale with document collection size. A map operation over 10,000 documents means 10,000 LLM calls. Estimate costs before running on large collections.
• The resolve operator for entity deduplication requires careful prompt engineering. Ambiguous entity descriptions produce poor deduplication results.
• Pipeline YAML syntax errors fail at runtime, not at parse time. Validate your YAML structure before running on expensive datasets.