Unstructured — Document ETL for LLM Pipelines

What it is

Unstructured is an open-source library that extracts and transforms data from unstructured documents into clean, structured formats suitable for LLM processing. It handles PDFs, Word documents, HTML pages, images (via OCR), emails, and many other file types. The library auto-detects document types and applies the appropriate parsing strategy.

It targets developers building RAG (Retrieval-Augmented Generation) pipelines, knowledge bases, and any AI application that needs to ingest real-world documents.

How it saves time or tokens

Unstructured handles the messy work of document parsing that would otherwise require multiple specialized libraries. Instead of writing separate code for PDFs, Word docs, and HTML, you call one function. The output is chunked and cleaned for LLM consumption, reducing token waste from formatting artifacts, headers, and footers. For RAG pipelines, properly chunked documents mean better retrieval accuracy and lower token costs.

How to use

1. Install the library:

pip install unstructured

1. Parse any document type:

from unstructured.partition.auto import partition

# Auto-detect and parse any document
elements = partition(filename='report.pdf')

for element in elements:
    print(f'{type(element).__name__}: {str(element)[:100]}')

1. Use with specific document types:

from unstructured.partition.pdf import partition_pdf
from unstructured.partition.html import partition_html

# PDF with OCR for scanned documents
elements = partition_pdf('scanned_report.pdf', strategy='ocr_only')

# HTML page
elements = partition_html(url='https://example.com/article')

Example

from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title

# Parse a PDF report
elements = partition(filename='quarterly_report.pdf')

# Chunk for RAG ingestion
chunks = chunk_by_title(
    elements,
    max_characters=1000,
    combine_text_under_n_chars=200
)

# Each chunk is ready for embedding and vector storage
for chunk in chunks:
    print(f'Type: {chunk.category}')
    print(f'Text: {chunk.text[:200]}')
    print(f'Metadata: {chunk.metadata.to_dict()}')
    print('---')

Related on TokRepo

• AI tools for RAG -- RAG pipeline tools and frameworks
• AI tools for documents -- Document processing and analysis tools

Common pitfalls

• Some document types require extra system dependencies. PDF parsing needs poppler-utils and tesseract for OCR. Install them via your system package manager before using those features.
• The 'auto' strategy may not always choose the best parser. For production pipelines, specify the partition function explicitly (partition_pdf, partition_html) and set the strategy parameter.
• Large documents can produce thousands of elements. Use the chunking utilities to combine small elements and split large ones before sending to your LLM or vector database.