Haystack — AI Orchestration for Search & RAG

What it is

Haystack is an open-source Python framework by deepset for building AI applications focused on retrieval-augmented generation (RAG) and semantic search. It provides a pipeline architecture where you compose components (document stores, retrievers, generators, readers) into end-to-end workflows. Haystack handles the plumbing between components: data serialization, error propagation, and execution ordering.

The framework supports multiple LLM providers (OpenAI, Anthropic, Cohere, local models), multiple document stores (Elasticsearch, Qdrant, Pinecone, Weaviate, ChromaDB), and multiple embedding models. This flexibility lets you swap backends without rewriting pipeline logic.

How it saves time or tokens

Building a RAG pipeline from scratch involves connecting a document store, embedding model, retriever, prompt builder, and LLM generator. Each connection requires data transformation, error handling, and configuration. Haystack provides pre-built components for each step and a pipeline abstraction that handles data flow between them.

The framework also includes production-critical features that developers often forget to build: document deduplication, metadata filtering, hybrid retrieval (combining semantic and keyword search), and evaluation tools for measuring pipeline quality. These features prevent the common failure mode where a RAG system works in demos but fails in production due to poor retrieval quality.

How to use

1. Install Haystack:

```bash

pip install haystack-ai

```

1. Build a simple RAG pipeline:

```python

from haystack import Pipeline

from haystack.components.generators import OpenAIGenerator

from haystack.components.builders import PromptBuilder

prompt = PromptBuilder(

template='Answer: {{query}}\nContext: {{documents}}'

)

generator = OpenAIGenerator(model='gpt-4o')

pipeline = Pipeline()

pipeline.add_component('prompt', prompt)

pipeline.add_component('generator', generator)

pipeline.connect('prompt', 'generator')

```

1. Add a document store and retriever for full RAG.

Example

Complete RAG pipeline with document indexing and retrieval:

from haystack import Pipeline, Document
from haystack.document_stores.in_memory import InMemoryDocumentStore
from haystack.components.embedders import (
    OpenAIDocumentEmbedder,
    OpenAITextEmbedder,
)
from haystack.components.retrievers.in_memory import (
    InMemoryEmbeddingRetriever,
)
from haystack.components.generators import OpenAIGenerator
from haystack.components.builders import PromptBuilder

# 1. Index documents
doc_store = InMemoryDocumentStore()
embedder = OpenAIDocumentEmbedder()
docs = [
    Document(content='Haystack is a RAG framework by deepset.'),
    Document(content='Pipelines compose components into workflows.'),
]
embedded = embedder.run(documents=docs)
doc_store.write_documents(embedded['documents'])

# 2. Build query pipeline
query_pipeline = Pipeline()
query_pipeline.add_component('embedder', OpenAITextEmbedder())
query_pipeline.add_component('retriever',
    InMemoryEmbeddingRetriever(document_store=doc_store))
query_pipeline.add_component('prompt', PromptBuilder(
    template='Context: {{documents}}\nQuestion: {{query}}\nAnswer:'))
query_pipeline.add_component('generator', OpenAIGenerator())

query_pipeline.connect('embedder.embedding', 'retriever.query_embedding')
query_pipeline.connect('retriever', 'prompt.documents')
query_pipeline.connect('prompt', 'generator')

result = query_pipeline.run({
    'embedder': {'text': 'What is Haystack?'},
    'prompt': {'query': 'What is Haystack?'},
})
print(result['generator']['replies'][0])

Related on TokRepo

• AI Tools for RAG — Compare Haystack with other RAG frameworks and tools.
• AI Tools for Research — Explore research tools that use semantic search and retrieval.

Common pitfalls

• Confusing Haystack 1.x with 2.x APIs. Version 2.0 was a complete rewrite with a new pipeline architecture and component API. Tutorials and code from version 1.x will not work with 2.x. Always check that examples match your installed version.
• Using in-memory document stores for production. InMemoryDocumentStore is great for development but loses all data on restart. For production, use a persistent backend like Elasticsearch, Qdrant, or Pinecone.
• Not evaluating retrieval quality. A RAG pipeline is only as good as its retriever. Haystack includes evaluation components to measure retrieval recall and answer quality. Use these before deploying to catch poor retrieval performance.