LLMLingua — Compress Prompts 20x with Minimal Loss

What it is

LLMLingua is a prompt compression tool from Microsoft Research. It reduces the token count of prompts sent to large language models by up to 20x while maintaining the quality of model outputs. It addresses the lost-in-the-middle problem in RAG pipelines where long contexts degrade retrieval accuracy.

Developers running RAG systems, long-context applications, or any LLM workflow where token costs matter will benefit from LLMLingua's compression capabilities.

How it saves time or tokens

LLMLingua directly reduces token consumption. A 10,000-token prompt compressed to 500 tokens saves API costs proportionally. For RAG pipelines, compression also improves result quality by removing redundant context that causes the model to lose focus on relevant passages.

How to use

1. Install LLMLingua via pip.
2. Load a small compression model (e.g., LLaMA-based).
3. Pass your prompt through the compressor before sending to the target LLM.

from llmlingua import PromptCompressor

compressor = PromptCompressor(
    model_name='microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank',
    use_llmlingua2=True,
)

original_prompt = 'Your very long prompt with lots of context...'
compressed = compressor.compress_prompt(
    original_prompt,
    rate=0.5,  # Keep 50% of tokens
    force_tokens=['important_keyword'],
)
print(f'Compressed from {compressed["origin_tokens"]} to {compressed["compressed_tokens"]} tokens')
print(compressed['compressed_prompt'])

Example

In a RAG pipeline, compress retrieved documents before sending them as context:

# Retrieved documents from vector search
documents = retrieve_relevant_docs(query)
context = '\n'.join(documents)

# Compress before sending to LLM
compressed = compressor.compress_prompt(context, rate=0.3)
response = llm.generate(f'{compressed["compressed_prompt"]}\n\nQuestion: {query}')

Related on TokRepo

• AI tools for RAG — Retrieval-augmented generation tools and techniques
• AI tools for content — Content processing and optimization

Common pitfalls

• Compression requires running a small model locally, which adds latency. Balance compression time against token cost savings.
• Aggressive compression rates (below 0.2) can remove critical information. Start with conservative rates and test output quality.
• The compression model needs GPU memory. For CPU-only environments, expect slower processing on long prompts.