# LocalGPT — Chat with Your Documents 100% Privately on Your Hardware

> Open-source tool that lets you ask questions about your documents using local LLMs with no data leaving your machine, powered by LangChain and GPU-accelerated inference.

## Install

Save in your project root:

# LocalGPT — Chat with Your Documents 100% Privately on Your Hardware

## Quick Use
```bash
git clone https://github.com/PromtEngineer/localGPT.git
cd localGPT
pip install -r requirements.txt
python ingest.py --source_directory /path/to/docs
python run_localGPT.py
```

## Introduction
LocalGPT is a privacy-focused document Q&A tool that runs entirely on your local hardware. It ingests documents, builds a vector store, and uses a locally running LLM to answer questions without sending any data to external servers. It is built on LangChain and supports multiple open-source models.

## What LocalGPT Does
- Ingests PDFs, CSVs, text files, and other document formats into a local vector database
- Chunks and embeds documents using HuggingFace sentence-transformer models
- Stores embeddings in Chroma for fast similarity-based retrieval
- Runs inference through quantized LLMs like Llama 2 using llama.cpp or HuggingFace
- Provides a Streamlit-based web UI for interactive question answering

## Architecture Overview
The system follows a retrieval-augmented generation pattern. Documents are split into chunks, embedded with a local sentence-transformer, and stored in a ChromaDB collection. When a user asks a question, the query is embedded, similar chunks are retrieved, and the combined context is passed to a locally running LLM to generate an answer.

## Self-Hosting & Configuration
- Requires Python 3.10+ and optionally a CUDA-capable GPU for faster inference
- Configure the embedding model and LLM in constants.py
- Supports GGUF-quantized models through llama-cpp-python for CPU or GPU inference
- Set SOURCE_DIRECTORY to point at your document folder before ingesting
- Run with the --device_type flag to choose between cpu, cuda, or mps backends

## Key Features
- Complete data privacy with zero external API calls during inference
- Supports multiple LLM backends including llama.cpp, HuggingFace, and GPTQ models
- Handles diverse file types through LangChain document loaders
- Includes both CLI and web-based interfaces for interaction
- GPU acceleration with CUDA and Apple Silicon MPS support

## Comparison with Similar Tools
- **PrivateGPT** — similar local-first RAG approach; LocalGPT adds a Streamlit UI and broader model support
- **Ollama + Open WebUI** — general local LLM chat; LocalGPT specializes in document retrieval
- **Quivr** — cloud-oriented RAG platform; LocalGPT stays fully offline
- **AnythingLLM** — desktop app with multi-provider support; LocalGPT is Python-native and more customizable

## FAQ
**Q: What hardware do I need?**
A: A machine with 16 GB RAM can run 7B-parameter models on CPU. A GPU with 8 GB VRAM speeds up inference significantly.

**Q: Which document formats are supported?**
A: PDF, CSV, TXT, DOCX, EPUB, HTML, and Markdown are supported through LangChain loaders.

**Q: Can I use it with my own fine-tuned model?**
A: Yes. Point the model path in constants.py to any HuggingFace or GGUF-compatible model.

**Q: Is there multi-user support?**
A: The default setup is single-user. For multi-user access, deploy the Streamlit app behind an authentication proxy.

## Sources
- https://github.com/PromtEngineer/localGPT
- https://localGPT.readthedocs.io

---
Source: https://tokrepo.com/en/workflows/asset-ce846b2a
Author: AI Open Source