Thanos — Global Prometheus with Unlimited Retention and High Availability

What it is

Thanos extends Prometheus with global query capabilities, unlimited storage via object storage (S3, GCS, Azure), and high-availability replication. It is the proven way to run Prometheus at multi-cluster, multi-year scale without changing your existing Prometheus setup. Thanos runs as a set of sidecar and gateway components alongside your Prometheus instances.

Thanos is designed for SRE and platform teams running Prometheus across multiple clusters who need a unified query interface and long-term metric retention.

How it saves time or tokens

Prometheus alone has limited local storage and no built-in multi-cluster querying. Running Prometheus at scale requires either over-provisioning disk or losing old metrics. Thanos solves both problems: a sidecar uploads Prometheus blocks to cheap object storage for indefinite retention, and the Query component federates queries across all Prometheus instances. You keep your existing Prometheus setup and add Thanos components alongside it.

How to use

1. Run the Thanos sidecar next to each Prometheus instance:

thanos sidecar \
  --tsdb.path=/prometheus \
  --prometheus.url=http://localhost:9090 \
  --objstore.config-file=bucket.yml

1. Configure object storage in bucket.yml:

type: S3
config:
  bucket: thanos-metrics
  endpoint: s3.amazonaws.com
  access_key: '...'
  secret_key: '...'

1. Run the Thanos Query component for global queries:

thanos query \
  --store=sidecar-1:10901 \
  --store=sidecar-2:10901 \
  --store=store-gateway:10901

Access the global Prometheus UI at the Thanos Query endpoint.

Example

A complete Thanos deployment architecture:

Cluster A:  Prometheus + Thanos Sidecar ─┐
                                          ├─> Thanos Query ─> Grafana
Cluster B:  Prometheus + Thanos Sidecar ─┤
                                          │
Object Storage (S3) <── Sidecar uploads ──┘
         │
         └──> Thanos Store Gateway ──> Thanos Query
                (serves historical data)

Grafana points at Thanos Query as its Prometheus data source and gets a unified view across all clusters and time ranges.

Related on TokRepo

• Monitoring tools — Browse observability and monitoring tools
• DevOps tools — Explore infrastructure tooling

Common pitfalls

• Not deploying a Store Gateway for historical queries. Without it, Thanos Query can only reach live Prometheus instances. The Store Gateway serves data from object storage for long-term queries.
• Forgetting to configure compaction. Thanos Compact merges and downsamples historical blocks in object storage. Without it, storage costs grow linearly and queries over long time ranges slow down.
• Running Thanos Query without deduplication. If you run HA Prometheus pairs, enable --query.replica-label to deduplicate metrics from replica instances.
• Starting with an overly complex configuration instead of defaults. Begin with the minimal setup, verify it works, then customize incrementally. This approach catches configuration errors early and keeps troubleshooting straightforward.