Anubis — Stop AI Crawlers from Overwhelming Your Web Server

Introduction

Anubis is an open-source reverse-proxy that sits in front of your web server and challenges incoming requests with a lightweight proof-of-work puzzle. Legitimate browsers solve the challenge in milliseconds, while high-volume AI scrapers and crawlers are effectively throttled or blocked. It protects self-hosted services from being overwhelmed by automated traffic.

What Anubis Does

• Intercepts HTTP requests before they reach your origin server
• Issues proof-of-work challenges that browsers solve automatically
• Blocks or rate-limits bots that cannot complete the challenge
• Passes verified traffic transparently to the upstream backend
• Logs blocked and allowed requests for visibility

Architecture Overview

Anubis runs as a single Go binary acting as a reverse proxy. When a new request arrives, Anubis serves a small JavaScript challenge page. The browser computes a SHA-256 hash meeting a configurable difficulty target, submits it back, and receives a signed cookie granting access. Subsequent requests with a valid cookie pass through without re-challenge. Non-browser clients that cannot execute JavaScript are blocked.

Self-Hosting & Configuration

• Deploy via Docker image or compile from source with Go
• Set ANUBIS_TARGET to point at your backend service
• Configure challenge difficulty level to balance security vs. user wait time
• Supports allowlists for known-good bots (Googlebot, etc.)
• Place behind Nginx, Caddy, or any existing reverse proxy

Key Features

• Proof-of-work challenge blocks automated scrapers effectively
• Transparent to real users with JavaScript-capable browsers
• Single binary with minimal resource footprint
• Configurable difficulty and cookie expiry
• Open source with an active community and regular updates

Comparison with Similar Tools

• Cloudflare Bot Management — managed SaaS; Anubis is self-hosted and free
• BunkerWeb — WAF with broader scope; Anubis focuses specifically on PoW challenges
• fail2ban — IP-based blocking after detection; Anubis proactively challenges before access
• CrowdSec — reputation-based blocking; Anubis uses computational proof instead
• SafeLine — full WAF; Anubis is a lightweight single-purpose middleware

FAQ

Q: Does Anubis break search engine indexing? A: You can allowlist known search engine bots by user-agent or IP range so they bypass the challenge.

Q: How long does the challenge take for a real user? A: At default difficulty, browsers solve it in under one second. The delay is barely noticeable.

Q: Can Anubis run alongside Cloudflare or another CDN? A: Yes. Place Anubis between your CDN and your origin server.

Q: Does it work with API traffic? A: API clients that send a pre-shared token can be allowlisted. The PoW challenge is designed for browser-based access.

Sources

• https://github.com/TecharoHQ/anubis