Open R1 — Fully Open Reproduction of DeepSeek-R1

Introduction

Open R1 is Hugging Face's initiative to create a fully open reproduction of the DeepSeek-R1 reasoning model. The project provides training scripts, curated datasets, and model checkpoints so that anyone can replicate and extend chain-of-thought reasoning capabilities without proprietary dependencies.

What Open R1 Does

• Provides open training recipes for reproducing DeepSeek-R1 reasoning capabilities
• Includes curated math and reasoning datasets for reinforcement learning from human feedback
• Implements GRPO (Group Relative Policy Optimization) for training reasoning models
• Publishes intermediate and final model checkpoints on Hugging Face Hub
• Offers evaluation scripts for benchmarking reasoning quality across standard tests

Architecture Overview

Open R1 uses a multi-stage training pipeline. The base model is first fine-tuned on curated reasoning traces using supervised learning, then further refined with GRPO, a variant of reinforcement learning that groups completions and scores them relative to each other. The training framework is built on top of the TRL (Transformer Reinforcement Learning) library, with DeepSpeed ZeRO-3 for distributed training across multiple GPUs.

Self-Hosting & Configuration

• Requires Python 3.10+ and PyTorch with CUDA support
• Install dependencies via pip from the project requirements
• Training configs are YAML files specifying model, dataset, and hyperparameters
• Multi-GPU training uses DeepSpeed with configurable ZeRO stages
• Model checkpoints can be pushed to or loaded from Hugging Face Hub

Key Features

• Fully open training pipeline with no proprietary components
• Reproducible GRPO training with documented hyperparameters and seeds
• Multi-stage pipeline: SFT distillation followed by reinforcement learning
• Compatible with any Hugging Face-supported base model architecture
• Community-driven dataset curation with quality filters and deduplication

Comparison with Similar Tools

• DeepSeek-R1 — The original closed-training model; Open R1 aims for comparable quality with full transparency
• Qwen-2.5 — Strong open model but training recipe is not fully documented; Open R1 emphasizes reproducibility
• LLaMA 3 — General-purpose open weights; Open R1 specifically targets chain-of-thought reasoning
• Sky-T1 — Another open reasoning reproduction; Open R1 benefits from Hugging Face infrastructure and community scale

FAQ

Q: What hardware is needed to train Open R1 models? A: The full training pipeline requires multiple A100 or H100 GPUs. Smaller-scale experiments can run on a single 80GB GPU with reduced batch sizes.

Q: Can I use Open R1 checkpoints for commercial purposes? A: Check the license on each checkpoint's Hugging Face model card, as licenses may vary by base model.

Q: How does GRPO differ from standard RLHF? A: GRPO eliminates the need for a separate reward model by scoring groups of completions against each other, simplifying the training setup.

Q: Are the training datasets included in the repository? A: Datasets are hosted on Hugging Face Hub and can be downloaded with the huggingface-cli tool.

Sources

• https://github.com/huggingface/open-r1
• https://huggingface.co/collections/huggingface/open-r1