OpenMythos — Open-Source Claude Architecture Reconstruction from Research

Introduction

OpenMythos is a theoretical open-source reconstruction of Claude's rumored Mythos architecture, built from first principles using publicly available research literature on looped transformers and related techniques. Created by kyegomez, it provides PyTorch and JAX implementations that explore what such an architecture might look like based on published papers and architectural patterns.

What It Does

• Implements a looped transformer architecture inspired by research literature on efficient deep networks
• Provides both PyTorch and JAX implementations for flexibility across research environments
• Offers configurable model components for experimenting with depth, width, and loop parameters
• Includes training scripts and example configurations for running experiments
• Enables researchers to test hypotheses about looped transformer designs at various scales

Architecture Overview

OpenMythos implements a transformer variant that uses weight-sharing loops to achieve effective depth without proportional parameter growth. The core module wraps a transformer block set that is iterated multiple times, with adaptive routing controlling how information flows through repeated passes. The design draws from published research on universal transformers and looped architectures, synthesizing these ideas into a cohesive implementation. Both PyTorch and JAX backends share the same architectural logic with framework-specific optimizations.

Self-Hosting & Configuration

• Clone the repository and install with pip in editable mode
• Requires PyTorch or JAX depending on chosen backend
• Configure model dimensions, loop counts, and training hyperparameters via YAML configs
• GPU recommended for training; CPU sufficient for small-scale inference experiments
• No external API keys or services required

Key Features

• Dual PyTorch and JAX implementations of the looped transformer concept
• Modular design allowing researchers to swap components and test variations
• Configurable loop depth and routing mechanisms for architecture exploration
• Training utilities with logging and checkpoint support
• Open-source codebase encouraging community contributions and experimentation

Comparison with Similar Tools

• Hugging Face Transformers — provides standard architectures; OpenMythos explores experimental looped designs
• x-transformers — offers transformer variants; OpenMythos focuses specifically on the looped architecture concept
• NanoGPT — minimal GPT training code; OpenMythos targets a different architectural paradigm entirely

FAQ

Q: Is this the actual Claude architecture? A: No. This is a theoretical reconstruction based on publicly available research. It is not affiliated with or endorsed by Anthropic.

Q: Can I train production models with this? A: It is a research tool for exploring architectural ideas. Production use would require significant additional engineering and validation.

Q: What hardware do I need? A: A single GPU is sufficient for small experiments. Larger runs benefit from multi-GPU setups with standard distributed training.

Sources

• https://github.com/kyegomez/OpenMythos