Apache ShardingSphere — Distributed Database Middleware Ecosystem

Introduction

Apache ShardingSphere is a distributed database middleware ecosystem that transforms any database into a distributed database system. It provides data sharding, read-write splitting, data encryption, and shadow database testing without requiring application code changes, supporting MySQL, PostgreSQL, and other databases transparently.

What ShardingSphere Does

• Shards data horizontally across multiple database instances with configurable sharding strategies
• Splits read and write traffic automatically between primary and replica databases
• Encrypts sensitive columns transparently with pluggable encryption algorithms
• Provides shadow database routing for production-environment stress testing without affecting real data
• Offers distributed transaction support via XA and BASE transaction modes

Architecture Overview

ShardingSphere operates in two deployment modes. ShardingSphere-JDBC is a lightweight Java library that intercepts JDBC calls and routes SQL to the correct shards, running in-process with your application. ShardingSphere-Proxy is a standalone database proxy that speaks the MySQL or PostgreSQL wire protocol, acting as a transparent middleware layer. Both modes share the same sharding engine that parses SQL, rewrites queries for target shards, executes them in parallel, and merges results.

Self-Hosting & Configuration

• Choose JDBC mode (embedded library) for Java applications or Proxy mode for any language
• Define sharding rules in YAML specifying sharding columns, algorithms, and table mappings
• Configure read-write splitting rules with primary and replica data source definitions
• Set up data encryption rules mapping logical columns to cipher columns and algorithms
• Use DistSQL (ShardingSphere DDL) to manage rules dynamically via SQL statements at runtime

Key Features

• Pluggable sharding algorithms: hash, range, time-based, or custom strategies
• DistSQL for managing sharding, encryption, and splitting rules via SQL without config files
• Multi-database support: MySQL, PostgreSQL, openGauss, and SQL Server as backend databases
• Online schema migration for resharding data between old and new sharding configurations
• Cluster governance with ZooKeeper or etcd for distributed rule synchronization

Comparison with Similar Tools

• Vitess — MySQL sharding by PlanetScale; ShardingSphere supports more databases and offers encryption and shadow DB features
• Citus — PostgreSQL-native sharding extension; ShardingSphere works as middleware across multiple database types
• ProxySQL — MySQL proxy for query routing; ShardingSphere adds sharding logic, encryption, and distributed transactions
• MyCat — MySQL middleware; ShardingSphere has a larger community, active Apache governance, and richer feature set
• Native partitioning — Database-level partitioning; ShardingSphere distributes across separate instances for true horizontal scaling

FAQ

Q: Should I use JDBC mode or Proxy mode? A: JDBC mode has lower latency (no extra network hop) and suits Java applications. Proxy mode supports any language via standard database protocols and simplifies operations.

Q: Can ShardingSphere handle cross-shard joins? A: Yes, for simple joins. ShardingSphere supports cross-shard queries by fetching data from multiple shards and merging in memory. Complex joins may require binding table rules.

Q: Does ShardingSphere support distributed transactions? A: Yes. It supports XA transactions (via Atomikos or Narayana) and BASE transactions (via Seata) for cross-shard consistency.

Q: Can I migrate an existing database to ShardingSphere? A: Yes. The online migration feature can reshard data from a single database to a sharded topology with minimal downtime.

Sources

• https://github.com/apache/shardingsphere
• https://shardingsphere.apache.org/document/current/en/overview/