DVC — Data Version Control for Machine Learning

What it is

DVC (Data Version Control) brings Git-like versioning to datasets, machine learning models, and ML pipelines. Large binary files are stored in remote storage (S3, GCS, Azure, SSH, HDFS) while lightweight pointer files (.dvc files) are tracked in Git. This gives you reproducible experiments, auditable model lineage, and the ability to switch between data versions with git checkout.

DVC targets ML engineers, data scientists, and teams that need to track data and model artifacts alongside code without bloating the Git repository.

How it saves time or tokens

Without DVC, teams resort to naming files model_v2_final_FINAL.pkl, storing data on shared drives with no versioning, or using custom scripts to manage artifacts. DVC integrates with Git to provide dvc push, dvc pull, and dvc checkout commands that work like their Git equivalents for data. Reproducing an experiment from six months ago is a git checkout plus dvc checkout away.

How to use

1. Install DVC: pip install dvc[s3] (or [gcs], [azure] for your storage backend).
2. Initialize in a Git repo: git init && dvc init, add a remote: dvc remote add -d storage s3://my-bucket/dvcstore.
3. Track data: dvc add data/train.csv, commit the .dvc file to Git, and push data: dvc push.

Example

# Initialize DVC in a Git repository
pip install dvc[s3]
git init && dvc init

# Configure remote storage
dvc remote add -d storage s3://my-bucket/dvcstore

# Track a large dataset
dvc add data/train.csv
git add data/train.csv.dvc .gitignore
git commit -m 'track training data'

# Push data to remote storage
dvc push

# On another machine, pull the data
git clone <repo>
dvc pull

# Create a reproducible pipeline
dvc run -n train -d data/train.csv -d train.py -o model.pkl \
  python train.py

Related on TokRepo

• Automation Tools -- ML pipeline and build tools
• Coding Tools -- Developer productivity libraries

Common pitfalls

• Forgetting to run dvc push after dvc add. The data is tracked locally but not uploaded to remote storage, so teammates cannot pull it.
• Not committing the .dvc file and updated .gitignore to Git. Without the pointer file in Git history, you lose the ability to reproduce that data version.
• Using DVC without configuring a remote for team projects. Local-only DVC tracking works for solo use but breaks collaboration.