# Hyperopt — Distributed Hyperparameter Optimization in Python

> Hyperopt uses Tree of Parzen Estimators and random search to efficiently optimize hyperparameters for machine learning models, with optional distributed execution via MongoDB.

## Install

Save as a script file and run:

# Hyperopt — Distributed Hyperparameter Optimization in Python

## Quick Use
```bash
pip install hyperopt
python -c "
from hyperopt import fmin, tpe, hp
best = fmin(
    fn=lambda x: (x - 2) ** 2,
    space=hp.uniform('x', -10, 10),
    algo=tpe.suggest,
    max_evals=100
)
print(best)
"
```

## Introduction
Hyperopt is a Python library for serial and parallel optimization over search spaces that may include real-valued, discrete, and conditional dimensions. Its Tree of Parzen Estimators (TPE) algorithm is widely used in machine learning to find optimal hyperparameter configurations faster than grid or random search.

## What Hyperopt Does
- Optimizes any black-box function over complex search spaces
- Implements TPE, random search, and adaptive TPE algorithms
- Supports conditional and nested hyperparameter definitions
- Distributes trials across workers via MongoDB for parallel search
- Stores trial history for analysis and warm-starting future runs

## Architecture Overview
Hyperopt separates the objective function, search space definition, and optimization algorithm. The fmin driver iterates by asking the algorithm (e.g., TPE) to suggest a point, evaluating the objective, and recording the result in a Trials object. For distributed operation, MongoTrials replaces the in-memory store with a MongoDB-backed queue that multiple workers consume.

## Self-Hosting & Configuration
- Install via pip; add pymongo for distributed trials
- Define search spaces using hp.uniform, hp.choice, hp.loguniform, etc.
- Set max_evals to control the total number of evaluations
- Use MongoTrials with a running MongoDB instance for parallel workers
- Launch hyperopt-mongo-worker processes on each machine

## Key Features
- TPE algorithm finds good configurations with fewer evaluations than grid search
- Expressive search space language with conditional parameters via hp.choice
- Trials object stores all results for post-hoc analysis and plotting
- Scales horizontally with MongoDB-backed distributed trials
- Lightweight dependency footprint suitable for any ML framework

## Comparison with Similar Tools
- **Optuna** — more modern API with pruning and dashboard; Hyperopt's TPE is well-established in academic literature
- **Ray Tune** — broader scope with scheduler integration; Hyperopt is simpler for single-machine use
- **Scikit-Optimize** — Bayesian optimization with Gaussian processes; Hyperopt's TPE handles categorical parameters more naturally
- **Nevergrad** — gradient-free optimization focused on numerical problems; Hyperopt is tuned for ML hyperparameters

## FAQ
**Q: What is TPE and why use it?**
A: Tree of Parzen Estimators models the search space probabilistically, sampling more from regions that produced good results. It is more sample-efficient than random search.

**Q: Can Hyperopt optimize neural network architectures?**
A: Yes. Use hp.choice to define conditional spaces that represent different layer configurations.

**Q: How do I resume an interrupted search?**
A: Pass a previously saved Trials object to fmin. It will continue from where it left off.

**Q: Does Hyperopt support early stopping?**
A: Not natively. Use Optuna or wrap your objective to raise an exception for unpromising trials.

## Sources
- https://github.com/hyperopt/hyperopt
- http://hyperopt.github.io/hyperopt/

---
Source: https://tokrepo.com/en/workflows/asset-9c1b4b6e
Author: Script Depot