Hyperopt — Distributed Hyperparameter Optimization in Python

Introduction

Hyperopt is a Python library for serial and parallel optimization over search spaces that may include real-valued, discrete, and conditional dimensions. Its Tree of Parzen Estimators (TPE) algorithm is widely used in machine learning to find optimal hyperparameter configurations faster than grid or random search.

What Hyperopt Does

• Optimizes any black-box function over complex search spaces
• Implements TPE, random search, and adaptive TPE algorithms
• Supports conditional and nested hyperparameter definitions
• Distributes trials across workers via MongoDB for parallel search
• Stores trial history for analysis and warm-starting future runs

Architecture Overview

Hyperopt separates the objective function, search space definition, and optimization algorithm. The fmin driver iterates by asking the algorithm (e.g., TPE) to suggest a point, evaluating the objective, and recording the result in a Trials object. For distributed operation, MongoTrials replaces the in-memory store with a MongoDB-backed queue that multiple workers consume.

Self-Hosting & Configuration

• Install via pip; add pymongo for distributed trials
• Define search spaces using hp.uniform, hp.choice, hp.loguniform, etc.
• Set max_evals to control the total number of evaluations
• Use MongoTrials with a running MongoDB instance for parallel workers
• Launch hyperopt-mongo-worker processes on each machine

Key Features

• TPE algorithm finds good configurations with fewer evaluations than grid search
• Expressive search space language with conditional parameters via hp.choice
• Trials object stores all results for post-hoc analysis and plotting
• Scales horizontally with MongoDB-backed distributed trials
• Lightweight dependency footprint suitable for any ML framework

Comparison with Similar Tools

• Optuna — more modern API with pruning and dashboard; Hyperopt's TPE is well-established in academic literature
• Ray Tune — broader scope with scheduler integration; Hyperopt is simpler for single-machine use
• Scikit-Optimize — Bayesian optimization with Gaussian processes; Hyperopt's TPE handles categorical parameters more naturally
• Nevergrad — gradient-free optimization focused on numerical problems; Hyperopt is tuned for ML hyperparameters

FAQ

Q: What is TPE and why use it? A: Tree of Parzen Estimators models the search space probabilistically, sampling more from regions that produced good results. It is more sample-efficient than random search.

Q: Can Hyperopt optimize neural network architectures? A: Yes. Use hp.choice to define conditional spaces that represent different layer configurations.

Q: How do I resume an interrupted search? A: Pass a previously saved Trials object to fmin. It will continue from where it left off.

Q: Does Hyperopt support early stopping? A: Not natively. Use Optuna or wrap your objective to raise an exception for unpromising trials.

Sources

• https://github.com/hyperopt/hyperopt
• http://hyperopt.github.io/hyperopt/