What is Hyperparameter Tuning?

A fraud team retrains a gradient-boosted model and tries twenty combinations of tree depth and learning rate. The team must choose the best-looking combination without letting the final test records influence its decision.

The scikit-learn guide to tuning hyper-parameters describes hyperparameters as parameters that are not directly learned within estimators. The guide documents GridSearchCV, which considers every combination in a grid, and RandomizedSearchCV, which samples a fixed number of candidates.

Bergstra and Bengio (2012) found that random search over the same domain found models as good as or better than grid search in a small fraction of the computation time in their experiments. Their explanation was that only a few hyperparameters matter for most datasets.

In CUBIG’s UCI Wilt experiment, published October 2026, each model’s configuration and the default 0.5 prediction threshold stayed fixed before and after data preparation. The comparison therefore measured the data change rather than a new tuning run, though a validation-selected threshold was not tested.

Common approaches include:

  • Grid search, which is exhaustive and becomes expensive as settings multiply
  • Random search, which samples settings and often covers the important ones faster
  • Sequential methods such as Bayesian optimization, which use earlier results to choose the next candidate
  • Nested cross-validation, which keeps tuning and final evaluation on separate data

How is hyperparameter tuning different from model training?

  • Training learns parameters such as weights or split points from the training data.
  • Tuning chooses the settings that control training, and each candidate setting requires its own training run.
  • Tuning decisions should use validation data, while the holdout set is kept for the final estimate.

Where does hyperparameter tuning break in AI work?

Tuning breaks when the test set guides the choice. The scikit-learn cross-validation guide notes that checking test-set performance while tuning parameters lets knowledge about the test set leak into the model, so evaluation metrics no longer report on generalization performance.

Tuning can also cloud comparisons between datasets. When a team retunes each model after changing the data, a better score can come from the new settings, the new data or both. Holding settings and evaluation records fixed while the data changes makes the effect easier to assess.

Frequently asked questions

What is the difference between a parameter and a hyperparameter?

A parameter, such as a weight, is learned from the training data. A hyperparameter, such as learning rate or tree depth, is set before training and controls how learning happens.

Is random search better than grid search?

In Bergstra and Bengio's 2012 experiments, random search found equally good or better models with much less computation time, mainly because only a few hyperparameters mattered for most datasets.

Should you tune hyperparameters on the test set?

No. Tune on validation data or cross-validation folds and keep the test set for the final estimate; otherwise, the reported score can overstate performance.