A churn model scores well on last quarter’s customers and noticeably worse on this quarter’s. The team must decide whether the model is too simple for the pattern or too closely fitted to its training sample.
Geman, Bienenstock and Doursat (1992) framed the bias-variance dilemma for neural networks. They described how estimation error can be decomposed into bias and variance, with incorrect models leading to high bias and model-free inference suffering from high variance.
A later paper by Belkin and colleagues (2019) describes a “double descent” curve in which test error can fall again after very flexible models pass the point where they fit the training data exactly. The classical tradeoff remains a useful diagnostic, though it does not predict every modern model’s behavior.
Common signs and responses include:
- High bias: Training and evaluation errors are both high, so teams may try more informative features or a more flexible model.
- High variance: Training error is low while evaluation error is much higher, so teams may try more data, regularization or a simpler model.
- A small gap with a poor score can point to missing information in the inputs that tuning alone will not fix.
- Cross-validation can give a steadier estimate of evaluation error than a single split.
How is the bias-variance tradeoff different from overfitting?
- Overfitting describes a model learning noise in the training sample, a high-variance outcome.
- Underfitting describes a model missing structure in the data, a high-bias outcome.
- The tradeoff is a framework that connects the two and explains why addressing one can worsen the other.
Where does the bias-variance tradeoff break in AI work?
A bias-variance diagnosis depends on an evaluation set that reflects real use. When evaluation data comes from a different period or population, a gap between training and evaluation scores can reflect model drift more than variance.
The diagnosis also becomes harder when data changes between comparisons. Teams that change features and model settings at the same time cannot tell which change moved the error, so evaluation records and settings should stay fixed while one factor changes. Data leakage can make training and evaluation scores look good together and hide both problems.