Bias vs Variance: The Bias-Variance Tradeoff Explained with Code

By Pushpjeet Cholkar, 17 years of corporate experience, starting in databases and now working in cloud, AI and ML.

Last updated: October 12, 2026

Short answer: Bias is error from a model that is too simple, so it misses the real pattern (underfitting). Variance is error from a model that is too sensitive to its training data, so it learns noise (overfitting). Total error is bias² plus variance plus irreducible noise; you tune model complexity to minimize the sum on validation data.

Prefer to see it first? The interactive bias-variance video and practice games explain the same idea in 74 seconds with “too cold, too hot, just right” scenes.

Bias vs variance at a glance

What is bias in machine learning?

Bias is the gap between what your model predicts on average and the true value. Imagine training the same kind of model many times on different samples from the same source. If the average of all those models is still off, that systematic error is bias.

A straight line fit to a curved relationship is the classic example. It doesn’t matter how many samples you give it: a line can’t bend, so it is wrong in the same places every time. That is underfitting.

Note: this is statistical bias. It is different from fairness bias (a model treating groups unequally) and from the “bias” term (intercept) inside a neural network layer.

What is variance in machine learning?

Variance measures how much the model’s predictions change when you train it on a different sample. A very flexible model, such as a high-degree polynomial or a deep decision tree, can bend to pass through every training point, including the random noise. Train it on another sample and you get a quite different curve.

That model looks great on training data and disappoints on new data. That is overfitting.

What is the bias-variance tradeoff?

For squared-error loss, the expected prediction error at a point splits into three parts:

Expected error = Bias² + Variance + Irreducible noise

• Bias²: how far the average model is from the truth.
• Variance: how much individual models scatter around that average.
• Irreducible noise: randomness in the data itself. No model can remove it.

As you make a model more flexible, bias usually falls and variance usually rises. Total validation error therefore tends to follow a U shape: high on the simple end (bias), high on the complex end (variance), lowest somewhere in between. The “tradeoff” is choosing complexity, features and regularization to sit near the bottom of that U. The Elements of Statistical Learning (Hastie, Tibshirani, Friedman) covers the derivation in detail.

How to tell if your model has high bias or high variance

You don’t need the formula to diagnose a real model. You need two numbers: training score and validation score.

A learning curve (score vs training-set size) makes this clearer. With high bias, both curves flatten out early at a poor score and adding data does little. With high variance, there is a wide gap that narrows as data grows. scikit-learn’s learning_curve and validation_curve utilities plot both. Post 07 in this series covers learning curves in depth.

Python demo: measuring bias and variance

Bias and variance are defined over many possible training sets, so you can only measure them directly in a simulation where you know the true function. This script does exactly that with NumPy:

• True function: sin(2πx) plus Gaussian noise (standard deviation 0.3, so noise variance is 0.09).
• 200 training sets of 30 points each.
• Polynomial models of degree 1, 4 and 8, evaluated at 50 fixed test points inside the training range.

"""Bias-variance decomposition demo (NumPy only).
Fits polynomials of degree 1, 4 and 8 to 200 noisy training sets drawn
from the same true function, then measures bias^2, variance and test error
at fixed test points. Run: python3 bias_variance_demo.py
"""
import numpy as np

rng = np.random.default_rng(42)
NOISE_SD = 0.3
N_TRAIN, N_SETS = 30, 200
x_test = np.linspace(0.05, 0.95, 50)  # stay inside the training range

def true_f(x):
    return np.sin(2 * np.pi * x)

y_true = true_f(x_test)
print(f"{'degree':>6} {'bias^2':>8} {'variance':>9} {'noise':>6} {'expected MSE':>13} {'train MSE':>10}")
for degree in (1, 4, 8):
    preds = np.empty((N_SETS, x_test.size))
    train_mse = []
    for i in range(N_SETS):
        x = rng.uniform(0, 1, N_TRAIN)
        y = true_f(x) + rng.normal(0, NOISE_SD, N_TRAIN)
        model = np.polynomial.Polynomial.fit(x, y, degree)
        preds[i] = model(x_test)
        train_mse.append(np.mean((model(x) - y) ** 2))
    mean_pred = preds.mean(axis=0)
    bias2 = np.mean((mean_pred - y_true) ** 2)
    variance = np.mean(preds.var(axis=0))
    noise = NOISE_SD ** 2
    print(f"{degree:>6} {bias2:>8.3f} {variance:>9.3f} {noise:>6.3f} {bias2 + variance + noise:>13.3f} {np.mean(train_mse):>10.3f}")

Output (Python 3.13.5, NumPy 2.5.3, run on October 12, 2026):

degree   bias^2  variance  noise  expected MSE  train MSE
     1    0.154     0.022  0.090         0.266      0.264
     4    0.003     0.018  0.090         0.111      0.078
     8    0.004     0.442  0.090         0.536      0.063

How to read it:

• Degree 1 (a straight line): bias² is 0.154, much larger than its variance of 0.022. Training MSE (0.264) is close to expected test MSE (0.266). Classic high bias.
• Degree 4: bias² drops to 0.003 and variance stays low at 0.018. Lowest expected error, 0.111, which is close to the 0.090 noise floor.
• Degree 8: bias² stays tiny (0.004), but variance jumps to 0.442. Training MSE is the lowest of the three (0.063) while expected test error is the highest (0.536). Classic high variance.

Two lessons sit in those numbers. First, training error alone keeps falling as you add complexity, so it can’t pick your model. Second, no model got below the 0.090 noise line, which is exactly what the formula predicts. (If you push the degree higher or test outside the training range, variance grows explosively; that is why the test points stay inside it.)

How to fix high bias

• Add features that carry real signal (domain-driven features, interactions).
• Use a more flexible model (polynomial terms, tree ensembles, a larger network).
• Reduce regularization strength.
• Train longer if the model hasn’t converged.

How to fix high variance

• Get more training data, or use data augmentation where it makes sense.
• Add or strengthen regularization (L2/ridge, L1/lasso, dropout, weight decay).
• Simplify the model or drop noisy features.
• Use early stopping on a validation set.
• Use bagging-style ensembles such as random forests, which average away part of the variance.
• Use cross-validation to choose hyperparameters instead of a single lucky split.

Common mistakes

• Picking the model with the best training score. The demo shows training error keeps falling while test error rises.
• Throwing more data at an underfit model. It rarely moves the needle; change the model or features instead.
• Tuning on the test set. Then the test score no longer measures generalization. Keep a validation set (or cross-validation) for tuning and the test set for one final check.
• Ignoring data problems. Leakage, label noise and a validation set from a different distribution than training can all look like bias or variance problems.
• Confusing statistical bias with fairness bias. Both matter, but they are different problems with different fixes.

Frequently asked questions

What is the difference between bias and variance?

Bias is error from a model that is too simple to capture the real pattern, so it is wrong in the same way on every dataset. Variance is error from a model that is too sensitive to its training data, so its predictions change a lot when the training sample changes.

Is high bias underfitting and high variance overfitting?

Yes, in practice. High bias shows up as underfitting: training and validation errors are both high and close together. High variance shows up as overfitting: training error is low but validation error is much higher.

How do I tell if my model has high bias or high variance?

Compare training and validation scores. Both poor and close together points to high bias. Training much better than validation points to high variance. A learning curve, which plots both scores as the training set grows, makes the pattern easier to read.

Does more training data fix high bias?

Usually not. A model that is too simple stays too simple with more rows. More data mainly helps high variance. For high bias, add useful features, use a more flexible model or reduce regularization.

Can you have low bias and low variance at the same time?

You can get both low enough for the task, but not zero. Irreducible noise sets a floor on error, and for a fixed amount of data, making a model more flexible usually trades bias for variance. The goal is the lowest total validation error.

Does the bias-variance tradeoff apply to deep learning?

The idea of diagnosing errors with training and validation scores still applies. Very large neural networks can behave differently from the classic U-shaped curve, a pattern researchers call double descent, so treat the tradeoff as a diagnostic tool rather than a strict law.

Sources

All checked on October 12, 2026.

• scikit-learn: Underfitting vs. Overfitting example
• scikit-learn: Validation curves: plotting scores to evaluate models
• Google Machine Learning Crash Course: Overfitting
• Hastie, Tibshirani and Friedman: The Elements of Statistical Learning (bias-variance decomposition, model assessment and selection)
• NumPy: Polynomial.fit

Learn machine learning with guided practice. New to ML? Start with AI/ML fundamentals and the what to learn in AI/ML roadmap. For live instructor-led classes, see the live AWS and AI/ML training page. If you are in the United States and want online AWS and AI/ML training in the USA, that page covers the US offering. Working with LLMs next? Read RAG vs fine-tuning.

More machine learning guides and tutorials: Machine learning tutorials and guides.