Overfitting

Overfitting happens when a model fits the training data so closely that it captures noise and coincidental patterns specific to that dataset, rather than the underlying relationship that would generalize to new data. The model looks highly accurate on the data it was trained on, but its performance drops sharply on data it has not seen.

It is diagnosed by comparing performance on the training set with performance on a held-out set, using a train-test split or cross-validation: a large gap, strong training performance but weak validation performance, is the signature of overfitting. It typically arises from a model that is too flexible relative to the amount of data available, for example a very deep decision tree or a neural network with too many parameters and too few training examples.

The opposite failure, underfitting, is when a model is too simple to capture the real pattern in the data at all. Common remedies for overfitting include gathering more training data, simplifying the model, adding regularization that penalizes complexity, stopping training early, and tuning model settings through hyperparameter tuning using validation performance, rather than training performance, as the guide.