Gradient boosting

Gradient boosting builds a classification or regression model as a sequence of small decision trees, added one at a time. Each new tree is trained to predict the residual errors of the combined model so far, effectively moving predictions a small step in the direction that reduces the overall loss, which is why the method is described as "gradient" boosting. The final prediction is the sum of all trees' contributions, each typically scaled down by a learning rate.

This differs from a random forest, which trains trees independently on random subsets and averages them; boosting trains trees sequentially and dependently, so each tree is deliberately biased toward the cases the ensemble currently gets wrong. Popular implementations, such as XGBoost, LightGBM and CatBoost, add regularization and engineering optimizations for speed on large tabular datasets.

Gradient boosting frequently produces the best accuracy of any method on structured, tabular data, and is a mainstay of prediction competitions and production risk, pricing and ranking models. It is more sensitive to hyperparameter tuning than a random forest, more prone to overfitting if the number of trees or learning rate is not controlled, and slower to train because trees are built one after another rather than in parallel.