January 13, 2021
Boosting is a ensemble approach that generally employs Decision Trees, arranged in a consecutive a fashion, to make prediction. The output from all the Decision Trees is considered while making final prediction but more weightage is given to the tree that has higher performance i.e. the output from high performing trees are boosted(amplified) while aggregating the outputs to make the final prediction.
Like Bagging, Boosting also employs bootstrapping but the resampling in done is a strategic manner to ensure that the consecutive Decision Tree gets more informative training data. By more informative I mean that training instances against whom wrong predictions were made are highlighted. In other words, while building the next Decision Tree learnings from previous tree are incorporated.
A distinguishing factor of Boosting algorithm is that it combines weak predictors into a strong one. A weak predictor is that algorithm which performs slightly better than random guess. The concept of Boosting came as an answer to the question “Can a set of weak learners create a single strong learner?”. Note that in practical applications an algorithm with 80% accuracy would considered weak compared to one with 90% accuracy and thus become a suitable candidate for Boosting.
Here we’ve focused on Decision trees, but other algorithms can also be ensembled using Boosting.
by : Monis Khan
Quick Summary:
Boosting is a ensemble approach that generally employs Decision Trees, arranged in a consecutive a fashion, to make prediction. The output from all the Decision Trees is considered while making final prediction but more weightage is given to the tree that has higher performance i.e. the output from high performing trees are boosted(amplified) while aggregating […]