Boosting

This article will introduce:

• What is boosting.
• Common boosting algorithms, including:
  • AdaBoost
  • Gradient Boosting

Boosting

Boosting usually refers to a family of algorithms that combine weak learners to a single strong learner. ¹

SPAM example

Take the example from Sunil’s article.

Consider the problem of classifying SPAM. We have serval rules for classifying:

1. It’s a SPAM if the email has only one image file.
2. It’s a SPAM if the email has only links.
3. It’s a SPAM if the email contains sentence like “You won a prize money of $XXXX”.
4. It’s not a SPAM if the source of email is in your contract list.
5. …

Obviously, these rules will not work very well individually. We call these rules weak learners.

To improve it, we combine the prediction of each weak learner. We can get the final prediction by voting. For example, if 7 of 10 weak learners think a email is SPAM, then we will predict it’s a SPAM. If only 1 of weak learners think a email is SPAM, we couldn’t reject that email.