Building more accurate decision trees with the additive tree
Open Access
- 16 September 2019
- journal article
- research article
- Published by Proceedings of the National Academy of Sciences in Proceedings of the National Academy of Sciences of the United States of America
- Vol. 116 (40), 19887-19893
- https://doi.org/10.1073/pnas.1816748116
Abstract
The expansion of machine learning to high-stakes application domains such as medicine, finance, and criminal justice, where making informed decisions requires clear understanding of the model, has increased the interest in interpretable machine learning. The widely used Classification and Regression Trees (CART) have played a major role in health sciences, due to their simple and intuitive explanation of predictions. Ensemble methods like gradient boosting can improve the accuracy of decision trees, but at the expense of the interpretability of the generated model. Additive models, such as those produced by gradient boosting, and full interaction models, such as CART, have been investigated largely in isolation. We show that these models exist along a spectrum, revealing previously unseen connections between these approaches. This paper introduces a rigorous formalization for the additive tree, an empirically validated learning technique for creating a single decision tree, and shows that this method can produce models equivalent to CART or gradient boosted stumps at the extremes by varying a single parameter. Although the additive tree is designed primarily to provide both the model interpretability and predictive performance needed for high-stakes applications like medicine, it also can produce decision trees represented by hybrid models between CART and boosted stumps that can outperform either of these approaches.Keywords
Funding Information
- HHS | NIH | National Institute of Biomedical Imaging and Bioengineering (K08 EB 026500 - 01A1)
This publication has 24 references indexed in Scilit:
- Intelligible Models for HealthCarePublished by Association for Computing Machinery (ACM) ,2015
- Fifty Years of Classification and Regression TreesInternational Statistical Review, 2014
- Improving the precision of classification treesThe Annals of Applied Statistics, 2009
- An empirical comparison of supervised learning algorithmsPublished by Association for Computing Machinery (ACM) ,2006
- Greedy function approximation: A gradient boosting machine.The Annals of Statistics, 2001
- Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors)The Annals of Statistics, 2000
- On the Boosting Ability of Top–Down Decision Tree Learning AlgorithmsJournal of Computer and System Sciences, 1999
- A Decision-Theoretic Generalization of On-Line Learning and an Application to BoostingJournal of Computer and System Sciences, 1997
- Induction of fuzzy decision treesFuzzy Sets and Systems, 1995
- A survey of decision tree classifier methodologyIEEE Transactions on Systems, Man, and Cybernetics, 1991