We apply the concept of structured sparsity to improve boosted decision trees with general loss functions. The existing approach to this problem is Friedman's gradient boosting procedure using a regression tree base learner. Although this method has led to many successful industrial applications, it suffers from several theoretical and practical drawbacks. By employing the idea of structured greedy search, we are able to design a regularized greedy forest procedure to address these issues. The resulting method constructs tree ensembles more effectively than gradient boosting, and achieves better performance on most datasets we have tested on. This work suggests that we can view boosted decision trees as procedures that construct high order nonlinear interactions through the concept of structured sparsity regularization, and this general view can guide us to design nonlinear learning algorithms that are more effective than existing methods.