You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For small datasets in singlemachine implementations, the split candidates for each continuous
72
+
For small datasets in single-machine implementations, the split candidates for each continuous
66
73
feature are typically the unique values for the feature. Some implementations sort the feature
67
74
values and then use the ordered unique values as split candidates for faster tree calculations.
68
75
69
-
Finding ordered unique feature values is computationally intensive for large distributed
70
-
datasets. One can get an approximate set of split candidates by performing a quantile calculation
71
-
over a sampled fraction of the data. The ordered splits create "bins" and the maximum number of such
72
-
bins can be specified using the `maxBins` parameters.
76
+
Sorting feature values is expensive for large distributed datasets.
77
+
This implementation computes an approximate set of split candidates by performing a quantile
78
+
calculation over a sampled fraction of the data.
79
+
The ordered splits create "bins" and the maximum number of such
80
+
bins can be specified using the `maxBins` parameter.
73
81
74
82
Note that the number of bins cannot be greater than the number of instances `$N$` (a rare scenario
75
83
since the default `maxBins` value is 100). The tree algorithm automatically reduces the number of
76
84
bins if the condition is not satisfied.
77
85
78
86
**Categorical features**
79
87
80
-
For `$M$` categorical feature values, one could come up with `$2^(M-1)-1$` split candidates. For
81
-
binary classification, we can reduce the number of split candidates to `$M-1$` by ordering the
88
+
For a categorical feature with `$M$` possible values (categories), one could come up with
89
+
`$2^{M-1}-1$` split candidates. For binary classification and regression,
90
+
we can reduce the number of split candidates to `$M-1$` by ordering the
82
91
categorical feature values by the proportion of labels falling in one of the two classes (see
83
92
Section 9.2.4 in
84
93
[Elements of Statistical Machine Learning](http://statweb.stanford.edu/~tibs/ElemStatLearn/) for
85
94
details). For example, for a binary classification problem with one categorical feature with three
86
-
categories A, B and C with corresponding proportion of label 1 as 0.2, 0.6 and 0.4, the categorical
87
-
features are ordered as A followed by C followed B or A, C, B. The two split candidates are A \| C, B
95
+
categories A, B and C whose corresponding proportions of label 1 are 0.2, 0.6 and 0.4, the categorical
96
+
features are ordered as A, C, B. The two split candidates are A \| C, B
88
97
and A , C \| B where \| denotes the split. A similar heuristic is used for multiclass classification
89
-
when `$2^(M-1)-1$` is greater than the number of bins -- the impurity for each categorical feature value
90
-
is used for ordering.
98
+
when `$2^{M-1}-1$` is greater than the `maxBins` parameter: the impurity for each categorical feature value
99
+
is used for ordering. In multiclass classification, all `$2^{M-1}-1$` possible splits are used
100
+
whenever possible.
101
+
102
+
Note that the `maxBins` parameter must be at least `$M_{max}$`, the maximum number of categories for
103
+
any categorical feature.
91
104
92
105
### Stopping rule
93
106
94
107
The recursive tree construction is stopped at a node when one of the two conditions is met:
95
108
96
-
1. The node depth is equal to the `maxDepth` training parameter
109
+
1. The node depth is equal to the `maxDepth` training parameter.
97
110
2. No split candidate leads to an information gain at the node.
98
111
99
112
### Max memory requirements
100
113
101
-
For faster processing, the decision tree algorithm performs simultaneous histogram computations for all nodes at each level of the tree. This could lead to high memory requirements at deeper levels of the tree leading to memory overflow errors. To alleviate this problem, a 'maxMemoryInMB' training parameter is provided which specifies the maximum amount of memory at the workers (twice as much at the master) to be allocated to the histogram computation. The default value is conservatively chosen to be 128 MB to allow the decision algorithm to work in most scenarios. Once the memory requirements for a level-wise computation crosses the `maxMemoryInMB` threshold, the node training tasks at each subsequent level is split into smaller tasks.
114
+
For faster processing, the decision tree algorithm performs simultaneous histogram computations for
115
+
all nodes at each level of the tree. This could lead to high memory requirements at deeper levels
116
+
of the tree, leading to memory overflow errors. To alleviate this problem, a `maxMemoryInMB`
117
+
training parameter specifies the maximum amount of memory at the workers (twice as much at the
118
+
master) to be allocated to the histogram computation. The default value is conservatively chosen to
119
+
be 128 MB to allow the decision algorithm to work in most scenarios. Once the memory requirements
120
+
for a level-wise computation cross the `maxMemoryInMB` threshold, the node training tasks at each
121
+
subsequent level are split into smaller tasks.
102
122
103
123
### Practical limitations
104
124
105
125
1. The implemented algorithm reads both sparse and dense data. However, it is not optimized for sparse input.
106
-
2. Python is not supported in this release.
126
+
2. Computation scales approximately linearly in the number of training instances,
127
+
in the number of features, and in the `maxBins` parameter.
107
128
108
129
## Examples
109
130
@@ -114,35 +135,101 @@ perform classification using a decision tree using Gini impurity as an impurity
114
135
maximum tree depth of 5. The training error is calculated to measure the algorithm accuracy.
0 commit comments