{"id":146,"date":"2018-08-11T08:37:52","date_gmt":"2018-08-11T06:37:52","guid":{"rendered":"http:\/\/blog.hwr-berlin.de\/codeandstats\/?p=146"},"modified":"2020-09-08T07:28:32","modified_gmt":"2020-09-08T05:28:32","slug":"variable-importance-in-random-forests","status":"publish","type":"post","link":"https:\/\/blog.hwr-berlin.de\/codeandstats\/variable-importance-in-random-forests\/","title":{"rendered":"Variable Importance in Random Forests"},"content":{"rendered":"
\n

<\/p>\n

\n

Variable Importance in Random Forests can suffer from severe overfitting<\/h1>\n<\/div>\n
\n

Predictive vs.\u00a0interpretational overfitting<\/h2>\n

There appears to be broad consenus that random forests rarely suffer from \u201coverfitting\u201d which plagues many other models. (We define overfitting<\/em> as choosing a model flexibility which is too high for the data generating process at hand resulting in non-optimal performance on an independent test set.) By averaging many (hundreds) of separately grown deep trees -each of which inevitably overfits the data – one often achieves a favorable balance in the bias variance tradeoff.
\nFor similar reasons, the need for careful parameter tuning also seems less essential than in other models.<\/p>\n

This post does not attempt to contribute to this long standing discussion (see e.g.\u00a0https:\/\/stats.stackexchange.com\/questions\/66543\/random-forest-is-overfitting<\/a>) but points out that random forests\u2019 immunity to overfitting is restricted to the predictions only and not to the default variable importance measure!<\/p>\n

We assume the reader is familiar with the basic construction of random forests which are averages of large numbers of individually grown regression\/classification trees. The random nature stems from both “row and column subsampling\u2019\u2019: each tree is based on a random subset of the observations, and each split is based on a random subset of candidate variables. The tuning parameter \u2013 which for popular software implementations has the default \\(\\lfloor p\/3 \\rfloor\\)<\/span> for regression and \\(\\sqrt{p}\\)<\/span> for classification trees \u2013 can have profound effects on prediction quality as well as the variable importance measures outlined below.<\/p>\n

At the heart of the random forest library is the CART algorithm which chooses the split for each node such that maximum reduction in overall node impurity is achieved. Due to the CART bootstrap row sampling, \\(36.8\\%\\)<\/span> of the observations are (on average) not used for an individual tree; those \u201cout of bag\u201d (OOB) samples can serve as a validation set to estimate the test error, e.g.:
\n\\[\\begin{equation}
\nE\\left( Y – \\hat{Y}\\right)^2 \\approx OOB_{MSE} = \\frac{1}{n} \\sum_{i=1}^n{\\left( y_i – \\overline{\\hat{y}}_{i, OOB}\\right)^2}
\n\\end{equation}\\]<\/span><\/p>\n

where \\(\\overline{\\hat{y}}_{i, OOB}\\)<\/span> is the average prediction for the \\(i\\)<\/span>th observation from those trees for which this observation was OOB.<\/p>\n

\n

Variable Importance<\/h3>\n

The default method to compute variable importance is the mean decrease in impurity<\/em> (or gini importance<\/em>) mechanism: At each split in each tree, the improvement in the split-criterion is the importance measure attributed to the splitting variable, and is accumulated over all the trees in the forest separately for each variable. Note that this measure is quite like the \\(R^2\\)<\/span> in regression on the training set.<\/p>\n

The widely used alternative as a measure of variable importance or short permutation importance<\/strong> is defined as follows:
\n\\[\\begin{equation}
\n\\label{eq:VI}
\n\\mbox{VI} = OOB_{MSE, perm} – OOB_{MSE}
\n\\end{equation}\\]<\/span><\/p>\n

(more pointers in https:\/\/stackoverflow.com\/questions\/15810339\/how-are-feature-importances-in-randomforestclassifier-determined<\/a>)<\/p>\n<\/div>\n

\n

Gini importance can be highly misleading<\/h3>\n

We use the well known titanic data set to illustrate the perils of putting too much faith into the Gini importance which is based entirely on training data – not on OOB samples – and makes no attempt to discount impurity decreases in deep trees that are pretty much frivolous and will not survive in a validation set.<\/p>\n

In the following model we include passengerID<\/em> as a feature along with the more reasonable Age, Sex and Pclass: randomForest(Survived ~ Age + Sex + Pclass + PassengerId, data=titanic_train[!naRows,], ntree=200,importance=TRUE,mtry=2)<\/code><\/p>\n

The figure below shows both measures of variable importance and surprisingly passengerID<\/em> turns out to be ranked number 2 for the Gini importance (right panel). This unexpected result is robust to random shuffling of the ID.<\/p>\n

The permutation based importance (left panel) is not fooled by the irrelevant ID feature. This is maybe not unexpected as the IDs shold bear no predictive power for the out-of-bag samples.<\/p>\n

\"\"<\/a><\/p>\n<\/div>\n

\n

Noise Feature<\/h3>\n

Let us go one step further and add a Gaussian noise feature, which we call PassengerWeight:<\/p>\n

titanic_train$PassengerWeight = rnorm(nrow(titanic_train),70,20)\nrf4 =randomForest(Survived ~ Age + Sex + Pclass + PassengerId + PassengerWeight, data=titanic_train[!naRows,], ntree=200,importance=TRUE,mtry=2)<\/code><\/pre>\n
\"\"<\/a><\/p>\n
Again, the blatant \u201coverfitting\u201d of the Gini variable importance is troubling whereas the permutation based importance (left panel) is not fooled by the irrelevant features. (Encouragingly, the importance measures for ID and weight are even negative!)<\/p>\n
In the remainder we investigate if other libraries suffer from similar spurious variable importance measures.<\/p>\n<\/div>\n<\/div>\n
\n
h2o library<\/h2>\n
Unfortunately, the h2o random forest implementation does not offer permutation importance:<\/p>\n
https:\/\/stackoverflow.com\/questions\/51584970\/permutation-importance-in-h2o-random-forest\/51598742#51598742<\/a><\/p>\n
Coding passenger ID as integer is bad enough:<\/p>\n
\"\"<\/a><\/p>\n
Coding passenger ID as factor makes matters worse:<\/p>\n
Let\u2019s look at a single tree from the forest:<\/p>\n
\"\"<\/a><\/p>\n
If we scramble ID, does it hold up?<\/p>\n
\"\"<\/a><\/p>\n<\/div>\n
\n
partykit<\/h2>\n
conditional inference trees are not being fooled by ID:<\/p>\n
\"\"<\/a><\/p>\n
And the variable importance in cforest<\/em> is indeed unbiased<\/p>\n
\"\"<\/a><\/p>\n<\/div>\n
\n
python\u2019s sklearn<\/h2>\n
Unfortunately, like h2o the python random forest implementation offers only Gini importance, but this insightful post offers a solution:<\/p>\n
http:\/\/explained.ai\/rf-importance\/index.html<\/a><\/p>\n<\/div>\n
\n
Gradient Boosting<\/h2>\n
Boosting is highly robust against frivolous columns:<\/p>\n
mdlGBM = gbm(Survived ~ Age + Sex + Pclass + PassengerId +PassengerWeight, data= titanic_train, n.trees = 300, shrinkage = 0.01, distribution = \"gaussian\")<\/code><\/pre>\n
\"\"<\/a><\/p>\n<\/div>\n
\n
Conclusion<\/h2>\n
Sadly, this post is 12 years behind:<\/p>\n
It has been known for while now that the Gini importance tends to inflate the importance of continuous or high-cardinality categorical variables:<\/p>\n
the variable importance measures of Breiman\u2019s original Random Forest method \u2026 are not reliable in situations where potential predictor variables vary in their scale of measurement or their number of categories.<\/p><\/blockquote>\n
(Strobl et al, 2007 Bias in random forest variable importance measures: Illustrations, sources and a solution<\/a>)<\/p>\n<\/div>\n
\n
Single Trees<\/h2>\n
I am still struggling with the extent of the overfitting. It is hard to believe that passenger ID could be chosen as a split point early<\/strong> in the tree building process given the other informative variables! Let us inspect a single tree<\/p>\n
\"\"<\/a><\/p>\n
##   rowname left daughter right daughter   split var split point status\n## 1       1             2              3      Pclass         2.5      1\n## 2       2             4              5      Pclass         1.5      1\n## 3       3             6              7 PassengerId        10.0      1\n## 4       4             8              9         Sex         1.5      1\n## 5       5            10             11         Sex         1.5      1\n## 6       6            12             13 PassengerId         2.5      1\n##   prediction\n## 1       <NA>\n## 2       <NA>\n## 3       <NA>\n## 4       <NA>\n## 5       <NA>\n## 6       <NA><\/code><\/pre>\n
This tree splits on passenger ID at the second level !! Let us dig deeper:<\/p>\n
The help page states<\/p>\n
For numerical predictors, data with values of the variable less than or equal to the splitting point go to the left daughter node.<\/p><\/blockquote>\n
So we have the 3rd class passengers on the right branch. Compare subsequent splits on (i) sex, (ii) Pclass and (iii) passengerID:<\/p>\n
Starting with a parent node Gini impurity of 0.184<\/p>\n
Splitting on sex yields a Gini impurity of 0.159<\/p>\n\n\n\n\n\n\n
<\/th>\n1<\/th>\n2<\/th>\n<\/tr>\n<\/thead>\n
0<\/td>\n72<\/td>\n303<\/td>\n<\/tr>\n
1<\/td>\n71<\/td>\n50<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Splitting on passengerID yields a Gini impurity of 0.183<\/p>\n\n\n\n\n\n\n
<\/th>\nFALSE<\/th>\nTRUE<\/th>\n<\/tr>\n<\/thead>\n
0<\/td>\n2<\/td>\n373<\/td>\n<\/tr>\n
1<\/td>\n3<\/td>\n118<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
And how could passenger ID accrue more importance than sex ?<\/p>\n
\"\"<\/a><\/p>\n
\n<\/div>\n<\/div>\n