Scikit-learn guide
Feature Importance in Scikit-learn
Scikit-learn gives you several ways to estimate which inputs influenced a model. The two most common are impurity-based importance from tree models and permutation importance, which tests how much model performance drops when one feature is shuffled.
Quick example
For a fitted random forest, feature_importances_ returns one
score per input column. Higher values mean that feature was used more
often, and more effectively, to reduce split impurity across the forest.
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
import pandas as pd
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
X, y, random_state=42, stratify=y
)
model = RandomForestClassifier(random_state=42)
model.fit(X_train, y_train)
importance = pd.Series(
model.feature_importances_,
index=X.columns,
).sort_values(ascending=False)
print(importance.head(10))Which method should you use?
In scikit-learn, the practical default is simple: use built-in tree importance for fast debugging, and use permutation importance when you need a ranking you can explain to someone else.
| Method | Use it for | Be careful about |
|---|---|---|
feature_importances_ | Fast tree-model diagnostics | Bias toward some feature types and training-set behavior |
permutation_importance | Held-out, metric-based importance | Correlated features and unstable rankings with too few repeats |
| Linear coefficients | Interpretable linear models with scaled inputs | Comparing unscaled coefficients or treating coefficients as causal |
How it's calculated
The calculation depends on the method. This matters because the score is not a universal property of the feature. It is a property of a fitted model, a dataset, and an importance calculation.
Impurity-based tree importance
A tree chooses splits that reduce a criterion such as Gini impurity, entropy, or squared error. Each time a feature is used for a split, the tree records how much that split reduced impurity, weighted by how many samples reached the node. A random forest averages those reductions across all trees, then normalizes the values so the feature importances sum to 1.
This explains why the method is fast: the information is already produced while fitting the forest. It also explains the weakness: the score reflects training-time split behavior, not necessarily how much validation performance depends on the feature.
Permutation importance
First, score the fitted model on a baseline dataset, usually a validation or test set. Then choose one feature, randomly shuffle that column, and score the model again. The importance is the drop in score caused by breaking that feature's relationship to the target. Repeat the shuffle several times and average the drops.
This is slower than built-in tree importance because the model must be scored repeatedly. The tradeoff is that the result is tied to the metric and dataset you care about.
Tree model importance
Tree-based estimators such as random forests and gradient boosting
models can expose feature_importances_. This is fast and
convenient, but it measures the model's internal split behavior rather
than the real-world causal value of a feature.
Use with care
Impurity-based importance can favor high-cardinality or highly variable features. Correlated features can also split credit in ways that make an individually useful feature look less important.
Permutation importance
Permutation importance is usually the better first explanation to show another person. It asks a simple question: if we break the information in this feature, how much worse does the model get?
from sklearn.inspection import permutation_importance
result = permutation_importance(
model,
X_test,
y_test,
n_repeats=20,
random_state=42,
scoring="roc_auc",
)
permutation_scores = pd.Series(
result.importances_mean,
index=X.columns,
).sort_values(ascending=False)
print(permutation_scores.head(10))Because this uses held-out data, it is closer to the question most practitioners care about: which features matter to predictive performance on data the model did not train on?
Compare the rankings
Do not expect the two rankings to be identical. Built-in tree importance describes how the forest used features while fitting trees. Permutation importance describes how much the fitted model's validation score drops when a feature is broken.
comparison = pd.concat(
[importance.rename("tree_importance"), permutation_scores.rename("permutation")],
axis=1,
).sort_values("permutation", ascending=False)
print(comparison.head(10))Large disagreements are useful. They can point to correlated predictors, features that helped splits without improving validation performance, or a metric that does not match the question you are trying to answer.
Plot the results
A horizontal bar chart is usually the clearest way to show feature importance. Sort the values, show only the top features, and label the chart with the method so readers know what the numbers mean.
import matplotlib.pyplot as plt
top_n = 12
ax = importance.head(top_n).sort_values().plot.barh(
figsize=(8, 6),
color="#2563eb",
)
ax.set_title("Random forest feature importance")
ax.set_xlabel("Impurity-based importance")
ax.set_ylabel("")
plt.tight_layout()
plt.show()For permutation importance, include uncertainty from repeated shuffles.
plot_data = pd.DataFrame(
zip(X.columns, result.importances_mean, result.importances_std),
columns=["feature", "mean", "std"],
).sort_values("mean", ascending=False).head(top_n)
plot_data = plot_data.sort_values("mean")
fig, ax = plt.subplots(figsize=(8, 6))
ax.barh(
plot_data["feature"],
plot_data["mean"],
xerr=plot_data["std"],
color="#2563eb",
)
ax.set_title("Permutation importance")
ax.set_xlabel("Mean score decrease")
ax.set_ylabel("")
plt.tight_layout()
plt.show()If the error bars are large or the order changes across runs, treat the ranking as unstable. That is a result, not a charting problem.
How to interpret the result
- Importance is model-specific. A linear model, random forest, and boosted tree can rank the same columns differently.
- Importance is not causality. It says the model used a feature, not that changing the feature would change the outcome.
- Correlated features complicate ranking. Two similar features may share credit or hide each other's contribution.
- Always compare importance against validation performance. A bad model can still produce a neat-looking ranking.
What to report
A feature-importance chart is not enough by itself. Include the context that lets another practitioner judge whether the ranking is meaningful.
- The model type and validation score.
- The dataset split used to compute importance.
- The scoring metric, especially for permutation importance.
- The number of permutation repeats and whether rankings were stable.
- Known correlated features, leakage risks, and excluded columns.
Sources: scikit-learn permutation importance documentation and scikit-learn forest importance example.