Feature Importance in Python

Recommended workflow

If you need a default, use permutation importance on held-out data. It works across model classes and ties the ranking to a scoring metric. Use built-in importances for fast model debugging, and use SHAP when you need to explain individual predictions or direction of contribution.

1. Train a model and confirm validation performance is good enough to explain.
2. Compute permutation importance on validation or test data.
3. Compare against model-specific importance for tree models.
4. Use SHAP if local explanations or contribution direction matter.
5. Document the metric, split, repeats, and limitations.

Permutation importance

Permutation importance scores the fitted model, shuffles one feature, scores again, and measures the score drop. It is slower than built-in tree importance, but easier to connect to the question most teams care about: what hurts validation performance?

from sklearn.inspection import permutation_importance
import pandas as pd

result = permutation_importance(
    model,
    X_valid,
    y_valid,
    scoring="roc_auc",
    n_repeats=30,
    random_state=42,
    n_jobs=-1,
)

importance = pd.DataFrame(
    zip(X_valid.columns, result.importances_mean, result.importances_std),
    columns=["feature", "mean", "std"],
).sort_values("mean", ascending=False)

print(importance.head(15))

The score is only meaningful relative to the metric. A feature can rank differently under AUC, log loss, accuracy, R squared, or mean absolute error.

Scikit-learn tree models

Tree estimators such as random forests expose feature_importances_. This is fast and useful for debugging, but it reflects split behavior inside the fitted trees.

from sklearn.ensemble import RandomForestClassifier
import pandas as pd

model = RandomForestClassifier(
    n_estimators=500,
    random_state=42,
    n_jobs=-1,
)
model.fit(X_train, y_train)

tree_importance = pd.Series(
    model.feature_importances_,
    index=X_train.columns,
).sort_values(ascending=False)

print(tree_importance.head(15))

XGBoost

XGBoost's scikit-learn API exposes feature_importances_, and the result depends on the model's importance_type. For tree models, common options include gain, weight, cover, total gain, and total cover.

from xgboost import XGBClassifier
import pandas as pd

model = XGBClassifier(
    n_estimators=300,
    learning_rate=0.05,
    max_depth=3,
    importance_type="gain",
    eval_metric="logloss",
    random_state=42,
)
model.fit(X_train, y_train)

xgb_importance = pd.Series(
    model.feature_importances_,
    index=X_train.columns,
).sort_values(ascending=False)

print(xgb_importance.head(15))

Use gain-like measures when you care about loss reduction. Use weight when you want to know how often a feature appears in splits. Do not mix them without naming the importance type.

LightGBM

LightGBM's scikit-learn API also exposes feature_importances_. The importance_type parameter controls whether the values represent split counts or total gains from splits.

from lightgbm import LGBMClassifier
import pandas as pd

model = LGBMClassifier(
    n_estimators=300,
    learning_rate=0.05,
    importance_type="gain",
    random_state=42,
)
model.fit(X_train, y_train)

lgbm_importance = pd.Series(
    model.feature_importances_,
    index=X_train.columns,
).sort_values(ascending=False)

print(lgbm_importance.head(15))

SHAP

SHAP is useful when a global ranking is not enough. It can show how feature values contribute to individual predictions, then aggregate those local contributions into global views.

import shap

explainer = shap.TreeExplainer(model)
shap_values = explainer(X_valid)

shap.plots.beeswarm(shap_values)

For classification models, check the shape and output being explained. You may need to select the class or output of interest before plotting.

Plotting

Keep feature-importance charts simple. Sort the values, use horizontal bars, show only the top features, and name the method in the axis label.

import matplotlib.pyplot as plt

top_n = 15
plot_data = importance.head(top_n).sort_values("mean")

fig, ax = plt.subplots(figsize=(8, 6))
ax.barh(
    plot_data["feature"],
    plot_data["mean"],
    xerr=plot_data["std"],
    color="#2563eb",
)
ax.set_title("Permutation importance")
ax.set_xlabel("Mean validation score decrease")
ax.set_ylabel("")
plt.tight_layout()
plt.show()

What to report

A ranking is not self-explanatory. Include enough detail for another practitioner to understand what the numbers mean and whether the result should be trusted.

• The model type and validation score.
• The importance method and metric.
• The data split used for the calculation.
• The number of permutation repeats, if applicable.
• Known correlated features, leakage risks, and excluded fields.

Sources: scikit-learn permutation importance, XGBoost Python API, LightGBM LGBMClassifier API, and SHAP TreeExplainer.