What is Feature Selection?

Feature selection is the process of choosing which input variables, or features, a machine learning model uses, keeping those that help predict the target and removing ones that are irrelevant, redundant, noisy or unavailable at prediction time, so the model is simpler and less likely to learn misleading patterns.

Feature Selection: Definition and Methods

A churn dataset may hold dozens of columns: tenure, plan, number of complaints, a customer ID and a cancellation reason recorded after the customer left. Some columns help prediction, some add noise, and the cancellation reason gives away the answer. Feature selection decides which columns the model receives.

The scikit-learn guide to feature selection covers several families of methods, including univariate selection, which keeps the features with the highest scores on a statistical test, and recursive feature elimination, which trains a model, removes the least important features and repeats on smaller and smaller sets.

Approaches are also commonly grouped as follows:

filter methods, which score each feature on its own, such as by correlation or a statistical test, before any model is trained
wrapper methods, which compare subsets of features by training and scoring a model on each
embedded methods, where the model selects features as part of training, as with regularized linear models
rule-based exclusion, which removes identifiers and fields that would not exist at the moment of prediction

How is feature selection different from feature engineering?

Feature engineering creates or transforms inputs, such as turning a signup date into tenure in months.
Feature selection chooses among the inputs that exist, keeping some and removing others.
Dimensionality reduction, such as principal component analysis, is a third option: it replaces the original features with a smaller set of new combined ones.

Where does feature selection break in AI work?

Feature selection breaks when it is fitted on all the data before the train-test split. The scikit-learn guide to common pitfalls illustrates data leakage with exactly this case: selecting features on the full dataset and then splitting it produces a much higher accuracy than expected, because the selection already used the test records.

Selection also breaks when a field that will not exist at prediction time stays in. In CUBIG’s Syntitan, Ready for Model, which supports binary classification, leaves ID-like key columns and columns it identifies as giving away the answer out of the model inputs, and compares results before and after data preparation on the same evaluation records, as described in how to evaluate a churn prediction model. Choosing which business fields are available at the decision point still needs someone who knows the process.

Frequently asked questions

Why is feature selection important?

Fewer, more relevant features can make a model easier to train and explain, reduce noise, and lower the risk of learning from fields that leak the answer or will not be available when the model is used.

Should feature selection be done before or after the train-test split?

After. Selection should be fitted on training data only, ideally inside the cross-validation loop, so that the test records play no part in choosing features.

What is the difference between feature selection and dimensionality reduction?

Feature selection keeps a subset of the original features. Dimensionality reduction, such as principal component analysis, builds new features that combine the originals.