Strange behavior in: Example 7 -- Sequential Feature Selection and GridSearch

[defaultusername0]

I'm running the code from the tutorial http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/#example-7-sequential-feature-selection-and-gridsearch

As in the example, I print the "best parameters" using:
print("Best parameters via GridSearch", gs.best_params_)_

However, the "best parameter" seems to change when changing the order of the values in the param_grid, e.g.:

[...]

param_grid = [
  {'sfs__k_features': [1, 2, 3, 4],
   'sfs__estimator__n_neighbors': [4, 3, 2, 1]} # order reversed
  ]

[...]

print("Best parameters via GridSearch", gs.best_params_)_

will yield

('Best parameters via GridSearch', {'sfs__k_features': 3, 'sfs__estimator__n_neighbors': 4})

instead of

('Best parameters via GridSearch', {'sfs__k_features': 3, 'sfs__estimator__n_neighbors': 1})

Is this a bug?

[rasbt]

Good question! Turns out it's not a bug but the dataset is a bit too simple and there are several different combinations that lead to the same "best" performance (here: 94% over the test folds). Hence, if there is a tie, it would automatically select the first instance with the highest performance.

You can show the more detailed results after fitting the GridSearchCV object as follows:

for i in range(len(gs.cv_results_['mean_test_score'])):
    print(gs.cv_results_['mean_test_score'][i], gs.cv_results_['params'][i]) 

0.9 {'sfs__estimator__n_neighbors': 4, 'sfs__k_features': 1}
0.9 {'sfs__estimator__n_neighbors': 4, 'sfs__k_features': 2}
0.94 {'sfs__estimator__n_neighbors': 4, 'sfs__k_features': 3}
0.93 {'sfs__estimator__n_neighbors': 4, 'sfs__k_features': 4}
0.9 {'sfs__estimator__n_neighbors': 3, 'sfs__k_features': 1}
0.9 {'sfs__estimator__n_neighbors': 3, 'sfs__k_features': 2}
0.94 {'sfs__estimator__n_neighbors': 3, 'sfs__k_features': 3}
0.93 {'sfs__estimator__n_neighbors': 3, 'sfs__k_features': 4}
0.9 {'sfs__estimator__n_neighbors': 2, 'sfs__k_features': 1}
0.9 {'sfs__estimator__n_neighbors': 2, 'sfs__k_features': 2}
0.94 {'sfs__estimator__n_neighbors': 2, 'sfs__k_features': 3}
0.93 {'sfs__estimator__n_neighbors': 2, 'sfs__k_features': 4}
0.9 {'sfs__estimator__n_neighbors': 1, 'sfs__k_features': 1}
0.9 {'sfs__estimator__n_neighbors': 1, 'sfs__k_features': 2}
0.94 {'sfs__estimator__n_neighbors': 1, 'sfs__k_features': 3}
0.93 {'sfs__estimator__n_neighbors': 1, 'sfs__k_features': 4}

As you can see, there are 3 0.94, and it would pick the first one in the list by default, i.e.,

0.94 {'sfs__estimator__n_neighbors': 4, 'sfs__k_features': 3}

However, if you flip the order sfs__estimator__n_neighbors': [4, 3, 2, 1]} to
sfs__estimator__n_neighbors': [1, 2, 3, 4]}, the parameter list will be reversed and the first in the list is then

0.94 {'sfs__estimator__n_neighbors': 1, 'sfs__k_features': 3}

Hope this makes sense! :)

[defaultusername0]

Thank you for your quick reply.
In the output you show, there is a reoccurring pattern in the accuracy score (0.9, 0.9, 0.94, 0.93) -- the score seems to depend only on k_features, but not on n_neighbors. I observe this pattern-behavior in many settings (e.g when adding extreme n_neighbors values in the param_grid) and also for larger, more complex datasets.

This leads me to believe that n_neighbors are not being taken into account correctly (?)

0.9 {'sfs__estimator__n_neighbors': 4, 'sfs__k_features': 1}
0.9 {'sfs__estimator__n_neighbors': 4, 'sfs__k_features': 2}
0.94 {'sfs__estimator__n_neighbors': 4, 'sfs__k_features': 3}
0.93 {'sfs__estimator__n_neighbors': 4, 'sfs__k_features': 4}

0.9 {'sfs__estimator__n_neighbors': 3, 'sfs__k_features': 1}
0.9 {'sfs__estimator__n_neighbors': 3, 'sfs__k_features': 2}
0.94 {'sfs__estimator__n_neighbors': 3, 'sfs__k_features': 3}
0.93 {'sfs__estimator__n_neighbors': 3, 'sfs__k_features': 4}

0.9 {'sfs__estimator__n_neighbors': 2, 'sfs__k_features': 1}
0.9 {'sfs__estimator__n_neighbors': 2, 'sfs__k_features': 2}
0.94 {'sfs__estimator__n_neighbors': 2, 'sfs__k_features': 3}
0.93 {'sfs__estimator__n_neighbors': 2, 'sfs__k_features': 4}

0.9 {'sfs__estimator__n_neighbors': 1, 'sfs__k_features': 1}
0.9 {'sfs__estimator__n_neighbors': 1, 'sfs__k_features': 2}
0.94 {'sfs__estimator__n_neighbors': 1, 'sfs__k_features': 3}
0.93 {'sfs__estimator__n_neighbors': 1, 'sfs__k_features': 4}

[rasbt]

hard to tell. if your classes are well separated, one would assume k wouldn't have a large influence on the classification results. I agree with your concern though, this looks very suspicious. I currently don't have time to look into that more closely, but I would say this is a potential bug until proven otherwise with a counter example.

[rasbt]

Oh, I think everything is ok and this was more of a misunderstanding. I see that you used both a KNN for feature selection and evaluation, which made this a bit harder to see.

So, the performance reported by the grid search is the performance on the test folds. This is a bit tricky to explain, but what happened in your case was basically is

• you have an inner KNN and an outer KNN. The inner KNN is used for feature selection, and the outer KNN is for the predictive performance.
• So via your grid, you changed the number of neighbors of the inner KNN, hence, the performance that is reported (the outer KNN) is always the same given that the same feature are selected.
• You would expect different results though for the same feature set if you were changing the number of neighbors in the outer KNN, too.

I.e, you had something like this:

param_grid = [
  {'sfs__k_features': [1, 2, 3, 4],
   'sfs__estimator__n_neighbors': [9, 6, 5, 3, 1],
   'knn__n_neighbors': [3]}
  ]

where the sfs__estimator neighbors only influence the selection of features.

You can try something like

param_grid = [
 {'sfs__k_features': [1, 2, 3, 4],
  'sfs__estimator__n_neighbors': [9, 6, 5, 3, 1],
  'knn__n_neighbors': [9, 6, 5, 3, 1]}
 ]

and would see then that this "pattern" should go away. If it makes sense to tune both the inner and the outer is another question though.

[defaultusername0]

Hi, thanks a lot for your help on this.

My aim is to use SFS+GridSearch to find the both the "optimal" k_features and n_neighbors hyperparameter for this dataset "in one go" -- while also taking into account that any choice of one might influence the optimal choice of the other.

To achieve this, if I understand correctly, would I have to use something along the lines of your second param_grid?

I guess I expected that the optimal k_features from the "inner" KNN would be passed up to the "outer" KKN, but that doesn't seem to be the case (?)

[rasbt]

My aim is to use SFS+GridSearch to find the both the "optimal" k_features and n_neighbors hyperparameter for this dataset "in one go" -- while also taking into account that any choice of one might influence the optimal choice of the other.

Yes, that would be accomplished via the parameter grid I posted

param_grid = [
{'sfs__k_features': [1, 2, 3, 4],
'sfs__estimator__n_neighbors': [9, 6, 5, 3, 1],
'knn__n_neighbors': [9, 6, 5, 3, 1]}
]

I guess I expected that the optimal k_features from the "inner" KNN would be passed up to the "outer" KKN, but that doesn't seem to be the case (?)

This is actually the case, the k_features are then passed to the "outer" KNN. The choice of n_neighbors is not passed on to the outer KNN though. The reason is that it gives you more flexibility this way. I.e., you could use different algorithms for inner and outer, say you could use a linear model for quick model selection in the SFS and then use a more expensive model as the "outer" one.

[MMron]

Hi @rasbt ,
I have run into the question “If it makes sense to tune both the inner and the outer is another question though”
Since I have generated more features than the original dataset I want to evaluate the predictive power of the model with these new features with the help of SFS. I want to do this within a nested cross validation in accordance with Varma, S. & Richard, S., 2006. Bias in error estimation when using cross-validation for model selection, Bethesda MD, USA: Biometric Research Branch, National Cancer Institute, as your book refers to.

Therefore, to my understanding, I need to apply the sfs as a preproccessing step, into the desired pipeline, whereas the pipeline is input to a gridsearch. Subsequently, a cross validation is applied to the gridsearch. As a result, 3 different cross validations are made. However, it would be many undesirable hyperparameter combinations since both the hyperparameters in the sfs estimator are being tuned, in addition to the hyper parameters in the estimator in the pipeline. These hyperparameters will rarely be in sync.
What are your thoughts on this? – Do you know of a more optimal solution?

A neat solution could be if both the estimator in sfs and the estimator in the pipline (e.i the last process) could have the same hyperparameters input at once.- I see this is also commented here #518,

[MMron]

A hardcoded solution could be to manually change the hyperparameters seperatly:
knn = KNeighborsClassifier(n_neighbors=2)

sfs1 = SFS(estimator=knn,
k_features=3,
forward=True,
floating=False,
scoring='accuracy',
cv=5)

pipe = Pipeline([('sfs', sfs1),
('knn', knn)])

param_grid = [{'sfs__k_features': [(1,4)],
'sfs__estimator__n_neighbors': [1],
'knn___neighbors': [1]},
{'sfs__k_features': [(1,4)],
'sfs__estimator__n_neighbors': [10],
'knn___n_neighbors': [10]}
]

gs = GridSearchCV(estimator=pipe,
param_grid=param_grid,
scoring='accuracy',
n_jobs=1,
cv=5,
refit=False)

gs = gs.fit(X_train, y_train)

[rasbt]

Hi there,

yeah, I agree that this will be expensive to tune 3 loops. The hardcoded approach you outlined should work though. I currently don't have time to implement the simultaneous tuning of the inner and outer estimator, but hopefully, I can get to it one day (possibly some time later this summer). That would be a nice improvement.

[Micha-93]

Any updates on the simultaneous tuning of the inner and outer estimator? Or is the hardcoded solution still the one to go? Sorry if i missed the update elsewhere.
(I want to test not only the n_features and hyperparameters, but also different estimatiors)