Hi all,
I was wondering whether any knew a way to perform nested CV using each one of the tuning parameters specified with GridSearchCV. At the moment if you do something like this (I'm copying from Sci-kit learn):
The problem I see with this is that the generalisation error is estimated, but as far as I know the outer CV layer is performed with the optimal tuned parameters automatically selected by GridSearchCV method. What I want is to be able to generalise error performance for EACH component of the grid so I can compare error rates on the test data.
Thanks!
I was wondering whether any knew a way to perform nested CV using each one of the tuning parameters specified with GridSearchCV. At the moment if you do something like this (I'm copying from Sci-kit learn):
# Number of random trials NUM_TRIALS = 30 # Load the dataset iris = load_iris() X_iris = iris.data y_iris = iris.target # Set up possible values of parameters to optimize over p_grid = {"C": [1, 10, 100], "gamma": [.01, .1]} # We will use a Support Vector Classifier with "rbf" kernel svm = SVC(kernel="rbf") # Arrays to store scores nested_scores = np.zeros(NUM_TRIALS) # Loop for each trial for i in range(NUM_TRIALS): # Nested CV with parameter optimization nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv) nested_scores[i] = nested_score.mean()
The problem I see with this is that the generalisation error is estimated, but as far as I know the outer CV layer is performed with the optimal tuned parameters automatically selected by GridSearchCV method. What I want is to be able to generalise error performance for EACH component of the grid so I can compare error rates on the test data.
Thanks!