Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Nested CV
#1
Hi all,

I was wondering whether any knew a way to perform nested CV using each one of the tuning parameters specified with GridSearchCV. At the moment if you do something like this (I'm copying from Sci-kit learn):

# Number of random trials
NUM_TRIALS = 30

# Load the dataset
iris = load_iris()
X_iris = iris.data
y_iris = iris.target

# Set up possible values of parameters to optimize over
p_grid = {"C": [1, 10, 100],
          "gamma": [.01, .1]}

# We will use a Support Vector Classifier with "rbf" kernel
svm = SVC(kernel="rbf")


# Arrays to store scores
nested_scores = np.zeros(NUM_TRIALS)

# Loop for each trial
for i in range(NUM_TRIALS):

# Nested CV with parameter optimization
    nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv)
    nested_scores[i] = nested_score.mean()


The problem I see with this is that the generalisation error is estimated, but as far as I know the outer CV layer is performed with the optimal tuned parameters automatically selected by GridSearchCV method. What I want is to be able to generalise error performance for EACH component of the grid so I can compare error rates on the test data.

Thanks!
Reply


Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020