Hello all,
I am having a really frustrating problem I cannot figure out.
I have run a cross validation for a machine learning model I am working on. I am generating metrics for how well the model performs and each iteration (each fold in a for loop) generates a new metric. I want to plot these metrics on the same plot so that for each class in the machine learning model, there is a plot that contains 'x' number of lines corresponding to the metric at each fold.
The problem is, when looping, one plot is created with all lines (as expected), however there are 19 classes in the model, so there should be 19 plots. Even when looping through 'i' only one plot is generated (the last 'i' element in the loop.
The code is long however, the problem may be in indentation which is why i am showing it all.. the problem is with the last part of the loop.. i would really appreciate help as I am stuck on this seemingly simple problem. i have looked online with no luck..
This produces a plot (atatched) where there are 3 test lines and 3 train lines- 6 in total (correct).. but it only produces the graph for the last i element, not all 19. please ignore legend as i am aware it needs work, but need to get the plotting correct before i fix it!
If i indent further (so plot_roc in line with the for i in number_classes), it produces 19 plots for EACH fold.. and does not combine the fold together.
thank you!
I am having a really frustrating problem I cannot figure out.
I have run a cross validation for a machine learning model I am working on. I am generating metrics for how well the model performs and each iteration (each fold in a for loop) generates a new metric. I want to plot these metrics on the same plot so that for each class in the machine learning model, there is a plot that contains 'x' number of lines corresponding to the metric at each fold.
The problem is, when looping, one plot is created with all lines (as expected), however there are 19 classes in the model, so there should be 19 plots. Even when looping through 'i' only one plot is generated (the last 'i' element in the loop.
The code is long however, the problem may be in indentation which is why i am showing it all.. the problem is with the last part of the loop.. i would really appreciate help as I am stuck on this seemingly simple problem. i have looked online with no luck..
#AUC containers fpr = dict() tpr = dict() fprTrain=dict() tprTrain=dict() roc_aucTest = dict() roc_aucTrain=dict() inputs=np.array(data) targets=trainingtarget # Define the K-fold Cross Validator kfold = KFold(n_splits=num_folds, shuffle=True, random_state=42) #earlystopping earlystop=tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=5,verbose=1,mode='auto') #optimizer optimizer=tf.keras.optimizers.Adam(lr=0.0001) fold_no = 1 for train, test in kfold.split(inputs, targets): #model model=tf.keras.models.Sequential() model.add(tf.keras.layers.Dense(units=700, input_dim=20767, activation="relu")) model.add(tf.keras.layers.Dense(units=900, activation="relu")) model.add(tf.keras.layers.Dense(units=800, activation="relu")) model.add(tf.keras.layers.Dense(units=19, activation="sigmoid")) # Compile the model model.compile(loss="binary_crossentropy", optimizer=optimizer, metrics=['accuracy']) # Fit data to model history=model.fit(inputs[train], targets[train], epochs=200, batch_size=32, verbose=1, validation_data=(inputs[test], targets[test]), callbacks=[earlystop], shuffle=True) # Increase fold numbers fold_no = fold_no + 1 #generate AUROC for individual classes and plot AUROC for each fold on plot test_predictions=model.predict(inputs[test]) train_predictions=model.predict(inputs[train]) #generate predictions number_classes=range(0,19) for i in number_classes: fpr[i], tpr[i], _ = roc_curve(targets[test][:, i], test_predictions[:, i]) roc_aucTest[i] = auc(fpr[i], tpr[i]) fprTrain[i], tprTrain[i], _ = roc_curve(targets[train][:, i], train_predictions[:, i]) roc_aucTrain[i] = auc(fprTrain[i], tprTrain[i]) # Plot all AUROC curves plot_roc(labels= targets[train][:, i], predictions= train_predictions[:,i], name='train ROC fold {}'.format(fold_no-1) + ':' + '(AUC={0:0.2f})' ''.format(roc_aucTrain[i])) plot_roc(labels= targets[test][:, i], predictions= test_predictions[:,i], linestyle='-.', name='test ROC fold {}'.format(fold_no-1)+ ':' +'(AUC={0:0.2f})' ''.format(roc_aucTest[i])) plt.show()The last lines starting with for i in number_classes:... is where the problem lies. the code itself works fine, but the call to plt.show ONLY shows one plot, and not 19 as it should. (please not plot_roc is defined and not shown in this to not increase this post size but happy to provide if it its necessary, i am not sure it is though)
This produces a plot (atatched) where there are 3 test lines and 3 train lines- 6 in total (correct).. but it only produces the graph for the last i element, not all 19. please ignore legend as i am aware it needs work, but need to get the plotting correct before i fix it!
If i indent further (so plot_roc in line with the for i in number_classes), it produces 19 plots for EACH fold.. and does not combine the fold together.
thank you!