Python Forum
Miss-Classification problem with shuffled batch
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Miss-Classification problem with shuffled batch
#1
Sad 
Hello all,

I am totally new to this forum and to python as well as pytorch. I watched a lot of tutorials and rebuild examples. But now I tried my first own net and I just don't get the results I would expect. So there must be obviously something wrong, but I don't know what and where ist the problem. And since I cannot really tell, where I can fix it, I add you the whole code. I tried to comment it as good as I can!!

Basically this net shall do the following:

I have a lot of different profiles for plants. All of these plants can be calculated by a software. For this calculation I insert some parameters. With these parameters the software generates a kind of fingerprint (1-D) containing a lot of letters, they stand for the Parameter in use (i.e. the height of the plant is divided in 5 different ranges, so A-E and my actual profile shall be B). So you get a not complete (since some parameters will be calculated from the software) representation in this form: ABSBBSCD%%%%%....AB%%B (% stands for a not set parameter).

To start the software I need the representation of another profile, which will be my start_value. So for this one we expect much less '%' characters. I took 33 profiles and tested them all against each other, to find out, which will be able to converge from which starting point. Now I want to train a net with input of the two Representations to the target of which converging class this combination will be.

My problem was now:
1. Everything was send to a single class (total miss classification), so I added the sogmoid function in the forward section of the net
2. then there were still many errors, so I tried to shuffle the data order every epoch of training. I tried that in a way that it was totally random but still go sure to shuffle the input and target in the same order.
3. Then I noticed, that suddenly I cannot find any longer all classes, but randomly only 2 will be matched. And what is really weird:
4. I test if the shuffled tensor is the same as the input tensor and python tells me yes??? that doesn't make any sense or is my coding totally wrong in logic? And if I test the net in the evaluation part at the end, then suddenly all the targets changed... even weirder....

I add the code below here and the data as well... but somehow I cannot add the ods file....

So I really have no idea where my misstake is... Probably there are some basic rules or function I don't understand in the right way which making an obvious misstake for you here in the forum. That would be the best way I can imagine, cause maybe then someone here can help me. I would be so thankful and if I forgot to add something ot you need more information or anything other I can contribute, please tell me and I will do as fast as I can!!!! Thank you very much!!

# import all necessary stuff
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import pandas as pd
import random

Text = open('Daten.txt', 'r') # Load Data, all in Form of i.e. ACBBDEJJSAU%%SBSNAV...A%EB
FP_Input = []
FP_Profile = []
FP_Art = 0
FP_Input_Index = 0
FP_Profile_Index = 0
Zeile_leer = 0
for zeile in Text:   # read all lines of data
    if zeile == '\n':
        Zeile_leer += 1   # skip empty lines and count them
    elif 'FP_' in zeile:
        FP_Art = FP_Art + 1  # notice switch of kind of data
    elif FP_Art == 2: # profile to be calculated
        FP_Input.append(zeile)
        FP_Input_Index = FP_Input_Index + 1
    elif FP_Art == 3: # profile from which calculation will be started
        FP_Profile.append(zeile)
        FP_Profile_Index = FP_Profile_Index + 1
    elif FP_Art == 1: # max positions of representation of every data
        FP_Max = zeile

# print(FP_Input)
# print(FP_Profile)
# print(FP_Max)


def max_fp_to_int(max_fp):  # calculate from the letters in max_fp how many positions for this parameter is needed
    fp = max_fp.strip()
    fp = fp.split()
    fp = ' '.join(fp)
    num_of_vars = fp.count(' ') + 1
    fp_int = np.zeros(num_of_vars)
    fp = fp.split()
    for i in range(len(fp)):
        if len(fp[i]) == fp[i].count('%'):
            fp_int[i] = 0 if len(fp[i]) == 1 else -1
        elif len(fp[i]) == 1:
            char_int = ord(fp[i]) - 64
            if char_int <= 0 or char_int > 26:  # check for bold letters
                print("ERROR: Fehler bei Max-FP, Buchstaben nicht alle Großgeschrieben!")
                exit()
            else:
                fp_int[i] = char_int
        else:
            help_int = 1
            for char in fp[i]:
                char_int = ord(char) - 64
                if char_int <= 0 or char_int > 26:
                    print("ERROR: Fehler bei Max-FP, Buchstaben nicht alle Großgeschrieben!")
                    exit()
                elif help_int == 1:
                    help_int = 26 * (char_int)
                else:
                    fp_int[i] = help_int + char_int
    return fp_int   # number of positions for every parameter, e.g. [57, 16, 5, 4, 11, 12, ... 8, 112, 57]


FP_Max_Int = max_fp_to_int(FP_Max)
# print(FP_Max_Int)


def fp_to_int(fp, max_fp):  # convert a profile fingerprint (ABDDEBD%%%ABAB...ABA) with information of fp_max
                            # to an integer representation. for every parameter all positions are 0, except the chosen
                            # one, i.e. first letter of profile = 0, max_fp at this position is 5 (=E) -> 1 0 0 0 0
    fp_num_of_int = int(sum(max_fp))
    num_of_double_fp_pos = 0
    for i in range(len(max_fp)):
        if max_fp[i] == -1:
            fp_num_of_int += 1
        elif max_fp[i] > 26:
            num_of_double_fp_pos += 1
    fp_int = np.zeros(fp_num_of_int)
    pos_in_fp = 0
    char_fp = [char for char in fp]
    i_skip = 0
    char_skip = 0
    for i in range(len(max_fp) + num_of_double_fp_pos):
        # print(i)
        if i_skip == 1:
            i_skip = 0
        elif max_fp[i - char_skip] == 0:
            pos_in_fp = pos_in_fp
            # print(char_fp[i], ' in Fp in Range von max ', max_fp[i -char_skip])
        elif max_fp[i - char_skip] == -1:
            i_skip = 1
            # print(char_fp[i], char_fp[i + 1], ' FP in Range von max ', max_fp[i - char_skip])
            char_skip += 1
        elif char_fp[i] == '%':
            if max_fp[i - char_skip] <= 26:
                pos_in_fp = int(pos_in_fp + max_fp[i - char_skip])
                # print(char_fp[i], ' in Fp in Range von max ', max_fp[i - char_skip])
            else:
                pos_in_fp = int(pos_in_fp + max_fp[i - char_skip])
                i_skip = 1
                # print(char_fp[i], char_fp[i + 1], ' FP in Range von max ', max_fp[i - char_skip])
                char_skip += 1
        elif max_fp[i - char_skip] <= 26:
            char_int = ord(char_fp[i]) - 64
            if char_int <= 0 or char_int > 26:
                print("ERROR: Fehler bei Max-FP, Buchstaben nicht alle Großgeschrieben!")
                exit()
            else:
                fp_int[pos_in_fp + char_int - 1] = 1
                pos_in_fp = int(pos_in_fp + max_fp[i - char_skip])
                # print(char_fp[i], ' in FP in Range von max ', max_fp[i -char_skip])
        else:
            char_int1 = ord(char_fp[i]) - 64
            char_int2 = ord(char_fp[i + 1]) - 64
            if char_int1 <= 0 or char_int1 > 26 or char_int2 <= 0 or char_int2 > 26:
                print("ERROR: Fehler bei Max-FP, Buchstaben nicht alle Großgeschrieben!")
                exit()
            else:
                fp_int[pos_in_fp + char_int1 * 26 + char_int2 - 1] = 1
            pos_in_fp = int(pos_in_fp + max_fp[i - char_skip])
            i_skip = 1
            # print(char_fp[i], char_fp[i + 1], ' in FP in Range von max ', max_fp[i - char_skip])
            char_skip += 1
    return fp_int


FP_Input_Int = []
FP_Profile_Int = []

for i in range(len(FP_Input)):
    FP_Input_Int.append(fp_to_int(FP_Input[i], FP_Max_Int))    # build tensor with all profiles to be calculated
    FP_Profile_Int.append((fp_to_int(FP_Profile[i], FP_Max_Int))) # build tensor with all profiles as starting points

FP_Input_Torch = torch.FloatTensor(FP_Input_Int)
FP_Profile_Torch = torch.FloatTensor(FP_Profile_Int)
Anstart_Verhalten = pd.read_excel("Anstart.ods", engine="odf")


# model
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.fc1 = nn.Linear(1721 * 2, 1721)  # 1721 positions from max_fp, 2 profiles combined
        self.fc2 = nn.Linear(1721, 500)
        self.fc3 = nn.Linear(500, 50)
        self.fc4 = nn.Linear(50, 5) # 5 possibilities if and how profiles can converge
        # self.fc4 = nn.Linear(20, 2)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        x = torch.sigmoid(x)
        return x


net = Net()

params = list(net.parameters())


def target_class(matrix, i, j):
    cell_value = matrix.iat[i, j]
    if cell_value == -3:
        target = [0, 0, 0, 0, 1] # singularity
    elif cell_value == -2:
        target = [0, 0, 0, 1, 0] # negative result
    elif cell_value == -1:
        target = [0, 0, 1, 0, 0] # not converged
    elif cell_value == 0:
        target = [0, 1, 0, 0, 0] # negative but not a big deal
    else:
        target = [1, 0, 0, 0, 0]  # converged

    return torch.FloatTensor(target)


for i in range(len(FP_Input_Torch)): # buil a big batch of all profile combination possible and the classification
                                     # as in target_class
    for j in range(len(FP_Profile_Torch)):
        if j == 0 and i == 0:
            input = torch.cat((FP_Input_Torch[i], FP_Profile_Torch[j]), 0)
            input = input.unsqueeze(0)
            target = target_class(Anstart_Verhalten, i, j)
            target = target.unsqueeze(0)
        else:
            new_input = torch.cat((FP_Input_Torch[i], FP_Profile_Torch[j]), 0)
            input = torch.cat((input, new_input.unsqueeze(0)), 0)
            target = torch.cat((target, target_class(Anstart_Verhalten, i, j).unsqueeze(0)), 0)
            # print(target_class(Anstart_Verhalten, i, j))
# print(input)
# print(target)


# input = torch.cat((FP_Input_Torch[1], FP_Profile_Torch[1]), 0)
output = net(input)
criterion = nn.MSELoss()
loss = criterion(output, target)
net.train()

net.zero_grad()     # zeroes the gradient buffers of all parameters
loss.backward()

learning_rate = 0.1
for f in net.parameters():
    f.data.sub_(f.grad.data * learning_rate)

# create your optimizer
optimizer = optim.SGD(net.parameters(), lr=learning_rate)


def get_shuffled_data(input_s, target_s):   # mix the profile combinations and the target_class in the same way
                                            # to get different shuffles in each training epoch
    for i in range(len(input)):
        j = random.randint(0, len(input) - 1)
        k = random.randint(0, len(input) - 1)
        memory_input = input_s[j]
        memory_target = target_s[j]
        input_s[j] = input_s[k]
        target_s[j] = target_s[k]
        input_s[k] = memory_input
        target_s[k] = memory_target
    return input_s, target_s


target_backup = target  # check if raw_data get changed

# in your training loop:
for epoch in range(1, 1000):  # train net
    optimizer.zero_grad()   # zero the gradient buffers
    input_s, target_s, = get_shuffled_data(input, target)
    output = net(input_s)
    loss = criterion(output, target_s)
    loss.backward()
    optimizer.step()    # Does the update

    if epoch % 10 == 0:
        print(loss, ' bei ', epoch/10, '%')


if torch.all(target_backup.eq(target)):  # check if target still same order as before training started (should be ok)
    print('okay')
else:
    print('different')
if torch.all(target_backup.eq(target_s)): # check if the shuffle worked, should be different....
    print('okay')
else:
    print('different')
net.eval()


def test_eval(test, target, i):  # test if all classifications were right. Here somehow target is different ot start
    test = test.detach().numpy()
    target = target.detach().numpy()
    if np.argmax(test) == np.argmax(target):
        print(i, ' right ', test, ' ', target)
        return 0
    else:
        print(i, ' wrong ', test, ' ', target)
        return 1


miss_classification = 0

for i in range(len(input)):
    test_input = input[i]
    test_target = target[i]
    test = net(test_input)
    miss_classification += test_eval(test, test_target, i)

print(miss_classification)

Attached Files

.txt   Daten.txt (Size: 11.37 KB / Downloads: 3)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Probabilities of binary classification problem Troublesome1996 0 2,398 Apr-19-2021, 06:40 PM
Last Post: Troublesome1996
  GridSearchCV for multi-label classification mapypy 0 3,648 Mar-29-2021, 01:58 AM
Last Post: mapypy
  GNN For Graph "Classification" BennyS 1 1,760 Feb-09-2021, 12:09 PM
Last Post: BennyS
  Help with multiclass classification in perceptron code Nimo_47 0 3,702 Nov-09-2020, 10:32 PM
Last Post: Nimo_47
  Classification and Regression tree (CART) kumarants 2 2,690 May-26-2020, 11:04 AM
Last Post: Larz60+
  Help batch converting .json chosen file to MySQL BrandonKastning 2 2,287 Mar-14-2020, 09:19 PM
Last Post: BrandonKastning
  Classification of Request PythonLearner703 8 3,886 Dec-09-2019, 08:56 PM
Last Post: micseydel
  CNN Speech Classification Mitchie87 0 1,583 Dec-06-2019, 06:17 PM
Last Post: Mitchie87
  Keras: Time series classification midarq 0 1,965 Sep-25-2019, 09:03 AM
Last Post: midarq
  Matplolib graphic from batch achilledue 2 2,489 Sep-11-2019, 08:43 AM
Last Post: buran

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020