Python Forum

I am trying to make the below code work since morning. I have read tutorials on google about python, I have done 0 progress. Can you help me with this error:

Using TensorFlow backend.
Traceback (most recent call last):
  File "source_code_modified.py", line 65, in <module>
    dict1 = pickle.load(f1,encoding='bytes')
TypeError: file must have 'read' and 'readline' attributes

The code that explodes is this part:

class load_train_data:
    os.open("/home/mojito/Desktop/CNN/datasets/training_data/images", os.O_RDONLY)
    def __enter__(self):
        return self
    def __exit__(self, type, value, traceback):
        pass


class load_test_data:
    os.open("/home/mojito/Desktop/CNN/datasets/test_data/images",os.O_RDONLY) 
    def __enter__(self):
        return self
    def __exit__(self, type, value, traceback):
        pass


with load_train_data() as f1:
   dict1 = pickle.load(f1,encoding='bytes')

I tried to fix this with the code below and I got a new error:

Using TensorFlow backend.
Traceback (most recent call last):
  File "source_code_modified.py", line 74, in <module>
    with open_train_data() as f1:
  File "source_code_modified.py", line 47, in open_train_data
    return open('/home/mojito/Desktop/CNN/datasets/training_data/images','rb') 
IsADirectoryError: [Errno 21] Is a directory: '/home/mojito/Desktop/CNN/datasets/training_data/images'

the code explodes in this part:

def open_train_data():
    return open('/home/mojito/Desktop/CNN/datasets/training_data/images','rb') <--- explodes here

def open_test_data():
    return open('/home/mojito/Desktop/CNN/datasets/test_data/images','rb')

with open_train_data() as f1:
   dict1 = pickle.load(f1)   <--- explodes here

What I am trying to do is to load a custom dataset of images for a CNN stored to my PC in the format (trainX, trainY), (testX, testY) = ...

Another question: Can I transform the folder with the images to a pickle (.P) file?

What it looks like you are trying to do is 'open' a directory to get all the images, so you can use them elsewhere.
I would recommend the glob module for that.

import glob

def open_train_data():
    return glob.glob('/home/mojito/Desktop/CNN/datasets/training_data/images/*') #the '*' is a 'wildcard' and it means to match one or more characters
 
for f in open_train_data():
   dict1 = pickle.load(f)
   ###do whatever else you need to

(Jun-12-2020, 09:01 AM)DreamingInsanity Wrote: [ -> ]What it looks like you are trying to do is 'open' a directory to get all the images, so you can use them elsewhere.
I would recommend the glob module for that.

import glob

def open_train_data():
    return glob.glob('/home/mojito/Desktop/CNN/datasets/training_data/images/*') #the '*' is a 'wildcard' and it means to match one or more characters
 
for f in open_train_data():
   dict1 = pickle.load(f)
   ###do whatever else you need to

I ran the code you posted. I got this error:

Using TensorFlow backend.
Traceback (most recent call last):
  File "source_code_modified.py", line 78, in <module>
    dict1 = pickle.load(f)
TypeError: file must have 'read' and 'readline' attributes

It explodes at this point:

dict1 = pickle.load(f)

Sorry, my mistake - it should work if you do this:

import glob
 
def open_train_data():
    return glob.glob('/home/mojito/Desktop/CNN/datasets/training_data/images/*') #the '*' is a 'wildcard' and it means to match one or more characters
  
for f in open_train_data():
   dict1 = pickle.load(open(f, 'rb'))
   ###do whatever else you need to

I forgot to open the file.

(Jun-12-2020, 10:20 AM)DreamingInsanity Wrote: [ -> ]Sorry, my mistake - it should work if you do this:

import glob
 
def open_train_data():
    return glob.glob('/home/mojito/Desktop/CNN/datasets/training_data/images/*') #the '*' is a 'wildcard' and it means to match one or more characters
  
for f in open_train_data():
   dict1 = pickle.load(open(f, 'rb'))
   ###do whatever else you need to

I forgot to open the file.

I am getting this error:

Using TensorFlow backend.
Traceback (most recent call last):
  File "source_code_modified.py", line 80, in <module>
    dict1 = pickle.load(open(f, 'rb'))
_pickle.UnpicklingError: A load persistent id instruction was encountered,
but no persistent_load function was specified.

I am trying to solve it though, but I thought posting it for more help. Thank you...

Any idea?

(Jun-12-2020, 05:05 PM)hobbyist Wrote: [ -> ]Any idea?

It seems pickle.load() is for use when unpickling. If you are trying to pickle an image, you want to use pickle.dumps() which returns a pickled object.
If you want to save the pickled object to a file, you could use pickle.dump(f, open("myfile.data", "wb"))