Python Forum
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Reduce code run time
#1
Hey guys, I'm trying to run a few files through a loop, but the run time is way to high. I am wondering if there is an issue with the way I coded it, that could cut down the time it takes to run the file.

Thanks!


This is a class that the files are read into and concatenated into a single file to be accessed by row.
class TextFiles:
   '''
   This Class will read in all the text data files. These files will then be
   used to compare data from the read in binary files. 
   '''
   def __init__(self, txt1, txt2, txt3, txt4, txt5):
       text_data = self._textfile_data(txt1, txt2, txt3, txt4, txt5) 
       self._text_data = text_data
   
   def _textfile_data(self, file1, file2, file3, file4, file5):
       '''
       This method places the read in text files into a 2D numpy array. The 
       zero column is then deleted removing the scan number. The arrays are
       then concatenated together, making a single numpy array of the data.
       '''
       data1 = np.loadtxt(file1, delimiter = '\t', skiprows = 1)
       data1 = np.delete(data1, [0], axis = 1)
       data2 = np.loadtxt(file2, delimiter = '\t', skiprows = 1)
       data2 = np.delete(data2, [0], axis = 1)
       data3 = np.loadtxt(file3, delimiter = '\t', skiprows = 1)
       data3 = np.delete(data3, [0], axis = 1)
       data4 = np.loadtxt(file4, delimiter = '\t', skiprows = 1)
       data4 = np.delete(data4, [0], axis = 1)
       data5 = np.loadtxt(file5, delimiter = '\t', skiprows = 1)
       data5 = np.delete(data5, [0], axis = 1)
       text_data_array = np.concatenate((data1, data2, data3, data4, data5), axis = 1)
       return text_data_array
This is where the files are being used.
 for x in range(len(dictionary._key_data(key_filename))-1):
           plt.plot(range(0,n), dictionary[next(current_key)], linewidth = 0.25)
           plt.plot(range(0,n), files._textfile_data(textfile1, textfile2, textfile3, textfile4, textfile5)[:,x], ':', linewidth = 0.25)
           ei = (sum(dictionary[dictionary._key_data(key_filename)[x]]) - (sum(files._textfile_data(textfile1, textfile2, textfile3, textfile4, textfile5)[:,x])))
           MAE = abs(ei)/n 
Reply
#2
I don't see where you instantiate TextFiles, but  that's another issue.
you have _textfile_data in a loop, so you are loading each file once for every iteration of the loop
I would expect this to be excruciatingly slow.

You need to rethink this.
Reply
#3
Larz you are totally right. The file read in through the looping was the issue. Thank you very much!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  reduce time series based on sum condition amdi40 0 1,079 Apr-06-2022, 09:09 AM
Last Post: amdi40
  Reduce four for loops or parallelizing code in Python cee878 1 1,166 Feb-10-2022, 10:02 AM
Last Post: Larz60+
  how to reduce running time of the code dilmailid 6 3,943 May-18-2018, 02:49 AM
Last Post: scidam

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020