How to read in mulitple files efficiently

[garynewport]

I have multiple files, all containing 10s of thousands of lines of data. I cannot alter the way the data is presented and it appears as seen in the attached screenshot.

The first 2 lines can be ignored; I am only interested in the lines beginning with "hydro".

Currently, my program reads the data in from each file selected as and when I select a graph...

for line in file:
                    if keyword.casefold() in line.casefold():
                        time     = self.convert(line[21:35])                   # Timestep
                        delta_t  = self.convert(line[36:50])                   # Change in timestep
                        mass     = self.convert(line[51:65])                   # Mass of star (excluding envelope)
                        radius   = self.convert(line[66:80])                   # Radius of star (excluding envelope)
                        lum_core = self.convert(line[81:95])                   # Luminosity of star (excluding envelope)
                        lum_tot  = self.convert(line[96:110])                  # Total luminosity (including enveloping cloud)
                        flux     = self.convert(line[111:125])                 # Mass flux
                        ratio    = float(line[125:137])                        # Ratio of star mass against mass of the Sun
                
                        # Store the data into the Numpy array 'data'
                        data = np.append(data, np.array([[time, delta_t, mass, radius, lum_core, lum_tot, flux, ratio]]), axis = 0)

The problem here is that this takes some time (several minutes) and is required each time I create a new graph based upon this data.

I did consider reading the data into a list as and when each file was selected, but this causes a delay between selecting a file and being able to select the next file, as the data is read into the list.

I thought of reading in on a separate thread, but then how do I ensure that the data added to the list is in the correct order?

I would appreciate any and all suggestions on how I might approach this problem.