[Tkinter] acceleration of data output in treeview tkinter

[Vladimir1984]

Good afternoon!

I am trying to come up with a mechanism to output data from the database in treeview tkinter (tabular form).
I read the table from the database, divide the number of rows by the number of processor cores.
I create streams and try to write data in parallel to treeview via write function.

1 2 3 4 5 6 7 8 9 10 11 12 13 14

def trvwinsert(lstb_donnes, numstrt, numfnsh):     brak = 0     colbrak = []     goden = 0     colgoden = []       for n in range(numstrt, numfnsh):         if lstb_donnes[n][6] == 'defect':             listBox.insert("", "end", values=(lstb_donnes[n][:]), tags=('defect',))             brak = brak + 1         else:             listBox.insert("", "end", values=(lstb_donnes[n][:]))             goden = goden + 1     return colgoden.append(goden), colbrak.append(brak)

Function code with dialog box (top level):

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

# mlticpu = multiprocessing.cpu_count()         lenprc = len(lstb_donnes)           nstrt = []         nfnsh = []         thread_list = []           for j in range(mlticpu):             if j == 0:                 nstrt.append(0)             else:                 nstrt.append(nfnsh[j - 1] + 1)               if j == mlticpu - 1:                 nfnsh.append(lenprc)             else:                 nfnsh.append(nstrt[j] + math.ceil(lenprc / mlticpu))           start = time.time()           for h in range(mlticpu):             numstrt = nstrt[h]             numfnsh = nfnsh[h]             t = threading.Thread(target=trvwinsert, name='thread{}'.format(h), args=(lstb_donnes, numstrt, numfnsh))             thread_list.append(t)             t.start()             print('{} has started'.format(t.name))           for t in thread_list:             t.join()           end = time.time()           print("cycle time = " + str(end - start))         print('all threads done')           first_item.entryconfig(1, label="Сохранить отчет", state="normal")         dop_window.after(1000)         dop_window.destroy()

As a result of execution, even with one thread, I get an error:

1 2 3 4 5 6 7 8 9 10 11

Exception in thread thread0: Traceback (most recent call last):   File "C:\Python\lib\threading.py", line 932, in _bootstrap_inner     self.run()   File "C:\Python\lib\threading.py", line 870, in run     self._target(*self._args, **self._kwargs)   File "C:/TestInstruments/Reports/report_4_new variant.py", line 68, in trvwinsert     listBox.insert("", "end", values=(lstb_donnes[n][:]))   File "C:\Python\lib\tkinter\ttk.py", line 1369, in insert     res = self.tk.call(self._w, "insert", parent, index, *opts) RuntimeError: main thread is not in main loop

As a result of searching for a solution to the problem, I did not come to a consensus on whether it is possible to speed up the writing of data to treeview using the threading module. It may be necessary to use the multiprocessing module.
The main question is whether it is possible to speed up the output of large tables in treeview tkinter. Now the output takes 40-45 minutes and only loads one core of the CPU.

[Larz60+]

Now the output takes 40-45 minutes and only loads one core of the CPU

How many rows of data are involved?
This seems like it might be an SQL query issue more than tkinter (treeview in all of my experience has been fast)

Have you tried running your query without any GUI overhead?
Can you share the SQL?

[Vladimir1984]

Good afternoon!

There are a million lines of data to output to such a table, maybe more. I first read the data from the database already prepared for display on the screen. I bring them out of memory and not from the base. The output line by line to the table takes time, if it is 5 thousand lines then it is seconds, but if 500 000 and more it is minutes.

I wanted to write data in multiple lines from multiple streams to a parallel.

I read the sources on this topic, I realized that I was mistaken - I had to build an interface on PySide2 (PyQT5). If necessary, I will redo it (tables are better implemented in it and output multithreading is supported).

[Larz60+]

I think you should try PYQT5 or (my preference) wxpython.
Both are much newer code than tkinter and probably (I don't know this for a fact) faster.

I don't know that multithreading would help, can't wrap my mind around how you would synchronize the writing to the widget, as I'm pretty sure there's no way to write more than one row at a time.

You're just putting the data key in the treeview, correct?
the actual data should be in memory and accessed by the key, then displayed in some other widget.
Tkinter is not the best choice for this type of display as the widget set is quite limited.

[Vladimir1984]

I think the topic can be closed or moved to a solved one.