Sep-05-2024, 07:20 PM
I'm trying to build a small GUI app with a logger that will, at some point, be doing some time consuming manipulation of multiple sets of data (up to several hundred). Naturally, I wanted to use multiprocessing to help speed things up a bit. I was following the example given in the Logging Cookbook (second example) in the Python docs and trying to figure out how to work it into my code. In this stripped down minimal example, clicking the build button should simply log a few messages. The problem, which is hopefully obvious to someone more learned in the topics at hand, is that it doesn't work as intended. The application only prints 3 of the 5 messages to the console and exactly zero are added to the log file.
What I was expecting, obviously, was that all 5 messages would be logged with the logger instance created in gui.py.
I've tried merging methods, moving methods out of the class into module level functions, creating the Queue/loggers in different places, and passing the first logger instance around as an argument. Everything I've tried up to this point either leads to the same results or throws a pickling error and eventually ends in an EOFError. The code given is just the most recent revision that doesn't throw an exception.
I'm just trying to get some direction on "where" I'm messing up. In case it matters, this is on Windows 10 using Python 3.12.
What I was expecting, obviously, was that all 5 messages would be logged with the logger instance created in gui.py.
I've tried merging methods, moving methods out of the class into module level functions, creating the Queue/loggers in different places, and passing the first logger instance around as an argument. Everything I've tried up to this point either leads to the same results or throws a pickling error and eventually ends in an EOFError. The code given is just the most recent revision that doesn't throw an exception.
I'm just trying to get some direction on "where" I'm messing up. In case it matters, this is on Windows 10 using Python 3.12.
# gui.py
from multiprocessing import Queue
import tkinter as tk
from tkinter import ttk
import builder
import logger
class Gui(tk.Tk):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
btn = ttk.Button(self, text='Build', command=lambda: builder.Build(q))
btn.pack()
log.configure(q)
self.mainloop()
if __name__ == '__main__':
q = Queue()
log = logger.Logger()
Gui()# logger.py
import logging
import logging.handlers
class Logger:
def __init__(self):
self.logger = logging.getLogger('dcre')
self.logger.setLevel('DEBUG')
self.log = self.logger.log
def configure(self, q):
self.q = q
self.qh = logging.handlers.QueueHandler(self.q)
self.file = logging.FileHandler('log.log')
self.file.setLevel('DEBUG')
self.logger.addHandler(self.qh)
self.logger.addHandler(self.file)
logging.basicConfig(level='INFO')# builder.py
import logging
from multiprocessing import Process
import threading
class Build:
def __init__(self, q):
self.queue = q
self.logger = logging.getLogger('dcre')
workers = []
for i in range(5):
wp = Process(target=self.foo, args=(i,))
workers.append(wp)
wp.start()
lp = threading.Thread(target=self.logger_thread)
lp.start()
for wp in workers:
wp.join()
self.queue.put(None)
lp.join()
def logger_thread(self):
while True:
record = self.queue.get()
if record is None:
break
self.logger.handle(record)
def foo(self, i):
msgs = (
(10, "This is a DEBUG message. You shouldn't see this."),
(20, 'This is an INFO message. Just so you know.'),
(30, 'This is a WARNING message. Be careful, yo.'),
(40, 'This is an ERROR message. Man, you done messed up.'),
(50, 'This is a CRITICAL message. Game over!')
)
self.logger.log(*msgs[i])Note: The configure method for the logger only existed to delay the configuration until after the GUI was created so that it had access to a Text widget for a custom handler to write to.
