Python Forum
How to use Logging with multiprocessing in Python3
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
How to use Logging with multiprocessing in Python3
#1
I am trying to use python built in logging with multiprocessing. Goal -- is to have errors logged to a file called "error.log" Issue -- The errors are printed in the console instead of the log file. see code below

import concurrent.futures
from itertools import repeat
import logging


def data_logging():
    error_logger = logging.getLogger("error.log")
    error_logger.setLevel(logging.ERROR)
    formatter = logging.Formatter('%(asctime)-12s %(levelname)-8s %(message)s')
    file_handler = logging.FileHandler('error.log')
    file_handler.setLevel(logging.ERROR)
    file_handler.setFormatter(formatter)
    error_logger.addHandler(file_handler)

    return error_logger


def check_number(error_logger, key):
    if key == 1:
        print ("yes")
    else:
        error_logger.error(f"{key} is not = 1")


def main():
    key_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 4, 5, 4, 3, 4, 5, 4, 3, 4, 5, 4, 3, 4, 3]
    error_logger = data_logging()
    with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
        executor.map(check_number, repeat(error_logger), key_list)


if __name__ == '__main__':
        main()
Function check_number checks if numbers in list key_list is 1 or not if key = 1, prints yes to the console, if not i would like the program to add {key} is not = 1 to the log file. instead with the code above it prints it to the console. please help if u can. this is a mini example to my program so don't change the logic
Reply
#2
your code works for me as expected.

console:
Output:
yes yes
error.log
Output:
2020-12-29 11:03:34,853 ERROR 2 is not = 1 2020-12-29 11:03:34,854 ERROR 3 is not = 1 2020-12-29 11:03:34,854 ERROR 4 is not = 1 2020-12-29 11:03:34,854 ERROR 5 is not = 1 2020-12-29 11:03:34,854 ERROR 6 is not = 1 2020-12-29 11:03:34,854 ERROR 7 is not = 1 2020-12-29 11:03:34,855 ERROR 8 is not = 1 2020-12-29 11:03:34,855 ERROR 9 is not = 1 2020-12-29 11:03:34,855 ERROR 10 is not = 1 2020-12-29 11:03:34,855 ERROR 4 is not = 1 2020-12-29 11:03:34,856 ERROR 5 is not = 1 2020-12-29 11:03:34,856 ERROR 4 is not = 1 2020-12-29 11:03:34,856 ERROR 3 is not = 1 2020-12-29 11:03:34,856 ERROR 4 is not = 1 2020-12-29 11:03:34,857 ERROR 5 is not = 1 2020-12-29 11:03:34,857 ERROR 4 is not = 1 2020-12-29 11:03:34,857 ERROR 3 is not = 1 2020-12-29 11:03:34,857 ERROR 4 is not = 1 2020-12-29 11:03:34,858 ERROR 5 is not = 1 2020-12-29 11:03:34,858 ERROR 4 is not = 1 2020-12-29 11:03:34,858 ERROR 3 is not = 1 2020-12-29 11:03:34,858 ERROR 4 is not = 1 2020-12-29 11:03:34,858 ERROR 3 is not = 1
Not sure why would you set max_workers=1 though :-)
If you can't explain it to a six year old, you don't understand it yourself, Albert Einstein
How to Ask Questions The Smart Way: link and another link
Create MCV example
Debug small programs

Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Gnuradio python3 is not compatible python3 xmlrpc library How Can I Fix İt ? muratoznnnn 3 4,876 Nov-07-2019, 05:47 PM
Last Post: DeaD_EyE

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020