Python Forum

Full Version: How to use Logging with multiprocessing in Python3
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
I am trying to use python built in logging with multiprocessing. Goal -- is to have errors logged to a file called "error.log" Issue -- The errors are printed in the console instead of the log file. see code below

import concurrent.futures
from itertools import repeat
import logging


def data_logging():
    error_logger = logging.getLogger("error.log")
    error_logger.setLevel(logging.ERROR)
    formatter = logging.Formatter('%(asctime)-12s %(levelname)-8s %(message)s')
    file_handler = logging.FileHandler('error.log')
    file_handler.setLevel(logging.ERROR)
    file_handler.setFormatter(formatter)
    error_logger.addHandler(file_handler)

    return error_logger


def check_number(error_logger, key):
    if key == 1:
        print ("yes")
    else:
        error_logger.error(f"{key} is not = 1")


def main():
    key_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 4, 5, 4, 3, 4, 5, 4, 3, 4, 5, 4, 3, 4, 3]
    error_logger = data_logging()
    with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
        executor.map(check_number, repeat(error_logger), key_list)


if __name__ == '__main__':
        main()
Function check_number checks if numbers in list key_list is 1 or not if key = 1, prints yes to the console, if not i would like the program to add {key} is not = 1 to the log file. instead with the code above it prints it to the console. please help if u can. this is a mini example to my program so don't change the logic
your code works for me as expected.

console:
Output:
yes yes
error.log
Output:
2020-12-29 11:03:34,853 ERROR 2 is not = 1 2020-12-29 11:03:34,854 ERROR 3 is not = 1 2020-12-29 11:03:34,854 ERROR 4 is not = 1 2020-12-29 11:03:34,854 ERROR 5 is not = 1 2020-12-29 11:03:34,854 ERROR 6 is not = 1 2020-12-29 11:03:34,854 ERROR 7 is not = 1 2020-12-29 11:03:34,855 ERROR 8 is not = 1 2020-12-29 11:03:34,855 ERROR 9 is not = 1 2020-12-29 11:03:34,855 ERROR 10 is not = 1 2020-12-29 11:03:34,855 ERROR 4 is not = 1 2020-12-29 11:03:34,856 ERROR 5 is not = 1 2020-12-29 11:03:34,856 ERROR 4 is not = 1 2020-12-29 11:03:34,856 ERROR 3 is not = 1 2020-12-29 11:03:34,856 ERROR 4 is not = 1 2020-12-29 11:03:34,857 ERROR 5 is not = 1 2020-12-29 11:03:34,857 ERROR 4 is not = 1 2020-12-29 11:03:34,857 ERROR 3 is not = 1 2020-12-29 11:03:34,857 ERROR 4 is not = 1 2020-12-29 11:03:34,858 ERROR 5 is not = 1 2020-12-29 11:03:34,858 ERROR 4 is not = 1 2020-12-29 11:03:34,858 ERROR 3 is not = 1 2020-12-29 11:03:34,858 ERROR 4 is not = 1 2020-12-29 11:03:34,858 ERROR 3 is not = 1
Not sure why would you set max_workers=1 though :-)