Python Forum
Logger file rotation not working when python code started from windows service as exe
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Logger file rotation not working when python code started from windows service as exe
#1
I am starting a python application (exe version created using pyInstaller) from a windows service (created in python, converted to exe using pyinstaller and installed using sc) but the log files generated by my application are not getting rotated. The code creates one log file and after it gets filled upto set limit for rotation, the logs just get stuck there while the code keeps on working perfectly fine.

So I have actually used a logger.conf file which has logger configuration with rotatingFileHandler to rotate file after every 10KBs (for testing purpose). The configuration in conf file looks like this:

[handler_fileRotationHandler]
class=logging.handlers.RotatingFileHandler
level=NOTSET
formatter=simpleFormatter
args=('<absolute path of log file>','a',10240,5)
Inside python code I am using this config to create and rotate log files like below code:

import common
import os
import sys

#current working directory path of script
if getattr(sys, 'frozen', False):
	# we are running in a bundle
	common.COMMON_PATH = sys._MEIPASS
else:
	# we are running in a normal Python environment
	common.COMMON_PATH = os.path.dirname(os.path.abspath(__file__))
	
#locations
common.LOG_FILE=common.COMMON_PATH+'/logging.conf'

#logger config
logging.config.fileConfig(common.LOG_FILE)
common.logger = logging.getLogger('<name of logger from logging.conf>')
logger = common.logger
In above code snippet, common.py is a python file created for declaring and initializing some common vars of whole source code.

Here is a list of cases when proper log file rotation works:
1- The python version started using python command
2- The exe version (created using PyInstaller) works fine when launched by double clicking directly
3- The exe version when started from a windows service also created in python, if service is installed using python commands like below:
MyService.py install

Now here is when it does not work:
I am converting the windows service code to exe (using pyInstaller again) and install the service with sc using below command:
sc create MyService binPath= "<absolute path of service exe file>"

When started using this service, the application works fine, and log file is generated too, but after reaching the max size defined in fileHandler for log, it does not create another log file and thus gets stuck in terms of log there only. The application keeps on working perfectly fine, just logs doesn't get recorded.

Here is what I have tried and observed:
1- In both cases, I have launched my app exe version using subprocess.Popen() command and my application does not have any UI element, so it runs perfectly in windows session 0 in background. Just for the information, in case it is relevant.
2- If I remove existing log statements and empty log file, logs start to get logged in the file and again stop when max size reach.
3- I have used os.getcwd() command to get directory where my app runs when launched in both cases for which I found below directories:

in python service installed using python case, app runs in "C:\Users\nirvantosh\AppData\Local\Programs\Python\Python36\lib\site-packages\win32"

while in exe version launched from service installed using sc case, app runs in "C:\Windows\system32"

Although in both cases, the logging.conf file provides the log file creation path, so I am assuming this should not be any issue (in fact, log file does gets created at expected location, just file rotation does not work, so this I guess is not relevant)

I need to use the exe version installed by sc only and not python service version. How to solve this issue, any help or guidance or direction is appreciated.

Just in case required, I am using:
python 3.6.3
pyInstaller 3.4
Reply
#2
[For anybody interested to know the solution]

So, the issue I was facing is a common one in RotationFileHandler which is of two handlers dealing with same file simultaneously thus, hindering RotationFileHandler to rename filled log file.

To elaborate above, I put exception handling in RotationFileHandler source code in handler.py and printed exception in a file. From there I got the issue as:

Error:
... [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\<path to log>\\test.log' -> 'C:\\Users\\<path to log>\\test.log.1' ...
On searching more, I found it is a common scenario with FileRotationHandler in logger in windows where sometimes multiple handles are open to same log file due to many reasons possible and thus, rotation file handler is not able to rename the file.

thread with extensive discussion on similar things including scenarios where this happens

I used procexp tool to check whether my log file is having multiple handles (opened by other applications). I found that since, I was using same logging.conf for both service code and application code, both of them were having open handles to log files for both. I created a separate config file for service code and the issue was gone. :)

The only thing not yet known to me is why this was only happening in exe versions of both service and application and not in python versions, cuz ideally the issue should have occurred when run as python code too.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Python logging RotatingFileHandler writes to random file after the first log rotation rawatg 0 338 Feb-15-2024, 11:15 AM
Last Post: rawatg
  working directory if using windows path-variable chitarup 2 680 Nov-28-2023, 11:36 PM
Last Post: chitarup
  New to Python - Not sure why this code isn't working - Any help appreciated TheGreatNinx 4 908 Jul-22-2023, 10:21 PM
Last Post: Pedroski55
  code not working when executed from flask app ThomasDC 1 834 Jul-18-2023, 07:16 AM
Last Post: ThomasDC
  New to python/coding Need help on Understanding why this code isn't working. Thanks! mat3372 8 1,662 May-09-2023, 08:47 AM
Last Post: buran
  logger behaviour setdetnet 1 853 Apr-15-2023, 05:20 AM
Last Post: Gribouillis
  Cannot get started standenman 4 1,147 Feb-22-2023, 05:25 PM
Last Post: standenman
  I am new to python and Could someone please explain how this below code is working? kartheekdas 2 971 Dec-19-2022, 05:24 PM
Last Post: kartheekdas
  how to write exception error into logger mg24 3 947 Nov-15-2022, 04:20 PM
Last Post: insharazzak
  python insert blank line in logger mg24 1 2,723 Nov-02-2022, 08:36 AM
Last Post: snippsat

Forum Jump:

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020