[issue4749] Issue with RotatingFileHandler logging handler on Windows
Lowell Alleman added the comment: I've ran into the same problem before. I've found that due to differences between the way Unix and Windows handles files (inodes vs file handles), this problem is more apparent on Windows, but it isn't handled 100% correctly on Unix systems either. I think the root of the problem is that there is nothing in the code to handle multiple concurrent processes writing to a single log file; and I'm not sure there is a simple fix for this. I tried several easy solutions to this problem by retrying failed file renames and re-opening closed files, but I ultimately discovered all all such approaches are inadequate and can actually result in losing old log files too (in the worse-case scenario). I final got frustrated enough to just take the time to write my own. It is based on the built-in one and aims to be a "drop-in" replacement. I use file locking to safely write to a single log file from multiple python processes concurrently. If you would like to give it a try, it is located here: http://pypi.python.org/pypi/ConcurrentLogHandler I agree that it would be nice for the built in logging handlers to do this for you, but in the mean time this may be option for you. -- nosy: +lowell87 ___ Python tracker <http://bugs.python.org/issue4749> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Lowell Alleman added the comment: Robert, please provide the Python version and distribution that your are using. This should do the trick: import sys print sys.version -- Added file: http://bugs.python.org/file14227/unnamed ___ Python tracker <http://bugs.python.org/issue4749> ___Robert, please provide the Python version and distribution that your are using. This should do the trick:import sysprint sys.version ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Lowell Alleman added the comment: I tested this against a number of different python installs that I have laying around. Here are the results that I found: Releases that reproduce this bug: Python 2.3.5 (#62, Feb 8 2005, 16:23:02) [MSC v.1200 32 bit (Intel)] on win32 Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on win32 Python 2.4.5 (#2, Jul 31 2008, 18:51:48) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 (Ubuntu 8.04) Python 2.4.6 (#2, Mar 19 2009, 10:00:53) [GCC 4.3.3] on linux2 (Ubuntu 9.04) Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32 The following release worked fine: Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 (Ubuntu 8.04) Python 2.5.4 (r254:67916, Apr 4 2009, 17:55:16) [GCC 4.3.3] on linux2 (Ubuntu 9.04) Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41) [GCC 4.3.3] on linux2 (Ubuntu 9.04) -- ___ Python tracker <http://bugs.python.org/issue4749> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Lowell Alleman added the comment: I must say that Vinay's findings are most interesting. Thanks Vinay for tracking this down! Just a thought, but has anybody tried this using the subprocess module? I've glanced through subprocess.py and it certainly does not use os.system(). Instead it uses CreateProcess() on windows and os.execvp*() on all other platforms. It also appears to go to great effort to properly deal with file handles, so I'm wondering if that would resolve this issue. (The current 2.6 docs, state that the subprocess "module is preferable" over the os.system() call, and that "All of the popen*() functions are obsolete. Use the subprocess module.") I'm quite curious to see if my ConcurrentLogHandler will fare any better with this scenario. I would really like for it to be able to handle this scenario, since it's design goals are to be resilient to this type of thing. But I'm relying on the threading system and locks too, so it's hard to say what will happen. Robert, I agree that submitting a new bug on this would be a good idea if none currently exists. I also think it would be good to to put a warning in the docs about this scenario if there is nothing that can be done to correct the situation. Even it if is not Python-specific thing, I think it is good to let people know about gotchas whenever possible. -- ___ Python tracker <http://bugs.python.org/issue4749> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4749] Issue with RotatingFileHandler logging handler on Windows
Lowell Alleman added the comment: For anyone who is interested, I did some testing with the ConcurrentRotatingFileHandler from my ConcurrentLogHandler package (v 0.8.3). I've found that it does not fail when dropped into the revised-logthred.py script. However, it does spend most of the time in what I call "degraded" mode because the file rotation fails--this means that the log file is constantly opened and closed around each write() call so that eventually the log file can be successfully rotated. This is very inefficient, but the design goal of the ConcurrentRotatingFileHandler is to never loose log events at all cost, whereas the RotatingFileHandler is a more general purpose solution and can therefore adhere to more size-based file rotation, and yields a better and more consistent performance. What I found interesting is that the ConcurrentRotatingFileHandler seems to be behaves the same with or without the thread locking around the os.system() calls. This is something that I need to look into when I have some more time. (I haven't actually timed or traced anything yet, so it's hard to say for sure what's really going on; or if something else was going on during my tests) Robert, one alternate approach to consider given all the complexities you have going on (by that I mean, multi-threaded process spawning programs that you need to have the ability to redirect the stdin/stdout streams...) You might want to look into adding one more process in the equation. You could have your script start off by launching a "central logging" socket listener (or use a system wide listener), and then have that single process handle writing to and rotating your log files. Then it's just a mater of redirecting the logging system to push the events to your logging process. I know this might not seem like the most appealing approach, but it may end up being the one that requires the least amount of modifications to your existing code... just a thought. (Vinay, I know you've recommended this approach before, are there any samples laying around to help people get started with a simple central logging solution based on (e.g. built on top of) Python's logging package?) -- ___ Python tracker <http://bugs.python.org/issue4749> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
