[issue4749] Issue with RotatingFileHandler logging handler on Windows

Changes by Piotr Dobrogost : -- nosy: +piotr.dobrogost ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Changes by Yongzhi Pan : -- nosy: +fossilet ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Thom Nichols added the comment: I just ran into this issue, clearly it is old and Python 2.6, but I just experienced it in a single-process linux app running on a production embedded system. root@at91sam9g20ek:~# python Python 2.6.5 (r265:79063, Sep 28 2011, 11:52:59) [GCC 4.3.3] on linux2

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: João Paulo Farias added the comment: > Ok.. I was hoping for a solution that did not involve running a > separate process just for logging. It needn't *just* be for logging - as long as it doesn't spawn any child processes. And the need to run a separate proc

[issue4749] Issue with RotatingFileHandler logging handler on Windows

João Paulo Farias added the comment: Ok.. I was hoping for a solution that did not involve running a separate process just for logging. Gabriel: Yeah, I have a python application that uses threads and spawns processes, even tho the processes it spawns have nothing related to the log file. A

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: > From: João Paulo Farias > I dont see the resolution for this problem yet... What should I > do to not have it happen? Don't use RotatingFileHandler in any process which spawns a subprocess. In a multiprocess situation where you want to use rotating log files

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Gabriel Genellina added the comment: @jpfarias: could you be more specific? Which error(s) do you have? in which scenario? -- ___ Python tracker ___

[issue4749] Issue with RotatingFileHandler logging handler on Windows

João Paulo Farias added the comment: I dont see the resolution for this problem yet... What should I do to not have it happen? -- nosy: +jpfarias ___ Python tracker ___

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Changes by Gabriel Genellina : -- nosy: +gagenellina ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Erik Antelman added the comment: The really annoying this about handle inheritance is that even if a subprocess is never referencing or using logging there can be an open file handle conflict due to the default inheratence. Another tack from modifying Popen, I looked at the forking.py of mu

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: Lowell, there's a working sample right there in the docs: I mentioned it in msg78619 in this issue. The link is http://docs.python.org/library/logging.html#sending-and-receiving-logging-events-across-a-network This should get anyone started for a multiprocess lo

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Lowell Alleman added the comment: For anyone who is interested, I did some testing with the ConcurrentRotatingFileHandler from my ConcurrentLogHandler package (v 0.8.3). I've found that it does not fail when dropped into the revised-logthred.py script. However, it does spend most of the time in

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: One more possibly related bug is issue2320 where subprocesses are spawned from multiple threads. They found an interesting workaround that I found seems to help our problem too: "on Windows, if close_fds is true then no handles will be inherited by the child p

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: I found Issue1425127 which may be a different symptom of this core problem. I suggested that we create a bug that documents the core problem here as described by Vinay in msg89174 and links to these two bugs (along with any others we find) as examples of the t

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: You may want to re-test with Popen(..., close_fds=True) with the latest Python 2.6. From the latest docs: http://docs.python.org/library/subprocess.html "If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is execu

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: I changed the script to use subprocess (attached file) and got the same rollover errors as before. I had to change cd and del to be cd.bat and del.bat which contained cd %1 and del %1 respectively since it appears subprocess can't run internal commands like cd

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Lowell Alleman added the comment: I must say that Vinay's findings are most interesting. Thanks Vinay for tracking this down! Just a thought, but has anybody tried this using the subprocess module? I've glanced through subprocess.py and it certainly does not use os.system(). Instead it uses

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: The problem may occur just because of the way file handles work with child processes. I'm not sure if it's a bug, or even if it's specific to Python, so I'm not going to raise another issue. It's worth searching to see if there's an existing issue about it, though.

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: Vinay - that's great news! Are you going to create a new bug for this issue with a proper title? It would seem to me that the fix for this would be to put locks internal to the os.system() call around where it spawns cmd so multiple spawns don't occur simulta

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Changes by Vinay Sajip : -- resolution: -> invalid status: open -> closed Added file: http://bugs.python.org/file14249/python-io-capture.log ___ Python tracker ___ __

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Changes by Vinay Sajip : Added file: http://bugs.python.org/file14248/thredio.py ___ Python tracker ___ ___ Python-bugs-list mailing list Unsub

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: I've just run a test several times - it's your original script with joins added at the end. I kept the acquire_lock and release_lock calls but made them only do anything if a constant USE_LOCK was True. I set this to False, so that no locking is actually occurring.

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: I turned off anti-virus again as well as file indexing and google desktop too and still got the errors when I disabled the locks around the os.system() calls. Vinay - when the locks aren't around the os.system() calls, do you get the rotating log errors? I'm

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: I'll thoroughly look through every piece of software that's running to see if I can turn eveything off that might be causing the problem. Were you able to reproduce the problem with my original script? I'm sure you have all of your virus/searching/etc. stuff t

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: Debugging threading problems is never as simple as it seems. I don't believe logging is breaking because of some interaction with the system calls - rather, the presence of the system calls is changing the real-time characteristics of the system and exposing the fa

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: Thanks Vinay. I ran the newest revised script with virus protection turned off and got the same failures as before (see console output below). If you comment out the os.system() calls, everything works just fine. Uncomment them and logging breaks. The os.sy

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: The last sentence in my last comment got truncated - it's getting late here ;-) I meant to say: If you are seeing a WindowsError with an error code of 32 (this might get lost in all the other output, but it's the first error I found - the other messages look like

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: On Ubuntu, without the locks around the os.system calls, I was not getting any errors other than file-not-found errors on blah.txt - which you would expect if another thread had just deleted the file. You didn't post the errors which your test script was generating

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: Whoops, my revised test script still had some tabs. Replacing. Further information on Windows - you sometimes get error 32 (cannot rename file because owned by another process) because of anti-virus or Windows indexing software having the file open. Can you please

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Changes by Vinay Sajip : Removed file: http://bugs.python.org/file14238/revised-logthred.py ___ Python tracker ___ ___ Python-bugs-list mailing

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: I didn't care about the os.system() call contention because that's what caused the logging problem and that blah.txt file contention should not cause logging to fail. I also had the join calls originally but took them out to simplify the code since it seemed t

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: I did some tests on the zip files which Frans attached to this post. The reason you are getting errors is at least partly that you are not following best practices in the test scripts. Specifically, the threads are making system calls to update and delete blah.txt

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Changes by R. David Murray : -- priority: -> normal status: closed -> open type: crash -> behavior versions: -Python 2.4, Python 2.5 ___ Python tracker ___ _

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: I just upgraded to 2.6.2 windows from python.org and it fails as well: Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on win32 I hope Vinay can track this down in case it's a race condition that's just moving around between versio

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: Thanks Lowell - good information. You have many more versions of Python "laying around" than I do. ;) -- ___ Python tracker ___ ___

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: P.S. Frans - It's good to get these other data points from you. So this is reproducible from another person and on different versions of python AND on different platforms! I wasn't expecting that at all. Thanks Frans. Is there a way we can reopen this bug?

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Lowell Alleman added the comment: I tested this against a number of different python installs that I have laying around. Here are the results that I found: Releases that reproduce this bug: Python 2.3.5 (#62, Feb 8 2005, 16:23:02) [MSC v.1200 32 bit (Intel)] on win32 Python 2.4.2 (#67, Sep 2

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: >>> import sys >>> print sys.version 2.6.1 (r261:67517, Dec 4 2008, 16:51:00) [MSC v.1500 32 bit (Intel)] I have seen this behavior in older versions as well. Interesting to see it fail in linux as well. -- ___ Pyt

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Lowell Alleman added the comment: Robert, please provide the Python version and distribution that your are using. This should do the trick: import sys print sys.version -- Added file: http://bugs.python.org/file14227/unnamed ___ Python tracker

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: P.S. The above script and failure is running on winxp sp3. Also, if you comment out the two os.system() calls, it works just fine. They seem like they should be unrelated to the logging though. You'll also see some errors about access to the blah.txt file wh

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: I have a small script that reproduces the problem. I couldn't reproduce it until I added some os.system() calls in the threads that were logging. Here's the output using python 2.6.1: Traceback (most recent call last): File "C:\Python26\lib\logging\handler

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Frans added the comment: Hi Vinay, Thanks for your info. I have just shared my experience. I ran into a problem (apperently, there is one) and found a work-around that fits my needs. If I find the real fix, I will surely follow up on it. Regards, Frans 2009/6/7 Vinay Sajip > > Vinay Sajip

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: ConcurrentLogHandler is for multiple *processes* writing to the same file, not multiple threads in a single process. Python logging does not support multiple processes writing to the same file because there is no portable IPC locking across all platforms supported

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Frans added the comment: I ran into the same problem with RotatingFileHandler from a multithreaded daemon under Ubuntu. I Googled around and found the ConcurrentLogHandler on pypi (http://pypi.python.org/pypi/ConcurrentLogHandler). It solved the problem. -- nosy: +Frans ___

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: I will go through the code and make sure I am not mistaken, but I do believe I have a single process, multiple threads, and only one handler for this file and I'm getting the same types of error messages shown on this page. I'm probably doing something wrong.

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: This exact problem? Single process, multiple threads, one single handler for a given file? Can you post an example which demonstrates your issue? Multiple threads *are* supported by the logging package, as mentioned in the documentation. If you can post a an examp

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Robert Cronk added the comment: I have had this problem with 2.6.1 on windows from multiple _threads_ instead of multiple processes. Is that not supported either? If not, what is the workaround for logging from multiple threads? Shouldn't it be easy to use a semaphore within the logging co

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Neyro added the comment: @vsajip thank you very much for correcting me. I always thought that since I was not using that FileHandler it didn't count, as if the handler was not created. Now I see that there are no reasons for the config loader to ignore it, especially since it is declared in the

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: Neyro, your problem is caused by having two handlers (a FileHandler and a RotatingFileHandler) pointing to the same file. Because of this, rollover fails (the file is still open because there is a handle open to it from the FileHandler). If you remove the FileH

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Neyro added the comment: I have the same problem with a program I'm writing. The attached script is made using parts of it's code. The logger is set up using a configuration file. By the way, the error is: Traceback (most recent call last): File "C:\Python26\lib\logging\handlers.py", line 74

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: Closed, as no further feedback received. -- status: pending -> closed ___ Python tracker ___ ___ Python

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Vinay Sajip added the comment: Can you (Mohammad) say which specific tickets raised the same issue? I downloaded and ran your script (Windows XP Pro, ActivePython 2.5.2.2) and had no problems. I created a .bat file with 100 invocations of the .py file. Lowell Alleman refers to concurrent use of

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Changes by Benjamin Peterson : -- assignee: -> vsajip nosy: +vsajip ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscri

[issue4749] Issue with RotatingFileHandler logging handler on Windows

Lowell Alleman added the comment: I've ran into the same problem before. I've found that due to differences between the way Unix and Windows handles files (inodes vs file handles), this problem is more apparent on Windows, but it isn't handled 100% correctly on Unix systems either. I think the

[issue4749] Issue with RotatingFileHandler logging handler on Windows

New submission from Mohammad Al : It seems that RotatingFileHandler is experiencing an issue when attempting to rotate log files under Windows. The following errors are received: Traceback (most recent call last): File "C:\Python25\lib\logging\handlers.py", line 73, in emit if self.shou