I've got an application which makes fairly heavy use of daemon threads to
perform 'background' processing and various other long-running tasks that are
likely to block.
My original understanding of threading.Thread's daemon threads was that I could
safely fire them off and essentially forget about managing them from the main
thread's perspective as long as they don't do anything that's not thread safe
-- eg I can fire them off, let them do their background twiddling and safely
let the threading machinery manage their lifespan assuming that
1. the 'background twiddling' is threadsafe and
2. the thread can safely 'die' at any point without requiring a
shutdown procedure
Some coding later and I learn this isn't exactly the case, as there as in an
additional requirement on the use of daemon threads -- they can't reference
anything in the global namespace else they may raise exceptions at interpreter
shutdown. This is because as part of the shutdown procedure, the interpreter
sets all global variables to None. A daemon thread may run while this is
occuring/after it occured but before the process exits, attempt to access a
global variable and then throw an exception. The exception may get printed if
the interpreter catches/prints it before the process exits.
I garnered this understanding from this problem description -- (although all
mistakes in description are my own)
http://bugs.python.org/issue1722344
In this bug report they are discussing an interpreter problem which affects
non-daemon threads -- I'm not attempting to claim that I'm being affected by an
interpreter bug, I reference this link only because it contains good
descriptions of the interpreter shutdown process as well as the additional
requirements the interpreter places on 'daemon' threads (beyond just being
'thread-safe'):
> When Python begins to shutdown it takes
> each module and sets each variable in the global namespace to None. If a
> thread has not terminated before the interpreter terminates then the
> thread tries to use a global variable which has been set to None.
>
> This is not about to change since this occurs because of coding
> "errors". You must make sure that either your thread is as safe as a
> __del__ method (which means no global namespace access) or you can't let
> the app exit until you are positive all of your threads have terminated,
> not just asked them to shutdown since this is all asynchronous.
> > > which means no global namespace access
> > Does that mean that you cannot use len and range in a Thread?
>
> No, it means you have to be careful if you do. Shutting down properly
> will take care of things. Otherwise you need to save a reference
> locally (either on an object or as a local variable) and use that
> reference instead of relying on the one defined in the global
> namespace.
Here's an example of one such traceback from one of my application runs:
> Exception in thread Thread-11 (most likely raised during interpreter
> shutdown):
> (pydev-2.6) ncohen$
In this run, the process exited before even one traceback finished printing but
this will be timing dependent, -- sometimes I'll see tracebacks from many
backgrounds threads and sometimes just snippets like this. They will typically
be AttributeError's or TypeError's resulting from failed attempts to access
variables from the global namespace. Here's a longer example from another run:
> Traceback (most recent call last):
> File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
> line 525, in __bootstrap_inner
> File "/Users/ncohen/software/viis/viis/apps/permassh.py", line 184, in run
> File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/SocketServer.py",
> line 224, in serve_forever
> <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute
> 'select'
>
> Traceback (most recent call last):
> File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
> line 525, in __bootstrap_inner
> File "/Users/ncohen/software/viis/viis/apps/permassh.py", line 184, in run
> File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/SocketServer.py",
> line 224, in serve_forever
> <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute
> 'select'
> Exception in thread Thread-7 (most likely raised during interpreter shutdown):
> Traceback (most recent call last):
> File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
> line 525, in __bootstrap_inner
> File
> "/Users/ncohen/pydev-2.6/lib/python2.6/site-packages/paramiko/transport.py",
> line 1571, in run
> File
> "/Users/ncohen/pydev-2.6/lib/python2.6/site-packages/paramiko/transport.py",
> line 1386, in _log
> File
> "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py",
> line 1105, in log
> <type 'exceptions.AttributeError'>: 'NoneType' object has no attribute
> 'IntType'
> (pydev-2.6)breathe-wifi:viis.webapp ncohen$
> (pydev-2.6) ncohen$
My question concerns whether or not daemon threads have the inconvenient 'no
references to globals()' requirement imposed only by the interpreter shutdown
process or if there are other additional reasons for the requirement.
0. is it possible for me to consider it 'safe' to have code which can trigger
the above exceptions during interpreter shutdown?
1. if it is possible, and it turns out I'm certain what I'm doing is safe, is
there a good way to squelch those error reports?
Is there perhaps some way to customize the value that all the globals get set
to during interpreter shutdown? -- i think it would work for me to replace the
None to which all globals are set with an object which
1. returns None as usual if accessed from a non-daemon thread
2. blocks forever if accessed from a daemon thread, or maybe calls an
'emergency exit' function, (but I don't think the emergency exit function would
get any guarantees that it would complete running before the process exited) ...
In my case I don't always have control over the daemon threads creation or
lifespan -- in this particular app, I'm using the paramiko library which uses
daemon thread(s) 'behind the scenes' to simplify its networking code ...
Looking forward to any comments/advice or name calling.
Thanks,
Ben
Ben Cohen
Programmer/Analyst (STS)
Scripps Institution of Oceanography
[email protected]
--
http://mail.python.org/mailman/listinfo/python-list
