Hi, I wrote a quick module for spawning a subprocess and run the
function, just as an excercise, after I read your post...
It will spawn a subprocess, then pass the serialized return values to
the parent. So, if you want to do:
lala = os.listdir(".")
you do
lala = fmg(os.listdir, ".")
This will not block, but it will run only on UNIX/Linux, unless there is
a version of python that includes support for cygwin, which has fork()
for Win32.
The module is below:
###################################################################
"""
fmg.py
As all threads will block on a call to some os functions, this implements a
function for doing thread-safe syscalls/C library functions using a child
process.
Availability: UNIX
"""
import os
import cPickle
def fork_me_gently(myfunct, *myargs):
"""fork_me_gently(callable, arguments) -> retval
Make an arbitrary function pseudo thread-safe by running
it in a child process. Return what the passed callable returns."""
#prepare a pipe for child-parent communication
r_end, w_end = os.pipe()
pid = os.fork()
if pid == 0: #child
retval = myfunct(*myargs)
#Serialize list for sending to pipe
#dump() needs a file object, not a fd
w_end_fo = os.fdopen(w_end, 'w')
cPickle.dump(retval, w_end_fo)
w_end_fo.close()
# DIE, CHILD!
raise SystemExit
else: #parent
# load() needs a file object
r_end_fo = os.fdopen(r_end)
retval = cPickle.load(r_end_fo)
#clean up after yourself before continuing
os.waitpid(pid, 0)
return(retval)
fmg = fork_me_gently
if __name__ == "__main__":
#temp = ts_listdir(".")
temp = fmg(os.listdir, ".")
for i in temp:
print i
Wolfgang Braun wrote:
> Hello List,
>
>
> I try to read large directories off network shares (samba3,NT) with
> os.listdir(). The listdir() takes up to 5 minutes and blocks the rest of
> the program (gui refreshes, ...)
>
> The idea now is to put the listdir call into a separate thread so the
> program can go on doing stuff (see below).
>
> Unfortunately, as soon as I start a rdir instance the whole python
> process locks up and waits for the os.listdir to return.
>
> Plan b was to popen('/bin/ls -1 '%dir) in the rdir thread which works
> better but is a bit kludgy (needs to work on NT, too) and the program
> cannot be shudown cleanly unless the rdir thread has finished.
>
>
> Obviously I'm doing something wrong? What would be the Right Way to
> handle this situation?
>
>
> Thanks,
> Wolfgang
>
>
> # --- skeleton listdir in a thread
>
> class rdir(threading.Thread):
> def __init__(self,dir,glob=None):
> super(rdir,self).__init__()
> self.dir=dir
> self.content=()
>
> def start(self):
> self.setDaemon(True)
> super(dircache,self).start()
>
> def run(self):
> self.content=os.listdir(self.dir)
> # alternatively os.popen(' /bin/ls -1U '%dir)
> # stuff to keep run() from returning
>
> _______________________________________________
> Tutor maillist - Tutor@python.org
> http://mail.python.org/mailman/listinfo/tutor
_______________________________________________
Tutor maillist - Tutor@python.org
http://mail.python.org/mailman/listinfo/tutor
