Re: [Python-Dev] pthreads, fork, import, and execvp
Guido van Rossum wrote: > Yeah, I think imports inside functions are overused. Rotem took the time to explain the problem to me more fully in private email, and it appears that using import statements inside functions is currently fundamentally unsafe in a posix environment where both multiple threads and fork() are used. If any thread other than the one invoking fork() holds the import lock at the time of the fork then import statements in the spawned process will deadlock. So I believe fixing this properly would indeed require assigning a new lock object in one of the two places Rotem suggested. Cheers, Nick. > > On 5/9/06, Rotem Yaari <[EMAIL PROTECTED]> wrote: >> Hello everyone! >> >> We have been encountering several deadlocks in a threaded Python >> application which calls subprocess.Popen (i.e. fork()) in some of its >> threads. >> >> This has occurred on Python 2.4.1 on a 2.4.27 Linux kernel. >> >> Preliminary analysis of the hang shows that the child process blocks >> upon entering the execvp function, in which the import_lock is acquired >> due to the following line: >> >> def _ execvpe(file, args, env=None): >> from errno import ENOENT, ENOTDIR >> ... >> >> It is known that when forking from a pthreaded application, acquisition >> attempts on locks which were already locked by other threads while >> fork() was called will deadlock. >> >> Due to these oddities we were wondering if it would be better to extract >> the above import line from the execvpe call, to prevent lock >> acquisition attempts in such cases. >> >> Another workaround could be re-assigning a new lock to import_lock >> (such a thing is done with the global interpreter lock) at PyOS_AfterFork or >> pthread_atfork. >> >> We'd appreciate any opinions you might have on the subject. >> >> Thanks in advance, >> >> Yair and Rotem >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r45925 - in python/trunk: Lib/tempfile.py Lib/test/test_os.py Misc/NEWS Modules/posixmodule.c
Martin v. Löwis wrote: > M.-A. Lemburg wrote: >> I was leaving those out already - only the codes named 'ERROR_*' >> get included (see attached parser and generator). > > Right. One might debate whether DNS_INFO_AXFR_COMPLETE (9751L) > or WSAEACCES (10013L) should be included as well. The WSA codes are already included in the errno module. Not sure about the DNS codes, but it would be easy to add them as well. > I got a smaller source file as I included only forward mappings, > and used a loop to create the backwards mappings. > >> Using a lookup object is not really clumsy - you can still access >> all the values by attribute access. The only difference is that >> they don't live in the module namespace, but get accessed via >> an object. > > So how much space would that save? I'll have to write the lookup code first. I expect a savings of between 2-3 times since the data will be stored in C static data. This is only swapped in as needed and can be shared between processes, so quantifying the savings is difficult. >> I'm not worried about the disk space being used. The heap >> memory usage is what's worrying: the import of the module lets >> the non-shared memory size of the process grow by 700kB >> on my AMD64 box. > > That number must be misleading somehow. The number does look a bit high - it is possible that the process has to swap in some of the shared memory. But then again, the shared memory also increases: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Before import: lemburg 26515 0.0 0.4 19936 4620 pts/3S+ 13:37 0:00 python After: lemburg 26515 0.0 0.5 20780 5320 pts/3S+ 13:37 0:00 python > There are 1510 strings, > with a total length of 39972. There are 1510 integers also, > and they all get added into three dictionaries. Well, the strings and integers count twice: once in the module namespace and once in the errorcode dictionary. > On a 32-bit machine, these should consume 76968 bytes for the > strings (*), 18120 bytes for the integers, and 10 bytes > for the dict entries (**), for a total of 20 bytes > at run-time. > > On a 64-bit machine, the strings should consume 101128 bytes (***), > the integers 24160, and the dict entries 20 bytes, > for a total of 325000 bytes. Given that the code strings and integers are created twice in my version of the module, the numbers sound about right. I agree that creating only one dictionary statically and the other mapping dynamically will already be a saving of 50% simply by sharing the string and integer objects. >>From that, I would conclude that one should avoid 64-bit machines > if one is worried about memory usage :-) > > Regards, > Martin > > (*) assuming 20 bytes string header, 1 byte null-termination, > and a rounding-up to the next multiple of 8 > (**) assuming 12 bytes per dict entry in three dictionaries > (winerror.__dict__, winerror.errorcode, interning dict), > and assuming an average fill ratio of the dicts of 50% > (***) assuming 40 bytes string header, provided long is > a 64-bit type on that platform -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 15 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Building with VS 2003 .NET
I recently checked out the 2006-02-04 python trunk, but I can't get it to build in Visual Studio 2003 .NET. When I open up the PCbuild\pcbuild.sln file in VS2003 .NET and then try to build the solution, I get the following errors: fatal error C1083: Cannot open include file: 'bzlib.h': No such file or directory fatal error C1083: Cannot open include file: 'tcl.h': No such file or directory fatal error C1083: Cannot open include file: 'tcl.h': No such file or directory error PRJ0019: A tool returned an error code from "Performing Makefile project actions" fatal error C1083: Cannot open include file: 'dbl.h': No such file or directory I'm surprised, and mystified, by this--since it seems to me that the .sln file ought to build properly. Is there something else I need to check out from subversion in order to get the code to build? I'd appreciate any help that anyone can offer. Thanks, Chris ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Building with VS 2003 .NET
On 5/15/06, R. Christian Call <[EMAIL PROTECTED]> wrote: > I recently checked out the 2006-02-04 python trunk, but I can't get it > to build in Visual Studio 2003 .NET. > > When I open up the PCbuild\pcbuild.sln file in VS2003 .NET and then try > to build the solution, I get the following errors: > > fatal error C1083: Cannot open include file: 'bzlib.h': No such file or > directory > fatal error C1083: Cannot open include file: 'tcl.h': No such file or > directory > fatal error C1083: Cannot open include file: 'tcl.h': No such file or > directory > error PRJ0019: A tool returned an error code from "Performing Makefile > project actions" > fatal error C1083: Cannot open include file: 'dbl.h': No such file or > directory > > I'm surprised, and mystified, by this--since it seems to me that the > .sln file ought to build properly. > > Is there something else I need to check out from subversion in order to > get the code to build? > > I'd appreciate any help that anyone can offer. Python depends on a number of external libraries (TCL, BZip2, Berkeley DB). You need to obtain and set up these libraries, or disable these extensions. Full details are in the "readme.txt" file in the PCBuild directory. You should read and follow the instructions there. You should probably post questions like this to the comp.lang.python newsgroup. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Building with VS 2003 .NET
[R. Christian Call] > I recently checked out the 2006-02-04 python trunk, but I can't get it > to build in Visual Studio 2003 .NET. > > When I open up the PCbuild\pcbuild.sln file in VS2003 .NET and then try > to build the solution, I get the following errors: > > fatal error C1083: Cannot open include file: 'bzlib.h': No such file or > directory [etc] > I'm surprised, and mystified, by this--since it seems to me that the > .sln file ought to build properly. > > Is there something else I need to check out from subversion in order to > get the code to build? Read PCbuild/readme.txt. Then do what it says ;-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r46005 - in python/trunk: Lib/tarfile.py Lib/test/test_tarfile.py Misc/NEWS
> Author: georg.brandl > Date: Mon May 15 21:30:35 2006 > New Revision: 46005 > > Modified: >python/trunk/Lib/tarfile.py >python/trunk/Lib/test/test_tarfile.py >python/trunk/Misc/NEWS > Log: > [ 141 ] tarfile.py: support for file-objects and bz2 (cp. #1488634) "Something went wrong" here on Windows. The Windows buildbot slaves other than mine eventually died with: """ ... test_tarfile command timed out: 1200 seconds without output """: I was working on my box at the time, and it became dramatically unusable: took about two minutes just to swap enough of the OS back in to _bring up_ Task Manager to see what was going on. The Python test process was close to 2GB in size, and of course the box was swapping madly. So I killed python_d.exe and waited to see how the other buildbots fared. Sure appears to be a Windows-specific problem -- or maybe all the Linux boxes come with a terabyte of RAM these days ;-) I'll take a peek later, but won't be disappointed if someone else figures it out first. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r46005 - in python/trunk: Lib/tarfile.py Lib/test/test_tarfile.py Misc/NEWS
[Tim] > "Something went wrong" here on Windows. The Windows buildbot slaves > other than mine eventually died with: > > """ > ... > test_tarfile > > command timed out: 1200 seconds without output > """: > > I was working on my box at the time, and it became dramatically > unusable: took about two minutes just to swap enough of the OS back > in to _bring up_ Task Manager to see what was going on. ... OK, I fixed that, but unfortunately the checkin comment was correct: The Windows buildbot slaves shouldn't swap themselves to death anymore. However, test_tarfile may still fail because of a temp directory left behind from a previous failing run. Windows buildbot owners may need to remove that directory by hand. I did that on my box, and test_tarfile passes there now, but test_tarfile is still failing on other Windows slaves; e.g., http://www.python.org/dev/buildbot/all/x86%20W2k%20trunk/builds/694/step-test/0 ... WindowsError: [Error 183] Cannot create a file when that file already exists: 'c:\\docume~1\\trentm~1.act\\locals~1\\temp\\testtar.dir\\directory' ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r46005 - in python/trunk: Lib/tarfile.py Lib/test/test_tarfile.py Misc/NEWS
On 5/15/06, Tim Peters <[EMAIL PROTECTED]> wrote: [Tim]> "Something went wrong" here on Windows. The Windows buildbot slaves> other than mine eventually died with:>> """> ...> test_tarfile>> command timed out: 1200 seconds without output > """:>> I was working on my box at the time, and it became dramatically> unusable: took about two minutes just to swap enough of the OS back> in to _bring up_ Task Manager to see what was going on. ... OK, I fixed that, but unfortunately the checkin comment was correct:The Windows buildbot slaves shouldn't swap themselves to deathanymore. However, test_tarfile may still fail because of a temp directory left behind from a previous failing run.Windows buildbot owners may need to remove that directoryby hand.I did that on my box, and test_tarfile passes there now, buttest_tarfile is still failing on other Windows slaves; e.g.,http://www.python.org/dev/buildbot/all/x86%20W2k%20trunk/builds/694/step-test/0...WindowsError: [Error 183] Cannot create a file when that file already exists: 'c:\\docume~1\\trentm~1.act\\locals~1\\temp\\testtar.dir\\directory'___ Ignorant question probably, but why can't the test just check for the directory first, and remove it if it exists? -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r46005 - in python/trunk: Lib/tarfile.py Lib/test/test_tarfile.py Misc/NEWS
[Brett Cannon] > Ignorant question probably, but why can't the test just check for the > directory first, and remove it if it exists? Because it's a stupid hack that should never be necessary: the test cleans up after itself in a "finally" clause. It only looks attractive right now because an earlier bug caused the Windows boxes to swap themselves to death, and the "finally" clause never executed. So, what the heck, sure, I checked that in :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] FYI: building on OS X
I just built 2.5 SVN on my OS X 10.4 laptop. I ran into the missing sys/statvfs.h problem that has been reported here in the past. To solve the problem I needed to upgrade my XCode install from 1.1 to 2.2.1 - the missing header files (there was actually a whole lot of them) were then installed. I don't know whether this needs to be mentioned somewhere in the build docs. Richard ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] FYI: building on OS X
On 5/15/06, Richard Jones <[EMAIL PROTECTED]> wrote: I just built 2.5 SVN on my OS X 10.4 laptop. I ran into the missingsys/statvfs.h problem that has been reported here in the past. Tosolve the problem I needed to upgrade my XCode install from 1.1 to2.2.1 - the missing header files (there was actually a whole lot of them) were then installed.I don't know whether this needs to be mentioned somewhere in thebuild docs.Wouldn't hurt.-Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r45925 - in python/trunk: Lib/tempfile.py Lib/test/test_os.py Misc/NEWS Modules/posixmodule.c
M.-A. Lemburg wrote: > Well, the strings and integers count twice: once in the module > namespace and once in the errorcode dictionary. That shouldn't be the case: the strings are interned (as they are identifier-like), so you have the very same string object in both dictionaries. The numbers shouldn't be duplicated because they occur in the co_consts array of the global code object, and because the compiler should share them there. > Given that the code strings and integers are created > twice in my version of the module, the numbers sound about > right. If they are indeed created twice, something is wrong. > I agree that creating only one dictionary statically > and the other mapping dynamically will already be a > saving of 50% simply by sharing the string and integer > objects. No, they should be shared already, so that shouldn't save anything. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
Nick Coghlan wrote:
> Rotem took the time to explain the problem to me more fully in private email,
> and it appears that using import statements inside functions is currently
> fundamentally unsafe in a posix environment where both multiple threads and
> fork() are used. If any thread other than the one invoking fork() holds the
> import lock at the time of the fork then import statements in the spawned
> process will deadlock.
>
> So I believe fixing this properly would indeed require assigning a new lock
> object in one of the two places Rotem suggested.
I still don't understand the problem fully. According to POSIX, fork does:
"A process is created with a single thread. If a multi-threaded process
calls fork(), the new process contains a replica of the calling thread
and its entire address space, possibly including the states of mutexes
and other resources."
So if the the import lock was held during the fork call, the new thread
will hold the import lock of the new process, and subsequent imports
will block.
However, then the problem is not with the execve implementation, but
with the fact that the import lock was held when the process forked.
Rotem should simply avoid to fork() in the toplevel code of a module.
This problem should be fixed in Python 2.5, whose PyOS_AfterFork now
reads
void
PyOS_AfterFork(void)
{
#ifdef WITH_THREAD
PyEval_ReInitThreads();
main_thread = PyThread_get_thread_ident();
main_pid = getpid();
_PyImport_ReInitLock();
#endif
}
This was added in
r39522 | gvanrossum | 2005-09-14 20:09:42 +0200 (Mi, 14 Sep 2005) | 4 lines
- Changes donated by Elemental Security to make it work on AIX 5.3
with IBM's 64-bit compiler (SF patch #1284289). This also closes SF
bug #105470: test_pwd fails on 64bit system (Opteron).
Unfortunately, that fix would apply to AIX only, though:
void
_PyImport_ReInitLock(void)
{
#ifdef _AIX
if (import_lock != NULL)
import_lock = PyThread_allocate_lock();
#endif
}
If we want to support fork while an import is going on, we should
release the import lock if it is held. Alternatively, the code
could throw away the old import lock on all systems; that seems
like a waste of resources to me, though. One should also reset
import_lock_thread and import_lock_level.
I'd be curious if Elemental Security added this code only to solve
the very same problem, i.e. a fork that happened with the import
lock held.
Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
