[issue2320] Race condition in subprocess using stdin

[Adam Olsen]

Adam Olsen <[EMAIL PROTECTED]> added the comment:

This is messy.  File descriptors from other threads are leaking into
child processes, and if the write end of a pipe never gets closed in all
of them the read end won't get EOF.

I suspect "cat"'s stdin is getting duplicated like that, but I haven't
been able to verify - /proc//fd claims fd 0 is /dev/pts/2.  Maybe
libc does some remapping.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3107] memory leak in make test (in "test list"), 2.5.2 not 2.5.1, Linux 64bit

[Tony Wallace]

Tony Wallace <[EMAIL PROTECTED]> added the comment:

> how do you know

Here is the story, sorry I skipped it before- I was at work then. 

I was doing the basic build-from-source on RHEL (Centos) Linux, because
I don't have root and I need to install it in $HOME/something.  I don't
try to change anything except the install location.  See the ./configure
command given before. 

I tried 2.5.2 because it was the latest + greatest.  When I ran "make
test",  it ran OK until it got to "test_list".  Then it got stuck until
I killed it.  Its very repeatable.  In another window, "top" shows the
python process's memory growing and growing, until it owns basically all
available memory (8GB) after less than a minute or so.  

Then I tried python 2.5.1, built exactly the same way, and make test
works OK- test_list completes in seconds.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3104] overzealous garbage collector (dict)

[Paul Melis]

Paul Melis <[EMAIL PROTECTED]> added the comment:

The script is still not a test case, as it doesn't *demonstrate* the
problem when run. You need to provide more information for this to be
reproducable by others. 

- what exact input did you use? (e.g. include the IRC log file on which
you claim a bug is exposed)
- what output/behaviour did you expect for the given input?
- how was the actual output/behaviour different from what was expected?

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3107] memory leak in make test (in "test list"), 2.5.2 not 2.5.1, Linux 64bit

[Martin v. Löwis]

Martin v. Löwis <[EMAIL PROTECTED]> added the comment:

Are you by any chance using gcc 4.3? That compiler is known to
miscompile Python, see issue3019 or issue2626.

Also, I'm puzzled that your prompt says Python-2.5.1 when configuring
with a prefix suggesting 2.5.2.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3107] memory leak in make test (in "test list"), 2.5.2 not 2.5.1, Linux 64bit

[Tony Wallace]

Tony Wallace <[EMAIL PROTECTED]> added the comment:

> are you using gcc 4.3
No, I don't think so. 
[EMAIL PROTECTED] tony]$ gcc --version
gcc (GCC) 3.2.3 20030502 (Red Hat Linux 3.2.3-53)
Copyright (C) 2002 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

And make is definitely using gcc, not something else.


> prompt 2.5.1
Good eye. I grabbed the file-python info from python 2.5.1 after I
installed it (because 2.5.1 passed make test, I am planning to use
that).  Its true, I should have got that from the python 2.5.2, which I
never did install- but unfortunately, it was deleted too soon.  If it is
important I can rebuild 2.5.2 again.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3107] memory leak in make test (in "test list"), 2.5.2 not 2.5.1, Linux 64bit

[Martin v. Löwis]

Martin v. Löwis <[EMAIL PROTECTED]> added the comment:

Are you willing to debug this? Otherwise, I'll have to close this as
"works-for-me" (which it does, as it did for many other people).

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3107] memory leak in make test (in "test list"), 2.5.2 not 2.5.1, Linux 64bit

[Tony Wallace]

Tony Wallace <[EMAIL PROTECTED]> added the comment:

> are you willing
Yes, so long as I don't need root, I can follow instructions OK.
By the way, the same thing (memory leak 2.5.2) occurred on Centos 4.6, a
different Linux box.  Lets proceed on that Centos 4.6 box.  Here are the
particulars:

Python-2.5.2]$ file python
python: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for
GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

[EMAIL PROTECTED] Python-2.5.2]$ gcc --version
gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-9)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[EMAIL PROTECTED] Python-2.5.2]$ cat /etc/[EMAIL PROTECTED] CentOS
release 4.6 (Final)

[EMAIL PROTECTED] Python-2.5.2]$ uname -a
Linux hathi.ambric.local 2.6.9-67.0.15.ELsmp #1 SMP Thu May 8 10:50:20
EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

(after make test)
test_largefile
test_linuxaudiodev
test_linuxaudiodev skipped -- Use of the `audio' resource not enabled
test_list
(stalls here so I dumped it)
make: *** [test] Quit

(another window, just before I dumped it)
top - 01:36:31 up 4 days, 15:07,  5 users,  load average: 2.70, 0.84, 0.32
Tasks:  87 total,   1 running,  86 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.8% us,  9.7% sy,  0.0% ni,  0.0% id, 86.0% wa,  0.0% hi,  0.5% si
Mem:  15639112k total, 15610836k used,28276k free,  280k buffers
Swap: 24579440k total,  1533172k used, 23046268k free,57676k cached

  PID USER  PR  NI %CPUTIME+  %MEM  VIRT  RES  SHR S COMMAND
17190 tony  24   0   17   1:05.49 90.9 32.2g  13g 7200 D python
   69 root  16   06   0:12.97  0.0 000 D kswapd0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3107] memory leak in make test (in "test list"), 2.5.2 not 2.5.1, Linux 64bit

[Martin v. Löwis]

Martin v. Löwis <[EMAIL PROTECTED]> added the comment:

Run "python Lib/test/test_lists.py" separately, and find out which
specific test fails (to complete). Report that, and also try to separate
that test's code into its own file, and see whether you can reproduce it
there. If so, try to further strip it down as much as you can.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3104] overzealous garbage collector (dict)

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

> The original text file was an IRC log. Shoving Charles Dickens' "Great
> Expectations" 17 times in a text file and then parsing it doesn't show
> this problem for some weird reason.

I'd say the "weird reason" is probably a bug in your script. For example
the following appears very dubious:

for o in self.wlist:
if len(o) > 0xFF:
o = o[:0xFF]
fp.write(chr(len(o)))
fp.write(o)
for s in self.wlist[o]:

In any case, the idea that one of Python's built-in containers would
silently *drop* values (rather than segfault or produce a MemoryError)
is in itself quite unbelievable, due to the way those containers funciton.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2650] re.escape should not escape underscore

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Talking about performance, why use a loop to escape special characters
when you could use a regular expression to escape them all at once?

--
nosy: +pitrou

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3088] test_multiprocessing hangs on OS X 10.5.3

[roudkerk]

roudkerk <[EMAIL PROTECTED]> added the comment:

I suspect the problems with WithManagerTestCondition.notify_all() may
have to do with the thread safety of the proxies.  If you replace
Thread(...) by self.Process(...) in that test then the problem may go away.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3088] test_multiprocessing hangs on OS X 10.5.3

[Adam Olsen]

Changes by Adam Olsen <[EMAIL PROTECTED]>:


--
nosy: +Rhamphoryncus

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3109] test_multiprocessing seems fragile

[Jesse Noller]

Changes by Jesse Noller <[EMAIL PROTECTED]>:


--
nosy: +roudkerk

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3109] test_multiprocessing seems fragile

[Jesse Noller]

Changes by Jesse Noller <[EMAIL PROTECTED]>:


--
assignee:  -> jnoller

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3100] weakref subclass segfault

[Jesse Noller]

Changes by Jesse Noller <[EMAIL PROTECTED]>:


--
nosy: +roudkerk

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3110] Multiprocessing package build problem on Solaris 10

[Jesse Noller]

New submission from Jesse Noller <[EMAIL PROTECTED]>:

Per skip's email:
FWIW, it appears that Solaris doesn't define SEM_VALUE_MAX but does 
define
_SEM_VALUE_MAX in sys/params.h.

   .../Modules/_multiprocessing/multiprocessing.c: In function 
'init_multiprocessing':
   .../Modules/_multiprocessing/multiprocessing.c:253: error: 
'SEM_VALUE_MAX' undeclared (first use in this function)
   .../Modules/_multiprocessing/multiprocessing.c:253: error: (Each 
undeclared identifier is reported only once
   .../Modules/_multiprocessing/multiprocessing.c:253: error: for each 
function it appears in.)

On Windows the author simple #defines SEM_VALUE_MAX to be LONG_MAX.  I 
used
a little cpp action to define it:

   #ifndef SEM_VALUE_MAX
   #  ifdef _SEM_VALUE_MAX
   #define SEM_VALUE_MAX _SEM_VALUE_MAX
   #  else
   #define SEM_VALUE_MAX INT_MAX
   #  endif
   #endif

--
components: Extension Modules
messages: 68210
nosy: jnoller, roudkerk, skip.montanaro
severity: normal
status: open
title: Multiprocessing package build problem on Solaris 10
type: compile error
versions: Python 2.6, Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3111] multiprocessing pkg Debian compilation error

[Jesse Noller]

New submission from Jesse Noller <[EMAIL PROTECTED]>:

Both debian ia64 and PPC are failing the mp test suite with the 
following error:

Re-running test 'test_multiprocessing' in verbose mode
test test_multiprocessing crashed -- : [Errno 
38] Function not implemented
Traceback (most recent call last):
  File "./Lib/test/regrtest.py", line 554, in runtest_inner
indirect_test()
  File "/home/pybot/buildarea/trunk.klose-debian-
ppc/build/Lib/test/test_multiprocessing.py", line 1764, in test_main
ProcessesMixin.pool = multiprocessing.Pool(4)
  File "/home/pybot/buildarea/trunk.klose-debian-
ppc/build/Lib/multiprocessing/__init__.py", line 226, in Pool
return Pool(processes, initializer, initargs)
  File "/home/pybot/buildarea/trunk.klose-debian-
ppc/build/Lib/multiprocessing/pool.py", line 84, in __init__
self._setup_queues()
  File "/home/pybot/buildarea/trunk.klose-debian-
ppc/build/Lib/multiprocessing/pool.py", line 131, in _setup_queues
self._inqueue = SimpleQueue()
  File "/home/pybot/buildarea/trunk.klose-debian-
ppc/build/Lib/multiprocessing/queues.py", line 315, in __init__
self._rlock = Lock()
  File "/home/pybot/buildarea/trunk.klose-debian-
ppc/build/Lib/multiprocessing/synchronize.py", line 106, in __init__
SemLock.__init__(self, SEMAPHORE, 1, 1)
  File "/home/pybot/buildarea/trunk.klose-debian-
ppc/build/Lib/multiprocessing/synchronize.py", line 38, in __init__
sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented
[674193 refs]
make: *** [buildbottest] Error 1
program finished with exit code 2

Buildbots:
http://www.python.org/dev/buildbot/stable/ppc%20Debian%20unstable%20trun
k/builds/1607/step-test/0
http://www.python.org/dev/buildbot/stable/ia64%20Ubuntu%20trunk/builds/1
89/step-test/0

--
components: Extension Modules
messages: 68211
nosy: jnoller, roudkerk
severity: normal
status: open
title: multiprocessing pkg Debian compilation error
versions: Python 2.6, Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3111] multiprocessing ppc Debian/ ia64 Ubuntu compilation error

[Jesse Noller]

Changes by Jesse Noller <[EMAIL PROTECTED]>:


--
title: multiprocessing pkg Debian compilation error -> multiprocessing ppc 
Debian/ ia64 Ubuntu compilation error

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3111] multiprocessing ppc Debian/ ia64 Ubuntu compilation error

[Jesse Noller]

Jesse Noller <[EMAIL PROTECTED]> added the comment:

Sorry, should have been debian PPC and ubuntu ia64

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3108] Implicit exception chaining (PEP 3134)

[Guido van Rossum]

Guido van Rossum <[EMAIL PROTECTED]> added the comment:

Submitted as r64281.

We still need a doc patch and changes to the exception printing (perhaps
also to traceback.py).  Please open new issues for those.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3112] implement PEP 3134 exception reporting

[Benjamin Peterson]

New submission from Benjamin Peterson <[EMAIL PROTECTED]>:

Traceback reporting needs to be altered to support exception chaining as
per PEP 3134.

--
components: Interpreter Core
messages: 68214
nosy: benjamin.peterson
priority: critical
severity: normal
status: open
title: implement PEP 3134 exception reporting
type: feature request
versions: Python 2.6

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3113] Document exception chaining

[Benjamin Peterson]

New submission from Benjamin Peterson <[EMAIL PROTECTED]>:

PEP 3134's features need to be documented.

--
assignee: georg.brandl
components: Documentation
messages: 68215
nosy: benjamin.peterson, georg.brandl
priority: high
severity: normal
status: open
title: Document exception chaining
versions: Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3112] implement PEP 3134 exception reporting

[Benjamin Peterson]

Changes by Benjamin Peterson <[EMAIL PROTECTED]>:


--
versions: +Python 3.0 -Python 2.6

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3108] Implicit exception chaining (PEP 3134)

[Benjamin Peterson]

Benjamin Peterson <[EMAIL PROTECTED]> added the comment:

See #3112 and #3113.

--
nosy: +benjamin.peterson

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3112] implement PEP 3134 exception reporting

[Antoine Pitrou]

Changes by Antoine Pitrou <[EMAIL PROTECTED]>:


--
nosy: +pitrou

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3113] Document exception chaining

[Antoine Pitrou]

Changes by Antoine Pitrou <[EMAIL PROTECTED]>:


--
nosy: +pitrou

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3112] implement PEP 3134 exception reporting

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Thinking about this, I realized that an exception can become its own
context if it is explicitly re-raised in its except handler (with "raise
variable", not bare "raise").
Not a critical bug, but it should be fixed in order to avoid delayed
garbage collection of such exceptions.

(as for the exception reporting code, it will have to detect recursivity)

>>> try: 1/0
... except Exception as e: raise e
... 
Traceback (most recent call last):
  File "", line 2, in 
  File "", line 1, in 
ZeroDivisionError: int division or modulo by zero
>>> e = sys.last_value
>>> e.__context__
ZeroDivisionError('int division or modulo by zero',)
>>> e.__context__ is e
True

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3114] bus error on lib2to3

[Benjamin Peterson]

New submission from Benjamin Peterson <[EMAIL PROTECTED]>:

I was just merging 2to3 work (and you'll have to do svnmerge.py merge in
Lib/lib2to3 for this to work) into py3k and was running the tests when
this rather alarming thing happended:

$ ./python.exe Lib/test/regrtest.py test_lib2to3
...
test test_lib2to3 crashed -- : local variable
'result' referenced before assignment
Traceback (most recent call last):
  File "Lib/test/regrtest.py", line 601, in runtest_inner
Bus error

Thinking this had something to do with r64281, I reverted it and got:
test test_lib2to3 crashed -- : local variable
'result' referenced before assignment
Fatal Python error: GC object already tracked
Abort trap

This probably isn't a lib2to3 problem, but that's how I got it.

--
components: Interpreter Core
messages: 68218
nosy: benjamin.peterson
priority: release blocker
severity: normal
status: open
title: bus error on lib2to3
type: crash
versions: Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3109] test_multiprocessing seems fragile

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Apparently the error can happen in test_pickling:

test_pickling (__main__.WithProcessesTestPicklingConnections) ...
Process Process-24:
Traceback (most recent call last):
  File "/home/antoine/py3k/traceback/Lib/multiprocessing/process.py",
line 237, in _bootstrap
self.run()
  File "/home/antoine/py3k/traceback/Lib/multiprocessing/process.py",
line 93, in run
self._target(*self._args, **self._kwargs)
  File "Lib/test/test_multiprocessing.py", line 1390, in _remote
client.connect(address)
socket.error: [Errno 111] Connection refused

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3109] test_multiprocessing seems fragile

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Oh, in another regrtest run, test_multiprocessing just hanged without
eating any CPU, this is what it printed after I pressed Ctrl-C:

test_multiprocessing
Process PoolWorker-5:4:
Traceback (most recent call last):
  File "/home/antoine/py3k/traceback/Lib/multiprocessing/process.py",
line 237, in _bootstrap
Process PoolWorker-5:3:
Traceback (most recent call last):
  File "/home/antoine/py3k/traceback/Lib/multiprocessing/process.py",
line 237, in _bootstrap
Process PoolWorker-5:2:
Traceback (most recent call last):
  File "/home/antoine/py3k/traceback/Lib/multiprocessing/process.py",
line 237, in _bootstrap
Process PoolWorker-5:1:
Traceback (most recent call last):
  File "/home/antoine/py3k/traceback/Lib/multiprocessing/process.py",
line 237, in _bootstrap

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3114] bus error on lib2to3

[Benjamin Peterson]

Benjamin Peterson <[EMAIL PROTECTED]> added the comment:

I have determined that this was caused by r64121.

--
nosy: +pitrou

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3114] bus error on lib2to3

[Benjamin Peterson]

Benjamin Peterson <[EMAIL PROTECTED]> added the comment:

I should mention the Bus error happens at test_future_builtins
(lib2to3.tests.test_fixers.Test_filter).

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3115] os.listdir randomly fails on occasions when it shouldn't

[philipspencer]

New submission from philipspencer <[EMAIL PROTECTED]>:

Python's os.listdir function has the following code in
Modules/posixmodule.c:

   errno = 0
   ...
for (;;) {
 Py_BEGIN_ALLOW_THREADS
 ep = readdir(dirp);
 Py_END_ALLOW_THREADS
 if (ep == NULL)
  break;
 ...
 a bunch of other stuff, including PyString_FromStringAndSize
 which calls malloc
 ...
}
if (errno != 0 && d != NULL) {

The assumption is that errno will be nonzero only if readdir failed.
However, this is not the case. GLibc's malloc will, in some rare cases,
set errno even when it succeeds. So, during one pass through the loop
errno gets set to ENOMEM and then it is still set to ENOMEM when the
final readdir returns null at the end of the directory listing.

The fix is to move the line "errno = 0" from outside the loop to
right before the readdir inside the loop. That is the only way to
ensure that, when the loop is exited with readdir returning null,
the condition "errno != 0" is equivalent to "readdir actually failed."

The attached patch does this. Without this patch, we experience
frequent failures with the python tools "creatrepo" and "repomanage"
when run on very large directories (> 5000 packages) on Fedora 8
systems; see RedHat bugzilla entry #451494. With the patch, the commands
work as expected.

The patch is against 2.5.1 but from what I can see of the posixmodule.c
code from SVN it should apply cleanly there too.

--
components: Extension Modules
messages: 68223
nosy: philipspencer
severity: normal
status: open
title: os.listdir randomly fails on occasions when it shouldn't
type: behavior
versions: Python 2.5

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3114] bus error on lib2to3

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

The following patch fixes it, which is somehow a bit scary...

--
keywords: +patch
Added file: http://bugs.python.org/file10629/3114.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3114] bus error on lib2to3

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

And a test.

Added file: http://bugs.python.org/file10630/test_3114.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3114] bus error on lib2to3

[Benjamin Peterson]

Benjamin Peterson <[EMAIL PROTECTED]> added the comment:

Thanks very much. Committed in r64384.

--
resolution:  -> fixed
status: open -> closed

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3114] bus error on lib2to3

[Adam Olsen]

Adam Olsen <[EMAIL PROTECTED]> added the comment:

I'm not sure that fix is 100% right - it fixes safety, but not
correctness.  Wouldn't it be more correct to move all 3 into
temporaries, assign from tstate, then XDECREF the temporaries? 
Otherwise you're going to expose just the value or traceback, without a
type set.

--
nosy: +Rhamphoryncus

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue2475] Popen.poll always returns None

[Mike Lisanke]

Mike Lisanke <[EMAIL PROTECTED]> added the comment:

Isn't this a critical problem. The .poll() function serves as a means to
check the status of the process started. When it continues to report
'None' to a process which has already terminated, it creates a false
positive of a hung process. Dealing with recovery from an actual hung
process is difficult enough. Having to deal with a bad detection that
the process ran to completion on top of this, makes the use of
subprocess difficult. 

Maybe I'm miss applying the .poll() function. I'm trying to detect that
a process has hung, prior to calling .stdout.readlines(). The
readlines() will hang my python script if the process is hung. Is there
another way I should be doing this?

Thanks, 

Mike

--
nosy: +lisanke

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue3114] bus error on lib2to3

[Guido van Rossum]

Guido van Rossum <[EMAIL PROTECTED]> added the comment:

N.B. Benjamin made a typo in the revision, it's r64284.

--
nosy: +gvanrossum
resolution: fixed -> 
status: closed -> open

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com