[issue9532] pipe.read hang, when calling commands.getstatusoutput in multi-threading code of python 2.4
New submission from denny : Hi all My environment is python 2.4.4 on linux. I am encountering hang of pipe.read() in commands.getstatusoutput for multi-threading code. I have spawned several threads which will call commands.getstatusoutput to run cli. However, pipe.read may hang sometimes. >From lsof, we know some pipe handles are not closed in the parent process, >after the child process is stopped. ;; -- Reproduce steps -- Below are reproduce steps. # Create a python script of /tmp/hang.py, whose content is given below. # Run "service crond stop; python /tmp/hang.py" several times. # The script may probably hang. From lsof of main.py and crond service, we may find one pipe existing in both processes. # If we stop crond to close the pipe, the hang of hang.py will be resolved. ;; -- Code of hang.py -- #!/usr/bin/python import commands import datetime import time import os import thread import threading def thread_run1(): cmd = "hostname" print "begin to run command:%s" % (cmd) status, output = commands.getstatusoutput(cmd) print "pid:%d, name:%s, status:%d, output:%s" % \ (os.getpid(), threading.currentThread().getName(), status, output) cmd = "ifconfig eth0" print "begin to run command:%s" % (cmd) status, output = commands.getstatusoutput(cmd) print "pid:%d, name:%s, status:%d, output:%s" % \ (os.getpid(), threading.currentThread().getName(), status, output) cmd = "ifconfig eth1" print "begin to run command:%s" % (cmd) status, output = commands.getstatusoutput(cmd) print "pid:%d, name:%s, status:%d, output:%s" % \ (os.getpid(), threading.currentThread().getName(), status, output) # cmd = "sh /tmp/subprocess.sh" cmd = "echo here1; sleep 2; echo here2; sleep 5" print "begin to run command:%s" % (cmd) status, output = commands.getstatusoutput(cmd) print "pid:%d, name:%s, status:%d, output:%s" % \ (os.getpid(), threading.currentThread().getName(), status, output) def thread_run2(): cmd = "service crond start" print "begin to run command:%s" % (cmd) status, output = commands.getstatusoutput(cmd) print "pid:%d, name:%s, status:%d, output:%s" % \ (os.getpid(), threading.currentThread().getName(), status, output) if __name__=='__main__': print "main function begins." thread_list = [] for i in xrange(1, 10): my_thread = threading.Thread(target = thread_run1) thread_list.append(my_thread) my_thread = threading.Thread(target = thread_run2) thread_list.append(my_thread) for t in thread_list: t.start() time.sleep(10) for t in thread_list: t.join() print "main function ends." -- components: Library (Lib) messages: 113086 nosy: denny priority: normal severity: normal status: open title: pipe.read hang, when calling commands.getstatusoutput in multi-threading code of python 2.4 type: behavior versions: 3rd party ___ Python tracker <http://bugs.python.org/issue9532> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue9532] pipe.read hang, when calling commands.getstatusoutput in multi-threading code of python 2.4
denny added the comment:
Hi David
I have tried in another testbd with python 2.6.5, and the problem of hang
doesn't reproduce, after retrying for several times.
The original hang happens in pipe.read of commands module.
After comparing the code of python 2.4.4 and python 2.6.5, I noticed two
enhancements in posimodule.c:posix_read.
These defensive coding add precheck, before invoking read function.
David, without these enhancements, would it cause the potential hang problem,
in your opinion?
Recompiling python 2.4.4 with some manual changes is a little scaring to me.
,--- python 2.4.4 posixmodule.c
| static PyObject *
| posix_read(PyObject *self, PyObject *args)
| {
| int fd, size, n;
| PyObject *buffer;
| if (!PyArg_ParseTuple(args, "ii:read", &fd, &size))
| return NULL;
| buffer = PyString_FromStringAndSize((char *)NULL, size);
| if (buffer == NULL)
| return NULL;
| Py_BEGIN_ALLOW_THREADS
| n = read(fd, PyString_AsString(buffer), size);
| Py_END_ALLOW_THREADS
| if (n < 0) {
| Py_DECREF(buffer);
| return posix_error();
| }
| if (n != size)
| _PyString_Resize(&buffer, n);
| return buffer;
| }
`---
,--- 2.6.5 posixmodule.c
| static PyObject *
| posix_read(PyObject *self, PyObject *args)
| {
| int fd, size, n;
| PyObject *buffer;
| if (!PyArg_ParseTuple(args, "ii:read", &fd, &size))
| return NULL;
| if (size < 0) {
| errno = EINVAL;
| return posix_error();
| }
| buffer = PyString_FromStringAndSize((char *)NULL, size);
| if (buffer == NULL)
| return NULL;
| if (!_PyVerify_fd(fd))
| return posix_error();
| Py_BEGIN_ALLOW_THREADS
| n = read(fd, PyString_AsString(buffer), size);
| Py_END_ALLOW_THREADS
| if (n < 0) {
| Py_DECREF(buffer);
| return posix_error();
| }
| if (n != size)
| _PyString_Resize(&buffer, n);
| return buffer;
| }
`---
--
___
Python tracker
<http://bugs.python.org/issue9532>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize
Denny Weinberg added the comment: Any news here? 3.6.0 is also affected by this bug. -- ___ Python tracker <http://bugs.python.org/issue27400> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17408] second python execution fails when embedding
Denny Weinberg added the comment: Can we please reopen this issue? -- ___ Python tracker <http://bugs.python.org/issue17408> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17408] second python execution fails when embedding
Denny Weinberg added the comment: Ok, thank you very much for your comments. See Issue27400 -- ___ Python tracker <http://bugs.python.org/issue17408> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize
New submission from Denny Weinberg:
After calling Py_Finalize and Py_Initialize I get the message "attribute of
type 'NoneType' is not callable" on the datetime.strptime method.
Example:
from datetime import datetime
s = '20160505 16'
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')
The first call works fine but it crashes after the re initialization.
Workaround:
from datetime import datetime
s = '20160505 16'
try:
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')
except TypeError:
import time
refdatim = datetime.fromtimestamp(time.mktime(time.strptime(s, '%Y%m%d
%H%M%S')))
Related Issue: Issue17408 ("second python execution fails when embedding")
--
components: Interpreter Core
messages: 269379
nosy: Denny Weinberg, palm.kevin
priority: normal
severity: normal
status: open
title: Datetime NoneType after calling Py_Finalize and Py_Initialize
type: behavior
versions: Python 3.5
___
Python tracker
<http://bugs.python.org/issue27400>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27400] Datetime NoneType after calling Py_Finalize and Py_Initialize
Denny Weinberg added the comment: Just to be clear: The error happens after these steps: 1. Call strptime 2. Call cpython function "Py_Finalize" and "Py_Initialize" 3. Call strptime again Now we get the error "attribute of type 'NoneType' is not callable" -- ___ Python tracker <http://bugs.python.org/issue27400> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27516] Wrong initialization of python path with embeddable distribution
Changes by Denny Weinberg : -- nosy: +Denny Weinberg ___ Python tracker <http://bugs.python.org/issue27516> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27625] "make install" fails when no zlib support available
Changes by Denny Weinberg : -- nosy: +Denny Weinberg ___ Python tracker <http://bugs.python.org/issue27625> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue17408] second python execution fails when embedding
Denny Weinberg added the comment: Hi, I think that the problem exists also in python 3.5.1 After calling Py_Finalize and Py_Initialize I get the message "attribute of type 'NoneType' is not callable" on the datetime.strptime method. Example: from datetime import datetime s = '20160505 16' refdatim = datetime.strptime(s, '%Y%m%d %H%M%S') The first call works find but it crashes after the re initialization. Workaround: from datetime import datetime s = '20160505 16' try: refdatim = datetime.strptime(s, '%Y%m%d %H%M%S') except TypeError: import time refdatim = datetime.fromtimestamp(time.mktime(time.strptime(s, '%Y%m%d %H%M%S'))) Can anyone confirm this bug? Can you tell me if this will be fixed for python 3.5/3.6/...? All the other modules are working find after the re initialization but datetime.strptime. -- nosy: +Denny Weinberg versions: +Python 3.5 ___ Python tracker <http://bugs.python.org/issue17408> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
