[issue27725] Use Py_SIZE(x) instead of x->ob_size
New submission from REIX Tony:
I'm porting Python 2.7.* on AIX.
I am now using a version 13.1.3.2 of XLC.
I was previously using version 12.1.0.14 .
It appears that this new compiler shows a potential issue in Python v2 code.
This issue seems to appear when -O2 is used.
I was porting Python 2.7.11 with -O2 instead of -O0 when I got it.
The issue appears when running tests involving power() with Decimals where the
x_sub() routine is used.
The following Python code shows the issue:
from decimal import *
c = Context(7, ROUND_UP)
d = c.power(Decimal(0.7), Decimal(3.4))
print d
Good result is: 0.2973948 .
The bug returns: 2.440099
However, I have about 22 errors when running:
./python Lib/test/regrtest.py -v test_decimal
It is random in 64bit and quite constant in 32bit.
The issue deals with using x->ob-size.
There is a macro:
Include/object.h:#define Py_SIZE(ob) (((PyVarObject*)(ob))->ob_size)
The issue appears in last lines of Objects/longobject.c:x_sub() routine, when
changing the sign: z->ob_size = -(z->ob_size);
I have looked at version 2.7.12 and 2.7.10 of Python: they contain the same
code as in 2.7.11.
And ->ob_size is used about 45 times, mainly in Objects/longobject.c. However,
Py_SIZE(...) is used ten times more (475).
- Looking at version 3.5.1 of Python, I see that ->ob_size is now used only
once, in: Objects/object.c .
And Py_SIZE(...) is now used 616 times.
And I also see that the Python 2.7.11-12 x_sub() code:
if (sign < 0)
z->ob_size = -(z->ob_size);
return long_normalize(z);
has been replaced in Python 3.5.1 by:
if (sign < 0) {
_PyLong_Negate(&z);
if (z == NULL)
return NULL;
}
return long_normalize(z);
/* If a freshly-allocated int is already shared, it must
be a small integer, so negating it must go to PyLong_FromLong */
Py_LOCAL_INLINE(void)
_PyLong_Negate(PyLongObject **x_p)
{
PyLongObject *x;
x = (PyLongObject *)*x_p;
if (Py_REFCNT(x) == 1) {
Py_SIZE(x) = -Py_SIZE(x);
return;
}
*x_p = (PyLongObject *)PyLong_FromLong(-MEDIUM_VALUE(x));
Py_DECREF(x);
}
So, it looks to me that Python v2 version should do the same that Python v3 has
done: replace x->ob-size by Py_SIZE(x) everywhere, and improve the code of
x_sub() by using some _PyLong_Negate() routine or macro.
--
components: Library (Lib)
messages: 272324
nosy: trex58
priority: normal
severity: normal
status: open
title: Use Py_SIZE(x) instead of x->ob_size
type: behavior
versions: Python 2.7
___
Python tracker
<http://bugs.python.org/issue27725>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27725] Use Py_SIZE(x) instead of x->ob_size
REIX Tony added the comment: Hi Raymond I've got several email exchanges with the IBM XLC expert. From his own study of my issue, his conclusion is that this kind of Python v2 coding is not ANSI-aliasing safe. It seems that there is a standard that requires C code to NOT do some kinds of coding so that any C compiler optimizer can do its best. The issue was not there with XLC v12.1.0.14 and -O2. It appeared with XLC v13.1.3.2 and -O2 since XLC v13 optimizer is more agressive. About GCC, I've not experimented yet with it for now (will do later today I hope), but the impact should be the same according to the optimizer level and improvements. Here is what IBMer Steven said: "I found the problem. It is not a problem with the compiler, but a problem with the source code/option set. It is an ansi aliasing violation. I'll try to provide as much detail as I can to explain it. At line 2512 of Objects/longobject.c, we have the following code: if (sign < 0) z->ob_size = -(z->ob_size); return long_normalize(z); Note that we use z->ob_size to access size, and the type of z is "PyLongObject *". This value is loaded in long_normalization. After we inline this function call, the compiler moves the load done in long_normalization above the if statement (past the store that writes to it), which is why we ends up with the wrong sign. Now the question is why does the compiler think that this is legal ? In long_normalize, the size is obtained using a macro Py_SIZE(v) (line 47). This macro expands to: (((PyVarObject*)(v))->ob_size) Notice that the pointer is cast to something of type PyVarObject*. PyVarObject and PyLongObject are not compatible types, and, because ansi aliasing is assumed, the compiler believes they do not reference the same memory. Therefore it is safe to move. A simple solution is to use "-qalias=noansi" when compiling. That will work, but could also hurt performance. The other solution is to use either Py_SIZE all of the time to access the memory or never. Do not mix and match. This will require some code changes. I'll leave it to you to figure out how to handle it, but my guess is that Py_SIZE is supposed to always be used. The comments in "object.h" lines 11-17 include this phrase "they must be accessed through special macros and functions only." -- ___ Python tracker <http://bugs.python.org/issue27725> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27725] Use Py_SIZE(x) instead of x->ob_size
REIX Tony added the comment: Thanks a lot Stefan, that should completely explain my issues. -fno-strict-aliasing -fwrapv for gcc So, that means that you would get better performance if you applied on Python v2.7 what Python v3.5 did about Py_SIZE(x) . However, there are probably other places where the aliasing issue still appears in v2.7 . Hummm I'll use -qalias=noansi with XLC and see what happens. -- ___ Python tracker <http://bugs.python.org/issue27725> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27725] Use Py_SIZE(x) instead of x->ob_size
REIX Tony added the comment: With XLC v13 -O2, using -qalias=noansi for building Objects/longobject.o only and not for all the other .o files did fix the 10 more failed tests I see with -O2 compared to -O0 (7-8 failed tests). So, ANSI-aliasing in Objects/longobject.c is the issue. About -fwrapv , I have to find an equivalent for XLC. I've given a first try with GCC 4.8.4 . I've got about 44 failed tests compared to 7-8 with XLC. To be improved. -- ___ Python tracker <http://bugs.python.org/issue27725> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27725] Use Py_SIZE(x) instead of x->ob_size
REIX Tony added the comment: OK. However, compiling ONLY the file Objects/longobject.c with -qalias=noansi did fix the issue on AIX. That could be the same on Linux. I haven't tried to use Py_SIZE() in all places where it should be used. Now trying to figure out why GCC behaves worst than XLC. Anyway, I have a satisfactory work-around. Thanks for your help ! -- ___ Python tracker <http://bugs.python.org/issue27725> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24741] Hangs and errors while testing on Ubuntu/Intel
New submission from REIX Tony: I'm now porting Python (2.7.10 and 3.4.3) on AIX. Before that, I wanted to build and test on reference machines in order to see what is expected during building and testing. However, on Ubuntu/i586 and Ubuntu/x86_64, I face hangs and errors during test phase on the 2 environments. * Ubuntu / i586 : - v. 2.7.10 : - [152/401] test_gdb Blocked a full night in: (See details in file P1 : 1) - v. 3.4.3 : - [292/390/1] test_tk : blocked during hours - [111/390] test_capi : blocked during more than 1/2h * Ubuntu / x86_64 : - 3.4.3 - [137/390] test_ftplib : full night See details in P1: 2) - Errors: - 3 tests failed: test_ssl test_urllib2_localnet test_urllib2net - test_ssl : test_connect_ex (test.test_ssl.NetworkedTests) ... FAIL - test_urllib2_localnet : test_basic_auth_success (test.test_urllib2_localnet.BasicAuthTests) ... ERROR - [141/390/2] test_venv test test_venv failed -- Traceback (most recent call last): File "/home/reixt/PYTHON-3/Python-3.4.3/Lib/test/test_venv.py", line 390, in test_with_pip self.assertEqual(err, "") AssertionError: "/tmp/tmpu049e_1j/lib/python3.4/site-pack[181 chars])>\n" != '' - /tmp/tmpu049e_1j/lib/python3.4/site-packages/pip/__init__.py:217: ResourceWarning: unclosed I'm new with building Python. I've followed instructions from: https://docs.python.org/devguide/ Download: Python-3.4.3.tar ./configure --with-pydebug && make -j2 ./python -m test -j3 I've found: http://buildbot.python.org/all/waterfall?category=3.4.stable and it looks like there are NO known issue on Ubuntu/x86_64 with '3.4'. However, I'm using source code of 3.4.3 . Difference between 3.4 and 3.4.3 ? Are there more instructions I must follow in order to improve my test environment on Ubuntu/x86_64 ? -- components: Tests files: Py1 messages: 247506 nosy: trex58 priority: normal severity: normal status: open title: Hangs and errors while testing on Ubuntu/Intel type: behavior versions: Python 3.4 Added file: http://bugs.python.org/file40042/Py1 ___ Python tracker <http://bugs.python.org/issue24741> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24741] Hangs and errors while testing on Ubuntu/Intel
REIX Tony added the comment: run on Ubuntu/i586 : $ make test > Go.res3 2>&1 & $ tail -f Go.res [164/390] test_cgi Silent ! ^C [1]+ Stoppedmake test > Go.res3 2>&1 reixt@b017569-ux:~/FromOldPC/FromOldPC/Projets/ToolBox/PYTHON-3/Python-3.4.3$ reixt@b017569-ux:~/FromOldPC/FromOldPC/Projets/ToolBox/PYTHON-3/Python-3.4.3$ reixt@b017569-ux:~/FromOldPC/FromOldPC/Projets/ToolBox/PYTHON-3/Python-3.4.3$ ps PID TTY TIME CMD 1239 pts/42 00:00:05 python 1243 pts/42 00:00:00 python 1390 pts/42 00:00:02 python 1443 pts/42 00:00:00 python 1452 pts/42 00:00:00 gdb 1454 pts/42 00:00:00 bash 4837 pts/42 00:00:00 ps 29566 pts/42 00:00:00 bash 29645 pts/42 00:00:00 make 29654 pts/42 00:00:00 python 29818 pts/42 00:00:00 python 29981 pts/42 00:00:00 python Looks like the main "make" stops. -- ___ Python tracker <http://bugs.python.org/issue24741> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24741] Hangs and errors while testing on Ubuntu/Intel
REIX Tony added the comment: Thnaks for the information. I'm surprised that: $ make test > Go.res3 2>&1 & makes issues that do not appear with: $ nohup make test > Go.res3 2>&1 & Anyway, we know the root cause now, and there is a work-around. So, closing the defect is OK for me. About the comment "Python has been used on AIX by other", I'm sure that Python2 has been ported on AIX. About Python3, I have found no one for AIX. I'll have another look. -- ___ Python tracker <http://bugs.python.org/issue24741> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24741] Hangs and errors while testing on Ubuntu/Intel
REIX Tony added the comment: Hum I'm now building Python 2.7.10 RPMs on AIX. The .spec file works fine. However, when adding make test in the .spec file, I've got the tests blocked in test_io. The same with 2.7.6 . Going into the directory and trying "make quicktest", it is now blocked after test_signal : # ps -edf | grep python root 154010881 0 11:54:03 pts/1 4:49 ./python -Wd -3 -E -tt ./Lib/test/regrtest.py -l Retrying with 2.7.6, it is blocked again with test_io. OK. Either it is a general issue with Python tests, or there is an issue on AIX. Investigating. -- ___ Python tracker <http://bugs.python.org/issue24741> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24741] Hangs and errors while testing on Ubuntu/Intel
REIX Tony added the comment: I got strange things: Tests blocked in: [300/400/22] test_signal and: # ps -edf | grep python root 7405612 12320954 0 14:52:04 pts/2 0:00 ./python -m -c -l -x test_io root 7929900 34209932 0 14:52:04 pts/2 0:00 ./python -m -c -l -x test_io root 12320954 34209932 0 14:52:04 pts/2 0:00 ./python -m -c -l -x test_io root 16384082 34209932 0 14:52:04 pts/2 0:00 ./python -m -c -l -x test_io root 34209932 4325570 0 14:46:14 pts/2 3:28 ./python -m -c -l -x test_io root 43712512 34209932 0 14:52:04 pts/2 0:00 ./python -m -c -l -x test_io root 53477604 12320954 0 14:52:04 pts/2 0:00 ./python -m -c -l -x test_io root 56819916 12320954 0 14:52:04 pts/2 0:00 ./python -m -c -l -x test_io root 57409628 12320954 0 14:52:04 pts/2 0:00 ./python -m -c -l -x test_io root 58458148 34209932 0 14:52:04 pts/2 0:00 ./python -m -c -l -x test_io -- ___ Python tracker <http://bugs.python.org/issue24741> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24741] Hangs and errors while testing on Ubuntu/Intel
REIX Tony added the comment: About test_io, when running it alone by: ./python ./Lib/test/test_io.py > /tmp/test_io.res I got NO error ! Done 2 times on each of my 2 AIX machines. Hu Why these tests do block in the middle ? -- ___ Python tracker <http://bugs.python.org/issue24741> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24741] Hangs and errors while testing on Ubuntu/Intel
REIX Tony added the comment: Yes. I'm on my own for AIX debugging. I'm afraid too. ;) OK. What I need is build more skills about how to get into deep details when things run badly. Lot of fun. ;) Thx -- ___ Python tracker <http://bugs.python.org/issue24741> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24046] Incomplete build on AIX
REIX Tony added the comment: Fresh openssl versions are now available for AIX: 1.0.1p and 1.0.2d . See: http://www.bullfreeware.com/search.php?package=openssl -- nosy: +trex58 ___ Python tracker <http://bugs.python.org/issue24046> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20600] test_create_server_ssl_verify_failed() failure on "PPC64 AIX 3.x" buildbot
REIX Tony added the comment: I'm now working about the port of Python 3.4.3 on AIX. So, I'm interesting with issues dealing with AIX. I'm now building a view of the issues on AIX 6.1 . -- nosy: +trex58 ___ Python tracker <http://bugs.python.org/issue20600> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
