[Python-Dev] Get rid of strerror.c and memmove.c?
I ran LLVM's under-development C front-end, clang (http://clang.llvm.org), against Python today. While a ton of message crop up thanks to lacking header files, I did come across a handful of semi-useful warnings (e.g., the change I checked into Python/compile.c). One thing it picked up is that on OS X Python/strerror.c disagrees with the declaration of sys_nerr and such. But then I thought I remembered strerror() being in C89. I checked configure-in to make sure that it wasn't special for some reason, it it isn't; only used when the function is not normally available. Same goes for memmove(). Since both strerror() and memmove() are both in C89 (they are listed in K&R), can we ditch these files? -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Get rid of strerror.c and memmove.c?
-On [20080207 09:05], Brett Cannon ([EMAIL PROTECTED]) wrote: >Since both strerror() and memmove() are both in C89 (they are listed >in K&R), can we ditch these files? You got my vote. I am assuming the minimum needed compiler is C90 (ISO over ANSI, sorry :P) and you might even need the AM1 from 1994 for multibyte support anyway (so effectively C94). -- Jeroen Ruigrok van der Werven / asmodai イェルーン ラウフロック ヴァン デル ウェルヴェン http://www.in-nomine.org/ | http://www.rangaku.org/ We have met the enemy and they are ours... ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] int/float freelists vs pymalloc
I note that in r60567 Christian checked in support for compacting the
int and float freelists.
I have no problems with the implementation, but would like to note
that I have been experimenting with an alternate approach which doesn't
require the addition of a knob to initiate the compacting.
Probably in response to the same stimulus as Christian it occurred to me
that the freelist approach had been adopted long before PyMalloc was
enabled as standard (in 2.3), and that much of the performance gains
between 2.2 and 2.3 were in fact due to PyMalloc.
So I've been testing with the freelists ripped out and ints and floats
reverted to fairly standard PyObject_New allocation (retaining the small
int interning) and thus relying on PyMalloc to do any compaction.
The results have been somewhat surprising...
The short version is that:
- for ints, the freelist is ahead of PyMalloc by a very small margin
(<4%)
- for floats, the freelist is a long way behind PyMalloc (>35% slower)
This without invoking freelist compaction by the way (though PyMalloc
is releasing empty arenas).
I don't know what's pessimising the float freelist, but the results are
similar on 2 boxes:
- AMD Athlon XP-1600+, 512MB RAM, FreeBSD 6.1, gcc 3.4.4
(~27k pystones)
- AMD Athlon XP-2500+, 512MB RAM, OS/2 v4 with EMX, gcc 2.8.1
(~38k pystones)
If there's interest in following this up, I can put my patches to
intobject.c and floatobject.c into the tracker.
Test scripts:
a) int
# test routine - integer
import time
def b(time_now=time.clock):
limit_val = 200
start_time = time_now()
for j in xrange(25):
d = {}
for i in xrange(limit_val):
d[i] = i + limit_val
return time_now() - start_time
if __name__ == '__main__':
print 'elapsed: %s s' % b()
b) float
# test routine - float
import time
def b(time_now=time.clock):
limit_val = 100
vals = range(limit_val)
offset = limit_val * 1.0
start_time = time_now()
for j in xrange(25):
d = {}
for i in [float(x) for x in vals]:
d[i] = i + offset
return time_now() - start_time
if __name__ == '__main__':
print 'elapsed: %s s' % b()
The times I've obtained were:
case XP1600/Fbsd XP2500/OS2 (*)
1) int freelist 38.1s 24.6s
2) int pymalloc 39.0s 25.3s
3) float freelist
(with int freelist) 75s 46.1s
4) float pymalloc
-with int freelist55s 29.0s
-with int pymalloc55.5s 29.5s
(*) OS/2 tests based on patched 2.5.1 sources rather than svn trunk.
FreeBSD tests based on svn trunk checked out at 1050UTC 7Feb08.
--
-
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: [EMAIL PROTECTED] (pref) | Snail: PO Box 370
[EMAIL PROTECTED] (alt) |Belconnen ACT 2616
Web:http://www.andymac.org/ |Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Can't decode \x876F character encoded in Shift JIS charset ?
>>> unicode('\x87\x6F', "shift jis")
Traceback (most recent call last):
File "", line 1, in
UnicodeDecodeError: 'shift_jis' codec can't decode bytes in position 0-1:
illegal multibyte sequence
Still, \x87\x6F is a valid Shift-JIS character :
http://demo.icu-project.org/icu-bin/convexp?conv=ibm-943_P15A-2003&b=87&s=MIME#layout,
it is "㎜"...
Thanks,
--
Nicolas Dumazet — NicDumZ
Deuxième année ENSIMAG.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
Andrew MacIntyre wrote: > So I've been testing with the freelists ripped out and ints and floats > reverted to fairly standard PyObject_New allocation (retaining the small > int interning) and thus relying on PyMalloc to do any compaction. Nice! What do you mean with int interning? Are you talking about the small int cache? > The results have been somewhat surprising... > > The short version is that: > - for ints, the freelist is ahead of PyMalloc by a very small margin >(<4%) > - for floats, the freelist is a long way behind PyMalloc (>35% slower) > > This without invoking freelist compaction by the way (though PyMalloc > is releasing empty arenas). > > I don't know what's pessimising the float freelist, but the results are > similar on 2 boxes: > - AMD Athlon XP-1600+, 512MB RAM, FreeBSD 6.1, gcc 3.4.4 >(~27k pystones) > - AMD Athlon XP-2500+, 512MB RAM, OS/2 v4 with EMX, gcc 2.8.1 >(~38k pystones) That's interesting. I wouldn't have thought that floats are much faster with pymalloc. I'd bet that pymalloc is a tiny bit slower until the new free list is filled. > If there's interest in following this up, I can put my patches to > intobject.c and floatobject.c into the tracker. I've uploaded my patch at http://bugs.python.org/issue2039. It keeps a standard free list with 8192 elements for floats and ints. Does your patch also use a free list or does it alloc/free directly? Can your profile my patch and compare it to your results? It may be a bit faster if your patch doesn't use a free list. Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot failures
On Thu, 7 Feb 2008 09:08:26 -0500, "A.M. Kuchling" <[EMAIL PROTECTED]> wrote: >On Wed, Feb 06, 2008 at 08:34:21PM -0500, Raymond Hettinger wrote: >> Also, test_docxmlrpc hasn't been happy. One of the tests isn't >> getting the exact response string it expected. Any ideas what is >> causing this? > >My fault; it should be fixed now. > >> There is also a recurring failure in SocketServer.py returning >> "ValueError: list.remove(x): x not in list" during attempts to >> remove a PID from the list of active_children. Any ideas about what >> is causing this? > >I couldn't find a current build that was showing this error, but >searching python.org turned up one that had been indexed: > >http://www.python.org/dev/buildbot/trunk/ppc%20Debian%20unstable%20trunk/builds/726/step-test/0 > >I don't see what could be causing this failure, though; the test isn't >starting any subprocesses outside of what the ForkingServer class >does. I don't see how this could be an artifact of the buildbot >environment, either. > >It would be easy to add an 'if pid in self.active_children' to the >code, but I don't want to do that without understanding the problem. You could instrument fork() so that it logs the call stack and the child PID and instrument ForkingServer so that it reports which PID it is about to try to remove from active_children. Perhaps this will point to the problem. Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Buildbot failures
On Wed, Feb 06, 2008 at 08:34:21PM -0500, Raymond Hettinger wrote: > Also, test_docxmlrpc hasn't been happy. One of the tests isn't > getting the exact response string it expected. Any ideas what is > causing this? My fault; it should be fixed now. > There is also a recurring failure in SocketServer.py returning > "ValueError: list.remove(x): x not in list" during attempts to > remove a PID from the list of active_children. Any ideas about what > is causing this? I couldn't find a current build that was showing this error, but searching python.org turned up one that had been indexed: http://www.python.org/dev/buildbot/trunk/ppc%20Debian%20unstable%20trunk/builds/726/step-test/0 I don't see what could be causing this failure, though; the test isn't starting any subprocesses outside of what the ForkingServer class does. I don't see how this could be an artifact of the buildbot environment, either. It would be easy to add an 'if pid in self.active_children' to the code, but I don't want to do that without understanding the problem. --amk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can't decode \x876F character encoded in Shift JIS charset ?
> UnicodeDecodeError: 'shift_jis' codec can't decode bytes in position > 0-1: illegal multibyte sequence As Amaury says, this is CP932 and Shift-Jis-2004, but not Shift-Jis. The IBM database must be incorrect: it lists as "all aliases" both csShiftJIS and csWindows31J, yet, according to http://www.iana.org/assignments/character-sets they are different (MIBEnum 17 vs MIBEnum 2024). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can't decode \x876F character encoded in Shift JIS charset ?
Hello,
Nicolas Dumazet :
> >>> unicode('\x87\x6F', "shift jis")
> Traceback (most recent call last):
> File "", line 1, in
> UnicodeDecodeError: 'shift_jis' codec can't decode bytes in position 0-1:
> illegal multibyte sequence
>
> Still, \x87\x6F is a valid Shift-JIS character :
> http://demo.icu-project.org/icu-bin/convexp?conv=ibm-943_P15A-2003&b=87&s=MIME#layout,
> it is "�L"...
It is possible that the encoding is actually "shift jis 2004" or
"cp932", which are both extensions to the original shift jis.
Please continue this discussion on comp.lang.python; or fill a bug request.
Cheers quand même,
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
On 2008-02-07 14:09, Andrew MacIntyre wrote: > Probably in response to the same stimulus as Christian it occurred to me > that the freelist approach had been adopted long before PyMalloc was > enabled as standard (in 2.3), and that much of the performance gains > between 2.2 and 2.3 were in fact due to PyMalloc. One of the hopes of having a custom allocator for Python was to be able to get rid off all free lists. For some reason that never happened. Not sure why. People were probably too busy with adding new features to the language at the time ;-) Something you could try to make PyMalloc perform better for the builtin types is to check the actual size of the allocated PyObjects and then make sure that PyMalloc uses arenas large enough to hold a good quantity of them, e.g. it's possible that the float types fall into the same arena as some other type and thus don't have enough "room" to use as free list. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 08 2008) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
On Feb 7, 2008 5:09 AM, Andrew MacIntyre <[EMAIL PROTECTED]> wrote: > > So I've been testing with the freelists ripped out and ints and floats > reverted to fairly standard PyObject_New allocation (retaining the small > int interning) and thus relying on PyMalloc to do any compaction. > > The results have been somewhat surprising... > > The short version is that: > - for ints, the freelist is ahead of PyMalloc by a very small margin >(<4%) > - for floats, the freelist is a long way behind PyMalloc (>35% slower) Martin did some profiling of ints in py3k 1.5 years ago. The results of his profiling are here: http://svn.python.org/projects/python/branches/py3k/INTBENCH I think Martin wrote a mail to describe his work in more detail. But the only mails I could find are not what I remember: http://mail.python.org/pipermail/python-3000/2006-August/003185.html http://mail.python.org/pipermail/python-3000/2006-August/003064.html I don't remember if he did any work on head or if he remembers any more that might be relevant here. n ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] int/float freelists vs pymalloc
> One of the hopes of having a custom allocator for Python was to be > able to get rid off all free lists. For some reason that never happened. > Not sure why. People were probably too busy with adding new > features to the language at the time ;-) Probably not. It's more that the free lists still outperformed pymalloc. > Something you could try to make PyMalloc perform better for the builtin > types is to check the actual size of the allocated PyObjects and then > make sure that PyMalloc uses arenas large enough to hold a good quantity > of them, e.g. it's possible that the float types fall into the same > arena as some other type and thus don't have enough "room" to use > as free list. I don't think any improvements can be gained here. PyMalloc carves out pools of 4096 bytes from an arena when it runs out of blocks for a certain size class, and then keeps a linked list of pools of the same size class. So when many float objects get allocated, you'll have a lot of pools of the float type's size class. IOW, PyMalloc has always enough room. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
