[Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-21 Thread Artur Siekielski
Hi. The problem with reference counters is that they are very often incremented/decremented, even for read-only algorithms (like traversal of a list). It has two drawbacks: 1. CPU cache lines (64 bytes on X86) containing a beginning of a PyObject are very often invalidated, resulting in loosing man

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-23 Thread Artur Siekielski
Ok, I managed to make a quick but working patch (sufficient to get working interpreter, it segfaults for extension modules). It uses the "ememoa" allocator (http://code.google.com/p/ememoa/) which seems a reasonable pool allocator. The patch: http://dpaste.org/K8en/. The main obstacle was that ther

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-23 Thread Artur Siekielski
2011/5/23 Guido van Rossum : >> Anyway, I'd like to have working copy-on-write in CPython - in the >> presence of GIL I find it important to have multiprocess programs >> optimized (and I think it's a common idiom that a parent process >> prepares some big data structure, and child "worker" process

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Artur Siekielski
2011/5/24 Sturla Molden : >> Oh, and using explicit shared memory or mmap is much harder, because >> you have to map the whole object graph into bytes. > > It sounds like you need PYRO, POSH or multiprocessing's proxy objects. PYRO/multiprocessing proxies isn't a comparable solution because of ORD

Re: [Python-Dev] CPython optimization: storing reference counters outside of objects

2011-05-24 Thread Artur Siekielski
2011/5/24 Sturla Molden : > Den 24.05.2011 11:55, skrev Artur Siekielski: >> >> PYRO/multiprocessing proxies isn't a comparable solution because of >> ORDERS OF MAGNITUDE worser performance. You compare here direct memory >> access vs serialization/message pass