Re: [Python-Dev] [Python-checkins] cpython: Following issue #13390, fix compilation --without-pymalloc, and make
Looks like Windows buildbots broken by this commit.
On Tue, Dec 18, 2012 at 12:07 AM, antoine.pitrou
wrote:
> http://hg.python.org/cpython/rev/a85673b55177
> changeset: 80923:a85673b55177
> user:Antoine Pitrou
> date:Mon Dec 17 23:05:59 2012 +0100
> summary:
> Following issue #13390, fix compilation --without-pymalloc, and make
> sys.getallocatedblocks() return 0 in that situation.
>
> files:
> Doc/library/sys.rst | 15 ---
> Lib/test/test_sys.py | 7 ++-
> Objects/obmalloc.c | 7 +++
> 3 files changed, 21 insertions(+), 8 deletions(-)
>
>
> diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst
> --- a/Doc/library/sys.rst
> +++ b/Doc/library/sys.rst
> @@ -396,16 +396,17 @@
> .. function:: getallocatedblocks()
>
> Return the number of memory blocks currently allocated by the interpreter,
> - regardless of their size. This function is mainly useful for debugging
> - small memory leaks. Because of the interpreter's internal caches, the
> - result can vary from call to call; you may have to call
> - :func:`_clear_type_cache()` to get more predictable results.
> + regardless of their size. This function is mainly useful for tracking
> + and debugging memory leaks. Because of the interpreter's internal
> + caches, the result can vary from call to call; you may have to call
> + :func:`_clear_type_cache()` and :func:`gc.collect()` to get more
> + predictable results.
> +
> + If a Python build or implementation cannot reasonably compute this
> + information, :func:`getallocatedblocks()` is allowed to return 0 instead.
>
> .. versionadded:: 3.4
>
> - .. impl-detail::
> - Not all Python implementations may be able to return this information.
> -
>
> .. function:: getcheckinterval()
>
> diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
> --- a/Lib/test/test_sys.py
> +++ b/Lib/test/test_sys.py
> @@ -7,6 +7,7 @@
> import operator
> import codecs
> import gc
> +import sysconfig
>
> # count the number of test runs, used to create unique
> # strings to intern in test_intern()
> @@ -616,9 +617,13 @@
> "sys.getallocatedblocks unavailable on this build")
> def test_getallocatedblocks(self):
> # Some sanity checks
> +with_pymalloc = sysconfig.get_config_var('WITH_PYMALLOC')
> a = sys.getallocatedblocks()
> self.assertIs(type(a), int)
> -self.assertGreater(a, 0)
> +if with_pymalloc:
> +self.assertGreater(a, 0)
> +else:
> +self.assertEqual(a, 0)
> try:
> # While we could imagine a Python session where the number of
> # multiple buffer objects would exceed the sharing of references,
> diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c
> --- a/Objects/obmalloc.c
> +++ b/Objects/obmalloc.c
> @@ -1316,6 +1316,13 @@
> {
> PyMem_FREE(p);
> }
> +
> +Py_ssize_t
> +_Py_GetAllocatedBlocks(void)
> +{
> +return 0;
> +}
> +
> #endif /* WITH_PYMALLOC */
>
> #ifdef PYMALLOC_DEBUG
>
> --
> Repository URL: http://hg.python.org/cpython
>
> ___
> Python-checkins mailing list
> python-check...@python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
--
Thanks,
Andrew Svetlov
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More compact dictionaries with faster iteration
On Dec 11, 2012, at 1:13 AM, Antoine Pitrou wrote: >> >> On Dec 10, 2012, at 2:48 AM, Christian Heimes >> wrote: >> >>> On the other hand every lookup and collision checks needs an >>> additional multiplication, addition and pointer dereferencing: >>> >>> entry = entries_baseaddr + sizeof(PyDictKeyEntry) * idx >> >> >> Currently, the dict implementation allows alternative lookup >> functions based on whether the keys are all strings. >> The choice of lookup function is stored in a function pointer. >> That lets each lookup use the currently active lookup >> function without having to make any computations or branches. > > An indirect function call is technically a branch, as seen from the CPU > (and not necessarily a very predictable one, although recent Intel > CPUs are said to be quite good at that). FWIW, we already have an indirection to the lookup function. I would piggyback on that, so no new indirections are required. My plan now is to apply the space compaction idea to sets. That code is less complex than dicts, and set operations stand to benefit the most from improved iteration speed. The steps would be: * Create a good set of benchmarks for set operations for both size and speed. * Start with the simplest version of the idea: separate the entries table from the hash table. Keep the hash table at Py_ssize_t, and pre-allocate the entry array to two-thirds the size of the hash table. This should give about a 25% space savings and speed-up iteration for all the set-to-set operations. * If that all works out, I want to trim the entry table for frozensefs so that the entry table has no over-allocations. This should give a much larger space savings. * Next, I want to experiment with the idea of using char/short/long sizes for the hash table. Since there is already an existing lookup indirection, this can be done with no additional overhead. Small sets will get the most benefit for the space savings and the cache performance for hash lookups should improve nicely (for a hash table of size 32 could fit in a single cache line). At each step, I'll run the benchmarks to make sure the expected speed and space benefits are being achieved. As a side-effect, sets without deletions will retain their insertion order. If this is of concern, it would be no problem to implement Antoine's idea of scrambling the entries during iteration. Raymond P.S. I've gotten a lot of suggestions for improvements to the proof-of-concept code. Thank you for that. The latest version is at: http://code.activestate.com/recipes/578375/ In that code, entries are stored in regular Python lists and inherit their over-allocation characteristics (about 12.5% overallocated for large lists). There are many other possible allocation strategies with their own respective speed/space trade-offs. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Understanding the buffer API
On 08/08/2012 11:47, Stefan Krah wrote: Nick Coghlan wrote: It does place a constraint on consumers that they can't assume those fields will be NULL just because they didn't ask for them, but I'm struggling to think of any reason why a client would actually *check* that instead of just assuming it. Can we continue this discussion some other time, perhaps after 3.3 is out? I'd like to respond, but need a bit more time to think about it than I have right now (for this issue). Those who contributed to the design of it through discussion here may be interested in how this has turned out in Jython. Although Jython is still at a 2.7 alpha, the buffer API has proved itself in a few parts of the core now and feels reasonably solid. It works for bytes in one dimension. There's a bit of description here: http://wiki.python.org/jython/BufferProtocol Long story short, I took the route of providing all information, which makes the navigational parts of the flags argument unequivocally a statement of what navigation the client is assuming will be sufficient. (The exception if thrown says explicitly that it won't be enough.) It follows that if two clients want a view on the same object, an exporter can safely give them the same one. Buffers take care of export counting for the exporter (as in the bytearray resize lock), and buffers can give you a sliced view of themselves without help from the exporter. The innards of memoryview are much simpler for all this and enable it to implement slicing (as in CPython 3.3) in one dimension. There may be ideas worth stealing here if the CPython buffer is revisited. N dimensional arrays and indirect addressing, while supported in principle, have no implementation. I'm fairly sure multi-byte items, as a way to export arrays of other types, makes no sense in Java where type security is strict and a parallel but type-safe approach will be needed. Jeff Allen ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] More compact dictionaries with faster iteration
Good plan! On Tue, Dec 18, 2012 at 11:35 PM, Raymond Hettinger wrote: > > On Dec 11, 2012, at 1:13 AM, Antoine Pitrou wrote: > > > On Dec 10, 2012, at 2:48 AM, Christian Heimes > wrote: > > On the other hand every lookup and collision checks needs an > additional multiplication, addition and pointer dereferencing: > > entry = entries_baseaddr + sizeof(PyDictKeyEntry) * idx > > > > Currently, the dict implementation allows alternative lookup > functions based on whether the keys are all strings. > The choice of lookup function is stored in a function pointer. > That lets each lookup use the currently active lookup > function without having to make any computations or branches. > > > An indirect function call is technically a branch, as seen from the CPU > (and not necessarily a very predictable one, although recent Intel > CPUs are said to be quite good at that). > > > FWIW, we already have an indirection to the lookup function. > I would piggyback on that, so no new indirections are required. > > My plan now is to apply the space compaction idea to sets. > That code is less complex than dicts, and set operations > stand to benefit the most from improved iteration speed. > > The steps would be: > > * Create a good set of benchmarks for set operations >for both size and speed. > > * Start with the simplest version of the idea: separate the >entries table from the hash table. Keep the hash table at >Py_ssize_t, and pre-allocate the entry array to two-thirds the size >of the hash table. This should give about a 25% space savings >and speed-up iteration for all the set-to-set operations. > > * If that all works out, I want to trim the entry table for frozensefs >so that the entry table has no over-allocations. This should >give a much larger space savings. > > * Next, I want to experiment with the idea of using char/short/long >sizes for the hash table. Since there is already an existing >lookup indirection, this can be done with no additional overhead. >Small sets will get the most benefit for the space savings and >the cache performance for hash lookups should improve nicely >(for a hash table of size 32 could fit in a single cache line). > > At each step, I'll run the benchmarks to make sure the expected > speed and space benefits are being achieved. > > As a side-effect, sets without deletions will retain their insertion > order. If this is of concern, it would be no problem to implement > Antoine's idea of scrambling the entries during iteration. > > > Raymond > > > P.S. I've gotten a lot of suggestions for improvements to the > proof-of-concept code. Thank you for that. The latest version > is at: http://code.activestate.com/recipes/578375/ > In that code, entries are stored in regular Python lists > and inherit their over-allocation characteristics (about > 12.5% overallocated for large lists). There are many > other possible allocation strategies with their own > respective speed/space trade-offs. > > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com > -- Thanks, Andrew Svetlov ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Mention OSError instead of IOError in the docs.
On Wed, Dec 19, 2012 at 7:16 AM, andrew.svetlov
wrote:
> http://hg.python.org/cpython/rev/a6ea6f803017
> changeset: 80934:a6ea6f803017
> user:Andrew Svetlov
> date:Tue Dec 18 23:16:44 2012 +0200
> summary:
> Mention OSError instead of IOError in the docs.
>
> files:
> Doc/faq/library.rst | 4 ++--
> 1 files changed, 2 insertions(+), 2 deletions(-)
>
>
> diff --git a/Doc/faq/library.rst b/Doc/faq/library.rst
> --- a/Doc/faq/library.rst
> +++ b/Doc/faq/library.rst
> @@ -209,7 +209,7 @@
> try:
> c = sys.stdin.read(1)
> print("Got character", repr(c))
> - except IOError:
> + except OSError:
> pass
> finally:
> termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm)
> @@ -222,7 +222,7 @@
> :func:`termios.tcsetattr` turns off stdin's echoing and disables
> canonical
> mode. :func:`fcntl.fnctl` is used to obtain stdin's file descriptor
> flags
> and modify them for non-blocking mode. Since reading stdin when it is
> empty
> - results in an :exc:`IOError`, this error is caught and ignored.
> + results in an :exc:`OSError`, this error is caught and ignored.
>
With any of these changes in the docs, please don't forget to include
appropriate "versionchanged" directives. Many people using the Python 3
docs at "docs.python.org/3/" will still be on Python 3.2, and thus relying
on the presence of such directives to let them know that while the various
OS-related exception names are now just aliases for OSError in 3.3+, the
distinctions still matter in 3.2.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
