Re: [Python-Dev] PEP 3121, 384 Refactoring Issues
Brett Cannon wrote: > No, the PEPs were fine and were accepted properly. A huge portion of the open > issues are from Robin Schreiber who as part of GSoC 2012 -- https:// > www.google-melange.com/gsoc/project/details/google/gsoc2012/robin_hood/ > 5668600916475904 -- went through and updated the stdlib to follow the new > practices introduced in the two PEPs. Not sure if there was some policy > decision made that updating the code wasn't worth it or people simply didn't > get around to applying the patches. Due to the frequent state lookups there is a performance problem though, which is quite significant for _decimal. Otherwise I think I would have implemented the changes already. http://bugs.python.org/issue15722 I think for speed sensitive applications it may be an idea to create a new C function (METH_STATE flag) which gets the state passed in by ceval. Other than that, looking up the state inside the module but cache it (like it's done for the _decimal context) also has reasonable performance. Also I hit the same issues that Eli mentioned here a while ago: https://mail.python.org/pipermail/python-dev/2013-August/127862.html Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [libmpdec] mpdecimal-2.4.1 released
Hi, I've released mpdecimal-2.4.1: http://www.bytereef.org/mpdecimal/changelog.html da74d3cfab559971a4fbd4fb506e1b4498636eb77d0fd09e44f8e546d18ac068 mpdecimal-2.4.1.tar.gz Starting with Python 3.4.2, this version should be used for an external libmpdec. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Cross compiling Python (for Android)
Frank, Matthew I intel.com> writes: > 4. Module _decimal is failing to compile. The problem is that it has > a header called memory.h. Android's libc has the problem that > /usr/include/stdlib.h includes . But the build system > puts -I. on the include path before the system dirs (as it should) > so when compiling _decimal, Modules/_decimal/libmpdec/memory.h gets > found instead of /usr/include/memory.h. Shiz has a patch here: > https://github.com/rave-engine/python3-android/blob/master/mk/python/3.3.5/p\ > ython-3.3.5-android-libmpdec.patch > (which renames memory.h -> mpmemory.h) but I don't know > > a. Is there a tracker for this yet? and > b. Is Shiz's fix the desired one or should I be looking for > another approach? (Maybe modifying the -I flags for the build > of just the build of _decimal or something?) I think using "memory.h" in an application is standard conforming. Since _decimal compiles on all other Linux platforms, it may be worth reporting this to the Android developers and see if they can fix it (possibly by not including memory.h in stdlib.h). FWIW, OCaml also has a "memory.h" header. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.4): Issue #23446: Use PyMem_New instead of PyMem_Malloc to avoid possible integer
On Mon, Feb 16, 2015 at 11:35:52AM +, serhiy.storchaka wrote:
> diff --git a/Modules/_testbuffer.c b/Modules/_testbuffer.c
> --- a/Modules/_testbuffer.c
> +++ b/Modules/_testbuffer.c
> @@ -850,7 +850,7 @@
> Py_ssize_t *dest;
> Py_ssize_t x, i;
>
> -dest = PyMem_Malloc(len * (sizeof *dest));
> +dest = PyMem_New(Py_ssize_t, len);
> if (dest == NULL) {
> PyErr_NoMemory();
> return NULL;
This, too, was already protected by len == ndim <= 64.
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 550 v4
On Thu, Sep 07, 2017 at 09:41:10AM -0400, Elvis Pranskevichus wrote: > threading.local(), the isolation mechanism, is *implicit*. > decimal.localcontext() is an *explicit* resource manager that relies on > threading.local() magic. PEP 550 simply provides a threading.local() > alternative that works in tasks and generators. That's it! If there only were a name that would make it explicit, like TaskLocalStorage. ;) Seriously, the problem with 'context' is that it is: a) A predefined set of state values like in the Decimal (I think also the OpenSSL) context. But such a context is put inside another context (the ExecutionContext). b) A theoretical concept from typed Lambda calculus (in the context 'gamma' the variable 'v' has type 't'). But this concept would be associated with lexical scope and would extend to functions (not only tasks and generators). c) ``man 3 setcontext``. A replacement for setjmp/longjmp. Somewhat related in that it could be used to implement coroutines. d) The .NET flowery language. I do did not fully understand what the .NET ExecutionContext and its 2881 implicit flow rules are. ... Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 554 v2 (new "interpreters" module)
On Fri, Sep 08, 2017 at 04:04:27PM -0700, Eric Snow wrote: > * "stdlib support for subinterpreters adds extra burden > on C extension authors" > > In the ``Interpreter Isolation`` section below we identify ways in > which isolation in CPython's subinterpreters is incomplete. Most > notable is extension modules that use C globals to store internal > state. PEP 3121 and PEP 489 provide a solution for most of the > problem, but one still remains. [petr-c-ext]_ Until that is resolved, > C extension authors will face extra difficulty to support > subinterpreters. It's a bit of a hassle, and the enormous slowdown in some of the existing solutions is really a no go [1]. In the case of _decimal, the tls-context is already subinterpreter safe and reasonable fast due to caching. The most promising model to me is to put *all* globals in a tls structure and cache the whole structure. Extrapolating from my experiences with the context, this might have a slowdown of "only" 4%. Still, the argument "who uses subinterpreters?" of course still remains. Stefan Krah [1] I'm referring to the slowdown of heaptypes + module state. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 557: Data Classes
On Thu, Sep 14, 2017 at 10:24:52AM -0700, Mike Miller wrote: > An elegant name can make the difference between another obscure > module thrown in the stdlib to be never seen again and one that gets > used every day. Which is more intuitive? > > from collections import record I'd expect something like a C struct or an ML record. > from dataclass import dataclass This is more intuitive, since the PEP example also has attached methods like total_cost(). I don't think this is really common for records. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 557: Data Classes
On Thu, Sep 14, 2017 at 11:06:15AM -0700, Mike Miller wrote: > On 2017-09-14 10:45, Stefan Krah wrote: > >I'd expect something like a C struct or an ML record. > > Struct is taken, and your second example is record. *If* the name were collections.record, I'd expect collections.record to be something like a C struct or an ML record. I'm NOT proposing "record". > > from dataclass import dataclass > > > >This is more intuitive, since the PEP example also has attached methods > >like total_cost(). I don't think this is really common for records. > > Every class can be extended, does that mean they can't be given appropriate > names? A class is not a record. This brief conversation already convinced me that "record" is a bad name for the proposed construct. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Does Cygwin still have broken slot initialization?
Hi, The docs have this rule for slot initialization for the benefit of Cygwin: https://github.com/python/cpython/commit/db6a569de7ae595ada53b618fce6bbbd1c98d350 Synopsis -PyType_GenericNew, /* tp_new */ +noddy_NoddyType.tp_new = PyType_GenericNew; +if (PyType_Ready(&noddy_NoddyType) < 0) +return; This is absolutely not required by C99 (and probably never was). 'PyType_GenericNew' is an address constant, and MSVC supports it just fine -- at least since VS 2008. Does anyone know if Cygwin still misbehaves? I would like to get rid of this arcane rule. https://bugs.python.org/issue31443 Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] If aligned_alloc() is missing on your platform, please let us know.
Hello, we want to add aligned versions of allocation functions to 3.7: https://bugs.python.org/issue18835 C11 has aligned_alloc(). Linux, BSD, OSX, MSVC, Android all have either posix_memalign() or _aligned_malloc(). Cygwin apparently has posix_memalign(). MinGW has: https://github.com/Alexpux/mingw-w64/blob/master/mingw-w64-crt/misc/mingw-aligned-malloc.c Victor wrote a patch and would like to avoid adding a (probably unnecessary) emulation function. I agree with that. So if any platform does not have some form of aligned_alloc(), please speak up. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Migrate python-dev to Mailman 3?
Barry Warsaw wrote: > In fact, we come with plugins for mail-archive.com and MHonarc. MHonarc output is nice, practically the same as pipermail: https://lists.debian.org/debian-x/2010/12/threads.html If it is possible to enable (and maintain!) a MHonarc archive with just blue links along with the new interface, everyone should be happy. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] If aligned_alloc() is missing on your platform, please let us know.
On Sat, Oct 28, 2017 at 03:43:47PM +1000, Nick Coghlan wrote: > 1. CPython's own support for platforms where we don't have a native aligned > memory allocation API to call is covered by PEP 11, so if all current > buildbots still work, then it will be up to the folks interested in a > platform that *doesn't* offer aligned allocations to provide both a > suitable emulation and a buildbot to test it on. Indeed, the feature is backed up by PEP 11. > 2.While all of the CPython-provided memory allocators will implement the > new slots, the folks implementing their own custom allocators will need a > defined upgrade path in the "Porting" section of the What's New guide. For > that, an explicit error on 3.7+ that says "Configured custom allocator > doesn't implement aligned memory allocations" is likely going to be easier > to debug than subtle issues with the way an implicit emulation layer > interacts with the other memory allocator slots. > > To appropriately address 2, we need more info not about which platforms > natively support aligned memory allocations, but rather from folks that > actually are implementing their own custom allocators. It may be that > making the low level cross-platform raw alligned allocators available as a > public API (independently of the switchable allocator machinery) will be a > more appropriate way of handling that case than providing an emulation > layer in terms of the old slots. I don't have an opinion whether new slots should be available. For my use case I just need PyMem_AlignedAlloc(), PyMem_AlignedFree() that automatically use the faster allocator for 'sizeof(void *) <= align <= ALIGNMENT' and 'size <= SMALL_REQUEST_THRESHOLD'. So yes, it would be nice to hear from people who implement custom allocators. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Migrate python-dev to Mailman 3?
On Mon, Oct 30, 2017 at 12:00:22PM +0100, Victor Stinner wrote: > Except of Antoine Pitrou, does everybody else like the new UI? :-) No, I don't like it. If there is a promise to keep an additional, MHonArc or Pipermail archive *with an implicit promise of long term support*, I don't care. Despite the mentioned shortcomings of Pipermail, it is 5 times faster for me to navigate and less stressful to look at. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Migrate python-dev to Mailman 3?
On Mon, Oct 30, 2017 at 07:46:42AM -0700, Guido van Rossum wrote: > I love MM3 and hyperkitty. But I rarely peruse the archives -- I only go to > pipermail to get a link to a specific message from the past so I can copy > it into a current message. IIUC that functionality is actually better in > hyperkitty because when a pipermail archive is rebuilt the message numbers > come out differently. Yes, I use the archives differently. When I'm temporarily unsubscribed due to overload I scan the archives for interesting topics and indeed sometimes read whole threads. I think Pipermail is great for that. Quiet design, nice font, good contrast for speed reading. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Guarantee ordered dict literals in v3.7?
Hello, would it be possible to guarantee that dict literals are ordered in v3.7? The issue is well-known and the workarounds are tedious, example: https://mail.python.org/pipermail/python-ideas/2015-December/037423.html If the feature is guaranteed now, people can rely on it around v3.9. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Sun, Nov 05, 2017 at 01:14:54PM -0500, Paul G wrote:
> I'm not entirely sure I understand the full set of reasoning for this - I
> couldn't really tell what the problem with OrderedDict is from the link
> Stefan provided. It seems to me like a kind of huge change for the language
> to move from arbitrary-ordered to guaranteed-ordered dict. The problem I see
> is that this introduces a huge backwards compatibility burden on all
> implementations of Python.
Scientific applications want something like
{'a': 10, 'b': "foo", 'c': {'this': b'123'}}
as an ordered initializer for unboxed or typed (or both) data. In general,
if dicts are ordered, they can be used for example as initializers for
(nested) C structs.
> 2. Someone invents a new arbitrary-ordered container that would improve on
> the memory and/or CPU performance of the current dict implementation
I would think this is very unlikely, given that the previous dict implementation
has always been very fast. The new one is very fast, too.
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Sun, Nov 05, 2017 at 09:01:40PM +0200, Serhiy Storchaka wrote: > Do you suggest to make dictionary displays producing OrderedDict > instead of dict? No, this is essentially a language spec doc issue that would guarantee the ordering properties of the current dict implementation. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Sun, Nov 05, 2017 at 09:09:37PM +0200, Serhiy Storchaka wrote: > 05.11.17 20:39, Stefan Krah пише: > >On Sun, Nov 05, 2017 at 01:14:54PM -0500, Paul G wrote: > >>2. Someone invents a new arbitrary-ordered container that would improve on > >>the memory and/or CPU performance of the current dict implementation > > > >I would think this is very unlikely, given that the previous dict > >implementation > >has always been very fast. The new one is very fast, too. > > The modification of the current implementation that don't preserve > the initial order after deletion would be more compact and faster. How much faster? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Sun, Nov 05, 2017 at 09:35:38PM +0200, Serhiy Storchaka wrote:
> 05.11.17 21:20, Stefan Krah пише:
> >On Sun, Nov 05, 2017 at 09:01:40PM +0200, Serhiy Storchaka wrote:
> >>Do you suggest to make dictionary displays producing OrderedDict
> >>instead of dict?
> >
> >No, this is essentially a language spec doc issue that would guarantee
> >the ordering properties of the current dict implementation.
>
> Wouldn't be enough to guarantee just the ordering of dicts before
> first deletion? Or before first resizing (the maximal size of
> dictionary displays is known at compile time, so they can be
> presized)?
Yes, for my use case that would be sufficient and that's what
I had in mind initially.
A luxury syntax addition like {a = 10, b = {c = "foo"}} that is read
as an OrderedDict (where the keys a, b, c are implicitly strings) would
of course also be sufficient for my use case.
But I suspect many users who have other use cases find it tantalizing
not to be able to use the properties of the current regular dict.
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Mon, Nov 06, 2017 at 12:18:17PM +0200, Paul Sokolovsky wrote: > MicroPython hashmap implementation is effectively O(n) (average and > worst case) due to the algorithm parameters chosen (like the load factor > of 1). Of course, parameters could be tweaked, but the ones chosen are > so because the memory usage is far more important for MicroPython > systems than performance characteristics (all due to small amounts of > memory). Like, MicroPython was twice as fast than Python 3.3 on > average, and 1000 times more efficient in the memory usage. $ cat xxx.py def pi_float(): """native float""" lasts, t, s, n, na, d, da = 0, 3.0, 3, 1, 0, 0, 24 while s != lasts: lasts = s n, na = n+na, na+8 d, da = d+da, da+32 t = (t * n) / d s += t return s for i in range(10): x = pi_float() $ time ./micropython xxx.py real0m4.424s user0m4.406s sys 0m0.016s $ $ time ../../cpython/python xxx.py real0m1.066s user0m1.056s sys 0m0.010s Congratulations ... Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Guarantee ordered dict literals in v3.7?
On Mon, Nov 06, 2017 at 01:03:05PM +0200, Paul Sokolovsky wrote: > > $ time ./micropython xxx.py > > $ time ../../cpython/python xxx.py > > > > > Congratulations ... > > That's why I wrote "Python 3.3", it's hard to argue CPython doesn't do > anything about the "Python is slow" proverb. It's still shouldn't be > hard to construct a testcase where MicroPython is faster (by not > implementing features not needed by 90% of Python programs of course, > not "for free"). Sorry, that was a slightly mischievous benchmark indeed. -- Whether the proposal is surreal or not depends on the likelihood that a) a substantially faster dict algorithm will emerge and b) CPython, PyPy and Jython will switch to it. My proposal was based on the fact that for almost two release cycles the ordering implementation detail hasn't changed. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] The current dict is not an "OrderedDict"
This is just a reminder that the current dict is not an "OrderedDict": >>> from collections import OrderedDict >>> OrderedDict(a=0, b=1) == OrderedDict(b=1, a=0) False >>> dict(a=0, b=1) == dict(b=1, a=0) True The recent proposal was primarily about guaranteeing the insertion order of dict literals. If further guarantees are proposed, perhaps it would be a good idea to open a new thread and state what exactly is being proposed. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The current dict is not an "OrderedDict"
On Wed, Nov 08, 2017 at 12:01:04AM +1000, Nick Coghlan wrote: > > The recent proposal was primarily about guaranteeing the insertion order of > > dict literals. > > > > If further guarantees are proposed, perhaps it would be a good idea to > > open a new thread and state what exactly is being proposed. > > "Insertion ordered until the first key removal" is the only guarantee > that's being proposed. > > OrderedDict just comes into the discussion because reaching for its > stronger guarantees is currently the only way to obtain that guarantee > in a formally implementation-independent and future-proof way. Ok good, I was primarily worried about collections.UnorderedDict coming up and users thinking that OrderedDict could be replaced entirely by dict(). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Add Py_SETREF and Py_XSETREF to the stable C API
On Fri, Nov 10, 2017 at 12:09:12PM +1000, Nick Coghlan wrote: > I'm with Antoine on this - we should be pushing folks writing > extension modules towards code generators like Cython, cffi, SWIG, and > SIP, support libraries like Boost::Python, or safer languages like > Rust (which can then be wrapped with cffi), rather than encouraging > more bespoke C/C++ extensions modules with handcrafted refcount > management. There's a reason the only parts of > https://packaging.python.org/guides/packaging-binary-extensions/ that > have actually been filled in are the ones explaining how to use a tool > to write the extension module for you :) They will be slower and in my experience not easier to maintain -- quite the opposite. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 565: Show DeprecationWarning in __main__
On Mon, Nov 13, 2017 at 10:37:46PM +1100, Chris Angelico wrote: > >> https://bugs.python.org/issue1539925 > >> https://github.com/ipython/ipython/issues/6611 > > > > Depends what you call "better". Personally, I don't want to see > > warnings each and every time I use a deprecated or questionable > > construct or API from the REPL. > > Isn't that the entire *point* of warnings? When you're working at the > REPL, you're the one in control of which APIs you use, so you should > be the one to know about deprecations. I haven't followed the long discussions, so this is probably not a very novel observation. But it seems to me that we have a problem getting users to treat the python command like e.g. gcc. If I want gcc warnings, I use -Wall -Wextra. I think the whole problem is that python warnings are a bit of an obscure feature (they were to me for a long time). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Use a queue in Tkinter (was: Dealing with tone in an email)
Steven D'Aprano wrote: >> What exactly didn't work? I don't understand. > > https://bugs.python.org/issue33412 Isn't the standard solution to use a queue for updating the GUI? At least I didn't have any problems at all with my one TKinter app, I think the method is described in the Python Cookbook. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Discussion related to memory leaks requested
Petr Viktorin gmail.com> writes: > The hairy details on why the global variables haven't yet gone away are > on import-sig [0]. Nick suggested a workable solution there that I > really need to go back to and implement. > > [0] https://mail.python.org/pipermail/import-sig/2015-July/001022.html I want to add here that existing schemes for eliminating global variables are inefficient (20% speed hit for _decimal), so a complete solution would have to address that as well. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update PEP 7 to require curly braces in C
Brett Cannon python.org> writes: > Anyone object if I update PEP 7 to remove the optionality of curly braces in PEP 7? I strongly prefer braces everywhere, but I'm -1 on enforcing it. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update PEP 7 to require curly braces in C
M.-A. Lemburg egenix.com> writes: > > Currently this thread stands at: > > Make that: > > > +1 > > Brett > > Ethan > > Robert > > Georg > > Nick > > Maciej Szulik > > +0 > > Guido > > -0 > > Serhiy > > -1 > MAL > > Victor (maybe; didn't specifically vote) > > Larry > > Stefan I want to clarify my position a bit: Personally, in _decimal/* I've always used braces and I prefer that. But from reading the Python sources in general, I got the impression that the default style at least for one-liner if-statements is to omit braces. So, in memoryview.c, I adapted to that style. I think enforcing braces won't do anything for security. DJB (who had a single exploit found in qmail in 20 years) even uses nested for-loops without braces. IMO secure code can only be achieved by auditing it quietly in a terminal, not being distracted by peripheral things like version control and web interfaces (green merge buttons!) and trying to do formal proofs (if time allows for it). So I would not want to enforce a style if it makes some people unhappy. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] BAD Benchmark Results for Python Default 2016-01-26
IMO the timings of the benchmark suite are a bit unstable -- this is not the fault of Intel's setup, I noticed it also when running the suite myself. On Tue, Jan 26, 2016 at 06:48:54PM +, Stewart, David C wrote: > Wow, what happened to Python default to cause such a regression? > > > > > On 1/26/16, 7:31 AM, "lp_benchmark_robot" > wrote: > > >Results for project Python default, build date 2016-01-26 03:07:40 + > >commit: cbd4a6a2657e > >previous commit: f700bc0412bc > >revision date: 2016-01-26 02:54:37 + > >environment: Haswell-EP > > cpu:Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz 2x18 cores, > > stepping 2, LLC 45 MB > > mem:128 GB > > os: CentOS 7.1 > > kernel: Linux 3.10.0-229.4.2.el7.x86_64 > > > >Baseline results were generated using release v3.4.3, with hash b4cbecbc0781 > >from 2015-02-25 12:15:33+00:00 > > > >-- > > benchmark relative change since change since current > > rev run > > std_dev* last run baseline > > with PGO > >-- > >:-) django_v2 0.21% -2.93% 8.95% > >16.19% > >:-| pybench 0.10% 0.05% -1.87% > >5.40% > >:-(regex_v8 2.72% -0.02% -4.67% > >4.57% > >:-| nbody 0.13% -0.92% -1.33% > >7.40% > >:-|json_dump_v2 0.20% 0.87% -1.59% > >11.48% > >:-| normal_startup 0.90% -0.57% 0.10% > >5.35% > >-- > >* Relative Standard Deviation (Standard Deviation/Average) > > > >If this is not displayed properly please visit our results page here: > >http://languagesperformance.intel.com/bad-benchmark-results-for-python-default-2016-01-26/ > > > >Note: Benchmark results are measured in seconds. > > > >Subject Label Legend: > >Attributes are determined based on the performance evolution of the workloads > >compared to the previous measurement iteration. > >NEUTRAL: performance did not change by more than 1% for any workload > >GOOD: performance improved by more than 1% for at least one workload and > >there > >is no regression greater than 1% > >BAD: performance dropped by more than 1% for at least one workload and there > >is > >no improvement greater than 1% > >UGLY: performance improved by more than 1% for at least one workload and also > >dropped by more than 1% for at least one workload > > > > > >Our lab does a nightly source pull and build of the Python project and > >measures > >performance changes against the previous stable version and the previous > >nightly > >measurement. This is provided as a service to the community so that quality > >issues with current hardware can be identified quickly. > > > >Intel technologies' features and benefits depend on system configuration and > >may > >require enabled hardware, software or service activation. Performance varies > >depending on system configuration. > ___ > Python-checkins mailing list > python-check...@python.org > https://mail.python.org/mailman/listinfo/python-checkins ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3)
Guido van Rossum python.org> writes: > I don't care too much either way, but I think passing underscores to the constructor shouldn't be affected by the context -- the underscores are just removed before parsing the number. But if it's too complicated to implement I'm fine with punting. Just removing the underscores would be fine. The problem is that per the PEP the conversion should happen according the Python float grammar but the actual decimal grammar is the one from the IBM specification. I'd much rather express the problem like you did above: A preprocessing step followed by the IBM specification grammar. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 515: Underscores in Numeric Literals (revision 3)
Guido van Rossum python.org> writes:
> So should the preprocessing step just be s.replace('_', ''), or should
> it reject underscores that don't follow the rules from the PEP
> (perhaps augmented so they follow the spirit of the PEP and the letter
> of the IBM spec)?
>
> Honestly I think it's also fine if specifying this exactly is left out
> of the PEP, and handled by whoever adds this to Decimal. Having a PEP
> to work from for the language spec and core builtins (int(), float()
> complex()) is more important.
I'd keep it simple for Decimal: Remove left and right whitespace (we're
already doing this), then remove underscores from the remaining string
(which must not contain any further whitespace), then use the IBM grammar.
We could add a clause to the PEP that only those strings that follow
the spirit of the PEP are guaranteed to be accepted in the future.
One reason for keeping it simple is that I would not like to slow down
string conversion, but thinking about two grammars is also a problem --
part of the string conversion in libmpdec is modeled in ACL2, which
would be invalidated or at least complicated with two grammars.
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Most 3.x buildbots are green again, please don't break them and watch them!
Victor Stinner gmail.com> writes: > Maybe it's time to move more 3.x buildbots to the "stable" category? > http://buildbot.python.org/all/waterfall?category=3.x.stable +1 I think anything that is actually stable should be in that category. > By the way, I don't understand why "AMD64 OpenIndiana 3.x" is > considered as stable since it's failing with multiple issues since > many months and nobody is working on these failures. I suggest to move > this buildbot back to the unstable category. +1 The bot was very stable and fast for some time but has been unstable for at least a year. > - PPC64 AIX 3.x: failing tests: test_httplib, test_httpservers, > test_socket, test_distutils, test_asyncio, (...); random timeout > failure in test_eintr, etc. I don't have access to AIX and I'm not > interested to acquire an AIX license, nor to install it. I'm not sure > that it's useful to have an AIX buildbot and no core developer have > access to AIX, and nobody is working on AIX failures. Maybe HP wants > to help us to support AIX? (Provide manpower, access to AIX servers, > or something like that.) Well, I think in this case it's the gcc AIX maintainer running it, so... I think we should have a policy to stop reporting issues on unstable bots unless someone has a concrete fix OR the bot maintainers are known to fix issues fast (but that does not seem to be the case). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
Xavier de Gaye gmail.com> writes: > Starting with API level 21 (Android 5.0), the build of python3 with the > official android toolchains (that is, without resorting to external libraries > for wide character support) runs correctly. With the set of patches described > in the patches/Makefile file at [1], the cpython test suite runs[2] on the > android x86 and armv7 emulators with only few errors[3]. Those errors are > listed with their corresponding error messages, this may give a raw idea of > the effort needed to support this platform. > > Xavier > > [1] https://bitbucket.org/xdegaye/pyona/src > [2] To reproduce these results, follow the instructions found in INSTALL > at https://bitbucket.org/xdegaye/pyona/wiki/install > [3] https://bitbucket.org/xdegaye/pyona/wiki/testsuite This looks great, very clean! As I understand the patches, the locale.h and langinfo.h problems are solved. Do you think the following issues on the Python bug tracker could be closed? http://bugs.python.org/issue20305 http://bugs.python.org/issue22747 http://bugs.python.org/issue17905 Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
Eric Snow gmail.com> writes: > On Sun, Apr 24, 2016 at 1:20 AM, Xavier de Gaye gmail.com> wrote: > > Starting with API level 21 (Android 5.0), the build of python3 with the > > official android toolchains (that is, without resorting to external > How does this relate to http://bugs.python.org/issue23496? As I understand, that issue seems abandoned and the patches are (despite core devs asking otherwise) against 3.4. If Xavier is willing to do so, I think it would be best to start over with a new issue that integrates his work into 3.6. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
Xavier de Gaye gmail.com> writes: > This code, or part of it, could be used to setup a buildbot and in this case > there would not be any conflict between the GPL v3 license and the Python > license, I think. I don't see how it can be combined with Python 3. For the patches on the tracker I just went by your contributor agreement. I didn't check the lineage of the patches. Can I assume that either you are re-licensing GPL-stuff written by yourself to the PSF (which is a perfectly valid use case of the agreement) or rewriting from scratch? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] support of the android platform
Xavier de Gaye gmail.com> writes: > Yes, I am re-licensing GPL code to the PSF for all the patches written by me > in the issues listed on http://bugs.python.org/issue26865#msg264310. I have > only rewritten the patches from scratch in the following issues: Thanks, this all sounds good. > issue #26854: missing header on android for the ossaudiodev module >(actually it's difficult to rewrite such an obvious patch) Indeed. :) Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Hrvoje Niksic avl.com> writes: > This inconsistency has caused bugs (or, more fairly, potential leaks) > before, see http://bugs.python.org/issue1782 > > Unfortunately, the suggested Python 3 change to PyModule_AddObject was > not accepted. First, these "leaks" only potentially show up when you already have much bigger problems (i.e. on Linux the machine would already freeze due to overallocation). Second, these "leaks" don't even show up as "definitely lost" in Valgrind (yes, I checked). On the bright side, Python must be in a very healthy state if we can afford to spend time on issues such as this one. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Serhiy Storchaka gmail.com> writes: > No impact except emitting a deprecation warning at build time. But we > can remove a deprecation warning and add it in future release if this is > annoying. > > But are you sure, that your code uses PyModule_AddObject() correctly? > Only two modules in the stdlib (_json and _tkinter) used it correctly. > Other modules have bugs even in tries to use PyModule_AddObject() > correctly for some operations. Could you perhaps stop labeling this as a bug? Usually we are talking about a *single* "leak" that a) does not even show up in Valgrind and b) only occurs under severe memory pressure when the OOM-killer is already waiting. I'm honestly mystified by your terminology and it's beginning to feel that you need to justify this patch at all costs. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Serhiy Storchaka gmail.com> writes: > But are you sure, that your code uses PyModule_AddObject() correctly? > Only two modules in the stdlib (_json and _tkinter) used it correctly. > Other modules have bugs even in tries to use PyModule_AddObject() > correctly for some operations. For the list, this is the extent of this horrible "bug": diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -5804,8 +5804,7 @@ PyObject_CallObject((PyObject *)&PyDecContext_Type, NULL)); init_basic_context(basic_context_template); Py_INCREF(basic_context_template); -CHECK_INT(PyModule_AddObject(m, "BasicContext", - basic_context_template)); +CHECK_INT(-1); $ valgrind --suppressions=Misc/valgrind-python.supp ./python -c "import decimal" [...] ==16945== LEAK SUMMARY: ==16945==definitely lost: 0 bytes in 0 blocks ^^^^ [...] Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Random832 fastmail.com> writes: > On Thu, Apr 28, 2016, at 05:05, Stefan Krah wrote: > > $ valgrind --suppressions=Misc/valgrind-python.supp ./python -c "import > > decimal" > > > > [...] > > ==16945== LEAK SUMMARY: > > ==16945==definitely lost: 0 bytes in 0 blocks > > > > Well, the obvious flaw with your test case is that a reference is > retained forever in the C static variable basic_context_template. For actual users of Valgrind this is patently obvious and was pretty much the point of my post. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Serhiy Storchaka gmail.com> writes: > 2. Most code that use PyModule_AddObject() doesn't work as intended. > Since the bahavior of PyModule_AddObject() contradicts the documentation > and is contrintuitive, we can't blame authors in this. > > I don't say this is a high-impacting bug, I even agree that there is no > need to fix the second part in maintained releases. But this is a bug > unless you propose different definition for a bug. Why do you think that module authors don't know that? For _decimal, I was aware of the strange behavior. Yes, a single reference can "leak" on failure. The problem is that we don't seem to have any common ground here. Do you accept the following? 1) PyModule_AddObject() can only fail if malloc() fails. a) Normally (for small allocations) this is such a serious problem that the whole application fails anyway. b) Say that you're lucky and the application continues. i) The import fails. In some cases ImportError is caught and a fallback is imported (example _pydecimal). In that case you leak an entire DSO and something small like a single context object. What is the practical difference between the two? ii) The import fails and there's no fallback. Usually the application stops, otherwise DSO+small leak again. iii) Retry the import (I have never seen this): while(1): try: import leftpad except (ImportError, MemoryError): continue break You could have a legitimate leak here, but see a). Module initializations are intricate and boring. I suspect that if we promote wide changes across PyPI packages we'll see more additional segfaults than theoretically plugged memory leaks. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Inconsistency of PyModule_AddObject()
Random832 fastmail.com> writes: > A more relevant point would be that _decimal does *not* use the API in a > way *which would be broken by the proposed change*, regardless of > whether the way in which it uses it is subjectively correct or can cause > leaks. And the ultimate point is that I don't want to spend about a week per year to evaluate the effect of needless code changes on a highly audited module. And no, this isn't theoretical... Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Yearly PyPI breakage
Hello, Could someone enlighten me which hoops I have to jump through this year in order to keep pip downloads working? Collecting cdecimal Could not find a version that satisfies the requirement cdecimal (from versions: ) No matching distribution found for cdecimal You are using pip version 7.1.2, however version 8.1.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. If this continues, I'm going to release a premium version that's 50% faster and only available from bytereef.org or Anaconda. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yearly PyPI breakage
Nathaniel Smith pobox.com> writes: > > If this continues, I'm going to release a premium version that's > > 50% faster and only available from bytereef.org or Anaconda. > > There's no point in making threats -- you're threatening the air. PyPI > is maintained by one overloaded developer (Donald Stufft, sponsored by > HPE as part of their openstack work) with help from a few overloaded, > burned-out volunteers. Everyone wants PyPI to be awesome and useful; > your frustration is totally understandable, and lots of us share it. > But the limitation here is that we have ~one person running a website > that serves ~100,000,000 requests/day, and running as fast as they can > just to keep things from breaking more than they already have. As long > as that's the case it doesn't matter what "should" happen, there's > no-one to do it. This wasn't so much of a threat, it was resignation. At some point it seems like the rational thing to do. I don't fully understand your explanation. Judging by the link that Donald posted (thanks!) it seems that PEP 470 introduced extra work for him that would not have been present had things been left in place. Also, I did not get any notification and now that I searched for PEP 470 it seems that it wasn't announced here: https://mail.python.org/pipermail/python-dev/2015-October/141838.html But if the majority prefers PyPI that way, I'll stop arguing. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yearly PyPI breakage
Łukasz Langa langa.pl> writes: > > I don't fully understand your explanation. Judging by the link that > > Donald posted (thanks!) it seems that PEP 470 introduced extra work > > for him that would not have been present had things been left in place. > > IIRC the PyPI maintainers were constantly nagged about “PyPI reliability issues” that were instead > external hosting issues. Everybody was affected every now and then whenever tummy.com or other external > servers for popular packages were down. Or at least I know *I was*. Way too often. But making them completely unreachable does not increase reliability. :) > > But if the majority prefers PyPI that way, I'll stop arguing. > > I’m not sure what you mean here but if you want to argue for reverting PEP 470, I wouldn’t hold my breath. No, I mean what we're doing now. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yearly PyPI breakage
> [cut overlong post] Glyph, nice sneaky way to try to divert from the original issue. Your whole post is invalidated by the simple fact that the URL was protected by a hash (which I repeatedly asked to be upgraded to sha256). This was the official scheme promoted by PEP-438, which you should know. But of course your actual intention here is character assassination, pretending to "rescue" cdecimal and trying to divert from the fact that the transition to PEP 470 was handled suboptimally. The very reason for this thread is that the security was silently disabled WITHOUT me getting a notification. What is on PyPI *now* is not what I configured! Please believe me when I say I do not mean the following to be insulting -- people who have done *actual* cryptography to varying degrees often tend to focus on the important parts and aren't impressed by regurgitating catch phrases like SSL and man-in-the-middle: http://cr.yp.to/ecdh.html The amount of security "experts" in the Python community that pontificate on any occasion is pretty annoying. What do you think djb thinks of Twisted? > If anyone wants package-index access to this name to upload Windows or manylinux wheels just let me know; however, as this is just a proof of concept, I do not intend to maintain it long-term. That apparently all you can do: Move bits from place A to place B and not care how long it took to produce them. You are a real hero. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yearly PyPI breakage
sdamon.com> writes: > You posted a mean-spirited complaint about a policy that is nearly exactly > two years old, to the wrong list, and call out the people calmly trying to > explain what happened and why, and how you can mitigate it for your own work Yeah, right. This is distutils-sig level. Read Glyph's "calm" and "nice" post again. Also note that *I* had already bowed out of the thread when Glyph unnecessarily decided to spread complete misinformation. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yearly PyPI breakage
Nick Coghlan gmail.com> writes: > I know you're not happy with myself and the other distutils-sig folks > regarding the decision to deprecate and remove automatic link > spidering, More accurately: Not happy with the removal of the checksummed "explicit" mode. What I would have preferred is a FreeBSD-like ports system. FreeBSD has been used in huge and secure installations, so the I don't buy the reliability and security arguments that are used in favor of centralization. But centralization seems to be a goal in and of itself these days (and that isn't limited to Python). > nor with the PSF regarding the current Terms of Service > around uploads to PyPI, but that doesn't make it OK to start off-topic > threads on python-dev just because you're a CPython core developer in > addition to being a PyPI user. Alexander thought otherwise: https://mail.python.org/pipermail/python-dev/2015-October/141840.html "Given that ensurepip is part of stdlib, I am not sure this is an accurate statement. Even if it was, did you make any effort to discuss the proposal outside of a small group subscribed to distutils ML?" I completely agree with that. Fredrik Lundh is also affected (and might not have received any mail, same as me): https://pypi.python.org/pypi/PIL > It *definitely* doesn't make it OK to accuse people of conspiring > against you when they answer your question in good faith, just because > their answer is the official distutils-sig/PyPA one (which was > approved through the PEP process in PEP 470). I'm not sure what you are referring to. Donald posted a link to PEP 470, in my response to Nathaniel I acknowledged this. In my exchange with Łukasz we both came to the conclusion (I think) that further discussion would be futile. IMO all responses from Brett, Donald, Nathaniel and Łukasz were reasonable and I haven't accused them of conspiring in the slightest. I see that the PEP was accepted by Paul Moore. I couldn't even dream of accusing Paul Moore of any kind of conspiracy. He's one of the most reasonable (and *genuinely* polite) people on these mailing lists. Or are you referring to a super-condescending flame bait where someone cloned a private website, assumed general ignorance and then proceeded to offer a hostile fork to anyone who would be interested? Well, I accepted the flame bait. > - writing to psf-legal to let them know whether or not Van Lindberg's > draft updates to the Terms of Service would be sufficient to make you > comfortable with uploading cdecimal to PyPI in addition to bundling it > with the standard library under your existing Contributor Licensing > Agreement: https://bitbucket.org/vanl/pypi/src/default/templates Okay, so there was recent progress here. This is actual news to me. Do you remember what kind of derision I went through for just suggesting something like that a couple of years ago? Yes, you supported me, but what about the others? Or that Van Lindberg was in the merry group of Twitter heroes who were gloating about the fact that (in their opinion) I could not do anything about the first hostile fork? Technically, he didn't gloat, but suddenly legal advice was apparently readily available. https://pypi.python.org/pypi/m3-cdecimal I can assure you that CoCs or "diversity statements" won't help you at all. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yearly PyPI breakage
Chris Barker noaa.gov> writes: > Indeed -- Fredrik never made any effort to support pypi, pip, etc. -- that's why the Pillow fork was started in the first place. Maybe, but he created PIL, so thank you, Fredrik! Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yearly PyPI breakage
Donald Stufft stufft.io> writes: > > Technically, he > > didn't gloat, but suddenly legal advice was apparently readily available. > > > > https://pypi.python.org/pypi/m3-cdecimal > > > > I'm not going to respond to the rest of this, because I don't think it's > possible for you and me to have a constructive discussion on these other > issues. However, I do want to point out that the PSF is the legal entity behind > PyPI itself and Van is both a board member and a lawyer. I don't recall what > exactly was discussed with who about m3-cdecimal, but it's completely > reasonable that when a legal question comes up with PyPI that we consult the > PSF board members, particularly the ones who are lawyers. It is not reasonable at all. A normal human being would try to consider first if an author has moral rights as opposed to legal rights and maybe pick up the phone or use private email instead of escalating everything to Twitter. Also, what was discussed was not whether the PSF had any right to remove m3-cdecimal but rather whether *I* had any rights to *demand* the removal. People concluded gleefully that I hadn't any (I still think they're mistaken and an enterprising Oracle lawyer could come to a different conclusion, but that's besides the point). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Yearly PyPI breakage
Brett Cannon python.org> writes: > This is whole thread is off-topic precisely because all of this is discussed -- in the open -- on distutils-sig and decided there. If people feel changes need to be made like broadcasting to a wider audience when a change occurs, then please bring it up on distutils-sig. But if people choose not to participate then they are implicitly delegating decision powers to those who do participate (which is totally fine and I'm not implying that if people don't participate they are doing something wrong). Participating there, especially for a non-native speaker, is basically a full-time job. You're met with a large number of extremely lengthy posts that would require an incredible amount of time to respond to in a careful manner. And ultimately it is the persons that Guido delegated the authority to who decide everything. I think many people have concluded that the influence/time ratio is too low to be worth it. Also I don't know any other development group who is a) that quick in trying to suppress any "off-topic" discussions and b) constantly uses venues outside of the Python mailing lists to steer and manipulate public opinion. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19330: Handle the no-docstrings case in tests
nick.coghlan wrote:
> http://hg.python.org/cpython/rev/a9bbc2d0c1dc
> -HAVE_DOCSTRINGS = (check_impl_detail(cpython=False) or
> - sys.platform == 'win32' or
> - sysconfig.get_config_var('WITH_DOC_STRINGS'))
> +# Rather than trying to enumerate all the cases where docstrings may be
> +# disabled, we just check for that directly
> +
> +def _check_docstrings():
> +"""Just used to check if docstrings are enabled"""
> +
> +HAVE_DOCSTRINGS = (_check_docstrings.__doc__ is not None)
>
> requires_docstrings = unittest.skipUnless(HAVE_DOCSTRINGS,
I think that does not detect --without-doc-strings (i.e. the C docstrings are
empty).
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 454 - tracemalloc - accepted
Victor Stinner wrote: > 2013/11/21 Nick Coghlan : > > Huzzah! Thanks to you both for getting this ready for inclusion :) > > I now hope that someone will use it :-) Congratulations! I hope pyfailmalloc can go into 3.4, too. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [buildbot] Increase number of stored builds
Hello, is it possible to increase the number of stored builds? Sometimes I want to see a build that might be a week or even 6 months old. Examples: 1) Recently I was wondering if the HPUX CFLAGS changed at some point. 2) The HPUX build has the following error: "*** WARNING: renaming "_curses" since importing it failed: dynamic module does not define init function (PyInit__curses)" I need to see if http://hg.python.org/cpython/rev/97fb852c5c26 is responsible. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] How do we feel about using pragmas and __attribute__ in C code?
Brett Cannon wrote: > So is there any reason to not use pragmas sparsely in the code? I'm +1 on using pragmas for suppressing selected warnings. > Tying into this and using compiler-specific things in C code, what about > __attribute__? http://bugs.python.org/issue19298 proposes an idea that Daniel > Stutzbach originally came up with where we could use __atribute__ (behind a > nicer macro) to help detect refleaks on PyObject* stack variables. Would > __attribute__ usage be okay in that situation? I would have to see an example function, mostly to get an idea of how readable the code is. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cffi in stdlib
Maciej Fijalkowski wrote: > I would like to discuss on the language summit a potential inclusion > of cffi[1] into stdlib. This is a project Armin Rigo has been working > for a while, with some input from other developers. I've tried cffi (admittedly only in a toy script) and find it very nice to use. Here's a comparison (pi benchmark) between wrapping libmpdec using a C-extension (_decimal), cffi and ctypes: +---+--+--+-+ | | _decimal | ctypes | cffi | +===+==+==+=+ | cpython-tip (with-system-ffi) | 0.19s | 5.40s | 5.14s | +---+--+--+-+ | cpython-2.7 (with-system-ffi) |n/a | 4.46s | 5.18s | +---+--+--+-+ | Ubuntu-cpython-2.7 |n/a | 3.63s |-| +---+--+--+-+ | pypy-2.2.1-linux64 |n/a | 125.9s | 0.94s | +---+--+--+-+ | pypy3-2.1-beta1-linux64 |n/a | 264.9s | 2.93s | +---+--+--+-+ I guess the key points are that C-extensions are hard to beat and that cffi performance on pypy-2 is outstanding. Additionally it's worth noting that Ubuntu does something in their Python build that we should do, too. +1 for cffi in the stdlib. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cffi in stdlib
Gregory P. Smith wrote: > Ubuntu compiles their Python with FDO (feedback directed optimization / > profile > guided optimization) enabled. All distros should do this if they don't > already. > It's generally 20% interpreter speedup. Our makefile already supports it but > it > isn't the default build as it takes a long time given that it needs to compile > everything twice and do a profiled benchmark run between compilations. Yes, I didn't know we already had `make profile-opt`. With that option the self-compiled results are nearly the same as with the Ubuntu version, the remaining difference might be due to Ubuntu's use of -flto, as Matthias suggests in http://bugs.python.org/issue17781 . Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19976: Argument Clinic METH_NOARGS functions now always
Probably Rietveld did not send mail, so I mention my review comments again: larry.hastings wrote: > +#ifdef __GNUC__ > +#define Py_UNUSED(name) _unused_ ## name __attribute__((unused)) > +#else > +#define Py_UNUSED(name) _unused_ ## name > +#endif > + The Intel compiler defines __GNUC__ but chokes on the __attribute__(). This works: #if defined(__GNUC__) && !defined(__INTEL_COMPILER) > +_pickle_Pickler_clear_memo(PyObject *self, PyObject *Py_UNUSED(ignored)) I'm not a native speaker, but UNUSED(ignored) reads strange to me. I would prefer UNUSED(args). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #19976: Argument Clinic METH_NOARGS functions now always
Stefan Behnel wrote: > """ > #ifndef CYTHON_UNUSED > # if defined(__GNUC__) > # if !(defined(__cplusplus)) || (__GNUC__ > 3 || (__GNUC__ == 3 && > __GNUC_MINOR__ >= 4)) > # define CYTHON_UNUSED __attribute__ ((__unused__)) > # else > # define CYTHON_UNUSED > # endif > # elif defined(__ICC) || (defined(__INTEL_COMPILER) && !defined(_MSC_VER)) > # define CYTHON_UNUSED __attribute__ ((__unused__)) > # else > # define CYTHON_UNUSED > # endif > #endif > """ > > I wonder why this works, though, given that you say Intel doesn't support > "__attribute__". The only difference I can spot is the space behind it. You're right, icc version 12.0 supports the attribute. It must have been some earlier version that failed. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [RELEASE] libmpdec-2.4.0 (suitable for compiling _decimal)
Hi, I've released libmpdec-2.4.0, which can be used to compile _decimal with the "--with-system-libmpdec" configure option: http://www.bytereef.org/mpdecimal/download.html libmpdec-2.4.0 is exactly the same as the one in the CPython source tree. The API is stable and the libmpdec-2.x.y series will stay binary compatible [1]. Stefan Krah [1] Starting from 2.4.0: 2.4.0 is not binary compatible with 2.3. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposed: The Great Argument Clinic Conversion Derby
Nick Coghlan wrote: > > I had that working at one point. Guido said no, keep it all in one file. > > I'm flexible but first you'd have to convince him. > > It's also not something we're stuck with forever - we can start with it inline > (which has the advantage of keeping all the code in the same place), and later > move to having the helpers in a separate file included from the implementation > file if we decide it makes sense to do so. If we move big chunks of code around twice, I guess "hg blame" will break twice, too. That is another thing worth considering. I agree with Serhiy, but that is probably known at this point. :) Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] RFC: PEP 460: Add bytes % args and bytes.format(args) to Python 3.5
Antoine Pitrou wrote: > > Very nice, thanks. If I was to make a blasphemous suggestion I would > > even target it for Python 3.4. (No, seriously, this is a big issue > > - see the recent discussion by Armin - and the big names involved show > > that it is a major holdup of 3.x uptake.) It would of course depend > > a lot on how much code from unicode formatting can be retained or > > adapted as opposed to a rewrite from scratch. > > From what I've seen of the unicode formatting code, a lot would have to > be rewritten or refactored. It is a non-trivial task, definitely > inappropriate for 3.4. I do not know the stringlib well enough, so I have a silly question: Would it be possible to re-use the 2.x stringlib just for the bytes type, name it byteslib and disable features as appropriate? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Changing Clinic's output
Antoine Pitrou wrote: > Several solutions have been proposed: > - move all generated code to separate C files, which would then be > #included'd into the main module file +1 for the reasons that Serhiy has listed. Additionally, if custom parsers are implemented, the generated code will take up even more space (look e.g. at Cython's custom parsers). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Changing Clinic's output
Brett Cannon wrote: > I personally don't care about this whole discussion (and I suspect people > being > quiet don't either). At this point the amount of arguing on this topic could > have been used more constructively converting code and then, if necessary, > tweaking the output of Argument Clinic later. Serhiy, who started the discussion in another thread, is converting modules at a rapid pace. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python3 "complexity"
Nick Coghlan wrote: > One idea we're considering for Python 3.5 is to have a report of > "ascii" on a POSIX OS imply the surrogateescape error handler (at > least for the standard streams, and perhaps in other contexts), since > the OS reporting the POSIX/C locale almost certainly indicates a > configuration error rather than intentional behaviour. On FreeBSD users apparently get the C locale by default. I don't think I've configured anything special during the install: freebsd-amd64# adduser Username: testuser Full name: Uid (Leave empty for default): Login group [testuser]: Login group is testuser. Invite testuser into other groups? []: Login class [default]: Shell (sh csh tcsh bash rbash nologin) [sh]: Home directory [/home/testuser]: Home directory permissions (Leave empty for default): Use password-based authentication? [yes]: no Lock out the account after creation? [no]: Username : testuser Password : Full Name : Uid: 1003 Class : Groups : testuser Home : /home/testuser Home Mode : Shell : /bin/sh Locked : no OK? (yes/no): yes adduser: INFO: Successfully added (testuser) to the user database. Add another user? (yes/no): no Goodbye! freebsd-amd64# su - testuser $ locale LANG= LC_CTYPE="C" LC_COLLATE="C" LC_TIME="C" LC_NUMERIC="C" LC_MONETARY="C" LC_MESSAGES="C" LC_ALL= Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Changing Clinic's output
Larry Hastings wrote: > https://bitbucket.org/larry/clinic-buffer-samples/src Thanks for doing this! +1 for the sidefile, -1 for the current approach, +-0 for the rest. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .clinic.c vs .c.clinic
Serhiy Storchaka wrote: > Now generated files have suffixes .clinic.c. I think it will be better, if > they > will end at special suffix (.c.clinic or even just .clinic). Can the output not go into a header file with static inline functions? I'd rather see memoryview.h than memoryview.clinic.c. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .clinic.c vs .c.clinic
> I'd rather see memoryview.h than memoryview.clinic.c. Or, if this collides with Include/*, one of the following: memoryview_func.h // public functions memoryview_if.h // public interface Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .clinic.c vs .c.clinic
Antoine Pitrou wrote: > Objects/memoryview.clinic.h should be fine. Last attempt: Objects/memoryview.api.h That is more neutral and describes what the file contains. IOW, it's easier to ignore the name (which is good in this case). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .clinic.c vs .c.clinic
Antoine Pitrou wrote: > > Objects/memoryview.api.h > > > > > > That is more neutral and describes what the file contains. > > Disagreed. It's not an API in the sense that it's something that's > designed to be called directly by third-party code. Right. Objects/memoryview.ac.h perhaps? I sort of dislike reading full words in filename extensions. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .clinic.c vs .c.clinic
Serhiy Storchaka wrote: > .ac is well known suffix of autoconf related files. I know, but unless someone writes Objects/configure.c I think this won't be a problem. > And tail .h has same disadvantages as .c. I'm not strongly inconvenienced by those you listed. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .clinic.c vs .c.clinic
Larry Hastings wrote: > Contestant 4: "Put in clinic directory, add .h" > > foo.c -> clinic/foo.c.h > foo.h -> clinic/foo.h.h +1 for this, 0 for the rest. Bonus points for any other directory name that is more self-descriptive. ;) Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] .clinic.c vs .c.clinic
Stefan Krah wrote: > Larry Hastings wrote: > > Contestant 4: "Put in clinic directory, add .h" > > > > foo.c -> clinic/foo.c.h > > foo.h -> clinic/foo.h.h > > +1 for this, 0 for the rest. Bonus points for any other directory name that > is > more self-descriptive. ;) On second thought, I do find that having Modules/cjkcodecs Modules/clinic looks kind of weird. So I'm +1 for __clinic__, 0 on the rest. +2 for something more self-explanatory like: Modules/__arghandlers__ Modules/__autogen__ Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The docstring hack for signature information has to go
Larry Hastings wrote:
> So here's the problem. Let's say you want to write an extension that will
> work
> with Python 3.3 and 3.4, using the stable ABI. If you don't add this line,
> then in 3.4 you won't have introspection information, drat. But if you *do*
> add this line, your docstring will look mildly stupid in 3.3, because it'll
> have this unsightly "sig=(" line at the top. And it *won't* have a nice
> handwritten docstring. (And if you added both a sig= signature *and* a
> handwritten signature, in 3.4 it would display both. That would also look
> dumb.)
I think we may slowly get into PEP territory here. Just imagine that
we settle on X, then decide at a later point to have a standard way of
adding type annotations, then find that X does not work because of (unknown).
I'm mentioning this because signatures get really interesting for me
if they contain type information.
Stefan Krah
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Start writing inlines rather than macros?
Skip Montanaro wrote: > I do have one question though. Suppose you encounter a compiler that > doesn't understand the inline keyword, so you choose the static > declaration as Kristján suggested. The resulting Python executable > should be functionally correct, but if the optimizer doesn't happen to > inline a given static function you might be stuck with some bad > performance across-the-board (if it never inlines, or doesn't inline > where we really need it to), or only under some circumstances (as a > hypothetical example, inlining in dictobject.c, but not in ceval.c). > Is there a configurable way to tell if a compiler will inline > functions which are declared static, and possibly under what > conditions they might not? It might still be necessary to maintain > macros for those platforms. I think that all modern compilers can handle "static inline" in header files. If you have a compiler that cannot, chances are that the platform is horribly outdated and this particular performance issue will be relatively benign compared to other ones. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] freeze build slave
"Martin v. L?wis" wrote: > C: pro: compared to B, build time is reduced (need only >to build once per branch); disk space is also reduced >con: it would test a debug build, not a release build It would be an option to run half of the Unix slaves (especially the ones with the more aggressive compilers) in release mode. I think that is beneficial anyway. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] ref leaks
Ethan Furman wrote: > >>Any words of wisdom for tracking those leaks? Often the easiest way is to compile --with-valgrind and run the test under Valgrind (obviously). In the Valgrind output, search for "definitely lost" and ignore the rest. If that does not turn up anything, consider a bug in regrtest.py: It happens that certain caches are initialized incrementally in each repetition of the test, so the reported leaks are spurious. That is why some caches like small integers are "warmed up" in regrtest.py. Perhaps there is some recently added cache that needs attention. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 2.7.7. on Windows
Mike Miller wrote: > I have to say I'm a bit baffled. I expected disagreement, but > didn't expect that multiple reasons against would be made up > seemingly at random? I and a company I work for (that distributes > Py) have been installing Python to ProgramFiles for almost a decade, > and can assure that none of those things you mention have yet > happened. Relax, I don't think Steve is making things up. That said, I can confirm what you wrote: I've always installed Python to "Program Files" and I've never had any issues (then again, I'm mostly using Linux). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Just a warning, in case any of the new packaging team forgot to contact http://cve.mitre.org/ . Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Victor Stinner wrote: > I don't understand your email. Can you please elaborate? There is nothing wrong with the package. The remark is a joke provoked by a long history of a campaign [1] against external packages on distutils-sig. Many tools (like crate.io, when it was still up) have made derogatory remarks about external packages. Now the latest version of the officially sanctioned download tool (pip) spits out copious warnings, one of which is the subject of this thread. External packages are being singled out unfairly: 1) Anyone can upload any package to PyPI (i.e. the index is not curated at all). 2) Last time I looked, access credentials (via "lost login") were sent out in plaintext. 3) AFAIK people can upload a different (malicious) version of a package with *the exact same name*. 4) pip generally downloads the latest version, so a malicious person can provide a good package for several years until people trust him, then change to a trojaned version. 5) Looking at the list of certificates that is in my default cert store, I don't find SSL trustworthy at all. 6) D.J. Bernstein, who is somewhat security minded, has been shipping his software *for years* with just plain HTTP and published checksums. To sum it up: 1) Don't use pip to install packages directly from PyPI if security really matters. 2) The best security we currently get is either a) with package signatures (*if* you can get the author's key via a trustworthy channel, which is rarely the case). b) with decent checksums that are recorded on public mailing lists at the time the package is announced (it would be hard for an attacker to modify all mailing list archives.) Whether a package is internal or external is orthogonal to both points. With all these points, I find it questionable for an "official" install tool to make security related remarks about just one category of weaknesses. After all, people might be led to believe that pip is some sort of apt-get and all uploaded packages are safe. Stefan Krah [1] Note that the joke is quite innocent in comparison to what I've read on distutils-sig about the subject. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > There is support for trusted externally hosted packages, you put the URL in > PyPI and include a hash in the fragment like so: > > http://www.bytereef.org/software/mpdecimal/releases/cdecimal-2.3.tar.gz#md5=655f9fd72f7a21688f903900ebea6f56 That is exactly the mode I was using until today. This mode produced the subject's warning message. Today I've switched to manual install mode with manual sha256sum verification which is *far* safer than anything you get via pip right now. > [2] For the definition of safe that PyPI/pip operate under, which is that the > author of a package is assumed to be trusted by the person electing to > download their package. No, there are other holes, which you have conceded in your previous mail. > I don't think the warning is FUD, and it doesn't mention anything security > related at all. The exact text of the warning is in the subject of the email > here: > > cdecimal an externally hosted file and may be unreliable > > Which is true as far as I can tell, it is externally hosted, and it may be > unreliable[1]. If there is a better wording for that I?m happy to have it and > will gladly commit it myself to pip. Do you honestly not see a difference between the cited warning and the *intended* warning "the server's availability may be unreliable"? Even the latter is FUD or a truism (it applies to any server). The real question is: Why is there a warning if the person running pip has explicitly allowed external packages? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > hosted packages are brittle and more prone to failure. Every single external > server adds *another* SPOF into any particular install set. Even if every > external server has a 99.9% uptime, when you combine multiple of them the > total > uptime of any particular set of requirements drops quickly. This has been a > major complaint that people have had over time. We have been through that many times; to me this is an indication that people are using pip under circumstances when they should not. pip is not apt-get. [I am aware that *you* know that, just stating it again for the broader audience.] > > 2) Last time I looked, access credentials (via "lost login") were sent > > out in plaintext. > > The existence of other security issues is not an excuse to not fix a security > issue. There are other problems and we're slowly working on trying to clear > them out. It is, however, a reason to avoid error messages that could *imply* (rightly or wrongly) to users that the combination of pip and internal packages is safe. > > 3) AFAIK people can upload a different (malicious) version of a > > package with *the exact same name*. > > Yes, a malicious author is needfully outside of the threat model for PyPI/pip. How so? I'm avoiding this attack by publishing sha256sums at release time. The point is that I *cannot* change cdecimal-2.3.tar.gz without a user digging up a checksum mismatch from the mailing list archives. > > 6) D.J. Bernstein, who is somewhat security minded, has been shipping > > his software *for years* with just plain HTTP and published checksums. > > Argument from authority doesn't really hold up very well when DJB doesn't > distribute is software in a way that is intended to be consumed mechanically. > Also while he may be a crypto expert he is far from an expert on successfully > distributing software, unless you also think that the signature checking in > most OS provided package managers is pointless. That is sort of a strawman. The whole point *is* that certain distribution models don't lend themselves to mechanical consumption. I cannot speak for DJB, perhaps he is just thinking that GPG signing is pointless if users can't validate the signature and SSL is pointless because one cannot trust governments. OS package signing is useful since the packages are curated. If anyone could upload a package to Debian, whereupon it would be signed with the official key, apt-get would lose its usefulness. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > > Today I've switched to manual install mode with manual sha256sum > > verification > > which is *far* safer than anything you get via pip right now. > > It is not safer in any meaingful way. > > If someone is in a position to compromise the integrity of PyPI's TLS, they > can replace the hash on that page with something else. Now you've attempted to > work around this by telling people to go look up the release announcement > hash. However if someone can compromise the integrity of PyPI's TLS, they can > also compromise the integrity of https://mail.python.org/, or GMane, or any > other TLS based website[1]. Of course it is safer. Suppose a file is stored on PyPI: 1) Attacker guesses my username (or is it even visible, I'm not sure). 2) Clicks on "lost login". 3) Intercepts mail (difficult, but far from the TLS attack category). Maybe on a home or university network. Or a rogue person at a mail provider. 4) Changes the uploaded file together with the hash. pip would be perfectly happy, checking the hash via Google would turn up a mismatch. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > I said ?meaningful?. Almost nobody is going to ever bother googling it and > the likelihood that someone is able to MITM *you* specifically is far lesser > than the likelihood that someone is going to MITM one of the cdecimal users. I'm doing this for important installs. -- That is how I installed qmail and djbdns. > Additionally your messages aren?t signed and email isn?t an authenticated > profile so if someone was able to get your password they could simply spoof > and email from you to the mailing list with new hashes, or edit out the > description > telling people to go google some stuff. Signing messages is pointless if the key isn't well connected. Also, I'm reading the lists and would notice a "release". Most importantly, the checksum mismatch would still be found, since the old messages with the correct sum would still exist under the scenario we're talking about (i.e. not GHCQ hacking into Belgacom routers). Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald Stufft wrote: > I?m unsure if you?re being willfully dense or if you?re just not understanding > what I mean when I say ?almost?. Of course there are going to be a few > outliers > where people do bother to do that, but it?s not going to be common place at > all. I suggest that you confine that style of discussion to distutils-sig. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Paul Moore wrote: > You're claiming that Mercurial moved to hosting on PyPI solely because > users suddenly needed to add a flag to install from pip? As opposed to > because PyPI gave them a more reliable hosting platform, for example? > OK. I certainly can't give any evidence to dispute that claim, > although I'm surprised. That is my understanding of the posted statistics, too. PyPI was overloaded, the improved infrastructure fixed that, so more projects now host on PyPI. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]
Donald, I'm out of his discussion. I have one last request: please don't gossip about core devs in public as long as you have commit privs: https://botbot.me/freenode/python-requests/ Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Internal representation of strings and Micropython
Paul Sokolovsky wrote: > In this regard, I'm glad to participate in mind-resetting discussion. > So, let's reiterate - there's nothing like "the best", "the only right", > "the only correct", "righter than", "more correct than" in CPython's > implementation of Unicode storage. It is *arbitrary*. Well, sure, it's > not arbitrary, but based on requirements, and these requirements match > CPython's (implied) usage model well enough. But among all possible > sets of requirements, CPython's requirements are no more valid that > other possible. And other set of requirement fairly clearly lead to > situation where CPython implementation is rejected as not correct for > those requirements at all. Several core-devs have said that using UTF-8 for MicroPython is perfectly okay. I also think it's the right choice and I hope that you guys come up with a very efficient implementation. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
dw+python-...@hmmz.org wrote: > * Has Python ever hit a showstopper release issue as a result of a bug > in MSVC? (I guess probably not). Yes, a PGO issue: http://bugs.python.org/issue15993 To be fair, in that issue I did not look if there's some undefined behavior in longobject.c. > * Will VS 14 be golden prior to Python 3.5's release? It would suck to > rely on a beta compiler.. :) This is my only concern, too. Otherwise, +1 for the switch. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Moving Python 3.5 on Windows to a new compiler
Stefan Krah wrote: > > * Will VS 14 be golden prior to Python 3.5's release? It would suck to > > rely on a beta compiler.. :) > > This is my only concern, too. Otherwise, +1 for the switch. One more thing: Will the SDK 64-bit tools be available for the Express Versions? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Tone it down on Twitter?
Apparently I have made it into "club of three who don't care much about opinions of others" for the crime of a single +0.5 for PEP-572 without participating in the discussion at all (neither did Jason). https://twitter.com/SerhiyStorchaka/status/1014274554452209665 This is a new high for Twitter gossip. Well done. Perhaps in the next vote the politbureau can indicate the intended outcome beforehand so we know how to vote. Thanks, Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
Victor Stinner wrote: > Moreover, I failed to find anyone who can explain me how the C API is used > in the wild, which functions are important or not, what is the C API, etc. In practice people desperately *have* to use whatever is there, including functions with underscores that are not even officially in the C-API. I have to use _PyFloat_Pack* in order to be compatible with CPython, I need PySlice_Unpack() etc., I need PyUnicode_KIND(), need PyUnicode_AsUTF8AndSize(), I *wish* there were PyUnicode_AsAsciiAndSize(). In general, in daily use of the C-API I wish it were *larger* and not smaller. I often want functions that return C instead of Python values ot functions that take C instead of Python values. The ideal situation for me would be a lower layer library, say libcpython.a that has all those functions like _PyFloat_Pack*. It would be an enormous amount of work though, especially since the status quo kind of works. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Experiment an opt-in new C API for Python? (leave current API unchanged)
On Mon, Nov 19, 2018 at 04:08:07PM +0100, Victor Stinner wrote: > Le lun. 19 nov. 2018 à 13:18, Stefan Krah a écrit : > > In practice people desperately *have* to use whatever is there, including > > functions with underscores that are not even officially in the C-API. > > > > I have to use _PyFloat_Pack* in order to be compatible with CPython, > > Oh, I never used this function. These functions are private (name > prefixed by "_") and excluded from the limited API. > > For me, the limited API should be functions available on all Python > implementations. Does it make sense to provide PyFloat_Pack4() in > MicroPython, Jython, IronPython and PyPy? Or is it something more > specific to CPython? I don't know the answer. If yes, open an issue to > propose to make this function public? It depends on what the goal is: If PyPy wants to be able to use as many C extensions as possible, then yes. The function is just one example of what people have to use to be 100% compatible with CPython (or copy these functions and maintain them ...). Intuitively, it should probably not be part of a limited API, but I never quite understood the purpose of this API, because I regularly need any function that I can get my hands on. > > I need PyUnicode_KIND() > > IMHO this one should not be part of the public API. The only usage > would be to micro-optimize, but such API is very specific to one > Python implementation. For example, PyPy doesn't use "compact string" > but UTF-8 internally. If you use PyUnicode_KIND(), your code becomes > incompatible with PyPy. > > What is your use case? Reading typed strings directly into an array with minimal overhead. > I would prefer to expose the "_PyUnicodeWriter" API than PyUnicode_KIND(). > > > need PyUnicode_AsUTF8AndSize(), > > Again, that's a micro-optimization and it's very specific to CPython: > result cached in the "immutable" str object. I don't want to put it in > a public API. PyUnicode_AsUTF8String() is better since it doesn't > require an internal cache. > > > I *wish* there were PyUnicode_AsAsciiAndSize(). > > PyUnicode_AsASCIIString() looks good to me. Sadly, it doesn't return > the length, but usually the length is not needed. Yes, these are all just examples. It's also very useful to be able to do PyLong_Type.tp_as_number->nb_multiply or grab as_integer_ratio from the float PyMethodDef. The latter two cases are for speed reasons but also because sometimes you *don't* want a method from a subclass (Serhiy was very good in finding corner cases :-). Most C modules that I've seen have some internals. Psycopg2: PyDateTime_DELTA_GET_MICROSECONDS PyDateTime_DELTA_GET_DAYS PyDateTime_DELTA_GET_SECONDS PyList_GET_ITEM Bytes_GET_SIZE Py_BEGIN_ALLOW_THREADS Py_END_ALLOW_THREADS floatobject.h and longintrepr.h are also popular. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] C API changes
Armin Rigo wrote: > The C API would change a lot, so it's not reasonable to do that in the > CPython repo. But it could be a third-party project, attempting to > define an API like this and implement it well on top of both CPython > and PyPy. IMHO this might be a better idea than just changing the API > of functions defined long ago to make them more regular (e.g. stop > returning borrowed references); by now this would mostly mean creating > more work for the PyPy team to track and adapt to the changes, with no > real benefits. I like this idea. For example, when writing two versions of a C module, one that uses CPython internals indiscriminately and another that uses a "clean" API, such a third-party project would help. I'd also be more motivated to write two versions if I know that the project is supported by PyPy devs. Do you think that such an API might be faster than CFFI on PyPy? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] C API changes
Steve Dower wrote: > My proposed marketing pitch is: "rewrite your existing code to be > forward-compatible today and faster in the future without more work, or > be prepared to rewrite/update your source code for each CPython release > to remain compatible with the low level API". The promise of "faster in > the future" needs to be justified (and I think there's plenty of > precedent in PyPy, Larry's Gilectomy and the various JavaScript VMs to > assume that we can do it). It's hard to discuss this in the abstract without knowing how big the breakage between each version is going to be. But for the scientific ecosystem this sounds a bit like a potential Python-4.0 breakage, which was universally rejected (so far). In the extreme case I can imagine people staying on 3.7. But it really depends on the nature of the changes. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Making PyInterpreterState an opaque type
Victor Stinner wrote: > Premature optimization is the root of all evil. Most C extensions use > premature optimization which are causing us a lot of troubles nowadays > when we want to make the C API evolve and cause issues to PyPy which > has issues to reimplement the C API on top of their different object > model with a different GC. I'm intimately familiar with several C extensions in the math/science space and this is not the impression I get at all. Most people are happy if they get things to work, because the actual problems are much harder than focusing on private vs. public API. As for PyPy, if I understood correctly, Armin Rigo was skeptical of the proposed plan and favored publishing an API as a third party package. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Point of building without threads?
Yury V. Zaytsev wrote: > On Mon, 2012-05-07 at 21:49 +0200, Antoine Pitrou wrote: > > > > I guess a long time ago, threading support in operating systems wasn't > > very widespread, but these days all our supported platforms have it. > > Is it still useful for production purposes to configure > > --without-threads? Do people use this option for something else than > > curiosity of mind? > > I hope that the intent behind asking this question was more of being > curious, rather then considering dropping --without-threads: > unfortunately, multithreading was, still is and probably will remain > troublesome on many supercomputing platforms. > > Often, once a new supercomputer is launched, as a developer you get a > half-baked C/C++ compiler with threading support broken to the point > when it's much easier to not use it altogether [*] rather than trying to > work around the compiler quirks. Out of curiosity: Do these incomplete compilers have any problem with either stdint.h or static inline functions in header files? Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing "Argument Clinic", a new way of specifying arguments to builtins for CPython
Stefan Behnel wrote: > > you'd just pass in this string: > > > >(arg1 : int, arg2 : float, arg3 : ExtType, *, arg4=None) -> ExtType2 I've mentioned this proposal in http://bugs.python.org/issue16612 , but it wasn't sufficient for the task. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] peps: Pre-alpha draft for PEP 435 (enum). The name is not important at the moment, as
eli.bendersky wrote: > +Ordered comparisons between enumeration values are *not* supported. Enums > are > +not integers! Hmm. I think this limits interoperation with C libraries and prototyping C code. Actually all I want from a Python enum is to be like a C enum with symbolic representations for the values. Stefan Krah ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
