Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

[Glenn Linderman]

I'm generally +1, with opinions noted below on these two topics.

On 6/26/2014 3:59 PM, Ben Hoyt wrote:

Should there be a way to access the full path?
--

Should ``DirEntry``'s have a way to get the full path without using
``os.path.join(path, entry.name)``? This is a pretty common pattern,
and it may be useful to add pathlib-like ``str(entry)`` functionality.
This functionality has also been requested in `issue 13`_ on GitHub.

.. _`issue 13`:https://github.com/benhoyt/scandir/issues/13


+1


Should it expose Windows wildcard functionality?


Should ``scandir()`` have a way of exposing the wildcard functionality
in the Windows ``FindFirstFile`` / ``FindNextFile`` functions? The
scandir module on GitHub exposes this as a ``windows_wildcard``
keyword argument, allowing Windows power users the option to pass a
custom wildcard to ``FindFirstFile``, which may avoid the need to use
``fnmatch`` or similar on the resulting names. It is named the
unwieldly ``windows_wildcard`` to remind you you're writing power-
user, Windows-only code if you use it.

This boils down to whether ``scandir`` should be about exposing all of
the system's directory iteration features, or simply providing a fast,
simple, cross-platform directory iteration API.

This PEP's author votes for not including ``windows_wildcard`` in the
standard library version, because even though it could be useful in
rare cases (say the Windows Dropbox client?), it'd be too easy to use
it just because you're a Windows developer, and create code that is
not cross-platform.


Because another common pattern is to check for name matches pattern, I 
think it would be good to have a feature that provides such. I do that 
in my own private directory listing extensions, and also some command 
lines expose it to the user.  Where exposed to the user, I use -p 
windows-pattern and -P regexp. My implementation converts the 
windows-pattern to a regexp, and then uses common code, but for this 
particular API, because the windows_wildcard can be optimized by the 
window API call used, it would make more sense to pass windows_wildcard 
directly to FindFirst on Windows, but on *nix convert it to a regexp. 
Both Windows and *nix would call re to process pattern matches except 
for the case on Windows of having a Windows pattern passed in. The 
alternate parameter could simply be called wildcard, and would be a 
regexp. If desired, other flavors of wildcard bsd_wildcard? could also 
be implemented, but I'm not sure there are any benefits to them, as 
there are, as far as I am aware, no optimizations for those patterns in 
those systems.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

[Glenn Linderman]

On 6/29/2014 5:28 AM, Nick Coghlan wrote:

There'd still be a slight window of discrepancy (since the filesystem
state may change between reading the directory entry and making the
lstat() call), but this could be effectively eliminated from the
perspective of the Python code by making the result of the lstat()
call authoritative for the whole DirEntry object.


+1 to this in particular, but this whole refresh of the semantics sounds 
better overall.


Finally, for the case where someone does want to keep the DirEntry 
around, a .refresh() API could rerun lstat() and update all the data.


And with that (initial data potentially always populated, or None, and 
an explicit refresh() API), the data could all be returned as 
properties, implying that they aren't fetching new data themselves, 
because they wouldn't be.


Glenn
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 -- os.scandir() function -- a better and faster directory iterator

[Glenn Linderman]

On 6/30/2014 4:25 PM, Devin Jeanpierre wrote:

On Mon, Jun 30, 2014 at 3:07 PM, Tim Delaney
 wrote:

On 1 July 2014 03:05, Ben Hoyt  wrote:

So, here's my alternative proposal: add an "ensure_lstat" flag to
scandir() itself, and don't have *any* methods on DirEntry, only
attributes.

...


Most importantly, *regardless of platform*, the cached stat result (if
not None) would reflect the state of the entry at the time the
directory was scanned, rather than at some arbitrary later point in
time when lstat() was first called on the DirEntry object.


I'm torn between whether I'd prefer the stat fields to be populated on
Windows if ensure_lstat=False or not. There are good arguments each way, but
overall I'm inclining towards having it consistent with POSIX - don't
populate them unless ensure_lstat=True.

+0 for stat fields to be None on all platforms unless ensure_lstat=True.

This won't work well if lstat info is only needed for some entries. Is
that a common use-case? It was mentioned earlier in the thread.


If it is, use ensure_lstat=False, and use the proposed (by me) 
.refresh() API to update the data for those that need it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My summary of the scandir (PEP 471)

[Glenn Linderman]

On 7/1/2014 2:20 PM, Paul Moore wrote:

Please, let's stick to a low-level wrapper round the OS API for the
first iteration of this feature. Enhancements can be added later, when
real-world usage has proved their value.


I almost wrote this whole message this morning, but didn't have time.  
Thanks, Paul, for digging through the details.


+1
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Exposing the Android platform existence to Python modules

[Glenn Linderman]

On 7/31/2014 5:59 PM, Shiz wrote:

Hi folks,

I’m working on porting CPython to the Android platform, and while making decent 
progress, I’m currently stuck at a higher-level issue than adding #ifdefs for 
__ANDROID__ to C extension modules.

The idea is, not only CPython extension modules have some assumptions that 
don’t seem to fit Android’s mold, some default Python-written modules do as 
well. However, whereas CPython extensions can trivially check if we’re building 
for Android by checking the __ANDROID__ compiler macro, Python modules can do 
no such check, and are left wondering how to figure out if the platform they 
are currently running on is an Android one. To my knowledge there is no 
reliable way to detect if one is using Android as a vehicle for their journey 
using any other way.

Now, the main question is: what would be the best way to ‘expose’ the 
indication that Android is being ran on to Python-living modules? My own 
thought was to add sys.getlinuxuserland(), or platform.linux_userland(), in 
similar vein to sys.getwindowsversion() and platform.linux_distribution(), 
which could return information about the userland of running CPython instance, 
instead of knowing merely the kernel and the distribution.


I've no idea what you mean by "userland" in your suggestions above or 
below, but doesn't the Android environment qualify as a 
(multi-versioned) platform independently of its host OS? Seems I've read 
about an Android reimplementation for Windows, for example. As long as 
all the services expected by Android are faithfully produced, the host 
OS may be irrelevant to an Android application... in which case, I would 
think/propose/suggest the platform name should change from win32 or 
linux to Android (and the Android version be reflected in version parts).



This way, code could trivially check if it ran on the GNU(+associates) 
userland, or under a BSD-ish userland, or Android… and adjust its behaviour 
accordingly.

I would be delighted to hear comments on this proposal, or better yet, 
alternative solutions. :)

Kind regards,
Shiz

P.S.: I am well aware that Android might as well never be officially supported 
in CPython. In that case, consider this a thought experiment of how it /would/ 
be handled. :)


Is your P.S. suggestive that you would not be willing to support your 
port for use by others?  Of course, until it is somewhat complete, it is 
hard to know how complete and compatible it can be.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Surely "nullable" is a reasonable name?

[Glenn Linderman]

On 8/4/2014 12:35 AM, Stephen Hansen wrote:
On Mon, Aug 4, 2014 at 12:12 AM, Larry Hastings > wrote:



Several people have said they found the name "nullable"
surprising, suggesting I use another name like "allow_none" or
"noneable".  I, in turn, find their surprise surprising;
"nullable" is a term long associated with exactly this concept. 
It's used in C# and SQL, and the term even has its own Wikipedia page:



The thing is, "null" in these languages are not the same thing. If you 
look to the various database wrappers there's a lot of controversy 
about just how to map the SQL NULL to Python: simply mapping it to 
Python's None becomes strange because the semantics of a SQL NULL or 
NULL pointer and Python None don't exactly match. Not all that long 
ago someone was making an argument on this list to add a SQLNULL type 
object to better map SQL NULL semantics (regards to sorting, as I 
recall -- but its been awhile)


Python has None. Its definition and understanding in a Python context 
is clear. Why introduce some other concept? In Python its very common 
you pass None instead of an other argument.


Before you say "the term 'nullable' will confuse end users", let
me remind you: this is not user-facing.  This is a parameter for
an Argument Clinic converter, and will only ever be seen by
CPython core developers.  A group which I hope is not so easily
confused


Yet, my lurking observation of argument clinic is it is all about 
clearly defining the C-side of how things are done in Python API's. It 
may not confuse 'end users', but it may confuse possible contributors, 
and simply add a lack of clarity to the situation.


Passing None in place of another argument is a very Pythonic thing to 
do; why confuse that by using other words which imply other semantics? 
None is a Python thing with clear semantics in Python; allow_none 
quite accurately describes the Pythonic thing described here, while 
'nullable' expects for domain knowledge beyond Python and makes 
assumptions of semantics.


/re-lurk

--S


Thanks, Stephen.  +1 to all you wrote.

There remains, of course, one potential justification for using 
"nullable", that you didn't make 100% clear. Because "argument clinic is 
it is all about clearly defining the C-side of how things are done in 
Python API's." and that is that C uses NULL (but it is only a 
convention, not a language feature) for missing reference parameters on 
occasion. But I think it is much more clear that if C NULL gets mapped 
to Python None, and we are talking about Python parameters, then a 
NULLable C parameter should map to an "allow_none" Python parameter.


The concepts of C NULL, C# NULL, SQL NULL, and Python None are all 
slightly different, even the brilliant people on python-dev could better 
spend their energies on new features and bug fixes rather than being 
slowed by the need to remember yet another unclear and inconsistent 
terminology issue, of which there are already too many.


Glenn
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] sum(...) limitation

[Glenn Linderman]

On 8/10/2014 1:24 AM, Stephen J. Turnbull wrote:

Actually ... if I were a fan of the "".join() idiom, I'd seriously
propose 0.sum(numeric_iterable) as the RightThang{tm].  Then we could
deprecate "".join(string_iterable) in favor of "".sum(string_iterable)
(with the same efficient semantics).
Actually, there is no need to wait for 0.sum() to propose "".sum... but 
it is only a spelling change, so no real benefit.


Thinking about this more, maybe it should be a class function, so that 
it wouldn't require an instance:


str.sum( iterable_containing_strings )

[ or  str.join( iterable_containing_strings ) ]
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Glenn Linderman]

On 8/21/2014 3:42 PM, Paul Moore wrote:

I wonder how badly a Unix system would break if you specified UTF16 as
the system encoding...?
Paul


Does Unix even support UTF-16 as an encoding? I suppose, these days, it 
probably does, for reading contents of files created on Windows, etc. 
(Unicode was just gaining traction when I last used Unix in a 
significant manner; yes, my web host runs Linux, and I know enough to do 
what can be done there... but haven't experimented with encodings other 
than ASCII & UTF-8 on the web host, and don't intend to).


If it allows configuration of UTF-16 or UTF-32 as system encodings, I 
would consider that a bug, though, as too much of Unix predates Unicode, 
and would be likely to fail.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Glenn Linderman]

On 8/21/2014 3:54 PM, Antoine Pitrou wrote:


Le 21/08/2014 18:27, Cameron Simpson a écrit :

As
remarked, codes 0 (NUL) and 47 (ASCII slash code) _are_ special to UNIX
filename bytes strings.


So you admit that POSIX mandates that file paths are expressed in an 
ASCII-compatible encoding after all? Good. I've nothing to add to your 
rant.


Antoine.


0 and 47 are certainly originally derived from ASCII.  However, there 
could be lots of encodings that are not ASCII compatible (but in 
practice, probably very few, since most encodings _are_ ASCII 
compatible) that could be fit those constraints.


So while as a technical matter, Cameron is correct that Unix only treats 
0 & 47 as special, and that is insufficient to declare that encodings 
must be ASCII compatible, as a practical matter, since most encodings 
are ASCII compatible anyway, it would be hard to find very many that 
could be used successfully with Unix file names that are not ASCII 
compatible, that could comply with the 0 & 47 requirements.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Glenn Linderman]

On 8/22/2014 8:51 AM, Oleg Broytman wrote:

What encoding does have a text file (an HTML, to be precise) with
text in utf-8, ads in cp1251 (ad blocks were included from different
files) and comments in koi8-r?
Well, I must admit the HTML was rather an exception, but having a
text file with some strange characters (binary strings, or paragraphs
in different encodings) is not that exceptional.
That's not a text file. That's a binary file containing (hopefully 
delimited, and documented) sections of encoded text in different encodings.


If it is named .html and served by the server as UTF-8, then the server 
is misconfigured, or the file is incorrectly populated.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Glenn Linderman]

On 8/22/2014 9:52 AM, Oleg Broytman wrote:

On Fri, Aug 22, 2014 at 09:37:13AM -0700, Glenn Linderman 
 wrote:

On 8/22/2014 8:51 AM, Oleg Broytman wrote:

What encoding does have a text file (an HTML, to be precise) with
text in utf-8, ads in cp1251 (ad blocks were included from different
files) and comments in koi8-r?
Well, I must admit the HTML was rather an exception, but having a
text file with some strange characters (binary strings, or paragraphs
in different encodings) is not that exceptional.

That's not a text file. That's a binary file containing (hopefully
delimited, and documented) sections of encoded text in different
encodings.

Allow me to disagree. For me, this is a text file which I can (and
do) view with a pager, edit with a text editor, list on a console,
search with grep and so on. If it is not a text file by strict Python3
standards then these standards are too strict for me. Either I find a
simple workaround in Python3 to work with such texts or find a different
tool. I cannot avoid such files because my reality is much more complex
than strict text/binary dichotomy in Python3.

Oleg.


I was not declaring your file not to be a "text file" from any 
definition obtained from Python3 documentation, just from a common sense 
definition of "text file".


Looking at it from Python3, though, it is clear that when opening a file 
in "text" mode, an encoding may be specified or will be assumed.  That 
is one encoding, applying to the whole file, not 3 encodings, with 
declarations on when to switch between them. So I think, in general, 
Python3 assumes or defines a definition of text file that matches my 
"common sense" definition. Also, if it is an HTML file, I doubt the 
browser will use multiple different encodings when interpreting it, so 
it is not clear that the file is of practical use for its intended 
purpose if it contains text in multiple different encodings, but is 
served using only a single encoding, unless there is javascript or some 
programming in the browser that reencodes the data.


On the other hand, Python3 provides various facilities for working with 
such files.


The first I'll mention is the one that follows from my description of 
what your file really is: Python3 allows opening files in binary mode, 
and then decoding various sections of it using whatever encoding you 
like, using the bytes.decode() operation on various sections of the 
file. Determination of which sections are in which encodings is beyond 
the scope of this description of the technique, and is application 
dependent.


The second is to specify an error handler, that, like you, is trained to 
recognize the other encodings and convert them appropriately. I'm not 
aware that such an error handler has been or could be written, myself 
not having your training.


The third is to specify the UTF-8 with the surrogate escape error 
handler. This allows non-UTF-8 codes to be loaded into memory. You, or 
algorithms as smart as you, could perhaps be developed to detect and 
manipulate the resulting "lone surrogate" codes in meaningful ways, or 
could simply allow them to ride along without interpretation, and be 
emitted as the original, into other files.


There may be other technique that I am not aware of.

Glenn
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path support

[Glenn Linderman]

On 8/22/2014 11:50 AM, Oleg Broytman wrote:

On Fri, Aug 22, 2014 at 10:09:21AM -0700, Glenn Linderman 
 wrote:

On 8/22/2014 9:52 AM, Oleg Broytman wrote:

On Fri, Aug 22, 2014 at 09:37:13AM -0700, Glenn Linderman 
 wrote:

On 8/22/2014 8:51 AM, Oleg Broytman wrote:

What encoding does have a text file (an HTML, to be precise) with
text in utf-8, ads in cp1251 (ad blocks were included from different
files) and comments in koi8-r?
Well, I must admit the HTML was rather an exception, but having a
text file with some strange characters (binary strings, or paragraphs
in different encodings) is not that exceptional.

That's not a text file. That's a binary file containing (hopefully
delimited, and documented) sections of encoded text in different
encodings.

Allow me to disagree. For me, this is a text file which I can (and
do) view with a pager, edit with a text editor, list on a console,
search with grep and so on. If it is not a text file by strict Python3
standards then these standards are too strict for me. Either I find a
simple workaround in Python3 to work with such texts or find a different
tool. I cannot avoid such files because my reality is much more complex
than strict text/binary dichotomy in Python3.

I was not declaring your file not to be a "text file" from any
definition obtained from Python3 documentation, just from a common
sense definition of "text file".

And in my opinion those files are perfect text. The files consist of
lines separated by EOL characters (not necessary EOL characters of my OS
because it could be a text file produced in a different OS), lines
consist of words and words of characters.


Until you know or can deduce the encoding of a file, it is binary. If it 
has multiple, different, embedded encodings of text, it is still binary. 
In my opinion. So these are just opinions, and naming conventions. If 
you call it text, you have a different definition of text file than I do.





Looking at it from Python3, though, it is clear that when opening a
file in "text" mode, an encoding may be specified or will be
assumed.  That is one encoding, applying to the whole file, not 3
encodings, with declarations on when to switch between them. So I
think, in general, Python3 assumes or defines a definition of text
file that matches my "common sense" definition.

I don't have problems with Python3 text. I have problems with Python3
trying to get rid of byte strings and treating bytes as strict non-text.


Python3 is not trying to get rid of byte strings. But to some extent, it 
is wanting to treat bytes as non-text... bytes can be encoded text, but 
is not text until it is decoded. There is some processing that can be 
done on encoded text, but it has to be done differently (in many cases) 
than processing done on (non-encoded) text.


One difference is the interpretation of what character is what varies 
from encoding to encoding, so if the processing requires understanding 
the characters, then the character code must be known.


On the other hand, if it suffices to detect blocks of opaque text 
delimited by a known set of delimiters codes (EOL: CR, LF, combinations 
thereof) then that can be done relatively easily on binary, as long as 
the encoding doesn't have data puns where a multibyte encoded character 
might contain the code for the delimiter as one of the bytes of the code 
for the character.



On the other hand, Python3 provides various facilities for working
with such files.

The first I'll mention is the one that follows from my description
of what your file really is: Python3 allows opening files in binary
mode, and then decoding various sections of it using whatever
encoding you like, using the bytes.decode() operation on various
sections of the file. Determination of which sections are in which
encodings is beyond the scope of this description of the technique,
and is application dependent.

This is perhaps the most promising approach. If I can open a text
file in binary mode, iterate it line by line, split every line of
non-ascii bytes with .split() and process them that'd satisfy my needs.
But still there are dragons. If I read a filename from such file I
read it as bytes, not str, so I can only use low-level APIs to
manipulate with those filenames. Pity.


If the file names are in an unknown encoding, both in the directory and 
in the encoded text in the file listing, then unless you can deduce the 
encoding, you would be limited to doing manipulations with file APIs 
that support bytes, the low-level ones, yes.  If you can deduce the 
encoding, then you are freed from that limitation.



Let see a perfectly normal situation I am quite often in. A person
sent me a directory full of MP3 files. The transport doesn't matter; it
could be FTP, or rsync, or a zip file sent by email, or bittorrent. What
matters is that filenames and content are in alien encodings. Mo

Re: [Python-Dev] Bytes path support

[Glenn Linderman]

On 8/27/2014 5:16 AM, Nick Coghlan wrote:

On 27 August 2014 08:52, Nick Coghlan  wrote:

On 27 Aug 2014 02:52, "Terry Reedy"  wrote:

Nick, I think the first half of your post is one of the clearest
expositions yet of 'why Python 3' (in particular, the str to unicode
change).  It is worthy of wider distribution and without much change, it
would be a great blog post.

Indeed, I had the same idea - I had been assuming users already understood
this context, which is almost certainly an invalid assumption.

The blog post version is already mostly written, but I ran out of weekend.
Will hopefully finish it up and post it some time in the next few days :)

Aaand, it's up:
http://www.curiousefficiency.org/posts/2014/08/multilingual-programming.html

Cheers,
Nick.



Indeed, I also enjoyed and found enlightening your response to this 
issue, including the broader historical context. I remember when Unicode 
was first published back in 1991, and it sounded interesting, but far 
removed from the reality of implementations of the day. I was intrigued 
by UTF-8 at the time, and even wrote an encoder and decoder for it for a 
software package that eventually never reached any real customers.


Your blog post says:


Choosing UTF-8 aims to treat formatting text for communication with 
the user as "just a display issue". It's a low impact design that will 
"just work" for a lot of software, but it comes at a price:


  * because encoding consistency checks are mostly avoided, data in
different encodings may be freely concatenated and passed on to
other applications. Such data is typically not usable by the
receiving application.



I don't believe this is a necessary result of using UTF-8. It is a 
possible result, and I guess some implementations are using it this way, 
but a proper language could still provide and/or require proper usage of 
UTF-8 data through its type system just as Python3 is doing with PEP 
393.  In fact, if it were not for the requirement to support passing 
character strings in other formats (UTF-16, UTF-32) to historical APIs 
(in CPython add-on packages) and the resulting practical performance 
considerations of converting to/from UTF-8 repeatedly when calling those 
APIs, Python3 could have evolved to using UTF-8 as its underlying data 
format, and obtained equal encoding consistency as it has today.


Of course, nothing can be "required" if the user chooses to continue 
operating in the encoded domain, and manipulate data using the necessary 
byte-oriented features of of whatever language is in use.


One of the choices of Python3, was to retain character indexing as an 
underlying arithmetic implementation citing algorithmic speed, but that 
is a seldom needed operation, and of limited general applicability when 
considering grapheme clusters. An iterator based approach can solve both 
problems, but would have been best introduced as part of Python3.0, 
although it may have made 2to3 harder, and may have made it less 
practical to implement six and other "run on both Py2 and Py3" type 
solutions harder, without introducing those same iterative solutions 
into Python 2.6 or 2.7.


Such solutions could still be implemented as options. Even PEP 393 
grudgingly supports some use of UTF-8 when requested by the user, as I 
understand it. Whether such an implementation would be better based on 
bytes or str is uncertain without further analysis, although type 
checking would probably be easier if based on str. A high-performance 
implementation would likely need to be implemented at least partly in C 
rather than CPython, although it could be prototyped in Python for proof 
of functionality. The iterators could obviously be implemented to work 
based on top of solutions such as PEP 393, by simply using indexing 
underneath, when fixed-width characters are available, and other 
techniques when UTF-8 is the only available format (rather than 
converting from UTF-8 to fixed-width characters because of calling the 
iterator).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path related questions for Guido

[Glenn Linderman]

On 8/26/2014 4:31 AM, MRAB wrote:

On 2014-08-26 03:11, Stephen J. Turnbull wrote:

Nick Coghlan writes:

  > "purge_surrogate_escapes" was the other term that occurred to me.

"purge" suggests removal, not replacement.  That may be useful too.

neutralize_surrogate_escapes(s, remove=False, replacement='\uFFFD')


How about:

replace_surrogate_escapes(s, replacement='\uFFFD')

If you want them removed, just pass an empty string as the replacement.


And further, replacement could be a vector of 128 characters, to do 
immediate transcoding, or a single character to do wholesale replacement 
with some gibberish character, or None to remove (or an empty string).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path related questions for Guido

[Glenn Linderman]

On 8/27/2014 6:08 PM, Stephen J. Turnbull wrote:

Glenn Linderman writes:
  > On 8/26/2014 4:31 AM, MRAB wrote:
  > > On 2014-08-26 03:11, Stephen J. Turnbull wrote:
  > >> Nick Coghlan writes:

  > > How about:
  > >
  > > replace_surrogate_escapes(s, replacement='\uFFFD')
  > >
  > > If you want them removed, just pass an empty string as the
  > > replacement.

That seems better to me (I had too much C for breakfast, I think).

  > And further, replacement could be a vector of 128 characters, to do
  > immediate transcoding,

Using what encoding?


The vector would contain the transcoding. Each lone surrogate would map 
to a character in the vector.



If you knew that much, why didn't you use
(write, if necessary) an appropriate codec?  I can't envision this
being useful.


If the data format describes its encoding, possibly containing data from 
several encodings in various spots, then perhaps it is best read as 
binary, and processed as binary until those definitions are found.


But an alternative would be to read with surrogate escapes, and then 
when the encoding is determined, to transcode the data. Previously, a 
proposal was made to reverse the surrogate escapes to the original 
bytes, and then apply the (now known) appropriate codec. There are not 
appropriate codecs that can convert directly from surrogate escapes to 
the desired end result. This technique could be used instead, for 
single-byte, non-escaped encodings. On the other hand, writing specialty 
codecs for the purpose would be more general.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path related questions for Guido

[Glenn Linderman]

On 8/28/2014 12:30 AM, MRAB wrote:

On 2014-08-28 05:56, Glenn Linderman wrote:

On 8/27/2014 6:08 PM, Stephen J. Turnbull wrote:

Glenn Linderman writes:
  > On 8/26/2014 4:31 AM, MRAB wrote:
  > > On 2014-08-26 03:11, Stephen J. Turnbull wrote:
  > >> Nick Coghlan writes:

  > > How about:
  > >
  > > replace_surrogate_escapes(s, replacement='\uFFFD')
  > >
  > > If you want them removed, just pass an empty string as the
  > > replacement.

That seems better to me (I had too much C for breakfast, I think).

  > And further, replacement could be a vector of 128 characters, to do
  > immediate transcoding,

Using what encoding?


The vector would contain the transcoding. Each lone surrogate would map
to a character in the vector.


If you knew that much, why didn't you use
(write, if necessary) an appropriate codec?  I can't envision this
being useful.


If the data format describes its encoding, possibly containing data from
several encodings in various spots, then perhaps it is best read as
binary, and processed as binary until those definitions are found.

But an alternative would be to read with surrogate escapes, and then
when the encoding is determined, to transcode the data. Previously, a
proposal was made to reverse the surrogate escapes to the original
bytes, and then apply the (now known) appropriate codec. There are not
appropriate codecs that can convert directly from surrogate escapes to
the desired end result. This technique could be used instead, for
single-byte, non-escaped encodings. On the other hand, writing specialty
codecs for the purpose would be more general.


There'll be a surrogate escape if a byte couldn't be decoded, but just
because a byte could be decoded, it doesn't mean that it's correct.

If you picked the wrong encoding, the other codepoints could be wrong
too.


Aha! Thanks for pointing out the flaw in my reasoning. But that means it 
is also pretty useless to "replace_surrogate_escapes" at all, because it 
only cleans out the non-decodable characters, not the incorrectly 
decoded characters.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bytes path related questions for Guido

[Glenn Linderman]

On 8/28/2014 10:41 AM, R. David Murray wrote:

On Thu, 28 Aug 2014 10:15:40 -0700, Glenn Linderman  
wrote:

On 8/28/2014 12:30 AM, MRAB wrote:

On 2014-08-28 05:56, Glenn Linderman wrote:

On 8/27/2014 6:08 PM, Stephen J. Turnbull wrote:

Glenn Linderman writes:
   > On 8/26/2014 4:31 AM, MRAB wrote:
   > > On 2014-08-26 03:11, Stephen J. Turnbull wrote:
   > >> Nick Coghlan writes:

   > > How about:
   > >
   > > replace_surrogate_escapes(s, replacement='\uFFFD')
   > >
   > > If you want them removed, just pass an empty string as the
   > > replacement.

That seems better to me (I had too much C for breakfast, I think).

   > And further, replacement could be a vector of 128 characters, to do
   > immediate transcoding,

Using what encoding?

The vector would contain the transcoding. Each lone surrogate would map
to a character in the vector.


If you knew that much, why didn't you use
(write, if necessary) an appropriate codec?  I can't envision this
being useful.

If the data format describes its encoding, possibly containing data from
several encodings in various spots, then perhaps it is best read as
binary, and processed as binary until those definitions are found.

But an alternative would be to read with surrogate escapes, and then
when the encoding is determined, to transcode the data. Previously, a
proposal was made to reverse the surrogate escapes to the original
bytes, and then apply the (now known) appropriate codec. There are not
appropriate codecs that can convert directly from surrogate escapes to
the desired end result. This technique could be used instead, for
single-byte, non-escaped encodings. On the other hand, writing specialty
codecs for the purpose would be more general.


There'll be a surrogate escape if a byte couldn't be decoded, but just
because a byte could be decoded, it doesn't mean that it's correct.

If you picked the wrong encoding, the other codepoints could be wrong
too.

Aha! Thanks for pointing out the flaw in my reasoning. But that means it
is also pretty useless to "replace_surrogate_escapes" at all, because it
only cleans out the non-decodable characters, not the incorrectly
decoded characters.

Well, replace would still be useful for ASCII+surrogateescape.


How?


Also for
cases where the data stream is *supposed* to be in a given encoding, but
contains undecodable bytes.  Showing the stuff that incorrectly decodes
as whatever it decodes to is generally what you want in that case.
Sure, people can learn to recognize mojibake for what it is, and maybe 
even learn to recognize it for what it was intended to be, in limited 
domains. But suppressing/replacing the surrogates doesn't help with 
that... would it not be better to replace the surrogates with an escape 
sequence that shows the original, undecodable, byte value?  Like  \xNN ?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed schedule for 3.4.2

[Glenn Linderman]

Well, this thread seems to be top-posted so...

Why not provide _urlopen_with_scary_keyword_parameter as the 
monkey-patch option?


So after the (global to the module) monkeypatch, they would _still_ have 
to add the keyword parameter.



On 9/8/2014 4:31 PM, Guido van Rossum wrote:
I still prefer having a parameter on urlopen (or thereabouts) -- it 
feels wrong to make it easier to change this globally than on a 
per-call basis, and if you don't understand monkey-patching, it's 
impossible to debug if you put the patch in the wrong place.


For the poor soul who has a script with many 
urlopen("https"//") calls, well, they probably don't mind 
the busywork of editing each and every one of them.


I'm fine with giving the actual keyword parameter a scary-sounding 
ugly name.


On Mon, Sep 8, 2014 at 3:48 PM, Donald Stufft > wrote:




On Sep 8, 2014, at 6:43 PM, Nick Coghlan mailto:ncogh...@gmail.com>> wrote:


On 9 Sep 2014 08:30, "Donald Stufft" mailto:don...@stufft.io>> wrote:
>
> If someone wants to do this, can’t they write their own 6 line
function?

Unfortunately not, as the domain knowledge required to know what
those six lines should look like is significant.

Keeping the old unsafe behaviour around with a more obviously
dangerous name is much simpler than explaining to people "Here,
copy this chunk of code you don't understand".

If we were starting with a blank slate there's no way we'd offer
such a thing, but as Jim pointed out, we do want to make it
relatively easy for Standard Operating Environment maintainers to
hack around it if necessary.

Cheers,
Nick.

>
> import ssl
> import urllib.request
> _real_urlopen = urllib.request.urlopen
> def _unverified(*args, **kwargs):
> if not kwargs.keys() & {“context”, “cafile”, “capath”,
“cadefault”}:
> ctx = ssl.create_default_context()
> ctx.verify_mode = CERT_NONE
> ctx.verify_hostname = False
> kwargs[“context”] = ctx
> return _real_urlopen(*args, **kwargs)
>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>



Why isn’t documentation with appropriate red warnings a suitable
place if we really must have it? That sounds like a much better
solution that some weird function people monkeypatch. It gives
them more control over things (maybe they have a valid certificate
chain, but an invalid host name!), it’ll work across all Python
implementations, and most importantly, it gives us a place where
there is some long form location to be like “yea you really
probably don’t want to be doing this” in big red letters.

Overall I’m -1 on either offering the function or documenting it
at all, but if we must do something then I think documentation is
more than enough.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Proposed schedule for 3.4.2

[Glenn Linderman]

On 9/8/2014 8:41 PM, Stephen J. Turnbull wrote:

  > Why not provide _urlopen_with_scary_keyword_parameter as the
  > monkey-patch option?
  >
  > So after the (global to the module) monkeypatch, they would_still_  have
  > to add the keyword parameter.

I understand the hardline position, though I don't like it: "if you
don't know how to do it yourself, we won't help you do it at all."[1]

But this "defense in depth" suggestion really violates the "consenting
adults" principle.  One warning in the docs and another in the name
itself should be enough, and if it isn't, Mommy should take Jimmy's
RaspberryPi away.
I was assuming, because of the suggestion for a monkey patch at all, in 
response to Guido's suggestion of a keyword parameter, that there was a 
problem adding a keyword parameter to urlopen. If there is, then the 
combo above could be useful in making them track down and adjust the 
places that need it, without forcing them to adjust the places that 
don't need it? If there is not, then no need for the monkey patch at 
all, they can just change add the keyword parameter.


If the alternate function doesn't have an extra keyword parameter, the 
monkeypatch solution would be the "easy" way to apply the change 
globally, even to places that don't need it, or optionally don't need 
it... too big a hammer. Having the extra parameter also might make them 
not apply it globally, and think more about what and why they are doing 
what they are doing.


Whatever, the idea is out there. If no one likes it, let it die.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Multilingual programming article on the Red Hat Developer blog

[Glenn Linderman]

On 9/16/2014 5:21 PM, Stephen J. Turnbull wrote:

It isn't, because the bytes/str problem was that given a str object
out of context you could not tell whether it was a binary blob or
text, and if text, you couldn't tell if it was external encoded text
or internal abstract text.

That is not true here because the representations of characters vs.
smuggled bytes in str are disjoint sets.


Actually, while it may be true that for the email headers case, all 
characters are characters, just the encoding is unknown, it is not 
necessarily true that they are in disjoint sets. Some bytes may decode 
into characters without needing to be smuggled... maybe not in 
text-protocols like email, but in the general case. So then some of the 
bytes that should be interpreted as binary data are not in a disjoint 
set from characters.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Glenn Linderman]

On 9/24/2014 6:59 PM, Chris Angelico wrote:

That is, could pip
defer the declaration until it's parsed its command line args and
decided that it'll be installing into the system directory, but NOT
make that declaration if it's given --user, or if it's running inside
a venv, or anything else that means it doesn't need that power?

Other programs do it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] mUTF-7 support?

[Glenn Linderman]

On 10/9/2014 7:41 PM, R. David Murray wrote:

Specifically, it is about what we might better term mailbox
*folders*...that is, not what you would normally think of as the
'mailbox name', which is usually understood to be the thing before the @
in the email address (and can't contain non-ASCII yet...we need RFC 6855
support for that, and I'm not sure*anybody*  has that yet).


There are still lots of idiotic web sites that assume everything in 
front of the @ must be a letter, digit, dot, or hyphen, and even some 
that only permit one dot after the @... even though for 30 years or so, 
the RFCs have permitted a nice variety of other special characters, 
although not all of them.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Glenn Linderman]

On 10/28/2014 6:45 AM, Stephen J. Turnbull wrote:

because it's a fork, it's a different name
I think this is an important point, and first brought to this discussion 
here. A fork is _not_ called Python, but something else... but if it is 
kept extremely compatible and up-to-date in the hopes of reintegration 
once it proves that it solves a problem, and that there are sufficient 
users, then such hopes seem reasonable.


And targeting the new "future compatible" MSVCRT sounds like the right 
approach, although it won't solve today's problems today, but it may 
solve tomorrow's problems for a long time into the future... if the 
MinGW people can be convinced to support that new MSVCRT as well.


In addition to all the components that are enabled by MinGW 
(particularly Fortran based extensions), one must remember that the 
current Windows Python also has extensions that interface to MSVC 
libraries that have never been ported to MinGW or Linux, and may never 
be. So an incompatible MinGW-built fork will lose some of those 
extensions... they may not be important to the folks that need MinGW, 
but that would be where & why the community would be split if the MinGW 
fork is not compatible with (some version of MSVC).  Of course, the 
current MSVC-based community is _already_ having issues with 
incompatible versions of MSVC (not limiting that community to Python, 
but broader users of MSVC)... enough problems that even M$ has noticed 
that their incompatibilities are problematical, and are attempting to 
address... not just for Python, but for many other systems and libraries 
as well. So gathering around and supporting their attempts to achieve 
that, by using their new system early, when feedback can still have a 
chance of improving that new "future compatible" system, will benefit 
everyone... but that time is limited, from what Steve Dower reports of 
the schedule... hoping to be ready for Python 3.5.


So now is an excellent time for this discussion to be happening, and if 
some work can be done to fork, pull the patches together, and tweak them 
to work with the new MSVC, in the Python 3.5 timeframe, you can have a 
phenomenal result, even if it is still a fork at that time.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Impact of Windows PowerShell OneGet ?

[Glenn Linderman]

New package manager from M$... article here 
.


It seems doubtful that M$ will eliminate .msi (their obscure, hard to 
configure and use, installation format), so it seems doubtful that the 
addition of OneGet will _force_ any changes to Python packaging.


However, it does open the question in my mind about whether there will 
be any _benefits_ of OneGet that would inspire helpful, useful changes 
to Python packaging. They speak of "trusted repositories", and the like, 
and it sounds like a the various *nix package managers (apt-get, et 
alia), but perhaps allowing multiple repositories rather than just a 
single source vendor repository (I'm actually not sure if *nix package 
managers allow multiple repositories or not, but from the way people 
talk about them, it always sounds like a "distribution" also provides "a 
repository" of additional packages).


"trusted repositories" sounds more like Perl's CPAN.

One of the links 
 
contains this quote: "This first version of OneGet installs and searches 
software from Chocolatey repositories.  Support of additional 
repositories will come in subsequent versions."


I have no clue what a Chocolatey repository is (yet, will Google), but 
unknown others will come, it says... whether it is possible to write a 
"repository plugin" such that Perl's CPAN or Python's PyPI or other 
preexisting repositories can be accessed is not clear.


The relationship between PowerShell and OneGet is not clear either... is 
OneGet written in PowerShell, or is PowerShell just one way to invoke 
OneGet, or???


Just a heads up.






___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Impact of Windows PowerShell OneGet ?

[Glenn Linderman]

On 10/30/2014 7:30 AM, "Martin v. Löwis" wrote:

Most likely, OneGet won't replace pip/PyPI, any more than apt or yum
does; but it may be worth having Python itself available that way.
That might simply mean having someone package up Python and put it on
an appropriate server, or maybe python.org could end up hosting a
repo.

Python is already available in Chocolatey:

https://chocolatey.org/packages/python

Given that OneGet intends to support Chocolatey as a repository,
it might just work already. All python.org would have to guarantee
is stable URLs for the Python msi files.

It might. Thanks for that information.

Poking around the link, I discover a weirdness: the claim that the 
package to install 32-bit Python on 64-bit systems is different than 
installing the 32-bit Python on 32-bit systems. While the instructions 
are explicit on what to do inside the chocolatey environment for all 3 
cases (the third being 64-bit install on 64-bit systems), I'm baffled as 
to why there is a difference, because there isn't when downloading 
32-bit Python from python.org...


And there is a weird reference to chocolatey's -x86 parameter, and the 
explanation seems to be that chocolatey has provision for installing 
32-bit or 64-bit packages on 64-bit systems, but  that the way Python is 
included in chocolatey, that provision can't be used.  Sounds very 
strange, like whoever set this up either didn't understand Python, or 
didn't understand chocolatey, or there is some limitation to the 
chocolatey implementation of 32-bit vs 64-bit packages.


Maybe if I understand chocolatey, this wouldn't seem so weird.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Glenn Linderman]

On 11/23/2014 3:18 AM, Nick Coghlan wrote:
Patches getting held up in the review queue for weeks or months is a 
*huge* barrier to contribution, as it prevents the formation of the 
positive feedback cycle where having a contribution accepted feels 
good, so folks are more likely to want to contribute again.
Exactly. None of the patches I care about have gone anywhere for years. 
Two of them were done by others, apparently using the right processes, 
and still went nowhere, so it doesn't encourage me to attempt to even 
learn the process to make a more appropriate contribution of mine 
(which, sadly, I fixed too many different bugs in one file or two files, 
and have not split them into individual patches). Another has recently 
been making some progress... I guess that means I don't care enough 
about them, to fight to force them through the system, it is easier just 
to apply the patches to my installation... but that is a sad state of 
affairs.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Glenn Linderman]

General comments:

I was a bit concerned when Steve first posted his plans for the Windows 
installer and making a web installer forcing re-downloads for every 
install, and expressed those privately. I'm no longer concerned, this 
outlined scheme is good, but I have some suggestions to make it great :)


With these other options available, if the web installer can do the 
/layout, especially from a checkbox, I'd almost be tempted to agree that 
the 20MB installers wouldn't be needed.


But here's another idea:  automatically keep all the .msi and .cab files 
used for the first installation of Python with it in the directory from 
which it runs (naming convention... prefix them all with 
python-3.5.0a1..(msi|cab)  It is very likely that a reinstall 
will use the same components (if more are needed on a later install, add 
them to the directory also).  And a good naming convention makes it 
obvious what to delete when done with the installer.


And a related idea: on the first install page, have a check box 
"download all installation components", that would do that, even if they 
are not used, and even if either of the one-click installs are chosen.


And a related idea: for custom installs, record the choices made in a 
metadata file in that same directory, and after the first install, 
subsequent installs could have a 3rd single-click install: same custom 
install as last time.  This would be kept in the directory with the 
installer, so could be applied to a zillion machines, and an install 
option /ditto would allow those choices from the command line. That way, 
the administrator could use the friendly interface to install the first 
machine, making the appropriate choices, and then just run 
"python-3.5.0a1.exe /ditto" on all the other zillion-1 machines, without 
needing to learn any other obscure command line parameters. I don't care 
how you spell /ditto, as long as there is documentation.


Regarding /layout, to me, that name has no mnemonic meaning of "download 
all these installation components and save them". Documentation could 
provide that, of course, but choosing a name like /download might be 
nicer.  Saving to the same directory as the installer lives in seems 
easier than needing to specify a directory... the documentation can warn 
that users of the option should put the web installer in the desired, 
shared or private, installation directory prior to running the option...


On 1/3/2015 4:16 PM, Antoine Pitrou wrote:

One comment: I would find it a bit confusing if the default install
path changes when using the customized install. OTOH, maybe you can't
choose another default there.


This could be cured by defaulting to either the new or old install 
location, or to a blank box, and having a couple buttons to "set install 
location to C:\Python35" or "set install location to "C:\Program 
Files", as well as the browse button and the option to type into the 
box.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Glenn Linderman]

On 1/4/2015 2:56 PM, Steve Dower wrote:

Unfortunately, I'm fairly restricted on naming convention. For some reason 
Windows Installer requires CAB files to be 8.3 names, where some of the 8 are 
reserved, so the names are pretty meaningless.
Least common denominator, I'm sure. But a pretty stupid limitation at 
this point in time. It would convince me to use a different installer 
technology, along with the many obscure and arcane features of .msi and 
.cab files that already convinced me of that years ago.


So then I'd suggest storing all the files in a directory named 
"python-3.5.0a1-install-files" unless that would also have to be limited 
to 8.3 :(



However, Windows does automatically cache all the files (typically in 
C:\ProgramData\Package Cache) so that repairs and uninstalls don't need to 
redownload anything, and this cache is automatically managed to preserve disk 
space. If you add new features later then you won't have the files in the 
cache, which is about the only way that this doesn't already fit your 
suggestion. I think it's fairly unlikely that people will be surprised by this, 
though when the first complaints come it about it I'll happily reconsider.
Handy, I was unaware of this, being turned off long ago by the stupid 
features of .cab and .msi, so the good features have escaped me. If 
there is a way to convince it to download the rest of the files based on 
the checkbox, that would be handier. Checkbox could be worded "download 
install components even for features not installed" which should keep 
people from checking it unnecessarily.  You mentioned a help link... 
couldn't the help text be included (text compresses great!) and popped 
up in a window?



>And a related idea: for custom installs, record the choices made in a metadata
>file in that same directory, and after the first install, subsequent installs
>could have a 3rd single-click install: same custom install as last time. This
>would be kept in the directory with the installer, so could be applied to a
>zillion machines, and an install option /ditto would allow those choices from
>the command line. That way, the administrator could use the friendly interface
>to install the first machine, making the appropriate choices, and then just run
>"python-3.5.0a1.exe /ditto" on all the other zillion-1 machines, without 
needing
>to learn any other obscure command line parameters. I don't care how you spell
>/ditto, as long as there is documentation.

This is really interesting, and certainly possible. I'll keep it in mind as 
something to look into after alpha. (What may be easier is a message at the end 
showing the command line to use that will perform the same installation 
silently... hmm...)

If you can display the command line (which would be interesting), then 
you can save it in a file, and invoke it from the 3rd single-click 
install button! Or if you can't, at least make sure the command can be 
copied to the clipboard, and the administrator could paste it to 
python-3.4.0a3.ditto.cmd :)



Incidentally, whoever came up with the py.exe launcher deserves a medal. It's 
made dealing with multiple versions amazingly easy.


Totally agree!

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Glenn Linderman]

On 1/9/2015 12:34 PM, David Anthoff wrote:

Nothing is merged in yet and everything can still change, so I'm keen to

hear whatever

feedback people have. I've tried to make improvements fairly for

first-time users

through to sysadmins, but if I've missed something big I'd like to hear

about it before

we get too close to alpha 1.

It would be great if the new installer supported silent, portable installs.
What I have in mind is a silent installation that drops all the necessary
files for a working python into a folder, but does not put ANY file anywhere
else and does not register anything anywhere on the system. So no PATH
modification, no registering of this install as one of the available python
interpreters, no uninstall entry or anything like that. By "all the
necessary files" I mean that it drops things like the MSVC runtime dlls etc
into that folder into which I'm installing python, but again doesn't try to
install things like that system or even user wide.

I'll give you my very specific use-case, but I think this might be useful
for others as well: we are trying to build an installer for some other
product X that internally requires a python instance to work. This python
instance should NOT be visible to anything other than product X, i.e. no
user should ever start it directly or anything like that, it really is an
implementation detail of product X. If we had an official python installer
that supported silent, portable installs, I could just call that python
installer inside the setup program for X, and it would drop a fully working
python installation into a sub-directory of the install directory of product
X. And we would be happy :)

The old MSI installer sort of had something like that with the MSI
administrative install option. But it never really worked because the
administrative install didn't drop the MSVC runtime dlls anywhere, as far as
I could tell. At some point I hacked around that by modifying the MSI file
in my setup program to also drop the MSVC runtime dlls, but that was VERY
hacky...

Thanks,
David

Sounds like a nice feature.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Pydoc Replacement for Python's help()?

[Glenn Linderman]

On 1/27/2015 8:04 AM, Cyd Haselton wrote:

Noted. Will proceed with the 3.x releass.
I had been excited you were working on Android Python until I realized 
you were working on 2.x. I started with Python 3, and have only dabbled 
in 2.x for a couple projects that had unported dependency needs. One of 
them, reportlab, was ported in the last year, and the other is my web 
server CGI ports which is still constrained by a couple libraries, and 
although I think they may have recently been ported, I haven't had time 
to upgrade it. Probably will by April, when I have to ditch my Google 
OpenID login system, because they have abandoned it.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Glenn Linderman]

On 2/12/2015 9:39 AM, Ethan Furman wrote:

On 02/12/2015 12:05 AM, Thomas Heller wrote:


Could not py.exe be extended so that it allows starting scripts in a
somewhat similar way?  'py-script -2.7 myscript foo bar baz' ???
Which would execute the script myscript.exe, myscript.bat, myscript.py,
myscript.cmd or whatever is in the Scripts directory on the Python 2.7
installation, without changing PATH presistently?

I'm pretty sure py.exe should only start .py files, not .bat, .cmd, or 
.anything-else.



IIRC, it already can be configured to start other things. I think I even 
used it for that early on in Vinay's development cycle, just to prove 
it, although the things I used it for then were not things I'm still doing.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Glenn Linderman]

On 2/13/2015 1:13 AM, Paul Moore wrote:

On 13 February 2015 at 06:59, Thomas Heller  wrote:

To make it clear: My suggestion is (or was, maybe it isn't a good idea)
to have some way to start 'something' that is in the Scripts
directory of a Python installation (which is usually a python script
or an exe-wrapper for it), without typing in the full path.  And without
changing the PATH envrionment variable.

People using the old-style "scripts" keyword to setup() can put pretty
much anything into the Scripts directory - I've seen .py files, Python
files with no suffix, bat files, even Unix shell files (not much use
on Windows...) in there. I don't think it's reasonable for the
launcher to try to run all of those things.

Maybe restricting it to running ".py" files or ".exe" files would be
reasonable. That covers entry points (which should be the norm for
newer projects) and Python scripts (which are the most likely things
to be portable).
The WINDOWS py launcher hardly needs to be portable.  Batch/CMD files 
also seem useful on WINDOWS. And Powershell?


If a launcher is developed for or enhanced for other systems, shell 
scripts may be useful, and Batch/CMD not.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Glenn Linderman]

On 2/17/2015 10:52 AM, Barry Warsaw wrote:

Probably the best thing to do (on *nix at least) is, if the path is absolute,
use the given string verbatim.  If the path is relative, search for the given
executable on $PATH and use the first one found.  If nothing is found, use
what's given explicitly.
I think searching would be inappropriate.  Just use what is given, as a 
string.  If the user wants a search, they can use `which`. Otherwise, 
setting to a relative path reliably would be hard (and sometimes desirable).
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Glenn Linderman]

On 2/17/2015 1:40 PM, Barry Warsaw wrote:

I agree, for deployment to environments where you can guarantee that the basic
Python infrastructure is available.  It may not be though, and perhaps we can
relegate that to py2exe, cx_freeze, and friends, although I would say that
having such support built into Python would be very cool, and of high value to
our users.

That's not to play down the usefulness of good executable zip support, pared
of the stdlib and interpreter.  OTOH, it would be nice to think ahead, where
the same tool could generate both types of single-file executables.


Thinking ahead?  Novel idea!  My dream scheme...

A scheme for putting all the pure- Python modules (mostly user code) 
into a .pyz, and a concurrent scheme for putting all the non-pure- 
Python modules (maybe even Python itself, for platforms that are missing 
the necessary version of Python 3), into some other platform-dependent 
bundle.


User instructions then become... try the .pyz, if it doesn't work, 
install the platform-dependent bundle and then it will.


So one could build up a platform-dependent bundle that contains all 
stuff used in development of various projects, and have easy user 
instructions for deployment of  various .pyz apps.


Longer instructions for people that think they know what they are doing 
would list the version of Python and the versions of the various 
dependencies in the platform-dependent bundle, if they wish to install 
them manually, etc.


I suppose licensing might, in some cases, conflict with making a single 
bundle for some modules and platforms.


I'm already doing the .pyz apps, using .py extensions, but don't have 
the other piece in place it's harder, and I haven't acquired the 
skill set for putting the binary chunks together... other than unzipping 
to the filesystem.


Glenn
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for Pronouncement: PEP 441 - Improving Python ZIP Application Support

[Glenn Linderman]

On 2/26/2015 9:05 AM, Paul Moore wrote:

On 24 February 2015 at 18:24, Guido van Rossum  wrote:

Here's my review. I really like where this is going but I have a few
questions and suggestions (I can't help myself :-).

OK, I've updated both the PEP and the patch based on follow-up
discussions. I think (again!) it is ready to go.

I've included the updated PEP inline below, it'll be available at
peps.python.org as soon as someone has a chance to upload it.

Thanks to everyone for the various comments. If I've missed anything
that someone thinks I'd said I would change, please let me know.


Three very, very, very minor nits below.



Paul

PEP: 441
Title: Improving Python ZIP Application Support
Version: $Revision$
Last-Modified: $Date$
Author: Daniel Holth ,
 Paul Moore 
Discussions-To:
https://mail.python.org/pipermail/python-dev/2015-February/138277.html
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30 March 2013
Post-History: 30 March 2013, 1 April 2013, 16 February 2015

Improving Python ZIP Application Support


Python has had the ability to execute directories or ZIP-format
archives as scripts since version 2.6 [1]_.  When invoked with a zip
file or directory as its first argument the interpreter adds that
directory to sys.path and executes the ``__main__`` module.  These
archives provide a great way to publish software that needs to be
distributed as a single file script but is complex enough to need to
be written as a collection of modules.

This feature is not as popular as it should be mainly because it was
not promoted as part of Python 2.6 [2]_, so that it is relatively
unknown, but also because the Windows installer does not register a
file extension (other than ``.py``) for this format of file, to associate
with the launcher.

This PEP proposes to fix these problems by re-publicising the feature,
defining the ``.pyz`` and ``.pyzw`` extensions as "Python ZIP Applications"
and "Windowed Python ZIP Applications", and providing some simple
tooling to manage the format.

A New Python ZIP Application Extension
==

The terminology "Python Zip Application" will be the formal term used
for a zip-format archive that contains Python code in a form that can
be directly executed by Python (specifically, it must have a
``__main__.py`` file in the root directory of the archive).  The
extension ``.pyz`` will be formally associated with such files.

The Python 3.5 installer will associate ``.pyz`` and ``.pyzw`` "Python
Zip Applications" with the platform launcher so they can be executed.
A ``.pyz`` archive is a console application and a ``.pyzw`` archive is a
windowed application, indicating whether the console should appear
when running the app.

On Unix, it would be ideal if the ``.pyz`` extension and the name
"Python Zip Application" were registered (in the mime types database?).
However, such an association is out of scope for this PEP.

Python Zip applications can be prefixed with a ``#!`` line
pointing to the correct Python interpreter and an optional
explanation::

 #!/usr/bin/env python3
 #  Python application packed with zipapp module
 (binary contents of archive)

On Unix, this allows the OS to run the file with the correct
interpreter, via the standard "shebang" support.  On Windows, the
Python launcher implements shebang support.

However, it is always possible to execute a ``.pyz`` application by
supplying the filename to the Python interpreter directly.

As background, ZIP archives are defined with a footer containing
relative offsets from the end of the file.  They remain valid when
concatenated to the end of any other file.  This feature is completely
standard and is how self-extracting ZIP archives and the bdist_wininst
installer format work.


Minimal Tooling: The zipapp Module
==

This PEP also proposes including a module for working with these
archives.  The module will contain functions for working with Python
zip application archives, and a command line interface (via ``python
-m zipapp``) for their creation and manipulation.

More complete tools for managing Python Zip Applications are
encouraged as 3rd party applications on PyPI.  Currently, pyzzer [5]_
and pex [6]_ are two such tools.

Module Interface


The zipapp module will provide the following functions:

``create_archive(source, target=None, interpreter=None, main=None)``


Create an application archive from *source*.  The source can be any
of the following:

* The name of a directory, in which case a new application archive
   will be created from the content of that directory.
* The name of an existing application archive file, in which case the
   file is copied to the target.  The file name should include the
   ``.pyz`` extension, if required.


Or   ".pyzw", I presume.


* A file object ope

Re: [Python-Dev] backwards and forwards compatibility, the exact contents of pickle files, and IntEnum

[Glenn Linderman]

On 3/14/2015 10:52 PM, Ethan Furman wrote:

I'm not sure exactly how to phrase this inquiry, so please bear with me.

What exactly does backwards compatibility mean as far as pickle goes?  We have 
the various protocols, we have the
contents of pickle files created at those protocols, and we have different 
versions of Python.

Should a pickle file created in Python 3.4 with protocol 3 and the contents of 
socket.AF_INET be unpicklable with a
Python 3.3 system?  Because currently it cannot be as socket.AF_INET became an 
IntEnum in 3.4.

I'm thinking the answer is yes, it should be.

So how do we fix it?  There are a couple different options:

   - modify IntEnum pickle methods to return the name only

   - modify IntEnum pickle methods to pickle just like the int they represent

The first option has the advantage that in 3.4 and above, you'll get back the 
IntEnum version.

The second option has the advantage that the actual pickle contents are the 
same [1] as in previous versions.

So, the final question:  Do the contents of a pickle file at a certain protocol 
have to be the some between versions, or
is it enough if unpickling them returns the correct object?
In general, it would seem to me that each version of Python could add 
object types that are foreign to prior versions, and so as long as 
version N can unpickle anything produced by versions <=N, all would be well.


While I am _not_ an expert in pickle, this would just be my expectation 
based on years of experience with compatibility issues in non-Python 
non-pickle systems.  Pickle may have stronger compatibility goals than I 
would expect, and so must you, if you are thinking the answer to the 
above is "yes, it should be". :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 0-base and 1-base indexed iterables? Custom slicing rules?

[Glenn Linderman]

On 3/21/2015 10:46 PM, pedro santos wrote:


Hi, I'm an Art and CG student learning Python and today's exercise was 
about positions in a tiled room. The fact that I had to check if a 
position was inside the room and given that in a 1x1 room, 0.0 was 
considered in and 1.0 was considered out, it kept me thinking about 
0-base indexing iterables and slicing.


This sort of discussion is better held on python-l...@python.org which 
is for learning and using the current and past versions of Python.


This mailing list is for coordinating the development of future versions 
of Python.


If your idea is to start a discussion to attempt to affect future 
versions of Python, that sort of discussion belongs on 
python-id...@python.org, and ideas that are appropriate for 
implementation in future versions of Python eventually come to 
python-dev when they are accepted for implementation.


There are lots of friendly people on all these lists, but it is best to 
use the correct one, so that people can focus on the intended topics of 
the lists when reading each of them.


Cheers!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Installing Python to non-ASCII paths

[Glenn Linderman]

On 3/22/2015 8:12 AM, Tim Golden wrote:
I'll create a £££ user (which is the easiest non-ASCII name to create 
on a UK keyboard) to see how cleanly the latest installer works. 


You can also copy/paste.  A path with a Cyrillic, Greek, Chinese, 
Tibetan, Japanese, Armenian, and Romanian character, none of which are 
in the "Windows ANSI" character set, should suffice...  Here ya go...


ț硕բ文བོདΘ

In my work with Windows, I've certainly seen that £ is much more 
acceptable to more programs than ț or these other ones.
 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keyword-only parameters

[Glenn Linderman]

On 4/14/2015 2:14 PM, Guido van Rossum wrote:

Also, why do you think we added the 'lone star' syntax? :-)


Hint: Not because Guido is from Texas
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Surely "nullable" is a reasonable name?

[Glenn Linderman]

On 4/19/2015 1:19 AM, Larry Hastings wrote:



On 08/07/2014 09:41 PM, Larry Hastings wrote:
Well!  It's rare that the core dev community is so consistent in its 
opinion.  I still think "nullable" is totally appropriate, but I'll 
change it to "allow_none".


(reviving eight-month-old thread)



  * Zen: "There should be one (and preferably only one) obvious way to
do it." We have a way of specifying the types this parameter
should accept; "allow_none" adds a second.
  * Zen: "Special cases aren't special enough to break the rules". 
"allow_none" was really just a special case of one possible type

for "types".




Is argument clinic a special case of type annotations?  (Quoted and 
worded to be provocative, intentionally but not maliciously.)


OK, I know that argument clinic applies to C code and I know that type 
annotations apply to Python code. And I know that C code is a lot more 
restrictive /a priori/ which clinic has to accommodate, and type 
annotations are a way of adding (unenforced) restrictions on Python 
code.  Still, from a 50,000' view, there seems to be an overlap in 
functionality... and both are aimed at Py 3.5... I find that 
interesting... I guess describing parameter types is the latest Python 
trend :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: No new syntax is required

[Glenn Linderman]

On 4/26/2015 4:32 PM, Paul Sokolovsky wrote:

Then, is the only logic for proposing __aenter__ is to reinsure against
a situation that someone starts to write async context manager, forgets
that they write async context manager, and make an __enter__ method
there. Then your implementation will announce that "async context
manager lacks __aenter__", whereas "my" approach would announce
"Async's manager __enter__ did not return awaitable value".

Again, is that the distinction you're shooting for, or do I miss
something?


Seems like the missing __aenter__ can easily be detected by the 
interpreter at compile time, but the wrong type returned would be at run 
time, or after a complex type-analysis done at compile time (unlikely to 
be practical).


So I think you've nailed the distinction... but I'm not the expert.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A macro for easier rich comparisons

[Glenn Linderman]

On 4/28/2015 2:13 AM, Victor Stinner wrote:

#define Py_RETURN_RICHCOMPARE(val1, val2, op)   \
> do {\
> switch (op) {   \
> case Py_EQ: if ((val1) == (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
> case Py_NE: if ((val1) != (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
> case Py_LT: if ((val1) < (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;   \
> case Py_GT: if ((val1) > (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;   \
> case Py_LE: if ((val1) <= (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
> case Py_GE: if ((val1) >= (val2)) Py_RETURN_TRUE; Py_RETURN_FALSE;  \
> }   \
> Py_RETURN_NOTIMPLEMENTED;   \
> } while (0)

I would prefer a function for that:

PyObject *Py_RichCompare(long val1, long2, int op);
Why would you prefer a function?  As a macro, when the op is a constant, 
most of the code would be optimized away by a decent compiler.


I suppose when the op is not a constant, then a function would save code 
space.


So I suppose it depends on the predominant use cases.


You should also handle invalid operator. PyUnicode_RichCompare() calls
PyErr_BadArgument() in this case.
One can quibble over the correct error return, but the above code does 
handle invalid operators after the switch.


Glenn
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492: async/await in Python; version 4

[Glenn Linderman]

On 5/1/2015 9:59 AM, Guido van Rossum wrote:


I think coroutine is the name of a concept, not a specific
implementation.

Cheers,

 Cheers indeed! I agree that the *concept* is best called coroutine -- 
and we have used this term ever since PEP 342. But when we're talking 
specifics and trying to distinguish e.g. a function declared with 
'async def' from a regular function or from a regular generator 
function, using 'async function' sounds right. And 'async method' if 
it's a method.


Exactly. The async function/method is an implementation technique for a 
specific kind/subset of coroutine functionality.  So the term coroutine, 
qualified by a description of its best usage and limitationsof async 
function, can be used in defining async function, thus appealing to what 
people know or have heard of and vaguely understand and can read more 
about in the literature.


A glossary entry for coroutine in the docs seems appropriate, which 
could point out the 16† ways to implement coroutine-style 
functionalities in Python, and perhaps make recommendations for 
different types of usage.


†OK, not 16 ways, but it is 3 now, or 4?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unicode literals in Python 2.7

[Glenn Linderman]

On 5/9/2015 5:39 AM, Adam Bartoš wrote:
I already have a solution in Python 3 (see 
https://github.com/Drekin/win-unicode-console, 
https://pypi.python.org/pypi/win_unicode_console), I was just 
considering adding support for Python 2 as well. I think I have an 
working example in Python 2 using ctypes.


Is this going to get released in 3.5, I hope?  Python 3 is pretty 
limited without some solution for Unicode on the console... probably the 
biggest deficiency I have found in Python 3, since its introduction. It 
has great Unicode support for files and processing, which convinced me 
to switch from Perl, and I like so much else about it, that I can hardly 
code in Perl any more (I still support a few Perl programs, but have 
ported most of them to Python).


I wondered if all your recent questions about Py 2 were as a result of 
porting the above to Py 2... I only have one program left that I was 
forced to write in Py 2 because of library dependencies, and I think 
that library is finally being ported to Py 3, whew!  So while I laud 
your efforts, and no doubt they will benefit some folks for a few years 
yet, I hope never to use your Py 2 port myself!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Unicode literals in Python 2.7

[Glenn Linderman]

On 5/11/2015 1:09 AM, Nick Coghlan wrote:

On 10 May 2015 at 23:28, Adam Bartoš  wrote:

Glenn Linderman wrote:

Is this going to get released in 3.5, I hope?  Python 3 is pretty
limited without some solution for Unicode on the console... probably the
biggest deficiency I have found in Python 3, since its introduction. It
has great Unicode support for files and processing, which convinced me
to switch from Perl, and I like so much else about it, that I can hardly
code in Perl any more (I still support a few Perl programs, but have
ported most of them to Python).

I'd love to see it included in 3.5, but I doubt that will happen. For one
thing, it's only two weeks till beta 1, which is feature freeze. And mainly,
my package is mostly hacking into existing Python environment. A proper
implementation would need some changes in Python someone would have to do.
See for example my proposal http://bugs.python.org/issue17620#msg234439. I'm
not competent to write a patch myself and I have also no feedback to the
proposed idea. On the other hand, using the package is good enough for me so
I didn't further bring attention to the proposal.

Right, and while I'm interested in seeing this improved, I'm not
especially familiar with the internal details of our terminal
interaction implementation, and even less so when it comes to the
Windows terminal. Steve Dower's also had his hands full working on the
Windows installer changes, and several of our other Windows folks
aren't C programmers.

PEP 432 (the interpreter startup sequence improvements) will be back
on the agenda for Python 3.6, so the 3.6 time frame seems more
plausible at this point.

Cheers,
Nick.


Wow!  Another bug that'll reach a decade in age before being fixed...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553: Built-in debug()

[Glenn Linderman]

On 9/7/2017 4:19 PM, Terry Reedy wrote:
A somewhat separate point: the name breakpoint() is slightly 
misleading, which has consequences if it is (improperly) called more 
than once. While breakpoint() acts as a breakpoint, what it does (at 
least in the default pdb case) is *initialize* and start a *new* 
debugger, possibly after an import. 


So maybe the original debug() call should be renamed debugger() [but 
with the extra optional parameters discussed], and an additional 
breakpoint() call could be added that would be much more like hitting a 
breakpoint in the already initialized debugger.


There seem to be two directions to go here: If breakpoint is called 
without a prior call to debugger, it should (1) call debugger implicitly 
with defaults, or (2) be a noop.


I prefer the latter, but either is workable.  Or, one could add all 
those parameters to every breakpoint() call, which makes (1) more 
workable, but code more wordy.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 557: Data Classes

[Glenn Linderman]

On 9/18/2017 11:37 AM, Ethan Furman wrote:

On 09/11/2017 03:28 PM, Guido van Rossum wrote:

Oddly I don't like the enum (flag names get too long that way), but I 
do agree with everything else Barry said (it

should be a trivalue flag and please don't name it cmp).


Hmmm, named constants are one of the motivating factors for having an 
Enum type.  It's easy to keep the name a reasonable length, however: 
export them into the module-level namespace.  re is an excellent 
example; the existing flags were moved into a FlagEnum, and then (for 
backwards compatibility) aliased back to the module level:

snip

Likewise, if we had:

    class Compare(enum.Enum):
    NONE = 'each instance is an island'
    EQUAL = 'instances can be equal to each other'
    ORDERED = 'instances can be ordered and/or equal'
    globals().update(Compare.__members__)

then we can still use, for example, EQUAL, but get the more 
informative repr and str when we need to.


I tend to like the Enum also. But maybe this is a case where there 
should be multiple different decorators:


@dataclass
@equality_dataclass
@ordered_datatclass
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Investigating time for `import requests`

[Glenn Linderman]

On 10/1/2017 7:34 PM, Nathaniel Smith wrote:

Another major slowness comes from compiling regular expression.
I think we can increase cache size of `re.compile` and use ondemand cached
compiling (e.g. `re.match()`),
instead of "compile at import time" in many modules.

In principle re.compile() itself could be made lazy -- return a
regular exception object that just holds the string, and then compiles
and caches it the first time it's used. Might be tricky to do in a
backwards compatibility way if it moves detection of invalid regexes
from compile time to use time, but it could be an opt-in flag.
Would be interesting to know how many of the in-module, compile time 
re.compile calls use dynamic values, versus string constants. Seems like 
string constant parameters to re.compile calls could be moved to 
on-first-use compiling without significant backwards incompatibility 
impact if there is an adequate test suite... and if there isn't an 
adequate test suite, should we care about the deferred detection?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 553

[Glenn Linderman]

On 10/4/2017 5:22 PM, Yarko Tymciurak wrote:

Barry suggested I bring this up here.

It seems the right time to at least discuss this:

RE:  PEP 553 enabling / disabling breakpoints ---

I've recently started using a simple conditional breakpoint in 
ipython, and wonder if  - in addition to Nick Coghlan's request for 
the env 'PYTHONBREAKPOINT'  (boolean?), it would make sense (I _think_ 
so) to add a condition parameter to the breakpoint() call.  This does 
raise several questions, but it seems that it could make for a simple 
unified way to conditionally call an arbitrary debugger.  What I found 
useful (in the contecxt of ipython - but general enough) you can see 
in this gist: 
https://gist.github.com/yarko/bdaa9d3178a6db03e160fdbabb3a9885


If PEP 553's breakpoint() were to follow this sort of interface (with 
"condition"), it raises a couple of questions:

- how would a missing (default) parameter be done?
- how would parameters to be passed to the debugger "of record" be 
passed in (named tuple? - sort of ugly)
- would PYTHONBREAKPOINT be a global switch (I think yes), vs a 
`condition` default.


I have no dog in the fight, but to raise the possibility (?) of having 
PEP 553 implement simple conditional breakpoint processing.


Any / all comments much appreciated.

breakpoint() already accepts arguments. Therefore no change to the PEP 
is needed to implement your suggestion. What you are suggesting is 
simply a convention among debuggers to handle a parameter named 
"condition" in a particular manner.


It seems to me that

if condition:
    breakpoint()

would be faster and clearer, but there is nothing to prevent a debugger 
from implementing your suggestion if it seems useful to the developers 
of the debugger. If it is useful enough to enough people, the users will 
clamor for other debuggers to implement it also.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Add a developer mode to Python: -X dev command line option

[Glenn Linderman]

On 11/16/2017 7:39 AM, Barry Warsaw wrote:

On Nov 16, 2017, at 09:47, Yury Selivanov  wrote:


Let's keep it simple.  I'm big -1 on adding different "debug levels",
they are always confusing.

Oh, this one’s easy.

-X dev   == some debugging
-X deve  == a little more
-X devel == give it to me!


This is the   "devel level", where you solve those bugs from hell...


-X develo== now you’re talking (literally)
-X develop   == thank you sir, may I have another?
-X develope  == here comes the flood
-X developer == needle inna haystack!

Cheers,
-Barry



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Glenn Linderman]

On 11/23/2017 5:31 PM, Nick Coghlan wrote:
- a consolidated list of *all* the APIs that can safely be called 
before Py_Initialize
So it is interesting to know that list, of course, but the ones that are 
to be supported and documented might be a smaller list. Or might not.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Accepting PEP 562 -- Module __getattr__ and __dir__

[Glenn Linderman]

The word "a" is extraneous/confusing-in-grammar in the first line of the 
abstract also. I can't fix it.


On 12/4/2017 9:46 AM, Victor Stinner wrote:

Link for lazy people like me:
https://www.python.org/dev/peps/pep-0562/

I changed the PEP status to fix a typo in the abstract:
https://github.com/python/peps/commit/a87417b22bf15bc4382daeaef6d32886c687ad19

Victor


2017-12-04 17:58 GMT+01:00 Guido van Rossum :

Ivan,

Congrats on your PEP. I believe the outstanding issues are now resolved and
I am hereby accepting it.

PS. Sorry, Larry, PEP 549 is rejected. But that happened a while ago.

--
--Guido van Rossum (python.org/~guido)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Glenn Linderman]

On 12/5/2017 8:07 PM, INADA Naoki wrote:

Oh, revised version is really short!

And I have one worrying point.
With UTF-8 mode, open()'s default encoding/error handler is
UTF-8/surrogateescape.

Containers are really growing.  PyCharm supports Docker and many new Python
developers use Docker instead of installing Python directly on their system,
especially on Windows.

And opening binary file without "b" option is very common mistake of new
developers.  If default error handler is surrogateescape, they lose a chance
to notice their bug.


"b" mostly matters on Windows, correct? And Windows doesn't use C or 
POSIX locale, correct? And if these are correct, then is this an issue? 
And if so, why?



On the other hand, it helps some use cases when user want byte-transparent
behavior, without modifying code to use "surrogateescape" explicitly.

Which is more important scenario?  Anyone has opinion about it?
Are there any rationals and use cases I missing?

Regards,

INADA Naoki  


On Wed, Dec 6, 2017 at 12:17 PM, INADA Naoki  wrote:

I'm sorry about my laziness.
I've very busy these months, but I'm back to OSS world from today.

While I should review carefully again, I think I'm close to accept PEP 540.

* PEP 540 really helps containers and old Linux machines PEP 538 doesn't work.
   And containers is really important for these days.  Many new
Pythonistas who is
   not Linux experts start using containers.

* In recent years, UTF-8 fixed many mojibakes.  Now UnicodeError is
more usability
   problem for many Python users.  So I agree opt-out UTF-8 mode is
better than opt-in
   on POSIX locale.

I don't have enough time to read all mails in ML archive.
So if someone have opposite opinion, please remind me by this weekend.

Regards,

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/v%2Bpython%40g.nevcal.com



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Glenn Linderman]

On 12/7/2017 4:48 PM, Victor Stinner wrote:


Ok, now comes the real question, open().

For open(), I used the example of a code snippet *writing* the content
of a directory (os.listdir) into a text file. Another example is to
read filenames from a text files but pass-through undecodable bytes
thanks to surrogateescape.

But Naoki explained that open() is commonly misused to open binary
files and Python should somehow fail badly to notify the developer of
their mistake.


So the real problem here is that open has a default mode of text. 
Instead of forcing the user to specify either "text" or "binary" when 
opening, text is used as a default, binary as an option to be specified.


I understand that default has a long history in Unix-land, dating at 
last as far back as 1977 when I first learned how to use the Unix open() 
function.


And now it would be an incompatible change to change it.

The real question is whether or not it is a good idea to change it... at 
this point in time, with Unicode and UTF-8 so prevalent, text and binary 
modes are far different than back in 1977, when they mostly just 
documented that this was a binary file that was being opened, and that 
one could more likely expect to see read() than fgets() in the following 
code.


If it were to be changed, one could add a text-mode option in 3.7, say 
"t" in the mode string, and a PendingDeprecationWarning for open calls 
without the specification of either t or b in the mode string.


In 3.8, the warning would be changed to DeprecationWarning.

In 3.9, all open calls would need to have either t or b, or would fail.

Meanwhile, back on the PEP 540 ranch, text mode open calls could 
immediately use surrogateescape, binary mode open calls would not, and 
unspecified open calls would not.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 540: Add a new UTF-8 mode (v2)

[Glenn Linderman]

On 12/7/2017 5:45 PM, Jonathan Goble wrote:
On Thu, Dec 7, 2017 at 8:38 PM Glenn Linderman <mailto:v%2bpyt...@g.nevcal.com>> wrote:


If it were to be changed, one could add a text-mode option in 3.7,
say "t" in the mode string, and a PendingDeprecationWarning for
open calls without the specification of either t or b in the mode
string.


"t" is already supported in open()'s mode argument [1] as a way to 
explicitly request text mode, though it's essentially ignored right 
now since text is the default anyway. So since the option is already 
present, the only thing needed at this stage for your plan would be to 
begin deprecating not using it.


*goes back to lurking*

[1] https://docs.python.org/3/library/functions.html#open


Thanks for briefly de-lurking.

So then for PEP 540... use surrogateescape immediately for t mode.

Then, when the user encounters an encoding error, there would be three 
solutions: switch to t mode, explicitly switch to surrogateescape, or 
fix the locale.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of the Android platform

[Glenn Linderman]

On 12/15/2017 8:29 AM, Xavier de Gaye wrote:

On 12/14/2017 02:59 PM, Victor Stinner wrote:
> It seems like Android is evolving quickly, would say quicker than
> Python releases. I'm asking if it's a good idea to put a recipe aside
> the Python source code for one specific Android API version? Would it
> still make sense to build for NDK v14 in 2 or 5 years?

NDK 14 has been released in march 2017 and the latest release is NDK 
16. There are sometimes major changes between releases and I think it 
is critical to ensure that the builds all use the same NDK release for 
that reason. Supporting another NDK release is just a substitution in 
one of the files of the build system and I am sure that in 2 or 5 
years there would have been a core developer smart enough to make that 
substitution (this could even have been me, I will only be 71 years 
old in 5 years :-)). Anyway if this is a problem, this should have 
been discussed in a review of the PR.


There are concerns, including a concern raised by me, about supporting 
Android with that build system or to supporting Android at all. It has 
been interesting and gratifying to work on this build system and to 
get the Python test suite running on Android without failures. Given 
these concerns and the lack of interest in the support of Android it 
is time for me to switch to something else, maybe improve the bdb 
module, why not ?


Xavier


I, for one, would love to see Android become a supported platform.

My understanding is that APIs are generally backward compatible, so that 
programs created with one API continue to work on future APIs... there 
may be new features they don't use and maybe can't support, because they 
don't know about newer APIs, but that is true of Windows and Linux and 
Mac also. It doesn't seem like it would be necessary to "support" or 
"release" an official Python for every new Android version, but it would 
be nice to have ongoing support for a recent version each time Python is 
released, if Xavier or someone can do it.  It is certainly a mainstream 
platform for development these days.


I see a variety of Python apps on Android, but I have no idea how well 
they work, or what their source is, or if they are supported, or what 
features of Python or its standard library might not be available, etc. 
I've been too busy on other projects to take time to investigate them.


Glenn
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Glenn Linderman]

On 12/19/2017 5:32 PM, Nathaniel Smith wrote:

On Tue, Dec 19, 2017 at 4:56 PM, Steve Dower  wrote:

On 19Dec2017 1004, Chris Barker wrote:

Nathaniel Smith has pointed out that eval(pprint(a_dict)) is supposed to
return the same dict -- so documented behavior may already be broken.


Two relevant quotes from the pprint module docs:


The pprint module provides a capability to “pretty-print” arbitrary
Python data structures in a form which can be used as input to the
interpreter
Dictionaries are sorted by key before the display is computed.

It says nothing about the resulting dict being the same as the original one,
just that it can be used as input. So these are both still true (until
someone deliberately breaks the latter).

This is a pretty fine hair to be splitting... I'm sure you wouldn't
argue that it would be valid to display the dict {"a": 1} as
'["hello"]', just because '["hello"]' is a valid input to the
interpreter (that happens to produce a different object than the
original one) :-). I think we can assume that pprint's output is
supposed to let you reconstruct the original data structures, at least
in simple cases, even if that isn't explicitly stated.


Any dict object read in from pprint is going to be a different object, 
not the original one. And, unless the original insertion order was 
sorted by the same key as pprint uses to sort, the iteration order will 
be different from the original.


As pointed out below, it will compare equal to the original dict.

pprint has always allowed you to reconstruct the original data 
structures, but not the iteration order of dicts.


With the new insertion order guarantee, nothing has changed, here.

A far more interesting question than what pprint does to dict order is 
what marshal and pickle do (and have done) with the dict order, although 
I can't figure that out from the documentation.





In any case, there are so many ways
to spoil the first point for yourself that it's hardly worth treating as an
important constraint.

I guess the underlying issue here is partly the question of what the
pprint module is for. In my understanding, it's primarily a tool for
debugging/introspecting Python programs, and the reason it talks about
"valid input to the interpreter" isn't because we want anyone to
actually feed the data back into the interpreter, but to emphasize
that it provides an accurate what-you-see-is-what's-really-there view
into how the interpreter understands a given object. It also
emphasizes that this is not intended for display to end users; making
the output format be "Python code" suggests that the main intended
audience is people who know how to read, well, Python code, and
therefore can be expected to care about Python's semantics.


(though I assume order is still ignored when comparing dicts, so:
eval(pprint(a_dict)) == a_dict will still hold.


Order had better be ignored when comparing dicts, or plenty of code will
break. For example:


{'a': 1, 'b': 2} == {'b': 2, 'a': 1}

True

Yes, this is never going to change -- I expect that in the long run,
the only semantic difference between dict and OrderedDict will be in
their __eq__ methods.


Saying that "iter(dict)" will produce keys in the same order as they were
inserted is not the same as saying that "dict" is an ordered mapping. As far
as I understand, we've only said the first part.

(And the "nerve" here is that I disagreed with even the first part, but
didn't fight it too strongly because I never relied on the iteration order
of dict. However, I *do* rely on nobody else relying on the iteration order
of dict either, and so proposals to change existing semantics that were
previously independent of insertion order to make them rely on insertion
order will affect me. So now I'm pushing back.)

I mean, I don't want to be a jerk about this, and we still need to
examine things on a case-by-case basis but... Guido has pronounced
that Python dict preserves order. If your code "rel[ies] on nobody
else relying on the iteration order", then starting in 3.7 your code
is no longer Python.

Obviously I like that change more than you, but to some extent it's
just something we have to live with, and even if I disagreed with the
new semantics I'd still rather the standard library handle them
consistently rather than being half-one-thing-and-half-another.

-n



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 567 v2

[Glenn Linderman]

On 1/3/2018 11:16 AM, Guido van Rossum wrote:
Maybe I should clarify again what run() does. Here's how I think of it 
in pseudo code:


def run(self, func, *args, **kwds):
    old = _get_current_context()
    new = old.copy()
    _set_current_context(new)
    try:
    return func(*args, **kwds)
    finally:
    _set_current_context(old)




I find it interesting that self isn't used in the above pseudo-code. I 
thought that Context.run() would run the function in the "context" of 
self, not in the context of a copy of "current context".



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Making "-j0" the default setting for the test suite?

[Glenn Linderman]

On 1/28/2018 9:15 PM, Terry Reedy wrote:

The speedup would be even better but for the last very long running test.
Could the last very long running test be started first, instead? (maybe 
it is, or maybe there are reasons not to)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Friendly reminder: be kind to one another

[Glenn Linderman]

On 1/29/2018 6:16 PM, Brett Cannon wrote:
Over the last 3 days I have had two situations come up where I was 
asked for my opinion in regards to possible CoC violations. I just 
wanted to take this opportunity to remind everyone that open source 
does not work if we are not open, considerate, and respectful to one 
another (which also happens to be the PSF CoC that we are all expected 
to follow when working on Python). When we stop being kind to each 
other is when open source falls apart because it drives people away, 
and for a project that is driven by volunteers like Python that will 
be what ends this project (not to say people should be rude to 
corporate open source projects, but they can simply choose to switch 
to a core dump approach of open source).


I gave a talk at PyCascades this past week on setting expectations for 
open source participation: https://youtu.be/HiWfqMbJ3_8?t=7m24s . I 
had at least one person who was upset about no one getting to their 
pull request quickly come up to me afterwards and apologize for ever 
feeling that way after watching my talk, so do please watch it if you 
have ever felt angry at an open source maintainer or contributor to 
help keep things in perspective.


I also wanted to say that I think core developers should work extra 
hard to be kind as we help set the tone for this project which can 
leak into the broader community. People with commit privileges are not 
beyond rebuke and so people should never feel they are not justified 
speaking up when they feel a core developer has been rude to them.


Anyway, the key point is to remember is that people are what make this 
project and community work, so please make sure that you do what you 
can to keep people wanting to participate.


Thanks Brett, I'll have to watch that. But even before I do, let me 
comment that being kind is not something you will have to regret later.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dataclasses and correct hashability

[Glenn Linderman]

On 2/4/2018 9:49 PM, Guido van Rossum wrote:
A frozen class requires a lot of discipline, since you have to compute 
the values of all fields before calling the constructor. A mutable 
class allows other initialization patterns, e.g. manually setting some 
fields after the instance has been constructed, or having a separate 
non-dunder init() method. There may be good reasons for using these 
patterns, e.g. the object may be part of a cycle (e.g. parent/child 
links in a tree). Or you may just use one of these patterns because 
you're a pretty casual coder. Or you're modeling something external.


My point is that once you have one of those patterns in place, 
changing your code to avoid them may be difficult. And yet your code 
may treat the objects as essentially immutable after the 
initialization phase (e.g. a parse tree). So if you create a dataclass 
and start coding like that for a while, and much later you need to put 
one of these into a set or use it as a dict key, switching to 
frozen=True may not be a quick option. And writing a __hash__ method 
by hand may feel like a lot of busywork. So this is where 
[unsafe_]hash=True would come in handy.


I think naming the flag unsafe_hash should take away most objections, 
since it will be clear that this is not a safe thing to do. People who 
don't understand the danger are likely to copy a worse solution from 
StackOverflow anyway. The docs can point to frozen=True and explain 
the danger.


This is an interesting use case. I haven't got the internals knowledge 
to know just how just different mutable and immutable classes and 
objects are under the hood. But this use case makes me wonder if, even 
at the cost of some performance that "normal" immutable classes and 
objects might obtain, if it would be possible to use the various 
undisciplined initialization patterns as desired, followed by as 
declaration "This OBJECT is now immutable" which would calculate its 
HASH value, and prevent future mutations of the object?


Yes, I'm aware that the decision for immutability has historically been 
done at the class level, not the object level, but in my ignorance of 
the internals, I wonder if that is necessary, for performance or more 
importantly, for other reasons.


And perhaps the implementation is internally almost like two classes, 
one mutable, and the other immutable, and the declaration would convert 
the object from one to the other.  But if I say more, I'd just be babbling.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dataclasses and correct hashability

[Glenn Linderman]

On 2/5/2018 12:11 AM, Nathaniel Smith wrote:

On Sun, Feb 4, 2018 at 11:28 PM, Glenn Linderman  wrote:

This is an interesting use case. I haven't got the internals knowledge to
know just how just different mutable and immutable classes and objects are
under the hood. But this use case makes me wonder if, even at the cost of
some performance that "normal" immutable classes and objects might obtain,
if it would be possible to use the various undisciplined initialization
patterns as desired, followed by as declaration "This OBJECT is now
immutable" which would calculate its HASH value, and prevent future
mutations of the object?

It would be technically possible to support something like

@dataclass(freezable=True)
class Foo:
 blah: int

foo = Foo()
# Initially, object is mutable, and hash(foo) raises an error
foo.blah = 1
assertRaises(hash, foo)

# This method is automatically generated for classes with freezable=True
foo.freeze()

# Now object is immutable, and hash(foo) is allowed
assertRaises(foo.__setattr__, "blah", 2)
hash(foo)

I don't know if it's worth the complexity, but I guess it would cover
at least some of the use cases Guido raised.

-n

Thanks, Nathaniel, for confirming that what I was suggesting is not 
impossible, even if it turns out to be undesirable for some reason, or 
unwanted by anyone else. But I have encountered a subset of the use 
cases Guido mentioned, and had to make a 2nd class to gather/hold the 
values of the eventual immutable class, before I could make it, because 
pieces of the data for the class values were obtained from different 
sources at different times. Once all collected, then the immutability 
could be obtained, the rest of the processing performed. Thrashes the 
allocator pretty well doing it that way, but the job got done.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Dataclasses, frozen and __post_init__

[Glenn Linderman]

On 2/19/2018 7:02 PM, Guido van Rossum wrote:

But how?

On Mon, Feb 19, 2018 at 5:06 PM, Chris Barker - NOAA Federal 
mailto:chris.bar...@noaa.gov>> wrote:


... maybe it would be helpful to be able to
freeze an instance after creation for multiple use-cases?



And there's the crux of the issue... if the creator of Python can't 
figure out how to make mutable objects immutable by fiat after creation, 
then it is unlikely anyone else can!


Yet it seems there are use cases for that sort of ability.

Is it not possible to strip an object of its operations that mutate it? 
Probably it is, but, in doing so, it might strip the whole class of the 
operations that mutate it, rendering it impossible to the create 
additional mutable instances of the same class. In another message, 
Guido suggests that the solution of having a flag for each object, that 
presumably would be checked by each mutation operation, is not a 
wonderful idea:


On 2/18/2018 6:30 PM, Guido van Rossum wrote:
The underlying issue here is that we don't want an extra state flag in 
the object to indicate "this object is currently [im]mutable". Using 
__class__ assignment to signal this is clever way to add this state, 
though not without risks.


My only suggestion here is to add a new type of class to the language: 
one that would effectively be two nearly identical classes under the 
covers, having all the same interfaces and internal data structures.  
Both internal classes would be subclasses of  the new class, for is-a 
purposes.  But they would be defined in such a way (if there is such a 
way) as to minimize the risks referred to just above (I don't know what 
those risks are), so that mutable instances could become immutable via 
__class__ assignment.


So just to play with this a bit in high-level syntax:

freezable_class Foo:  # or maybe     class Foo( freezable ):
   int bar;

All instances of Foo would first be created as __mutable__Foo.  Both 
__mutable__Foo and __immutable__Foo would be instance_of( Foo ).


Foo xyz

xyz.bar = 17  # works fine
xyz.bar += 1 # works fine
if xyz.bar == 18:
 pass  # True
if instance_of( xyz, Foo ):
 pass  # True
if instance_of( xyz, __mutable__Foo ):
pass   # True

Foo would have an implicitly defined method (or defined on the parent 
freezable class) that would convert a class from __mutable__Foo  to  
__immutable__Foo.  There would be no reverse conversion.


xyz.__immutable__()  # under the covers, assigns xyz.__class__ to 
__immutable__Foo


Mutating methods applied to __immutable__Foo instances would cause 
exceptions, because they wouldn't exist for the __immutable__Foo subclass.


if xyz.bar == 18:
 pass  # True
if instance_of( xyz, __mutable__Foo ):
pass   # False
else:
    print("Can't change it any more.")

xyz.bar  = 19  # throws: xyz is now immutable

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Enum, Flag, __contains__, and False vs TypeError

[Glenn Linderman]

On 4/4/2018 11:32 AM, Ethan Furman wrote:

API question.

Background:
--

When doing checks such as

--> 3 in [4, 5, 6]
--> 'test' in {'test':True, 'live':False}

the result is True or False.

When doing checks such as

--> 3 in 'hello world'
--> [4, 5, 6] in {'test':True, 'live':False}

the result is a TypeError.

The general rule seems to be that if it is impossible for the 
in-question object to be in the container object then a TypeError is 
raised, otherwise True or False is returned.



Question 1:
--

(A) A standard Enum class is a container of Enum members.  It cannot 
hold anything else.  However, it has been returning False both in 
cases where the in-question object was an Enum of a different class (a 
Fruit in a Color, for example) and when the in-question object was not 
even an Enum ('apple' in Fruit, for example).


(B) The waters get even more muddied when Fruit has a str mixin, so 
`Fruit.APPLE == 'apple' is True` -- in that case, should `'orange' in 
Fruit` return True, False, or raise TypeError?



Question 2:
--

(A) The new Flag type allows `in` tests on the members themselves; so, 
for example:


--> SomeFlag.ONE in SomeFlag.One|SomeFlag.TWO
True

The question, of course, is what to do when a non-Flag member is 
tested for:


--> 'apple' in SomeFlag.ONE
# False or TypeError?

--> 2 in SomeFlag.TWO
# True or TypeError?

(B) And, of course, the same muddier question arises with IntFlag, 
where SomeFlag.TWO == 2 is True.



My thoughts:
---

For question 1A (pure Enum):  I'm thinking a TypeError should be 
raised when the in-question object is not an Enum member of any kind 
-- it simply is not possible for that object to ever be in an Enum, 
and is surely a bug.


For question 1B (mixed Enum):  if 1A is TypeError, then 1B should also 
be TypeError at least for non-mixin in-question types (so checking for 
1 in StrEnum would be a TypeError), but I'm torn between TypeError and 
True/False for cases where the in-question type matches the mixin type 
('apple' in StrEnum)  On the one hand, even though an Enum member 
might be equal to some other type, that other type will not have the 
Enum attributes, etc, and a True answer would lead one to believe you 
could access `.name` and `.value`, etc., while a False answer would 
lead one to believe there was no match even though equality tests 
pass; on the other hand, how strong is the "container" aspect of a 
mixed Enum?  How often is the test `'apple' in Fruit` meant to 
discover if you have a Fruit member vs whether you have something that 
could be a Fruit member?  Also, how important is it to be consistent 
with IntFlag, which I definitely think should return True/False for 
int checks?


For question 2A (pure Flag):  I'm comfortable sticking with a 
TypeError (assuming we switch to TypeError for 1A).


For question 2B (int Flag):  I think a TypeError if the in-question 
object is not an int is appropriate (assuming TypeError for 1A and 
2A), but if it is an int, True/False seems the better path.  My 
reasoning being that Flag and IntFlag are more similar to sets than 
lists, and IntFlag is specifically meant to work with ints, and a test 
of `2 in some_int_flags` is more concerned with a flag being set than 
with .name or .value attributes that may or may not exist on the 
in-question object.


Any and all thoughts appreciated.


I think the "in" test should raise TypeError if tested against 
_anything_ that is not an Enum member.


Why? I see a parallel between Enum and mappings.

x = {'test':True, 'live':False}

So it is True that  'test' in x   and  'live' in x  and False that True 
in x and False in x.


It is False that  'foo' in x  and  3 in x

It is TypeError that [4,5,6] in x, or {'foo': 'bar'} in x.

Note that it is False that (4,5,6) in x  which is a little surprising 
given the above two, but not when you realize the differences between 
this and the above two.


So with mappings, you can have any hashable type as a key: with Enum, 
you can only have Enum members as keys.


So I have no idea why you would want to return False, rather than 
TypeError, other than (1) the distinction probably doesn't matter to 
most people (2) backward compatibility.


I would find   2 in some_int_flags  being True surprising.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Is it useful to update cgitb module?

[Glenn Linderman]

On 4/7/2018 9:45 PM, Alex Walters wrote:

Are there people still actively developing new cgi scripts in python?  I know 
some modern HTTPDs don’t even support classic cgi without some kind of fastcgi 
daemon in between.  I am aware that some parts of various wsgi tools use the 
cgi module, but is the cgitb module useful for them?


Yes. I have several web sites and applications built as Python CGI 
scripts.  cgitb is extremely useful.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Glenn Linderman]

On 4/17/2018 12:46 AM, Chris Angelico wrote:

 gen = (x for x in rage(10)) # NameError
 gen = (x for x in 10) # TypeError (not iterable)
 gen = (x for x in range(1/0)) # Exception raised during evaluation

This brings such generator expressions in line with a simple translation to
function form::

 def ():
 for x in rage(10):
 yield x
 gen = () # No exception yet
 tng = next(gen) # NameError

Detecting these errors more quickly is nontrivial. It is, however, the exact
same problem as generator functions currently suffer from, and this proposal
brings the genexp in line with the most natural longhand form.


Open questions
==

Can the outermost iterable still be evaluated early?


As of Python 3.7, the outermost iterable in a genexp is evaluated early, and
the result passed to the implicit function as an argument.  With PEP 572, this
would no longer be the case. Can we still, somehow, evaluate it before moving
on? One possible implementation would be::

 gen = (x for x in rage(10))
 # translates to
 def ():
 iterable = iter(rage(10))
 yield None
 for x in iterable:
 yield x
 gen = ()
 next(gen)



I think "rage" is supposed to be "range" in four places in the quoted block.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Assignment Expressions

[Glenn Linderman]

On 4/23/2018 1:01 PM, Ethan Furman wrote:

On 04/22/2018 10:44 PM, Tim Peters wrote:

[Guido]
In reality there often are other conditions being applied to the 
match for
which `if expr as name` is inadequate. The simplest would be 
something like


   if ...:
   
   elif (m := re.match('(.*):(.*)', line)) and m.group(1) == 
m.group(2):

 

And the match() call may not even be the first thing to check -- 
e.g. we

could have

 elif line is not None and (m := re.match('(.*):(.*)', line)) 
and m.group(1) == m.group(2):


I find myself warming more to binding expressions the more I keep them
in mind while writing new code.  And I think it may be helpful to
continue showing real examples where they would help.

Today's example:  I happened to code this a few hours ago:

diff = x - x_base
if diff:
 g = gcd(diff, n)
 if g > 1:
 return g

It's not really hard to follow, but two levels of nesting "feels
excessive", as does using the names "diff" and "g" three times each.
It's _really_ an "and" test:  if the diff isn't 0 and gcd(diff, n) >
1, return the gcd.  That's how I _thought_ of it from the start.

Which this alternative expresses directly:

if (diff := x - x_base) and (g := gcd(diff, n)) > 1:
 return g


So I really like being able to make the assignment in the expression, 
but I have a really hard time parsing it with the name first.


Attempting to read just the names first:

  if
    diff
  scan for ending right paren
  found
  and
 g
  scan for ending right paren
    oops, found opening left paren
    scan for ending right paren
    found
  resume scanning for final right paren
  found
 > 1:
 return g


Attempting to read expressions first:

  if
    x - x_base
  and
    gcd(diff, n)
  what's diff?
    scan backwards
  diff is x - x_base
    > 1:
    return g
  what's g?
    scan up and backwards
    g is gcd(diff, n)


Attempting to read interleaved:

  if
    skip diff
  x - x_base
  back to diff
    as diff
  and
    skip g
  gcd(diff, n)
  back to g
    as g
  > 1:
  return g


On the other hand, if it were using the "as" keyword:

if (x - xbase as diff) and (gcd(diff, n) as g) > 1:
    return g

I would parse as:

  if
    x - x_base
    as diff
  and
    gcd(diff, n)
    as g
  > 1:
  return g

For me at least, the last is much more readable.  Thinking about it 
some more, the problem (or maybe just my problem) is that I see an 
"if" or "while" and the I look for the thing that is True or False, 
and using the ":=" syntax the first thing I see is a placeholder for a 
result that doesn't exist yet, making me constantly scan backwards and 
forwards to put all the pieces in the correct place.


With "as", it just flows forwards.


You need to borrow the time machine, and get with those early 
mathematicians that first said:


Let x be the sum of y and z

and convince them that what they should have said was:

Let the sum of y and z be called x.

:)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow

[Glenn Linderman]

On 4/24/2018 8:56 PM, Tim Peters wrote:

  The alternative is typing the
sub-expression(s) of interest by hand at the debugger prompt, or
adding print()s, both of which are prone to introducing typos, or
changing results radically due to triggering side effects in the code
invoked by the duplicated sub-expression(s).  Adding a binding
construct wouldn't change anything about how the code worked (apart
from possibly clobbering a local name).
I've done both subexpression mangling (attempts at duplication) and 
added print statements and experienced these negative side effects, 
taking twice as long (or more) as intended to get the debugging 
information needed.


While I'm no core developer, and would have a mild appreciation of 
avoiding those while True: loops so was generally in favor of this PEP, 
but not enough to be inpsired to speak up about it, I would frequently 
benefit from this capability... adding extra binding names, and printing 
_them_ instead of the duplicated subexpressions.


+1

Glenn
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow

[Glenn Linderman]

On 4/24/2018 6:10 PM, Tim Peters wrote:

Luckily, I only have to write code for me now, so am free to pick the
perfect compromise in every case;-)

QOTD !  I'm in the same situation.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] (name := expression) doesn't fit the narrative of PEP 20

[Glenn Linderman]

On 4/27/2018 2:11 PM, Tim Peters wrote:

That's easy:  any time there's a long thread to which Guido has
contributed at least twice, it will be seen as a Golden Opportunity to
re-litigate every decision that's ever been made;-)

You're getting pretty good at that QOTD thing, Tim :)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python startup time

[Glenn Linderman]

On 5/2/2018 8:56 PM, Gregory Szorc wrote:
Nobody in the project is seriously talking about a complete rewrite in 
Rust. Contributors to the project have varying opinions on how 
aggressively Rust should be utilized. People who contribute to the C 
code, low-level primitives (like storage, deltas, etc), and those who 
care about performance tend to want more Rust. One thing we almost 
universally agree on is that we want to rewrite all of Mercurial's C 
code in Rust. I anticipate that figuring out the balance between Rust 
and Python in Mercurial will be an ongoing conversation/process for 
the next few years.

Have you considered simply rewriting CPython in Rust?

And yes, the 4th word in that question was intended to produce peals of 
shocked laughter. But why Rust? Why not Go? http://esr.ibiblio.org/?p=7724
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A fast startup patch (was: Python startup time)

[Glenn Linderman]

On 5/5/2018 10:30 AM, Toshio Kuratomi wrote:
On Fri, May 4, 2018, 7:00 PM Nathaniel Smith > wrote:


What are the obstacles to including "preloaded" objects in regular
.pyc files, so that everyone can take advantage of this without
rebuilding the interpreter?


Would this make .pyc files arch specific?


Lots of room in the __pycache__ folder.

As compilation of the .py module proceeds, could it be determined if 
there is anything that needs to be architecture specific, and emit an 
architecture-specific one or an architecture-independent one as 
appropriate?  Data structures are mostly bitness-dependent, no?


But if an architecture-specific .pyc is required, could/should it be 
structured and named according to the OS conventions also:  .dll .so  .etc ?


Even if it doesn't contain executable code, the bytecode could be 
contained in appropriate data sections, and there has been talk about 
doing relocation of pointer in such pre-compiled data structures, and 
the linker _already_ can do that sort of thing...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Slow down...

[Glenn Linderman]

On 5/7/2018 7:59 AM, Eric Snow wrote:

On Sun, May 6, 2018 at 8:25 PM, Nick Coghlan  wrote:

I'm inclined to agree that a Python 3.8 PEP in the spirit of the PEP 3003
language moratorium could be a very good idea.

Note that the PEP specifically applies to "syntax, semantics, and
built-ins".  Here's the full abstract [1]:

 This PEP proposes a temporary moratorium (suspension) of all changes to the
 Python language syntax, semantics, and built-ins for a period of
at least two
 years from the release of Python 3.1. In particular, the moratorium would
 include Python 3.2 (to be released 18-24 months after 3.1) but allow Python
 3.3 (assuming it is not released prematurely) to once again include 
language
 changes.

 This suspension of features is designed to allow non-CPython 
implementations
 to "catch up" to the core implementation of the language, help ease 
adoption
 of Python 3.x, and provide a more stable base for the community.

-eric

Here's my "lightning" response to a "lightning talk" about a moratorium:

So if other implementations didn't catch up during the last moratorium, 
either the moratorium then was lifted too soon, or the other 
implementations don't really want to catch up, or the thought that they 
should catch up was deemed less important than making forward progress 
with the language. Have any of those opinions changed?


While async is a big change that I personally haven't grasped, but which 
has little impact (a couple keywords) on code that doesn't use it, a new 
moratorium wouldn't impact it, and nothing else that is happening seems 
too much or too fast from my perspective.


Dan's original comment about language versus library is interesting, 
though. It is probably true that one should resist adding language where 
library suffices, but sometimes a lack of certain expressiveness in the 
language causes cumbersome implementations of library routines to 
achieve the goal.


f-strings and binding expressions are cases where (1) the functionality 
is certainly possible via library (2) there is a large amount of code 
that uses the functionality, and (3) said code is more cumbersome 
without the expressiveness of the newer syntax.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Glenn Linderman]

On 5/28/2015 12:26 PM, Paul Moore wrote:

On 28 May 2015 at 19:22, Chris Angelico  wrote:

Unfortunately (and believe me, I've been down this road many times) on
Windows *only* the exe format is a "first-class" executable.
Executable scripts and shebangs are very useful, but there are always
corner cases where they don't work *quite* like an exe. On Windows,
you have to be prepared to ship an exe if you want to compete with
languages that generate exes.

I'm aware of that. When I said "a tiny stub", I was thinking in terms
of a small executable. The idea is that its sole purpose is to locate
Python someplace else, and chain to it; that has to be actual
executable code, complete with the 512-byte "MZ" header and
everything, to ensure compatibility. But it should be able to be
small, tight, and easy to verify correctness of, so there aren't (in
theory!) security exploits in the header itself.

OK, cool. I'm sort of working on that as a bit of a side project - a
tiny stub exe that you can prepend to a Python zipapp which runs it
via the standard embedding APIs. It's little more than an idea at the
moment, but I don't think it'll be too hard to implement...


Paul,

I've been using zipapps for some years now, and it would be really cool 
to have what I think you are talking about here:


1) Build zipapp as normal.  It likely depends on some minimal Python 
version.
2) Prepend stub .exe (Windows) or !# line (Unix) that declares a version 
of Python to actually use. This can be as broad as Python 2 or Python 3 
(not sure how to specify that either works, especially on Windows), or 
more detailed/restrictive by specifying  2.n or 3.n.  On Windows, it 
would find the newest qualifying installation and run it, and Unix, the 
symlinks are set up to do that already, is my understanding, so the 
proper !# line does that for you.


This would be something I could use and benefit from immediately upon it 
being available, so I laud your idea, and hope you have a successful 
implementation, and look forward to using it.  It would largely replace 
the need for the py.exe launcher for some classes of applications.


Of course, per other disccusions, this doesn't solve the problem for:

A) machine without Python installed
B) programs that need binary extensions

Other discussions have suggested:

3) The stub could offer to download and install Python

A corollary:

4) The stub could offer to download and install the needed binary 
extensions as well as Python. This would require the installation 
uniformity of something like pip, so perhaps would be restricted to 
extensions available via pip.  And it would be much enhanced by some 
technique where the zipapp would contain metadata readable by the stub, 
that would declare the list of binary extensions required. Or, of 
course, it could even declare non-binary extension that are not packaged 
with the zipapp, if the process is smooth, the modules available via 
pip, etc., as a tradeoff.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Glenn Linderman]

On 5/29/2015 3:33 AM, Paul Moore wrote:

On 28 May 2015 at 22:09, Glenn Linderman  wrote:

This would be something I could use and benefit from immediately upon it
being available, so I laud your idea, and hope you have a successful
implementation, and look forward to using it.  It would largely replace the
need for the py.exe launcher for some classes of applications.

The following proof-of-concept works as is (based on my pretty minimal
testing), and only uses the limited API, so it should work with any
version of Python 3 (I've not tested it with Python 2, but I think the
only "new" API is PySys_SetArgvEx, which could be replaced with
PySys_SetArgv at a pinch). Excuse the dreadful coding style and lack
of error handling, I hacked it up in about an hour :-)


I have no particular interest in Python 2, having started with Python 3, 
and only used Python 2 for cases where dependent packages required it, 
and I've now reached the nirvana of all my dependency packages being 
ported to Python 3, although I have yet to port/test one remaining 
application to prove that. So I only mentioned Python 2 because it still 
could be useful for other people :)



(Actually, I just tried building on Python 2 - guess what - Unicode
:-) SetProgramName and SetArgvEx won't take Unicode values. The easy
fix is just not to use Unicode, the hard one is to do the encoding
dance, but I'm not going to bother...).


One approach would be to support Unicode arguments only for Python 3, 
but that would really be only paying lip service to Python 2 support.  
Another approach might be to not #define UNICODE for the Python 2 
version, and use the 8-bit Windows APIs, allowing Windows and the C 
runtime to do the encoding dance for you? Although I'm not sure what 
Python 2 requires in that respect.



#define UNICODE
#define _UNICODE
#include 
#include 

int
main()
{
 TCHAR program[MAX_PATH];
 LPWSTR *argv;
 int argc;
 PyObject *runpy;
 PyObject *ret;

 argv = CommandLineToArgvW(GetCommandLineW(), &argc);
 GetModuleFileName(NULL, program, MAX_PATH);
 Py_SetProgramName(program);  /* optional but recommended */
 Py_Initialize();
 PySys_SetArgvEx(argc, argv, 0);
 runpy = PyImport_ImportModule("runpy");
 if (!runpy) PyErr_Print();
 ret = PyObject_CallMethod(runpy, "run_path", "u", program);
 if (!ret) PyErr_Print();
 Py_Finalize();
 return 0;
}


That looks interesting, I wonder what compilation environment it would 
need?  I don't think I've even installed a C compiler on my last couple 
boxes, and the only version of a C compiler I have is, umm... M$VC++6.0, 
since I've moved to using Python for anything a 5 line batch file can't 
do...



One mildly annoying thing is that python3.dll is only installed in
\DLLs, which typically isn't on PATH.
Ah, linking so I guess if I figured out how to create this binary, 
it would contain a reference to python3.dll that would attempt to be 
resolved via the PATH, from what you say, and typically fail, due to 
PATH seldom containing python3.dll.  The python launcher gets around 
that by (1) being installed in %windir%, and going and finding the 
appropriate Python (per its own configuration file, and command line 
parameters), and setting up the path to that Python, which, when 
executed, knows its own directory structure and can thus find its own 
python3.dll.


The launcher, of course, adds an extra layer of process between the 
shell and the program, because it launches the "real" Python executable.



So actually using the limited API from your own application fails by default.
Fixing that's mostly a user admin issue, though (and you can just link
to the full API and avoid the whole problem).


Do I understand correctly that the "user admin issue" means "add the 
appropriate \DLLs to the PATH"?


What I don't understand here is how linking to the full API avoids the 
problem... it must put more python library code into the stub 
executable? Enough to know how to search the registry to find the 
 for the version of Python from which the full API 
was obtained? Or something else?


Are there other alternatives?  Assuming that the reference to the 
missing DLL is not required until the point at which a symbol from it is 
first referenced, so that the stub would have some ability to do 
something before that first call, maybe...


1. The stub above could be enhanced to contained a "hard coded"
   directory that it adds to the PATH itself?
2. The stub above could be enhanced to define that its first parameter
   is the , and tweak its PATH.
3. These days, the Python installer does offer to optionally add itself
   to the PATH. Is that sufficient to make the stub work?
4. The launcher could be used, assuming it is installed, but then you
   don't need a stub, and you get the extra pr

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Glenn Linderman]

On 5/29/2015 2:45 PM, Paul Moore wrote:

On 29 May 2015 at 21:49, Glenn Linderman  wrote:

That looks interesting, I wonder what compilation environment it would need?
I don't think I've even installed a C compiler on my last couple boxes, and
the only version of a C compiler I have is, umm... M$VC++6.0, since I've
moved to using Python for anything a 5 line batch file can't do...


One mildly annoying thing is that python3.dll is only installed in
\DLLs, which typically isn't on PATH.

Ah, linking so I guess if I figured out how to create this binary, it
would contain a reference to python3.dll that would attempt to be resolved
via the PATH, from what you say, and typically fail, due to PATH seldom
containing python3.dll.  The python launcher gets around that by (1) being
installed in %windir%, and going and finding the appropriate Python (per its
own configuration file, and command line parameters), and setting up the
path to that Python, which, when executed, knows its own directory structure
and can thus find its own python3.dll.

The launcher, of course, adds an extra layer of process between the shell
and the program, because it launches the "real" Python executable.


So actually using the limited API from your own application fails by
default.
Fixing that's mostly a user admin issue, though (and you can just link
to the full API and avoid the whole problem).


Do I understand correctly that the "user admin issue" means "add the
appropriate \DLLs to the PATH"?

What I don't understand here is how linking to the full API avoids the
problem... it must put more python library code into the stub executable?
Enough to know how to search the registry to find the 
for the version of Python from which the full API was obtained? Or something
else?

Sorry, I assumed more Windows/C knowledge than you have.


It is mostly the C/Python interface knowledge that I lack... although my 
Windows/C knowledge is getting rusty.



I'll work on this and produce proper binaries in due course, so you
can always wait for them. But you can build the stub with pretty much
anything, I suspect - I managed with MSVC 2010 and mingw. I'll add
some build docs and get it on github.

Using mingw

 gcc -Wall -O2 -o stub.exe stub.c -I \Include
C:\Windows\system32\python34.dll
 strip -s stub.exe

Using MSVC

 cl /Festub.exe /O2 stub.c /I\Include \libs\python34.lib


Github sounds good.  Binaries sound good.  I would have to download the 
free MSVC10 or Ming and install and learn to use them, etc., to make 
progress... probably doable, but (1) I'm surviving at the moment with 
the launcher + zipapp, but it'd be nice for folks I code for to have 
.exe things, and (2) I'm backlogged in my other projects which don't 
need me to download a C compiler to make progress.



Regarding the DLLs, yes the "user admin issue" is adding the right
directory to PATH. I used the phrase "admin issue" as it's the aspect
that's likely to be far harder than any of the technical issues :-)
The reason using the full API helps is that the full API references
python34.dll rather than python3.dll. And the Python installer puts
python34.dll on PATH automatically, as it's what the "python" command
uses. (For the people with more Windows knowledge, I know this is a
simplification, but it's close enough for now).

So there are two  options with the code I posted.

1. Build an exe that uses a specific version of Python, but which will
"just work" in basically the same way that the "python" command works.
2. Build an exe that works with any version of Python, but requires
some setup from the user.

Either approach requires that the Python DLL is on PATH, but that's
far more likely with the version-specific one, just because of how the
installer does things.


I don't presently see any C:\Python34\DLLs or C:\Python34 on my path, 
but I didn't ask the installer to put it there either. So I'm guessing 
your option 1 assumes asking the Python installer to put it there? Not 
"automatically" but "on request", I think?


In my c:\Python34\DLLs, I don't see a python34.dll, only python3.dll... 
so I'm somewhat unclear on your simplified explanation.




With extra code, the stub could locate an appropriate Python DLL
dynamically, which would simplify usage at the cost of a bit of fiddly
code in the stub.

This might be a useful addition to the zipapp module for Python 3.6.


Indeed.  Especially with extra fiddly code if you or someone on Github 
has the time, it could be very useful for the zipapp module.




Paul

PS Current launchers (py.exe, the entry point launchers from
pip/setuptools, etc) tend to spawn the actual python program in a
subprocess. I believe there are *technically* some differences in the
runtime environment when you use an

Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)

[Glenn Linderman]

On 5/29/2015 3:28 PM, Paul Moore wrote:

On 29 May 2015 at 23:15, Glenn Linderman  wrote:

I don't presently see any C:\Python34\DLLs or C:\Python34 on my path, but I
didn't ask the installer to put it there either. So I'm guessing your option
1 assumes asking the Python installer to put it there? Not "automatically"
but "on request", I think?

In my c:\Python34\DLLs, I don't see a python34.dll, only python3.dll... so
I'm somewhat unclear on your simplified explanation.


I'm definitely batting zero today :-(

OK, let's try to be clear. I typically do "all users" installs. The
"for me only" install is slightly different, and the new install for
3.5 may be different again. But what I see is:

1. C:\Python34\DLLs containins python3.dll, which is *never* on PATH
(and doesn't need to be for normal use). Anything that wants to use
python3.dll needs that directory manually adding to PATH.
2. python34.dll is in C:\Windows\System32. This is always available to
all processes, as it's in the Windows system directory.

If you say "add Python to my PATH" you get C:\Python34 added to PATH.
For a user install, I believe python34.dll may be in there rather than
in C:\Windows\system32, so technically, for an app that uses
python34.dll to work, you need *either* an admin install, *or* to have
done "add Python to PATH".



Interesting.

In C:\, I have directories  Python27, Python32, Python33, Python34.  I 
can't be 100% sure how I answered the install questions, even Python34 
was a couple months ago.


In C:\Windows\System32, I have python27.dll, python32.dll, python33.dll, 
pythoncom32.dll, pythoncom33.dll, pythoncomloader32.dll, and 
pythoncomloader33.dll.  But not python34.dll!  I finally found that in 
c:\Windows\SysWOW64, which I guess means that I "accidentally" installed 
a 32-bit Python 3.4.  Or maybe I had a reason at the time.  But does 
that add another dimension to the picture for the stub?




I hope that made sense. Sorry for my garbled previous version.


This matches reality better... but whether it makes sense or not is 
another question.



On 5/29/2015 3:33 PM, Steve Dower wrote:
> Paul Moore wrote:
>> One mildly annoying thing is that python3.dll is only installed in 
\DLLs, which
>> typically isn't on PATH. So actually using the limited API from your 
own application fails by default.
>> Fixing that's mostly a user admin issue, though (and you can just 
link to the full API and avoid the whole problem).

>
> I didn't even notice that 3.4 (and earlier?) were doing that, so I 
changed/fixed it by accident :)



Indeed, and earlier. It apparently started in 3.2 with the definition of 
the Stable ABI.



> Python 3.5 installs python3.dll alongside python35.dll, so it'll go 
into the user's Python directory by default or into the system directory 
for an all-users installation. The embeddable distro includes 
python3.dll alongside the other DLLs as well.

>
> Cheers,
> Steve


This makes more sense... but will it cause problems with something? It 
seems to me like it was a bug to put it in the  
rather than %windir%\System32 back in Python 3.2 days when it was 
invented. What good is it to have a stable ABI that is hidden away where 
it cannot (easily) be used, and the more dynamic API is easier to get to?


Sadly, PEP 384 is silent on the location of python3.dll.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Where are bugs with the web site reported?

[Glenn Linderman]

On 7/16/2015 12:11 PM, Ryan Gonzalez wrote:
I have encountered this weird issue on Chrome for Android where 
scrolling up just a little causes the page to dart to the top. I was 
going to report it in the bug tracker, but I didn't see a label for 
the web site itself.


Worst part is, this is stopping me from reading the humor page!


Sounds more like a bug in Chrome than on the site, unless it is 
repeatable using other browsers, or unless the site has Chrome-specific 
code.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status on PEP-431 Timezones

[Glenn Linderman]

On 7/27/2015 1:42 PM, Lennart Regebro wrote:

On Mon, Jul 27, 2015 at 9:47 PM, Ethan Furman  wrote:

On 07/27/2015 07:46 AM, Lennart Regebro wrote:

Well, OK, let's propose these wordings: It looks like a date
operation, ie, add one to the date, but in reality it's a time
operation, ie add 86400 seconds to the time. These things sound
similar but are very different.

I have to disagree.  If I have my alarm at 7am (localtime ;) so I can be at
work at 8am I don't care exactly how many seconds have passed, that alarm
better go off at 7am local time.

Right. And then adding 86400 seconds to it is not the right thing to do.


It is the right thing to do... but one also adds/subtracts 3600 seconds 
from it before going to bed 2 days a year, due to government 
interference, unless it is an atomic clock or cell-phone, which do those 
updates automatically.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 495 Was: PEP 498: Literal String Interpolation is ready for pronouncement

[Glenn Linderman]

On 9/11/2015 5:40 PM, Alexander Belopolsky wrote:


The insanity I am dealing with now

...
But the decision to allow interzone t - s was made long time ago and 
it is a PEP 495 goal to change that.


The first few paragraphs you wrote, which I elided, are a great 
explanation of why things work in ways that might be unexpected, and by 
including in the descriptions other things that might be unexpected, it 
helps people realize that the need to understand what the operators 
really mean, when applied to classes, rather than numbers.  Of course, 
even floating point number operations and integer division only 
approximate mathematical reality, if you are looking for more examples.


But the beginning phrase about "insanity" should probably be elided in 
documentation, yet the body could very well be appropriate for tutorial 
documentation, even if not reference documentation, although I'd not 
object to finding it there.


The last phrase, about it being a PEP 495 goal to change that, might be 
true, but if it changes it, then it would be a confusing and backward 
incompatible change.




Yes, but are we willing to accept that datetimes have only partial order?


That's what the politicians gave us. These are datetime objects, not 
mathematical numbers.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 495 Was: PEP 498: Literal String Interpolation is ready for pronouncement

[Glenn Linderman]

On 9/11/2015 6:39 PM, Alexander Belopolsky wrote:
On Fri, Sep 11, 2015 at 9:12 PM, Glenn Linderman 
mailto:v+pyt...@g.nevcal.com>> wrote:


That's what the politicians gave us. These are datetime objects,
not mathematical numbers.


That's an argument for not defining mathematical operations like <, > 
or - on them, but you cannot deny the convenience of having those. 


It wasn't intended to argue for not defining the operations, just 
intended to justify that it is partial ordering... if the associated 
timezone implements daylight saving.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] If you shadow a module in the standard library that IDLE depends on, bad things happen

[Glenn Linderman]

On 10/29/2015 12:30 PM, Paul Moore wrote:

On 29 October 2015 at 18:45, Donald Stufft  wrote:

So I don=E2=80=99t think it=E2=80=99s true that people don=E2=80=99t shad=
ow the standard library, they just have various ways to do it that have s=
everal gotchas and require people to generally hack around the limitation=
.=C2=A0

(Your mailer or mine seems to have gone weird with encoding...)

There's a difference between the opting into using (say) pdb++ at
runtime, and at install time. The problem with the sort of shadowing
eggs did (and people try to emulate) is that simply *installing* a
package changes the behaviour of programs that don't use that package,
by virtue of the fact that it's there at all.

I'd much rather pdb++ or anything similar allowed the user to opt in
on demand. For example, it could put a file pdb.py containing "from
pdbpp import *" into a subdirectory, and direct users who want to
replace pdb with pdbpp to add that directory to PYTHONPATH. Just don't
do that at install time, leaving the user needing to uninstall pdbpp
to opt out of the behaviour.

As I said, I'm not against people being *able* to shadow the stdlib, I
just don't want to see it being *easy*, nor do I want it to be the
norm.

Paul

PS My experience with a similar case is pyreadline on Windows, which
hooks into the stdlib readline functionality when installed. That's a
PITA, because mostly I don't want pyreadline (I prefer the default
Windows command line editing, despite it having less functionality)
but I want pyreadline when using ipython. There's no way for me to get
that behaviour because (py)readline is enabled by installing it, not
by user choice at runtime.


+1

Although, I wouldn't mind if it were easy & documented, like say having 
both shadow-packages before stdlib and site-packages after it, so that 
multiple, super-tricky and annoying ways of doing the shadowing were not 
needed.  But, say, packages shadow-packages would only be used if there 
were an extra directive somewhere to enable it, or a subset of the 
packages that reside there.


Glenn
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

[Glenn Linderman]

On 11/14/2015 3:21 PM, Serhiy Storchaka wrote:

On 15.11.15 00:56, Victor Stinner wrote:

These encodings are rarely used. I don't think that any text editor use
them. Editors use ascii, latin1, utf8 and... all locale encoding. But I
don't know any OS using UTF-16 as a locale encoding. UTF-32 wastes disk
space.


AFAIK the standard Windows editor Notepad uses UTF-16. And I often 
encountered Windows resource files in UTF-16. UTF-16 was more popular 
than UTF-8 on Windows some time. If this horse is dead I'll throw it away.


Just use UTF-8, ignoring an optional leading BOM. If someone wants to 
use something else, they can write a "preprocessor" to convert it to 
UTF-8 for use by Python.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

[Glenn Linderman]

On 11/14/2015 5:15 PM, Chris Angelico wrote:

Can the py.exe launcher handle a UTF-16 shebang? (I'm pretty sure Unix
program loaders won't.) That alone might be a reason for strongly
encouraging ASCII-compat encodings.


That raises an interesting question about if py.exe can handle a leading 
UTF-8 BOM.  I have my emacs-on-Windows configured to store UTF-8 without 
BOM, but Notepad would put a BOM when saving UTF-8, last I checked.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

[Glenn Linderman]

On 11/14/2015 5:15 PM, Chris Angelico wrote:

I think even Notepad defaults to UTF-8 for
files, now.


Just installed Windows 10 on a new machine, and upgraded to the latest 
Windows 10 release, 1511.


Notepad defaults to ANSI encoding, as I think it always has.  UTF-8 is 
an option, and it does seem to try to notice the original encoding of 
the file, when editing old files, but when creating a new one ANSI.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support of UTF-16 and UTF-32 source encodings

[Glenn Linderman]

On 11/14/2015 5:37 PM, Chris Angelico wrote:

On Sun, Nov 15, 2015 at 12:27 PM, Glenn Linderman  wrote:

Notepad defaults to ANSI encoding, as I think it always has.  UTF-8 is an
option, and it does seem to try to notice the original encoding of the file,
when editing old files, but when creating a new one ANSI.

Thanks. Is "ANSI" always an eight-bit ASCII-compatible encoding?


I wouldn't trust an answer to this question that didn't come from 
someone that used Windows with Chinese, Japanese, or Korean, as their 
default language for the install. So I don't have a trustworthy answer 
to give.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python Language Reference has no mention of list comÃprehensions

[Glenn Linderman]

On 12/3/2015 5:56 PM, Chris Angelico wrote:

You come across something syntactic that begins by opening a square
bracket, and you know that its semantics are: "construct a new list".
That's what's common here.

What goes*inside*  those brackets can be one of two things:

1) A (possibly empty) comma-separated sequence of expressions

2) One or more nested 'for' loops, possibly guarded by 'if's, and a
single expression

So we have two subforms of the same basic syntax. The first one
corresponds better to the output format, in the same way that a string
literal might correspond to its repr under specific circumstances.
Neither is a literal. Neither is a call to a constructor function
(contrast "list()" or "list.__new__(list)", which do call a
constructor). So what is this shared syntax? Whatever word is used,
it's going to be a bit wrong. I'd be happy with either "constructor"
or "display", myself.


Construction.  It includes an implicit constructor call and does more.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Asynchronous context manager in a typical network server

[Glenn Linderman]

On 12/18/2015 10:36 AM, Guido van Rossum wrote:


I was opted to the signal module because `signal` documentation
suggest that
it alos supports Windows while asyncio documentation states that
`loop.
add_signal_handler()` is UNIX only.


Unfortunately that's true, but using the signal module with asyncio 
the way you did is *not* safe. The only safe way is to use the 
loop.add_signal_handler() interface.


Does this mean Windows users should not bother trying to use asyncio ?

(I haven't yet, due to lack of time, but I'd hate to think of folks, 
including myself in the future, investing a lot of time developing 
something and then discover it can never be reliable, due to this sort 
of "unsafe" or "not-available-on-Windows" feature.)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Update PEP 7 to require curly braces in C

[Glenn Linderman]

On 1/18/2016 9:16 PM, Greg Ewing wrote:

Alexander Walters wrote:
I am not a core developer, but I just kind of feel its hypocritical 
to oppose always using brackets for the development of *python*


If we were being *really* pythonic, we would write all
the C code without any braces at all, and feed it through
a filter that adds them based on indentation. End of
argument. :-)



No argument, an extension:  If we were being *really* pythonic, we would 
re-implement CPython in Python, with a Python-to-C option (Cython?), and 
then we wouldn't need to worry about using braces for block delimiters.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 509: Add a private version to dict

[Glenn Linderman]

On 1/20/2016 10:36 AM, Maciej Fijalkowski wrote:

Why can't you simply use the id of the dict object as the globally unique
>dict ID? It's already globally unique amongst all Python objects which makes
>it inherently unique amongst dicts.
>
>___
>Python-Dev mailing list
>Python-Dev@python.org
>https://mail.python.org/mailman/listinfo/python-dev
>Unsubscribe:
>https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com
>

Brett, you need two things - the ID of the dict and the version tag.
What we do in pypy is we have a small object (called, surprisingly,
VersionTag()) and we use the ID of that. That way you can change the
version id of an existing dict and have only one field.
For the reuse case, can't you simply keep the ma_version "live" in dict 
items on the free list, rather than starting over at (presumably) 0 ?  
Then if the dict is reused, it bumps the ma_version, and the fallback 
code goes on with (presumably) relocating the original dict (oops, it's 
gone), and dealing with the fallout.


Then you can use the regular dict id as the globally unique dict id?  
And only need the one uint64, rather than a separately allocated extra 
pair of uint64?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 509: Add a private version to dict

[Glenn Linderman]

On 1/20/2016 12:50 PM, Brett Cannon wrote:


A global (shared between all dicts) unit64 ma_version is actually
quite
reliable -- if a program does 1,000,000 dict modifications per second,
it would take it 600,000 years till wrap-around.



But would invalidate everything, instead of just a fraction of things, 
on every update to anything that is monitored...
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 509: Add a private version to dict

[Glenn Linderman]

On 1/20/2016 4:08 PM, Brett Cannon wrote:



On Wed, 20 Jan 2016 at 15:46 Victor Stinner <mailto:victor.stin...@gmail.com>> wrote:


Hi,

2016-01-20 22:18 GMT+01:00 Glenn Linderman mailto:v%2bpyt...@g.nevcal.com>>:
> On 1/20/2016 12:50 PM, Brett Cannon wrote:
>>
>> A global (shared between all dicts) unit64 ma_version is
actually quite
>> reliable -- if a program does 1,000,000 dict modifications per
second,
>> it would take it 600,000 years till wrap-around.

I think that Yury found a bug in FAT Python. I didn't test the case
when the builtins dictionary is replaced after the definition of the
function. To be more concrete: when a function is executed in a
different namespace using exec(code, namespace). That's why I like the
PEP process, it helps to find all issues before going too far :-)

I like the idea of global counter for dictionary versions. It means
that the dictionary constructor increases this counter instead of
always starting to 0.

FYI a fat.GuardDict keeps a strong reference to the dictionary. For
some guards, I hesitated to store the object identifier and/or using a
weak reference. An object identifier is not reliable because the
object can be destroyed and a new object, completly different, or of
the same type, can get the same identifier.

> But would invalidate everything, instead of just a fraction of
things, on
> every update to anything that is monitored...

I don't understand this point.


I think Glenn was assuming we had a single, global version # that all 
dicts shared *without* having a per-dict version ID. The key thing 
here is that we have a global counter that tracks the number of 
mutations for *all* dictionaries but whose value we store as a 
*per-dictionary* value. That ends up making the version ID inherently 
both a token representing the state of any dict but also the 
uniqueness of the dict since no two dictionaries will ever have the 
same version ID.


This would work. You were correct about my assumptions.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FAT Python (lack of) performance

[Glenn Linderman]

On 1/27/2016 9:12 AM, Stephen J. Turnbull wrote:

Without that knowledge and effort, choosing a programming language
based on microbenchmarks is like choosing a car based on the
leg-length of the model sitting on the hood in the TV commercial.


+1  QOTD
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Speeding up CPython 5-10%

[Glenn Linderman]

On 1/27/2016 12:37 PM, Yury Selivanov wrote:




MicroPython also has dictionary lookup caching, but it's a bit
different to your proposal.  We do something much simpler: each opcode
that has a cache ability (eg LOAD_GLOBAL, STORE_GLOBAL, LOAD_ATTR,
etc) includes a single byte in the opcode which is an offset-guess
into the dictionary to find the desired element.  Eg for LOAD_GLOBAL
we have (pseudo code):

CASE(LOAD_GLOBAL):
key = DECODE_KEY;
offset_guess = DECODE_BYTE;
if (global_dict[offset_guess].key == key) {
 // found the element straight away
} else {
 // not found, do a full lookup and save the offset
 offset_guess = dict_lookup(global_dict, key);
 UPDATE_BYTECODE(offset_guess);
}
PUSH(global_dict[offset_guess].elem);

We have found that such caching gives a massive performance increase,
on the order of 20%.  The issue (for us) is that it increases bytecode
size by a considerable amount, requires writeable bytecode, and can be
non-deterministic in terms of lookup time.  Those things are important
in the embedded world, but not so much on the desktop.


That's a neat idea!  You're right, it does require bytecode to become 
writeable.


Would it?

Remember "fixup lists"?  Maybe they still exist for loading function 
addresses from one DLL into the code of another at load time?


So the equivalent for bytecode requires a static table of offset_guess, 
and the offsets into that table are allocated by the byte-code loader at 
byte-code load time, and the byte-code is "fixed up" at load time to use 
the correct offsets into the offset_guess table.  It takes one more 
indirection to find the guess, but if the result is a 20% improvement, 
maybe you'd still get 19%...



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Glenn Linderman]

On 2/3/2016 12:15 AM, Alexander Walters wrote:
On a very personal note (like the rest of this wasn't my personal 
issues with possibly making my life slightly more difficult), I would 
much rather see python stop touching the registry all together - but I 
have no strong argument for that. 


Me too. My opinions follow, some might call them arguments, strong or 
weak, some might call them foolishness.


I've been setting up a collection of tools and programs on Dropbox for 
far-flung project team members (users, not programmers) to share. It is 
nearly impossible to install a typical Windows program on Dropbox, 
because the installation information is partly in the installed 
directory structure, and partly in the registry. Exporting registry info 
from the machine that does the install to other machines is hard because 
other users have different paths to Dropbox. OK, for commercial 
software, installing to Dropbox probably violates some of the licensing, 
forcing each user to buy/install, but FOSS or in-house software doesn't 
have such restrictions: but still, most of it wants to use the registry, 
and Windows almost, but doesn't quite, force it to.


Portable software exists, but often is 3rd party hacks of popular FOSS 
rather than being directly supported by the FOSS development team.  
Python falls into this category. Happily, I recently found WinPython 
Zero, which hacks it (somehow) to work portably, and I've deployed it 
successfully on Dropbox. I'd rather Python were portable on its own, 
without hacks.


Portability requires not using the registry, so I agree with Alexander 
there.


Portability, as "Windows portable software" is generally defined, is 
focused on moving programs and data around on a flash drive, from one 
computer to another, and is focused on single-user, any (Windows) 
machine (with sufficient specs).


That doesn't quite fit the Dropbox environment. Most portable software 
reintroduces the idea of storing configuration information in the 
program folder, which is OK for "project" configuration info, done once, 
by one person, but not for "personal preferences".


The other thing Windows GUI lost is the concept of "current working 
directory", which hit me hard when I first started trying to set up 
project working areas on Dropbox. Many Windows programs want to run only 
one copy of themselves, in one Window, with one set of settings, and one 
"Start In" directory (which generally defaults to  the program 
directory, or sometimes to "My Documents".  This is why I went looking 
for a portable Python (and other portable things), and I finally 
realized that I was "fighting city hall" in trying to get an environment 
set up that was usable for the various teams (of users, not developers). 
Writeup at slashdot 
 for more details 
on the lack of a "current working directory" concept in Windows GUI 
programs.


The path to Dropbox folders is different for everyone, even the drive 
letter can be different.


So here are my preferences for CPython:

1) It would be best CPython itself were fully portable. That wherever 
you point the installer, it lives there, and if somehow you happen to 
execute it from that directory, it would use its invocation path as the 
basis for finding the rest of the installed pieces.


2) A script could be provided that sets the association for .py and the 
corresponding ftype to point to the python that executed the script... 
and which has a parameter to clear that back out, too. This would be to 
allow users to set a particular python for convenient script execution, 
since Windows does the association thing, instead of using #! lines.


3) A script could be provided (maybe the one Alexander referred to) that 
sets the registry so that the "apparently one true System Python" points 
to the python that executed the script... and which has a parameter to 
clear that back out, too.  This would be for compatible transition to 
the new registry-free Python versions for packages that have weird 
installers (such as Alexander alluded to). But with the registry-free 
Python available, those packages would hopefully see the future is 
registry free, and avoid requiring registry data to do installs.


4) A script could be provided to add the python that executed the script 
to the PATH, with an option to remove it.


5) A script could be provided to add the python that executed the script 
to the launcher .ini file, with an option to remove it.


6) A script could be provided to add the python that executed it to the 
Start menu, and/or Desktop icons, with an option to remove them.


Maybe scripts 2-6 are all the same one, with different options (or 
different invocation shortcuts for the clicky folk). Not everyone would 
probably need all the scripts, but none of them would be particularly 
large, such that they would be burdensome for others to ignore.


Such a collection of scripts would allow folks to

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

[Glenn Linderman]

On 2/10/2016 2:20 PM, Georg Brandl wrote:

This came up in python-ideas, and has met mostly positive comments,
although the exact syntax rules are up for discussion.

cheers,
Georg



PEP: 515
Title: Underscores in Numeric Literals
Version: $Revision$
Last-Modified: $Date$
Author: Georg Brandl
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 10-Feb-2016
Python-Version: 3.6

Abstract and Rationale
==

This PEP proposes to extend Python's syntax so that underscores can be used in
integral and floating-point number literals.

This is a common feature of other modern languages, and can aid readability of
long literals, or literals whose value should clearly separate into parts, such
as bytes or words in hexadecimal notation.

Examples::

 # grouping decimal numbers by thousands
 amount = 10_000_000.0

 # grouping hexadecimal addresses by words
 addr = 0xDEAD_BEEF

 # grouping bits into bytes in a binary literal
 flags = 0b_0011__0100_1110


+1

You don't mention potential restrictions that decimal numbers should 
permit them only every three places, or hex ones only every 2 or 4, and 
your binary example mentions grouping into bytes, but actually groups 
into nybbles.


But such restrictions would be annoying: if it is useful to the coder to 
use them, that is fine. But different situation may find other 
placements more useful... particularly in binary, as it might want to 
match widths of various bitfields.


Adding that as a rejected consideration, with justifications, would be 
helpful.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

[Glenn Linderman]

On 2/11/2016 11:01 AM, Ethan Furman wrote:

On 02/11/2016 10:50 AM, Serhiy Storchaka wrote:
> I have strong preference for more strict and simpler rule, used by
> most other languages -- "only between two digits". Main arguments:

> 2. Most languages use this rule. It is better to follow non-formal
> standard that invent the rule that differs from rules in every other
> language. This will help programmers that use multiple languages.

If Python followed other languages in everything:

1) Python would not need to exist; and
2) Python would suck  ;)

If our rule is more permissive that other languages then 
cross-language developers can still use the same style in both 
languages, without penalizing those who want to use the extra freedom 
in Python.


Ditto.

If people need an idea to shoot down, regarding literal constants, and 
because I couldn't find a Python-Non-Ideas list to post this in, here is 
one.  Note that it is unambiguous, does not conflict with existing 
binary literals, but otherwise sucks.  Please vote this idea down with 
emphasis:


Base 64 decoding literals:

print( 0b64_CjMy_NTM0_Mjkw_NQ )
325342905
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

[Glenn Linderman]

On 2/11/2016 12:22 AM, Georg Brandl wrote:

Hey all,

based on the feedback so far, I revised the PEP.  There is now
a much simpler rule for allowed underscores, with no exceptions.
This made the grammar simpler as well.


+1 overall


Examples::

 # grouping decimal numbers by thousands
 amount = 10_000_000.0

 # grouping hexadecimal addresses by words
 addr = 0xDEAD_BEEF

 # grouping bits into bytes in a binary literal
nybbles, not bytes, is shown... which is more readable, and does group 
into bytes also.

 flags = 0b_0011__0100_1110


+1 on 0b_ and 0X_ and, especially, 0O_ (but why anyone would use 
uppercase base designators is beyond me, as it is definitely less readable)



 # making the literal suffix stand out more
 imag = 1.247812376e-15_j


+1 on _j

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

[Glenn Linderman]

On 2/11/2016 4:16 PM, Steven D'Aprano wrote:

On Thu, Feb 11, 2016 at 06:03:34PM +, Brett Cannon wrote:

On Thu, 11 Feb 2016 at 02:13 Steven D'Aprano  wrote:


On Wed, Feb 10, 2016 at 08:41:27PM -0800, Andrew Barnert wrote:


And honestly, are you really claiming that in your opinion, "123_456_"
is worse than all of their other examples, like "1_23__4"?

Yes I am, because 123_456_ looks like you've forgotten to finish typing
the last group of digits, while 1_23__4 merely looks like you have no
taste.


OK, but the keyword in your sentence is "taste".

I disagree. The key *idea* in my sentence is that the trailing
underscore looks like a programming error. In my opinion, avoiding that
impression is important enough to make trailing underscores a syntax
error.

I've seen a few people vote +1 for things like 123_j and 1.23_e99, but I
haven't seen anyone in favour of trailing underscores. Does anyone think
there is a good case for allowing trailing underscores?



If we update PEP 8 for our
needs to say "Numerical literals should not have multiple underscores in a
row or have a trailing underscore" then this is taken care of. We get a
dead-simple rule for when underscores can be used, the implementation is
simple, and we get to have more tasteful usage in the stdlib w/o forcing
our tastes upon everyone or complicating the rules or implementation.

I think this is a misrepresentation of the alternative. As I see it, we
have two alternatives:

- one or more underscores can appear AFTER the base specifier or any digit;
- one or more underscores can appear BETWEEN two digits.

To describe the second alternative as "complicating the rules" is, I
think, grossly unfair. And if Serhiy's proposal is correct, the
implementation is also no more complicated:

# underscores after digits
octinteger: "0" ("o" | "O") "_"* octdigit (octdigit | "_")*
hexinteger: "0" ("x" | "X") "_"* hexdigit (hexdigit | "_")*
bininteger: "0" ("b" | "B") "_"* bindigit (bindigit | "_")*


# underscores after digits
octinteger: "0" ("o" | "O") (octdigit | "_")*
hexinteger: "0" ("x" | "X") (hexdigit | "_")*
bininteger: "0" ("b" | "B") (bindigit | "_")*


An extra side effect is that there are more ways to write zero.  0x, 0b, 
0o, 0X, 0B, 0O, 0x_, 0b_, 0o_, etc.
But most people write   0   anyway, so those would be bad style, anyway, 
but it makes the implementation simpler.




# underscores between digits
octinteger: "0" ("o" | "O") octdigit (["_"] octdigit)*
hexinteger: "0" ("x" | "X") hexdigit (["_"] hexdigit)*
bininteger: "0" ("b" | "B") bindigit (["_"] bindigit)*


The idea that the second alternative "forc[es] our tastes on everyone"
while the first does not is bogus. The first alternative also prohibits
things which are a matter of taste:

# prohibited in both alternatives
0_xDEADBEEF
0._1234
1.2e_99
-_1
1j_


I think that there is broad agreement that:

- the basic idea is sound
- leading underscores followed by digits are currently legal
   identifiers and this will not change
- underscores should not follow the sign - +
- underscores should not follow the decimal point .
- underscores should not follow the exponent e|E
- underscores will not be permitted inside the exponent (even if
   it is harmless, it's silly to write 1.2e9_9)
- underscores should not follow the complex suffix j

and only minor disagreement about:

- whether or not underscores will be allowed after the base
   specifier 0x 0o 0b


+1 to allow underscores after the base specifier.


- whether or not underscores will be allowed before the decimal
   point, exponent and complex suffix.


+1 to allow them. There may be cases where they are useful, and if it is 
not useful, it would not be used.  I really liked someone's style guide 
proposal: use of underscore within numeric constants should only be done 
to aid readability.  However, pre-judging what aids readability to one 
person's particular taste is inappropriate.



Can we have a show of hands, in favour or against the above two? And
then perhaps Guido can rule on this one way or the other and we can get
back to arguing about more important matters? :-)

In case it isn't obvious, I prefer to say No to allowing underscores
after the base specifier, or before the decimal point, exponent and
complex suffix.
I think it was obvious :)  And I think we disagree. And yes, there are 
more important matters. But it was just a couple days ago when I wrote a 
big constant in some new code that I was thinking how nice it would be 
if I could put a delimiter in there... so I'll be glad for the feature 
when it is available.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 515: Underscores in Numeric Literals

[Glenn Linderman]

On 2/11/2016 7:56 PM, David Mertz wrote:


Great PEP overall. We definitely don't want the restriction to 
grouping numbers only in threes. South Asian crore use grouping in twos.


https://en.m.wikipedia.org/wiki/Crore



Interesting... 3 digits in the least significant group, and _then_ by 
twos. Wouldn't have predicted that one! Never bumped into that notation 
before!
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com