Re: [Python-Dev] Python 2.7 patch levels turning two digit

[Steve Dower]

> Antoine Pitrou wrote:
> Le 23/06/2014 15:27, M.-A. Lemburg a écrit :
>>
>> Not sure what you mean. We've had binary wininst distributions for 
>> Windows for more than a decade, and egg and msi distributions for 8 
>> years :-)
>>
>> But without access to the VS 2008 compiler that is needed to compile 
>> those extensions,
> 
> It does seem to be available:
> http://www.microsoft.com/en-us/download/details.aspx?id=13276
> 
> What am I missing?

That's the service pack, which will only install if you already have VS 2008 
installed.

The only official source for VS 2008 these days is through an MSDN 
subscription, though there's a link floating around that will get to an ISO of 
VC 2008 Express (but it could disappear or move at any time, which would break 
the link for good).

It's also possible to get VC9 standalone through the Windows SDK for Windows 7 
and .NET 3.5, but this installer has bugs, and distutils does not detect VC 
installs properly (it only detects Visual Studio and then assumes VC). This is 
fixable with a few extra files and registry keys, but not simple.

The best answer here is making VC9 available in a long-term, unsupported manner 
(support is the main MSFT concern - simply throwing products out there and 
forgetting about them is very counter-cultural). I'm working on getting people 
to recognize the importance of keeping the old compilers available, but it's an 
uphill battle. Obviously I'll post here as soon as I have something I can 
officially share. :)

Cheers,
Steve

> Regards
> 
> Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remaining decisions on PEP 471 -- os.scandir()

[Steve Dower]

Victor Stinner wrote:
> 2014-07-20 18:50 GMT+02:00 Antoine Pitrou :
>> Have you tried modifying importlib's _bootstrap.py to use scandir() 
>> instead of listdir() + stat()?
>
> IMO the current os.scandir() API does not fit importlib requirements.
> importlib usually wants fresh data, whereas DirEntry cache cannot be
> invalidated. It's probably possible to cache some os.stat() result in
> importlib, but it looks like it requires a non trivial refactoring of
> the code. I don't know importlib enough to suggest how to change it.

The data is completely fresh at the time it is obtained, which is identical to 
using stat(). There will always be a race-condition between looking and doing, 
which is why we still use exception handling on actions.

> By the way, DirEntry constructor is not documented in the PEP. Should
> we document it? It might be a way to "invalidate the cache":
>
> entry = DirEntry(os.path.dirname(entry.path), entry.name)
>
> Maybe it is an abuse of the API. A clear_cache() method would be less
> ugly :-) But maybe Ben Hoyt does not want to promote keeping DirEntry
> for a long time?

DirEntry is a convenient way to return a tuple without returning a tuple, 
that's all. If you want up to date info, call os.stat() and pass in the path. 
This should just be a better (and ideally transparent) substitute for 
os.listdir() in every single context.

Personally I'd make it a string subclass and put one-shot properties on it 
(i.e. call/cache stat() on first access where we don't already know the 
answer), which I think is close enough to where it's landed that I'm happy. (As 
far as bikeshedding goes, I prefer "_DirEntry" and no docs :) )

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

[Steve Dower]

This sounds great, but the disable switch worries me if it's an ENVVAR=1 kind 
of deal. Those switches have a tendency on Windows of becoming "well known 
tricks" and they get set globally and permanently, often by application 
installers or sysadmins (PYTHONPATH suffers the exact same problem).

It sounds like the likely approach is a certificate name, which is fine, 
provided there's no option for "accept everything". I just wanted to get an 
early vote in against a boolean switch.

Cheers,
Steve

Top-posted from my Windows Phone

From: R. David Murray
Sent: ‎8/‎30/‎2014 6:33
To: python-dev@python.org
Subject: Re: [Python-Dev] PEP 476: Enabling certificate validation by default!

On Sat, 30 Aug 2014 14:03:57 +0200, "M.-A. Lemburg"  wrote:
> On 30.08.2014 12:55, Antoine Pitrou wrote:
> > On Sat, 30 Aug 2014 12:46:47 +0200
> > "M.-A. Lemburg"  wrote:
> >>> That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE
> >>> env vars (or, better, by specific settings *inside* the application).
> >>>
> >>> I'm against multiplying environment variables, as it makes it more
> >>> difficult to assess the actual security of a setting. The danger of an
> >>> ill-secure setting is much more severe than with hash randomization.
> >>
> >> You have a point there. So how about just a python run-time switch
> >> and no env var ?
> >
> > Well, why not, but does it have a value over letting the code properly
> > configure their SSLContext?
>
> Yes, because when Python changes the default to be validating
> and more secure, application developers will do the same as
> they do now: simply use the defaults ;-)

But neither of those addresses the articulated use case: someone *using*
a program implemented in python that does not itself provide a way to
disable the new default security (because it is *new*).  Only an
environment variable will do that.

Since the environment variable is opt-in, I think the "consenting
adults" argument applies to Alex's demure about "multiple connections".
It could still emit the warnings.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Steve Dower]

Larry Hastings wrote:
> 
> On 09/19/2014 03:31 PM, Barry Warsaw wrote:
> I think we need a Python 3.5 Release Schedule PEP.
> 
> Just checked it in as PEP 478.  It should show up here in a few minutes:
> http://legacy.python.org/dev/peps/pep-0478/
>
> Key facts:
> . Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 
> sprints.
> . Final release is September 13, 2015, just over a year from now.
>
> Comments?

Martin is no longer producing the Windows installers - that task has been 
handed to me. I'm planning to have a rewritten installer (also in the same 
repo) that should be easier to modify and maintain, as well as being able to 
produce alternative packages (such as a Python 3.5 or stdlib merge module, for 
example), though that doesn't necessarily need to go into the PEP.

I'm also considering/experimenting with installing into "Program Files" by 
default, but I suspect that isn't going to work out yet.

I'd like to move the Windows versions onto the next release of VC (currently 
"VC 14" until the branding team figures out what to call it). There isn't a 
promised RTM date for VC 14 yet, so it looks like the best available compiler 
by Beta 1 will be a "Go Live" RC. (The "Go Live" marking basically means "we 
think this is ready for use, but expect a round of minor updates/fixes soon - 
the compiler is least likely to be updated, my guess is that it'll be Visual 
Studio UI mostly).

I personally don't have any qualms about using the RC compiler for Beta 1, and 
Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed this 
topic that some people were concerned about having the final version available 
for Python 3.5 Beta. 

So far I've been building regularly with internal versions of VC and haven't 
been hitting any major issues with Python (OpenSSL has some issues, but I've 
been filing bugs on both sides so they should be worked out soon enough). My 
work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for anyone 
interested.

For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), but I 
obviously want to settle on one or the other by Beta. Last time we discussed 
it, there was strong support for changing compiler, but I have a better idea of 
the timeline now and it's tighter than I thought...

Thoughts, anyone?

Cheers,
Steve

> /arry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Steve Dower]

"This new compiler has the incredibly awesome feature of being forwards 
compatible
right? Like in 10 years stuff compiled with a newer compiler will still work?"

That's the promise at least :)

All the macros that leaked implementation details (like file descriptors) are 
now isolated so if they change it won't break existing applications. It'll 
still be possible for newer compiler versions to break them, but the design now 
obviously discourages it. There's also an official guarantee, so if it is 
broken in future it'll be treated as a bug. As much as I'd love to make solid 
promises, I can only pass on the promises that were made to me.

But yes, we should have forward compatibility with later MSVC versions, which 
will help avoid the situation where it's hard to get hold of the right 
compiler...

Top-posted from my Windows Phone

From: Donald Stufft<mailto:don...@stufft.io>
Sent: ‎9/‎23/‎2014 18:50
To: Nick Coghlan<mailto:ncogh...@gmail.com>
Cc: Steve Dower<mailto:steve.do...@microsoft.com>; 
python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] 3.5 release schedule PEP


On Sep 23, 2014, at 9:48 PM, Nick Coghlan 
mailto:ncogh...@gmail.com>> wrote:

On 24 September 2014 03:05, Steve Dower 
mailto:steve.do...@microsoft.com>> wrote:
Larry Hastings wrote:

On 09/19/2014 03:31 PM, Barry Warsaw wrote:
I think we need a Python 3.5 Release Schedule PEP.

Just checked it in as PEP 478.  It should show up here in a few minutes:
http://legacy.python.org/dev/peps/pep-0478/

Key facts:
. Beta 1 is May 24th 2015, about a month after the end of the PyCon US 2015 
sprints.
. Final release is September 13, 2015, just over a year from now.

Comments?

Martin is no longer producing the Windows installers - that task has been 
handed to me. I'm planning to have a rewritten installer (also in the same 
repo) that should be easier to modify and maintain, as well as being able to 
produce alternative packages (such as a Python 3.5 or stdlib merge module, for 
example), though that doesn't necessarily need to go into the PEP.

I'm also considering/experimenting with installing into "Program Files" by 
default, but I suspect that isn't going to work out yet.

I'd like to move the Windows versions onto the next release of VC (currently 
"VC 14" until the branding team figures out what to call it). There isn't a 
promised RTM date for VC 14 yet, so it looks like the best available compiler 
by Beta 1 will be a "Go Live" RC. (The "Go Live" marking basically means "we 
think this is ready for use, but expect a round of minor updates/fixes soon - 
the compiler is least likely to be updated, my guess is that it'll be Visual 
Studio UI mostly).

I personally don't have any qualms about using the RC compiler for Beta 1, and 
Beta 2 will almost certainly use VC 14 RTM, but I know when I proposed this 
topic that some people were concerned about having the final version available 
for Python 3.5 Beta.

So far I've been building regularly with internal versions of VC and haven't 
been hitting any major issues with Python (OpenSSL has some issues, but I've 
been filing bugs on both sides so they should be worked out soon enough). My 
work is at http://hg.python.org/sandbox/steve.dower (branch: VC14) for anyone 
interested.

For the alphas, I'm contemplating producing two builds (VC 10 and VC 14), but I 
obviously want to settle on one or the other by Beta. Last time we discussed 
it, there was strong support for changing compiler, but I have a better idea of 
the timeline now and it's tighter than I thought...

Thoughts, anyone?

It's ultimately up to Larry as RM, but I'd personally favour targeting
the newer compiler and runtime, even with the slight risk of
potentially needing to slip our schedule. There's also a fair amount
of wiggle room between the first beta and the first release candidate.

Regards,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com<mailto:ncogh...@gmail.com>   |   
Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org<mailto:Python-Dev@python.org>
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

This new compiler has the incredibly awesome feature of being forwards 
compatible
right? Like in 10 years stuff compiled with a newer compiler will still work?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Steve Dower]

> Paul Moore wrote:
> On 24 September 2014 14:16, Mike Miller  wrote:
>> It has been a supported option for just shy of 15 years on 2.X...
>> most if not all the bugs (setuptools) were fixed a decade ago, and
>> right now thousands, if not millions of people are running it under
>> Program Files right now. I can vouch for several thousand because a
>> company I work for distributes Python and pip there for its customers
>> all around the world w/o issue.
>
> One thing that I presume would be an issue. Isn't Program Files protected in
> newer versions of Windows? I haven't tested this myself, so I may be wrong 
> about
> this. So take the following with a pinch of salt.

It's protected very well in newer versions. You typically need to be an 
administrator AND have opted in to being able to modify system files without 
warning.

> Assuming so, that means that if Python is installed there, the standard "pip
> install XXX" would not work unless run in an elevated shell. We are currently
> trying to focus on a unified message for how users should install 
> distributions
> from PyPI, by using pip install.
> I'm not sure it's a good idea to complicate that message yet by adding
> provisos about managing the system Python (which is the only one most Windows
> users will use).

This is my main concern. Until pip install --user is the default (or the 
fallback if there are no write permissions on the destination), a default that 
locks users out of the simplest PyPI experience is a genuine problem. Yes, 
users can elevate to run pip, but I'd prefer pip to use elevation if it has it 
and to use per-user if not.

There also isn't a great story for per-user Python installs on Windows, but 
that becomes fairly cheap with the installer rewrite.

> I know this is only the same situation as Unix users have, but Windows users
> don't have a distro offering packaged versions of PyPI modules.
> I also know we should be moving towards promoting --user, but I don't think
> we're quite ready for that yet. And my speculation doesn't compete with your
> real-life experience, certainly. But I would suggest carefully checking before
> making the switch.

A good reason to decide early on a change like this, or at least to promote it 
as an option in 3.5 and make it the default in 3.6.

Cheers,
Steve

> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Steve Dower]

Donald Stufft wrote:
> One thing about *nix is even though you can’t write to your normal Python
> install location without root, invoking pip with permissions (assuming you 
> have
> them) is as easy as prefacing it with ``sudo`` in most cases. Does Windows 
> have
> an equivalent or do you need to launch a whole new shell?

Unfortunately not. The "easy way" is for the executable to declare that it 
needs administrative privileges, and then the OS will take over and let you 
approve/reject/sign-in/etc. according to your settings.

I don't believe this is the right solution anyway, as very many Windows users 
won't be able to do this (particularly in IT managed environments). Having 'pip 
install' do a per-user install automatically is something that will actually 
work, at the cost/benefit of not affecting other users.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Steve Dower]

> Mike Miller wrote:
> Paul Moore wrote:
>>> One thing that I presume would be an issue. Isn't Program Files
>>> protected in newer versions of Windows?
> 
> Yes, that's the feature that protects from malicious users/code editing 
> "import
> os" to run "format c:\", spam your address book, or look for credit card
> numbers, etc.
> 
> It is the same on Mac and other Unix systems, even Windows since Vista came 
> out,
> almost 10 years ago.
> 
>>
>> This is my main concern. Until pip install --user is the default (or
>> the fallback if there are no write permissions on the destination),
>>> real-life experience, certainly. But I would suggest carefully
>>> checking before making the switch.
> 
> Windows users know how to elevate now, especially developer types.

Sure, but there are plenty of not-quite developer types using Python, plenty 
using locked down machines, and plenty who simply don't trust arbitrary code 
when it wants to elevate. The ability to become an admin does not preclude the 
need to support non-admin users.

> It should be billed as a "feature for your protection" not something to be
> feared. Microsoft decided security was worth the pain of changing over its
> billions of users. Why not Python?
> 
> In my experience pip --user works just fine also. We use it on our unmanned
> media players successfully.

This is good to know. Maybe we aren't as far away as we think.

> We also write Windows services, which run under a system account, where it is
> imperative everything is running from a secure file system.
> 
>> A good reason to decide early on a change like this,
>> or at least to promote it as an option in 3.5 and make it the default in 3.6.
> 
> It's already an option! It always has been --> Custom install. I can't 
> remember
> a time when it didn't work perfectly.

Agreed. I mean it'll become an obvious option (one of the radio buttons at the 
start), and eventually the default option.

As a slight aside, do you enable the option to compile PYC/PYO files on 
install? Unless your users are running as admin, you won't get caching on these 
for the stdlib when it's installed into Program Files.

Cheers,
Steve

> -Mike
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Steve Dower]

M.-A. Lemburg wrote:
> 
> I'd rather be conservative here and wait for another Python release before
> switching VC versions. There are a few important questions that need answers
> before we can consider a new VC version:
> 
> * Will there be free versions available ?
> 
> * Will those free editions include the 64-bit compilers ?
> 
> * Will those free editions include the optimizing compilers ?
> 
> * Is there a roadmap for how long these free versions will remain
> officially available ?
> 
> * Are there issues compiling 3rd party libraries with it ?
> 
> E.g. the numeric and science stacks, the web stacks,
> the deployment stacks, etc.
> 
> * What license terms will the new version have ?
> 
> E.g. GPL compatibility issues, weird exceptions,
> 
> * What will the pricing structure look like ?
> 
> While core devs will get free MSDN licenses, most other 3rd party
> providers will have to buy licenses for the compiler, unless
> they can use the free versions.
> 
> An alternative would be targeting VC13 instead of VC14, in case it has good
> answers for the above questions. It's been around for a year now, so there
> should be more experience available with this version.

(Nit - it's actually VC12 a.k.a. "Visual Studio 2013" - VC13 was skipped. This 
is what happens when you have separate engineering and marketing teams :) )

I don't have good answers to all of these yet, but none of them are going to be 
any worse than for VC12. I've forwarded these questions to the people on the VC 
team who do get to choose the answers, and while I'm not expecting to hear 
specifics back from them, they are at least aware of the concerns and how 
important their product is to our community.

There will be free versions available, but I don't know what format they'll be 
in. Those free editions should include identical compilers to the paid ones - 
the cases where that hasn't been true have been bugs or due to assumptions that 
were proven to be incorrect.

The main improvement in this version is that all versions from VC14 should be 
binary compatible, and so there will always be a free compiler, but it may be 
VC15/16/etc. and not VC14.

There are certainly issues with 3rd party libraries, largely because all 
projects have a tendency to take dependencies on compiler/library internals. 
OpenSSL, for example, redefines the stdout/in/err macros based on the VC 
version, but the new definitions are no longer valid with VC14, and so they are 
fixing that. Python itself has a few issues that I have already fixed in my 
branch. There will certainly be other issues, but an advantage of starting 
early is that bugs in the compiler itself can be fixed in the compiler.

The license should not change significantly from previous versions. GPL 
incompatibilities are because the GPL wants to be incompatible with licenses 
based on different ideologies - AFAIK there's never been anything in the VC 
licenses preventing whatever redistribution license you like.

Part of my improvements to /PCBuild will help avoid the need for Visual Studio 
entirely, but the free versions should always be sufficient for building and 
debugging. I have no insight or control over the pricing structure.

Cheers,
Steve

> --
> Marc-Andre Lemburg
> eGenix.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Steve Dower]

Chris Angelico wrote:
> On Thu, Sep 25, 2014 at 6:50 AM, Steve Dower  
> wrote:
>> Donald Stufft wrote:
>>> One thing about *nix is even though you can’t write to your normal
>>> Python install location without root, invoking pip with permissions
>>> (assuming you have
>>> them) is as easy as prefacing it with ``sudo`` in most cases. Does
>>> Windows have an equivalent or do you need to launch a whole new shell?
>>
>> Unfortunately not. The "easy way" is for the executable to declare that it
> needs administrative privileges, and then the OS will take over and let you
> approve/reject/sign-in/etc. according to your settings.
> 
> When you say the executable declares this, is that a static (compile/link 
> time)
> declaration, or a run-time one? That is, could pip defer the declaration until
> it's parsed its command line args and decided that it'll be installing into 
> the
> system directory, but NOT make that declaration if it's given --user, or if 
> it's
> running inside a venv, or anything else that means it doesn't need that power?
> If so, that could actually be a very powerful feature: a user without admin
> privs (or choosing not to exercise them) is still able to install into a venv,
> but if s/he forgets to activate the venv, the "hey, I want to modify system
> files" prompt will be the alarm that says the situation is not what was
> expected. That's what happens on my Linux system - I get errors back if I 
> don't
> use sudo, but in a venv, everything works without being root.

It's a compile time option (it's specified in the manifest you can embed into 
an executable) but it is possible to explicitly launch a separate process with 
the flag. Most programs that look like they can elevate on-demand are really 
launching a background process to do the work - you can't change the user for a 
process once it's started in Windows.

This process can obviously be the same one you started with (maybe with 
different command line args), and the main observable difference is that you 
get fewer access denied errors.

So yes, pip can certainly do this, and if it's already running elevated then it 
shouldn't reprompt, but it's not entirely trivial to get this right ("are you 
denied write access to that directory because you're not admin or because it's 
on read-only media?") and it's considerably easier to try it, fail on access 
issues, but provide a flag for the user to force elevation. "pip --sudo install 
..." would be fine by me :)

Cheers,
Steve
 
> ChrisA
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 3.5 release schedule PEP

[Steve Dower]

> Paul Moore wrote:
> On 25 September 2014 17:05, Steve Dower  wrote:
>> So yes, pip can certainly do this, and if it's already running
>> elevated then it shouldn't reprompt, but it's not entirely trivial to
>> get this right ("are you denied write access to that directory because
>> you're not admin or because it's on read-only media?") and it's
>> considerably easier to try it, fail on access issues, but provide a
>> flag for the user to force elevation. "pip --sudo install ..." would
>> be fine by me :)
> 
> I thought one issue with running an elevated command line subprocess from a
> non-elevated one, was that the elevated one didn't have access to the
> non-elevated console, so it popped up its own independent console window, 
> which
> disappeared immediately the process completed (hence losing any error 
> messages).
> I definitely recall easy_install did that at one stage, and it was a real 
> pain.
> Or is that something the parent process can affect, and the cmd/easy_install
> pair just didn't do so?

I'm not sure you can do it automatically, but if you own both sides of the 
application you can set up a pipe between the two processes and let the 
unelevated side forward stdio.

Again, this isn't trivial to get right. The design for the elevation model 
seems to have focused mainly on GUI rather than console, probably assuming that 
people who need to elevate from the console will elevate the shell itself (this 
is a guess - I have no insight into how the Windows team designed this), so 
there are limitations we have to work within.

> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

Hi all,

(This is advance notice since people on this list will be interested. Official 
announcements are coming when setuptools makes their next release.)

Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). 
We've produced this package to help library developers build wheels for 
Windows, but also to help users unblock themselves when they need to build C 
extensions themselves.

The Microsoft Visual C++ Compiler for Python 2.7 is available from: 
http://aka.ms/vcpython27 

This package contains all the tools and headers required to build C extension 
modules for Python 2.7 32-bit and 64-bit (note that some extension modules 
require 3rd party dependencies such as OpenSSL or libxml2 that are not 
included). Other versions of Python built with Visual C++ 2008 are also 
supported.

You can install the package without requiring administrative privileges and, 
with the latest version of setuptools (when it releases), use tools such as 
pip, wheel, or a setup.py file to produce binaries on Windows.

Unfortunately, this package isn't necessarily going to help with building 
CPython 2.7 itself, as the build process is complicated and Visual Studio 2008 
is practically required. However, as most people aren't building CPython on 
Windows, this isn't a huge issue. This compiler package should be sufficient 
for most extension modules.

I should also point out that VC9 is no longer supported by Microsoft. This 
means there won't be any improvements or bug fixes coming, and there's no 
official support offered. Feel free to contact me directly 
 if there are issues with the package.

Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

It'll help with the numerical stack, but only a little. The devs involved have 
largely figured it out already and I can't provide a good Fortran compiler or 
BLAS library, which is what they need.

It'll be much more valuable for the small packages that have one vital C 
extension - currently those are basically unusable without a wheel or a 
compiler. Many DB and XML packages seem to fall into this category. It also 
works for Cython, so anything that uses Cython should work with just these 
compilers.

Cheers,
Steve

Top-posted from my Windows Phone

From: Christian Heimes<mailto:christ...@python.org>
Sent: ‎9/‎27/‎2014 7:19
To: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

On 26.09.2014 20:01, Steve Dower wrote:
> Hi all,
>
> (This is advance notice since people on this list will be interested. 
> Official announcements are coming when setuptools makes their next release.)
>
> Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). 
> We've produced this package to help library developers build wheels for 
> Windows, but also to help users unblock themselves when they need to build C 
> extensions themselves.
>
> The Microsoft Visual C++ Compiler for Python 2.7 is available from: 
> http://aka.ms/vcpython27

Awesome! :) Thanks a lot, Steve!

Is it possible to compile extensions from Python's numerical stack such
as NumPy, SciPy and SciKit, too?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

Plain distutils won't detect it. It would be easy enough to fix 2.7.9, but 
"update Python" is a big/impossible ask for a lot of people, whereas "update 
setuptools" is easy and also covers Python 2.6 and <3.3.

The compiler installer can't set the keys that distutils looks for without 
losing the per-user installation, and it may also corrupt actual installs of 
Visual Studio. A monkey patch via setuptools was the best way to handle this - 
covers pip and Cython and can be ported to other libraries that care but avoid 
setuptools.

Now that we have a patch, there's very limited value in fixing 2.7.9, IMO. But 
I'm willing to be convinced - we can always add a version check to the 
setuptools patch.

Cheers,
Steve

Top-posted from my Windows Phone

From: Antoine Pitrou<mailto:solip...@pitrou.net>
Sent: ‎9/‎27/‎2014 5:13
To: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

On Fri, 26 Sep 2014 18:01:31 +
Steve Dower  wrote:
> Hi all,
>
> (This is advance notice since people on this list will be interested. 
> Official announcements are coming when setuptools makes their next release.)

When you mention "setuptools", do you imply it doesn't work with plain
distutils?

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

The SDK compilers are not easily fixable (I've tried). The installation is 
basically corrupt as far as the non-x86 compilers are concerned.

The package we just put out is exactly the same files as the SDK with those 
issues fixed. There's no reason to encourage the SDK at all now (which was the 
point of the whole exercise from my point of view).

Cheers,
Steve

Top-posted from my Windows Phone

From: Antoine Pitrou
Sent: ‎9/‎27/‎2014 8:32
To: python-dev@python.org
Subject: Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

On Sat, 27 Sep 2014 14:10:48 +0100
Paul Moore  wrote:
> On 27 September 2014 14:01, Nick Coghlan  wrote:
> > I personally believe we should treat handling both this and the SDK
> > compilers properly as a platform-enablement bug for distutils and
> > ensure they work properly with the currently maintained branches
> > (including 2.7).
>
> +1

+1 as well.

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

Paul Moore wrote:
> File "C:\Apps\Python27\Lib\distutils\msvc9compiler.py", line 299, in
> query_vcvarsall
> 
> raise ValueError(str(list(result.keys(
> 
> ValueError: [u'path', u'include', u'lib']
> 
> 
> Failed building wheel for blist
> Failed to build blist
> Cleaning up...
> 
> 
> PS 15:36 [00:02] C:\Work\Projects\buildwheel
> >.\ve-2.7\Scripts\pip.exe list
> pip (1.5.6)
> setuptools (6.0.1)
> wheel (0.24.0)
> 
> This looks suspiciously like http://bugs.python.org/issue7511, which is
> worrying. Is this not something I should have expected setuptools to have
> patched?

Always embarrassing to have to admit this, but there is a bug in my code... :(

distutils validates that four environment variables are set after the 
vcvarsall.bat script - PATH, INCLUDE, LIB and LIBPATH. I neglected to set 
LIBPATH for 64-bit targets (LIBPATH is used by the compiler when compiling with 
/clr, which probably no Python extension has done ever).

If you have a dirty machine like mine, then it's probably already set globally, 
which can really mess up your testing.

I'm putting up an updated installer for the compilers that will set this 
variable, so if you haven't grabbed a copy yet hold off for the next 12 hours 
or so. Otherwise, you can set your LIBPATH variable to any valid path (or maybe 
just any value - the compiler may not even look if /clr is not passed).

Sorry,
Steve

> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

If you update setuptools to 6.0 or later, it shouldn't have any trouble 
detecting it. No need to set any environment variables.

FWIW, I put my vcvarsall.bat where I did because it's not the original version. 
All the files in VC and WinSDK after unmodified from their original release.

Cheers,
Steve

Top-posted from my Windows Phone

From: Georg Brandl<mailto:g.bra...@gmx.net>
Sent: ‎10/‎5/‎2014 3:23
To: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7

I just tried out the compiler and built wininst and wheel dists. Thanks!

distutils was *almost* fine using it, but for two snags:

* I had to set VS90COMNTOOLS

* distutils expects vcvarsall.bat in VC, while you have it in the parent dir

The first could be set by the installer of your package.  But if it is not
feasible to do so, setting an envvar is something that developers can surely
be expected to do with instruction.

The second is a little more inconvenient.  Wouldn't it be possible to put
the .bat file in the "right" folder, or if it has to be there, put *another*
one in "VC"?  (That is what I did.)

cheers,
Georg

On 09/27/2014 05:16 PM, Steve Dower wrote:
> Plain distutils won't detect it. It would be easy enough to fix 2.7.9, but
> "update Python" is a big/impossible ask for a lot of people, whereas "update
> setuptools" is easy and also covers Python 2.6 and <3.3.
>
> The compiler installer can't set the keys that distutils looks for without
> losing the per-user installation, and it may also corrupt actual installs of
> Visual Studio. A monkey patch via setuptools was the best way to handle this -
> covers pip and Cython and can be ported to other libraries that care but avoid
> setuptools.
>
> Now that we have a patch, there's very limited value in fixing 2.7.9, IMO. But
> I'm willing to be convinced - we can always add a version check to the
> setuptools patch.
>
> Cheers,
> Steve


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

>From Victor Stinner:
> I know that it's hard to replace Visual Studio. I don't want to do it right 
> now, but I would like to discuss that with you.


I have read the rest of the thread, but I want to start from this point. I'm 
probably going to run off in random directions since there are a lot of issues 
raised here (all of which I've heard before and given serious thought and 
discussion to), so please excuse the brain dump.

As one of the driving forces behind keeping MSVC a viable choice for building 
CPython, I'd be disappointed if we were to change away from it (as would my 
employer). That said, I'm involved with Python because I want to be involved - 
I was developing in Python long before Microsoft hired me and I'm incredibly 
lucky that my day job permits me to be involved in this community - and it's 
not going to directly hurt my career if everyone decides to move away from 
MSVC, so I don't believe I'm conflicted here.

The main way I'm currently trying to keep MSVC viable is by porting CPython 3.5 
to build with the latest version (VC14 - yet to be released, but there are 
previews available). My progress is in my sandbox at 
https://hg.python.org/sandbox/steve.dower (VC14 branch), I've been regularly 
testing with the latest (internal) builds, fixing issues in Python and getting 
issues fixed in OpenSSL, Tcl and VC. The known PGO bugs have been fixed for 
VC14, and one of my plans is to do some thorough testing to see if it's safe 
and worth re-enabling.

Some answers to points that were raised in this discussion:

* VC 14 Express for Desktop can build both 32-bit and 64-bit versions of CPython
 - I'm also making changes to Python's VC projects so they can build correctly 
without VS, though I don't know if we'll have a compiler only package for VC14

* The main advantage of VC14 is that the C runtime will be binary compatible 
into the future (rather than *mostly* source compatible)
 - most of the macros are now function calls and opaque types like FILE* are 
genuinely opaque

* Extensions built with VC14 or later will work with CPython built with VC14 or 
later
 - If other compilers begin to support the new CRT ABI and can match the 
calling convention, then it won't matter which compiler is used for extensions
 - The MS CRT has public sources - they ship with any version of VC14 - which 
should help 

* There are plenty of issues that prevent you from building with VC14 
currently, just like with any other compiler that isn't VC10.
 - Platforms and compilers need dedicated maintainers

I don't have any official confirmation, but my guess would be that the 64-bit 
compilers were omitted from the VC 2008 Express to save space (bearing in mind 
that WinXP was the main target at that time, which had poor 64-bit support, and 
very few people cared about building 64-bit binaries) and were not available in 
the IDE for VC 2010 Express by mistake. For building extensions, the former is 
resolved by the package at http://aka.ms/vcpython27, and the latter works fine 
since the 64-bit compiler is there, just not exposed in the IDE. Neither of 
these will be an issue with VC14 - 64-bit is far too important these days.

Cross compilation is a valid issue, but I hope that build services like 
Appveyor also help out here. There is regular talk about the PSF/PyPI providing 
something similar, though I have doubts about its feasibility under any model 
other than renting a preconfigured VM. I don't see any reason why projects 
couldn't apply to the PSF for a grant to cover Appveyor costs or the expenses 
of similar services.

As for extensions with Fortran or BLAS dependencies - the Python ABI 
requirements only extend to the Python interface. You can easily define a 
second interface within your project and build dependencies with whatever tools 
you like (and in theory if your compiler emits COFF modules, you can link 
objects from separate compilers, though that's probably easier said than done). 
When you control both sides of the ABI, the tooling is less important. It's 
just unfortunate that right now the ABI for Python is defined by the tooling - 
hopefully the VC14 CRT will help stabilise this.

Currently the official CPython 2.7 build from python.org is built with VS 2008 
Ultimate SP1 and I assume 3.4 is built with VS 2010 SP1, since I know Martin 
has access to the full version of VS. As Paul mentioned a few times, I'd also 
be quite happy to see all the effort towards building CPython with other 
compilers go towards building extensions with other compilers. On Windows, very 
very few people ever build their own binaries for anything - it's a totally 
different culture to *nix - so it's far more important that the development 
teams can build their own product. In this scenario, it's more important that 
extension developers know exactly what the ABI is so they can match it, than 
whether it's exactly the same as what their tools use by default. 

I don't think there's anything actionable 

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Is there some reason the Fortran part can't be separated out into a DLL? That's 
the C ABI Antoine was referring to, and most compilers can generate import 
libraries from binaries, even if the original compiler produced then in a 
different format.

Top-posted from my Windows Phone

From: Sturla Molden
Sent: ‎10/‎11/‎2014 7:22
To: python-dev@python.org
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

Antoine Pitrou  wrote:

> It sound like whatever MSVC produces should be the defacto standard
> under Windows.

Yes, and that is what Clang does on Windows. It is not as usable as MinGW
yet, but soon it will be. Clang also suffers fronthe lack of a Fortran
compiler, though.

Sturla

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

DLLs linked by import library at compile time (ie. not using LoadLibrary calls) 
and placed in the same directory as the .pyd should be exempt from DLL hell - 
Python already creates an activation context when importing pyds to let them 
load their own dependencies. Multiple CRTs are also okay as long as they don't 
try and share state (such as file descriptors) across boundaries.

I do understand the lack of knowledge and experience though. I've helped out in 
the past but I personally don't have the bandwidth to contribute more, nor the 
persuasiveness to get someone else to.

Top-posted from my Windows Phone

From: Sturla Molden<mailto:sturla.mol...@gmail.com>
Sent: ‎10/‎11/‎2014 9:59
To: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

Steve Dower  wrote:

> Is there some reason the Fortran part can't be separated out into a DLL?

DLL hell, I assume. Using the Python extension module loader makes it less
of a problem. If we stick with .pyd files where everything is statically
linked we can rely on the Python dev team to make sure that DLL hell does
not bite us. Most of the contributors to projects like NumPy and SciPy are
not computer scientists. So the KISS principle is important, which is why
scientific programmers often use Fortran in the first place. Making sure
DLLs are resolved and loaded correctly, or using stuff like COM or .NET to
mitigate DLL hell, is just in a different league. That is for computer
engineers to take care of, but we are trained as physicists, matematicians,
astronomers, chemists, biologists, or what ever... I am sure that engineers
at Microsoft could do this correctly, but we are not the kind of guys you
would hire :-)


OT: Contrary to common belief, there is no speed advantage of using Fortran
on a modern CPU, because the long pipeline and the hierarchical memory
alleviates the problems with pointer aliasing. C code tends to run faster
then Fortran, often 10 to 20 % faster, and C++ tends to be slightly faster
than C. In 2014, Fortran is only used because it is easier to program for
non-specialists. And besides, correctness is far more important than speed,
which is why we prefer Python or MATLAB in the first place. If you ever see
the argument that Fortran is used because of pointer aliasing, please feel
free to ignore it.


Sturla

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

[Steve Dower]

I was seeing this recently and had to run recover on my repo (not sure what the 
command line is for that - TortoiseHg had a menu). YMMV, but the symptoms sound 
the same.

Cheers,
Steve

Top-posted from my Windows Phone

From: David Bolen
Sent: ‎10/‎24/‎2014 22:01
To: python-dev@python.org
Subject: Re: [Python-Dev] XP buildbot problem cloning from hg.python.org

David Bolen  writes:

> which appears to die mid-stream while receiving the manifests.
>
> So I'm sort of hoping there might be some record server-side as to why
> things are falling apart mid-way.

Just to follow-up to myself, I get the same same error trying to do a
clone from my own personal XP machine rather than the buildbot (which
is a VM).  I've had the issue with hg 1.6.2, 2.5.2 and 3.1.2.

However, the same clones completely successfully under OSX and Linux.

So that's sort of strange.

-- David

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

(Apologies for the short reply, posting from my phone.)

"MSVC can continue
to be the default compiler used for Python on Windows, none of Roumen's
patches change that. They would merely open up the choice for packagers and
users to build CPython (and extension modules, thanks to separate patches)
with alternate compilers, in cross-compilation or otherwise."

Building CPython for Windows is not something that needs solving. The culture 
on Windows is to redistribute binaries, not source, and both the core team and 
a number of redistributors have this figured out (and it will only become 
easier with VC14 and Python 3.5).

I'd rather see this effort thrown behind compiling extensions, including cross 
compilation. The ABI is well defined enough that any compiler should be usable, 
especially once the new CRT is in use. However, there is work needed to update 
the various tool chains to link to VC14's CRT and we need to figure out the 
inconsistencies between tools so we can document and work through them.

Having different builds of CPython out there will only fragment the community 
and hurt extension authors far more than it may seem to help.

Cheers,
Steve

Top-posted from my Windows Phone

From: Tony Kelman
Sent: ‎10/‎25/‎2014 9:06
To: python-dev@python.org
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

I'm several weeks late to this discussion, but I'm glad to see that it
happened. I'm not a Python developer, and barely a user, but I have several
years of daily experience compiling complicated scientific software cross-
platform, particularly with MinGW-w64 for Windows. The Python community,
both core language and scientific package developers and users, needs to
act here. The problem is bad and getting worse. Luckily much of the work
to start solving it has already been done in bits and pieces, it needs
coordination and participation to come to a conclusion.

> Cross compilation is a valid issue, but I hope that build services like
> Appveyor also help out here. There is regular talk about the PSF/PyPI
> providing something similar

AppVeyor is better than nothing (I've been using it since beta), but it's
a far cry from build services and package management as the Linux world
knows them. Obtaining and setting up build dependencies quickly and
repeatably, and finishing the build of a complicated piece of software
such as CPython, or NumPy, SciPy, Julia (where most of my recent expertise
lies), etc. on a small single-core VM with limited memory and a restrictive
time limit is often not possible. These problems are solved within Linux
infrastructure like Koji, Open Build Service, buildd, etc.

MinGW-w64 is a mature, well-tested toolchain that is very capable of cross-
compiling a wide variety of libraries from Linux to Windows, in addition to
building conventionally on Windows for Windows. The MSYS2 collection of
MinGW-w64-compiled packages (https://github.com/Alexpux/MINGW-packages) has
been mentioned. Linux distributions including
- Fedora https://admin.fedoraproject.org/pkgdb/packages/mingw%2A/
- openSUSE https://build.opensuse.org/project/show/windows:mingw:win32
- Arch https://aur.archlinux.org/packages/?K=mingw
and others have projects for providing many hundreds of open-source
packages compiled for Windows. Debian has the cross-compilers available but
not many packages yet (https://packages.debian.org/search?keywords=mingw).

As a developer of a (compiled) open-source library or application, wouldn't
you love to be able to build binaries on Linux for Windows? With some work
and build system patches, you can. For many projects it's a simple matter of
./configure --host=x86_64-w64-mingw32. Not with CPython though. CPython is
only included in 2 of the above MinGW-w64 distribution projects, MSYS2 and
Arch. This is possible with a very, very long set of patches, many of which
have been submitted by Roumen Petrov to the Python bug tracker - see
http://bugs.python.org/issue17605 and other issues linked therein. Roumen
has done a huge amount of work, and he and others who consider the MinGW-w64
compiler important will continue to do so. (Thanks a million Roumen!)

> I could step in as maintainer for Cygwin and builds based on GCC using
> mingw* APIs.
>
> Regards,
> Roumen Petrov

A maintainer has volunteered. Others will help. Can any core developers
please begin reviewing some of his patches? Even if just to say they need
to be rebased. The lack of responses on the bug tracker is disheartening
from an outside perspective. The pile of patches accumulating in external
MinGW packaging projects is tantamount to a fork of CPython. It won't go
away since there are dedicated packagers working to keep their MinGW-w64
builds functional, even in the ad-hoc current state. The patches continue
piling up, making it more difficult for everyone - except for the core
Python developers if they continue to ig

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Ray Donnelly wrote:
> What is it that you
> are afraid of if CPython can be compiled out of the box using
> mingw/MinGW-w64? Why are you fighting so hard against having option.

I'm afraid of users having numpy crash because they're using an MSVC CPython 
instead of a mingw CPython. I'm afraid of users not being able to use library A 
and library B at the same time because A requires MSVC CPython and B requires 
mingw CPython. (I can produce more examples if you like, but the general 
concern is having a fragmented community, as I said in my previous post.)

I'm fighting against "having options" because it will suck up the precious 
volunteer time we have and direct it away from where it would be more useful, 
which is making it easier to build extensions with other compilers.

I would love to see extensions for Windows built on all platforms. I see no 
value in building Python itself for Windows from different platforms.

If other core developers agree with you that a more "pure" build of Python is 
worthwhile, then they can go ahead and merge the patches (though I suspect most 
would like to see some traction achieved on a fork first). I think it's 
important that I (as Windows build manager) make my own position clear, that's 
all.

(The rest of your email is purely unsubstantiated opinion, which is okay to 
have, but it doesn't demand any reply so I've omitted it.)

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Ray Donnelly wrote:
> On Sun, Oct 26, 2014 at 1:45 AM, Steve Dower  
> wrote:
>> Ray Donnelly wrote:
>>> Also, where are the publicly accessible specifications and other technical
>>> descriptions that MinGW-w64 would need to implement strong binary
>>> compatibility with MSVC? As a random example, those for C++ name
>>> mangling and the PDB file format would be very helpful.
>>
>> C++ name mangling is always an implementation detail and it changes from
>> version to version. Luckily, CPython is entirely in C, so that doesn't 
>> matter.
>> PDBs are another red herring - you can build a loadable PE file without PDBs.
>
> Of course C++ can be called from C and that is done in some CPython
> extensions, so it's not a red herring. If we want to talk about strong
> binary compatibility I'd expect the aim would be to intermix freely
> between compilers. We'd like people to be able to debug MinGW-w64 code
> using CDB in Visual Studio if they want to, and on the flipside, to
> have GDB able to read PDB files built by MSVC (actually there's a long
> standing problem when debugging MinGW-w64 code in GDB that stack
> unwinding out of MS built dlls is flaky at best) - so again this is
> not really a red herring. I'm also led to believe that MSVC has a very
> good optimizer so if some project wanted to build certain libraries or
> objects with that for their performance critical paths then I can see
> that as being useful to those projects and their users'.

Binary compatibility that strong is very unlikely to ever happen, and certainly
not with versions of compilers that are being actively developed. It would be 
far
too restrictive to both development teams.

The weaker compatibility of C DLL boundaries is far more achievable - we already
mostly have it, as evidenced by some Python packages working correctly with
mismatched compilers. Soon the CRT will be isolated along the same boundaries,
which is short-term pain for long-term gain.

>>
>> The full source code for the MSVCRT is available with any version of Visual
>> Studio (including the free editions, last time I checked), so feel free to
check
>> whatever you need to ensure compatibility. I've suggested to the VC team that
>> they could get in touch with the MinGW projects and offer to help them 
>> improve
>> compatibility with MSVC, but unfortunately I don't think anyone will take me 
>> up
>> on that. I'm happy to research what I can to answer specific questions, but
>> there's very little that isn't already publicly available other than direct
>> access to the devs.
>
> Under what license? We'd rather have open specifications that
> copyrighted, strictly licensed code that we can't look at for various
> tainting reasons.

As far as I can tell, it's covered by the Visual Studio license, which basically
means you can't redistribute the files (I'm not a lawyer, but I've spent plenty 
of
time talking about licenses to lawyers... not sure how much that counts for :) 
).

Most closed-source Microsoft code is not released under open-source-like 
licenses,
so there's no concept of derivative work, attribution or reciprocation, and 
that's
what appears to cover the CRT sources. "Using" the sources probably counts as 
using
VS, which may trigger some non-commercial clauses if you've got the free version
(but probably not the 30 day trial of the paid version... licenses are weird), 
but
reading them is well within the granted permissions.

The intention of including the sources is to help people with debugging... I 
don't
think it's even possible to rebuild the CRT from them. I do understand the taint
concerns though - until recently, I was operating under rules that made even 
some
documentation "unsafe"...

>> Cheers,
>> Steve
>>
>>>> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Ray Donnelly wrote:
> On Sat, Oct 25, 2014 at 11:44 PM, Paul Moore  wrote:
>> On 25 October 2014 23:22, Chris Angelico  wrote:
>>> On Sun, Oct 26, 2014 at 9:19 AM, Antoine Pitrou  wrote:
 My point is that your "Windows build" would not have the same behaviour
 as a MSVC-produced Windows build, and so testing it with it would not
 certify that your code would actually be compatible with genuine
 MSVC builds of CPython, which we will not stop supporting.

>>>
>>> So you're saying it's impossible to support two compilers?
>>
>> No, rather that Windows currently only has a single supported compiler
>> (excluding cygwin, which is essentially a different OS). Adding a
>> second compiler doesn't just involve adding support for it - which is
>> all that the people offering mingw patches are doing - but also
>> involves going through the whole CPython ecosystem locating the places
>> where there is an implicit assumption that "all Windows builds use the
>> same compiler" and fixing them. I've already pointed out where this is
>> a question for pip and wheel. Whoever wants to add support for a
>> second compiler needs to be willing to do this part of the job as
>> well.
>>
>> Handwaving arguments that "it's binary compatible" aren't enough. Prove it.
> 
> The msvcrt.dlls that MinGW-w64 depends on are those dating back to
> Windows XP SP3 / XP64. Ironically, the official Windows CPython
> doesn't come with any such crt guarantees and you must ensure that the
> same msvcr??.dll is used for *all* extensions. This puts considerable
> strain on extension developers to use the correct (or any) version of
> Visual Studio to build their extensions for CPython on Windows.

We're well aware of this, and it's a big part of why I'm currently migrating 
CPython to build with VC14, which will not have the same binary compatibility 
issues. For VC14, the entire CRT has been cleaned up and mostly hidden behind 
calls into DLLs, so provided the calling conventions match (which they must or 
everything would crash very quickly), it should be relatively easy to build 
compatible extensions with MinGW-w64.

> Also, where are the publicly accessible specifications and other technical
> descriptions that MinGW-w64 would need to implement strong binary
> compatibility with MSVC? As a random example, those for C++ name
> mangling and the PDB file format would be very helpful.

C++ name mangling is always an implementation detail and it changes from 
version to version. Luckily, CPython is entirely in C, so that doesn't matter. 
PDBs are another red herring - you can build a loadable PE file without PDBs.

The full source code for the MSVCRT is available with any version of Visual 
Studio (including the free editions, last time I checked), so feel free to 
check whatever you need to ensure compatibility. I've suggested to the VC team 
that they could get in touch with the MinGW projects and offer to help them 
improve compatibility with MSVC, but unfortunately I don't think anyone will 
take me up on that. I'm happy to research what I can to answer specific 
questions, but there's very little that isn't already publicly available other 
than direct access to the devs.

Cheers,
Steve

>> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Greg Ewing wrote:
> Nick Coghlan wrote:
>> That assumption will allow MinGW-w64 to link with the appropriate
>> MSVCRT versions for extention building without anything breaking.
>
> If that works, then the same technique should allow CPython itself to be built
> in a VS-compatible way with mingw, shouldn't it?
>
> Those objecting to a mingw-built python seem to be assuming that such a thing
> will necessarily be incompatible with VS builds, but I don't see why that has 
> to
> be the case.

That's true, and a good point that I missed. However, the main (practical) 
desire for building CPython with something other than VS seems to be to avoid 
having to be compatible with VS.

It's entirely possible that having two alternative builds of CPython would 
force everyone to be more compatible, but I think it's more likely to simply 
end up being two different worlds. Maybe I'm being unnecessarily cynical :)

Cheers,
Steve

> --
> Greg
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

> Paul Moore wrote:
> On 27 October 2014 20:45, Greg Ewing  wrote:
>> Nick Coghlan wrote:
>>>
>>> That assumption will allow MinGW-w64 to link with the appropriate
>>> MSVCRT versions for extention building without anything breaking.
>>
>>
>> If that works, then the same technique should allow CPython itself to
>> be built in a VS-compatible way with mingw, shouldn't it?
> 
> Yes.
> 
>> Those objecting to a mingw-built python seem to be assuming that such
>> a thing will necessarily be incompatible with VS builds, but I don't
>> see why that has to be the case.
> 
> No, we've been trying to establish whether the patches to build with mingw 
> were
> intended to produce such a compatible build. It's not clear, but so far it 
> seems
> that apparently that is *not* the intent (and worse, mingw-w64 may not even be
> able to build viable executables that link with msvcr100 without some heavy
> hacking, although that's still somewhat unclear).

Unless there is also opposition to moving to VC14, I'd rather see the mingw 
projects invest in linking to those libraries. I believe they'll have a much 
easier time of it than worrying about VC10, and the investment will be worth 
more in the future as the public API of the CRT stops changing.

Unfortunately, I'm not able to help out more than I've already offered 
(researching answers to specific questions). Largely because I have enough 
work-outside-work going on, but also because my employer won't like me getting 
involved with GPL'd software at all.

Cheers,
Steve

> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

"You might want to interact with Steve Dower off-list"

FWIW, I'm happy to talk specifics off list, and have already been involved in a 
number of discussions with the numpy and Scipy guys wrt figuring out specific 
technical challenges or clarifying non obvious parts of dealing with Windows. 
(As far as coding goes, practically all my spare time is taken up already, 
which is why I haven't contributed directly to those projects, much as I'd like 
to.)

Cheers,
Steve

Top-posted from my Windows Phone

From: Stephen J. Turnbull<mailto:step...@xemacs.org>
Sent: ‎10/‎28/‎2014 20:59
To: Tony Kelman<mailto:kel...@berkeley.edu>
Cc: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

Tony Kelman writes:

 > No, just hearing the words come out of my mouth they sound a little
 > nuts.  Maybe not, there are after all half a dozen or more
 > incompatible alternate Python implementations floating around. I
 > think most of them started as from-scratch rewrites rather than
 > source forks, but maybe that's irrelevant.

Well, they have different names and they clearly are not intended to
be ABI compatible, so noone expects that.  OTOH, there clearly is an
expectation among many (and not just in the Windows world, cf. all of
the distros that provide whole stacks of everything for each version
of Python) that downloaded packages will just work without
incompatibility.

 > > Well, for starters, most of python-dev would rather avoid any contact
 > > whatsoever with Windows.  I think part of the problem is that Windows
 > > developers *of* Python are *very* rare (fingers of one hand rare).
 >
 > In my opinion the MSVC toolchain makes that problem worse, as it's far
 > harder for unix developers to have any familiarity with how things
 > work.

I've used Cygwin, I've used MinGW, and I've used VC.  Sure, the former
two are GCC-based so I have a lot of muscle memory for command-line
switches.  But that's not very important; the pain of using Windows is
what drives me away from all of them.

 > But you do have involvement and core developers from Microsoft
 > which is obviously incredibly important. Maybe even mandatory for
 > Python on Windows to be viable in your eyes.

No, I don't think that's true.  What I think *is* true is that most
developers on Windows do have access to Microsoft tools, so we do need
to provide compatibility with them.  As you say, the VC toolchain is
not all things to all men, but what's visible to python-dev makes it
more important than Cygwin or MinGW.  See Paul Moore's post about
communications between the scientific Python community and python-dev
for what I mean by "visible".

 > > It should be evident by now that our belief is that the large
 > > majority of Windows users is well-served by the current model
 >
 > This is not the case at all in the scientific community. NumPy and
 > SciPy put in a lot of extra work to come up with something that is
 > compatible with the MSVC build of CPython because they have to, not
 > because they're "happy to" jump through the necessary hoops.

Agreed.  This is well-known to python-dev, and AFAICS it *is* a
concern for us.  However, as Paul points out, a bridge needs to be
built.  Your posts have been a contribution to building that bridge,
for sure, but more work on the bridge is needed.

 > Do python-dev and numpy-discussion not talk to one another?

Exactly the issue here.  To resolve this, we need to talk more.
Unfortunately, I'm not one to help build the bridge as I haven't
developed on Windows at all since about 2003.

 > I'm going to move the "extensions with MinGW-w64" part of this
 > conversation over to numpy-discussion,

As far as I can tell, that's a good idea right now.  They have the
need, they have the expertise, both of which are somewhat lacking
here.

 > Okay. I'll table the discussion with python-dev for now then.

I hope you'll be able to come pick it back up at some point.  You
might want to interact with Steve Dower off-list, as he's spearheading
the effort to move the official builds to the "stable ABI" version of
MSVC.  Once that's in place, the MinGW guys will have a stationary
target which is up to date to shoot at.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

Antoine Pitrou wrote:
> On Wed, 29 Oct 2014 11:07:53 -0400
> Tres Seaver  wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>> 
>> On 10/29/2014 10:31 AM, R. David Murray wrote:
>> 
>> > If you are writing code targeted for Windows, I think you are very 
>> > likely to have an MSDN subscription of some sort if your package 
>> > includes C code.  I'm sure it's not 100%, though.
>> 
>> My experience with distributing distributions-with-extensions 
>> indicates that the vast majority of Windows developers who are 
>> "downstream" users for those distributions *cannot* build them from 
>> source:  if there is no MSI / bdist_win (maybe now wheel), they won't use 
>> the project.
>>
>> (Note that "having an MSDN subscription" is not the same as "knowing 
>> how to configure which compiler such that it can bulid extensions 
>> against an installed Python binary").
> 
> I don't think you have to configure anything. Just install the right Visual 
> Studio version and it's done.

The tricky part here is the *right* Visual Studio version. As many have noted, 
there were bugs/missing components in some of the older Express editions (like 
the 64-bit compiler integration), and even the compiler pack we released 
recently doesn't actually help building Python itself, though it's good for 
extensions. In general, VS 2008 Professional and VS 2010 Professional are 
easiest to set up if you can get them, the C++ Express editions are fairly easy 
to set up, and the Windows SDK is difficult but possible (and won't currently 
build CPython either, as the build depends on VS - I'm also fixing this for 
3.5).

My ideal target (Utopia refined to be achievable) is for python-dev to handle 
the Python binaries themselves (already done) and to make them easy to bundle 
with applications/etc (I'm working on this for 3.5), and for PyPI to include 
pre-built wheels for Windows where necessary.

Note that I don't require package developers to build their own wheels, though 
I think that they are generally the right people to be doing it. It would be 
nice to create a culture of delegation, so that "someone" could be a trusted 
builder for a range of packages (for example, I'd love it if 
Continuum/Enthought/similar could provide their builds of numpy/scipy as wheels 
and those projects would be willing to have them be the official PyPI 
downloads). This is roughly how the python.org installers are handled, and 
while it may cause some lag between source and binary releases, I think the 
release cycles are slow enough that most users would only notice that "pip 
install numpy" now works.

Cheers,
Steve

> Regards
> 
> Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Status of C compilers for Python on Windows

[Steve Dower]

"For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)"

You could link to visualstudio.com for the trial versions, and maybe to a 
page/post about the PSF's grants process if such a page exists.

"And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly"

These are still at 
http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs#DownloadFamilies_4,
 which is the main download page. Hopefully they don't go away before 3.5, but 
I have no control over that unfortunately.

The link I posted for 2.7 was aka.ms/vcpython27, which is a redirect that I own 
and can update if necessary.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Moore<mailto:p.f.mo...@gmail.com>
Sent: ‎10/‎29/‎2014 15:48
To: Ethan Furman<mailto:et...@stoneleaf.us>
Cc: Python Dev<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] Status of C compilers for Python on Windows

On 29 October 2014 22:19, Ethan Furman  wrote:
>> Yeah, I wondered about that. I'll work up a patch for that. But the
>> more I think about it, it really is trivial:
>
> I am reminded of an interview question I was once asked which was prefaced
> with: "Here's an easy one..."
>
> My reply was, if you know the answer, it's easy!

Yeah, I know what you mean. My take on this is that I agree it's not
easy if you don't know and can't get access to the information, but if
you can, there's very little to it.

>> - For non-free MSVC, install the appropriate version, and everything just
>> works.
>> - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7
>> package and everything just works as long as you're using setuptools.
>> - For 32 bit Python 3.2-3.4, install Visual Studio Express and
>> everything just works.
>> - For 64 bit Python 3.2-3.4, install the SDK, set some environment
>> variables, and everything just works.
>> - For Python 3.5+, install the current Visual Studion Express and
>> everything just works.
>
>
> I would suggest
>   - where to get these packages

Conceded. Working out how to point people at stuff on MSDN is a
challenge, things seem to move around. Maybe Steve Dower could help
here with canonical URLs for some of them (IIRC, he provided one for
the VS compilers for Python 2.7 package when it was announced).

For the paid versions, I'm going to assume that anyone who paid for a
compiler and doesn't know where their copy is, probably can't be
helped ;-)

And of course there's the awkward problem that VS 2010 Express is
unsupported and therefore no longer available from *any* official
location. I can't solve that, sadly.

>   - where to get any dependencies

There are none. I could state that explicitly, I guess.

>   - any options to [not] specify during install

I'll have to go through the installs again just to be sure, but I'm
pretty certain "Select the default for everything" is correct.

>   - what environment variables to what value

None, except for the SDK which I did say I needed to test out and
cover in more detail.

>   - where one should be at when one starts the compile process

I don't understand this. It's just "pip wheel foo" to build a wheel
for foo (which will be downloaded), or "pip wheel ." or  "python
setup.py bdist_wheel" as you prefer for a local package. That's
standard distutils/setuptools/pip extension building. I don't propose
to cover that, just how to *set up* the environment.

With the sole exception of the SDK case, once installed, everything
just works everywhere, nothing to set up, no special environment to
configure. Start up a new cmd or powershell console (or use the one
you're already in) and go. Maybe Unix users expect more complexity
because it's not that simple on Unix? But I thought it was - install
the appropriate OS packages and that's it?

> and, of course, a gotchas section for uncommon but frustrating things that
> might go wrong.

Hmm, I see your point here, but I'm not sure what I might include. You
*can* get in a mess [1] but generally I don't as I'm an experienced
Windows user. I also don't want to offer to debug and fix everyone's
problems in getting things set up, so offering to collect and document
"common issues" that I've seen is impractical. Maybe a section
entitled "Common Issues and Their Solutions", with some placeholder
text saying "If you have any issues installing any of the compiler
packages mentioned, please document what

Re: [Python-Dev] Impact of Windows PowerShell OneGet ?

[Steve Dower]

>> Most likely, OneGet won't replace pip/PyPI, any more than apt or yum
>> does; but it may be worth having Python itself available that way.
>> That might simply mean having someone package up Python and put it on
>> an appropriate server, or maybe python.org could end up hosting a
>> repo.
> 
> Python is already available in Chocolatey:
> 
> https://chocolatey.org/packages/python
> 
> Given that OneGet intends to support Chocolatey as a repository, it might just
> work already. All python.org would have to guarantee is stable URLs for the
> Python msi files.

I'd like it if we guarantee stable URLs anyway. The 3.5 installer will have a 
single flag to switch between building a full bundle (~23MB) or a tiny 
downloader (<500KB), but the latter option is only viable if the URLs are 
stable. I can also include options in the downloader for 32/64-bit versions and 
debug symbols, which would really reduce the choices for a user (and yes - the 
entire installer is still automatable and I'll write up docs to go with it for 
sysadmin scenarios).

(I believe that OneGet does support dependencies, since one of the intended 
uses is setting up dev environments. If it gets backported, I'll see about 
setting up a Python build environment in there, unless someone else does it 
first.)

Cheers,
Steve

> Regards,
> Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Who's using VS/Windows to work on Python?

[Steve Dower]

Just wondering who is regularly/occasionally using VS 2010 to work on Python?

Also, who currently owns the Windows buildbots and are you willing/able to add 
a VS 2015 Preview installation (or give me access so I can do it)? I can also 
set up a VS 2015 buildbot, but the old ones will break if the projects change 
(I should do this anyway to at least become familiar with how the buildbots 
work...)


I want to get a feel for who will be disrupted when I merge in the changes to 
build with VS 2015, which will help influence when that merge occurs. VS 2015 
Preview was just released yesterday, and while it's not perfect yet, it's 
isolated enough from VS 2010 that there won't be any issues installing them 
both (I can't say the same for VS 2013...). If people are okay with this, then 
we can consider merging the changes sooner. Once VS 2015 RC is released, I'll 
be looking to merge asap., but I don't see a strong need to do it before then 
other than a longer bake time.

(In any case, the old project files will still be there, but right now it's 
easier to handle when those change. Once they are in their own directory, 
changes to the VS 2010 projects will need to be duplicated by hand in the VS 
2015 projects, and while the converse is true, it's not as critical.)

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Who's using VS/Windows to work on Python?

[Steve Dower]

Tim Golden wrote:
> Are the changes for VS2015 likely to hamper a build against VS2010?

I hadn't tried it, but it looks like it works fine with VS2010 (with a few 
extra warnings, but nothing fatal as far as the build is concerned). I'm a 
little surprised, tbh, but not overly upset :)

Guess I can go ahead and start merging the changes (via issues w/patches) 
whenever I like then. The most painful change will probably be moving the x86 
outputs from PCBuild to PCBuild\win32, but I'll get that into the buildbot 
scripts and it should be fine.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Who's using VS/Windows to work on Python?

[Steve Dower]

David Bolen wrote:
> Steve Dower  writes:
> 
>> Also, who currently owns the Windows buildbots and are you
>> willing/able to add a VS 2015 Preview installation (or give me access
>> so I can do it)? (...)
> 
> I've got several of the Windows buildbots, and could add this. Is there 
> benefit
> to just starting with one (I've got XP, Win7 and Win8) first to save some 
> time?

Starting with just the Win7 or Win8 one would be fine. Python 3.5 won't support 
XP, and VS 2015 doesn't support XP (though I believe it will still be able to 
build for XP, just not *on* XP).

> My only real concern is to verify that this will co-exist properly with the
> existing VS installations and buildbot build process. I note that the download
> page says not to use on production computers. That's probably just a CYA, and 
> I
> don't mind if VS itself has issues, but if it risks compromising other aspects
> of the buildbot, then I think testing on a different machine would be 
> preferred.

VS 2010 should be fine. The most likely issues are with VS 2013 (for teams that 
haven't updated their setup authoring yet), but these should have been ironed 
out already.

> Is anything aside from VS 2015 itself (e.g., the Ultimate 2015 download on the
> preview page) needed? I'm assuming not.

Nope, that's it. I'll make sure there's a free version that will work 
eventually, but right now only Ultimate is available. I will make sure to let 
you know when the merge happens, but if VS 2015 is already there then that 
should be smooth anyway.

Thanks,
Steve

> -- David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows Dedicated Mailing List

[Steve Dower]

I'd like to keep development *of* Python here, regardless of platform. 
Otherwise all the Linux and Mac people might forget about us :)

Top-posted from my Windows Phone

From: Tim Golden
Sent: ‎11/‎14/‎2014 7:11
To: Shorya Raj; Python 
Dev
Subject: Re: [Python-Dev] Windows Dedicated Mailing List


On 14/11/2014 08:34, Shorya Raj wrote:
> Hello
> I have been following the lists for a while, and have noticed that there
> do not seem to be any sort of specifics for Windows. Now, as someone
> developing python applications (and hoping to develop python itself on
> Windows), I think it may be prudent to consider adding a Windows
> specific mailing list regarding development on Windows, whether it is
> development of Python on windows, or developing Python applications on
> Windows. Please let me know what you guys think - if there is consensus,
> it would be great to get it up and running, and I would be more than
> willing to take steps from my side as required.

That would be this:

https://mail.python.org/mailman/listinfo/python-win32

TJG

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Who's using VS/Windows to work on Python?

[Steve Dower]

Nick Coghlan wrote:
> With the recent (excellent) announcements around switching from the current
> Visual Studio Express model to the far more full featured Visual Studio
> Community approach, what will the situation be in relation to using Visual
> Studio Community edition to work on CPython?

It "just works" :)

VS Community is indistinguishable from VS Pro apart from some of the branding 
and a few features so obscure that I hadn't heard of or used them until I was 
told they were missing. Also the license, which allows for free use on 
open-source projects regardless of the company where you work (the limits are 
for commercial use at companies that can afford the product).

> This thread, and Brian's post regarding renewing MSDN subscriptions made me
> realise we're going to need to be able to explain that to new Windows based
> contributors.

Replace "Express" with "Community" everywhere (once the project migrations go 
in, since VS Community *2013* isn't a lot of help right now).

I don't know exactly what the situation will be with VS 2015, but there will 
either be a free Community or a free Express for Desktop (without the 64-bit 
bug that was in VS 2010 Express), either of which will be fine for working on 
Python.

Cheers,
Steve

> Cheers,
> Nick.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Steve Dower]

> Antoine Pitrou wrote:
> On Fri, 21 Nov 2014 13:29:11 +
> Brett Cannon  wrote:
>>> If that "must be self-hosted" constraint is removed, then the
>>> obvious candidate for Mercurial hosting that supports online editing
>>> + pull requests is the PSF's BitBucket account.
>>>
>>
>> There's also CodePlex and (ironically) SourceForge for open-source hg
>> hosting.
> 
> I hope nobody's gonna suggest SourceForge seriously for anything. It's the
> SharePoint (or Lotus Notes) of software forges.

SourceForge can be Lotus Notes; CodePlex is SharePoint :-)

BitBucket is a much better option than either of those. Since I discovered 
hggit (and got it working), I'm less opposed to github than I have ever been, 
but I still prefer BitBucket. It just feels more pragmatic and less like a 
cattle market.

I'm not hugely for/against moving the repo, though I do like the general idea 
of reducing the amount of work committers have to do.

Cheers,
Steve

> Regards
> 
> Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PCBuild project updates

[Steve Dower]

​Hi all

Just attracting some attention to http://bugs.python.org/issue22919 for those 
of us who build Python out of the PCBuild folder. More details/patches there, 
but the tl;dr is

* Still works with VS 2010 (and now VS 2013 and VS 2015 too)
* Build time should be reduced
* Tools\buildbot\*.bat are also updated, so buildbots shouldn't notice any 
change

If you're concerned/interested, I'll be watching the tracker for comments. 
Ideally I'd like at least a couple of people to test build with whatever VS 
version they have, but if nobody can do that then I'll assume that nobody will 
notice anything break sooner than me :)

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Issue 22919: Update PCBuild for VS 2015

[Steve Dower]

FYI, I've just committed these changes (http://bugs.python.org/issue22919).

There shouldn't be any immediate failures, as the updated projects will still 
build with VS 2010, but our Windows developers/buildbots can migrate onto the 
later tools as they feel comfortable.

I know there are at least a few bugs coming out of the new compiler, so I'll be 
tracking those down and fixing things. Feel free to nosy me (or Windows) on the 
issue tracker if you find anything.

Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

[Steve Dower]

This is also my approach, and the one that I'm encouraging throughout Microsoft 
as we start putting out more Python packages for stuff.

Top-posted from my Windows Phone

From: Barry Warsaw
Sent: ‎12/‎13/‎2014 7:19
To: python-dev@python.org
Subject: Re: [Python-Dev] Python 2.x and 3.x use survey, 2014 edition

On Dec 13, 2014, at 12:29 AM, Donald Stufft wrote:

>For what it’s worth, I almost exclusively write 2/3 compatible code (and
>that’s with the “easy” subset of 2.6+ and either 3.2+ or 3.3+) and doing so
>does make the language far less fun for me than when I was writing 2.x only
>code.

For myself, the way I'd put it is:

With the libraries I maintain, I generally write Python 2/3 compatible code,
targeting Python 2.7 and 3.4, with 2.6, 3.3, and 3.2 support as bonuses,
although I will not contort too much to support those older versions.  Doing
so does make the language far less fun for me than when I am writing 3.x only
code.  All applications I write in pure Python 3, targeting Python 3.4, unless
my dependencies are not all available in Python 3, or I haven't yet had the
cycles/resources to port to Python 3.  Writing and maintaining applications in
Python 2 is far less fun than doing so in Python 3.

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] libffi embedded in CPython

[Steve Dower]

Maciej Fijalkowski wrote:
> After reading this http://bugs.python.org/issue23085 and remembering 
> struggling
> having our own patches into cpython's libffi (but not into libffi itself), I
> wonder, is there any reason any more for libffi being included in CPython?

We use it for ctypes, so there's certainly still a need. Are you asking whether 
we need a fork of it as opposed to treating it like an external (like OpenSSL)?

> Cheers,
> fijal
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

I've put together a short post showing where I've been taking the Windows 
installer for Python 3.5, since I know there are interested people on this list 
who will have valuable feedback.

http://stevedower.id.au/blog/the-python-3-5-installer/

Nothing is merged in yet and everything can still change, so I'm keen to hear 
whatever feedback people have. I've tried to make improvements fairly for 
first-time users through to sysadmins, but if I've missed something big I'd 
like to hear about it before we get too close to alpha 1.

Cheers,
Steve 
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

Thanks for all the comments. I've responded all at once, since I don't think 
any of the points raised are going to immediately turn into intense discussion. 
Feel free to trim the rest of the post if you want to respond to just one of 
the points.

Donald Stufft wrote:
> Does signing the .exe’s that ship with Python effect pip at all? I’m guessing
> not since pip.exe is actually a generic .exe that we use for any project and 
> is
> shipped inside of the .whl files but I figured i’d ask.
>
> Depending on the answer above, does it make sense to sign the generic .exe 
> (does
> that even work if we rename it?) which will get used for anything that uses
> entry points on Windows?

Actually, I don't think the generic .exe can be signed, since embedding the 
script within the file will invalidate the certificate. It's better not to have 
the certificate at all in this case. If we were still using the separate 
pip-script.py file then we could sign it.

> Is there any plan to backport this to 2.7 (presumably after some experience is
> had with it in 3.5)?

No. I'm very much in the "2.7 is finished" camp. Hitting rebuild every now and 
then for security updates is the extent of my interest (and in large part 
that's because if nobody is producing builds then I'll probably end up having 
to produce the builds to run on Azure anyway).



Antione Pitrou wrote:
> One comment: I would find it a bit confusing if the default install path
> changes when using the customized install. OTOH, maybe you can't choose
> another default there.

It's a tough one to choose, honestly. I really don't want separate "customize" 
buttons for per-user and all-user installs (too many up front decisions, 
especially for people who really should just choose the first option). My 
reasoning for having "C:\Python35" as the default there is to simplify the 
"just give me the old style install" case, but that may turn out to be a bad 
reason. I'm not sure yet.



ChrisA wrote:
> You talk of installing by default into Program Files, and having a
> separate per-user installation mode. How do these two installation
> targets interact? Suppose someone installs 3.5 globally, then installs
> 3.6 for self only? Or installs 3.5.1 for self only? I would normally
> expect a per-user installation to trump a global one, but this could
> make for a lovely dep-hell on a system that's used by one person, who
> then isn't sure what was installed as admin and what wasn't.

Yeah, it gets a little messy, especially if you keep adding all the Python 
versions to PATH (Windows will always put per-user PATH entries at the end). 
The py.exe launcher handles this best, and the system version will always be 
found first.

As for versions, 3.5 vs 3.6 should be distinguished explicitly ("py -3.5" vs 
"py -3.6") and 3.5.0 vs 3.5.1 can only be installed at the same time if one is 
system-wide and the other is per-user. In this case, the per-user one will be 
used by py.exe, even if it is older than the system-wide one.

Compared to the current situation, the per-user and all-user installs do not 
interfere as much.

> (Also: I'm assuming that pip would require admin privs if Python is in
> Program Files, and won't require admins if it's per-user, right? Same
> as the Python installer itself?)

Correct. The pip guys are already looking into handling installs in this case 
since it matters on *nix, and having read-only Python installs on Windows will 
only make it more similar. It seems likely in this case that the user 
site-packages folder will be used if your user can't write to the shared 
site-packages. If you can write to it (either per-user install or running pip 
as admin) then you'll install to that folder.

I am fully expecting to find the most issues in this area once we change the 
install location, and I don't think we can fix all these issues without 
actually releasing the change. So it's a bit of a risk, but one I feel is worth 
it. Of course, if the community and release manager disagree, I'll happily back 
it out and we can focus on testing it directly for a bit before inflicting it 
on our users.



Glenn Linderman wrote:
> But here's another idea: automatically keep all the .msi and .cab files used 
> for
> the first installation of Python with it in the directory from which it runs
> (naming convention... prefix them all with python-3.5.0a1..(msi|cab) It
> is very likely that a reinstall will use the same components (if more are 
> needed
> on a later install, add them to the directory also). And a good naming
> convention makes it obvious what to delete when done with the installer.

Unfortunately, I'm fairly restricted on naming convention. For some reason 
Windows Installer requires CAB files to be 8.3 names, where some of the 8 are 
reserved, so the names are pretty meaningless.

However, Windows does automatically cache all the files (typically in 
C:\ProgramData\Package Cache) so that repairs and uninstalls don't need to 
redownload anything, and

Re: [Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

Chris Angelico wrote:
> On Mon, Jan 5, 2015 at 9:56 AM, Steve Dower  wrote:
>> (Windows will always put per-user PATH entries at the end). The py.exe
>> launcher handles this best, and the system version will always be found 
>> first.
>>
>> As for versions, 3.5 vs 3.6 should be distinguished explicitly ("py -3.5" vs
>> "py -3.6") and 3.5.0 vs 3.5.1 can only be installed at the same time if one 
>> is
>> system-wide and the other is per-user. In this case, the per-user one will be
>> used by py.exe, even if it is older than the system-wide one.
>>
> 
> Wait, what?
> 
> If I'm reading this correctly, PATH is set such that an all-users
> version trumps a self-only version, but when you use the py.exe
> launcher, it's the other way around? That seems extremely confusing.
> But if that's not a deliberate design decision and isn't enforced by
> external code, would it be possible to simply invert that for PATH,
> and have a per-user installation always be found ahead of a
> system-wide one? That would make reasonable sense, if it can be done.

Unfortunately, Windows enforces the PATH ordering. It constructs the PATH from 
two registry keys, one is the system-wide value (that requires administrative 
privileges to modify) and it is followed by the user's value (that does not 
require administrative privileges). This is probably for security reasons and 
can't be changed.

PATH also suffers from including the most-recently installed Python version 
first, rather than the most recent version. Basically, py.exe gives the 
behaviour we want and PATH simply can't do it.

I think this means the best way to make multiple versions work properly is to 
rename python.exe to python3.5.exe, then install the launcher as python.exe and 
python3.exe (with some logic to look at its own name) so it can resolve the 
right version. Maybe we can even extend the launcher to resolve launchers in 
Scripts (pip.exe, etc.) and have consistent rules there too?

Or perhaps this is more trouble than it's worth, and we should let "py.exe" 
have a fixed set of rules and let PATH work as people have come to expect. 
Thoughts?

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

Mark Lawrence wrote:
> If I'm reading this correctly it means that py.exe gets picked up from PATH so
> it could be 32 or 64 bit. Does this mean that the launcher could be or needs
> enhancing so 32 or 64 bit can be selected? I'm not sure if anything can be 
> done
> about pyw.exe, perhaps you (plural) can throw some light on this for me.

We only ever install a 32-bit launcher (including with the current installer, 
though it doesn't quite work properly), so this isn't really an issue. To 
explicitly request a 32-bit interpreter you can pass "-3.5-32" - without the 
"-32" suffix you'll get 64-bit if it's there and 32-bit otherwise.

Paul Moore wrote:
> Steve is in essence saying that it's not possible to sanely manage PATH as 
> part
> of the new installer, but that py.exe makes that unnecessary.

It's actually not possible to sanely manage PATH from any installer - it really 
needs to be handled by a user directly (though I can't ever bring myself to 
tell anyone to use the UI for it). The old installers were less susceptible 
because they didn't support per-user installs, but you'd still get the "last 
install Python wins" behaviour.

> My point is that while py handles the interpreter, it doesn't handle things 
> like
> pip (unless we change the standard usage instructions to say "py -m pip" is 
> the
> supported approach on Windows).

Or we make pip3.5.exe launch against 3.5 and pip.exe and pip3.exe behave like 
the launcher, regardless of where they run from.

For example, say I have Python 3.5 and Python 3.6 installed and 
PATH=C:\Python35;C:\Python35\Scripts;C:\Python36;C:\Python36\Scripts;... (yes, 
those are the 'wrong' way around). If I just type "pip" it will run 
C:\Python35\Scripts\pip.exe, which may or may not be what I expect.

If pip behaved like the launcher, it would find the latest installed version 
(3.6) and launch pip3.6.exe, which would always run against 
C:\Python36\python.exe. (How does it do this? No idea yet. Technical details...)

This might also mean you could write "pip -3.5 install ..." to run against a 
specific version, but that doesn't have to be supported if it's likely to break 
scripts by stealing an argument. Since we'll install "pip", "pip3" and 
"pip3.5", the options are already there.

Would we need "pip3.5-32" as well? This could get out of hand pretty quickly. 
Perhaps the launcher could include "usepy -3.5" to update the user's PATH for 
the current session? Would that get any more use? I don't know, but it feels 
unlikely. There have been multiple requests for the Add to PATH option to be 
enabled by default in the installer, but frankly it concerns me too much to do 
so. There are too many effects that most users won't expect.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

Ethan Furman:
> On 01/04/2015 02:56 PM, Steve Dower wrote:
> 
>> ChrisA wrote:
>>> You talk of installing by default into Program Files, and having a 
>>> separate per-user installation mode. How do these two installation 
>>> targets interact? Suppose someone installs 3.5 globally, then 
>>> installs
>>> 3.6 for self only? Or installs 3.5.1 for self only? I would normally 
>>> expect a per-user installation to trump a global one, but this could 
>>> make for a lovely dep-hell on a system that's used by one person, who 
>>> then isn't sure what was installed as admin and what wasn't.
>> 
>> Yeah, it gets a little messy, especially if you keep adding all the 
>> Python versions to PATH (Windows will always put per-user PATH entries 
>> at the end). The py.exe launcher handles this best, and the system 
>> version will always be found first.
> 
> py.exe should *not* follow this behavior. User installs should be searched
> first, then system installs -- otherwise, what's the point in having a
> user-install? Filling up the gobs of disk-space we now have available?

Typo on my part. The py.exe launcher will always find the *user* install first. 
Searching PATH will always find the system version first.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

Paul Moore wrote:
> tl;dr We should have "Add this Python to PATH" as a user choice on the initial
> installer screen, applicable to whichever install type the user chooses. 
> Details
> as to why are below.

I agree. I'll work this up before alpha 1. (FWIW, it defaults to unselected.)

Displaying a warning about other Pythons already being on the path at this 
point is trickier, but I can probably figure something out.

> For myself, I'll probably have to use the custom install to get the control I
> want. But that's suboptimal, as I probably *want* to use the new locations, 
> and
> having to override the "Custom" default of C:\Python35 to specify "Program
> Files" is a nuisance. And I bet I'll install a 32-bit Python in "C:\Program
> Files" rather than "C:\Program Files (x86)" at least once, and probably more
> than once...

Inclined to agree, and it is possible to make the default on the customise page 
switch with the Install as Administrator checkbox until it's been manually 
edited. My main concern is people who want it the old way, but they may not 
actually exist. Perhaps making the old way trickier is the best way to flush 
them out sooner...

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

David Anthoff wrote:
> It would be great if the new installer supported silent, portable installs.
> What I have in mind is a silent installation that drops all the necessary 
> files
> for a working python into a folder, but does not put ANY file anywhere else 
> and
> does not register anything anywhere on the system. So no PATH modification, no
> registering of this install as one of the available python interpreters, no
> uninstall entry or anything like that. By "all the necessary files" I mean 
> that
> it drops things like the MSVC runtime dlls etc into that folder into which I'm
> installing python, but again doesn't try to install things like that system or
> even user wide.
>
> I'll give you my very specific use-case, but I think this might be useful for
> others as well: we are trying to build an installer for some other product X
> that internally requires a python instance to work. This python instance 
> should
> NOT be visible to anything other than product X, i.e. no user should ever 
> start
> it directly or anything like that, it really is an implementation detail of
> product X. If we had an official python installer that supported silent,
> portable installs, I could just call that python installer inside the setup
> program for X, and it would drop a fully working python installation into a
> sub-directory of the install directory of product X. And we would be happy :)

I'll look into this, but it probably isn't going to work as part of the 
installer. I have previously looked into being able to install arbitrary 
side-by-side copies of Python, but that's near impossible as well. Windows 
Installer doesn't really let you just copy files - it isn't part of its 
intended functionality. It isn't too difficult to build custom MSIs with 
certain parts of Python (such as the DLLs and the standard library, but no 
docs, headers or EXEs) in a way that won't conflict with other installs, but 
you're still using an MSI here which is not necessarily ideal.

The easiest way is still going to be to install a copy of Python without 
administrative privileges, which *currently* will drop all the files you want 
into the main install path (wherever you customise that to be), and then copy 
them directly into your installer, but that may change depending on the 
redistribution requirements of the CRT. There are still limitations within 
Python itself that would be nice to fix, such as looking at the registry too 
soon when it could resolve directories relative to its own location, but these 
are independent from the installer. If you don't need python.exe, then it 
should be fine as long as you put pythonXY.dll alongside the executable loading 
it (otherwise a system version may take priority).

As you can see, this is a fairly deep hole with lots of caveats :)

We could release a ZIP file containing all the Python files. The only reason I 
hesitate on this is that it could cause significant confusion for someone who 
doesn't really understand the implications, while people like yourself who have 
thought about this are also capable of finding workarounds and don't really 
need the ZIP file apart from convenience. Making some of the fixes to make 
python.exe more portable would relieve my concerns here.

> The old MSI installer sort of had something like that with the MSI
> administrative install option. But it never really worked because the
> administrative install didn't drop the MSVC runtime dlls anywhere, as far as I
> could tell. At some point I hacked around that by modifying the MSI file in my
> setup program to also drop the MSVC runtime dlls, but that was VERY hacky...

I'm a little surprised that worked at all for what you were trying to do. You'd 
be better off installing it once and then copying the files yourself. 

But overall, this is the sort of thing I do want to enable. I firmly believe 
that Python from python.org is for *developers*, and those who just want to run 
a Python application should be able to get a complete package. This is very 
different from the *nix approach, but it makes far more sense to Windows users. 
A good example is TortoiseHg, which bundles everything so that users never even 
know they have Python 2.7 or Mercurial installed on their machine. Making it 
easier for people to bundle Python into their own applications is a good thing, 
as far as I'm concerned.

Cheers,
Steve

> Thanks,
> David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

David Anthoff wrote:
>> I'll look into this, but it probably isn't going to work as part of
>> the installer. I have previously looked into being able to install arbitrary
>> side-by-side copies of Python, but that's near impossible as well.
>> Windows Installer doesn't really let you just copy files - it isn't
>> part of its intended functionality. It isn't too difficult to build custom
>> MSIs with certain parts of Python (such as the DLLs and the standard
>> library, but no docs, headers or EXEs) in a way that won't conflict with
>> other installs, but you're still using an MSI here which is not
>> necessarily ideal.
> 
> Are administrative MSI installs an option, though? They don't register 
> anything
> but just drop files, right? But see my comments below about a zip drop, which
> would be a much, much nicer option in my opinion.

Not to my knowledge. An administrative install puts the files in a shared 
location and allows users to run much faster installs that will then refer to 
that shared location rather than copying the files locally. As I said, MSI 
doesn't support plain file drops (often called "xcopy install" - you use that 
term later, but I'm not sure how well known it is).

>> We could release a ZIP file containing all the Python files.
> 
> That would be absolutely FANTASTIC and would solve all problems around this
> topic. In theory we could handle this ourselves on our end, but this is for a
> small open source project and we are really hesitant to take on another 
> software
> packaging job. Doing this right just for our product would be a fair amount of
> work, we would have to do this with every new Python release, we might mix
> things up etc., and really we are more interested in our piece of software 
> than
> packaging Python ;) I think it would be a much nicer model if there was a zip 
> to
> download from python.org.

Apart from the bit below, this would be identical to you installing Python once 
on your own machine and zipping up the files yourself. For the (considerably) 
less than 1% of Python users who want to do this, I don't think it's a big ask.

> There is another issue that keeps us from hosting our
> own files: we would have to figure out licensing issues, both related to 
> python
> and to the msvcr*.dll and msvcp*.dll. I would feel much more comfortable if we
> didn't distribute python or other binaries, but just downloaded stuff from
> python.org as part of the setup process.

IANAL, but I've dealt with enough licensing issues in my day job that this 
makes me even more concerned about making a ZIP file available. If you're 
planning to do this, putting up a ZIP file could potentially expose Python (and 
presumably the PSF indirectly) to the liability that you should be taking on as 
a redistributor, if it's that important to your product. Without an actual 
legal opinion, I'm now -1 on making a ZIP download available for this purpose.

>> The only reason I hesitate on this is that it could cause significant
>> confusion for someone who doesn't really understand the implications,
>> while people like yourself who have thought about this are also capable
>> of finding workarounds and don't really need the ZIP file apart from
>> convenience.
> 
> I was not clear in my previous email. I have NOT found a way to work around
> this. I have tried various hacks, but none really works. I got pretty far, but
> none really worked in all cases in a robust way. So I would certainly welcome 
> a
> downloadable zip file a great deal. Is there maybe a compromise for now to 
> have
> such a zip on the server, but not advertise it widely, and maybe put an
> "experimental"/"beta" moniker into the filename?

I gave you a workaround. Install Python just for yourself and zip up the 
install directory.

> I assume you would include the MS VC runtime files msvcr100.dll and 
> msvcp100.dll
> in such a zip file?

Either that or redistribute them using the tools provided by Microsoft (there 
are redistributable merge modules and executable installers available). The 
advantage of the latter approach is that your users will automatically get the 
latest versions and security fixes.

> Is there any chance this might even be done (as an experimental version) for
> Python 3.4?

You'd have to ask Martin, so probably not. But the workaround I gave you will 
work for 3.4.

>> Making some of the fixes to
>> make python.exe more portable would relieve my concerns here.
> 
> I see that. For our cases things seem to work, but I agree, it would be good 
> if
> python.exe would try hard to work in a xcopy mode.
> 
>>> The old MSI installer sort of had something like that with the MSI
>>> administrative install option. But it never really worked because
>>> the administrative install didn't drop the MSVC runtime dlls
>>> anywhere, as far as I could tell. At some point I hacked around that
>>> by modifying the MSI file in my setup program to also drop the MSVC
>>> runtime dlls, but
>>> that was VERY hack

Re: [Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

> On Sat, Jan 10, 2015 at 3:28 AM, Nick Coghlan  wrote:
>> For the time being, things like PyInstaller, PyRun, Portable Python,
>> etc are going to offer a better solution than anything we provide in
>> the standard installers.
> 
> See also Anaconda and Enthought Canopy. I think miniconda, for instance, may
> give you just what you need, if you don't want to go the py2exe/PyInstaller
> approach (though you probably do want to go that way, as far as I can tell 
> from
> your description of your use-case.

Actually, both Anaconda and Canopy suffer from the initialization process 
issues. You can't really use virtualenv with either of them unless they set the 
registry PythonPath value (since they won't find the libraries they need to 
launch), and if you install both then they'll fight over the registry key and 
you'll get Canopy launching with Anaconda's path or vice-versa.

> I'm inclined to think that this does not belong as part of the standard
> installer.

The problems are inherent to the standard python.exe, and are likely part of 
the standard pythonXY.dll. Nick's scared of fixing it, so I'm absolutely 
petrified :)

Cheers,
Steve

> -Chris
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] New Windows installer for Python 3.5

[Steve Dower]

Martin v. Löwis wrote:
> Am 04.01.15 00:34, schrieb Steve Dower:
>> so I'm keen to hear whatever feedback people have.
> 
> One issue that I always wanted to address is patch releases. There are two
> aspects I dislike about the current implementation
> 
> a) an upgrade install first uninstalls the previous installation (removing all
>files), and then reinstalls all "new" files. In many patch releases, a lot 
> of
>.py files remain unmodified, so it should speed up the upgrade if they 
> would not
>need to be replaced.
> b) Installer supports patch files (.msp); I always wanted to provide them in 
> the
>hope that this would reduce the download size. IIUC, it would require 
> stable
>component IDs for components to be upgraded, which I could not manage to
>provide.
> 
> So, do you have any plans on dealing with a or b?

So the stable component IDs is dealt with - WiX will auto-generate them based 
on the relative install path of each file and a user-provided ID (to 
distinguish between the 32-bit and 64-bit installs, for example. This ID is 
currently generated from the hash of a user provided URI, which is also used to 
generate upgrade codes and a few fixed component IDs).

Installing patches is something we can defer until 3.5.1, but I am interested 
in looking into it. My main concern is that it may hurt fresh installs (for 
example, 3.5.5 is actually 3.5.0.msi+.1.msp+.2.msp+.3.msp+.4.msp+.5.msp) and 
I'm not sure that true slipstreams are supported/recommended because it will 
affect future patches (different product codes, IIRC). So fresh installs of 
later versions may have a significantly increased download size.

Because of the split into multiple MSIs, it's also possible to independently 
version some parts of it. I suspect this will only apply to the py.exe 
launcher, but if that is unchanged between 3.5.0 and 3.5.1 then we can leave 
its version at 3.5.0 and it won't be redownloaded/installed. Not a huge saving, 
but it's a possibility. I expect most MSIs will change in some way between 
versions, so an MSP is the only good way to improve upgrades (the main benefit 
of the MSI split here is we will always install the latest 32-bit launcher, and 
if you install 3.5 32-bit and 64-bit and eventually 3.6 32-bit and 64-bit 
you'll only have one launcher installed).

Another possible problem is that MSI uses file version information to determine 
whether to update files. .py files don't have version information, so MSI to 
MSI updates are probably always going to replace everything - another reason 
why MSPs are the only good choice here.

I'll chat to some of the guys who work on the Visual Studio installer, since it 
uses the same technology and is far more complex than Python's one. They may 
have some suggestions about how to approach this.

> Other issues:
> - what MSI version do you require, and what is the minimum
>   Windows version supporting that MSI version?

I need to double-check the support still, but the aim is to support back to 
Windows Vista, so I think that's Windows Installer 3.0. Of more concern is the 
installer EXE, which is going to require a minimum level C runtime that may not 
be available before we've installed...

> - Since you are going to install into Program Files by default,
>   I think the library should be precompiled by default - there
>   is little chance that a regular user can save .pyc files when
>   running Python. It might be possible to even include the pyc
>   files in the distribution, if we can arrange to somehow support
>   relative paths in co_filename.

Agreed. The library will be precompiled by default if you choose the All Users 
button on the front page, and you can control this independently through the 
customize option (to precompile Just for Me or install without precompiling).

> Regards,
> Martin

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile Python on Windows (OpenSSL)

[Steve Dower]

Victor Stinner wrote:
> To compile Python on Windows, there are a few information in the Developer
> Guide:
> https://docs.python.org/devguide/setup.html#windows-compiling

I'm sorry, I've been slack and haven't updated this guide with the new changes. 
That said, there aren't that many changes to what is already in the guide - 
mostly new information that I'd like to add. I'll try and get something written 
up this week.

> Python 3.5 now requires Visual Studio 2010 *SP1*, or newer Visual Studio:
> http://bugs.python.org/issue22919#msg233637

Technically, Python 3.5 requires Visual Studio 2015, as the decision was made 
to move to the newer compiler for this release (and make whatever schedule 
changes were necessary to ensure everything will be available - it was on 
python-dev, though I don't have links handy).

It will still open and build with Visual Studio 2010 SP1 or later, however, 
buildbots should move to VS 2015 as they can, since that will be the official 
toolset. (I'm not pushing too hard until 2015 RC is released, since that's when 
Microsoft start supporting upgrades to newer versions. We do have one buildbot 
that's already using VS 2015 Preview though.)

> I found PCbuild\readme.txt which is not mentionned in the devguide :-/
> https://hg.python.org/cpython/file/56f717235c45/PCbuild/readme.txt
> (at least not on the Windows section of the setup page)

That should help, since I've updated that file. Though it's a little ahead of 
its time still, so a lot of the instructions do not yet have to be followed 
strictly, and I consider being able to build with VS 2010 SP1 a feature that we 
want to keep supporting for now while we transition.

> I found some clues to build OpenSSL to be able to build the Python ssl module,
> but I still have issues.
> 
> Is there a more complete documentation?
> 
> I found how to install svn.exe, perl.exe and nasm.exe, but not how to install
> nmake.exe. I don't know the command to build OpenSSL.

You don't require perl, you don't have to install nasm or build OpenSSL. 
Calling PCBuild/get_externals.bat will get nasm and the OpenSSL sources, and 
PCBuild/build.bat will rebuild OpenSSL as necessary.

nmake is part of Visual C++, so it should be available once you have your path 
configured (PCBuild/env.bat).

So a complete guide to check out and build Python on Windows should be:

1. Install SVN and put it on your path
2. Run PCBuild\get_externals.bat (this is the only step that requires SVN)
3. Install Visual Studio 2010 SP1 or later
4. Open PCBuild\pcbuild.sln in Visual Studio *or* run PCBuild\build.bat from 
the command line.

> I don't care of building OpenSSL, my goal is only to build the Python ssl
> module. Is there a way to install a development version of OpenSSL (.lib files
> if I remember correctly) from an installer/binary?

If you can find ssleay.lib and libeay.lib somewhere, they'll probably work. If 
not, those projects are in PCBuild and will be built automatically (if 
necessary) when you build _ssl.vcxproj or _hashlib.vcxproj.

Cheers,
Steve

> My draft notes:
> 
> +Compile CPython on Windows
> +==
> +
> +To build the Python ssl extension:
> +
> +Need:
> +
> +* Visual Studio 2010 SP1 or newer
> +* CPython source code (default branch: 3.5)
> +* perl binary: ActivePerl
> +* svn binary, ex: SilkSVN
> +* nasm and nmake binaries: compile NASM (install the binary doesn't
> install nmake)
> +
> +Read PCbuild/readme.txt.
> +
> +* Build Python (in debug mode)
> +* Type: PCbuild\get_externals.bat
> +* Type: PCbuild\win32\python_d.exe PCbuild\prepare_ssl.py
> externals\openssl-1.0.1j
> 
> Victor
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile Python on Windows (OpenSSL)

[Steve Dower]

Victor Stinner wrote:
> 2015-01-13 23:18 GMT+01:00 Steve Dower :
>> Technically, Python 3.5 requires Visual Studio 2015
> 
> For me, it's *very* difficult to find how to install Visual Studio.
> There are many different websites and web pages which mention Visual Studio 
> with
> a lot of versions and "flavors" (Express, Community, Ultimate, etc.).

http://www.visualstudio.com/ is the only website to look at, and it may 
redirect you to somewhere on microsoft.com. Anywhere else is sketchy.

> Visual Studio 2015 was not released yet :-/

Yes, the timing is a little bit awkward, which is why I'm keen to make sure VS 
2010 keeps working for now.

> My VM has only a disk of 40 GB. Only 12 GB are free. I already have VS
> 2008 Express and VS 2010 Express installed. I understood that "Ultimate"
> includes a *lot* of things, not only a C compiler.

I appreciate this, and I'm constantly trying to get a version released that is 
just the compiler and nothing else. Unfortunately, I'm one lone low-level 
engineer, and releasing a different package like this is a high-level decision 
that I have no control over. Because VS 2015 is in preview, the only available 
version is Ultimate, but when the RC is released I'd expect to be able to 
choose the more specific versions.

> I found a "free" Visual Studio which is in fact Visual Studio 2013 Community 
> and
> I read that it's not free.

Community Edition is certainly free (as in $0 - if you want to get ideological 
about "free" then you'll make it very hard for yourself to use any software 
produced by a business). The only requirement is that you log in with a 
Microsoft (a.k.a Live.com/Hotmail) account. It's a dumb requirement, but again, 
this is a high-level decision that I get no say in.

> I sent an email to Brian Curtin to ask to renew my MSDN account. He didn't 
> reply
> yet.

Give him a chance, I'm sure he's just busy :)

> Victor

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile Python on Windows (OpenSSL)

[Steve Dower]

Victor Stinner wrote:
> 2015-01-13 23:46 GMT+01:00 M.-A. Lemburg :
>> Just a note of caution: for older preview releases of VS the only way 
>> to get back to a clean system was to reinstall Windows.
>
> Does it mean that it's not possible to have VS 2008 and VS 2015 installed at 
> the same time?

It's possible, and that configuration will be fine.

Until VS 2015 RC is released, there's no guarantee that it will uninstall or 
upgrade cleanly. The testing so far looks like it'll be fine, but there's no 
guarantee.

There's also a chance that it will interfere with VS 2013, but any earlier 
versions will be fine.

Cheers,
Steve

> VS 2008 is required to build Python 2.7.
>
> Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Compile Python on Windows (OpenSSL)

[Steve Dower]

Victor Stinner wrote:
> 2015-01-15 22:39 GMT+01:00 Ryan Gonzalez :
> > http://www.microsoft.com/en-us/download/details.aspx?id=8279
> 
> "Microsoft Windows SDK for Windows 7 and .NET Framework 4"
> 
> Are you sure that it is SDK 7.1, and not 7.0?

The naming is horrible, and the bugs between VS and the SDK are also horrible. 
I'm sorry.

SDK 7.0 == "Windows SDK for Windows 7 and .NET Framework 3.5"
SDK 7.1 == "Windows SDK for Windows 7 and .NET Framework 4"

So this is the correct link for building Python 3.3 and 3.4. It's also the one 
you want to keep building the latest code with the old compiler.

> --
> 
> The SDK 7.0 works for Python 2.7 which is compiled with Visual Studio 2008.
> 
> I used the SDK 7.1 for Python 3.3 and 3.4 which are compiled with Visual 
> Studio
> 2010.
> 
> It looks likt SDK 8 is more for Visual Studio 2012.
> 
> If you use the wrong SDK, you will depend on a "MSVCRxxx.dll" which is not
> provided by Python x.x (ex: MSVCR100.dll for SDK 7.1/Python 3.3 & 3.4).

This information is all entirely correct. Worse, if you depend on the wrong 
runtime version, you'll get unexpected crashes and other errors.

Cheers,
Steve

> Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Azure event hub network access

[Steve Dower]

This would be much better posted on the github page for the project. I don't 
have the URL handy, but if you search github for "Python Azure SDK" you'll find 
it.

Cheers,
Steve

Sent from my Windows Phone

From: syed khalid
Sent: ‎2/‎5/‎2015 21:27
To: python-dev@python.org
Cc: python-l...@python.org
Subject: [Python-Dev] Azure event hub network access

I am getting http error 404. I am able to access the site via telnet which 
eliminates network issues. Here is the code and subsequent errors



user/bin/python
import sys
import azure
import socket

from azure.servicebus import (
  _service_bus_error_handler
  )

from azure.servicebus.servicebusservice import (
  ServiceBusService,
  ServiceBusSASAuthentication
  )

from azure.http import (
  HTTPRequest,
  HTTPError
  )

from azure.http.httpclient import _HTTPClient

class EventHubClient(object):

def sendMessage(self,body,partition):
eventHubHost = 
"pac-ns.servicebus.windows.net"

httpclient = _HTTPClient(service_instance=self)

sasKeyName = "pac-pl"
sasKeyValue = "IhkEepQPLfSy9jo6H2Y="

authentication = ServiceBusSASAuthentication(sasKeyName,sasKeyValue)

request = HTTPRequest()
request.method = "POST"
request.host = eventHubHost
request.protocol_override = "https"
#request.path = "/myhub/publishers/" + partition + "/messages?api-version=20
14-05"
request.body = body
request.headers.append(('Content-Type', 'application/atom+xml;type=entry;cha
rset=utf-8'))

authentication.sign_request(request, httpclient)

request.headers.append(('Content-Length', str(len(request.body
status = 0

try:
resp = httpclient.perform_request(request)
status = resp.status
except HTTPError as ex:
status = ex.status

return status

class EventDataParser(object):

  def getMessage(self,payload,sensorId):
host = socket.gethostname()
body = "{ \"DeviceId\" : \"" + host + "\",\"SensorData\": [ "

msgs = payload.split(",")
first = True

for msg in msgs:
# print msg
  sensorType = msg.split(":")[0]
sensorValue = msg.split(":")[1]
  if first == True:
first = False
  else:
body += ","

  body += "{ \"SensorId\" : \"" + sensorId + "\", \"SensorType\" : \"" + sen
sorType + "\", \"SensorValue\" : " + sensorValue + " }"
body += "]}"

return body

hubClient = EventHubClient()
parser = EventDataParser()
hostname = socket.gethostname()
sensor = sys.argv[2]

body = parser.getMessage(sys.argv[1],sensor)
hubStatus = hubClient.sendMessage(body,hostname)
# return the HTTP status to the caller
print hubStatus
print hostname
print sensor




~/IOT/AZURE$ python send.py temperature:22,humidity:20 deviceid

404
ubuntu
deviceid
{ "DeviceId" : "ubuntu","SensorData": [ { "SensorId" : "deviceid", "SensorType" 
: "temperature", "SensorValue" : 22 },{ "SensorId" : "deviceid", "SensorType" : 
"humidity", "SensorValue" : 20 }]}




--
Syed Khalid

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

I was also surprised as I was working on the installer, so +1 on changing it.

The installer will also need some changes to add it to PATH, which should be 
fairly straightforward, but ping me if you need/want pointers. (It's checked in 
now, so I consider it fair game for anyone who wants to change it.)

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Moore
Sent: ‎2/‎10/‎2015 5:40
To: Nick Coghlan
Cc: Python Dev
Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

On 10 February 2015 at 13:34, Nick Coghlan  wrote:
> I was actually surprised it didn't already work that way, given that almost
> everything else on Windows is version specific.

That's actually a point I missed making. While I wouldn't class myself
as "typical", when I went looking for the user-scripts directory on my
machine, I automatically looked in %APPDATA%\PythonXY. I suspect that
many people would assume this was how it worked anyway, and be
surprised by the current behaviour.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

One of my main engineering concerns is lack of shared knowledge a.k.a. the 
truck factor (not CPython specific, btw, just every project I work on). So I'm 
always keen to get multiple people working on new code asap. I also prefer 
installers to be treated as part of the product and updated with whatever 
changes affect it, though that basically never happens in my experience :)

I assume the PEP will need updating if it specifies the exact path, and the 
installer readme may have a reference too. Presumably it'll only affect 3.5, as 
previous versions wouldn't use the directory for anything other than installing 
scripts.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Moore<mailto:p.f.mo...@gmail.com>
Sent: ‎2/‎10/‎2015 6:41
To: Steve Dower<mailto:steve.do...@microsoft.com>
Cc: Nick Coghlan<mailto:ncogh...@gmail.com>; Python 
Dev<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

On 10 February 2015 at 14:13, Steve Dower  wrote:
> The installer will also need some changes to add it to PATH, which should be
> fairly straightforward, but ping me if you need/want pointers. (It's checked
> in now, so I consider it fair game for anyone who wants to change it.)

I was sort of hoping you'd be doing that independently, but never mind
:-) I'll give it a go and shout if I get stuck.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

Martin v. Löwis wrote:
> Am 10.02.15 um 16:41 schrieb Steve Dower:
>> One of my main engineering concerns is lack of shared knowledge a.k.a. the
>> truck factor (not CPython specific, btw, just every project I work on).
> 
> Wrt. the installer, I think this is a lost cause. IIUC, you aren't really
> getting familiar with msi.py, and you are the one who has the highest chances 
> -
> so I was the only real maintainer of it.
> Likewise, the Wise installer was only maintained by Tim Peters.
> 
> So chances are really high that you will be the only one making substantial
> changes to the WiX files, although some people might learn how to build the
> installer.

Aww, come on. Let me have a little bit of hope ;)

> Regards,
> Martin
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

So I've enumerated the problems with PATH on Windows before (and my subsequent 
dislike for modifying it through the installer), but here's the dot-point 
summary for those who haven't seen it.

* System-wide values always precede per-user values
* Most-recently-installed values always precede existing values (within the 
system/user separation above)
* It's impossible to specify per-user values from a system-wide installer
* The system-wide value has a length limit that silently breaks PATH if you 
exceed it
* It's difficult for users to modify PATH themselves (mostly due to bad UI in 
Windows)

The py.exe launcher solves many of these problems, but does not help with 
installed scripts. It's trivially easy to end up in a place where this would 
fail:

pip install --user spam
spam

The *only* case where this will work reliably (without manual user 
configuration) is where Python 3.5 is the only installed version on the 
machine, and it was installed by the current user just for them.

As we've seen from earlier discussions, the main beneficiaries of having Python 
on PATH are those using the command-line. Most scripts are going to make 
assumptions or work unnecessarily hard to find the actual location of the 
Python version they need.

My best idea for dealing with this is to add another script alongside the 
py.exe launcher called `activate-py`.[1] This script would take the same 
command-line version argument as py.exe and use it to update the current 
prompt's PATH correctly. Specifically, I expect the script would actually use 
py.exe to invoke the requested Python and get the paths directly from 
sysconfig, which will ensure that the order of precedence always matches.

Users would need to run `activate-py`, `activate-py -2` or `activate-py -3.5` 
(etc.) when they start their session.[2] This parallels venv's `activate` 
command, though because this would be a global command I believe the `-py` 
suffix is necessary. The example above becomes:

activate-py -3.5
pip install --user spam
spam

A `deactivate-py` command would also be added to revert the changes, though I 
don't expect that to be part of most workflows.

Another benefit is that shell scripts could call `activate-py` rather than 
trying to detect whether Python is already on the path or requiring a PYTHON 
variable to be set. (I seem to write these scripts all the time and know that 
I'd benefit from it.)

This is yet another attempt to try and change user behaviour, which I'm not 
thrilled about, but I'm really starting to feel that this is the best way out 
of a bad situation. It differs from the solutions used on Linux, though there 
may be some value in this approach there too. Thought/votes/suggestions?

Cheers,
Steve


[1]: Actually multiple scripts to handle cmd.exe/PowerShell, but this is an 
implementation detail. Users would simply type `activate-py` and the shells 
know which one to use.

[2]: We can easily add Start Menu shortcuts (e.g. "Python 3.5 (32-bit) Command 
Prompt") and run it for them, though that is not a critical part of this 
proposal.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

Paul Moore wrote:
> On 10 February 2015 at 20:50, Paul Moore  wrote:
>> On 10 February 2015 at 17:45, Steve Dower  wrote:
>>> This is yet another attempt to try and change user behaviour, which I'm not 
>>> thrilled
>>> about, but I'm really starting to feel that this is the best way out of a 
>>> bad situation.
>>> It differs from the solutions used on Linux, though there may be some value 
>>> in this
>>> approach there too. Thought/votes/suggestions?
>>
>> I have to admit, I agree with you. But before the "Add Python to PATH"
>> option was added to the installer, we repeatedly got requests for it
>> and it was apparently a constant source of confusion for beginners. If
>> we're going to revert that change, we'll need to be very careful how
>> we present it.
> 
> Drat - hit "Send" too soon.
> 
> Also, the proposed scripts won't be on PATH, so users will have to run
> them using explicit paths, which is both inconvenient and poses a
> discoverability issue (Tools/scripts/win_add2path.py has been round
> for ages, but AFAIK hardly anyone uses it). I wonder if there's a way
> to get the launcher (py.exe) to help? For powershell, we could have
> "py --setpath" write the appropriate path setting command to stdout,
> and users could use "Invoke-Expression $(py --setpath)", for example.
> That's a bit clumsy, and doesn't work for cmd.exe, but maybe there's
> possibilities... Essentially, py.exe is the only command guaranteed to
> be on PATH for any install (or is it? for a user install, is it stuck
> somewhere in the user's APPDATA?)

(For a user install, I've guaranteed that the launcher will be added to the
user's PATH. But a system-wide launcher will win.)

win_add2path.py does the same thing as the installer, so it isn't a useful
solution.

The `py --setpath` is basically my idea with `activate-py`, though with the
"Invoke-Expression" part handled in the script (and not actually using
Invoke-Expression, but that's implementation details). If these are part of the
launcher, then they'll always be on PATH, and if they're just thin wrappers
around py.exe then they can support all the same versions of Python (in effect,
all of them, though with the 32/64-bit per-user collisions that existed
pre-3.5).

> It's fair enough to say that installers shouldn't be in the business
> of messing with PATH, but nevertheless, people expect to run an
> installer and have the installed programs available for use. Command
> line utilities like Mercurial do it, so people will expect Python to.
> Unfortunately.

And Python does, as long as that installed program is "py". The problem is that
people expect "python" to be there and be the correct one (for a totally
arbitrary definition of "correct") as well as pip, easy_install and anything
installed by those two tools, whether they were installed for all users or just
the current one. It's far more complex than a single app (especially when you
realise that a system-wide install is effectively trying to make Python look
like part of the OS - we only get that about 50% right now...).

> Sorry, all problems, no real answers here.
> Paul

Yeah, same. `activate-py` is the best I've got.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

You're not alone, that's exactly how I do it too.

I'm writing up a PEP for the activate-py command I suggested earlier that will 
temporarily set your PATH. If that goes well then we'll be able to ship that 
with the py.exe launcher (I'm particularly excited for batch files to be able 
to use it).

Cheers,
Steve

Top-posted from my Windows Phone

From: Thomas Heller<mailto:thel...@ctypes.org>
Sent: ‎2/‎12/‎2015 0:03
To: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

Am 10.02.2015 um 18:45 schrieb Steve Dower:
> As we've seen from earlier discussions, the main beneficiaries of
> having Python on PATH are those using the command-line. Most scripts
> are going to make assumptions or work unnecessarily hard to find the
> actual location of the Python version they need.

Maybe I'm more or less alone with the way I work, but I don't like
python.exe on my PATH (and py.exe alloes me to do this).
I start python scripts from the command line either with 'script.py'
or 'py -3.4 script.py' or 'py -2.7 script.py'.

I even like running scripts in this way:
'py -2.7 -m pip install whatever'.  However, this only works when the
script also can be started as package.  Some packages allow this,
some don't.

Could not py.exe be extended so that it allows starting scripts in a
somewhat similar way?  'py-script -2.7 myscript foo bar baz' ???
Which would execute the script myscript.exe, myscript.bat, myscript.py,
myscript.cmd or whatever is in the Scripts directory on the Python 2.7
installation, without changing PATH presistently?

Thanks,
Thomas

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

If venv's activate script sets it, I say go ahead and write it up. If it's just 
virtualenv, I'd rather not explicitly depend on it, especially since PEP 397's 
stated aim is file associations and not command line.

I've been making changes to py.exe in hg.p.o, so I hope the standalone one is 
tracking. The msi for it as part of the official build can also standalone, so 
maybe we should merge the two?

Top-posted from my Windows Phone

From: Paul Moore
Sent: ‎2/‎12/‎2015 6:48
To: Thomas Heller
Cc: Python Dev
Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

On 12 February 2015 at 08:05, Thomas Heller  wrote:
> Maybe I'm more or less alone with the way I work, but I don't like
> python.exe on my PATH (and py.exe alloes me to do this).
> I start python scripts from the command line either with 'script.py'
> or 'py -3.4 script.py' or 'py -2.7 script.py'.

This is how I worked for a long time after py.exe became available.
But then I discovered virtualenv, and started using that, and needing
to use 2 completely different command sets for "when I'm in a
virtualenv" vs "when I'm not" became completely unusable.

If py.exe detected when the environment variable VIRTUAL_ENV was set,
and used that virtualenv as the default Python rather than the
"system" python it normally used, this would be perfect.

I think I'll write a PEP for this. Do people think it's a reasonable idea?

(On that note, is there any relationship between the py.exe shipped
with Python and the standalone version Vinay used to maintain but
which now appears to be hosted at
https://bitbucket.org/pypa/pylauncher? The two seem to be different,
for example the pypa version registers pyz/pyzw extensions).

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

Paul Moore wrote:
> 
> On 12 February 2015 at 15:37, Steve Dower  wrote:
>> If venv's activate script sets it, I say go ahead and write it up. If
>> it's just virtualenv, I'd rather not explicitly depend on it,
>> especially since PEP 397's stated aim is file associations and not command
> line.
> 
> Yep, venv uses it too (see
> https://hg.python.org/cpython/file/9e10c4255277/Lib/venv/scripts/nt/activate.bat).
> 
> As people like Thomas (and me, until this issue stopped me :-)) are using the
> launcher for command line use, I think it's fair to broaden the scope to make
> command line usage more convenient. I agree that PEP
> 397 was originally focused mainly on file associations, but I think it's worth
> looking beyond that now.

That's roughly the premise behind my activate-py proposal. Use `activate` to 
make a virtualenv/venv the default, and `activate-py [-x[.y[-32]]]` to make a 
non-venv the default. I have proof of concept implementations already for 
cmd.exe and PowerShell and it's fairly simple, but writing up the rationale is 
time consuming (and also the main point - it's worth having all the problems 
with PATH documented in one place).

>> I've been making changes to py.exe in hg.p.o, so I hope the standalone
>> one is tracking. The msi for it as part of the official build can also
>> standalone, so maybe we should merge the two?
> 
> Hmm, sadly I don't think it is. Originally the standalone one was (I
> believe) provided by Vinay for people using Pythons that didn't have it 
> bundled,
> and to add features (such as the new file extensions) on a quicker timescale
> than Python releases. But since he passed it to the pypa umbrella I don't 
> think
> he's been keeping the two in sync. I've copied him in case I'm completely 
> wrong
> on this.
> 
> Personally if we now have a standalone launcher MSI, I'd like to discontinue 
> the
> external one completely, and officially publish the standalone launcher MSI on
> python.org as a service for users of older Pythons. There seems little reason 
> to
> maintain 2 repos if we don't have to. We could merge in changes from the
> external repo before discontinuing it, but I'm not sure how controversial that
> would be (for example, would it need a PEP to include the 2 new extensions?)

None of my installer changes so far have had a PEP, and only a few people have 
complained about that :) (it does have more documentation than I've ever 
written for an installer before though)

IIRC, there was a PEP for executing ZIP files directly (2.6-era?), which I 
believe are the purpose of those extensions. If "py.exe spam.pyz" already 
works, I don't see any need for a PEP to add the association in the installer.

Cheers,
Steve

> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

Vinay Sajip wrote:
> By the way, although I turned over the standalone launcher to pypa, I forked
> that repo, and do development on my fork [1]. I remembered that I added a
> feature (with the help of Pawel Jasinski) to allow e.g. py -ipy to pick a
> command 'ipy' configured in the .ini file (this is used to launch IronPython,
> but it could be used for other things too). In addition, my version has some
> bugfixes (e.g. use of iswspace instead of isspace, which is wrong). Steve's
> changes seem to just be related to adding IP_VERSION - I'm not sure what 
> that's
> about and I haven't looked into it yet; I just skimmed the differences, and
> didn't see anything major in launcher.c (I compared with the version in the
> default branch of cpython).

I thought I did more than that, but maybe that was attempting to make things 
work :) The default branch is the most up to date.

The big change is that the registry key for 32-bit Python now includes the 
"-32" suffix, so that you can install 32-bit and 64-bit versions per-user at 
the same time and we don't have a collision. The launcher needed some updates 
to handle that, since it was assuming that registry key names would never be 
more than three characters long. Probably everything else I did was elsewhere 
in Python or in the installer. (Btw, it sounds like your installer will work 
better as a standalone tool, though my MSI has the advantage that it won't 
collide with a Python 3.5+ installation. I'm not fussed about whose 'wins' 
though.)

Cheers,
Steve

> Regards,
> 
> Vinay Sajip
> 
> [1] https://bitbucket.org/vinay.sajip/pylauncher
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation (full C or C+Python)

[Steve Dower]

I think posixmodule is a great candidate for splitting up by platform rather 
than function, as the whole file is packed with ifdef. It's really only lacking 
a volunteer to do it, but we could start here (ie. make posixmodule_nt.c for 
the Windows impl, etc.) and progressively move function implementations out 
over time? All the module setup and probably most of the Python layer can stay 
where it is.

More likely we're going to get bogged down discussing it again though, so if 
that happens my vote is to just make posixmodule.c 800 lines longer.

Cheers,
Steve

Top-posted from my Windows Phone

From: Antoine Pitrou
Sent: ‎2/‎13/‎2015 5:44
To: python-dev@python.org
Subject: Re: [Python-Dev] PEP 471 (scandir): Poll to choose the implementation 
(full C or C+Python)

On Fri, 13 Feb 2015 08:35:00 -0500
Ben Hoyt  wrote:
>
> If we go ahead with the all C approach, I'd be in favour of
> refactoring a little and putting the new scandir code into a separate
> C file. There are two ways to do this: a) sticking with a single
> Python module and just referencing the non-static functions in
> scandir.c from posixmodule.c, or b) sharing some functions but making
> _scandir.c its own importable module. Option (a) is somewhat simpler
> as there's not module setup stuff twice, but I don't know if there's a
> precedent for that way of doing things.

The _io module already does things that way (the (a) option, I mean).

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

[Steve Dower]

Launching non-Python scripts from py.exe is not going to work as well as you 
may hope. There will be numerous subtle bugs due to the cmd->py.exe->cmd 
wrapping (Powershell users would have seen this with some console apps or batch 
files). I think having a global command to add a particular Python to PATH for 
the current session will work best here, and yes, I'm working on it.

If py.exe starts defaulting to whatever is on PATH then I don't see the point 
of it. Knowing that you'll get the latest 2.x version by default is quite handy 
(I'd be quite okay changing that to 3.x, though it is specified in the original 
PEP). For me, the point of py.exe is to be able to ignore PATH completely.

Adding more commands to the launcher args (eg. -ipy) will make the parsing far 
more complex, especially when deciding which args to consume and which to 
forward. I think you can make a copy of the launcher called ipy.exe and add 
ipy.ini with the path to IronPython in it as default.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Moore
Sent: ‎2/‎13/‎2015 3:20
To: Glenn Linderman
Cc: Python Dev
Subject: Re: [Python-Dev] PEP 370 - per-user scripts directory on Windows

On 13 February 2015 at 10:15, Glenn Linderman  wrote:
> Maybe restricting it to running ".py" files or ".exe" files would be
> reasonable. That covers entry points (which should be the norm for
> newer projects) and Python scripts (which are the most likely things
> to be portable).
>
> The WINDOWS py launcher hardly needs to be portable.  Batch/CMD files also
> seem useful on WINDOWS. And Powershell?
>
> If a launcher is developed for or enhanced for other systems, shell scripts
> may be useful, and Batch/CMD not.

By "portable" in this sense I meant "not written by a Unix developer
who hadn't thought about Windows much". I was comparing Python files
with a .py extension and extensionless Python files, specifically (my
wording didn't make that clear, sorry). Yes, files with extensions of
.bat or .cmd or .ps1 are probably written for Windows only. But I
think it's getting out of scope of the launcher to decide to use
cmd.exe to launch .bat and .cmd files, and powershell to launch .ps1
files.

Regardless, my real point was that there can be all sorts of stuff in
the Scripts directory, and I'm not convinced that this is a direction
the launcher should be going in. Someone *could* make a case for
launching certain types of file that might be in there, but I'm not
going to be that person.

One thought - it's not hard using pkg_resources to enumerate all
available console_script entry points, pick one and run it. So you
could pretty simply write a script (that itself could be run with the
launcher) to run anything in Scripts that's built via entry points
(which is the recommended way these days).

# launch.py
import sys
import pkg_resources

# Get the script name
script = sys.argv[1]
# Remove our name from sys.argv
del sys.argv[0]

# There's probably a better API you can use
for ep in pkg_resources.working_set.iter_entry_points('console_scripts'):
if ep.name == script:
fn = ep.load()
sys.exit(fn())

# Couldn't find the entry point
print("No entry point called {} available".format(script), file=sys.stderr)
sys.exit(1)

Now you can do "py launch.py pip --help" and it will run pip for you.

If a project isn't exposing its scripts as entry points, ask them to :-)

Paul

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] subclassing builtin data structures

[Steve Dower]

"Instead of requiring *every* subclass to override all the methods,
couldn't we require the base classes (like int) to assume that the
signature is unchanged and call type(self), and leave it up to the
subclass to override all the methods *only* if the signature has
changed?"

I assumed everyone was just saying this point over and over, so I haven't been 
following the thread closely. This is precisely how inheritance works: 
subclasses are constrained by the base class. If you want to play, you *must* 
play by its rules. (Else, use composition.)

It's fine for base classes to assume a compatible constructor, and if builtins 
can do it without hurting performance for the non-subclassed case, I don't see 
why not.

Cheers,
Steve

Top-posted from my Windows Phone

From: Steven D'Aprano
Sent: ‎2/‎14/‎2015 4:24
To: python-dev@python.org
Subject: Re: [Python-Dev] subclassing builtin data structures

On Fri, Feb 13, 2015 at 06:03:35PM -0500, Neil Girdhar wrote:
> I personally don't think this is a big enough issue to warrant any changes,
> but I think Serhiy's solution would be the ideal best with one additional
> parameter: the caller's type.  Something like
>
> def __make_me__(self, cls, *args, **kwargs)
>
> and the idea is that any time you want to construct a type, instead of
>
> self.__class__(assumed arguments…)
>
> where you are not sure that the derived class' constructor knows the right
> argument types, you do
>
> def SomeCls:
>  def some_method(self, ...):
>return self.__make_me__(SomeCls, assumed arguments…)
>
> Now the derived class knows who is asking for a copy.

What if you wish to return an instance from a classmethod? You don't
have a `self` available.

class SomeCls:
def __init__(self, x, y, z):
...
@classmethod
def from_spam(cls, spam):
x, y, z = process(spam)
return cls.__make_me__(self, cls, x, y, z)  # oops, no self


Even if you are calling from an instance method, and self is available,
you cannot assume that the information needed for the subclass
constructor is still available. Perhaps that information is used in the
constructor and then discarded.

The problem we wish to solve is that when subclassing, methods of some
base class blindly return instances of itself, instead of self's type:


py> class MyInt(int):
... pass
...
py> n = MyInt(23)
py> assert isinstance(n, MyInt)
py> assert isinstance(n+1, MyInt)
Traceback (most recent call last):
  File "", line 1, in ?
AssertionError


The means that subclasses often have to override all the parent's
methods, just to ensure the type is correct:

class MyInt(int):
def __add__(self, other):
o = super().__add__(other)
if o is not NotImplemented:
o = type(self)(o)
return o


Something like that, repeated for all the int methods, should work:

py> n = MyInt(23)
py> type(n+1)



This is tedious and error prone, but at least once it is done,
subclasses of MyInt will Just Work:


py> class MyOtherInt(MyInt):
... pass
...
py> a = MyOtherInt(42)
py> type(a + 1000)



(At least, *in general* they will work. See below.)

So, why not have int's methods use type(self) instead of hard coding
int? The answer is that *some* subclasses might override the
constructor, which would cause the __add__ method to fail:

# this will fail if the constructor has a different signature
o = type(self)(o)


Okay, but changing the constructor signature is quite unusual. Mostly,
people subclass to add new methods or attributes, or to override a
specific method. The dict/defaultdict situation is relatively uncommon.

Instead of requiring *every* subclass to override all the methods,
couldn't we require the base classes (like int) to assume that the
signature is unchanged and call type(self), and leave it up to the
subclass to override all the methods *only* if the signature has
changed? (Which they probably would have to do anyway.)

As the MyInt example above shows, or datetime in the standard library,
this actually works fine in practice:

py> from datetime import datetime
py> class MySpecialDateTime(datetime):
... pass
...
py> t = MySpecialDateTime.today()
py> type(t)



Why can't int, str, list, tuple etc. be more like datetime?



--
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Can I replace distutils.msvccompiler?

[Steve Dower]

I'm about to start doing a rework on distutils.msvc[9]compiler.MSVCCompiler for 
Python 3.5 to be able to handle forward-compatibility better. Right now I've 
tweaked msvc9compiler enough for VS 2015, but it won't handle later VS versions 
automatically.


Is there any reason to keep the old msvccompiler and msvc9compiler modules 
around? If possible I'd like to drop them completely and just have one 
implementation (probably distutils.msvccompiler.MSVCCompiler again).


Obviously it will be behaviourally compatible with 3.4 and compatible with the 
CCompiler interface, but it may not be function-for-function compatible with 
distutils.msvc9compiler (e.g. get_build_version is not useful). Do we need a 
deprecation period? msvccompiler is mentioned in the docs, but not documented, 
and msvc9compiler is not even mentioned.


Thanks,

Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Steve Dower]

"Go ahead, make my pep."

We should make a python-dev t-shirt with this on it :)

Top-posted from my Windows Phone

From: Daniel Holth<mailto:dho...@gmail.com>
Sent: ‎2/‎15/‎2015 9:46
To: Nick Coghlan<mailto:ncogh...@gmail.com>
Cc: Paul Moore<mailto:p.f.mo...@gmail.com>; Steve 
Dower<mailto:steve.do...@microsoft.com>; Python 
Dev<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support


Go ahead, make my pep.

I will appreciate seeing it happen.

On Feb 15, 2015 8:47 AM, "Nick Coghlan" 
mailto:ncogh...@gmail.com>> wrote:
On 15 February 2015 at 23:21, Paul Moore 
mailto:p.f.mo...@gmail.com>> wrote:
> On 15 February 2015 at 08:59, Nick Coghlan 
> mailto:ncogh...@gmail.com>> wrote:
>> The other option would to cut PEP 441 way back to *just* be about
>> standardising and registering the file associations, and recommending
>> the use of pip to obtain the build machinery with (whether pyzaa,
>> pyzzer or Twitter's more comprehensive pex). It would be a short PEP,
>> but potentially still worth it for the improved visibility of the
>> decision when folks are trying to figure out what "pyz" and "pyzw"
>> files are later.
>
> Ok, thinking about this a little more.
>
> Getting the extension support is the key thing on Windows - at the
> moment, people are faced with adding their own file associations or
> putting binary data in a .py file, neither of which is a nice choice.
> Tooling is important, though - sure, you can zip the data up and put a
> header on, but it's fiddly.
>
> Which brings us full circle. A simple module, executable as "python -m
> zipapp" (see below re name) which exports a single function, pack()
> that creates the archive. If we want to provide a script to wrap the
> module, like pyvenv.py does for venv, I've no objection to that -
> presumably it would go in Tools/Scripts? If people (like me) want to
> experiment with a more programmatic API for building pyz files, they
> can do so on PyPI, and if such a thing becomes sufficiently mature we
> can look then at proposing it for inclusion in the stdlib, as an
> extension to the zipapp module.
>
> Regarding naming, I'm happy to go with zipapp if it's your preference.
> Presumably the wrapper in Tools/Scripts would be pyzipapp.py?

Or we just skip the wrapper and make "python -m zipapp" the
recommended invocation style. Adding a wrapper later is fairly easy,
but removing it would be difficult.

>
> So the usage would be something like
>
> python -m zipapp [options] dir_to_zip
>
> Options:
> -p The interpreter to use on the shebang line
> (defaulting to /usr/bin/env python)
> -o archive_name The name of the output file (defaulting to the
> source directory name with a .pyz extension)
> If the argument has no extension, add '.pyz'
> -m module:function  The entry point to call (written to __main__.py)
> Using this is an error if there is a
> __main__.py, and mandatory if there isn't
>
> If you want anything more complex, it's easy enough to write your own
> script based on zipfile, or use one of the modules on PyPI.
>
> Does this sound reasonable? If it's OK, I'll go ahead and prepare an
> update to the PEP and an implementation. (Steve, looks like I may be
> learning how to maintain the wix files after all - wish me luck :-))
> If I hear no objections in the next couple of days, I'll assume
> everyone's OK with it and I'll prepare a PEP update and a patch.

Sounds good to me.

Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates
it as described? It seems a bit tidier than allocating a new PEP
number and rejecting PEP 441, when the revised proposal is essentially
just a simplified and renamed version of your original idea.

Cheers,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com<mailto:ncogh...@gmail.com>   |   
Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Steve Dower]

Paul Moore wrote:
> On 15 February 2015 at 18:25, Petr Viktorin  wrote:
>> On POSIXy systems the "python3" symlink is created in all venvs. I
>> thought (perhaps naïvely) that Windows doesn't do shebangs natively,
>> so there's some Python-specific mechanism around them, which should
>> handle "python3".
> 
> Windows doesn't have "python2" or "python3" commands, just "python".
> To choose which version you use, set PATH or use an explicit path to the
> executable.
> 
> The launcher offers shebang handling for scripts, and special-cases the
> following values for Unix compatibility:
> 
> #!/usr/bin/env python
> #!/usr/bin/python
> #!/usr/local/bin/python
> 
> They default to the "default Python" which is normally the latest Python 2
> version available, but which can be configured by the user.
> They also support adding an explicit version (python2, python3, python2.7,
> python3.4, ...)
> 
> As far as I know, this is *identical* behaviour to Unix - even to the 
> incredibly
> annoying (to me) choice of Python 2 as a default. So I reconfigure the default
> in my personal settings to Python 3. Unix users can do this too (although it 
> may
> involve a symlink in a ~/bin directory rather than an ini file change).

We could also add special-cases for "#!/usr/...python3" in the launcher on 
Windows.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Python-checkins] cpython: Issue #23152: Implement _Py_fstat() to support files larger than 2 GB on

[Steve Dower]

Agreed. I've made both of these changes.

Thanks for the suggestions

Cheers,
Steve

Top-posted from my Windows Phone

From: Brett Cannon
Sent: ‎2/‎21/‎2015 14:13
To: Ben Hoyt; Python-Dev
Subject: Re: [Python-Dev] [Python-checkins] cpython: Issue #23152: Implement 
_Py_fstat() to support files larger than 2 GB on



On Sat Feb 21 2015 at 4:23:16 PM Ben Hoyt 
mailto:benh...@gmail.com>> wrote:
When merging some changes while working on scandir, I noticed a minor issue 
with this commit:

https://hg.python.org/cpython/rev/4f6f4aa0d80f

The definition of "struct win32_stat" has been moved to fileutils.h and renamed 
to "struct _Py_stat_struct", which is fine -- however, the old "struct 
win32_stat" definition is still present (but unused) in posixmodule.c.

So I think the old "struct win32_stat { ... }" definition can simply be removed 
from posixmodule.c now.

I don't think win32_stat is part of the stable ABI so as long as everything 
keeps working then I don't see why it needs to stick around.


Also, unrelated to this commit, I notice the _Py_attribute_data_to_stat 
function (was attribute_data_to_stat) can't fail and always returns 0, and all 
callers ignore its return value anyway. Can it be changed to return void?

Don't see why not since it's a private API.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat

[Steve Dower]

Thanks! Fixed now.

Cheers,
Steve

Top-posted from my Windows Phone

From: Serhiy Storchaka<mailto:storch...@gmail.com>
Sent: ‎2/‎21/‎2015 14:14
To: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat 
to _Py_attribute_data_to_stat

On 21.02.15 20:04, steve.dower wrote:
> https://hg.python.org/cpython/rev/307713759a62
> changeset:   94720:307713759a62
> parent:  94718:4f6f4aa0d80f
> user:Steve Dower 
> date:Sat Feb 21 10:04:10 2015 -0800
> summary:
>Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat

What about time_t_to_FILE_TIME?


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat

[Steve Dower]

Why does it do that?

Top-posted from my Windows Phone

From: Serhiy Storchaka<mailto:storch...@gmail.com>
Sent: ‎2/‎21/‎2015 22:57
To: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat 
to _Py_attribute_data_to_stat

On 22.02.15 01:26, Steve Dower wrote:
> Thanks! Fixed now.

Clinic removes the declaration of _Py_time_t_to_FILE_TIME.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat to _Py_attribute_data_to_stat

[Steve Dower]

Ah, that makes sense. Feel free to move it up a few lines (or into a header if 
that's preferred) if it's bothering you. Otherwise I can do it in about 6-8 
hours.

Cheers,
Steve

Top-posted from my Windows Phone

From: Serhiy Storchaka<mailto:storch...@gmail.com>
Sent: ‎2/‎22/‎2015 7:29
To: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] cpython: Issue #23152: Renames attribute_data_to_stat 
to _Py_attribute_data_to_stat

On 22.02.15 16:12, Steve Dower wrote:
> Why does it do that?

Because it is in the section of generated code.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available

[Steve Dower]

Thanks for finding this. I'm following up on the issue of anyone else is having 
the same issue.

As an aside, it'd be great to hear if it's worked for anyone at all :)

Cheers,
Steve

Top-posted from my Windows Phone

From: Ben Hoyt
Sent: ‎3/‎9/‎2015 5:29
To: Paul Moore
Cc: Python Dev
Subject: Re: [Python-Dev] [RELEASED] Python 3.5.0a2 is now available

I'm seeing the same issue (though I also get the missing-DLL error dialog when 
I run python.exe). -Ben

On Mon, Mar 9, 2015 at 6:20 AM, Paul Moore 
mailto:p.f.mo...@gmail.com>> wrote:
On 9 March 2015 at 09:34, Larry Hastings 
mailto:la...@hastings.org>> wrote:
> On behalf of the Python development community and the Python 3.5 release
> team, I'm thrilled to announce the availability of Python 3.5.0a2.   Python
> 3.5.0a2 is the second alpha release of Python 3.5, which will be the next
> major release of Python.  Python 3.5 is still under heavy development, and
> is far from complete.

Hmm, I just tried to install the 64-bit "full installer" version, for
all users with the default options. This is on a PC that hasn't had
3.5 installed before, and doesn't have Visual Studio 2015 installed.
When it got to the step "precompiling standard library" I got an error
window pop up saying "python.exe - system error. The program can't
start because api-ms-win-crt-runtime-l1-1-0.dll is missing from your
computer. Try reinstalling the program to fix this problem." All there
was was an "OK" button. Pressing that told me "Setup was successful"
but then "py -3.5 -V" gives me nothing (no error, no version, just
returns me to the command prompt). Same result if I do "& 'C:\Program
Files\Python 3.5\python.exe' -V".

Python 3.5.0a2 (64-bit) is showing in my "Add/Remove Programs".

This is Windows 7, 64-bit.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/benhoyt%40gmail.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc)

[Steve Dower]

Paul Moore wrote:
> I just thought I'd give installing Python 3.5 a go on my PC, now that
> 3.5a2 has come out. I didn't get very far (see earlier message) but it 
> prompted
> me to think about how I'd use it, and what habits I'd need to change.
> 
> I'd suggest that the "what's new in 3.5" document probably needs a section on
> the new installer that explains this stuff...

This is true. Right now I'm in experimentation mode, and being more aggressive 
about changing things than is probably a good idea (because it solicits 
feedback like this :) ). When things settle down I expect to end up closer to 
where we started, so there's not a huge amount of value in writing it all up 
right now. I'll get there.

> First of all, I always use "all users" installs, so I have Python in "Program
> Files" now. As a result, doing "pip install foo" requires elevation. As 
> that's a
> PITA, I probably need to switch to using "pip install --user". All that's 
> fine,
> and from there "py -3.5" works fine, as does "py -3.5 -m foo". But even if it
> is, not every entry point has a corresponding "-m" invocation (pygments'
> pygmentize command doesn't seem to, for example)

I know you're already involved in this Paul, but for everyone else there's a 
big discussion going on at https://github.com/pypa/pip/issues/1668 about 
changing pip's default behaviour to handle falling back to --user automatically.

> But suppose I want to put Python 3.5 on my PATH. The installer has an "add
> Python to PATH" checkbox, but (if I'm reading the WiX source right, I didn't
> select that on install) that doesn't add the user directory. So I need to add
> that to my PATH. Is that right? And of course, that means I need to *know* the
> user site directory ($env:LOCALAPPDATA\Python\Python35\Scripts), correct?

Correct. There's no way to add a per-user directory to PATH from an all-users 
installation (except for a few approaches that I expect/hope would trigger 
malware detectors...)

> Maybe the answer is that we simply start recommending that everyone on Windows
> uses per-user installs. It makes little difference to me (beyond the fact that
> when I want to look at the source of something in the stdlib, the location of
> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) 
> but
> I doubt it's what most people will expect.

I'm okay with this. Installing for all users is really something that could be 
considered an advanced option rather than the default, especially since the aim 
(AIUI) of the all-users install is to pretend that Python was shipped with the 
OS. (I'd kind of like to take that further by splitting things more sensibly 
between Program Files, Common Files and System32, but there's very little gain 
from that and much MUCH pain as long as people are still expecting C:\PythonXY 
installs...)

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Windows installer - File associations in "per user" installs

[Steve Dower]

It's a bug. File and assign to me please.

Top-posted from my Windows Phone

From: Paul Moore<mailto:p.f.mo...@gmail.com>
Sent: ‎3/‎10/‎2015 3:35
To: Steve Dower<mailto:steve.do...@microsoft.com>
Cc: Python Dev<mailto:python-dev@python.org>
Subject: Windows installer - File associations in "per user" installs

On 9 March 2015 at 15:37, Steve Dower  wrote:
>> Maybe the answer is that we simply start recommending that everyone on 
>> Windows
>> uses per-user installs. It makes little difference to me (beyond the fact 
>> that
>> when I want to look at the source of something in the stdlib, the location of
>> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) 
>> but
>> I doubt it's what most people will expect.
>
> I'm okay with this. Installing for all users is really something that could 
> be considered an advanced option rather than the default, especially since 
> the aim (AIUI) of the all-users install is to pretend that Python was shipped 
> with the OS. (I'd kind of like to take that further by splitting things more 
> sensibly between Program Files, Common Files and System32, but there's very 
> little gain from that and much MUCH pain as long as people are still 
> expecting C:\PythonXY installs...)

I've just tried a per-user install of Python 3.5a2. The machine in
question previously had (and still has) a system install of 3.4, with
"Make this Python the default" selected (so the .py extension is
associated with that version and specifically the 3.4 launcher).

I didn't get the option to associate .py files with 3.5 (there's *no
way* I'd consider that to be advanced usage - if I'm installing
Python, why wouldn't I want to associate it with .py files [1]) and I
still seem to have .py associated with the 3.4 launcher, not the 3.5
one that's in my %APPDATA% folder.

>cmd /c assoc .py
.py=Python.File
>cmd /c ftype python.file
python.file="C:\WINDOWS\py.exe" "%1" %*

I'm happy if a per-user install of 3.5 makes a per-user filetype
association (assuming such a thing is possible, I've never tried it
before) but it's absolutely not OK if we're planning on recommending
an install type that doesn't create the right association.

Paul

[1] Given that I have 3.4 and am installing an experimental 3.5
version, it's not actually at all clear cut which version I want as my
default. In all honesty, I don't think this decision is actually
something that should be defaulted. Maybe the "don't make the user
make any choices in the default selection" approach has gone a little
too far here?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] libffi embedded in CPython

[Steve Dower]

Zachary Ware wrote:
> On Mar 25, 2015 4:22 AM, "Paul Moore"  wrote:
>> On a related note, is there any information available on how the
>> "externals" repo is maintained? In particular, should things in there
>> be exact copies of upstream, or is it OK to include extra data (in
>> this case, the results of running "configure" for the Windows build)?
>> It works for me either way, it's just a matter of how the build
>> process would be structured and maintained.
> 
> Its not extremely clear how it's "supposed to be" done; look at the 
> differences
> between how we handle OpenSSL and Tcl/Tk, for example. One way or the other,
> though, we will store the configure output so that our build doesn't depend on
> any more than it absolutely has to.

It would be nice for at least one other person to know how to do it. I can't 
see the SVN logs, but I feel like you're the only person who's touched it in a 
while :)

As well as the configure output, we sometimes include very light patching (for 
example, Tcl/Tk currently has a patch to their makefile to work around a VC14 
bug). I'm not sure whether that's considered good practice or not, but I prefer 
only needing to do it once vs. building it into the build system.

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop it?

[Steve Dower]

We discussed that a while back and decided the PEP was sufficient, but I'm not 
entirely opposed to a reminder.

Probably best to repurpose this buildbot for 2.7. David has a Windows 8.1 one 
that should be marked as stable for 3.x (though it's not quite stable yet... 
AFAIK it's the only machine with VS 2015 other than mine).

Cheers,
Steve

Top-posted from my Windows Phone

From: Victor Stinner
Sent: ‎3/‎28/‎2015 16:14
To: David Bolen
Cc: python-dev@python.org
Subject: Re: [Python-Dev] Buildbot x86 XP-4 3.x doesn't compile anymore: drop 
it?


Le 28 mars 2015 21:15, "David Bolen" 
mailto:db3l@gmail.com>> a écrit :

> I'm assuming you aren't suggesting turning off the XP buildbot
> entirely, correct?

I'm talking about the 3.x slave, so only python 3.5.

Would it be possible to specify in What's New In Python 3.5 which Windows 
versions are no more supported?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Do we need to sign Windows files with GnuPG?

[Steve Dower]

Larry Hastings wrote:
> Steve's also changed the authentication process. His new installers rely on a
> Windows digital signature technology called Authenticode where the signature 
> is
> built right into the .exe file. Windows platforms will automatically
> authenticate executables signed with Authenticode, so this is both secure and
> convenient.
> 
> Martin's build process also digitally signed the files he built, but not using
> Authenticode (or at least I don't think so). Like the Mac and source code
> releases, his automation used GnuPG to produce separate ".asc" files 
> containing
> digital signatures. This meant authentication was a manual process.

Martin previously only signed the installer with Authenticode, and generated a 
signature with GnuPG for the installer. My change now signs every binary and 
MSI in the entire installation with Authenticode, and for now I've stopped 
creating a GPG signature for the installers. I'm still providing sizes and MD5 
hashes for the user-visible downloads (except for the last alpha release, 
thanks Larry for covering for me).

With the installer also being a downloader, there are now actually 30+ files 
uploaded for each Windows release. Most of these are never seen by users unless 
they run the installer with /layout (sorry for not having changed this to 
/download yet... it's not as easily customizable as I'd hoped, but /layout is 
the standard name for this command), and if they're being downloaded by the 
installer then both hashes (embedded in the installer) and Authenticode 
signatures (embedded in each file) are being checked and will be blocked if 
they don't match. So verifying the EXE installer should always be sufficient to 
trust the rest of the installable files.

> The Authenticode approach sounds great. But there are advantages to the GnuPG
> approach too:

For reference, the main advantage of Authenticode signing is shown at 
https://technet.microsoft.com/en-us/library/dd835561(v=ws.10).aspx - about 
halfway down there are screenshots of the various dialogs that are displayed 
when you run signed vs. unsigned vs. blocked applications.

It also helps bypass SmartScreen, which will block downloaded files until 
they've developed a minimum level of trust. Simply having an Authenticode 
signature on the initial download meets this level.

(The summary of my opinion is that these two checks are sufficient for the 
initial EXE download, and the embedded hashes and signatures are sufficient for 
the rest. Having python.exe et al signed is a bonus that we've never done in 
the past.)

> * Using GnuPG means we can authenticate the files from any platform, not just
> Windows. If there were a security breach on the Python content delivery 
> network,
> any developer could get GnuPG for their platform and authenticate that the
> installers are unmodified. If we use Authenitcode,

There are tools out there for validating Authenticode on Linux, though none of 
them seem to be as complete as on Windows (it really needs the OS certificate 
store to be completely reliable), so I can certainly see the value in being 
able to verify these against a signature. My only question is whether/how this 
is better with GPG compared to say a SHA hash? I don't currently have a GPG key 
(to my knowledge), so it's not like there's any preexisting trust to build from 
- or am I misunderstanding how GPG works here?

> * GnuPG is agnostic about the data it digitally signs. So, for example, 
> Martin's
> build process digitally signs the Windows help file--the ".chm" file--produced
> by his build process. The help file Steve builds is currently completely
> unsigned; Steve says he can try signing it but he's not sure it'll work. Note
> that .chm files actually can contain live code, so this is at least a 
> plausible
> vector for attack.

Authenticode is not supported for CHM files, unfortunately. If this is the only 
file that we decide needs GPG, I'd vote to stop offering the download apart 
from the interpreter :) (Among other things, I'm not supposed to use GPG 
without specific permission from the lawyers at work because of the license...)

> My Windows development days are firmly behind me. So I don't really have an
> opinion here. So I put it to you, Windows Python developers: do you care about
> GnuPG signatures on Windows-specific files? Or do you not care?

The later replies seem to suggest that they are general goodness that nobody on 
Windows will use. If someone convinces me (or steamrolls me, that's fine too) 
that the goodness of GPG is better than a hash then I'll look into adding it 
into the process. Otherwise I'll happily add hash generation into the upload 
process (which I'm going to do anyway for the ones displayed on the download 
page).

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/pyth

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

[Steve Dower]

The thing is, that's exactly the same goodness as Authenticode gives, except 
everyone gets that for free and meanwhile you're the only one who has admitted 
to using GPG on Windows :)

Basically, what I want to hear is that GPG sigs provide significantly better 
protection than hashes (and I can provide better than MD5 for all files if it's 
useful), taking into consideration that (I assume) I'd have to obtain a signing 
key for GPG and unless there's a CA involved like there is for Authenticode, 
there's no existing trust in that key.

Cheers,
Steve

Top-posted from my Windows Phone

From: M.-A. Lemburg<mailto:m...@egenix.com>
Sent: ‎4/‎3/‎2015 10:55
To: Steve Dower<mailto:steve.do...@microsoft.com>; Larry 
Hastings<mailto:la...@hastings.org>; Python Dev<mailto:python-dev@python.org>; 
python-committers<mailto:python-committ...@python.org>
Subject: Re: [python-committers] [Python-Dev] Do we need to sign Windows files 
with GnuPG?

On 03.04.2015 19:35, Steve Dower wrote:
>> My Windows development days are firmly behind me. So I don't really have an
>> opinion here. So I put it to you, Windows Python developers: do you care 
>> about
>> GnuPG signatures on Windows-specific files? Or do you not care?
>
> The later replies seem to suggest that they are general goodness that nobody 
> on Windows will use. If someone convinces me (or steamrolls me, that's fine 
> too) that the goodness of GPG is better than a hash then I'll look into 
> adding it into the process. Otherwise I'll happily add hash generation into 
> the upload process (which I'm going to do anyway for the ones displayed on 
> the download page).

FWIW: I regularly check the GPG sigs on all important downloaded
files, regardless of which platform they target, including the
Windows installers for Python or any other Windows installers
I use which provide such sigs.

The reason is simple:
The signature is a proof of authenticity which is not bound to
a particular file format or platform and before running .exes
it's good to know that they were built by the right people and
not manipulated by trojans, viruses or malicious proxies.

Is that a good enough reason to continue providing the GPG
sigs or do you need more proof of goodness ? ;-)

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

[Steve Dower]

"Authenticode does not have a PKI"

If you got that from this discussion, I need everyone to at least skim read 
this:  https://msdn.microsoft.com/en-us/library/ie/ms537361(v=vs.85).aspx

Authenticode uses the same certificate infrastructure as SSL (note: not the 
same certificates). As I see it, anyone running on Windows has access to 
verification that is at least as good as GPG, and the only people who would 
benefit from GPG sigs are those checking Windows files on another OS or those 
with an existing GPG workflow on Windows (before this thread, I knew nobody who 
used GPG on Windows for anything, so forgive me for thinking this is very rare).

Cheers,
Steve

Top-posted from my Windows Phone

From: Wes Turner<mailto:wes.tur...@gmail.com>
Sent: ‎4/‎4/‎2015 6:42
To: M. -A. Lemburg<mailto:m...@egenix.com>
Cc: Python-Dev<mailto:python-dev@python.org>; 
python-committers<mailto:python-committ...@python.org>; Larry 
Hastings<mailto:la...@hastings.org>; Steve 
Dower<mailto:steve.do...@microsoft.com>
Subject: Re: [Python-Dev] [python-committers] Do we need to sign Windows files 
with GnuPG?


So, AFAIU from this discussion:

* Authenticode does not have a PKI
* GPG does have PKI
* ASC signatures are signed checksums

As far as downstream packaging on Windows (people who should/could be 
subscribed to release ANNs):

For Choclatey NuGet:

* https://chocolatey.org/packages/python
* https://chocolatey.org/packages/python.x86
* https://chocolatey.org/packages/python2
* https://chocolatey.org/packages/python-x86_32
* https://chocolatey.org/packages/python3

Python(x,y):

* https://code.google.com/p/pythonxy/

For Anaconda (the MS Azure chosen python distribution):

* http://docs.continuum.io/anaconda/install.html#windows-install

...

These should/could/are checking GPG signatures for Windows packages downstream.

http://www.scipy.org/install.html

On Apr 3, 2015 5:38 PM, "M.-A. Lemburg" 
mailto:m...@egenix.com>> wrote:
On 04.04.2015 00:14, Steve Dower wrote:
> The thing is, that's exactly the same goodness as Authenticode gives, except 
> everyone gets that for free and meanwhile you're the only one who has 
> admitted to using GPG on Windows :)
>
> Basically, what I want to hear is that GPG sigs provide significantly better 
> protection than hashes (and I can provide better than MD5 for all files if 
> it's useful), taking into consideration that (I assume) I'd have to obtain a 
> signing key for GPG and unless there's a CA involved like there is for 
> Authenticode, there's no existing trust in that key.

Hashes only provide checks against file corruption (and then
only if you can trust the hash values). GPG provides all the
benefits of public key encryption on arbitrary files (not just
code).

The main benefit in case of downloadable installers is to
be able to make sure that the files are authentic, meaning that
they were created and signed by the people listed as packagers.

There is no CA infrastructure involved as for SSL certificates
or Authenticode, but it's easy to get the keys from key servers
given the key signatures available from python.org<http://python.org>'s download
pages.

If you want to sign a package file using GPG, you will need
to create your own key, upload it to the key servers and then
place the signature up on the download page.

Relying only on Authenticode for Windows installers would
result in a break in technology w/r to the downloads we
make available for Python, since all other files are (usually)
GPG signed:

https://www.python.org/ftp/python/3.4.3/

Cheers,
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/


> Cheers,
> Steve
>
> Top-posted from my Windows Phone
> 
> From: M.-A. Lemburg<mailto:m...@egenix.com<mailto:m...@egenix.com>>
> Sent: ‎4/‎3/‎2015 10:55
> To: Steve 
> Dower<mailto:steve.do...@microsoft.com<mailto:steve.do...@microsoft.com>>; 
> Larry Hastings<mailto:la...@hastings.org<mailto:la...@hastings.org>>; Python 
> Dev<mailto:python-dev@python.org<mailto:python-dev@python.org>>; 
> python-committers<mailto:

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

[Steve Dower]

"Relying only on Authenticode for Windows installers would result in a break in 
technology w/r to the downloads we make available for Python, since all other 
files are (usually) GPG signed"

This is the point of this discussion. I'm willing to make such a break because 
I believe Authenticode is so much more convenient for end users that it isn't 
worth producing GPG signatures. So far, the responses seem to be:

"I'd use them on Windows" x1
"I'd consider using them on another OS" x2-3
"Please don't change" everyone else

At least that's the impression I'm getting, so I hope that helps clarify why 
I'm still not convinced it's that critical.

Cheers,
Steve

Top-posted from my Windows Phone
____
From: M.-A. Lemburg<mailto:m...@egenix.com>
Sent: ‎4/‎3/‎2015 15:38
To: Steve Dower<mailto:steve.do...@microsoft.com>; Larry 
Hastings<mailto:la...@hastings.org>; Python Dev<mailto:python-dev@python.org>; 
python-committers<mailto:python-committ...@python.org>
Subject: Re: [python-committers] [Python-Dev] Do we need to sign Windows files 
with GnuPG?

On 04.04.2015 00:14, Steve Dower wrote:
> The thing is, that's exactly the same goodness as Authenticode gives, except 
> everyone gets that for free and meanwhile you're the only one who has 
> admitted to using GPG on Windows :)
>
> Basically, what I want to hear is that GPG sigs provide significantly better 
> protection than hashes (and I can provide better than MD5 for all files if 
> it's useful), taking into consideration that (I assume) I'd have to obtain a 
> signing key for GPG and unless there's a CA involved like there is for 
> Authenticode, there's no existing trust in that key.

Hashes only provide checks against file corruption (and then
only if you can trust the hash values). GPG provides all the
benefits of public key encryption on arbitrary files (not just
code).

The main benefit in case of downloadable installers is to
be able to make sure that the files are authentic, meaning that
they were created and signed by the people listed as packagers.

There is no CA infrastructure involved as for SSL certificates
or Authenticode, but it's easy to get the keys from key servers
given the key signatures available from python.org's download
pages.

If you want to sign a package file using GPG, you will need
to create your own key, upload it to the key servers and then
place the signature up on the download page.

Relying only on Authenticode for Windows installers would
result in a break in technology w/r to the downloads we
make available for Python, since all other files are (usually)
GPG signed:

https://www.python.org/ftp/python/3.4.3/

Cheers,
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/


::: Try our new mxODBC.Connect Python Database Interface for free ! 


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/


> Cheers,
> Steve
>
> Top-posted from my Windows Phone
> 
> From: M.-A. Lemburg<mailto:m...@egenix.com>
> Sent: ‎4/‎3/‎2015 10:55
> To: Steve Dower<mailto:steve.do...@microsoft.com>; Larry 
> Hastings<mailto:la...@hastings.org>; Python 
> Dev<mailto:python-dev@python.org>; 
> python-committers<mailto:python-committ...@python.org>
> Subject: Re: [python-committers] [Python-Dev] Do we need to sign Windows 
> files with GnuPG?
>
> On 03.04.2015 19:35, Steve Dower wrote:
>>> My Windows development days are firmly behind me. So I don't really have an
>>> opinion here. So I put it to you, Windows Python developers: do you care 
>>> about
>>> GnuPG signatures on Windows-specific files? Or do you not care?
>>
>> The later replies seem to suggest that they are general goodness that nobody 
>> on Windows will use. If someone convinces me (or steamrolls me, that's fine 
>> too) that the goodness of GPG is better than a hash then I'll look into 
>> adding it into the process. Otherwise I'll happily add hash generation into 
>> the upload process (which I'm going to do anyway for the ones displayed on 
>> the download page).
>
> FWIW: I regularly check the GPG sigs on all important

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

[Steve Dower]

Small clarification: there certificates *are* the same format as for SSL, and 
OpenSSL it's able to validate them in the same way as well as generate them 
(but not extract embedded ones, AFAICT). But generally SSL certificates are not 
marked as suitable for code signing so you need to buy a separate one.

Both Martin and I have the PSF's code signing cert private key, which is how we 
can sign with the "Python Software Foundation" name. The public key is embedded 
into every signed file, just like an SSL cert is attached to a site or an 
S/MIME cert is embedded in a signed email.

Cheers,
Steve

Top-posted from my Windows Phone
________
From: Steve Dower<mailto:steve.do...@microsoft.com>
Sent: ‎4/‎4/‎2015 7:25
To: Wes Turner<mailto:wes.tur...@gmail.com>; M. -A. 
Lemburg<mailto:m...@egenix.com>
Cc: python-committers<mailto:python-committ...@python.org>; 
Python-Dev<mailto:python-dev@python.org>
Subject: Re: [python-committers] [Python-Dev] Do we need to sign Windows files 
with GnuPG?

"Authenticode does not have a PKI"

If you got that from this discussion, I need everyone to at least skim read 
this:  https://msdn.microsoft.com/en-us/library/ie/ms537361(v=vs.85).aspx

Authenticode uses the same certificate infrastructure as SSL (note: not the 
same certificates). As I see it, anyone running on Windows has access to 
verification that is at least as good as GPG, and the only people who would 
benefit from GPG sigs are those checking Windows files on another OS or those 
with an existing GPG workflow on Windows (before this thread, I knew nobody who 
used GPG on Windows for anything, so forgive me for thinking this is very rare).

Cheers,
Steve

Top-posted from my Windows Phone

From: Wes Turner<mailto:wes.tur...@gmail.com>
Sent: ‎4/‎4/‎2015 6:42
To: M. -A. Lemburg<mailto:m...@egenix.com>
Cc: Python-Dev<mailto:python-dev@python.org>; 
python-committers<mailto:python-committ...@python.org>; Larry 
Hastings<mailto:la...@hastings.org>; Steve 
Dower<mailto:steve.do...@microsoft.com>
Subject: Re: [Python-Dev] [python-committers] Do we need to sign Windows files 
with GnuPG?


So, AFAIU from this discussion:

* Authenticode does not have a PKI
* GPG does have PKI
* ASC signatures are signed checksums

As far as downstream packaging on Windows (people who should/could be 
subscribed to release ANNs):

For Choclatey NuGet:

* https://chocolatey.org/packages/python
* https://chocolatey.org/packages/python.x86
* https://chocolatey.org/packages/python2
* https://chocolatey.org/packages/python-x86_32
* https://chocolatey.org/packages/python3

Python(x,y):

* https://code.google.com/p/pythonxy/

For Anaconda (the MS Azure chosen python distribution):

* http://docs.continuum.io/anaconda/install.html#windows-install

...

These should/could/are checking GPG signatures for Windows packages downstream.

http://www.scipy.org/install.html

On Apr 3, 2015 5:38 PM, "M.-A. Lemburg" 
mailto:m...@egenix.com>> wrote:
On 04.04.2015 00:14, Steve Dower wrote:
> The thing is, that's exactly the same goodness as Authenticode gives, except 
> everyone gets that for free and meanwhile you're the only one who has 
> admitted to using GPG on Windows :)
>
> Basically, what I want to hear is that GPG sigs provide significantly better 
> protection than hashes (and I can provide better than MD5 for all files if 
> it's useful), taking into consideration that (I assume) I'd have to obtain a 
> signing key for GPG and unless there's a CA involved like there is for 
> Authenticode, there's no existing trust in that key.

Hashes only provide checks against file corruption (and then
only if you can trust the hash values). GPG provides all the
benefits of public key encryption on arbitrary files (not just
code).

The main benefit in case of downloadable installers is to
be able to make sure that the files are authentic, meaning that
they were created and signed by the people listed as packagers.

There is no CA infrastructure involved as for SSL certificates
or Authenticode, but it's easy to get the keys from key servers
given the key signatures available from python.org<http://python.org>'s download
pages.

If you want to sign a package file using GPG, you will need
to create your own key, upload it to the key servers and then
place the signature up on the download page.

Relying only on Authenticode for Windows installers would
result in a break in technology w/r to the downloads we
make available for Python, since all other files are (usually)
GPG signed:

https://www.python.org/ftp/python/3.4.3/

Cheers,
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

[Steve Dower]

There's no problem, per se, but initially it was less trouble to use the 
trusted PSF certificate and native support than to add an extra step using a 
program I don't already use and trust, am restricted in use by my employer 
(because of the license and the fact there are alternatives), and developing 
the trust in a brand new certificate.

Eventually the people saying "do it" will win through sheer persistence, since 
I'll get sick of trying to get a more detailed response and just concede. Not 
sure if that's how we want to be running the project though...

Top-posted from my Windows Phone

From: Barry Warsaw<mailto:ba...@python.org>
Sent: ‎4/‎4/‎2015 9:11
To: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] [python-committers] Do we need to sign Windows files 
with GnuPG?

On Apr 04, 2015, at 02:41 PM, Steve Dower wrote:

>"Relying only on Authenticode for Windows installers would result in a break
>in technology w/r to the downloads we make available for Python, since all
>other files are (usually) GPG signed"

It's the "only" part I have a question about.

Does the use of Authenticode preclude detached GPG signatures of the exe file?
I can't see how it would, but maybe there's something (well, a lot of
somethings ;) I don't know about Windows.

If not, then what's the problem with also providing a GPG signature?

Cheers,
-Barry
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

[Steve Dower]

Nathaniel Smith wrote:
> And I suspect python-dev generally doesn't put much weight on the 
> extra effort required (release managers have all been using gpg for
> decades, it's pretty trivial)

I'm aware of this, but still don't see it as a reason to unnecessarily 
duplicate process.

> or see any reason why Microsoft's internal GPL-hate should have any
> effect on the PSF's behaviour.

Seems the "internal GPL-hate" has softened even more than I was aware. The 
history for GPG was spotty, but my request was automatically approved, so I 
guess the line has been moved far enough away that I've lost that excuse :)

Now I just have to find the time to learn how to use it...

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Do we need to sign Windows files with GnuPG?

[Steve Dower]

"One question, if you will - I don't think this was asked so far - is
authenticode verifiable from Linux, without Windows? And does it work
for users of WINE ?"

I've seen some info suggesting that it's verifiable, but you do need to extract 
the cert and calculate the hash against less than the signed file. Seemed like 
Mono had a tool for it, but OpenSSL can handle the cert.

Currently the new installer doesn't run on Wine because of missing APIs (since 
I want to discuss alternate distribution ideas I haven't treated this as a 
priority), and I've heard they haven't implemented enough crypto yet to handle 
it, but that could be outdated.

"GPG sigs will provide protection against replay attacks"

How does this work?

Cheers,
Steve

Top-posted from my Windows Phone

From: Robert Collins<mailto:robe...@robertcollins.net>
Sent: ‎4/‎4/‎2015 21:59
To: Steve Dower<mailto:steve.do...@microsoft.com>
Cc: M.-A. Lemburg<mailto:m...@egenix.com>; Larry 
Hastings<mailto:la...@hastings.org>; Python Dev<mailto:python-dev@python.org>; 
python-committers<mailto:python-committ...@python.org>
Subject: Re: [Python-Dev] [python-committers] Do we need to sign Windows files 
with GnuPG?

On 4 April 2015 at 11:14, Steve Dower  wrote:
> The thing is, that's exactly the same goodness as Authenticode gives, except
> everyone gets that for free and meanwhile you're the only one who has
> admitted to using GPG on Windows :)
>
> Basically, what I want to hear is that GPG sigs provide significantly better
> protection than hashes (and I can provide better than MD5 for all files if
> it's useful), taking into consideration that (I assume) I'd have to obtain a
> signing key for GPG and unless there's a CA involved like there is for
> Authenticode, there's no existing trust in that key.

GPG sigs will provide protection against replay attacks [unless we're
proposing to revoke signatures on old point releases with known
security vulnerabilities - something that Window software vendors tend
not to do because of the dramatic and immediate effect on the deployed
base...]

This is not relevant for things we're hosting on SSL, but is if anyone
is mirroring our installers around. They dont' seem to be so perhaps
its a bit 'meh'.

OTOH I also think there is value in consistency: signing all our
artifacts makes checking back on them later easier, should we need to.

One question, if you will - I don't think this was asked so far - is
authenticode verifiable from Linux, without Windows? And does it work
for users of WINE ?

-Rob


--
Robert Collins 
Distinguished Technologist
HP Converged Cloud
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python3 Stable ABI

[Steve Dower]

I don't see any obvious issues, but there may be some that don't need to be 
marked stable.

Given that a mismatch here will cause build errors for users, I'm +1 on 
checking this in.

Cheers,
Steve

Top-posted from my Windows Phone

From: Zachary Ware
Sent: ‎4/‎13/‎2015 17:29
To: Python-Dev
Subject: [Python-Dev] Python3 Stable ABI

In issue23903, I've created a script that will produce PC/python3.def
by scraping the header files in Include.  There are are many many
discrepencies between what my script generates and what is currently
in the repository (diff below), but in every case I've checked the
script has been right: what the script finds is actually exported as
part of the limited API, but due to not being in the .def file it's
not actually exported from python3.dll.  Almost all of the differences
are things that the script found that weren't present, but there are a
couple things going the other way.

The point of this message is to ask everybody who maintains anything
in C to take a look through and make sure everything in their area is
properly guarded (or not) by Py_LIMITED_API.  Alternately, if somebody
can find a bug in my script and brain that's finding too much stuff,
that would be great too.

Ideally, after this is all settled I'd like to add the script to both
the Makefile and the Windows build system, such that PC/python3.def is
always kept up to date and flags changes that weren't meant to be
made.

Regards,
--
Zach

(I'm afraid Gmail might mangle this beyond recognition, you can find the diff at
http://bugs.python.org/review/23903/diff/14549/PC/python3.def
if it does.)

diff -r 24f2c0279120 PC/python3.def
--- a/PC/python3.defMon Apr 13 15:51:59 2015 -0500
+++ b/PC/python3.defMon Apr 13 16:10:34 2015 -0500
@@ -1,13 +1,15 @@
 ; This file specifies the import forwarding for python3.dll
 ; It is used when building python3dll.vcxproj
+; Generated by python3defgen.py, DO NOT modify directly!
 LIBRARY"python3"
 EXPORTS
+  PyAST_FromNode=python35.PyAST_FromNode
+  PyAST_FromNodeObject=python35.PyAST_FromNodeObject
+  PyAST_Validate=python35.PyAST_Validate
   PyArg_Parse=python35.PyArg_Parse
   PyArg_ParseTuple=python35.PyArg_ParseTuple
   PyArg_ParseTupleAndKeywords=python35.PyArg_ParseTupleAndKeywords
   PyArg_UnpackTuple=python35.PyArg_UnpackTuple
-  PyArg_VaParse=python35.PyArg_VaParse
-  PyArg_VaParseTupleAndKeywords=python35.PyArg_VaParseTupleAndKeywords
   PyArg_ValidateKeywordArguments=python35.PyArg_ValidateKeywordArguments
   PyBaseObject_Type=python35.PyBaseObject_Type DATA
   PyBool_FromLong=python35.PyBool_FromLong
@@ -39,7 +41,6 @@
   PyCFunction_GetFlags=python35.PyCFunction_GetFlags
   PyCFunction_GetFunction=python35.PyCFunction_GetFunction
   PyCFunction_GetSelf=python35.PyCFunction_GetSelf
-  PyCFunction_New=python35.PyCFunction_New
   PyCFunction_NewEx=python35.PyCFunction_NewEx
   PyCFunction_Type=python35.PyCFunction_Type DATA
   PyCallIter_New=python35.PyCallIter_New
@@ -58,6 +59,7 @@
   PyCapsule_SetPointer=python35.PyCapsule_SetPointer
   PyCapsule_Type=python35.PyCapsule_Type DATA
   PyClassMethodDescr_Type=python35.PyClassMethodDescr_Type DATA
+  PyCmpWrapper_Type=python35.PyCmpWrapper_Type DATA
   PyCodec_BackslashReplaceErrors=python35.PyCodec_BackslashReplaceErrors
   PyCodec_Decode=python35.PyCodec_Decode
   PyCodec_Decoder=python35.PyCodec_Decoder
@@ -68,6 +70,7 @@
   PyCodec_IncrementalEncoder=python35.PyCodec_IncrementalEncoder
   PyCodec_KnownEncoding=python35.PyCodec_KnownEncoding
   PyCodec_LookupError=python35.PyCodec_LookupError
+  PyCodec_NameReplaceErrors=python35.PyCodec_NameReplaceErrors
   PyCodec_Register=python35.PyCodec_Register
   PyCodec_RegisterError=python35.PyCodec_RegisterError
   PyCodec_ReplaceErrors=python35.PyCodec_ReplaceErrors
@@ -122,6 +125,7 @@
   PyErr_Fetch=python35.PyErr_Fetch
   PyErr_Format=python35.PyErr_Format
   PyErr_FormatV=python35.PyErr_FormatV
+  PyErr_GetExcInfo=python35.PyErr_GetExcInfo
   PyErr_GivenExceptionMatches=python35.PyErr_GivenExceptionMatches
   PyErr_NewException=python35.PyErr_NewException
   PyErr_NewExceptionWithDoc=python35.PyErr_NewExceptionWithDoc
@@ -132,14 +136,25 @@
   PyErr_PrintEx=python35.PyErr_PrintEx
   PyErr_ProgramText=python35.PyErr_ProgramText
   PyErr_Restore=python35.PyErr_Restore
+  PyErr_SetExcFromWindowsErr=python35.PyErr_SetExcFromWindowsErr
+  
PyErr_SetExcFromWindowsErrWithFilename=python35.PyErr_SetExcFromWindowsErrWithFilename
+  
PyErr_SetExcFromWindowsErrWithFilenameObject=python35.PyErr_SetExcFromWindowsErrWithFilenameObject
+  
PyErr_SetExcFromWindowsErrWithFilenameObjects=python35.PyErr_SetExcFromWindowsErrWithFilenameObjects
+  PyErr_SetExcInfo=python35.PyErr_SetExcInfo
+  PyErr_SetExcWithArgsKwargs=python35.PyErr_SetExcWithArgsKwargs
   PyErr_SetFromErrno=python35.PyErr_SetFromErrno
   PyErr_SetFromErrnoWithFilename=python35.PyErr_SetFromErrnoWithFilename
   
PyErr_S

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.5.0a4 is now available

[Steve Dower]

Paul Moore wrote:
> On 20 April 2015 at 09:16, Larry Hastings  wrote:
>> There is now a third type of Windows installer for Python 3.5.  In addition
>> to the conventional installer and the web-based installer, Python 3.5 now
>> has an embeddable installer designed to be run as part of a larger
>> application's installer for apps using or extending Python.
> 
> Probably a question for Steve mostly, but how does this work, exactly?
> I see it's an exe - I was sort of expecting a zip file.

I made it a self-extracting RAR file so it could be signed, but I've already 
had multiple people query it so the next release will probably just be a plain 
ZIP file. I just need to figure out some reliable way of validating the 
download other than GPG, since I'd like installers to be able to do the 
download transparently and ideally without hard-coding hash values. I might add 
a CSV of SHA hashes to the zip too.

> Specifically, I was considering using this to make a build of Vim,
> with Python support, using a copy of Python included with Vim (so no
> reliance on there being a "system" install of Python). As Vim is open
> source, "build it yourself", I typically just compile vim and then zip
> up the application directory. So I'm not clear how an exe "embeddable"
> installer fits into that scheme. Apologies if the exe is just a
> self-extracting archive, and the answer is just "run it and tell it
> where to dump the files". I've not even had a chance to get to
> somewhere I can download the installer to try it yet :-)

It is indeed just a run-and-dump extractor. I haven't had a chance to write up 
any docs for it yet, but there are some open bugs I want to fix first 
(specifically http://bugs.python.org/issue23955) before this becomes too 
formalized.

Cheers,
Steve

> Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.5.0a4 is now available

[Steve Dower]

Donald Stufft wrote:
> Is this version statically linked by any chance?

No, there's no change to the compilation process, so you can get exactly the 
same result by using the regular installer and copying the files around.

Not sure if this is what you're referring to, but on Windows DLLs are always 
loaded from alongside the executable before searching in other locations. I 
learned at PyCon that some platforms embed full paths for .so's (I'd always 
just assumed that there was a similar resolution process), but this is not the 
case here. Dropping all the DLLs and PYDs in the same location is sufficient to 
make sure you always load the right files, unless sys.path has been filled in 
with incorrect registry values (the bug I referenced).

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] [RELEASED] Python 3.5.0a4 is now available

[Steve Dower]

Whoops, sorry. Yeah, I knew about that behavior, but in hindsight it's 
obviously going to be a surprise for others :)

One plain zip file coming up for the next release. All the binaries will be 
signed and paranoid people can check the GPG sig and embed their own hashes if 
not the files themselves.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Moore<mailto:p.f.mo...@gmail.com>
Sent: ‎4/‎22/‎2015 5:48
To: Steve Dower<mailto:steve.do...@microsoft.com>
Cc: Larry Hastings<mailto:la...@hastings.org>; Python 
Dev<mailto:python-dev@python.org>
Subject: Re: [python-committers] [RELEASED] Python 3.5.0a4 is now available

On 22 April 2015 at 13:45, Paul Moore  wrote:
> On 21 April 2015 at 23:05, Steve Dower  wrote:
>> I made it a self-extracting RAR file so it could be signed, but I've already 
>> had multiple people query it so the next release will probably just be a 
>> plain ZIP file. I just need to figure out some reliable way of validating 
>> the download other than GPG, since I'd like installers to be able to do the 
>> download transparently and ideally without hard-coding hash values. I might 
>> add a CSV of SHA hashes to the zip too.
>
> You could probably just leave it as is (or make it a self-extracting
> zip file) and just describe it on the web page as "Windows amd64
> embeddable self-extracting archive". People are (I think) pretty used
> to the idea that they can open a self-extracting archive in tools like
> 7-zip, so those who didn't want to run the exe could do that (and
> would know they could). Obviously extracting that way you don't get
> the signature check, but that's to be expected.

Whoops, no - I changed my mind. If you double click on the downloaded
file (which I just did) it unpacks it into the directory you
downloaded the exe to, with no option to put it anywhere else, and no
UI telling you what it's doing. That's going to annoy people badly.
Better make it a simple zipfile in that case.

Paul (off to tidy up his download directory :-()
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 492 quibble and request

[Steve Dower]

The high-level answer to this is, like yield, the function temporarily returns 
all the way up the stack to the caller who intends to advance the 
iterator/async function. This is typically the event loop, and the main 
confusion here comes when the loop is implicit.

If you explicitly define an event loop it's roughly equivalent to the for loop 
that is calling next on a generator. For pip, I expect that's what you'd have - 
a blocking function ("do_work()"?, "execute_plan()"?) that creates a loop and 
starts it's tasks running within it. Each task inside that call with be 
asynchronous with respect to each other (think about passing generators to 
zip() ) but the loop will block the rest of your code until they're all 
finished.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Moore
Sent: ‎4/‎30/‎2015 2:07
To: Greg Ewing
Cc: Python Dev
Subject: Re: [Python-Dev] PEP 492 quibble and request

On 30 April 2015 at 09:58, Greg Ewing  wrote:
> Ethan Furman wrote:
>>
>> Having gone through the PEP again, I am still no closer to understanding
>> what happens here:
>>
>>   data = await reader.read(8192)
>>
>> What does the flow of control look like at the interpreter level?
>
>
> Are you sure you *really* want to know? For the sake
> of sanity, I recommend ignoring the actual control
> flow and pretending that it's just like
>
>data = reader.read(8192)
>
> with the reader.read() method somehow able to be
> magically suspended.

Well, if I don't know, I get confused as to where I invoke the event
loop, how my non-async code runs alongside this etc.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 554 v3 (new interpreters module)

[Steve Dower]

On 03Oct2017 0755, Antoine Pitrou wrote:

On Tue, 3 Oct 2017 08:36:55 -0600
Eric Snow  wrote:

On Tue, Oct 3, 2017 at 5:00 AM, Antoine Pitrou  wrote:

On Mon, 2 Oct 2017 22:15:01 -0400
Eric Snow  wrote:


I'm still not convinced that sharing synchronization primitives is
important enough to be worth including it in the PEP.  It can be added
later, or via an extension module in the meantime.  To that end, I'll
add a mechanism to the PEP for third-party types to indicate that they
can be passed through channels.  Something like
"obj.__channel_support__ = True".


How would that work?  If it's simply a matter of flipping a bit, why
don't we do it for all objects?


The type would also have to be safe to share between interpreters. :)


But what does it mean to be safe to share, while the exact degree
and nature of the isolation between interpreters (and also their
concurrent execution) is unspecified?

I think we need a sharing protocol, not just a flag.


The easiest such protocol is essentially:

* an object can represent itself as bytes (e.g. generate a bytes object 
representing some global token, such as a kernel handle or memory address)

* those bytes are sent over the standard channel
* the object can instantiate itself from those bytes (e.g. wrap the 
existing handle, create a memoryview over the same block of memory, etc.)
* cross-interpreter refcounting is either ignored (because the kernel is 
refcounting the resource) or manual (by including more shared info in 
the token)


Since this is trivial to implement over the basic bytes channel, and 
doesn't even require a standard protocol except for convenience, Eric 
decided to avoid blocking the core functionality on this. I'm inclined 
to agree - get the basic functionality supported and let people build on 
it before we try to lock down something we don't fully understand yet.


About the only thing that seems to be worth doing up-front is some sort 
of pending-call callback mechanism between interpreters, but even that 
doesn't need to block the core functionality (you can do it trivially 
with threads and another channel right now, and there's always room to 
make something more efficient later).


There are plenty of smart people out there who can and will figure out 
the best way to design this. By giving them the tools and the ability to 
design something awesome, we're more likely to get something awesome 
than by committing to a complete design now. Right now, they're all 
blocked on the fact that subinterpreters are incredibly hard to start 
running, let alone experiment with. Eric's PEP will fix that part and 
enable others to take it from building blocks to powerful libraries.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Steve Dower]

On 03Nov2017 0915, Victor Stinner wrote:

Hi,

2017-11-03 15:36 GMT+01:00 Guido van Rossum :

Maybe we should remove typing from the stdlib?
https://github.com/python/typing/issues/495


I'm strongly in favor on such move.


I'm torn.

If not having typing installed means you can't use "Any" or "Optional" 
in an annotation, that basically kills the whole thing. Some primitives 
need to be there.


If annotations become glorified strings (which IMHO they should) and 
typing gains a function to parse those into type hints (which IMHO it 
should), then I'm in favour of splitting typing out. (Personally, if 
making every type hint a literal 'string' meant that I could avoid 
importing typing then I'd do it.)


However, if typing is split out then its API (specifically, the contents 
of typing.__all__ and their semantic meaning (e.g. "Iterable" means 
something with an "__iter__" method)) needs to stay in PEPs.


Static analysers using type hints encode much more information about 
these types than can be inferred statically from typing.py, which means 
the definitions should not change faster than Python x.*y*. Ideally, 
they would not change at all once released.


For example, my static analyser has an existing object representing 
iterables, since we've been inferring iterables for years. When I parse 
a type annotation and see "typing.Iterable", I'm going to just 
substitute my own implementation - the definition in typing.py isn't 
going to be used (or be useful). And because it has to map across 
languages, it has to be a hard-coded mapping that can't rely on 
typing.py at all.


Since the contents of typing.py will likely be completely ignored by my 
analyser, which means that I can't treat "whatever version of typing is 
installed" as ground truth. It needs to move slower or be purely 
additive. Being in the standard library is a nice easy way to ensure 
this - moving it out is a risk.


That said, because I don't care about the contents of the file, all the 
heavy execution stuff can totally be moved out. If typing in the stdlib 
became very trivial definitions or just a set of names to support 
searching/editors/forward-refs, and typing out of the stdlib had the 
ability to convert annotations into an object model that provides rich 
runtime introspection, I'd also be fine. At least then the interface is 
highly stable, even if the implementation (for those who use it) changes.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Remove typing from the stdlib

[Steve Dower]

On 03Nov2017 1124, Guido van Rossum wrote:

A side note (I'm reading all responses but staying out of the discussion):

No static checker should depend on the *contents* of typing.py, since 
it's just a bunch of runtime gymnastics to allow types to be evaluated 
at runtime without errors, with a secondary goal of making them 
introspectable (some folks don't even agree with the latter, e.g. Mark 
Shannon).


Static analyzers should be able to make strong *assumptions* about what 
things defined there mean -- in mypy such assumptions are all over the 
place, based on the full name of things -- it never reads typing.py. (It 
reads typing.pyi from typeshed, but what's there is ignored in many 
cases too.)


Thank you. Very glad to hear I understood it correctly.

Cheers,
Steve


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

[Steve Dower]

Since there is voting going on, -1 on changing the guarantees of `dict`. If 
ordering is important, OrderedDict is more explicit and more obvious to the 
over reading the code, even if the underlying implementation is identical.

Good luck teaching people about why Python went from OrderedDict to 
UnorderedDict within a version. I remember learning about this same thing in 
C++ and just thinking “wut?”. And they just added a new type and said to stop 
using the old one – not changing something that people already understand.

Better to underspecify the default dict and offer explicitly named extensions 
(DefaultDict, OrderedDict, etc.) for those who want more guarantees.

Cheers,
Steve

Top-posted from my Windows phone

From: Peter Ludemann via Python-Dev
Sent: Sunday, November 5, 2017 12:53
To: Sven R. Kunze
Cc: Python-Dev
Subject: Re: [Python-Dev] Guarantee ordered dict literals in v3.7?

Isn't ordered dict also useful for **kwargs? 

If it turns out that there's a dict implementation that's faster by not 
preserving order, collections.UnorderedDict could be added.
There could also be specialized implementations that pre-size the dict (cf: C++ 
unordered_map::reserve), etc., etc.
But these are all future things, which might not be necessary.

On 5 November 2017 at 12:44, Sven R. Kunze  wrote:
+1 from me too.

On 04.11.2017 21:55, Jim Baker wrote:
+1, as Guido correctly recalls, this language guarantee will work well with 
Jython when we get to the point of implementing 3.7+. 

On Sat, Nov 4, 2017 at 12:35 PM, Guido van Rossum  wrote:
This sounds reasonable -- I think when we introduced this in 3.6 we were 
worried that other implementations (e.g. Jython) would have a problem with 
this, but AFAIK they've reported back that they can do this just fine. So let's 
just document this as a language guarantee.

On Sat, Nov 4, 2017 at 10:30 AM, Stefan Krah  wrote:

Hello,

would it be possible to guarantee that dict literals are ordered in v3.7?


The issue is well-known and the workarounds are tedious, example:

   https://mail.python.org/pipermail/python-ideas/2015-December/037423.html


If the feature is guaranteed now, people can rely on it around v3.9.



Stefan Krah



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/jbaker%40zyasoft.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/pludemann%40google.com


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Partial support of a platform

[Steve Dower]

On 06Nov2017 0941, Victor Stinner wrote:

[SNIP]

But the question here is more about "partial" support.

While changes are usually short, I dislike applying them to Python 2.7
and/or Python 3.6, until a platform is fully support. I prefer to
first see a platform fully supported to see how much changes are
required and to make sure that we get someone involved to maintain the
code (handle new issues).

Example of platforms: MinGW, Cygwin, OpenBSD, NetBSD, xWorks RTOS, etc.


I appreciate the desire for changes to be made upstream, especially on 
code that changes frequently enough to make it difficult to patch 
reliably (this is basically the entire premise behind my PEP 551). At 
the same time, I don't like carrying the burden of code for platforms we 
do not support (hence PEP 551 doesn't really add any interesting code). 
There is a balance to be found here, though.


I don't believe CPython *partially* supports any platforms - either they 
are fully supported or they are not supported.


However, we can and should do things that help other people support 
their platforms, such as making sure build options can be specified by 
environment variables. At the same time, we can and should *exclude* 
things that require platform-specific testing in core (for example, 
predefined options for building for a specific platform).


Similarly, if someone wanted to add specific platform support to a 
stdlib module via "if sys.platform ...", I'd be hesitant. However, if 
they refactored it such that it was easier *for a custom build of 
CPython* to provide/omit certain features (e.g. 
https://github.com/python/cpython/blob/30f4fa456ef626ad7a92759f492ec7a268f7af4e/Lib/threading.py#L1290-L1296 
) then I'd be more inclined to accept it - but only if there was no 
behavioural change on supported platforms.


Does that make sense? I'm not sure whether we ought to capture some 
general guidelines somewhere on how to make decisions around this, but 
we should certainly have the discussion here first anyway.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python possible vulnerabilities in concurrency

[Steve Dower]

On 15Nov2017 2053, Guido van Rossum wrote:
On Wed, Nov 15, 2017 at 6:50 PM, Guido van Rossum > wrote:


So far I learned one thing from the report. They use the term 
"vulnerabilities" liberally, defining it essentially as "bug":


All programming languages contain constructs that are incompletely
specified, exhibit undefined behaviour, are
implementation-dependent, or are difficult to use correctly. The use
of those constructs may therefore give rise to /vulnerabilities/, as
a result of which, software programs can execute differently than
intended by the writer.


They then go on to explain that sometimes vulnerabilities can be 
exploited, but I object to calling all bugs vulnerabilities -- that's 
just using a scary word to get attention for a sleep-inducing document 
containing such gems as "Use floating-point arithmetic only when 
absolutely needed" (page 230).


I looked at this report the first time it was posted and came to the 
same conclusion.


It's only valuable in the sense that it makes clear just how perfect 
your code has to be to avoid being vulnerable, and since that level of 
perfection can never be achieved, the takeaway is that you can't achieve 
security solely within the application/framework/runtime. It is 
convenient to have formally researched and collated it, so the rest of 
us can just write blog posts/PEPs stating it as fact, but I think most 
people will intuitively get the main point without referring to the report.


(Yes, I'm still interested in pushing PEP 551 forward :) I've been 
trying to get some actual companies other than Microsoft using it for 
the real-world experience, and I have a couple of conference talks 
coming up about it. There are implementations against v3.7.0a2 is at 
https://github.com/zooba/cpython/tree/pep551 and against v3.6.3 at 
https://github.com/zooba/cpython/tree/pep551_36 )


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python initialization and embedded Python

[Steve Dower]

On 17Nov2017 1601, Victor Stinner wrote:

In short, it means that using the "Python runtime" before it's
initialized by _PyRuntime_Initialize() is now likely to crash. For
example, calling PyMem_RawMalloc(), before calling
_PyRuntime_Initialize(), now calls the function NULL: dereference a
NULL pointer, and so immediately crash with a segmentation fault.

I'm writing this email to ask if this change is an issue or not to
embedded Python and the Python C API. Is it still possible to call
"all" functions of the C API before calling Py_Initialize()?


I thought it was never possible to call most of the C API without 
initializing, except for certain APIs that are documented as being safe. 
I've certainly crashed many times calling C APIs before initialization. 
My intuition was that the only safe ones before were those that were 
used to initialize the runtime (Py_SetPath and such), which are also the 
ones being "upgraded" as part of this work.


If we have a good idea of which ones are [un]safe now, perhaps we should 
tag them explicitly in the docs? Do we know which ones are [un]safe?


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com