Re: [Python-Dev] csv changed from python 2.4 to 2.5
Nick Craig-Wood wrote: > Christian K <[EMAIL PROTECTED]> wrote: [...] >> Python 2.5.1 (r251:54863, May 2 2007, 16:56:35) >> [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >>>>> import csv >>>>> d = csv.excel >>>>> d.delimiter = ',' >>>>> > > Don't you want to do this anyway? > > import csv > class my_dialect(csv.excel): > delimeter = "," > I could probably do that, sure. I used to register my custom dialects and retrieve and modify them at another place, thus probably misusing the register mechanism as a replacement for a global symbol. Thanks, Christian ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 30.08.2014 17:22, Alex Gaynor wrote: > The Windows certificate store is used by ``load_default_certs``: > > * https://github.com/python/cpython/blob/master/Lib/ssl.py#L379-L381 > * https://docs.python.org/3.4/library/ssl.html#ssl.enum_certificates The Windows part of load_default_certs() has one major flaw: it can only load certificates that are already in Windows's cert store. However Windows comes only with a small set of default certs and downloads more certs on demand. In order to trigger a download Python or OpenSSL would have to use the Windows API to verify root certificates. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 30.08.2014 00:22, Antoine Pitrou wrote: > SSL_CERT_DIR and SSL_CERT_FILE are used, if set, when > SSLContext.load_verify_locations() is called. > > Actually, come to think of it, this allows us to write a better > test for that method. Patch welcome! The environment vars are used only when SSLContext.set_default_verify_paths() is called. load_verify_locations() loads certificates from a given file, directory or memory but it doesn't look at the env vars. create_default_context() calls SSLContext.load_default_certs() when neither cafile, capath nor cadata is given as an argument. SSLContext.load_default_certs() then calls SSLContext.set_default_verify_paths(). However there is a catch: SSLContext.set_default_verify_paths() is not called on Windows. In retrospective it was a bad decision by me to omit the call. http://hg.python.org/cpython/file/164a17eca081/Lib/ssl.py#l376 Christian PS: SSL_CERT_DIR and SSL_CERT_FILE are the default names. It's possible to change the names in OpenSSL. ssl.get_default_verify_paths() returns the names and paths to the default verify locations. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 16:16, R. David Murray wrote: > Self -signed certificates are not crazy in an internal corporate > environment even when properly playing the defense in depth game. Once > you've acked the cert the first time, you will be warned if it changes > (like an ssh host key). Sure, as Nick says the corp could set up an > internal signing authority and make sure everyone has their CA...and > they *should*...but realistically, that is probably relatively rare at > the moment, because it is not particularly easy to accomplish > (distributing the CA everywhere it needs to go is still a Hard Problem, > though it has gotten a lot better). It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. That's all. A self-signed certificate acts as its own root CA (so to speak). But there is a downside, too. The certificate is trusted for any and all connections. Python's SSL module has no way to trust a specific certificate for a host. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 08:24, Nick Coghlan wrote: > To answer David's specific question, the existing knobs at the OpenSSL > level (SSL_CERT_DIR and SSL_CERT_FILE ) let people add an internal CA, > opt out of the default CA system, and trust *specific* self-signed > certs. This works only on Unix platforms iff SSL_CERT_DIR and SSL_CERT_FILE are both set to a non-empty string that points to non-existing files or something like /dev/null. On Windows my enhancement will always cause the system trust store to kick in. There is currently no way to disable the Windows system store for ssl.create_default_context() and ssl._create_stdlib_context() with the functions' default arguments. On Mac OS X the situation is even more obscure. Apple's OpenSSL binaries are using Apple's Trust Evaluation Agent. You have to set OPENSSL_X509_TEA_DISABLE=1 in order to prevent the agent from adding trusted certs from OSX key chain. Hynek Schlawack did a deep dive into it. https://hynek.me/articles/apple-openssl-verification-surprises/ > A Python-specific user level cert store is something that could be > developed as a PyPI library for Python 2.7.9+ and 3.4+ (Is cert > management considered in scope for cryptography.io? If so, that could > be a good home). Python's SSL module is lacking some functionalities in order to implement a fully functional cert store. * no verify hook to verify each certificate in the chain like https://www.openssl.org/docs/ssl/SSL_CTX_set_cert_verify_callback.html http://linux.die.net/man/3/x509_store_ctx_set_verify_cb /api/ssl.html#OpenSSL.SSL.Context.set_verify * no way to get the full cert chain including the root certificate. * no API to get the subject public key information (SPKI). The SPKI hash can be used to identify a certificate. For example it's used in Google's CRLSet. http://dev.chromium.org/Home/chromium-security/crlsets * the cert validation exception could use some additional information. There are probably some more things mising. An X509 object would help, too. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 19:29, Antoine Pitrou wrote: > You certainly shouldn't do so. If an application has special needs that > require trusting a self-signed certificate, then it should expose a > configuration setting to let users specify the cert's location. Stuffing > self-signed certs into the system trust store is really a measure of > last resort. Correct! I merely wanted to state that OpenSSL can verify a self-signed certificate easily. The certificate 'just' have to be added to the SSLContext's store of trusted root certs. Somebody has to figure out how Python can accomplish the task. > There's another case which isn't solved by this, though, which is when a > cert is invalid. The common situation being that it has expired > (renewing certs is a PITA and therefore expired certs are more common > than it sounds they should be). In this case, there is no way to > whitelist it: you have to disable certificate checking altogether. This > can be exposed by the application as configuration option if necessary, > as well. It's possible to ignore errors with a verify callback. OpenSSL's wiki has an example for the expired certs http://wiki.openssl.org/index.php/Manual:X509_STORE_CTX_set_verify_cb%283%29#EXAMPLES Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 08:09, Nick Coghlan wrote: > As Antoine says here, I'm also opposed to adding more Python specific > configuration options. However, I think there may be something > worthwhile we can do that's closer to the way browsers work, and has > the significant benefit of being implementable as a PyPI module first > (more on that in a separate reply). I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand. There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc. Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 31.08.2014 22:30, Paul Moore wrote: > On 31 August 2014 21:15, Antoine Pitrou wrote: >> What do you call your local cert store? > > I was referring to Christian's comment >> It's very simple to trust a self-signed certificate: just download it and >> stuff it into the trust store. I was referring to the the trust store of the SSLContext object and not to any kind of cert store of the operating system. Sorry for the confusion. > a) Is there really no OS-level personal trust store? I'm thinking of > Windows here for my own personal use, but the same question applies > elsewhere. Windows and OSX have superior cert stores compared to Linux and BSD. They have means for user and system wide cert stores and trust settings Linux just have one central directory or file with all trusted certs. My KDE has some options to disable certs but I don't know how to make use of the configuration. Even worse: Linux distros doesn't make a different between purposes. On Windows a user can trust a certificate for S/MIME but not for server auth or client auth. Ubuntu just puts all certification in one directory but it's wrong. :( https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1207004 Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 01.09.2014 08:44, Nick Coghlan wrote: > Yes, it would have exactly the same security failure modes as > sitecustomize, except it would only fire if the application > imported the ssl module. > > The "-S" and "-I" switches would need to disable the implied > "sslcustomize", just as they disable "import site". A malicious package can already play havoc with your installation with a custom ssl module. If somebody is able to sneak in a ssl.py then you are screwed anyway. sslcustomize is not going to make the situation worse. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 01.09.2014 17:35, Nick Coghlan wrote: > Oh, now I get what you mean - yes, sitecustomize already poses the same > kind of problem as the proposed sslcustomize (hence the existence of the > related command line options). If an attacker is able to place a module like sitecustomize.py in an import directory or any .pth file in a site-packages directory than this Python installation is compromised. .pth files are insidious because they are always loaded and their code is always executed. I don't see how sslcustomize is going to make a difference here. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 02.09.2014 23:32, Antoine Pitrou wrote:
>> Furthermore, "disable verification" is a nonsensical thing to do with TLS.
>
> It's not. For example, if you have an expired cert, all you can do
> AFAIK is to disable verification.
It's possible to ignore or just warn about expired certs with simple
verify callback. The callback looks like this:
int verify_callback(int ok, X509_STORE_CTX *ctx)
{
if (X509_STORE_CTX_get_error(ctx) == X509_V_ERR_CERT_HAS_EXPIRED)
return 1;
return ok;
}
It's installed like this:
PySSLContext *self;
X509_STORE *store = SSL_CTX_get_cert_store(self->ctx);
X509_STORE_set_verify_cb(store, verify_callback);
The X509_STORE_CTX struct is created when a certificate chain is
verified. It holds all sorts of states like chain, leaf cert, current
cert that is tested, validation depth, error flags and more. In order to
write useful verify callbacks me or somebody else has to write a
X509_STORE_CTX type and X509 cert type. It's something I want to do for
more than a year but I don't find any spare time. :(
Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Sad status of Python 3.x buildbots
Hi all I am using buildbot now for some time and i would be willing to contribute on that. I had small work on openstack buildbot slave but had not the proper infrastructure to get more value out of it. I like that project and automation. Anyway, if i could be of help let me know (; Short to my person: i am christian staffa and actually living in germany. Chris Sent from my Sony Xperia™ smartphone Nick Coghlan wrote >___ >Python-Dev mailing list >Python-Dev@python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >https://mail.python.org/mailman/options/python-dev/christian.staffa%40gmx.de ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 03.09.2014 19:29, Ethan Furman wrote: > Excellent. Last question (I hope): it is possible to (easily) create an > SSLContext that will verify against a self-signed certificate? Yes: context = ssl.create_default_context(cafile="/path/to/selfsigned.pem") That works iff the certificate is valid, not expired and its CN or SAN matches the hostname of the service. When the hostname doesn't match then you have to set context.check_hostname = False Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 03.09.2014 19:54, Guido van Rossum wrote: > Let's take the plunge on this issue for the next 2.7 release (3.5 being > a done deal). Yes, some people will find that they have an old script > accessing an old service which breaks. Surely some of the other changes > in the same 2.7 bugfix release will also break some other scripts. > People deal with it. Probably 90% of the time it's an annoyance (but no > worse than any other minor-release upgrade -- you should test upgrades > before committing to them, and if all else fails, roll it back). But at > least some of the time it will be a wake-up call and an expired > certificate will be replaced, resulting in more security for all. I'm +1 for Python 3.5 but -1 for Python 2.7. The SSLContext backport will landed in Python 2.7.9 (to be released). No Python 2 user is familiar with the feature yet. But more importantly: None of the stdlib modules support the new feature, too. httplib, imaplib ... they all don't take a SSLContext object as an argument. PEP-466 does not include the backport for the network modules. Without the context argument there is simply no clean way to configure the SSL handshake properly. The default settings must stay until we decide to backport the context argument and have a way to configure the default behavior. Nick and me are planing a PEP. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 03.09.2014 21:37, Victor Stinner wrote: > Thanks, you replied before I asked the question :-) (If > certificates are validated by default, how do you disable the > checks?) > > Sorry, I didn't follow the whole discussion and Python 2.7 changes > related to security. Does Python 2.7 support loading > (automatically) system certificate authorities? Like the Windows > certificate store: > https://docs.python.org/dev/whatsnew/3.4.html#whatsnew34-win-cert-store Yes! Alex and others have ported all SSL features back to Python 2.7. The SSL module in Python 2.7.9 will have the same feature set as Python 3.4. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 476: Enabling certificate validation by default!
On 03.09.2014 21:37, Guido van Rossum wrote: > OK, that changes my position for 2.7 (but not for 3.5). I had > assumed there was a way to disable the cert check by changing one > parameter to the urlopen() call. (And I had wanted to add that > there should be a clear FAQ about the subject.) If this isn't > possible that changes the situation. (But I still think that once > we do have that simple change option we should do it, in a later > 2.7 upgrade.) > > I apologize for speaking before I had read all facts, and I'll > await what you and Nick come up with. You are welcome! :) I like to see cert checks in Python 2.7, too. Eventually Python 2.7 should have them enabled by default or at least have one very simple way to enable them globally. Right now we aren't there yet. Perhaps Python 2.7.10 or 2.7.11 will have the necessary bits and backports. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backwards compatibility after certificate autovalidation
On 09.09.2014 05:03, Nick Coghlan wrote: > > On 9 Sep 2014 10:48, "Jim J. Jewett" <mailto:jimjjew...@gmail.com>> wrote: >> I assume that adding _unverified_urlopen or urlopen(context=...) do >> provide incremental improvements compatible with the eventual full >> opt-in. If so, adding them is probably reasonable, but I think the >> PEP should explicitly list all such approved half-measures as a guard >> against API feature creep. > > From Guido's and your feedback, I think we may need two things to > approve this for 3.4.2 (putting 2.7 aside for now): > > 1. "context" parameter support in urllib.request (to opt out on a > per-call basis) > 2. a documented way to restore the old behaviour via sitecustomize > (which may involve monkeypatching) What's with our plan to introduce sslcustomize? Is the idea for a configuration module and named contexts off the table? For reference: I came up with the idea to introduce the module "sslcustomize" similar to sitecustomize. Contrary to sitecustomize the module is imported at the end of the ssl module. Based on my idea Nick proposed another addition to the SSL module. He proposed a ssl.named_contexts mapping from module names to factory functions that create SSLContext objects. http://permalink.gmane.org/gmane.comp.python.devel/149292 I still prefer the general idea over the monkey patching idea because it provides a clean but simple interface for structured configuration. Monkey patching of stdlib modules is ugly and error-prone. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP476: Enabling certificate validation by default
On 19.09.2014 18:53, Alex Gaynor wrote: > Hi all, > > I've just updated the PEP to reflect the API suggestions from Nick, and the > fact that the necessary changes to urllib were landed. > > I think this is ready for pronouncement, Guido? There is still the issue with SSL_CERT_DIR and SSL_CERT_FILE on Windows and Apple's OpenSSL builds on OSX. I've opened a bug report http://bugs.python.org/issue22449 tl;dr On Windows SSL_CERT_DIR and SSL_CERT_FILE are simply ignored by SSLContext.load_verify_locations. On OSX Apple's Trust Evaluation Agent adds certs behind the scene. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP476: Enabling certificate validation by default
On 21.09.2014 01:03, Nick Coghlan wrote:
> We may also need some clarification from Ned regarding the status of
> OpenSSL and the potential impact switching from dynamic linking to
> static linking of OpenSSL may have in terms of the
> "OPENSSL_X509_TEA_DISABLE" setting.
You may want to ask Hynek, too. He initially discovered the issue and
made me aware how Apple is tying keychain into OpenSSL. You may find the
code in:
http://opensource.apple.com/source/OpenSSL098/OpenSSL098-35.1/src/crypto/x509/x509_vfy_apple.h
http://opensource.apple.com/source/OpenSSL098/OpenSSL098-35.1/src/crypto/x509/x509_vfy_apple.c
The TEA (Trust Evaluation Agent) switch is global and affects all SSL
context instances. There is non canonical way to set the TEA flag for a
single context or socket.
/* -1: not set
* 0: set to false
* 1: set to true
*/
static tea_enabled = -1;
void
X509_TEA_set_state(int change)
{
tea_enabled = (change) ? 1 : 0;
}
int
X509_TEA_is_enabled()
{
if (tea_enabled < 0)
tea_enabled = (NULL == getenv(X509_TEA_ENV_DISABLE));
return tea_enabled != 0;
}
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Microsoft Visual C++ Compiler for Python 2.7
On 26.09.2014 20:01, Steve Dower wrote: > Hi all, > > (This is advance notice since people on this list will be interested. > Official announcements are coming when setuptools makes their next release.) > > Microsoft has released a compiler package targeting Python 2.7 (i.e. VC9). > We've produced this package to help library developers build wheels for > Windows, but also to help users unblock themselves when they need to build C > extensions themselves. > > The Microsoft Visual C++ Compiler for Python 2.7 is available from: > http://aka.ms/vcpython27 Awesome! :) Thanks a lot, Steve! Is it possible to compile extensions from Python's numerical stack such as NumPy, SciPy and SciKit, too? ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bytes-like objects
On 06/10/14 10:33, Georg Brandl wrote: > bytes-like object Howdy, two small comments: 1) just as a quick check, I did a Python search for exactly that phrase https://docs.python.org/3/search.html?q=bytes-like+object&check_keywords=yes&area=default with zero results. Maybe it would be a good thing if glossary entries were always found via the search page. 2) And about this glossary entry: "An object that supports the Buffer Protocol" - can I take that for granted, as a real definition, meaning """an object is bytes-like iff it supports the buffer protocol"""? cheers - Chris -- Christian Tismer :^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://www.pydica.net/ 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing 2.7.x
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 06.10.14 20:55, Zachary Ware wrote: > On Mon, Oct 6, 2014 at 12:24 PM, Ned Deily wrote: >> 3. security: "fixing issues exploitable by attackers such as crashes, >> privilege escalation and, optionally, other issues such as denial of >> service attacks. Any other changes are not considered a security risk >> and thus not backported to a security branch." >>= 3.2.x and 3.3.x > > 3.1 is still in this category, is it not? According to PEP 375, it's > a few months past due for its last release. > > http://legacy.python.org/dev/peps/pep-0375/#maintenance-releases > I don't think that the rules should be implicitly considered compatible between the 2.X and 3.X series. Python 2.X has a history that extends to X==6, X==5 and even X==4, as a really conservative POV with an extent over more than 10 years. I believe, such a thing does not exist for the Python 3.X series at all. My impression is that no 3.X user ever would want to stick with any older version. Is that true, or am I totally wrong? cheers -- Chris - -- Christian Tismer :^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://www.pydica.net/ 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUMur7AAoJEOcwEVD7e+4OW0UIAJk2ZTFX3OjBrt1G0RZc9nPb hHVxGJNXNeBellM9BpmoW9t9hgk94lAIgmh5hop5uMt32o9CH47s97rKw7K1ekl5 sML/4hl5/BLRiHXgwSB1ZltqZrvG/xsE6AE1v37BcPf/X3T4UfPhW30z+43eaBJw Q3b21EwxxUJGJ//GWwi2+buCfkfRuePBIB4MQiMm3/JI9h03EPbRoQ0/53huKLeW I7oAemVzprQHw7coaTf6EOHFTlmUfHvm5K9ywpabX10/Ediz1suJfPMPdzUigHG3 G3ABMKAA1YockpfIDU/7rpmDcliblpjU5MT4BsZycuYXOyUesV6uDaLMOdO5fEY= =ZuoT -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] shebang policy, and pip
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Howdy, this question is a bit about general policy which is not yet covered in the python recommendations: I see projects which do check-ins like "get rid of shebang lines" and they remove those lines from non-script sources. It is not always clear to me what to do, so I tend to leave those lines in per default, in order not to waste time thinking about it, but well, today I was confronted with that. Digging a bit deeper shows the following: python docs: No mention of shebang, but for Windows. https://docs.python.org/3/search.html?q=shebang&check_keywords=yes&area=default https://docs.python.org/3/using/windows.html?highlight=shebang Google's python style guide also says when a shebang is needed, but does not forbid it. Pep 394 explains how to use shebang, but still nothing about not using it. http://legacy.python.org/dev/peps/pep-0394/ So is there anything officially preferred, and should that go into pep 8? Special case with pip - -- I was looking through my installed packages and wondered quite much about pip: Pip has a shebang in the __init__ file, but no shebang in the __main__ file. I guess this is wrong and should be in the executable file, which is __main__ . cheers - Chris - -- Christian Tismer :^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://www.pydica.net/ 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUNQ8FAAoJEOcwEVD7e+4Of5MH/13YyXRzPVXfsLDbfe/CRkFF ORVAPkRB90OaEmbLTvBXPhFlAgEwcQnpdO+tmqigvlORd0cSXQKIxoPOiqq601gs XV56aREqBCT26XMaKTuoPdu4DaW+TkwyWSn70eq4U/P7YjF3ZlNt8IkA5mteM7an ycRYMnknEaIvP/xpZdGp+v4pq5LA42LWAY1awnBk4eMP04uDowSmcuLELpZrmSCr iMkw6wPUdZxGVtQNwSses0mh3DuaQuwrubhHMnLoOKn/lqRjckG2Ii2BIHlWy9lQ 5X3y8IdWPh7awio8xaibNqsWaP+0DT97g2H7QZf8YIG7/DkHL/iacSr7NAPBXXQ= =IODc -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] shebang policy, and pip
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 08.10.14 14:20, Donald Stufft wrote: > >> On Oct 8, 2014, at 6:16 AM, Christian Tismer wrote: >> >> >> ... >> >> So is there anything officially preferred, and should that go into pep 8? > > Some editors can use shebang lines to control syntax highlighting or linting > (mine for example will lint different for python2 vs python3 shebangs). > Interesting use case, of course! At first sight, at least ;-) But it becomes relatively awkward when virtualenv is used: There is no textual distinction between python 2 and 3, and the text editor would have to do more analysis to get things right than to guess from the shebang line. Which then makes it useless, and the editor would need to grab the real python interpreter to find things out. cheers - Chris - -- Christian Tismer :^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://www.pydica.net/ 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUNZV1AAoJEOcwEVD7e+4OlpcIAN3556PP4sBlFeDDPn4bq3fO ScXRhB8tPkS3Ftgyux2swE91yxOZFSV8Zyae04wTB+GlzKZ6v9aXMKn5mt+b6sIr aq1vfPxG7Qoz3q7zhaAfps8ErD9f1PriC7/4F2P4FOOko07eHt6/e8Sxg4qAgMPz nADLj8/2MtvQYFiw3m4Zsqs31wjCTTICH52FnRgla9+u5IhStdQE7OFTiHZaPNK3 tsUohUoKqhMPIhwuB669JE7rwcRB9dA5Iatiy3uU0wqCYkekT3I4DwCtAS+DZkJX Fe0wpLGGHOprwUTQu0SMFGQRxPX3HxL0RzTNLKjCcDNLDWRcwZRPOZ3K4DVoggQ= =i5dg -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] libffi embedded in CPython
On 19.12.2014 10:52, Paul Moore wrote: > Probably the easiest way of moving this forward would be for someone > to identify the CPython-specific patches in the current version, and > check if they are addressed in the latest libffi version. They haven't > been applied as they are, I gather, but maybe equivalent fixes have > been made. I've no idea how easy that would be (presumably not > trivial, or someone would already have done it). If the patches aren't > needed any more, upgrading becomes a lot more plausible. That's easy. All patches are tracked in the diff file https://hg.python.org/cpython/file/3de678cd184d/Modules/_ctypes/libffi.diff . The file *should* be up to date. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Starting CPython development w/ Docker
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 2015-04-20 15:52, Saul Shanabrook wrote: > I started trying some CPythong development a week ago at PyCon and > first got testing working using Docker on my mac. This had the > advantage of not having to worry about installing and dependencies, > and also let me test on different Python versions easily. > > If you are interested in trying it, I laid out all the steps here: > http://www.saulshanabrook.com/cpython-dev-w-docker/ Good work! May I suggest that you add ccache and the --config-cache option to configure? Both speed up the build process a lot. For ccache you have to install the package and put /usr/lib64/ccache in front of PATH. Christian -BEGIN PGP SIGNATURE- iQEcBAEBCgAGBQJVNRgSAAoJEIZoUkkhLbaJZF0IAJQHcVJvxCnWXjtnsgW7y4Rc sTccSCvBU6Qlwghnb8jn5+fMLGpgBpT1JQqvX8jdzXSkRyvFjPROdGTFom6qRm4F vrF2XAfA5lH2nwvtx1S8WsG6NaStGn/H1tfwZRujE5bxViuPpd3UXfYPJogSKWAZ OadnpnjdhS24EJ5GMnJe61KPZphBlNyTmSB60SXxu/Sk3AIVnM+RPKshvbsla1vg 3udBRgjP2LLSO6mzwIjfL8+blnpExkUTLiqHqpw9QX3i4ox4xkIozBj56KinfVvC lXHtMuK+dv34A43BffAX1ipEVd+HpCBYwhQEitievjidLXkoLcpAwsl/bSY7ToI= =JeXE -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fwd: Coverity Scan update
On 2015-05-12 17:28, Guido van Rossum wrote: > -- Forwarded message -- > From: "Dakshesh Vyas" mailto:dv...@coverity.com>> > Date: May 12, 2015 1:08 AM > Subject: Coverity Scan update > To: "gu...@python.org <mailto:gu...@python.org>" <mailto:gu...@python.org>> > Cc: > > Hello Guido van Rossum, > > Thank you for using Coverity Scan. > With more than 4000 open source projects now registered at Coverity > Scan, we are committed to help the open source community find quality > and security issues early in development. > > We would like to inform you that you can now nominate your favorite > defect! Tell us which defects you are glad were found using Coverity > Scan and get special gifts like a Samsung Gear 2 Smartwatch, Code Black > Drone or Tile for iOS and Android. Gifts will be distributed every month > based on defect nomination! > > We recently added new hardware resource and upgraded our Scan server to > our latest 7.6 version, which includes great new features: Thanks Guido, I didn't get the mail although I'm a project admin. Dakshesh must have sent it out to the initial project owner. I'm planning to do a final round of fixes when the first beta is out. Currently CPython is down to about 5 issues. The new version might reveal additional problems or false-positives. Let's see if I can get it down to zero again. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 553: Built-in debug()
On 2017-09-07 09:50, Barry Warsaw wrote: > On Sep 6, 2017, at 23:10, Terry Reedy wrote: >> >> Environmental variables are set to strings, not objects. It is not clear >> how you intend to handle the conversion. > > The environment variable names a module import path. Without quibbling about > the details of the syntax (because honestly, I’m not convinced it’s a useful > feature), it would work roughly like: > > * The default value is equivalent to PYTHONBREAKPOINTHOOK=pdb.set_trace > * breakpoint() splits the value on the rightmost dot > * modules on the LHS are imported, then the RHS is getattr’d out of that > * That’s the callable breakpoint() calls Setuptools' entry points [1] use colon between import and function, e.g. "pdb:set_trace" would import pdb and then execute set_trace. The approach can be augmented to allow calling a class method, too. So "package.module:myclass.classfunc" would do : from package.module import myclass myclass.classfunc Regards, Christian [1] https://setuptools.readthedocs.io/en/latest/setuptools.html#automatic-script-creation ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] make multissltests
For your information, You can now automatically compile and check the ssl module with multiple versions of OpenSSL and LibreSSL. The multissltest script downloads tar.gz, compiles the source and installs headers + shared lib into a local directory. It takes rather long the first time because OpenSSL does not support parallel builds. Be prepared to wait an hour. After the script has downloaded and installed OpenSSL/LibreSSL, it recompiles the ssl and hashlib module with every version and runs the SSL-related test suite. make multissltests compiles and runs all tests make multisslcompile only compiles See ./python Tools/ssl/multissltests.py --help for additional options. For now the script covers: * OpenSSL 0.9.8zc * OpenSSL 0.9.8zh * OpenSSL 1.0.1u * OpenSSL 1.0.2 * OpenSSL 1.0.2l * OpenSSL 1.1.0f * LibreSSL 2.3.10 * LibreSSL 2.4.5 * LibreSSL 2.5.3 * LibreSSL 2.5.5 Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Investigating time for `import requests`
On 2017-10-02 04:04, INADA Naoki wrote: > *3. ssl* > > import time: 2007 | 2007 | ipaddress > import time: 2386 | 2386 | textwrap > import time: 2723 | 2723 | _ssl > ... > import time: 306 | 988 | base64 > import time: 2902 | 11004 | ssl > > I already created pull request about removing textwrap dependency from ssl. > https://github.com/python/cpython/pull/3849 Thanks for the patch. I left a comment on the PR. Please update your patch and give me a chance to review patches next time. > ipaddress and _ssl module are bit slow too. But I don't know we can > improve them or not. The _ssl extension module has to initialize OpenSSL. It is expected to take a while. For 3.7 I'll replace ssl.match_hostname with OpenSSL function. The ssl module will no longer depend on re and ipaddress module. > ssl itself took 2.9 ms. It's because ssl has six enums. Why are enums so slow? Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python startup optimization: script vs. service
Hello python-dev, it's great to see that so many developers are working on speeding up Python's startup. The improvements are going to make Python more suitable for command line scripts. However I'm worried that some approaches are going to make other use cases slower and less efficient. I'm talking about downsides of lazy initialization and deferred imports. For short running command line scripts, lazy initialization of regular expressions and deferred import of rarely used modules can greatly reduce startup time and reduce memory usage. For long running processes, deferring imports and initialization can be a huge performance problem. A typical server application should initialize as much as possible at startup and then signal its partners that it is ready to serve requests. A deferred import of a module is going to slow down the first request that happens to require the module. This is unacceptable for some applications, e.g. Raymond's example of speed trading. It's even worse for forking servers. A forking HTTP server handles each request in a forked child. Each child process has to compile a lazy regular expression or important a deferred module over and over. uWSGI's emperor / vassal mode us a pre-fork model with multiple server processes to efficiently share memory with copy-on-write semantics. Lazy imports will make the approach less efficient and slow down forking of new vassals. TL;DR please refrain from moving imports into functions or implementing lazy modes, until we have figured out how to satisfy requirements of both scripts and long running services. We probably need a PEP... Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python startup optimization: script vs. service
On 2017-10-02 14:05, George King wrote: > I’m new to this issue, but curious: could the long-running server > mitigate lazy loading problems simply by explicitly importing the > deferred modules, e.g. at the top of __main__.py? It would require some > performance tracing or other analysis to figure out what needed to be > imported, but this might be a very easy way to win back response times > for demanding applications. Conversely, small scripts currently have no > recourse. That approach could work, but I think that it is the wrong approach. I'd rather keep Python optimized for long-running processes and introduce a new mode / option to optimize for short-running scripts. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python startup optimization: script vs. service
On 2017-10-02 15:26, Victor Stinner wrote:
> 2017-10-02 13:10 GMT+02:00 INADA Naoki :
>> https://github.com/python/cpython/pull/3796
>> In this PR, lazy loading only happens when uuid1 is used.
>> But uuid1 is very uncommon for nowdays.
>
> Antoine Pitrou added a new C extension _uuid which is imported as soon
> as uuid(.py) is imported. On Linux at least, the main "overhead" is
> still done on "import uuid". But Antoine change optimized a lot
> "import uuid" import time!
>
>> https://github.com/python/cpython/pull/3757
>> In this PR, singledispatch is lazy loading types and weakref.
>> But singledispatch is used as decorator.
>> So if web application uses singledispatch, it's loaded before preforking.
>
> While "import module" is fast, maybe we should use sometimes a global
> variable to cache the import.
>
> module = None
> def func():
>global module
>if module is None: import module
>...
>
> I'm not sure that it's possible to write an helper for such pattern.
I would rather like to see a function in importlib that handles deferred
imports:
modulename = importlib.deferred_import('modulename')
def deferred_import(name):
if name in sys.modules:
# special case 'None' here
return sys.modules[name]
else:
return ModuleProxy(name)
ModuleProxy is a module type subclass that loads the module on demand.
Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python startup optimization: script vs. service
On 2017-10-02 16:59, Barry Warsaw wrote: > On Oct 2, 2017, at 10:48, Christian Heimes wrote: >> >> That approach could work, but I think that it is the wrong approach. I'd >> rather keep Python optimized for long-running processes and introduce a >> new mode / option to optimize for short-running scripts. > > What would that look like, how would it be invoked, and how would that change > the behavior of the interpreter? I haven't given it much thought yet. Here are just some wild ideas: - add '-l' command line option (l for lazy) - in lazy mode, delay some slow operations (re compile, enum, ...) - delay some imports in lazy mode, e.g. with a deferred import proxy Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python startup optimization: script vs. service
On 2017-10-02 19:29, Brett Cannon wrote: > My current design for an opt-in lazy importing setup includes an > explicit function for importlib that's mainly targeted for the stdlib > and it's startup module needs, but could be used by others: > https://notebooks.azure.com/Brett/libraries/di2Btqj7zSI/html/Lazy%20importing.ipynb Awesome, thanks Brett! :) Small nit pick, you need to add a special case for blocked imports. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [edk2] Official port of Python on EDK2
On 2017-11-01 10:07, Thiebaud Weksteen wrote: > Hi, > > UEFI has become the standard for firmware (BIOS) interface. Intel has > provided an open source implementation under the name EDK2 (part of > the TianoCore initiative) [1] for some time. This implementation has > evolved significantly and now provides the functionalities of a small > OS with a standard library similar to POSIX. > > In 2011, a port of Python 2.7.1 was added to the EDK2 repository [2]. > This port then evolved to 2.7.2 which is still defined as the > reference port [3]. In 2015, another port was added of Python 2.7.10 > in parallel of 2.7.2 [4]. Since then, both implementations have > diverged from upstream and know vulnerabilities have not been fixed. > > I would like to bring support for edk2 in the official Python > repository to remediate this situation, that is officially support > edk2 as a platform. Technically, there would be three main aspects for > the on-boarding work: > > 1) Fix headers and source to resolve definition conflicts, similarly > to ABS definition in [5]; https://gist.github.com/tweksteen/ed516ca7ab7dfa8d18428f59d9c22a3e is a low-hanging fruit, e.g. ABS() should be replaced with Py_ABS() from pymacro.h. Why did you have to replace non-ASCII chars in comments? Does your build chain not support comments with UTF-8 chars? > 2) Add the edk2module.c [6] to handle platform-specific > functionalities, similarly to the posixmodule.c; edk2module.c duplicates a lot of functionality that is also exposed by posixmodule.c. We try to reduce duplicated code and functionality as much as possible. IMO the edk2 module should be folded into posixmodule.c. > 3) Add the build configuration file [7] and necessary modifications > within Python to handle the edk2 toolchain; Once you are able to build master for EDK2, we need to figure out how to integrate the new build flavor into our CI and buildbot system. Is the EDK2 build chain available for free on Linux? > This work would target the master branch (that is Python 3). I would > be interested in hearing your thoughts on this idea. In general your proposal sounds like a good idea. A new platform may require a PEP, though. You can start now by submitting pull requests for the header fixes. Even in the case we decide not to support EDK2, we make your life easier by reducing the amount of extra patches. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] unittest isolation and warnings
Hi guys,
when writing tests, I suddenly discovered that unittest
is not isolated to warnings.
Example:
One of my tests emits warnings when a certain condition is
met. Instead of reporting the error immediately, it uses
warnings, and at the end of the test, an error is produced
if there were warnings.
if hasattr(__main__, "__warningregistry__"):
raise RuntimeError("There are errors, see above.")
By chance, I discovered that an error was suddenly triggered without
a warning. That must mean the warning existed already from
another test as a left-over.
My question:
Is that known, and is that intended?
To what extent are the test cases isolated from each other?
I do admit that my usage of warnings is somewhat special.
But it is very convenient to report many errors on remote servers.
Cheers -- Chris
--
Christian Tismer :^) tis...@stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest isolation and warnings
Thanks a lot!
Good to know.
Ciao -- Chris
On 17.11.17 16:44, Brett Cannon wrote:
> Tests are not isolated from the warnings system, so things will leak
> out. Your best option is to use the context manager in the warnings
> module to temporarily make all warnings raise exceptions and test for
> the exception (I'm at the airport, hence why I don't know the name of
> the context manager; the warnings module docs actually have a sample on
> how best to write tests the involve warnings).
>
>
> On Fri, Nov 17, 2017, 01:34 Christian Tismer, <mailto:tis...@stackless.com>> wrote:
>
> Hi guys,
>
> when writing tests, I suddenly discovered that unittest
> is not isolated to warnings.
>
> Example:
> One of my tests emits warnings when a certain condition is
> met. Instead of reporting the error immediately, it uses
> warnings, and at the end of the test, an error is produced
> if there were warnings.
>
> if hasattr(__main__, "__warningregistry__"):
> raise RuntimeError("There are errors, see above.")
>
> By chance, I discovered that an error was suddenly triggered without
> a warning. That must mean the warning existed already from
> another test as a left-over.
>
> My question:
> Is that known, and is that intended?
> To what extent are the test cases isolated from each other?
>
> I do admit that my usage of warnings is somewhat special.
> But it is very convenient to report many errors on remote servers.
>
> Cheers -- Chris
>
> --
> Christian Tismer :^) tis...@stackless.com
> <mailto:tis...@stackless.com>
> Software Consulting : http://www.stackless.com/
> Karl-Liebknecht-Str. 121 : https://github.com/PySide
> 14482 Potsdam : GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776 fax +49 (30) 700143-0023
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org <mailto:Python-Dev@python.org>
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
--
Christian Tismer :^) tis...@stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] What's the status of PEP 505: None-aware operators?
On 2017-11-28 21:31, Raymond Hettinger wrote: > >> I also cc python-dev to see if anybody here is strongly in favor or against >> this inclusion. > > Put me down for a strong -1. The proposal would occasionally save a few > keystokes but comes at the expense of giving Python a more Perlish look and a > more arcane feel. > > One of the things I like about Python is that I can walk non-programmers > through the code and explain what it does. The examples in PEP 505 look like > a step in the wrong direction. They don't "look like Python" and make me > feel like I have to decrypt the code to figure-out what it does. > > timeout ?? local_timeout ?? global_timeout > 'foo' in (None ?? ['foo', 'bar']) > requested_quantity ?? default_quantity * price > name?.strip()[4:].upper() > user?.first_name.upper() Your examples have convinced me, -1 from me. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [ssl] The weird case of IDNA
Hi,
tl;dr
This mail is about internationalized domain names and TLS/SSL. It
doesn't concern you if you live in ASCII-land. Me and a couple of other
developers like to change the ssl module in a backwards-incompatible way
to fix IDN support for TLS/SSL.
Simply speaking the IDNA standards (internationalized domain names for
applications) describe how to encode non-ASCII domain names. The DNS
system and X.509 certificates cannot handle non-ASCII host names. Any
non-ASCII part of a hostname is punyencoded. For example the host name
'www.bücher.de' (books) is translated into 'www.xn--bcher-kva.de'. In
IDNA terms, 'www.bücher.de' is called an IDN U-label (unicode) and
'www.xn--bcher-kva.de' an IDN A-label (ASCII). Please refer to the TR64
document [1] for more information.
In a perfect world, it would be very simple. We'd only had one IDNA
standard. However there are multiple standards that are incompatible
with each other. The German TLD .de demands IDNA-2008 with UTS#46
compatibility mapping. The hostname 'www.straße.de' maps to
'www.xn--strae-oqa.de'. However in the older IDNA 2003 standard,
'www.straße.de' maps to 'www.strasse.de', but 'strasse.de' is a totally
different domain!
CPython has only support for IDNA 2003.
It's less of an issue for the socket module. It only converts text to
IDNA bytes on the way in. All functions support bytes and text. Since
IDNA encoding does change ASCII and IDNA-encoded data is ASCII, it is
also no problem to pass IDNA2008-encoded text or bytes to all socket
functions.
Example:
>>> import socket
>>> import idna # from PyPI
>>> names = ['straße.de', b'strasse.de', idna.encode('straße.de'),
idna.encode('straße.de').encode('ascii')]
>>> for name in names:
... print(name, socket.getaddrinfo(name, None, socket.AF_INET,
socket.SOCK_STREAM, 0, socket.AI_CANONNAME)[0][3:5])
...
straße.de ('strasse.de', ('89.31.143.1', 0))
b'strasse.de' ('strasse.de', ('89.31.143.1', 0))
b'xn--strae-oqa.de' ('xn--strae-oqa.de', ('81.169.145.78', 0))
xn--strae-oqa.de ('xn--strae-oqa.de', ('81.169.145.78', 0))
As you can see, 'straße.de' is canonicalized as 'strasse.de'. The IDNA
2008 encoded hostname maps to a different IP address.
On the other hand ssl module is currently completely broken. It converts
hostnames from bytes to text with 'idna' codec in some places, but not
in all. The SSLSocket.server_hostname attribute and callback function
SSLContext.set_servername_callback() are decoded as U-label.
Certificate's common name and subject alternative name fields are not
decoded and therefore A-labels. The *must* stay A-labels because
hostname verification is only defined in terms of A-labels. We even had
a security issue once, because partial wildcard like 'xn*.example.org'
must not match IDN hosts like 'xn--bcher-kva.example.org'.
In issue [2] and PR [3], we all agreed that the only sensible fix is to
make 'SSLContext.server_hostname' an ASCII text A-label. But this is an
backwards incompatible fix. On the other hand, IDNA is totally broken
without the fix. Also in my opinion, PR [3] is not going far enough.
Since we have to break backwards compatibility anyway, I'd like to
modify SSLContext.set_servername_callback() at the same time.
Questions:
- Is everybody OK with breaking backwards compatibility? The risk is
small. ASCII-only domains are not affected and IDNA users are broken anyway.
- Should I only fix 3.7 or should we consider a backport to 3.6, too?
Regards,
Christian
[1] https://www.unicode.org/reports/tr46/
[2] https://bugs.python.org/issue28414
[3] https://github.com/python/cpython/pull/3010
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [ssl] The weird case of IDNA
On 2017-12-30 11:28, Antoine Pitrou wrote: > On Fri, 29 Dec 2017 21:54:46 +0100 > Christian Heimes wrote: >> >> On the other hand ssl module is currently completely broken. It converts >> hostnames from bytes to text with 'idna' codec in some places, but not >> in all. The SSLSocket.server_hostname attribute and callback function >> SSLContext.set_servername_callback() are decoded as U-label. >> Certificate's common name and subject alternative name fields are not >> decoded and therefore A-labels. The *must* stay A-labels because >> hostname verification is only defined in terms of A-labels. We even had >> a security issue once, because partial wildcard like 'xn*.example.org' >> must not match IDN hosts like 'xn--bcher-kva.example.org'. >> >> In issue [2] and PR [3], we all agreed that the only sensible fix is to >> make 'SSLContext.server_hostname' an ASCII text A-label. > > What are the changes in API terms? If I'm calling wrap_socket(), can I > pass `server_hostname='straße'` and it will IDNA-encode it? Or do I > have to encode it myself? If the latter, it seems like we are putting > the burden of protocol compliance on users. Only SSLSocket.server_hostname attribute and the hostname argument to the SNI callback will change. Both values will be A-labels instead of U-labels. You can still pass an U-label to the server_hostname argument and it will be encoded with "idna" encoding. >>> sock = ctx.wrap_socket(socket.socket(), server_hostname='www.straße.de') Currently: >>> sock.server_hostname 'www.straße.de' Changed: >>> sock.server_hostname 'www.strasse.de' Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [ssl] The weird case of IDNA
On 2017-12-30 13:19, Skip Montanaro wrote: > Guido wrote: > > This being a security issue I think it's okay to break 3.6. might > even backport to 3.5 if it's easy? > > > Is it also a security issue with 2.x? If so, should a fix to 2.7 be > contemplated? IMO the IDNA encoding problem isn't a security issue per se. The ssl module just cannot handle internationalized domain names at all. IDN domains always fail to verify. Users may just be encouraged to disable hostname verification. On the other hand the use of IDNA 2003 and lack of IDNA 2008 support [1] can be considered a security problem for German, Greek, Japanese, Chinese and Korean domains [2]. I neither have resources nor expertise to address the encoding issue. Christian [1] https://bugs.python.org/issue17305 [2] https://www.unicode.org/reports/tr46/#Transition_Considerations ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] subprocess not escaping "^" on Windows
Hi Guys,
yes I know there was a lengthy thread on python-dev in 2014
called "subprocess shell=True on Windows doesn't escape ^ character".
But in the end, I still don't understand why subprocess does
escape the double quote when shell=True but not other special
characters like "^"?
Yes I know that certain characters are escaped under certain
Windows versions and others are not. And it is not trivial to make
that work correctly in all cases. But I think if we support
some escaping at all, then we should also support all special
cases. Or what sense should an escape make if it works sometimes
and sometimes not?
The user would have to know which cases work and which not. But
I thought we want to remove exactly that burden from him?
-
As a side note: In most cases where shell=True is found, people
seem to need evaluation of the PATH variable. To my understanding,
>>> from subprocess import call
>>> call(("ls",))
works in Linux, but (with dir) not in Windows. But that is misleading
because "dir" is a builtin command but "ls" is not. The same holds for
"del" (Windows) and "rm" (Linux).
So I thought that using shell=True was a good Thing on windows,
but actually it is the start of all evil.
Using regular commands like "git" works fine on Windows and Linux
without the shell=True parameter.
Perhaps it would be a good thing to emulate the builtin programs
in python by some shell=True replacement (emulate_shell=True?)
to match the normal user expectations without using the shell?
Cheers - Chris
--
Christian Tismer-Sperling:^) tis...@stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess not escaping "^" on Windows
That is true.
list2cmdline escapes partially, but on NT and Windows10, the "^" must
also be escaped, but is not. The "|" pipe symbol must also be escaped
by "^", as many others as well.
The effect was that passing a rexexp as parameter to a windows program
gave me strange effects, and I recognized that "^" was missing.
So I was asking for a coherent solution:
Escape things completely or omit "shell=True".
Yes, there is a list of chars to escape, and it is Windows version
dependent. I can provide it if it makes sense.
Cheers -- Chris
On 07.01.18 18:20, Guido van Rossum wrote:
> I assume you're talking about list2cmdline()? That seems to be used to
> construct a string that can be passed to `cmd /c "{}"` -- it gets
> substituted instead of the {}, i.e. surrounded by ". I honestly can't
> say I follow that code completely, but I see that it escapes double
> quotes. Why is there a need to escape other characters? Is there a
> definitive list of special characters somewhere?
>
> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer <mailto:tis...@stackless.com>> wrote:
>
> Hi Guys,
>
> yes I know there was a lengthy thread on python-dev in 2014
> called "subprocess shell=True on Windows doesn't escape ^ character".
>
> But in the end, I still don't understand why subprocess does
> escape the double quote when shell=True but not other special
> characters like "^"?
>
> Yes I know that certain characters are escaped under certain
> Windows versions and others are not. And it is not trivial to make
> that work correctly in all cases. But I think if we support
> some escaping at all, then we should also support all special
> cases. Or what sense should an escape make if it works sometimes
> and sometimes not?
>
> The user would have to know which cases work and which not. But
> I thought we want to remove exactly that burden from him?
>
> -
>
> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
>
> >>> from subprocess import call
> >>> call(("ls",))
>
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
>
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
>
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
>
> Cheers - Chris
>
> --
> Christian Tismer-Sperling :^) tis...@stackless.com
> <mailto:tis...@stackless.com>
> Software Consulting : http://www.stackless.com/
> Karl-Liebknecht-Str <http://www.stackless.com/ Karl-Liebknecht-Str>.
> 121 : https://github.com/PySide
> 14482 Potsdam : GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776 fax +49
> (30) 700143-0023
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org <mailto:Python-Dev@python.org>
> https://mail.python.org/mailman/listinfo/python-dev
> <https://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
> <https://mail.python.org/mailman/options/python-dev/guido%40python.org>
>
>
>
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
--
Christian Tismer-Sperling:^) tis...@stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess not escaping "^" on Windows
By "normal user expectations" I meant the behavior when the builtin commands
were normal programs.
Using "shell=True" is everywhere recommended to avoid, and I believe
we could avoid it by giving them replacements for build-ins.
But I don't care if the shell escaping is correct. And that is not
trivial, either.
On 07.01.18 18:22, Guido van Rossum wrote:
> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer <mailto:tis...@stackless.com>> wrote:
>
> As a side note: In most cases where shell=True is found, people
> seem to need evaluation of the PATH variable. To my understanding,
>
> >>> from subprocess import call
> >>> call(("ls",))
>
> works in Linux, but (with dir) not in Windows. But that is misleading
> because "dir" is a builtin command but "ls" is not. The same holds for
> "del" (Windows) and "rm" (Linux).
>
> So I thought that using shell=True was a good Thing on windows,
> but actually it is the start of all evil.
> Using regular commands like "git" works fine on Windows and Linux
> without the shell=True parameter.
>
> Perhaps it would be a good thing to emulate the builtin programs
> in python by some shell=True replacement (emulate_shell=True?)
> to match the normal user expectations without using the shell?
>
>
> That feels like a terrible idea to me. How do you define "normal user
> expectations" here? If people want shell builtins they should just use
> shell=True. (Also note IIUC there are several quite different shells
> commonly used on Windows, e.g. PowerShell.)
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
--
Christian Tismer-Sperling:^) tis...@stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess not escaping "^" on Windows
Ok, I thought only about Windows where people often use shell=True.
I did not see that as a Linux problem, too.
Not meant as a proposal, just loud thinking... :-)
But as said, the incomplete escaping is a complete mess.
Ciao -- Chris
On 07.01.18 19:54, Christian Tismer wrote:
> By "normal user expectations" I meant the behavior when the builtin commands
> were normal programs.
>
> Using "shell=True" is everywhere recommended to avoid, and I believe
> we could avoid it by giving them replacements for build-ins.
>
> But I don't care if the shell escaping is correct. And that is not
> trivial, either.
>
> On 07.01.18 18:22, Guido van Rossum wrote:
>> On Sun, Jan 7, 2018 at 8:17 AM, Christian Tismer > <mailto:tis...@stackless.com>> wrote:
>>
>> As a side note: In most cases where shell=True is found, people
>> seem to need evaluation of the PATH variable. To my understanding,
>>
>> >>> from subprocess import call
>> >>> call(("ls",))
>>
>> works in Linux, but (with dir) not in Windows. But that is misleading
>> because "dir" is a builtin command but "ls" is not. The same holds for
>> "del" (Windows) and "rm" (Linux).
>>
>> So I thought that using shell=True was a good Thing on windows,
>> but actually it is the start of all evil.
>> Using regular commands like "git" works fine on Windows and Linux
>> without the shell=True parameter.
>>
>> Perhaps it would be a good thing to emulate the builtin programs
>> in python by some shell=True replacement (emulate_shell=True?)
>> to match the normal user expectations without using the shell?
>>
>>
>> That feels like a terrible idea to me. How do you define "normal user
>> expectations" here? If people want shell builtins they should just use
>> shell=True. (Also note IIUC there are several quite different shells
>> commonly used on Windows, e.g. PowerShell.)
>>
>> --
>> --Guido van Rossum (python.org/~guido <http://python.org/~guido>)
>
>
--
Christian Tismer-Sperling:^) tis...@stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam: GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] subprocess not escaping "^" on Windows
Ok, then I'm happy to improve the escaping! I was confused because I could not understand that nobody than me should have run into this problem before. There are many special cases. I'll try my very best :) Cheers -- Chris On 07.01.18 21:59, Guido van Rossum wrote: > On Sun, Jan 7, 2018 at 12:30 PM, Gregory P. Smith <mailto:g...@krypto.org>> wrote: > > the best way to improve shell escaping on windows is to send a PR > against the list2cmdline code that escapes everything you believe it > should when running on windows. With hyperlinks to the relevant msdn > info about what might need escaping. > > > Agreed. FWIW the call to list2cmdline seems to compound the problem, > since it just takes args and puts double quotes around it, mostly > undoing the work of list2cmdline. For example if I use (args=['a', 'b > c'], shell=True) I think list2cmdline turns that to args='a "b c"', and > then the format() expression constructs the command: > > cmd.exe /c "a "b c"" > > I really have no idea what that means on Windows (and no quick access to > a Windows box to try it) but on Windows that would create *two* > arguments, the first one being 'a b' and the second one 'c'. > > At this point I can understand that Christian recommends against > shell=True -- it's totally messed up! But the fix should really be to > fix this, not inventing a new feature. > > -- > --Guido van Rossum (python.org/~guido <http://python.org/~guido>) > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com > -- Christian Tismer :^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
Hi, I'm still working on a ssl module PEP for 3.7 [1], but it's probably not going to be finished before beta 1 deadline. I have a bunch of fixes and improvements for the ssl module in queue, most of them require OpenSSL 1.0.2 features. The features are also present and working properly since LibreSSL 2.5.3 If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land bunch of useful goodies like proper hostname verification [2], proper fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate and PrivateKey types (support loading certs and keys from file and memory) [4], and simplified cipher suite configuration [5]. I can finally clean up _ssl.c during the beta phase, too. OpenSSL 1.0.1 is out of support since December 2016, 0.9.8 since 2015. These versions haven't received any security updates for more than a year! All major Linux and BSD distributions have at least 1.0.2 [6]. The only relevant exception is Ubuntu 14.04 LTS, because Travis CI is running 14.04. PR 3562 [7] contains a PoC to compile a custom build of OpenSSL on Travis. Builds are cached. Regards, Christian [1] https://github.com/tiran/peps/blob/sslmodule37/pep-.txt [2] https://bugs.python.org/issue31399 [3] https://bugs.python.org/issue32185 [4] https://bugs.python.org/issue18369 [5] https://bugs.python.org/issue31429 [6] https://gist.github.com/tiran/c5409bbd60a5f082f654d967add8cc79 [7] https://github.com/python/cpython/pull/3462 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
On 2018-01-13 14:23, Antoine Pitrou wrote: > On Sat, 13 Jan 2018 13:54:33 +0100 > Christian Heimes wrote: >> >> If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land >> bunch of useful goodies like proper hostname verification [2], proper >> fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate >> and PrivateKey types (support loading certs and keys from file and >> memory) [4], and simplified cipher suite configuration [5]. I can >> finally clean up _ssl.c during the beta phase, too. > > Given the annoyance of supporting old OpenSSL versions, I'd say +1 to > this. > > We'll have to deal with the complaints of users of Debian oldstable, > CentOS 6 and RHEL 6, though. It's more of an issue for Travis CI. The Python 3.7-dev target won't have a functional ssl module. Travis either has to update their build base to 16.04, provide a custom build of OpenSSL, or all packages have to use a container. [1] Christian [1] https://github.com/travis-ci/travis-ci/issues/5821#issuecomment-214452987 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Deprecate PEP 370 Per user site-packages directory?
Hi, PEP 370 [1] was my first PEP that got accepted. I created it exactly one decade and two days ago for Python 2.6 and 3.0. Back then we didn't have virtual environment support in Python. Ian Bicking had just started to create the virtualenv project a couple of months earlier. Fast forward 10 years... Nowadays Python has venv in the standard library. The user-specific site-packages directory is no longer that useful. I would even say it's causing more trouble than it's worth. For example it's common for system script to use "#!/usr/bin/python3" shebang without -s or -I option. I propose to deprecate the feature and remove it in Python 4.0. Regards, Christian [1] https://www.python.org/dev/peps/pep-0370/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate PEP 370 Per user site-packages directory?
On 2018-01-13 19:04, Random832 wrote: > On Sat, Jan 13, 2018, at 12:06, Christian Heimes wrote: >> Hi, >> >> PEP 370 [1] was my first PEP that got accepted. I created it exactly one >> decade and two days ago for Python 2.6 and 3.0. Back then we didn't have >> virtual environment support in Python. Ian Bicking had just started to >> create the virtualenv project a couple of months earlier. >> >> Fast forward 10 years... >> >> Nowadays Python has venv in the standard library. The user-specific >> site-packages directory is no longer that useful. I would even say it's >> causing more trouble than it's worth. For example it's common for system >> script to use "#!/usr/bin/python3" shebang without -s or -I option. >> >> I propose to deprecate the feature and remove it in Python 4.0. > > Where would pip install --user put packages, and how would one run scripts > that require those packages? Right now these things Just Work; I've never had > to learn how to use virtual environments. I see two option: 1) "pip install --user" is no longer supported. You have to learn how to use virtual envs. It's really easy: "python3 -m venv path; path/bin/pip install package". 2) "pip install --user" automatically creates or uses a custom virtual (~/.pip/virtualenv-$VERSION/) and links entry points to ~/.local/bin. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate PEP 370 Per user site-packages directory?
On 2018-01-13 20:08, Oleg Broytman wrote: > Hi! > > On Sat, Jan 13, 2018 at 06:06:16PM +0100, Christian Heimes > wrote: >> Hi, >> >> PEP 370 [1] was my first PEP that got accepted. I created it exactly one >> decade and two days ago for Python 2.6 and 3.0. Back then we didn't have >> virtual environment support in Python. Ian Bicking had just started to >> create the virtualenv project a couple of months earlier. >> >> Fast forward 10 years... >> >> Nowadays Python has venv in the standard library. The user-specific >> site-packages directory is no longer that useful. > >Can I disagree? > >> I would even say it's >> causing more trouble than it's worth. For example it's common for system >> script to use "#!/usr/bin/python3" shebang without -s or -I option. > >System scripts are run under user root which seldom has user-specific > site-packages so why worry? You'd be surprised how many tools and programs are using Python these days. You can easily break important user programs by installing a package with --user. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate PEP 370 Per user site-packages directory?
On 2018-01-13 19:57, Antoine Pitrou wrote: > On Sat, 13 Jan 2018 19:18:41 +0100 > Christian Heimes wrote: >> On 2018-01-13 19:04, Random832 wrote: >>> On Sat, Jan 13, 2018, at 12:06, Christian Heimes wrote: >>>> Hi, >>>> >>>> PEP 370 [1] was my first PEP that got accepted. I created it exactly one >>>> decade and two days ago for Python 2.6 and 3.0. Back then we didn't have >>>> virtual environment support in Python. Ian Bicking had just started to >>>> create the virtualenv project a couple of months earlier. >>>> >>>> Fast forward 10 years... >>>> >>>> Nowadays Python has venv in the standard library. The user-specific >>>> site-packages directory is no longer that useful. I would even say it's >>>> causing more trouble than it's worth. For example it's common for system >>>> script to use "#!/usr/bin/python3" shebang without -s or -I option. >>>> >>>> I propose to deprecate the feature and remove it in Python 4.0. >>> >>> Where would pip install --user put packages, and how would one run scripts >>> that require those packages? Right now these things Just Work; I've never >>> had to learn how to use virtual environments. >> >> I see two option: >> >> 1) "pip install --user" is no longer supported. You have to learn how to >> use virtual envs. It's really easy: "python3 -m venv path; path/bin/pip >> install package". >> 2) "pip install --user" automatically creates or uses a custom virtual >> (~/.pip/virtualenv-$VERSION/) and links entry points to ~/.local/bin. > > Option 2 doesn't work, since the installed package then isn't known to > the system Python. I see that as a benefit. User installed packages will no longer be able to break system-wide programs. These days a lot of packages are using setuptools' entry points to create console scripts. Entry point have no option to create a console script with -s or -I flag. On my system, only 40 out of 360 scripts in /usr/bin have -s or -I. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
On 2018-01-13 21:02, Brett Cannon wrote: > +1 from me as well for the improved security. Thanks, Brett! How should we handle CPython's Travis CI tests? The 14.04 boxes have OpenSSL 1.0.1. To the best of my knowledge, Travis doesn't offer 16.04. We could either move to container-based testing with a 16.04 container, which would give us 1.0.2 Or we could compile our own copy of OpenSSL with my multissl builder and use some rpath magic. In order to test all new features, Ubuntu doesn't cut it. Even current snapshot of Ubuntu doesn't contain OpenSSL 1.1. Debian Stretch or Fedora would do the trick, though. Maybe Barry's work on official test container could leveraged testing? Regards, Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
On 2018-01-14 01:03, Steven D'Aprano wrote: > On Sat, Jan 13, 2018 at 02:23:19PM +0100, Antoine Pitrou wrote: >> On Sat, 13 Jan 2018 13:54:33 +0100 >> Christian Heimes wrote: >>> >>> If we agree to drop support for OpenSSL 0.9.8 and 1.0.1, then I can land >>> bunch of useful goodies like proper hostname verification [2], proper >>> fix for IP address in SNI TLS header [3], PEP 543 compatible Certificate >>> and PrivateKey types (support loading certs and keys from file and >>> memory) [4], and simplified cipher suite configuration [5]. I can >>> finally clean up _ssl.c during the beta phase, too. >> >> Given the annoyance of supporting old OpenSSL versions, I'd say +1 to >> this. >> >> We'll have to deal with the complaints of users of Debian oldstable, >> CentOS 6 and RHEL 6, though. > > It will probably be more work for Christian, but is it reasonable to > keep support for the older versions of OpenSSL, but make the useful > goodies conditional on a newer version? It's much more than just goodies. For example the X509_VERIFY_PARAM_set1_host() API fixes a whole lot of issues with ssl.match_hostname(). The feature is OpenSSL 1.0.2+ and baked into the certificate validation system. I don't see a realistic way to perform the same task with 1.0.1. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
On 2018-01-14 03:48, Paul G wrote: > One thing to note is that if getting Travis working with Python 3.7 is a > pain, a huge number of libraries on PyPI probably just won't test > against Python 3.7, which is not a great situation to be in. > > It's probably worth contacting Travis to give them a head's up and see > how likely it is that they'll be able to support Python 3.7 if it > requires a newer version of these libraries. Unless my proposal isn't rejected, I'll contact Travis CI tomorrow. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
On 2018-01-14 11:17, Antoine Pitrou wrote: > On Sat, 13 Jan 2018 23:45:07 +0100 > Christian Heimes wrote: >> On 2018-01-13 21:02, Brett Cannon wrote: >>> +1 from me as well for the improved security. >> >> Thanks, Brett! >> >> How should we handle CPython's Travis CI tests? The 14.04 boxes have >> OpenSSL 1.0.1. To the best of my knowledge, Travis doesn't offer 16.04. >> We could either move to container-based testing with a 16.04 container, >> which would give us 1.0.2 Or we could compile our own copy of OpenSSL >> with my multissl builder and use some rpath magic. > > I don't think you need some rpath magic, just set LD_LIBRARY_PATH to > the right value. I prefer LD_RUN_PATH because it adds rpath to the ELF header of shared libraries and binaries. https://github.com/python/cpython/pull/5180 Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
On 2018-01-14 09:24, Matt Billenstein wrote: > Correct me if I'm wrong, but Python3 on osx bundles openssl since Apple has > deprecated (and no longer ships the header files for) the version shipped with > recent versions of osx. > > Perhaps this is an option to support the various flavors of Linux as well? AFAK Apple has decided to compile and statically link CPython's ssl with an ancient, customized LibreSSL version. Cory posted [1] a couple of months ago Can confirm: macOS 10.13 will ship a Python linked against LibreSSL 2.2.7. A downside: this continues to use the TEA, meaning you cannot choose to distrust the system roots with it. For TEA, see Hynek's blog post [2] I'm not going to add OpenSSL sources or builds to CPython. We just got rid of copies of libffi and other 3rd party dependencies. Crypto and TLS libraries are much, MUCH more complicated to handle than libffi. It's a constant moving targets of attacks. Vendors and distributions also have different opinions about trust store and policies. Let's keep build dependencies a downstream and vendor problem. Christian [1] https://twitter.com/lukasaoz/status/872085966579802112 [2] https://hynek.me/articles/apple-openssl-verification-surprises/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate PEP 370 Per user site-packages directory?
On 2018-01-14 04:16, Barry Warsaw wrote: > On Jan 13, 2018, at 12:06, Christian Heimes wrote: > >> These days a lot of packages are using setuptools' entry points to >> create console scripts. Entry point have no option to create a console >> script with -s or -I flag. On my system, only 40 out of 360 scripts in >> /usr/bin have -s or -I. > > -I should be the default for system scripts; it’s not on Debian/Ubuntu though > unfortunately. Same for most Fedora scripts. :/ Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
On 2018-01-14 16:54, Ned Deily wrote: > On Jan 14, 2018, at 08:39, Christian Heimes wrote: >> On 2018-01-14 09:24, Matt Billenstein wrote: >>> Correct me if I'm wrong, but Python3 on osx bundles openssl since Apple has >>> deprecated (and no longer ships the header files for) the version shipped >>> with >>> recent versions of osx. >>> >>> Perhaps this is an option to support the various flavors of Linux as well? >> >> AFAK Apple has decided to compile and statically link CPython's ssl with >> an ancient, customized LibreSSL version. Cory posted [1] a couple of >> months ago > > I think you're conflating some things here. Apple has not yet shipped a > version of Python 3 with macOS so the fact that Apple now links their > version of Python2.7 with a "private" copy of LibreSSL is irrelevant. > (It's private in the sense that they don't ship the header files for it; > the shared libs are there just for the use of the open source products > they ship with macOS that don't yet use the macOS native crypto APIs, > products like Python and Perl.) > > What Matt is likely thinking of is the Python 3 versions provided by the > python.org macOS binary installers where we do build and link with our > own 1.0.2 (and soon 1.1.0 for 3.7) versions of OpenSSL. Currently, > the OpenSSL (and several other third-party libs such as libxz which > is not shipped by Apple) are built as part of the installer build > script in the Mac section of the source repo. I would like to > refactor and generalize that so those third-party libs > could optionally be used for non-installer builds as well. But, in > any case, we don't have much choice for the installer builds until > such time as cPython has support for the Apple-provided crypto APIs. Yeah, that sounds useful for macOS and Windows development. Back when I was doing more Windows stuff, I used our buildbot scripts to provide local builds of dependencies such as expat and OpenSSL. >> I'm not going to add OpenSSL sources or builds to CPython. We just got >> rid of copies of libffi and other 3rd party dependencies. Crypto and TLS >> libraries are much, MUCH more complicated to handle than libffi. It's a >> constant moving targets of attacks. Vendors and distributions also have >> different opinions about trust store and policies. >> >> Let's keep build dependencies a downstream and vendor problem. > > That's not always an option, unfortunately. For Python.org macOS and Windows installers, I'm considering us as our own downstream vendors. I should rather say "Steve and you" instead of us. You are both doing the heavy lifting. Thanks for you hard work. :) Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >=2.5.3
On 2018-01-16 08:08, Steve Dower wrote: > From my perspective, we can’t keep an OpenSSL-like API and use Windows > platform libraries (we *could* do a requests-like API easily enough, but > even urllib3 is painfully low-level). > > > > We have to continue shipping our own copy of OpenSSL on Windows. Nothing > to negotiate here except whether OpenSSL releases should trigger a > Python release, and I think that decision can stay with the RM. 3.7 will no longer use static linking. We can offer out-of-bounds updates of the OpenSSL DLLs. And by "we", I'm talking about you. :) Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
On 2018-01-16 12:28, Wes Turner wrote: > > > On Tuesday, January 16, 2018, Steve Dower <mailto:steve.do...@python.org>> wrote: > > From my perspective, we can’t keep an OpenSSL-like API and use > Windows platform libraries (we *could* do a requests-like API easily > enough, but even urllib3 is painfully low-level). > > Support for Windows SChannel and Apple SecureTransport is part of the > TLS module. > > IDK how far along that work is (whether it'll be ready for 3.7 beta 1)? > Or where those volunteering to help with the TLS module can send PRs? You are misunderstanding the goal of PEP 543. It's not about providing implementations of various backends. The PEP merely defines an minimal abstraction layer. Neither the PEP nor the API are finalized or complete yet, too Some parts of the PEP must be changed before it can be finalized. Cory and I are discussion the matter. Python 3.7's ssl module won't be compatible with PEP 543. For 3.8 it *might* be possible to provide a 543 compatible implementation on top of the ssl module. I will not work on SChannel or SecureTransport, since I have neither expertise, knowledge, interest or resources to work on other implementations. AFAIK Steve would rather plug in Windows' cert validation API into OpenSSL than to provide another TLS implementation. For Apple ... no clue. How about you contact Apple support? Regards, Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL >= 2.5.3
FYI, master on Travis CI now builds and uses OpenSSL 1.1.0g [1]. I have created a daily cronjob to populate Travis' cache with OpenSSL builds. Until the cache is filled, Linux CI will take an extra 5 minute. Christian [1] https://github.com/python/cpython/pull/5180 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.7: Require OpenSSL >=1.0.2 / LibreSSL>=2.5.3
On 2018-01-16 22:47, Steve Dower wrote: > I think you mean out-of-band updates, and by “you” I'm going to pretend > you mean PyCA ;) Err, yes :) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] LibreSSL support
On 2018-01-16 21:17, Christian Heimes wrote: > FYI, master on Travis CI now builds and uses OpenSSL 1.1.0g [1]. I have > created a daily cronjob to populate Travis' cache with OpenSSL builds. > Until the cache is filled, Linux CI will take an extra 5 minute. I have messed up my initial research. :( When I was checking LibreSSL and OpenSSL for features, I draw a wrong conclusion. LibreSSL is *not* OpenSSL 1.0.2 compatible. It only implements some of the required features from 1.0.2 (e.g. X509_check_hostname) but not X509_VERIFY_PARAM_set1_host. X509_VERIFY_PARAM_set1_host() is required to perform hostname verification during the TLS handshake. Without the function, I'm unable to fix Python's hostname matching code [1]. LibreSSL upstream knows about the issue since 2016 [2]. I have opened another bug report [3]. We have two options until LibreSSL has addressed the issue: 1) Make the SSL module more secure, simpler and standard conform 2) Support LibreSSL I started a vote on Twitter [4]. So far most people prefer security. Christian [1] https://bugs.python.org/issue31399 [2] https://github.com/pyca/cryptography/issues/3247 [3] https://github.com/libressl-portable/portable/issues/381 [4] https://twitter.com/reaperhulk/status/953991843565490176 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] LibreSSL support
On 2018-01-18 19:42, Wes Turner wrote: > Is there a build flag or a ./configure-time autodetection that would > allow for supporting LibreSSL while they port X509_VERIFY_PARAM_set1_host? X509_VERIFY_PARAM_set1_host() is a fundamental and essential piece in the new hostname verification code. I cannot replace ssl.match_hostname() easily without the API. There might be a way to add a callback, but it would take a couple of days of R&D to implement it. It won't be finished for beta1 feature freeze. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] LibreSSL support
On 2018-01-18 20:54, Wes Turner wrote: > LibreSSL is not a pressing need for me; but fallback to the existing > insecure check if LibreSSL is present shouldn't be too difficult? Please give it a try and report back. Patches welcome :) Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] LibreSSL support
On 2018-01-18 21:49, Chris Jerdonek wrote: > > On Thu, Jan 18, 2018 at 7:34 AM Christian Heimes <mailto:christ...@python.org>> wrote: > > On 2018-01-16 21:17, Christian Heimes wrote: > We have two options until LibreSSL has addressed the issue: > > 1) Make the SSL module more secure, simpler and standard conform > 2) Support LibreSSL > > I started a vote on Twitter [4]. So far most people prefer security. > > > It’s not exactly the most balanced (neutral) presentation of a ballot > question though. :) It's more venting than voting :) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Drop support for old unsupported FreeBSD and Linux kernels?
On 2018-01-19 06:36, Benjamin Peterson wrote: > +1 to both of your specific proposals. > > More generally, I think it makes good sense to allow dropping support for a > platform in the next major Python release after vendor support for the > platform stops. Even we say we support something, it will break quickly > without buildbot validation. Do you mean minor release? We haven't done a major release in about a decade. :) I was going to suggest a similar policy for OpenSSL. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] LibreSSL support
On 2018-01-19 10:43, Steve Holden wrote: > On Fri, Jan 19, 2018 at 12:09 AM, Nathaniel Smith <mailto:n...@pobox.com>> wrote: > > On Jan 18, 2018 07:34, "Christian Heimes" <mailto:christ...@python.org>> wrote: > > On 2018-01-16 21:17, Christian Heimes wrote: > > FYI, master on Travis CI now builds and uses OpenSSL 1.1.0g [1]. I > have > > created a daily cronjob to populate Travis' cache with OpenSSL > builds. > > Until the cache is filled, Linux CI will take an extra 5 minute. > > I have messed up my initial research. :( When I was checking > LibreSSL > and OpenSSL for features, I draw a wrong conclusion. LibreSSL is > *not* > OpenSSL 1.0.2 compatible. It only implements some of the required > features from 1.0.2 (e.g. X509_check_hostname) but not > X509_VERIFY_PARAM_set1_host. > > X509_VERIFY_PARAM_set1_host() is required to perform hostname > verification during the TLS handshake. Without the function, I'm > unable > to fix Python's hostname matching code [1]. LibreSSL upstream knows > about the issue since 2016 [2]. I have opened another bug report > [3]. > > We have two options until LibreSSL has addressed the issue: > > 1) Make the SSL module more secure, simpler and standard conform > 2) Support LibreSSL > > > [...] > > > > We have *very* few people qualified to maintain the ssl module, so > given the new landscape I think we should focus on keeping our core > OpenSSL support solid and not worry about LibreSSL. If LibreSSL > wants to be supported as well then – like any other 2nd tier > platform – they need to find someone to do the work. And if people > are worried about supporting more diversity in SSL implementations, > then PEP 543 is probably the thing to focus on. > > Given the hard limit on resources it seems only sensible to focus on > the "industry standard" library. I'm rather disappointed that LibreSSL > isn't a choice, but given the lack of compatibility that's hardly > Python's problem. Thanks! I'd prefer to support LibreSSL, too. Paul Kehrer from PyCA summed up the issue with LibreSSL nicely: > It was marketed as an API compatible drop-in replacement and it is failing in that capacity. Additionally, it is missing features needed to improve the security and ease the maintenance burden of CPython’s dev team. Since I haven given up on LibreSSL, I spent some time and implemented some autoconf magic in https://github.com/python/cpython/pull/5242. It checks for the presence of libssl and X509_VERIFY_PARAM_set1_host() function family: ... checking whether compiling and linking against OpenSSL works... yes checking for X509_VERIFY_PARAM_set1_host in libssl... yes ... The ssl module will regain compatibility with LibreSSL as soon as a new release provides the necessary functions. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] LibreSSL support
On 2018-01-19 15:42, Christian Heimes wrote: > On 2018-01-19 10:43, Steve Holden wrote: >> On Fri, Jan 19, 2018 at 12:09 AM, Nathaniel Smith > <mailto:n...@pobox.com>> wrote: >> >> On Jan 18, 2018 07:34, "Christian Heimes" > <mailto:christ...@python.org>> wrote: >> >> On 2018-01-16 21:17, Christian Heimes wrote: >> > FYI, master on Travis CI now builds and uses OpenSSL 1.1.0g [1]. I >> have >> > created a daily cronjob to populate Travis' cache with OpenSSL >> builds. >> > Until the cache is filled, Linux CI will take an extra 5 minute. >> >> I have messed up my initial research. :( When I was checking >> LibreSSL >> and OpenSSL for features, I draw a wrong conclusion. LibreSSL is >> *not* >> OpenSSL 1.0.2 compatible. It only implements some of the required >> features from 1.0.2 (e.g. X509_check_hostname) but not >> X509_VERIFY_PARAM_set1_host. >> >> X509_VERIFY_PARAM_set1_host() is required to perform hostname >> verification during the TLS handshake. Without the function, I'm >> unable >> to fix Python's hostname matching code [1]. LibreSSL upstream knows >> about the issue since 2016 [2]. I have opened another bug report >> [3]. >> >> We have two options until LibreSSL has addressed the issue: >> >> 1) Make the SSL module more secure, simpler and standard conform >> 2) Support LibreSSL >> >> >> [...] >> >> >> >> We have *very* few people qualified to maintain the ssl module, so >> given the new landscape I think we should focus on keeping our core >> OpenSSL support solid and not worry about LibreSSL. If LibreSSL >> wants to be supported as well then – like any other 2nd tier >> platform – they need to find someone to do the work. And if people >> are worried about supporting more diversity in SSL implementations, >> then PEP 543 is probably the thing to focus on. >> >> Given the hard limit on resources it seems only sensible to focus on >> the "industry standard" library. I'm rather disappointed that LibreSSL >> isn't a choice, but given the lack of compatibility that's hardly >> Python's problem. > > Thanks! > > I'd prefer to support LibreSSL, too. Paul Kehrer from PyCA summed up the > issue with LibreSSL nicely: > >> It was marketed as an API compatible drop-in replacement and it is > failing in that capacity. Additionally, it is missing features needed to > improve the security and ease the maintenance burden of CPython’s dev team. > > > Since I haven given up on LibreSSL, I spent some time and implemented > some autoconf magic in https://github.com/python/cpython/pull/5242. It > checks for the presence of libssl and X509_VERIFY_PARAM_set1_host() > function family: > > ... > checking whether compiling and linking against OpenSSL works... yes > checking for X509_VERIFY_PARAM_set1_host in libssl... yes > ... > > The ssl module will regain compatibility with LibreSSL as soon as a new > release provides the necessary functions. No core developer has vetoed against my proposal. I also spoke to several members of Python Cryptographic Authority and Python Packaging Authority. They are all in favor of my proposal, too. There I have decided to move forward and require OpenSSL 1.0.2 API. This means Python 3.7 temporarily suspends support for LibreSSL until https://github.com/libressl-portable/portable/issues/381 is resolved. I have appended a lengthy explanation to my LibreSSL ticket, too. I also informed LibreSSL developers that Python 3.8 will most likely require an OpenSSL 1.1 compatible API. With OpenSSL 1.0.2 support, I can drop a considerable amount of legacy code, e.g. custom thread locking, initialization and a bunch of shim functions. Regards, Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why is Python for Windows compiled with MSVC?
On 2018-02-01 10:19, Oleg Sivokon wrote: > >> so why shouldn’t the one with the most users? > > Because it makes compilation difficult, and cross-compilatin completely > impossible? Why is it difficult: a package maintainer needs to (1) buy MS > Windows (2) create a special workflow for compiling on a different machine. > This is both costly and inconsistent with free-as-in-freedom... It makes > cross-compilation impossible because libraries produced by any tool that can > run on all target platforms are incompatible with Python binaries on MS > Windows. > > Again, many languages (i.e. projects similar in size an purpose to CPython) > took a different approach: they use GNU compilers to be able to compile > cross-platform. This is true for Ruby and Go at least. I would need to > investigate further, but I think these two examples should be enough. > >> I’m likely biased because I work there and I’m the main intermediary with >> python-dev, but these days Microsoft is one of the strongest supporters of >> CPython. We employ the most core developers of any private company and we >> all are allowed work time to contribute, we provide full access to our >> development tools and platforms to all core developers and some prominent >> projects, we’ve made fixes, enhancements and releases or core products such >> as the CRT, MSVC, Visual Studio, Visual Studio Code, and Azure SPECIFICALLY >> to support CPython development and users. As far as I know, ALL the Windows >> buildbots are running on Azure subscriptions that Microsoft provides (though >> managed by some awesome volunteers). You’ll see us at PyCon US under the >> biggest banner and we’ll have a booth filled with engineers and not >> recruiters. Crash reports from thousands of opted-in users come into our >> systems and have directly lead to both CPython and Windows bug fixes. > > Oh, so this is the real reason... well, corporate interests are hard to argue > against. But, this is an interesting statistic nevertheless. Thanks for > letting me know. You are drawing the wrong conclusion here. CPython has been using MSVC many years before Microsoft even started to sponsor MSDN subscriptions for core developers. MSVC is the default Windows compiler for over 20 years now. IIRC Microsoft started to donate MSDN subscription for about 5 years and Steve has been helping out with Windows improvement for about 5 years. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Deprecate crypt module and revert PR 3854
Hi, in PR 3854 [1] Serhiy added blowfish, extended DES and NT-Hash to Python's crypt mdodule. I vetoed against addition of the APIs because all these hashing algorithms are not state of the art. Their quality ranges from old to horribly, horriblye broken beyond any repair. Shortly after the PR has landed, I was made aware that glibc has deprecated crypt(3) API [2] and favor of an external library called libxcrypt [3] from OpenWall Linux. I have patched Python 3.7 [4] to support libxcrypt. In light of deprecation of crypt(3) glibc function and bad quality of hashing algorithms, I'd like to raise the motion to revert 3854 and deprecate the crypt module. The whole module should be rather moved into 3rd party library that wraps xcrypt. Regards, Christian [1] https://github.com/python/cpython/pull/3854 [2] https://www.phoronix.com/scan.php?page=news_item&px=Fedora-28-libxcrypt-Plans [3] https://github.com/besser82/libxcrypt [4] https://bugs.python.org/issue32635 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate crypt module and revert PR 3854
On 2018-02-02 18:05, Serhiy Storchaka wrote: > 02.02.18 18:18, Guido van Rossum пише: >> I'm all for nudging people in the direction of xcrypt. I assume we >> can't just switch the C-level crypt with xcrypt and leave the Python >> API unchanged? >> >> However until a usable solution exist (either in the stdlib or as 3rd >> party) I don't think we should deprecate anything (deprecating things >> before the replacement is ready is stressful for everyone involved). >> >> I'm also not sure I agree with removing support for old hashes. By all >> means put in the docs that they are unsafe. But if someone has a >> database full of old hashes it would be nice to be able to at least >> read/verify it, right? >> >> Was a release already made with blowfish, extended DES and NT-Hash? >> (And what's so bad with blowfish? It's mentioned in the heading of the >> xcrypt project too.) > > To clarify, extended DES and NT-Hash were not added. They were removed > from my PR after Christians request. Only the Blowfish method was added, > and it is so strong as SHA-2 methods. It is the only method supported on > OpenBSD. Ah, I misinterpreted the subject of the PR. The closed PR still mentions extended DES and NT-Hash. I'm sorry and blame my travel fatigue. The email was written at the airport after I had a conversion with somebody about new Python 3.7 features. > This PR is not a single enhancement made in the crypt module recently. I > also extended tests and added support for configuring SHA-2 methods. > There is an open PR (not merged before 3.7b1 unfortunately) for using > crypt_r() instead of crypt(): https://bugs.python.org/issue28503. In general I'm all for more tests and improvements of existing modules. However in this case Python 3.7 is sending wrong signals. For example additional of blowfish was prominently features on the largest German newsletter for IT. Both blowfish and SSHA (salted sha) are legacy password hashing algorithms. Glibc has moved them out of the main library for a good reason. (*) > If deprecate the crypt module, should modules pwd, grp and spwd be > deprecated too? The crypt module is needed for checking password hashes > provided by spwd. The pwd and grp module are fine. The modules use proper libc APIs that are internally backed by NSS (libc's Name Service Switch, not Mozilla's Network Security Service). APIs such getpwnam are defined and standardized since POSIX.1-2001. The pwd and grp automatically work with any configured user and group provider, even LDAP, IdM or Active Directory. Fun fact: Golang programs are usually statically compiled and don't even use libc. However Go's os/user package requires CGO and libc because it has to interface with libc and NSS to acquire user and group information. The spwd module is a different story. It's a direct interface to /etc/shadown using Linux-only APIs. The shadow DB API requires root permission. I think it even circumvents system security policies and identity provider. tl;dr pwd + grp == good, required crypt + spwd == bad Regards, Christian (*) Most Linux distros never had blowfish in libc anyway. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
On 03.02.18 08:12, Steven D'Aprano wrote: > On Sat, Feb 03, 2018 at 12:50:11AM -0500, Terry Reedy wrote: >> On 2/2/2018 7:01 PM, asrp asrp wrote: >>> I don't know if this is the right place to post this. Please redirect as >>> needed. >> >> This list is for development *of* cpython. Development *with* python in >> general belongs on python-list. > > This list is for development of Python the language, not just CPython > the interpreter. It seems to me that announcing a new Python > interpreter, especially one designed for the purpose of allowing rapid > experimentation with the language, is on topic for this list. > > Well spoken! -- Christian Tismer-Sperling:^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate crypt module and revert PR 3854
On 2018-02-02 21:31, Antoine Pitrou wrote: > On Fri, 2 Feb 2018 16:23:20 +0100 > Christian Heimes wrote: >> Hi, >> >> in PR 3854 [1] Serhiy added blowfish, extended DES and NT-Hash to >> Python's crypt mdodule. I vetoed against addition of the APIs because >> all these hashing algorithms are not state of the art. Their quality >> ranges from old to horribly, horriblye broken beyond any repair. >> >> Shortly after the PR has landed, I was made aware that glibc has >> deprecated crypt(3) API [2] and favor of an external library called >> libxcrypt [3] from OpenWall Linux. I have patched Python 3.7 [4] to >> support libxcrypt. >> >> In light of deprecation of crypt(3) glibc function and bad quality of >> hashing algorithms, I'd like to raise the motion to revert 3854 and >> deprecate the crypt module. > > Those are two separate proposals. > > On the topic of reverting PR #3854, I don't see the point. Is Blowfish > more fragile than the other algorithms? If not, it sounds ok to add it. Blowfish password hashing algorithms (also known as bcrypt) hasn't been state of the art of a long time. Here is an old blog post from 2012 that explains some of the issues with Blowfish: http://www.unlimitednovelty.com/2012/03/dont-use-bcrypt.html > On the topic of deprecating the crypt module, that doesn't sound like a > good idea right now. People may need to generate crypt()-compatible > output for various reasons, such as being able to automate system > administration tasks. IMO legacy support is not a good argument to keep the crypt module. The passlib [1] package provides an excellent selection of legacy and modern password derivation and hashing algorithms. As an admin you cannot rely on the crypt module. libc's crypt() function usually provides a subset. For example most Linux libc do not provide blowfish based hashing. Christian [1] https://pythonhosted.org/passlib/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate crypt module and revert PR 3854
On 2018-02-02 21:21, Nathaniel Smith wrote: > On Feb 2, 2018 7:24 AM, "Christian Heimes" <mailto:christ...@python.org>> wrote: > > Shortly after the PR has landed, I was made aware that glibc has > deprecated crypt(3) API [2] and favor of an external library called > libxcrypt [3] from OpenWall Linux. I have patched Python 3.7 [4] to > support libxcrypt. > > In light of deprecation of crypt(3) glibc function and bad quality of > hashing algorithms, I'd like to raise the motion to revert 3854 and > deprecate the crypt module. The whole module should be rather moved into > 3rd party library that wraps xcrypt. > > > If the crypt module can just as well use libxcrypt, and that's how the > distros are going to build it, then what's the advantage of deprecating > it? Won't it just continue to work indefinitely? libxcrypt is now an external library that must be installed separately. It bloats CPython's dependency tree and the size of Python container images for a legacy feature. For Python 4.0 I'd like to drop some of the old, dead batteries and include include useful batteries. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
Hi user with no real name yet, On 03.02.18 01:01, asrp asrp wrote: > Hello, > > I don't know if this is the right place to post this. Please redirect as > needed. > > I've made a small Python interpreter in Python with runtime AST node > semantics and edit-and-continue. I thought it could make prototyping language > changes more easily and visualize usage before writing them in C. > > Its here: https://github.com/asrp/python_terp > > So, for example, redefining the for_stmt function in the right scope changes > the behaviour of future for loops at runtime. > > Although from discussion I've read in PEPs, actual implementation always look > like a non-issue (which seems like magic to me) so maybe no-one here actually > needs this. > > (I really needed edit-and-continue for one of my projects but of course, > running it in this extra interpreter is much too slow.) > > asrp In the readme to python_terp you say: """ python_terp is intended to make language modification to Python easier to preview changes more quickly and is not intended for full CPython compatibility. However, a large subset of Python is already included. In particular, enough to run the first stage of its parser. """ This needs clarification. What do you mean by subset? A real subset or also things that are different and will stay different? To what extent are you planning to stay compatible, and where do you plan to deviate? The reason that I'm asking is that by compatible I mean the compatibility of PyPy. If you can reach that, and be it just by a subset, then it makes sense to speak of Python. Cheers - Chris -- Christian Tismer-Sperling:^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : https://github.com/PySide 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Deprecate crypt module and revert PR 3854
On 2018-02-02 17:18, Guido van Rossum wrote: > I'm all for nudging people in the direction of xcrypt. I assume we can't > just switch the C-level crypt with xcrypt and leave the Python API > unchanged? > > However until a usable solution exist (either in the stdlib or as 3rd > party) I don't think we should deprecate anything (deprecating things > before the replacement is ready is stressful for everyone involved). > > I'm also not sure I agree with removing support for old hashes. By all > means put in the docs that they are unsafe. But if someone has a > database full of old hashes it would be nice to be able to at least > read/verify it, right? > > Was a release already made with blowfish, extended DES and NT-Hash? (And > what's so bad with blowfish? It's mentioned in the heading of the xcrypt > project too.) I answered some of your questions in other replies and will answer the remaining concerns on Monday. You suggested a 3rd party module. I have cloned the crypt module with Serhiy's improvements and turned it into a standalone module with a ctypes interface, https://github.com/tiran/legacycrypt . I'll release the package as soon as I find time to polish the documentation and give Serhiy his will deserved credit for his work. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python Test suite hangining
On 2018-03-05 13:13, Brett Cannon wrote: > > > On Sun, 4 Mar 2018 at 11:38 TonyFlury via Python-Dev > mailto:python-dev@python.org>> wrote: > > All, > Sorry to trouble you all - but I am trying to get the Python 3.8 > test suite running on Ubuntu 16.0.4. > > As per the dev guide - having cloned the repo and run the build I am > running the test suite by : ./python -m test -j1 > > This runs through to test 414/415 - and then start getting this message > > running: test_poplib (nnn sec) > > every 30 seconds - nnn got to 1077 secs before I decided to stop it. > > > When I ran test_poplib on it's own - I got this Traceback : > > Traceback (most recent call last): > ... > ... > File "/home/tony/Development/python/cpython/Lib/ssl.py", line > 1108, in do_handshake > self._sslobj.do_handshake() > ssl.SSLError: [SSL: SSLV3_ALERT_CERTIFICATE_UNKNOWN] sslv3 alert > certificate unknown (_ssl.c:1038) > > And then lots of reports of unexpected EOFs. > > I have the latest CA-certificates installed > > Any advice - I couldn't find anything on google. > > > CI is testing, so this isn't a universal issue: > https://travis-ci.org/python/cpython/jobs/349398791 . So I would see if > you could diagnose whether your latest certs are so new compared to what > others who run the test suite are using that that's what is causing your > failure compared to others. Python's test suite should depend on public CAs. The issue in poplib is related to https://bugs.python.org/issue32706 . I'll address the problem when I'm back home in Germany next week. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Why is pickle.DEFAULT_PROTOCOL still 3?
On 2018-04-02 22:48, Lukasz Langa wrote: > Pickle protocol version 4.0 was originally defined back in PEP 3154 and > shipped as part of Python 3.4 back in 2011. Yet it's still not the default. > There's a number of things that would run faster with it like multiprocessing. > > This is too late for 3.7 which is a shame but can we at least bump it for 3.8? It sounds like a reasonable request. Python 3.4 is out of commission by then. I'm sure the release manager for 3.8 is going to agree with you, too. :) Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] ssl module and LibreSSL CVE-2018-8970
Hi, I like to share the story of a critical security bug with you. Contrary to other issues in TLS/SSL, it's a story with happy ending. Nobody was harmed. The bug was fixed before it affected the general population. Introduction Python's ssl.match_hostname() function was a source of several CVEs and other security bugs. After a long struggle, I decided to drop support for old OpenSSL releases and uses a new OpenSSL method to offload host name verification to OpenSSL. The improvement [1] eventually landed in Python 3.7. Nowadays OpenSSL verifies host name or IP address during the TLS/SSL handshake. Later I discovered that LibreSSL <= 2.6 did not have X509_VERIFY_PARAM_set1_host() [2]. We had to temporarily suspend support for LibreSSL. About two months later, LibreSSL caught up and released version 2.7.0 with support for the function. The bug --- One day after the release of LibreSSL 2.7.0, I started to port Python 3.7 to LibreSSL. In matter of minutes I got the ssl module to compile and work with LibreSSL. All tests were passing -- except for negative the host name verification tests. LibreSSL was accepting all invalid host names as correct! Python's vigorous test suite had discovered a critical security bug in LibreSSL. It turned out that LibreSSL copied the implementation of X509_VERIFY_PARAM_set1_host(param, name, namelen) from BoringSSL and the documentation from OpenSSL. BoringSSL's implementation didn't support the special case of 0 as namelen parammeter. OpenSSL supports namelen = 0, which is interpreted as namelen=strlen(name). It is documented in OpenSSL's man page and was even recommended on OpenSSL's wiki as preferred way. Happy Ending So I got in contact with LibreSSL's security team and BoringSSL's security team [3]. Less than a day later, both libraries released fixes for the bug [4]. Mitre has assigned CVE-2018-8970 [5] to the bug. Disaster averted! BoringSSL's security issue [3] contains more information. Adam Langley lifted the restriction about an hour ago. I like to thank Bob Beck (LibreSSL), Adam Langley (Google) and David Benjamin (Google) for their assistance and cooperation. Regards, Christian [1] https://bugs.python.org/issue31399 [2] https://github.com/libressl-portable/portable/issues/381 [3] https://bugs.chromium.org/p/chromium/issues/detail?id=824799 [4] https://www.libressl.org/releases.html [5] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-8970 ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Trying to build from source, test-poplib fails
On 2018-04-08 01:33, Skip Montanaro wrote: >> Do you have ca-certificates installed? > > It seems so: > > % apt search ca-certificates | grep installed > > ca-certificates/artful,artful,now 20170717 all [installed] > ca-certificates-mono/artful,artful,now 4.6.2.7+dfsg-1ubuntu1 all > [installed,automatic] > liblwp-protocol-https-perl/artful,artful,now 6.07-2 all [installed] Skip, it's a red herring. Python's test_poplib suite doesn't depend on public CA certs. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Trying to build from source, test-poplib fails
On 2018-04-09 10:30, Serhiy Storchaka wrote: >> I get the same issues too, and this has been happening for quite some >> time. The tests and/or poplib itself may be written in a fragile way. > > I get the different issue with test_poplib (running with the -j option). > All tests except test_multiprocessing_fork are passed, and test_poplib > hangs. I thought it is a misconfiguration on my computer. It's not your computer, it's the test suite. It uses the old asyncore library. The test has become unstable since the ssl module properly checks host names, see https://bugs.python.org/issue32706 and https://bugs.python.org/issue32753. In Python 3.6 and earlier, a hostname verification error resulted in a closed connection. Starting with Python 3.7, the ssl module checks the hostname earlier in the TLS handshake. A verification error results in a TLS handshake alert and an exception on the server side. Sometimes the old server code in asyncore runs into a race condition and the test fails. The whole test framework needs to be rewritten from scratch and replaced with something better. I haven't had time to work on it. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] bpo-28055: Fix unaligned accesses in siphash24(). (GH-6123)
On 2018-05-13 06:57, Serhiy Storchaka wrote: > https://github.com/python/cpython/commit/1e2ec8a996daec65d8d5a3d43b66a9909c6d0653 > commit: 1e2ec8a996daec65d8d5a3d43b66a9909c6d0653 > branch: master > author: Rolf Eike Beer > committer: Serhiy Storchaka > date: 2018-05-13T13:57:31+03:00 > summary: > > bpo-28055: Fix unaligned accesses in siphash24(). (GH-6123) > > The hash implementation casts the input pointer to uint64_t* and directly > reads > from this, which may cause unaligned accesses. Use memcpy() instead so this > code > will not crash with SIGBUS on sparc. > > https://bugs.gentoo.org/show_bug.cgi?id=636400 > > files: > A Misc/NEWS.d/next/Core and Builtins/2018-04-25-20-44-42.bpo-28055.f49kfC.rst > M Python/pyhash.c Hi Serhiy, I was against the approach a good reason. The PR adds additional CPU instructions and changes memory access pattern in a critical path of CPython. There is no reason to add additional overhead for the majority of users or X86 and X86_64 architectures. The memcpy() call should only be used on architectures that do not support unaligned memory access. See comment https://bugs.python.org/issue28055#msg276257 At least for latest GCC, the change seems to be fine. GCC emits the same assembly code for X86_64 before and after your change. Did you check the output on other CPU architectures as well as clang and MSVC, too? Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Hashes in Python3.5 for tuples and frozensets
On 2018-05-16 18:10, Raymond Hettinger wrote: > > >> On May 16, 2018, at 5:48 PM, Anthony Flury via Python-Dev >> wrote: >> >> However the frozen set hash, the same in both cases, as is the hash of the >> tuples - suggesting that the vulnerability resolved in Python 3.3 wasn't >> resolved across all potentially hashable values. > > You are correct. The hash randomization only applies to strings. None of > the other object hashes were altered. Whether this is a vulnerability or not > depends greatly on what is exposed to users (generally strings) and how it is > used. > > For the most part, it is considered a feature that integers hash to > themselves. That is very fast to compute :-) Also, it tends to prevent hash > collisions for consecutive integers. Raymond is 100% correct. Just one small nit pick: randomization applies to both string and bytes. Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Failing tests on master (asyncio, multiprocessing)
Hi, several of my PRs as well as local tests have started failing recently. On my local Fedora 27 machine, four sendfile related tests of test_asyncio's BaseLoopSockSendfileTests suite are failing reproducible. For example Travis CI job https://travis-ci.org/python/cpython/jobs/380852981 fails in: * test_stdin_broken_pipe (test.test_asyncio.test_subprocess.SubprocessFastWatcherTests) * test_stdin_broken_pipe (test.test_asyncio.test_subprocess.SubprocessSafeWatcherTests) * test_ignore (test.test_multiprocessing_forkserver.TestIgnoreEINTR) Could somebody have a look, please? Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Failing tests on master (asyncio, multiprocessing)
On 2018-05-19 11:29, Eitan Adler wrote: > On 19 May 2018 at 02:05, Christian Heimes wrote: >> Hi, >> >> several of my PRs as well as local tests have started failing recently. >> On my local Fedora 27 machine, four sendfile related tests of >> test_asyncio's BaseLoopSockSendfileTests suite are failing reproducible. >> > > https://bugs.python.org/issue33531 ? Yeah, that's it. Thanks! Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PyIndex_Check conflicts with PEP 384
Hi friends, when implementing the limited API for PySide2, I recognized a little bug where a function was replaced by a macro. The file number.rst on python 3.6 says """ .. c:function:: int PyIndex_Check(PyObject *o) Returns ``1`` if *o* is an index integer (has the nb_index slot of the tp_as_number structure filled in), and ``0`` otherwise. """ Without notice, this function was replaced by a macro a while ago, which reads #define PyIndex_Check(obj) \ ((obj)->ob_type->tp_as_number != NULL && \ (obj)->ob_type->tp_as_number->nb_index != NULL) This contradicts PEP 384, because there is no way for non-heaptype types to access the nb_index field. If nobody objects, I would like to submit a patch that adds the function back when the limited API is active. I think to fix that before Python 3.7 is out. Ciao -- Chris -- Christian Tismer-Sperling:^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://pyside.org 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PyIndex_Check conflicts with PEP 384
Hi Nate, I just did that, and I got some follow-up work, too, which is fine with me. Just as a note for you: Qt not itself, but PyQt5 did that already and submitted a stable ABI for Python 3.5 and up. I was challenged end of last December to try that, and I succeeded after a long struggle, because I also needed to convert all of PySide2 to using heaptypes, and that was really hard. Actually, I succeeded yesterday, after 5 months. And now I know all the subtle things that people need to know when converting existing types to heaptypes. Since QtC has adopted PySide2 in 2016, including myself as a consultant, now it is really a Qt product, and the limited API is due to my work. It comes naturally that I also should fix the problems which showed up during that process. I also think to submit a paper to python.org where I document all the subtle problems which occured during the conversion process. It looks simple, but it really is not. All the best -- Chris On 01.06.18 17:18, Nathaniel Smith wrote: > Indeed, that sounds like a pretty straightforward bug in the stable ABI. > You should file an issue on bugs.python.org <http://bugs.python.org> so > it doesn't get lost (and if it's the main new stable ABI break in 3.7 > then you should probably mark that bug as a release blocker so that Ned > notices it). > > Unfortunately, very few people use the stable ABI currently, so it's > easy for things like this to get missed. Hopefully it Qt starts using it > then that will help us shake these things out... But it also means we > need your help to catch these kinds of issues :-). Thanks! > > On Fri, Jun 1, 2018, 06:51 Christian Tismer <mailto:tis...@stackless.com>> wrote: > > Hi friends, > > when implementing the limited API for PySide2, I recognized > a little bug where a function was replaced by a macro. > > The file number.rst on python 3.6 says > > """ > .. c:function:: int PyIndex_Check(PyObject *o) > > Returns ``1`` if *o* is an index integer (has the nb_index slot > of the > tp_as_number structure filled in), and ``0`` otherwise. > """ > > Without notice, this function was replaced by a macro a while > ago, which reads > > #define PyIndex_Check(obj) \ > ((obj)->ob_type->tp_as_number != NULL && \ > (obj)->ob_type->tp_as_number->nb_index != NULL) > > This contradicts PEP 384, because there is no way for non-heaptype > types to access the nb_index field. > > If nobody objects, I would like to submit a patch that adds the > function back when the limited API is active. > I think to fix that before Python 3.7 is out. > > Ciao -- Chris > > -- > Christian Tismer-Sperling :^) tis...@stackless.com > <mailto:tis...@stackless.com> > Software Consulting : http://www.stackless.com/ > Karl-Liebknecht-Str. 121 : http://pyside.org > 14482 Potsdam : GPG key -> 0xFB7BEE0E > phone +49 173 24 18 776 fax +49 (30) 700143-0023 > > ___ > Python-Dev mailing list > Python-Dev@python.org <mailto:Python-Dev@python.org> > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/njs%40pobox.com > -- Christian Tismer-Sperling:^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://pyside.org 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable ABI
On 02.06.18 05:47, Nick Coghlan wrote: > On 2 June 2018 at 03:45, Jeroen Demeyer <mailto:j.deme...@ugent.be>> wrote: > > On 2018-06-01 17:18, Nathaniel Smith wrote: > > Unfortunately, very few people use the stable ABI currently, so it's > easy for things like this to get missed. > > > So there are no tests for the stable ABI in Python? > > > Unfortunately not. > > https://bugs.python.org/issue21142 is an old issue suggesting automating > those checks (so we don't inadvertently add or remove symbols for > previously published stable ABI definitions), but it's not yet made it > to the state of being sufficiently well automated that it can be a > release checklist item in PEP 101. > > Cheers, > Nick. Actually, I think we don't need such a test any more, or we could use this one as a heuristic test: I have written a script that scans all relevant header files and analyses all sections which are reachable in the limited API context. All macros that don't begin with an underscore which contain a "->tp_" string are the locations which will break. I found exactly 7 locations where this is the case. My PR will contain the 7 fixes plus the analysis script to go into tools. Preparind that in the evening. cheers -- Chris -- Christian Tismer-Sperling:^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://pyside.org 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable ABI
On 03.06.18 13:18, Ronald Oussoren wrote: > > >> On 3 Jun 2018, at 12:03, Christian Tismer wrote: ... >> >> I have written a script that scans all relevant header files >> and analyses all sections which are reachable in the limited API >> context. >> All macros that don't begin with an underscore which contain >> a "->tp_" string are the locations which will break. >> >> I found exactly 7 locations where this is the case. >> >> My PR will contain the 7 fixes plus the analysis script >> to go into tools. Preparind that in the evening. > > Having tests would still be nice to detect changes to the stable ABI when > they are made. > > Writing those tests is quite some work though, especially if those at least > smoke test the limited ABI by compiling snippets the use all symbols that > should be exposed by the limited ABI. Writing those tests should be fairly > simple for someone that knows how to write C extensions, but is some work. > > Writing a tests that complain when the headers expose symbols that shouldn’t > be exposed is harder, due to the need to parse headers (either by hacking > something together using regular expressions, or by using tools like gccxml > or clang’s C API). What do you mean? My script does that with all "tp_*" type fields. What else would you want to check? -- Christian Tismer-Sperling:^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://pyside.org 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PySequence_Check but no __len__
Hi friends, there is a case in the Python API where I am not sure what to do: If an object defines __getitem__() only but no __len__(), then PySequence_Check() already is true and does not care. So if I define no __len__, it simply fails. Is this intended? I was mislead and thought this was the unlimited case, but it seems still to be true that sequences are always finite. Can someone please enlighten me? -- Christian Tismer-Sperling:^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://pyside.org 14482 Potsdam: GPG key -> 0xE7301150FB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PySequence_Check but no __len__
Hi Brett,
because you did not understand me, I must have had a fundamental
misunderstanding. So I started a self-analysis and came to the
conclusion that this was my error since maybe a decade:
When iterators and generators came into existence, I somehow
fell into the trap to think that there are now sequences with
undetermined or infinite length. They would be exactly those sequences
which have no __len__ attribute.
I understand now that sequences are always of fixed length
and adjusted myself.
-
My problem is to find out how to deal with a class which has
__getitem__ but no __len__.
The documentation suggests that the length of a sequence can always
be obtained by len().
https://docs.python.org/3/reference/datamodel.html
But the existence of __len__ is not guaranteed or enforced.
And if you look at the definition of PySequence_Fast(), you find that
a sequence can be turned into a list with iteration only and no __len__.
So, is a sequence valid without __len__, if iteration is supported,
instead?
There is the whole chapter about sequence protocol
https://docs.python.org/3/c-api/sequence.html?highlight=sequence
but I cannot find out an exact definition what makes up a sequence?
Sorry if I'm again the only one who misunderstands the obvious :)
Best -- Chris
On 21.06.18 18:29, Brett Cannon wrote:
> Sorry, I don't quite follow.
>
> On Thu, 21 Jun 2018 at 08:50 Christian Tismer <mailto:tis...@stackless.com>> wrote:
>
> Hi friends,
>
> there is a case in the Python API where I am not sure what to do:
>
> If an object defines __getitem__() only but no __len__(),
> then PySequence_Check() already is true and does not care.
>
>
> Which matches
> https://docs.python.org/3/c-api/sequence.html#c.PySequence_Check .
>
> From Objects/abstract.c:
>
> int
> PySequence_Check(PyObject *s)
> {
> if (PyDict_Check(s))
> return 0;
> return s != NULL && s->ob_type->tp_as_sequence &&
> s->ob_type->tp_as_sequence->sq_item != NULL;
> }
>
>
>
>
> So if I define no __len__, it simply fails. Is this intended?
>
>
> What is "it" in this case that is failing? It isn't PySequence_Check()
> so I'm not sure what the issue is.
>
> -Brett
>
>
>
> I was mislead and thought this was the unlimited case, but
> it seems still to be true that sequences are always finite.
>
> Can someone please enlighten me?
> --
> Christian Tismer-Sperling :^) tis...@stackless.com
> <mailto:tis...@stackless.com>
> Software Consulting : http://www.stackless.com/
> Karl-Liebknecht-Str. 121 : http://pyside.org
> 14482 Potsdam : GPG key -> 0xE7301150FB7BEE0E
> phone +49 173 24 18 776 fax +49 (30)
> 700143-0023
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org <mailto:Python-Dev@python.org>
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com
>
--
Christian Tismer-Sperling:^) tis...@stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : http://pyside.org
14482 Potsdam: GPG key -> 0xE7301150FB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PySequence_Check but no __len__
Answering myself:
PySequence_Check determines a sequence. See the docs.
len() can but does not have to exist.
The size is always limited.
After evicting my initial fault, this is now obvious.
Sorry about the noise.
On 22.06.18 13:17, Christian Tismer wrote:
> Hi Brett,
>
> because you did not understand me, I must have had a fundamental
> misunderstanding. So I started a self-analysis and came to the
> conclusion that this was my error since maybe a decade:
>
> When iterators and generators came into existence, I somehow
> fell into the trap to think that there are now sequences with
> undetermined or infinite length. They would be exactly those sequences
> which have no __len__ attribute.
>
> I understand now that sequences are always of fixed length
> and adjusted myself.
>
> -
>
> My problem is to find out how to deal with a class which has
> __getitem__ but no __len__.
>
> The documentation suggests that the length of a sequence can always
> be obtained by len().
> https://docs.python.org/3/reference/datamodel.html
>
> But the existence of __len__ is not guaranteed or enforced.
> And if you look at the definition of PySequence_Fast(), you find that
> a sequence can be turned into a list with iteration only and no __len__.
>
> So, is a sequence valid without __len__, if iteration is supported,
> instead?
>
> There is the whole chapter about sequence protocol
> https://docs.python.org/3/c-api/sequence.html?highlight=sequence
>
> but I cannot find out an exact definition what makes up a sequence?
>
> Sorry if I'm again the only one who misunderstands the obvious :)
>
> Best -- Chris
>
>
> On 21.06.18 18:29, Brett Cannon wrote:
>> Sorry, I don't quite follow.
>>
>> On Thu, 21 Jun 2018 at 08:50 Christian Tismer > <mailto:tis...@stackless.com>> wrote:
>>
>> Hi friends,
>>
>> there is a case in the Python API where I am not sure what to do:
>>
>> If an object defines __getitem__() only but no __len__(),
>> then PySequence_Check() already is true and does not care.
>>
>>
>> Which matches
>> https://docs.python.org/3/c-api/sequence.html#c.PySequence_Check .
>>
>> From Objects/abstract.c:
>>
>> int
>> PySequence_Check(PyObject *s)
>> {
>> if (PyDict_Check(s))
>> return 0;
>> return s != NULL && s->ob_type->tp_as_sequence &&
>> s->ob_type->tp_as_sequence->sq_item != NULL;
>> }
>>
>>
>>
>>
>> So if I define no __len__, it simply fails. Is this intended?
>>
>>
>> What is "it" in this case that is failing? It isn't PySequence_Check()
>> so I'm not sure what the issue is.
>>
>> -Brett
>>
>>
>>
>> I was mislead and thought this was the unlimited case, but
>> it seems still to be true that sequences are always finite.
>>
>> Can someone please enlighten me?
>> --
>> Christian Tismer-Sperling :^) tis...@stackless.com
>> <mailto:tis...@stackless.com>
>> Software Consulting : http://www.stackless.com/
>> Karl-Liebknecht-Str. 121 : http://pyside.org
>> 14482 Potsdam : GPG key -> 0xE7301150FB7BEE0E
>> phone +49 173 24 18 776 fax +49 (30)
>> 700143-0023
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org <mailto:Python-Dev@python.org>
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>>
>>
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com
>>
>
>
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com
>
--
Christian Tismer-Sperling:^) tis...@stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : http://pyside.org
14482 Potsdam: GPG key -> 0xE7301150FB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 485 isclose() implementation review requested
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512
On 2015-05-18 01:02, Chris Barker wrote:
> * Is there a better way to create a False or True than::
>
> PyBool_FromLong(0) and PyBool_FromLong(1)
You can use the macros Py_RETURN_TRUE and Py_RETURN_FALSE instead of
return PyBool_FromLong(0).
> * Style question: should I put brackets in an if clause that has
> only one line?::
>
> if (some_check) { just_this_one_expression }
I prefer the extra brackets because they make the code more explicit.
It's really a matter of taste.
> * I can't find docs for PyDoc_STRVAR: but it looks like it should
> use it -- how?
PyDoc_STRVAR(functionname_doc,
"isclose(foo) -> bool\n\
\n\
long doc string.");
> * I'm getting a warning in my PyMethodDef clause::
>
> static PyMethodDef IsCloseMethods[] = { {"isclose", isclose_c,
> METH_VARARGS | METH_KEYWORDS, "determine if two floating point
> numbers are close"}, {NULL, NULL, 0, NULL}/* Sentinel */
> };
You have to type cast the function pointer to a PyCFunction here:
(PyCFunction)isclose_c
The type cast is required for KEYWORD functions and NOARGS functions.
Christian
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 485 isclose() implementation review requested
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 2015-05-18 17:49, Chris Barker wrote: > Thanks Cristian, that clears up a couple things -- got it > compiling without warning. > > But I also discovered that I must have not pushed the latest copy > yesterday. > > It's on a machine at home -- I'll push it tonight. But the copy on > gitHub now is mostly good -- I think the only changes are handling > the docsstrings better and some more/better tests. You're welcome! Does your latest patch handle NaN, too? I only noticed infinity checks but no explicit check for NaN. Christian -BEGIN PGP SIGNATURE- iQEcBAEBCgAGBQJVWgyrAAoJEIZoUkkhLbaJhBsH/3gH7Z1+otWwR6hIYjNU4OjK xjPmGeypisU6UDxfQa+lIHg1rEmyxlSbkXtn2DYysw9CMentK/XclF8GzWAA/ySV m0UE4+hzcWB7fsOLgbCxQKfNTEM/jp7D2z3qIZTknEecHENx552AaEfTXRTWQKHK QLh0sLA3QrOzUkOf+EXJQHlYvxf+F71PVyfOX8/m3XHhaQrpb70AGktsUPRDN3yc blY6SSQoV1uhw+/crqz34BoPGipAkZdq9abyz4Ja0adC8hT++7rbVldFdsrDIPdQ MX30atV+ZQ2Mb5NqJkmEjCKF5uXvwvlP8ijgz5nZKZ9db+9Z8YS0/e7UrPb85uM= =7N7z -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.7 is here until 2020, please don't call it a waste.
On 2015-05-29 23:14, Gregory P. Smith wrote: > > On Fri, May 29, 2015 at 12:24 AM Nick Coghlan <mailto:ncogh...@gmail.com>> wrote: > > > On 29 May 2015 11:01 am, "Victor Stinner" <mailto:victor.stin...@gmail.com>> wrote: > > > > Why not continue to enhance Python 3 instead of wasting our time with > > Python 2? We have limited resources in term of developers to maintain > > Python. > > > > (I'm not talking about fixing *bugs* in Python 2 which is fine > with me.) > > I'm actually OK with volunteers deciding that even fixing bugs in > 2.7 isn't inherently rewarding enough for them to be willing to do > it for free on their own time. > > > That is 100% okay. > > What is not okay is for python-dev representatives to respond to users > (in any list/forum/channel) reporting bugs in 2.7 or asking if a fix in > 3 can be backported to 2.7 with things akin to "just use Python 3" or > "sorry, 2.7 is critical fixes only. move to python 3 already." This is > actively driving our largest users away. I bring this up because a user > was bemoaning how useless they feel python core devs are because of this > attitude recently. Leading to feelings of wishing to just abandon > CPython if not Python all together. > > I'm sure I have even made some of those responses myself (sorry!). My > point here is: know it. recognize it. don't do it anymore. It harms the > community. > > A correct and accurate response to desires to make non-api-breaking > changes in 2.7 is "Patches that do not change any APIs for 2.7 are > welcome in the issue tracker." possibly including "I don't have the > bandwidth to review 2.7 changes, find someone on python-dev to review > and champion this for you if you need it." Finding someone may not > always be easy. But at least is still the "patches welcome" attitude and > suggests that the work can be done if someone is willing to do it. Lets > make a concerted effort to not be hostile and against it by default. > > Ex: Is someone with a python application that is a million of lines > supposed to have everyone involved in that drop the productive work they > are doing and spend that porting their existing application to python 3 > because we have so far failed to provide the tools to make that > migration easy? No. Empathize with our community. Feel their pain. > (and everyone who is working on tools to aid the transition: keep doing > that! Our users are gonna need it unless we don't want them as users > anymore.) > > We committed to supporting 2.7 until 2020 in 2014 per > https://hg.python.org/peps/rev/76d43e52d978. That means backports of > important bug or performance fixes should at least be allowed on the > table, even if hairy, even if you won't work on them yourselves on a > volunteer basis. This is the first long term support release of Python > ever. This is what LTS means. LTS could /also/ stand for Learn To > Support... Over the last years I have changed my mind a bit, too. For Python 2.7 LTS I welcome performance improving patches as well as security improvements (SSL module) and build related fixes. For performance patches we have to consider our responsibility for the environment. Every improvement means more speed and less power consumption. Python runs of hundreds of thousands of machines in the cloud. Python 2.7 will be used for at least half a decade, probably longer. Servers can be replaced with faster machines later and less fossil fuel must be burned to produce power. Let's keep Python green! :) Thanks to Benjamin, the patch has already landed. Antoine's improved GIL may be another improvement for Python 2.7. Servers are getting more cores every year. The new GIL helps to scale multiple CPU bound threads on machines with more cores, e.g. http://www.dabeaz.com/python/NewGIL.pdf Christian ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] 2.7 is here until 2020, please don't call it a waste.
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 2015-05-30 14:03, Antoine Pitrou wrote: > No, it's up to the proponent to prove that the effect exists, with > a magnitude large enough to make any interesting difference. That's > part of the process when suggesting a change. > > If it doesn't, or if it's entirely cosmetical, it may be an > important part of Christian's lifestyle (as are many individual > practices, including religious, militant, dietetic...), but it > certainly shouldn't brought up here. We don't want everyone trying > to inject their beliefs in the maintenance process. Antoine, now your are putting it over the top. You make it sound like I'm some crazy environmentalist or eco-warrior. Well, I'm not. I merely keep the environment in mind. Yes, I have a modern, power saving washing machine and LED lights at home. Mostly because they save money in the long run (Germany's electricity prices are high). That was also the point behind my comment. Increased performance result in better efficiency which lead to better utilization of hardware and less power consumption. Companies are interested in better efficiency, because they have to pay less for hardware, power and cooling. The obvious benefits for our environment are a side effect. A smaller CO2 foot print is not my main concern. But I wanted to bring it up anyway. For some it is an small additional motivator for performance improvements. For others it could be a marketing instrument. In Germany ads are full of crazy 'green' slogans. Christian -BEGIN PGP SIGNATURE- iQEcBAEBCgAGBQJVabPTAAoJEIZoUkkhLbaJTF0H+wb8ciikP762qc8u586H2AjV 2xV9AAamI1Z6RwlvKRM7YHVk48coYIKk9WQ6DZODNlVSIhnijexII1dai91gbQvy jEVkLK2P6/C7I4gz7Fp0/SoCwkpGCev2CiSJUhIoE4oIw+Mm4BRASpf5hn4n+pRI yqXixYf7h+QWHgN0FRU3GU8RxNYRe65zB/3YeDUhKLQdkf8Gq4NVX7rlTx1gvZrq DbaGjKtkT8uec6hnvZcXwWVODYW10VHTonhlV3ff0sReXw94sXOeQwQ3n+7uwKAb sqvy11k0r6JejNGFxJqfMyXH557LP5ucc2g9+J8M2Sw4SOs7L6E+caaX89FY754= =soyL -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backporting the 3.5+ Windows build project files to 2.7
Hi Zack, On 23.06.15 06:42, Zachary Ware wrote: > On Mon, Jun 22, 2015 at 10:37 PM, Nick Coghlan wrote: >> I'd suggest explicitly reaching out to the Stackless folks to get >> their feedback. As I believe the switched to a newer compiler and VC >> runtime for Windows a while back, I suspect it will make their lives >> easier rather than harder, but it's still worth asking for their >> input. > From a glance at the stackless repo, it looks like it still uses > VS2008's project files (which pretty much means using MSVC 9), but I > could have easily missed something. > > Christian, what say you? Would having the project files from 3.5 > backported to 2.7 (but still using MSVC 9) be positive, negative, or > indifferent for Stackless? > I am very positive about your efforts. We wanted to create a Stackless 2.8 version end of 2013 with support for newer compilers, to begin with. After a while, we stopped this, and I left the branch in a private, unmaintained repository. https://bitbucket.org/stackless-dev/stackless-private/branch/2.8-slp Maybe you can make use of this, use it as you like it. You will get an invitation. cheers - Chris -- Christian Tismer :^) tis...@stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://www.pydica.net/ 14482 Potsdam: GPG key -> 0xFB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
