Re: [Python-Dev] Python startup time
On 21 July 2017 at 23:53, David Mertz wrote: > I would guess that Windows users don't tend to run lots of command line > tools where startup time dominates, as *nix users do. Well, in the sense that many Windows users don't use the command line at all, this is true. However, startup time is a definite problem for Windows users who *do* use the command line, because process creation cost is a lot higher than on Unix, so starting new commands is *already* costly, and therefore minimising additional overhead is crucial. It's a bit of a chicken and egg problem - Windows users avoid excessive command line program invocation because startup time is high, so no-one optimises startup time because Windows users don't use short-lived command line programs. But I'm seeing a trend away from that - more and more Windows tools these days seem to be comfortable spawning subprocesses. I don't know what prompted that trend. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python startup time
> -Original Message- > From: Python-Dev [mailto:python-dev-bounces+tritium- > list=sdamon@python.org] On Behalf Of Paul Moore > Sent: Saturday, July 22, 2017 4:14 AM > To: David Mertz > Cc: Barry Warsaw ; Python-Dev d...@python.org> > Subject: Re: [Python-Dev] Python startup time > It's a bit of a chicken and egg problem - Windows users avoid > excessive command line program invocation because startup time is > high, so no-one optimises startup time because Windows users don't use > short-lived command line programs. But I'm seeing a trend away from > that - more and more Windows tools these days seem to be comfortable > spawning subprocesses. I don't know what prompted that trend. The programs I see that are comfortable spawning processes willy-nilly on windows are mostly .net, which has a lot of the runtime assemblies cached by the OS in the GAC - if you are spawning a second processes of yourself, or something that uses the same libraries as you, the compile step on those can be skipped. Unless you are talking about python/non-.NET programs, in which case, I have no answer. > Paul > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > list%40sdamon.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python startup time
I believe the trend is due to language like Python and Node.js, most of which aggressively discourage threading (more from the broader community than the core languages, but I see a lot of apps using these now), and also the higher reliability afforded by out-of-process tasks (that is, one crash doesn’t kill the entire app – e.g browser tabs). Optimizing startup time is incredibly valuable, and having tried it a few times I believe that the import system (in essence, stat calls) is the biggest culprit. The tens of ms prior to the first user import can’t really go anywhere. Cheers, Steve Top-posted from my Windows phone From: Alex Walters Sent: Saturday, July 22, 2017 1:39 Cc: 'Python-Dev' Subject: Re: [Python-Dev] Python startup time > -Original Message- > From: Python-Dev [mailto:python-dev-bounces+tritium- > list=sdamon@python.org] On Behalf Of Paul Moore > Sent: Saturday, July 22, 2017 4:14 AM > To: David Mertz > Cc: Barry Warsaw ; Python-Dev d...@python.org> > Subject: Re: [Python-Dev] Python startup time > It's a bit of a chicken and egg problem - Windows users avoid > excessive command line program invocation because startup time is > high, so no-one optimises startup time because Windows users don't use > short-lived command line programs. But I'm seeing a trend away from > that - more and more Windows tools these days seem to be comfortable > spawning subprocesses. I don't know what prompted that trend. The programs I see that are comfortable spawning processes willy-nilly on windows are mostly .net, which has a lot of the runtime assemblies cached by the OS in the GAC - if you are spawning a second processes of yourself, or something that uses the same libraries as you, the compile step on those can be skipped. Unless you are talking about python/non-.NET programs, in which case, I have no answer. > Paul > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > list%40sdamon.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help to fix urllib(.parse) vulnerabilities
Le 22 juil. 2017 8:04 AM, "Serhiy Storchaka" a écrit : I think the only reliable way of fixing the vulnerability is rejecting or escaping (as specified in RFC 2640) CR and LF inside sent lines. Adding the support of RFC 2640 is a new feature and can be added only in 3.7. And this feature should be optional since not all servers support RFC 2640. https://github.com/python/cpython/pull/1214 does the right thing. In that case, I suggest to reject newlines in ftplib, and maybe add an opt-in option to escape newlines. Java just rejected newlines, no? Or does Java allows to escape them? Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help to fix urllib(.parse) vulnerabilities
On Sat, Jul 22, 2017 at 6:38 PM, Victor Stinner wrote: > Le 22 juil. 2017 8:04 AM, "Serhiy Storchaka" a > écrit : > > I think the only reliable way of fixing the vulnerability is rejecting or > escaping (as specified in RFC 2640) CR and LF inside sent lines. Adding the > support of RFC 2640 is a new feature and can be added only in 3.7. And this > feature should be optional since not all servers support RFC 2640. > https://github.com/python/cpython/pull/1214 does the right thing. > > > In that case, I suggest to reject newlines in ftplib, and maybe add an > opt-in option to escape newlines. > > Java just rejected newlines, no? Or does Java allows to escape them? > > Victor > > OK, let's just reject \n then and be done with it. It's a rare use case after all. Java just rejects \n for all commands and does not support escaping (aka RFC 2640). -- Giampaolo - http://grodola.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python startup time
On Sat, Jul 22, 2017, 07:22 Steve Dower, wrote: > I believe the trend is due to language like Python and Node.js, most of > which aggressively discourage threading (more from the broader community > than the core languages, but I see a lot of apps using these now), and also > the higher reliability afforded by out-of-process tasks (that is, one crash > doesn’t kill the entire app – e.g browser tabs). > > > > Optimizing startup time is incredibly valuable, and having tried it a few > times I believe that the import system (in essence, stat calls) is the > biggest culprit. The tens of ms prior to the first user import can’t really > go anywhere. > Stat calls in the import system were optimized in importlib a while back to be cached in finders so at this point you will have to remove a stat call to lower that cost or cache more which goes into breaking abstractions or designing new APIs. -brett > > Cheers, > > Steve > > > > Top-posted from my Windows phone > > > > *From: *Alex Walters > *Sent: *Saturday, July 22, 2017 1:39 > *Cc: *'Python-Dev' > > > *Subject: *Re: [Python-Dev] Python startup time > > > > > -Original Message- > > > From: Python-Dev [mailto:python-dev-bounces+tritium- > > > list=sdamon@python.org] On Behalf Of Paul Moore > > > Sent: Saturday, July 22, 2017 4:14 AM > > > To: David Mertz > > > Cc: Barry Warsaw ; Python-Dev > > d...@python.org> > > > Subject: Re: [Python-Dev] Python startup time > > > > > > > It's a bit of a chicken and egg problem - Windows users avoid > > > excessive command line program invocation because startup time is > > > high, so no-one optimises startup time because Windows users don't use > > > short-lived command line programs. But I'm seeing a trend away from > > > that - more and more Windows tools these days seem to be comfortable > > > spawning subprocesses. I don't know what prompted that trend. > > > > The programs I see that are comfortable spawning processes willy-nilly on > > windows are mostly .net, which has a lot of the runtime assemblies cached > by > > the OS in the GAC - if you are spawning a second processes of yourself, or > > something that uses the same libraries as you, the compile step on those > can > > be skipped. Unless you are talking about python/non-.NET programs, in > which > > case, I have no answer. > > > Paul > > > ___ > > > Python-Dev mailing list > > > Python-Dev@python.org > > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > > > list%40sdamon.com > > > > ___ > > Python-Dev mailing list > > Python-Dev@python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org > > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help to fix urllib(.parse) vulnerabilities
On Sat, Jul 22, 2017 at 7:10 PM, Giampaolo Rodola' wrote: > > > On Sat, Jul 22, 2017 at 6:38 PM, Victor Stinner > wrote: > >> Le 22 juil. 2017 8:04 AM, "Serhiy Storchaka" a >> écrit : >> >> I think the only reliable way of fixing the vulnerability is rejecting or >> escaping (as specified in RFC 2640) CR and LF inside sent lines. Adding the >> support of RFC 2640 is a new feature and can be added only in 3.7. And this >> feature should be optional since not all servers support RFC 2640. >> https://github.com/python/cpython/pull/1214 does the right thing. >> >> >> In that case, I suggest to reject newlines in ftplib, and maybe add an >> opt-in option to escape newlines. >> >> Java just rejected newlines, no? Or does Java allows to escape them? >> >> Victor >> >> > OK, let's just reject \n then and be done with it. It's a rare use case > after all. > Java just rejects \n for all commands and does not support escaping (aka > RFC 2640). > I've just merged the PR. There's the question whether to backport this to older versions, considering there's a small chance this may break some code/apps, but considering the chance is small and this a security fix I'd probably be +0.5 for backporting it (2.7 + 3.x - not sure up 'till when). -- Giampaolo - http://grodola.blogspot.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Need help to fix urllib(.parse) vulnerabilities
I consider that it is a security vulneraibility and so should be fixed in all supported branches including 3.3 and 3.4. If someone is blocked for a legit usecase, an old Python version can be used until we decide how to handle it. I concur with you, I don't think that anyone uses filenames containing newlines on FTP. FTP protocol is text based and uses newlines as the command separator. I expect a lot of not fun issues if someone uses such filename on legit files. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python startup time
“Stat calls in the import system were optimized in importlib a while back” Yes, I’m aware of that, which is why I don’t have any specific suggestions off-hand. But given the differences in file systems between Windows and other OSs, it wouldn’t surprise me if there were a more optimal approach for NTFS to amortize calls better. Perhaps not, but it is still the most expensive part of startup that we have any ability to change, so it’s worth investigating. Cheers, Steve Top-posted from my Windows phone From: Brett Cannon Sent: Saturday, July 22, 2017 10:18 To: Steve Dower; Alex Walters Cc: Python-Dev Subject: Re: [Python-Dev] Python startup time On Sat, Jul 22, 2017, 07:22 Steve Dower, wrote: I believe the trend is due to language like Python and Node.js, most of which aggressively discourage threading (more from the broader community than the core languages, but I see a lot of apps using these now), and also the higher reliability afforded by out-of-process tasks (that is, one crash doesn’t kill the entire app – e.g browser tabs). Optimizing startup time is incredibly valuable, and having tried it a few times I believe that the import system (in essence, stat calls) is the biggest culprit. The tens of ms prior to the first user import can’t really go anywhere. Stat calls in the import system were optimized in importlib a while back to be cached in finders so at this point you will have to remove a stat call to lower that cost or cache more which goes into breaking abstractions or designing new APIs. -brett Cheers, Steve Top-posted from my Windows phone From: Alex Walters Sent: Saturday, July 22, 2017 1:39 Cc: 'Python-Dev' Subject: Re: [Python-Dev] Python startup time > -Original Message- > From: Python-Dev [mailto:python-dev-bounces+tritium- > list=sdamon@python.org] On Behalf Of Paul Moore > Sent: Saturday, July 22, 2017 4:14 AM > To: David Mertz > Cc: Barry Warsaw ; Python-Dev d...@python.org> > Subject: Re: [Python-Dev] Python startup time > It's a bit of a chicken and egg problem - Windows users avoid > excessive command line program invocation because startup time is > high, so no-one optimises startup time because Windows users don't use > short-lived command line programs. But I'm seeing a trend away from > that - more and more Windows tools these days seem to be comfortable > spawning subprocesses. I don't know what prompted that trend. The programs I see that are comfortable spawning processes willy-nilly on windows are mostly .net, which has a lot of the runtime assemblies cached by the OS in the GAC - if you are spawning a second processes of yourself, or something that uses the same libraries as you, the compile step on those can be skipped. Unless you are talking about python/non-.NET programs, in which case, I have no answer. > Paul > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > list%40sdamon.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
