[Python-Dev] Status of Python buildbots
Hi,
I spent last week working on fixing buildbots:
https://www.python.org/dev/buildbot/
It should now be able again to rely on them to detect regressions. Changes:
* Fix various bugs (I don't even recall which ones)
* Fix multiple random failures
* Fix dozen of warnings
I also enhanced the configuration of buildbots to search for warnings
in the output of the "tests" step (Python test suite). I use a regular
expression to search for patterns:
https://github.com/python/buildmaster-config/blob/master/master/master.cfg#L93-L125
If a warning is found, the buildbot now becomes orange. As I wrote, I
fixed dozens of warnings, but there are still a few ones (and many
unstable tests failing randomly).
The orange state notify for example when a test failed once, but
passed when it was run again in verbose mode. It helps to detect
unstable tests, but also real bugs.
The "Warning -- xxx was modified by test_xxx" warning now always log
the old and new value to ease debugging these warnings. Example on
Python 2.7:
Warning -- files was modified by test_sax
Before: []
After: ['@test_352_tmp']
TODO:
* Create a mailing list and send an email to this list when the state
of a buildbot changes (only from green to orange or red? similar to
what we have on #python-dev IRC channel). I suggest to start with a
whitelist of buildbots known to be stable... hum, like the list of
"stable buildbots" ;-) Let's start with the 3.x branch, and later
enable it on more branches. The mailing list should help to coordinate
when multiple developers work on buildbot issues in parallel.
* Fix FreeBSD buildbots which fail to compile Python because of
http://bugs.python.org/issue23404 I proposed a change to not
regenerate generated files based on file modification time anymore,
but require an explicit action ("make regen-all"). This change solves
many practical issues.
* Fix remaining warnings.
* Fix unstable tests.
* Add more warnings! I proposed to emit ResourceWarning in
multiprocessing.Queue and concurrent.futures executors ;-)
http://bugs.python.org/issue30171 and
http://bugs.python.org/issue30244
* Add more buildbots! Zachary Ware proposed to add a buildbot running
"regen-all" to check that generated files are up to date.
* Repeat ;-)
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of Python buildbots
On Wed, May 3, 2017 at 10:22 AM, Victor Stinner wrote: > Hi, > > I spent last week working on fixing buildbots: > >https://www.python.org/dev/buildbot/ > Thanks! > * Add more buildbots! Zachary Ware proposed to add a buildbot running > "regen-all" to check that generated files are up to date. > > * Repeat ;-) I have been saying for several months now that I would like to set up a Cygwin buildbot--an important step in making that platform supportable again. I now have the infrastructure available to do so (Windows VM on an OpenStack infrastructure at my university). I wanted to wait until the tests on Cygwin were more stable, but since you allow unstable buildbots I could include it among them for now. Is the buildbot setup documentation on the wiki page still accurate? Thanks, Erik ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Removing Python/thread_foobar.h
Hi, python-dev. I'm reading the part of threading feature to replace TLS API with TSS API (PEP 539). Python/thread_foobar.h hasn't be included to Python/thread.c (its include directive is comment out) and anywhere. This looks like dead code and we want to delete it, does anyone have a reason we should keep it? Regards, Masayuki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing Python/thread_foobar.h
Good catch! It's actually a template for writing a threading adaptation for new platforms. But it's probably hopelessly out of date so unless someone wants to adopt it and keep it up to date, we can delete it. (Also, new platforms with unique threading APIs don't come along at the rate they did in the 1990s. :-) If there's more discussion, it probably belongs on bugs.python.org. --Guido On Wed, May 3, 2017 at 1:59 AM, Masayuki YAMAMOTO wrote: > Hi, python-dev. > > I'm reading the part of threading feature to replace TLS API with TSS API > (PEP 539). Python/thread_foobar.h hasn't be included to Python/thread.c > (its include directive is comment out) and anywhere. This looks like dead > code and we want to delete it, does anyone have a reason we should keep it? > > Regards, > Masayuki > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing Python/thread_foobar.h
You can remove thread_foobar.h. I don't think that anyone still wants to use this template. The other thread_*.h files can be used as template as well. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of Python buildbots
On Wed, 3 May 2017 at 01:24 Victor Stinner wrote:
> Hi,
>
> I spent last week working on fixing buildbots:
>
>https://www.python.org/dev/buildbot/
>
> It should now be able again to rely on them to detect regressions.
Thanks, Victor!
> Changes:
>
> * Fix various bugs (I don't even recall which ones)
> * Fix multiple random failures
> * Fix dozen of warnings
>
> I also enhanced the configuration of buildbots to search for warnings
> in the output of the "tests" step (Python test suite). I use a regular
> expression to search for patterns:
>
>
> https://github.com/python/buildmaster-config/blob/master/master/master.cfg#L93-L125
>
> If a warning is found, the buildbot now becomes orange. As I wrote, I
> fixed dozens of warnings, but there are still a few ones (and many
> unstable tests failing randomly).
>
> The orange state notify for example when a test failed once, but
> passed when it was run again in verbose mode. It helps to detect
> unstable tests, but also real bugs.
>
>
> The "Warning -- xxx was modified by test_xxx" warning now always log
> the old and new value to ease debugging these warnings. Example on
> Python 2.7:
>
> Warning -- files was modified by test_sax
> Before: []
> After: ['@test_352_tmp']
>
>
> TODO:
>
> * Create a mailing list and send an email to this list when the state
> of a buildbot changes (only from green to orange or red? similar to
> what we have on #python-dev IRC channel). I suggest to start with a
> whitelist of buildbots known to be stable... hum, like the list of
> "stable buildbots" ;-) Let's start with the 3.x branch, and later
> enable it on more branches. The mailing list should help to coordinate
> when multiple developers work on buildbot issues in parallel.
>
SGTM.
-Brett
>
> * Fix FreeBSD buildbots which fail to compile Python because of
> http://bugs.python.org/issue23404 I proposed a change to not
> regenerate generated files based on file modification time anymore,
> but require an explicit action ("make regen-all"). This change solves
> many practical issues.
>
> * Fix remaining warnings.
>
> * Fix unstable tests.
>
> * Add more warnings! I proposed to emit ResourceWarning in
> multiprocessing.Queue and concurrent.futures executors ;-)
> http://bugs.python.org/issue30171 and
> http://bugs.python.org/issue30244
>
> * Add more buildbots! Zachary Ware proposed to add a buildbot running
> "regen-all" to check that generated files are up to date.
>
> * Repeat ;-)
>
> Victor
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Status of Python buildbots
On Wed, May 3, 2017 at 4:16 AM, Erik Bray wrote: > I have been saying for several months now that I would like to set up > a Cygwin buildbot--an important step in making that platform > supportable again. I now have the infrastructure available to do so > (Windows VM on an OpenStack infrastructure at my university). I > wanted to wait until the tests on Cygwin were more stable, but since > you allow unstable buildbots I could include it among them for now. That would be fine :) > Is the buildbot setup documentation on the wiki page still accurate? Probably not; the devguide should be more up-to-date: https://docs.python.org/devguide/buildslave.html I don't know what unique challenges you may run into setting up a Cygwin worker, but please submit PRs against that page if you find anything to be lacking. -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
My allergies have hit me hard so I'm not thinking at full capacity, but did we ever decide if supporting os.PathLike in the stdlib was viewed as an enhancement or bugfix? Specifically I'm thinking of https://bugs.python.org/issue30218 for adding support to shutil.unpack_archive() and whether it should be backported to 3.6. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On Wed, May 3, 2017 at 9:15 PM, Brett Cannon wrote: > My allergies have hit me hard so I'm not thinking at full capacity, but did > we ever decide if supporting os.PathLike in the stdlib was viewed as an > enhancement or bugfix? Specifically I'm thinking of > https://bugs.python.org/issue30218 for adding support to > shutil.unpack_archive() and whether it should be backported to 3.6. > Haha, not sure if you have actual allergy, or if you just remembered the fspath discussions nightmare ;). You're obviously not asking me, but I'd say anything in the 3.6 stdlib that takes a path, but not os.PathLike, should count as a bug. Otherwise there will just be surprises with code that works on 3.7 but not on the famously fspath-enabled 3.6. I'm not even 100% sure that backporting to 3.5 would be (or would have been) a bad idea; 3.5 will stick around for quite some time still. -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On 5/3/2017 2:15 PM, Brett Cannon wrote: My allergies have hit me hard so I'm not thinking at full capacity, but did we ever decide if supporting os.PathLike in the stdlib was viewed as an enhancement or bugfix? Specifically I'm thinking of https://bugs.python.org/issue30218 for adding support to shutil.unpack_archive() and whether it should be backported to 3.6. On the face of it, that particular issue looks like an enhancement that should have gone into 3.6 (if ever), but did not. I notice that https://www.python.org/dev/peps/pep-0519/#implementation did not include "Update shutil", so it was not done, at least not completely. Was shutil updated at all? Is unpack_archive the only shutil function not updated? If so, I could see the omission as a bug. If the patch for 30218 were applied in 3.6, would the doc https://docs.python.org/3/library/shutil.html#shutil.unpack_archive need to be changed, with a note "Added in 3.6.2: filename can be any pathlike object"? If so, it is an enhancement. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
If you start to backport support for the fspath protocol, be prepared to have to backport it *many* places. I expect that slowly in the near future, many functions will be patched to support the fspath protocol. I suggest to only do that in master. It's not that hard to cast manually to a string: it's just str(path), no? Victor 2017-05-03 20:15 GMT+02:00 Brett Cannon : > My allergies have hit me hard so I'm not thinking at full capacity, but did > we ever decide if supporting os.PathLike in the stdlib was viewed as an > enhancement or bugfix? Specifically I'm thinking of > https://bugs.python.org/issue30218 for adding support to > shutil.unpack_archive() and whether it should be backported to 3.6. > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On Thu, May 4, 2017 at 1:07 AM, Terry Reedy wrote: > On 5/3/2017 2:15 PM, Brett Cannon wrote: >> >> My allergies have hit me hard so I'm not thinking at full capacity, but >> did we ever decide if supporting os.PathLike in the stdlib was viewed as an >> enhancement or bugfix? Specifically I'm thinking of >> https://bugs.python.org/issue30218 for adding support to >> shutil.unpack_archive() and whether it should be backported to 3.6. > > > On the face of it, that particular issue looks like an enhancement that > should have gone into 3.6 Agreed. > (if ever), but did not. I notice that > https://www.python.org/dev/peps/pep-0519/#implementation > did not include "Update shutil", so it was not done, at least not > completely. Shutil was among the most important to be updated, IMO. I had made some sort of list of affected modules elsewhere [1]: ntpath, posixpath, os.scandir, os.[other stuff], DirEntry (tempted to say os.DirEntry, but that is not true), shutil.[stuff], (io.)open, fileinput, filecmp, zipfile, tarfile, tempfile (for the 'dir' keyword arguments), maybe even glob and fnmatch (are the patterns paths?) It looks like what made it to PEP519 was mainly this: "It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary)." > Was shutil updated at all? Is unpack_archive the only shutil function not > updated? If so, I could see the omission as a bug. > > If the patch for 30218 were applied in 3.6, would the doc > https://docs.python.org/3/library/shutil.html#shutil.unpack_archive > need to be changed, with a note "Added in 3.6.2: filename can be any > pathlike object"? If so, it is an enhancement. Regardless of bugfix vs enhancement semantics, that seems like a good thing to do. -- Koos [1] e.g. in this thread somewhere: https://mail.python.org/pipermail/python-ideas/2016-April/039827.html -- + Koos Zevenhoven + http://twitter.com/k7hoven + ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Is adding support for os.PathLike an enhancement or bugfix?
On 5/3/2017 7:13 PM, Koos Zevenhoven wrote: On Thu, May 4, 2017 at 1:07 AM, Terry Reedy wrote: On 5/3/2017 2:15 PM, Brett Cannon wrote: My allergies have hit me hard so I'm not thinking at full capacity, but did we ever decide if supporting os.PathLike in the stdlib was viewed as an enhancement or bugfix? Specifically I'm thinking of https://bugs.python.org/issue30218 for adding support to shutil.unpack_archive() and whether it should be backported to 3.6. On the face of it, that particular issue looks like an enhancement that should have gone into 3.6 Agreed. (if ever), but did not. I notice that https://www.python.org/dev/peps/pep-0519/#implementation did not include "Update shutil", so it was not done, at least not completely. Shutil was among the most important to be updated, IMO. I had made some sort of list of affected modules elsewhere [1]: ntpath, posixpath, os.scandir, os.[other stuff], DirEntry (tempted to say os.DirEntry, but that is not true), shutil.[stuff], (io.)open, fileinput, filecmp, zipfile, tarfile, tempfile (for the 'dir' keyword arguments), maybe even glob and fnmatch (are the patterns paths?) What did not get done for 3.6 should be proposed for 3.7. It looks like what made it to PEP519 was mainly this: "It is expected that most APIs in Python's standard library that currently accept a file system path will be updated appropriately to accept path objects (whether that requires code or simply an update to documentation will vary)." To me that says that not being updated (yet) is not a bug. Was shutil updated at all? Is unpack_archive the only shutil function not updated? If so, I could see the omission as a bug. If the patch for 30218 were applied in 3.6, would the doc https://docs.python.org/3/library/shutil.html#shutil.unpack_archive need to be changed, with a note "Added in 3.6.2: filename can be any pathlike object"? If so, it is an enhancement. Regardless of bugfix vs enhancement semantics, that seems like a good thing to do. Enhancing public APIs in normal (non-provisional) modules in bugfix releases has turned out to be a bad thing to do. Hence the policy to not do that. The few exceptions have been necessary to fix a bug that needed to be fixed, and could not reasonably be fixed otherwise. [1] e.g. in this thread somewhere: https://mail.python.org/pipermail/python-ideas/2016-April/039827.html -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8 based locale
Hi, Nick and all core devs who are interested in this PEP. I'm reviewing PEP 538 and I want to accept it in this month. It will reduces much UnicodeError pains which server-side OPs facing. Thank you Nick for working on this PEP. If you have something worrying about this PEP, please post a comment soon. If you don't have enough time to read entire this PEP, feel free to ask a question about you're worrying. Here is my comments: > > Relationship with other PEPs > > > This PEP shares a common problem statement with PEP 540 (improving Python > 3's > behaviour in the default C locale), but diverges markedly in the proposed > solution: > > * PEP 540 proposes to entirely decouple CPython's default text encoding from > the C locale system in that case, allowing text handling inconsistencies to > arise between CPython and other locale-aware components running in the same > process and in subprocesses. This approach aims to make CPython behave less > like a locale-aware application, and more like locale-independent language > runtimes like the JVM, .NET CLR, Go, Node.js, and Rust https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html says: > Every instance of the Java virtual machine has a default charset, which may > or may not be one of the standard charsets. The default charset is determined > during virtual-machine startup and typically depends upon the locale and > charset being used by the underlying operating system. I don't know about .NET runtime on Unix much. (mono and .NET Core). "Go, Node.js and Rust" seems enough examples. > New build-time configuration options > > > While both of the above behaviours would be enabled by default, they would > also have new associated configuration options and preprocessor definitions > for the benefit of redistributors that want to override those default > settings. > > The locale coercion behaviour would be controlled by the flag > ``--with[out]-c-locale-coercion``, which would set the > ``PY_COERCE_C_LOCALE`` > preprocessor definition. > > The locale warning behaviour would be controlled by the flag > ``--with[out]-c-locale-warning``, which would set the > ``PY_WARN_ON_C_LOCALE`` > preprocessor definition. "locale warning" means warning printed when C locale is used, am I right? As my understanding, "locale warning" is shown in these cases (all cases implies under C locale and PYTHONUTF8 is not enabled). a. C locale is used and locale coercion is disabled by ``--without-c-locale-coercion`` configure option. b. locale coercion is failed since there is none of C.UTF-8, C.utf8, nor UTF-8 locale. c. Python is embedded. locale coercion can't be used in this case. In case of (b), while warning about C locale is not shown, warning about coercion is still shown. So when people don't want to see warning under C locale and there is no (C.UTF-8, C.utf8, UTF-8) locales, there are three ways: * Set PYTHONUTF=1 (if PEP 540 is accepted) * Set PYTHONCOERCECLOCALE=0. * Use both of ``--without-c-locale-coercion`` and ``--without-c-locale-warning`` configure options. Is my understanding right? BTW, I prefer PEP 540 provides ``--with-utf8mode`` option which enables UTF-8 mode by default. And if it is added, there are too few use cases for ``--without-c-locale-warning``. There are some use cases people want to use UTF-8 by default in system wide. (e.g. container, webserver in Cent OS, etc...) On the other hand, most of C locale usage are "per application" basis, rather than "system wide." configure option is not suitable for such per application setting, off course. But I don't propose removing the option from PEP 538. We can discuss about reducing configure options later. > > On platforms where they would have no effect (e.g. Mac OS X, iOS, Android, > Windows) these preprocessor variables would always be undefined. > Why ``--with[out]-c-locale-coercion`` have no effect on macOS, iOS and Android? On Android, locale coercion fixes readline. Do you mean locale coercion happen always regardless this configuration option? On macOS, ``LC_ALL=C python`` doesn't make Python's stdio to ``ascii:surrogateescape``? Even so, locale coercion may fix libraries like readline, curses. While C locale is less common on macOS, I don't understand any reason to disable it on macOS. I know almost nothing about iOS, but it's similar to Android or macOS in my expectation. > Improving the handling of the C locale > -- > ... > locale settings for locale-aware operations. Both the JVM and the .NET CLR > use UTF-16-LE as their primary encoding for passing text between applications > and the underlying platform. JVM and .NET examples are misleading again. They just use UTF-16-LE for syscall on Windows, like Python. I don't know about them much, but I believe they don't use UTF-16 for system encoding on Linux. > Defaulting to "surrogateescape" e
