Re: [Python-Dev] understanding ubuntu's patches to cpython

[Matthias Klose]

On 11/16/2014 11:23 PM, Buck Evan wrote:

This particular patch caused virtualenv issue #118 and a subsequent
workaround:
http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/trusty/python2.7/trusty/view/head:/debian/patches/distutils-install-layout.diff#L281


Although this patch sits on the trusty branch of cpython patckaging, in
point of fact on that line i have `return
os.path.join(get_config_var('LIBPL'), "Makefile")`. I see that other
patches in the same project also touch this line, but none of them have
this LIBPL code that I find in my actual installation.


I'm not sure what issue you have here.  Indeed, as Ned suggested, best thing 
would be to open a bug report in Launchpad [1].



Can any of you all help me understand what's going on here? I'm trying to
pinpoint exactly when/how the problematic code (.replace("/usr/local","/usr")
went away.


See the README.Debian.  In the past people installed packages into the system 
path using "sudo python setup.py --install", and breaking applications shipped 
with Ubuntu.  This avoids accidental installation of such packages. 
Unfortunately you can do this again with pip, because pip acting on the system 
python silently overwrites system installed python modules.


  Matthias

[1] https://bugs.launchpad.net/ubuntu/+source/python2.7/+filebug

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Can Python Be Built Without Distutils

[Matthias Klose]

On 01/23/2015 06:30 PM, Cyd Haselton wrote:
> Related to my earlier question regarding building Python on Android
> and an undefined reference to dlopen error...I have the following
> question:  Is it possible to build and install Python without having
> to build and install...or use...distutils?
> 
> Some background:
> I can build the python interpreter on my device, and I can build a
> bunch of modules.  The problem appears when make reaches the part
> where setup.py is used to build and import modules...specifically when
> setup.py attempts to import distutils.core.

you can do this using Setup.local. This works for me building additional
extensions as builtins.  It might require some tweaking to build everything.
Otoh, I would like to get rid off the setup.py altogether (/me ducks ...).

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] libffi embedded in CPython

[Matthias Klose]

On 03/11/2015 06:27 PM, Brett Cannon wrote:
> On Mon, Dec 22, 2014 at 4:49 PM Jim J. Jewett  wrote:
> 
>>
>>
>> On Thu, Dec 18, 2014, at 14:13, Maciej Fijalkowski wrote:
>>> ... http://bugs.python.org/issue23085 ...
>>> is there any reason any more for libffi being included in CPython?
>>
>>
>> Paul Moore wrote:
>>> Probably the easiest way of moving this forward would be for someone
>>> to identify the CPython-specific patches in the current version ...
>>
>> Christian Heimes wrote:
>>> That's easy. All patches are tracked in the diff file
>>> https://hg.python.org/cpython/file/3de678cd184d/Modules/_
>> ctypes/libffi.diff
>>
>> That (200+ lines) doesn't seem to have all the C changes, such as the
>> win64 sizeof changes from issue 11835.
>>
>> Besides http://bugs.python.org/issue23085, there is at least
>> http://bugs.python.org/issue22733
>> http://bugs.python.org/issue20160
>> http://bugs.python.org/issue11835
>>
>> which sort of drives home the point that making sure we have a
>> good merge isn't trivial, and this isn't an area where we should
>> just assume that tests will catch everything.  I don't think it
>> is just a quicky waiting on permission.
>>
>> I've no doubt that upstream libffi is better in many ways, but
>> those are ways people have already learned to live with.
>>
>> That said, I haven't seen any objections in principle, except
>> perhaps from Steve Dower in the issues.  (I *think* he was
>> just saying "not worth the time to me", but it was ambiguous.)
>>
>> I do believe that Christian or Maciej *could* sort things out well
>> enough; I have no insight into whether they have (or someone else
>> has) the time to actually do so.
>>
> 
> Did anyone ever step forward to do this? I'm a bit worried about the
> long-term viability of ctypes if we don't have a maintainer or at least
> someone making sure we are staying up-to-date with upstream libffi.

my plan was to merge the next libffi release, something along the lines which is
released with GCC 5. But this release isn't yet done.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Some data points for the "annual release cadence" concept

[Matthias Klose]

On 12.06.2018 18:23, Guido van Rossum wrote:

So, to summarize, we need something like six for C?


there is https://github.com/encukou/py3c
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Some data points for the "annual release cadence" concept

[Matthias Klose]

On 12.06.2018 14:38, Nick Coghlan wrote:

Since we last seriously discussed potential release cadence changes back in
2012 (with PEPs 407 and 413 [2,3]), that means JEE joins GCC switching to
an annual release cadence from GCC 5 back in 2015 (see [4]),


no, GCC is doing yearly releases since 2001.  In 2015 they changed the 
versioning schema, but it's still one major release per year.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Computed Goto dispatch for Python 2

[Matthias Klose]

On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote:
> Hi All,
> 
> This is Vamsi from Server Scripting Languages Optimization team at Intel 
> Corporation.
> 
> Would like to submit a request to enable the computed goto based dispatch in 
> Python 2.x (which happens to be enabled by default in Python 3 given its 
> performance benefits on a wide range of workloads). We talked about this 
> patch with Guido and he encouraged us to submit a request on Python-dev 
> (email conversation with Guido shown at the bottom of this email).
> 
> Attached is the computed goto patch (along with instructions to run) for 
> Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin  at 
> http://bugs.python.org/issue4753). We built and tested this patch for Python 
> 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP 
> CPU with 18 cores, hyper-threading off, turbo off).
> 
> Below is a summary of the performance we saw on the "grand unified python 
> benchmarks" suite (available at https://hg.python.org/benchmarks/). We made 3 
> rigorous runs of the following benchmarks. In each rigorous run, a benchmark 
> is run 100 times with and without the computed goto patch. Below we show the 
> average performance boost for the 3 rigorous runs.
> 
> Python 2.7.10 (original) vs Computed Goto performance
> Benchmark

-1

As Gregory pointed out, there are other options to build the interpreter, and we
are missing data how these compare with your patch.

I assume, you tested with the Intel compiler, so it would be good to see results
for other compilers as well (GCC, clang).  Please could you provide the data for
LTO and profile guided optimized builds (maybe combined too)?  I'm happy to work
with you on setting up these builds, but currently don't have the machine
resources to do so myself.

If the benefits show up for these configurations too, then I'm +/-0 on this 
patch.

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Keeping competitive with Go (was Re: Computed Goto dispatch for Python 2)

[Matthias Klose]

On 05/28/2015 06:13 PM, Barry Warsaw wrote:
> Go seems to be popular where I work.  It is replacing Python in a number of
> places, although Python (and especially Python 3) is still a very important
> part of our language toolbox.
> 
> There are several reasons why Go is gaining popularity.  Single-file
> executables is definitely a reason; it makes deployment very easy, even if it
> increases the maintenance burden (e.g. without shared libraries, you have
> multiple copies of things so when a security fix is required for one of those
> things you have to recompile the world).

And the very same place where you are working is investing in getting shared
libraries working for Go.  Single binaries may be popular for distributing end
user applications, but definitely not for distributing a core OS or a SDK.
Sorry, you didn't yet arrive in distro land ...

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Sorry folks, minor hiccup for Python 3.5.0rc1

[Matthias Klose]

On 08/11/2015 02:56 AM, Larry Hastings wrote:
> On 08/10/2015 05:55 PM, Larry Hastings wrote:
>> I yanked the tarballs off the release page as soon as I suspected something. 
>> I'm rebuilding the tarballs and the docs now.  If you grabbed the tarball as
>> soon as it appeared, it's slightly out of date, please re-grab.
> 
> p.s. I should have mentioned--the Mac and Windows builds should be fine.  
> They,
> unlike me, updated their tree ;-)

didn't see any follow-up message. are the source tarballs now fixed?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Profile Guided Optimization active by-default

[Matthias Klose]

The current pgo target just uses a very specific task to train for the feedback.
For my Debian/Ubuntu builds I'm using the testsuite minus some problematic tests
to train. Otoh I don't know if this is the best way to do it, however it gave
better results at some time in the past.  What I would like is a benchmark / a
mixture of benchmarks on which to enable pgo/pdo. Based on that you could enable
pgo based on some static decisions based on autofdo. For that you don't need any
profile runs during your build; it just needs shipping the autofdo outcome
together with a Python release. This doesn't give you the same performance as
for for a GCC pgo build, but it would be a first step. And defining the probe
for any pgo build would be welcome too.

  Matthias


On 08/22/2015 06:25 PM, Brett Cannon wrote:
> On Sat, Aug 22, 2015, 09:17 Guido van Rossum  wrote:
> 
> How about we first add a new Makefile target that enables PGO, without
> turning it on by default? Then later we can enable it by default.
> 
> 
> I agree. Updating the Makefile so it's easier to use PGO is great, but we
> should do a release with it as opt-in and go from there.
> 
> Also, I have my doubts about regrtest. How sure are we that it represents a
> typical Python load? Tests are often using a different mix of operations
> than production code.
> 
> That was also my question. You said that "it provides the best performance
> improvement", but compared to what; what else was tried? And what
> difference does it make to e.g. a Django app that is trained on their own
> simulated workload compared to using regrtest? IOW is regrtest displaying
> the best across-the-board performance because it stresses the largest swath
> of Python and thus catches generic patterns in the code but individuals
> could get better performance with a simulated workload?
> 
> -Brett
> 
> 
> On Sat, Aug 22, 2015 at 7:46 AM, Patrascu, Alecsandru <
> alecsandru.patra...@intel.com> wrote:
> 
> Hi All,
> 
> This is Alecsandru from Server Scripting Languages Optimization team at
> Intel Corporation.
> 
> I would like to submit a request to turn-on Profile Guided Optimization or
> PGO as the default build option for Python (both 2.7 and 3.6), given its
> performance benefits on a wide variety of workloads and hardware.  For
> instance, as shown from attached sample performance results from the Grand
> Unified Python Benchmark, >20% speed up was observed.  In addition, we are
> seeing 2-9% performance boost from OpenStack/Swift where more than 60% of
> the codes are in Python 2.7. Our analysis indicates the performance gain
> was mainly due to reduction of icache misses and CPU front-end stalls.
> 
> Attached is the Makefile patches that modify the all build target and adds
> a new one called "disable-profile-opt". We built and tested this patch for
> Python 2.7 and 3.6 on our Linux machines (CentOS 7/Ubuntu Server 14.04,
> Intel Xeon Haswell/Broadwell with 18/8 cores).  We use "regrtest" suite for
> training as it provides the best performance improvement.  Some of the test
> programs in the suite may fail which leads to build fail.  One solution is
> to disable the specific failed test using the "-x " flag (as shown in the
> patch)
> 
> Steps to apply the patch:
> 1.  hg clone https://hg.python.org/cpython cpython
> 2.  cd cpython
> 3.  hg update 2.7 (needed for 2.7 only)
> 4.  Copy *.patch to the current directory
> 5.  patch < python2.7-pgo.patch (or patch < python3.6-pgo.patch)
> 6.  ./configure
> 7.  make
> 
> To disable PGO
> 7b. make disable-profile-opt
> 
> In the following, please find our sample performance results from latest
> XEON machine, XEON Broadwell EP.
> Hardware (HW):  Intel XEON (Broadwell) 8 Cores
> 
> BIOS settings:  Intel Turbo Boost Technology: false
> Hyper-Threading: false
> 
> Operating System:   Ubuntu 14.04.3 LTS trusty
> 
> OS configuration:   CPU freq set at fixed: 2.6GHz by
> echo 260 >
> /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
> echo 260 >
> /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
> Address Space Layout Randomization (ASLR) disabled (to
> reduce run to run variation) by
> echo 0 > /proc/sys/kernel/randomize_va_space
> 
> GCC version:gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)
> 
> Benchmark:  Grand Unified Python Benchmark (GUPB)
> GUPB Source: https://hg.python.org/benchmarks/
> 
> Python2.7 results:
> Python source: hg clone https://hg.python.org/cpython cpython
> Python Source: hg update 2.7
> hg id: 0511b1165bb6 (2.7)
> hg id -r 'ancestors(.) and tag()': 15c95b7d81dc (2.7) v2.7.10
> hg --debug id -i: 0511b1165bb6cf40ada0768a7efc7ba89316f6a5
> 
> Benchmarks  Speedup(%)
> simple_logging  20
> raytrace20
> silent_logging  19
> richards19
> cha

Re: [Python-Dev] buildbot's are needlessly compiling -O0

[Matthias Klose]

Am 24.11.2013 08:13, schrieb Gregory P. Smith:
> our buildbots are setup to configure --with-pydebug which also
> unfortunately causes them to compile with -O0... this results in a python
> binary that is excruciatingly slow and makes the entire test suite run take
> a long time.
> 
> given that nobody is ever going to run a gdb or another debugger on the
> buildbot generated transient binaries themselves how about we speed all of
> the buildbot's up by adding CFLAGS=-O2 to the configure command line?
> 
> Sure, the compile step will take a bit longer but that is dwarfed by the
> test time as it is:
> 
> http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3224
> http://buildbot.python.org/all/builders/ARMv7%203.x/builds/7
> http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/639
> 
> It should dramatically decrease the turnaround latency for buildbot results.

maybe check if the compiler understands -Og (GCC 4.8), and use that?  Otoh, it
would be good to check with -O2 or -O3 too, common values used when building
python for production use.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final

[Matthias Klose]

Am 17.02.2014 00:25, schrieb Larry Hastings:
> And my local branch will remain private until 3.4.0 final ships!

sorry, but this is so wrong. Is there *any* reason why to keep this branch 
private?

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final

[Matthias Klose]

Am 19.02.2014 00:46, schrieb Larry Hastings:
> On 02/18/2014 03:38 PM, Matthias Klose wrote:
>> Am 17.02.2014 00:25, schrieb Larry Hastings:
>>> And my local branch will remain private until 3.4.0 final ships!
>> sorry, but this is so wrong. Is there *any* reason why to keep this branch
>> private?
> 
> Yes.  It ensures that nobody can check something into it against my wishes. 
> Also, in the event that I cherry-pick revisions out-of-order, it allows me to
> rebase, making merging easier.
> 
> Is there *any* reason to make this branch public before 3.4.0 final?

 - Python is an open source project.  Why do we need to hide
   development for a month or more?

 - Not even four eyes looking at the code seems to be odd. You
   can make mistakes too.

This seems to be a social or a technical problem.  I assume making this branch
available read-only would address your concerns?  Does hg allow this?  And if
not, why not create this branch in the upstream repository, and tell people not
to commit to it?  Why shouldn't such a social restriction work?  Seems to work
well for other projects.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final

[Matthias Klose]

Am 19.02.2014 01:05, schrieb Larry Hastings:
> On 02/18/2014 03:56 PM, Matthias Klose wrote:
>> Am 19.02.2014 00:46, schrieb Larry Hastings:
>>> On 02/18/2014 03:38 PM, Matthias Klose wrote:
>>>> Am 17.02.2014 00:25, schrieb Larry Hastings:
>>>>> And my local branch will remain private until 3.4.0 final ships!
>>>> sorry, but this is so wrong. Is there *any* reason why to keep this branch
>>>> private?
>>> Yes.  It ensures that nobody can check something into it against my wishes.
>>> Also, in the event that I cherry-pick revisions out-of-order, it allows me 
>>> to
>>> rebase, making merging easier.
>>>
>>> Is there *any* reason to make this branch public before 3.4.0 final?
>>   - Python is an open source project.  Why do we need to hide
>> development for a month or more?
>>
>>   - Not even four eyes looking at the code seems to be odd. You
>> can make mistakes too.
>>
>> This seems to be a social or a technical problem.  I assume making this 
>> branch
>> available read-only would address your concerns?  Does hg allow this?  And if
>> not, why not create this branch in the upstream repository, and tell people 
>> not
>> to commit to it?  Why shouldn't such a social restriction work?  Seems to 
>> work
>> well for other projects.
> 
> When you are release manager for Python, you may institute this policy if you
> like.  Right now, I have enough to do as it is.

is it too much to ask for a public mirror / tarball / something of this branch?
 This seems to be a minor effort compared to the clinic work that went into 3.4.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 3.4: Cherry-picking into rc2 and final

[Matthias Klose]

Am 19.02.2014 22:18, schrieb Nick Coghlan:
> and the Ubuntu 14.04 deadline restricts our ability to add a 3rd rc.

well, I think it would be wrong to restrict that for only that reason.  I did
object to delay the release cycle a second time for completing a feature.  If
the release has to be delayed for QA reasons, that's a good reason. Having a
Debian background myself ;)

I'm fine to ship with a rc3 too, and update it post-release, after wading
through distribution bureaucracy ... but please don't use this as an example of
a distribution influencing an upstream.  There are "better" examples for this 
:-/

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Cherry-pick between Python 3.4 RC2 and final?

[Matthias Klose]

Am 04.03.2014 15:52, schrieb Brett Cannon:
> I have also filed http://bugs.python.org/issue20851 to make sure the
> devguide covers running tests from a tarball. If the way the release has
> been handled has still bugged you enough it can be discussed at the
> language summit, but it would be the first time we consider putting any
> restrictions on the RM.

I wouldn't put it this way, but instead ask how to enable the RM to do this kind
of work more publicly.  I really would like it more if we can extend our buildd
infrastructure to automatically test during the time where the trunk and the
release candidates don't match. I am aware that this was never done for any
release in the past, but maybe this is something that can be enabled for 3.5.
But before documenting a process which may change depending on the RM why not
try to align the process?

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Where is our official policy of what platforms we do support?

[Matthias Klose]

Am 14.05.2014 17:08, schrieb Brett Cannon:
> On Wed May 14 2014 at 11:02:50 AM, R. David Murray 
> wrote:
> 
>> On Wed, 14 May 2014 11:31:15 -0300, "Joao S. O. Bueno" <
>> jsbu...@python.org.br> wrote:
>>> +1 for an official policy that comes with a "permanent maintainer for
>>> this platform required"  as part of the list
>>> of requisites.
>>>
>>>   js
>>>  -><-
>>>
>>> On 14 May 2014 11:20, Brett Cannon  wrote:
 Over the past week or so there have been 2 patches to add support for
 various UNIX OSs. Now I thought we had stopped trying to add new
>> esoteric
 OSs (e.g. I had never heard of MirOS until the patch for it came in),
>> but I
 can't find a PEP that spells out what it takes to get a platform
>> supported
 (http://legacy.python.org/dev/peps/pep-0011/ is about removing
>> platforms,
 not keeping them or adding them unless you are re-adding one which
 apparently just takes a volunteer).

 Do we want an official policy written down in a PEP (yes, I can write
>> it)?
 Should I keep closing these patches and saying that we are not adding
 support for new operating systems and be hand-wavy about it?
>>
>> In addition to a maintainer (who I think doesn't have to be a committer,
>> though that would be ideal), I think a maintained buildbot should be a
>> requirement for formal support.
>>
> 
> I would think someone how is/would be a core dev and a *stable* buildbot
> are requirements.

so, are aarch64-linux-gnu and powerpc64le-linux-gnu supported?  I assume there
are no buildbots and there won't be any for a long time. Otoh various distros do
ship python on these architectures. Or are these just new architectures for an
existing platform?  If yes, then we should ask about architecture support too.
The most prominent linux example are some RTLD constants which differ across
some architectures.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] use cases for "python-config" versus "pkg-config python"

[Matthias Klose]

Am 02.06.2014 20:11, schrieb Michael Haubenwallner:
> Hi,
> 
> following up myself with a patch proposal:
> 
> On 05/28/2014 04:51 PM, Michael Haubenwallner wrote:
>> Stumbling over problems on AIX (Modules/python.exp not found) building 
>> libxml2 as python module
>> let me wonder about the intended use-cases for 'python-config' and 
>> 'pkg-config python'.
>>
>> FWIW, I can see these distinct use cases here, and I'm kindly asking if I 
>> got them right:
>>
>> * Build an application containing a python interpreter (like python$EXE 
>> itself):
>>   + link against libpython.so
>>   + re-export symbols from libpython.so for python-modules 
>> (platform-specific)
>>   + This is similar to build against any other library, thus
>>   = 'python.pc' is installed (for 'pkg-config python').
>>
>> * Build a python-module (like build/lib.-/*.so):
>>   + no need to link against libpython.so, instead
>>   + expect symbols from libpython.so to be available at runtime, 
>> platform-specific either as
>>   + undefined symbols at build-time (Linux, others), or
>>   + a list of symbols to import from "the main executable" (AIX)
>>   + This is specific to python-modules, thus
>>   = 'python-config' is installed.
>>
> 
> Based on these use-cases, I'm on a trip towards a patch improving AIX support 
> here,
> where the attached one is a draft against python-tip (next step is to have 
> python-config
> not print $LIBS, but $LINKFORMODULE only).
> 
> Thoughts?

there is http://bugs.python.org/issue15590

I think it is worth improving, together with adding documentation, and maybe
distinguishing the two use cases linking for a module or an embedded 
interpreter.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 3.8.0a1 is now available for testing

[Matthias Klose]

On 06.02.19 13:23, Petr Viktorin wrote:
> FWIW, we're preparing to rebuild all Fedora packages with the 3.8 
> alphas/betas,
> so everything's tested when 3.8.0 is released:
> https://fedoraproject.org/wiki/Changes/Python3.8
> 
> That should cover the main Python projects, too.

well, the real challenge is that all test suites of third party packages still
pass on all architectures.  From past transitions, I know that this costs the
most time and resources.  But yes, targeting 3.8 for Ubuntu 20.04 LTS as well.

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Matthias Klose]

On 13.02.19 17:20, Victor Stinner wrote:
> Hi,
> 
> I'm a (strong) supporter of providing a "python" command which would
> be the latest Python version!

This very much depends on what is working with the latest Python version, and
what amount of third party packages your distro has to support. It doesn't have
to be the newest version.

> As php does nowadays (after previous issues with "php4" vs "php5".) I
> don't recall that perl had "perl4" vs "perl5", the command was always
> "perl", no? Same for Ruby: it was still "ruby" after for Ruby 2, no?
> Only Python and PHP used different program names depending on the
> language version, no? And PHP now moved back to a single "php"
> program.

it's not only upstreams doing that kind of versioned names; distros are doing
that to ease the pain for larger transitions.

> In the container and virtualenv era, it's now easy to get your
> favorite Python version for the "python" command.
> 
> On my Windows VM, "python" is Python 3.7 :-) In virtual environments,
> "python" can also be Python 3 as well.

maybe the PEP should recommend to have python3 in virtual environments as well?

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Matthias Klose]

On 13.02.19 16:24, Petr Viktorin wrote:
> PEP 394 says:
> 
>> This recommendation will be periodically reviewed over the next few
>> years, and updated when the core development team judges it
>> appropriate. As a point of reference, regular maintenance releases
>> for the Python 2.7 series will continue until at least 2020.
> 
> I think it's time for another review.
> I'm especially worried about the implication of these:
> 
> - If the `python` command is installed, it should invoke the same
>   version of Python as the `python2` command
> - scripts that are deliberately written to be source compatible
>   with both Python 2.x and 3.x [...] may continue to use `python` on
>   their shebang line.
> 
> So, to support scripts that adhere to the recommendation, Python 2
> needs to be installed :(

Debian's concern about pointing python to python3 is that it will break software
after an upgrade.  The current state seems is still the same that Debian doesn't
want to ship a python symlink after the Python2 removal.

For Ubuntu, I'm not sure if I want a python executable at all, because there is
not much progress in handling more than one python installation, so just using
python3 for the distro sounds fine.  pypi.org now recommends unconditionally
installing with pip, and pip is still happy to modify system installed packages
when asked, messing around with the distro packages.  But probably that kind of
users then install their own python symlink anyway.

For the Ubuntu 20.04 LTS release and the Debian bullseye release (maybe 2021), I
am trying to make sure that the python shebang isn't used by distro packages
anymore (either by removing python2 altogether, or by using the
python2/python2.7 shebangs).

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Matthias Klose]

On 16.02.19 00:15, Gregory P. Smith wrote:
> On Thu, Feb 14, 2019 at 9:29 AM Barry Warsaw  wrote:
> 
>> On Feb 13, 2019, at 23:08, Matěj Cepl  wrote:
>>
>>> Is this relevant to the discussion at hand? We are talking about
>>> the binary /usr/bin/python3 which will be surely be provided
>>> even by Python 4, won't it?
>>
>> Why would it be?  Since this is all hypothetical anyway , I’d more
>> likely expect to only ship /usr/bin/python.
>>
> 
> Because nobody can use 'python' and expect that to be anything but a 2and3
> compatible interpreter for the next 5+ years given we live in a world where
> people routinely have a very real need to write #! lines that works with
> obsolete distributions.  python3 implies >=3.x, thus python 4, 5, 6, 2069,
> 3001, and 90210 should all have python3 point to them.  realistically
> people will stop referring to python3 by 2069 so we could consider removing
> the recommendation at that point.
> 
> 2020 is not the end of use or end of importance for Python 2.  It is merely
> the end of bugfixes applied by python-dev.
> 
> A thing I want to make sure we _don't_ do in the future is allow future
> pythonN binaries.  python4, python90210, etc.  those should never exist.
> python, python3, and pythonX.Y only.  If we were ever to go back on our
> promise and create another world-breaking python version, it could get its
> own canonical binary name.  But we're specifically planning _not_ to make
> that mistake again.
> 
> I suspect most of my opining below will be contentious to multiple people
> because I describe a state of the world that is at conflict with decisions
> multiple independent distros have already made.  Accept their mistakes and
> move on past it to the hack in that case:
> 
> A new "py" launcher isn't going to solve this problem - it is separate and
> should be its own PEP as it has its own set of requirements and reasons to
> be considered (especially on platforms with no concept of a #!).  Recommend
> "py" today-ish and nobody can rely on it for at least 10+ years in a wide
> support cross platform scripting type of situation because it won't be
> present on the obsolete or long term supported things that people have a
> need for such #!s to run on.
> 
> Not our problem?  Well, actually, it is.  Matthias speaking for Debian
> suggesting they don't want to have "python" at all if it isn't a synonym
> for "python2" because it'll break software is... wrong.  If software is not
> 3 compatible and uses "python", it'll also break when python is python3.
> Just in a different manner.  "python" should point to python3 when a distro
> does not require python2 for its core.  It should not _vary_ as to which of
> 2.7 or 3.7 it will point to within a given stable distribution (installing
> python2.7 should never suddenly redirect it back to python2).  But "python"
> itself should always exist when any python interpreter is core to the OS.
> That means if a distro no longer requires python2 as part of its base/core
> but does require python3... "python" must point to "python3".  If a posixy
> OS no longer requires python at all (surely there are some by now?) the
> question of what python should point to when an OS distro supplied optional
> python package gets installed is likely either "nothing at all" or ">=3.x"
> but should never be decided as "2.7" (which sadly may be what macOS does).

There is no notion of a "core" for Debian. So "core" applies to the whole
distro, as long as there are python shebangs found.

For Ubuntu, you don't have a python command on the default desktop install, just
python3.  Trying to invoke python, command-not-found tells you:

  $ python

  Command 'python' not found, but can be installed with:

  [...]

  You also have python3 installed, you can run 'python3' instead.

That tells you which way to go.

> Do we already have LTS _stable_ distributions making that mistake today?
> If so they've done something undesirable for the world at large and we're
> already screwed if that distro release is deemed important by masses of
> users: There is no way to write a *direct* #! line that works out of the
> box to launch a working latest version Python interpreter across all
> platforms.

If you count the above example towards this "mistake", probably yes. But there
is *no* way to have a sane way to have what you want.

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Another update for PEP 394 -- The "python" Command on Unix-Like Systems

[Matthias Klose]

On 16.02.19 18:25, Nick Coghlan wrote:
> While Matthias is still personally reluctant to add the alias for
> Debian/Ubuntu, the *only* thing preventing aliasing /usr/bin/python to
> /usr/bin/python3 right now on the Fedora & RHEL side of things is PEP
> 394, and Guido objected strongly when Petr last tried to get the PEP
> to even acknowledge that it was reasonable for distros to make that
> setting configurable on a system-wide basis:
> https://github.com/python/peps/pull/630

No, I'm not "personally reluctant" about this, it's the current majority view of
people on the debian-python ML.  Barry stepped back as a Debian maintainer, so
there are not many people supporting your view.

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Testing cross-compiled python (windows arm32) from build bot

[Matthias Klose]

On 18.03.19 21:10, Paul Monson via Python-Dev wrote:
> Hello,
> 
> For windows arm32 I would like to be able run tests in the buildbot.
> Windows arm32 programs built with MSVC are always cross-compiled.
> This means I need to build cpython on x86/x64 and then deploy the build 
> artifacts and run the tests on an ARM32 device.
> 
> Is there any documentation or examples of how to do this?
> If not, are there any recommendations for how to achieve this?

I'm not aware of any documentation. For arm-linux-gnueabihf I once had a setup
with two chroots, the first one for cross-building python, and then the second
one for running the tests using qemu.  Two separate chroots to make sure that I
don't run any target code during the build.  The build location bind mounted in
both chroots to avoid the copy step.

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP-582 and multiple Python installations

[Matthias Klose]

On 02.04.19 18:10, Steve Dower wrote:
> On 02Apr2019 0817, Calvin Spealman wrote:
>> (I originally posted this to python-ideas, where I was told none of this 
>> PEP's
>> authors subscribe so probably no one will see it there, so I'm posting it 
>> here
>> to raise the issue where it can get seen and hopefully discussed)
> 
> Correct, thanks for posting. (I thought we had a "discussions-to" tag with
> distutils-sig on it, but apparently not.)
> 
>> While the PEP does show the version number as part of the path to the actual
>> packages, implying support for multiple versions, this doesn't seem to be
>> spelled out in the actual text. Presumably __pypackages__/3.8/ might sit
>> beside __pypackages__/3.9/, etc. to keep future versions capable of 
>> installing
>> packages for each version, the way virtualenv today is bound to one version 
>> of
>> Python.
>>
>> I'd like to raise a potential edge case that might be a problem, and likely 
>> an
>> increasingly common one: users with multiple installations of the *same*
>> version of Python. This is actually a common setup for Windows users who use
>> WSL, Microsoft's Linux-on-Windows solution, as you could have both the 
>> Windows
>> and Linux builds of a given Python version installed on the same machine. The
>> currently implied support for multiple versions would not be able to separate
>> these and could create problems if users pip install a Windows binary package
>> through Powershell and then try to run a script in Bash from the same
>> directory, causing the Linux version of Python to try to use Windows python
>> packages.
>>
>> I'm not actually sure what the solution here is. Mostly I wanted to raise the
>> concern, because I'm very keen on WSL being a great entry path for new
>> developers and I want to make that a better experience, not a more confusing
>> one. Maybe that version number could include some other unique identify, 
>> maybe
>> based on Python's own executable. A hash maybe? I don't know if anything like
>> that already exists to uniquely identify a Python build or installation.
> 
> Yes, this is a situation we're aware of, and it's caught in the conflict of 
> "who
> is this feature meant to support".

This smells the same like mixing system installed python packages (deb/rpm) with
one managed by pip, and pip touching system installed packages.

> Since all platforms have a unique extension module suffix (e.g.
> "module.cp38-win32.pyd"), it would be possible to support this with "fat"
> packages that include all binaries (or some clever way of merging wheels for
> multiple platforms).

unfortunately not. The Android developers opted out of that, reverting that
change.  Also how would you differentiate win32 builds for different
architectures? But maybe this is already done.

> And since this is already in CPython itself, it leads to about the only
> reasonable solution - instead of "3.8", use the extension module suffix
> "cp38-win32". (Wheel tags are not in core CPython, so we can't use those.)
> 
> But while this seems obvious, it also reintroduces problems that this has the
> potential to fix - suddenly, just like installing into your global 
> environment,
> your packages are not project-specific anymore but are Python-specific. Which 
> is
> one of the major confusions people run into ("I pip installed X but now can't
> import it in python").
> 
> So the main points of discussion right now are "whose problem does this solve"
> and "when do we tell people they need a full venv". And that discussion is
> mostly happening at
> https://discuss.python.org/t/pep-582-python-local-packages-directory/963/
> 
> Cheers,
> Steve
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/doko%40ubuntu.com

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Matthias Klose]

On 24.04.19 18:02, Victor Stinner wrote:
> Hum, I found issues with libpython: C extensions are explicitly linked
> to libpython built in release mode. So a debug python loading a C
> extension may load libpython in release mode, whereas libpython in
> debug mode is already loaded.
> 
> When Python is built with --enable-shared, the python3.7 program is
> linked to libpython3.7m.so.1.0 on Linux. C extensions are explicitly
> linked to libpython3.7m as well:
> 
> $ python3.7-config --ldflags
> ... -lpython3.7m ...
> 
> Example with numpy:
> 
> $ ldd 
> /usr/lib64/python3.7/site-packages/numpy/core/umath.cpython-37m-x86_64-linux-gnu.so
> ...
> libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (...)
> ...
> 
> When Python 3.7 is compiled in debug mode, libpython gets a "d" flag
> for debug: libpython3.7dm.so.1.0.
> 
> I see 2 solutions:
> 
> (1) Use a different directory. If "libpython" gets the same filename
> in release and debug mode, at least, they must be installed in
> different directories. If libpython build in debug mode is installed
> in /usr/lib64/python3.7-dbg/ for example, python3.7-dbg should be
> compiled with -rpath /usr/lib64/python3.7-dbg/ to get the debug
> libpython.
> 
> (2) If "libpython" gets a different filename in debug mode, C
> extensions should not be linked to libpython explicitly but
> *implicitly* to avoid picking the wrong libpython. For example, remove
> "-lpython3.7m" from "python3.7-config --ldflags" output.
> 
> The option (1) rely on rpath which is discouraged by Linux vendors and
> may not be supported by all operating systems.
> 
> The option (2) is simpler and likely more portable.
> 
> Currently, C extensions of the standard library may or may not be
> linked to libpython depending on they are built. In practice, both
> work since python3.7 is already linked to libpython: so libpython is
> already loaded in memory before C extensions are loaded.

the purpose of python-config here is not clear. Whether it's intended to be used
for linking extensions, or embedded interpreters. Currently you are using the
same for both use cases.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Matthias Klose]

On 24.04.19 01:44, Victor Stinner wrote:
> Hi,
> 
> Two weeks ago, I started a thread "No longer enable Py_TRACE_REFS by
> default in debug build", but I lost myself in details, I forgot the
> main purpose of my proposal...
> 
> Let me retry from scratch with a more explicit title: I would like to
> be able to run C extensions compiled in release mode on a Python
> compiled in debug mode ("pydebug"). The use case is to debug bugs in C
> extensions thanks to additional runtime checks of a Python debug
> build, and more generally get a better debugging experiences on
> Python. Even for pure Python, a debug build is useful (to get the
> Pyhon traceback in gdb using "py-bt" command).
> 
> Currently, using a Python compiled in debug mode means to have to
> recompile C extensions in debug mode. Compile a C extension requires a
> C compiler, header files, pull dependencies, etc. It can be very
> complicated in practical (and pollute your system with all these
> additional dependencies). On Linux, it's already hard, but on Windows
> it can be even harder.
> 
> Just one concrete example: no debug build of numpy is provided at
> https://pypi.org/project/numpy/ Good luck to build numpy in debug mode
> manually (install OpenBLAS, ATLAS, Fortran compiler, Cython, etc.)
> :-)

there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;)  So
depending on the package maintainer, you already have that available, but it is
extra maintenance cost.  Simplifying that would be a good idea.  However I still
would like to be able to have "debug" and "non-debug" builds co-installable at
the same time.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Matthias Klose]

On 25.04.19 08:31, Nathaniel Smith wrote:
> You don't necessarily need rpath actually. The Linux loader has a
> bug/feature where once it has successfully loaded a library with a given
> soname, then any future requests for that soname within the same process
> will automatically return that same library, regardless of rpath settings
> etc. So as long as the main interpreter has loaded libpython.whatever from
> the correct directory, then extension modules will all get that same
> version. The rpath won't matter at all.
> 
> It is annoying in general that on Linux, we have these two different ways
> to build extension modules. It definitely violates TOOWTDI :-). It would be
> nice at some point to get rid of one of them.
> 
> Note that we can't get rid of the two different ways entirely though – on
> Windows, extension modules *must* link to libpython.dll, and on macOS,
> extension modules *can't* link to libpython.dylib. So the best we can hope
> for is to make Linux consistently do one of these, instead of supporting
> both.
> 
> In principle, having extension modules link to libpython.so is a good
> thing. Suppose that someone wants to dynamically load the python
> interpreter into their program as some kind of plugin. (Examples: Apache's
> mod_python, LibreOffice's support for writing macros in Python.) It would
> be nice to be able to load python2 and python3 simultaneously into the same
> process as distinct plugins. And this is totally doable in theory, *but* it
> means that you can't assume that the interpreter's symbols will be
> automagically injected into extension modules, so it's only possible if
> extension modules link to libpython.so.
> 
> In practice, extension modules have never consistently linked to
> libpython.so, so everybody who loads the interpreter as a plugin has
> already worked around this. Specifically, they use RTLD_GLOBAL to dump all
> the interpreter's symbols into the global namespace. This is why you can't
> have python2 and python3 mod_python at the same time in the same Apache.
> And since everyone is already working around this, linking to libpython.so
> currently has zero benefit... in fact manylinux wheels are actually
> forbidden to link to libpython.so, because this is the only way to get
> wheels that work on every interpreter.

extensions in Debian/Ubuntu packages are not linked against libpython.so, but
the main reason here is that sometimes you have to extensions built in
transition periods like for 3.6 and 3.7. And this is also the default when not
configuring with --enable-shared.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Matthias Klose]

On 25.04.19 13:26, Victor Stinner wrote:
> I looked how fonforge gets compiler and linker flags to embed Python:
> it seems like to "pkg-config --libs python-2.7" which returns
> "-lpython2.7". My PR doesn't change Misc/python.pc. Should I modify
> Misc/python.pc as well... or not? :-) I'm not used to pkg-config. I
> don't know if it's common that C extensions are built using
> pkg-config. I guess that distutils is more commonly used to build C
> extensions.

... except for all the software which is doing some embedding (e.g. vim), or is
building some bindings as part of the upstream software. So yes, there is some
stuff ...

The tendency seems to deprecate your own config helper in favor of pkgconfig.
However I'm not sure how this would do with the current MacOS python-config
python script.  If we want to differentiate between embedding and extensions,
then we need two different module names, maybe keeping the current one for
extensions, and having a new one for embedding.

Not sure about python-config, if we want a new helper for embedding, or add new
options for the existing script.

> Victor
> 
> Le jeu. 25 avr. 2019 à 12:53, Victor Stinner  a écrit :
>> Le jeu. 25 avr. 2019 à 09:30, Matthias Klose  a écrit :
>>> the purpose of python-config here is not clear. Whether it's intended to be 
>>> used
>>> for linking extensions, or embedded interpreters. Currently you are using 
>>> the
>>> same for both use cases.
>>
>> My PR 12946 removes libpython from distutils, python-config and
>> python-config.py:
>> https://github.com/python/cpython/pull/12946
>>
>> Do you mean that this change will break the build of applications
>> embedding Python? If yes, what can done to fix that?
>>
>> Provide a different script to the specific case of embedded Python? Or
>> add a new option to specify that you are embedding Python?
>>
>> In Python 3.7, the required linker flag is "-lpython3.7m". It's not
>> trivial to guess the "m" suffix. FYI Python 3.8 it becames just
>> "-lpython3.8": I removed the "m" suffix which was useless.
>>
>> Victor
>> --
>> Night gathers, and now my watch begins. It shall not end until my death.
> 
> 
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode

[Matthias Klose]

On 25.04.19 13:14, Victor Stinner wrote:
> Le jeu. 25 avr. 2019 à 09:34, Matthias Klose  a écrit :
>> there's a simple solution: apt install python3-numpy-dbg cython3-dbg ;)  So
>> depending on the package maintainer, you already have that available, but it 
>> is
>> extra maintenance cost.  Simplifying that would be a good idea.
> 
> Fedora provides "debuginfo" for all binarry packages (like numpy), but
> that's different from a debug build. Usually, C code of packages are
> optimized by gcc -O2 or even gcc -O3 which makes the debugging
> experience very painful: gdb fails to read C local variables and just
> say "". To debug internals, you want a debug build
> compiled by gcc -Og or (better IMHO) gcc -O0.
> 
> If you want to inspect *Python* internals but you don't need to
> inspect numpy internals, being able to run a release numpy on a debug
> Python is convenient.

yes, the Debian/Ubuntu packages contain both the debug build, and the debug info
for they normal build, e.g.

/usr/lib/debug/.build-id/3a/8ea2ab6ee85ff68879a48170966873eb8da781.debug
/usr/lib/debug/.build-id/78/5ff95f8d2d06c5990ae4e03cdff99452ca0de9.debug
/usr/lib/debug/.build-id/92/e008cffa3f09106214bfb6b80b7fd02ceab74f.debug
/usr/lib/debug/.build-id/ab/33160518c41acc0488bbc3af878995ef74e07f.debug
/usr/lib/debug/.build-id/bd/65896626a4c6566e96ad008362922cf6a39cd6.debug
/usr/lib/debug/.build-id/f1/e83b14a76dd9564e962dcdd2f70202e6fdb2b1.debug
/usr/lib/debug/.build-id/ff/5eab5fd2d14f4bfa6a1ef2300358efdc7dd800.debug
/usr/lib/python3/dist-packages/lxml/_elementpath.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/builder.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/etree.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/html/clean.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/html/diff.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/objectify.cpython-37dm-x86_64-linux-gnu.so
/usr/lib/python3/dist-packages/lxml/sax.cpython-37dm-x86_64-linux-gnu.so

> With an additional change on SOABI (I will open a separated issue for
> that), my PR 12946 (no longer link C extensions to libpython) allows
> to load lxml built in release mode in a Python built in debug mode!
> That's *very* useful for debugging. I show an example of the gdb
> experience with a release Python vs debug Python:
> 
> https://bugs.python.org/issue21536#msg340821
> 
> With a release Python, the basic function "py-bt" works as expected,
> but inspecting Python internals doesn't work: most local C variables
> are "optimized out" :-(
> 
> With a debug Python, the debugging experience is *much* better: it's
> possible to inspect Python internals!
> 
> 
>> However I still
>> would like to be able to have "debug" and "non-debug" builds co-installable 
>> at
>> the same time.
> 
> One option is to keep "d" flag in the SOABI so C extensions get a
> different SO filename (no change compared to Python 3.7):
> "NAME.cpython-38-x86_64-linux-gnu.so" for release vs
> "NAME.cpython-38d-x86_64-linux-gnu.so" for debug, debug gets "d"
> suffix ("cpython-38" vs "cpython-38d").
> 
> *But* modify importlib when Python is compiled in debug mode to look
> also to SO without the "d" suffix: first try load
> "NAME.cpython-38d-x86_64-linux-gnu.so" (debug: "d" suffix). If there
> is no match, look for "NAME.cpython-38-x86_64-linux-gnu.so" (release:
> no suffix). Since the ABI is now compatible in Python 3.8, it should
> "just work" :-)
> 
> From a Linux packager perspective, nothing changes ;-) We can still
> provide "apt install python3-numpy-dbg" (debug) which can is
> co-installable with "apt install python3-numpy" (release).
> 
> The benefit is that it will be possible to load C extensions which are
> only available in the release flavor with a debug Python ;-)

yes, that sounds good.  Are there use cases where you only want to load *some*
debug extensions, even if more are installed?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: Speeding up CPython

[Matthias Klose]

On 10/20/20 2:53 PM, Mark Shannon wrote:
> I'd love to hear your thoughts on this.

a VM needs a separate backend for each architecture (maybe even OS)

 - which architectures do you include into your proposal?
   what's your estimate for a new port?

 - do you plan for a fall-back to a slow non-VM mode, e.g.
   the existing one?  Do you plan to support that in parallel
   such that Python still can be used on architectures with
   a working compiler (and a libffi port).

   E.g. OpenJDK has the zero port, which is slow (interpreted),
   but runs on practically all architectures.

Thanks, Matthias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H4O5TFWGL5SDG4NSESNGNKIZMK6W7L7V/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Distro packagers: PEP 615 and the tzdata dependency

[Matthias Klose]

On 11/16/20 6:46 PM, Barry Warsaw wrote:
> That’s what I was going to suggest.  I’m not doing any Debian or Ubuntu work 
> these days, but Matthias Klose should be watching both lists, and should be 
> able to handle the Debuntu stack.

why is this dependency needed?  The tzdata package is marked with Priority
required. See Debian policy 2.5.

Matthias

> -Barry
> 
>> On Nov 16, 2020, at 07:45, Miro Hrončok  wrote:
>>
>> On 11/16/20 4:10 PM, Paul Ganssle wrote:
>>>> Maybe it would make sense to create a community mailing list for
>>>> packagers?
>>> That sounds like a good idea to me.
>>
>> I am following the Linux SIG mailing list. But it's mostly dead.
>>
>> https://mail.python.org/archives/list/linux-...@python.org/
>>
>> --
>> Miro Hrončok
>> --
>> Phone: +420777974800
>> IRC: mhroncok
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/DF3UXOGYRAIOLTRGSNGNSZEKZOUFMXGA/
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/23R5ZD5ZVA4R73ZTKQWWJR6CDLGRKXJN/
> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MF4J5HIBM37NRZ5I2RSXYLU7PIXVKXO2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Can lib2to3 and 2to3 be deprecated yet?

[Matthias Klose]

On 12/10/20 10:58 AM, Victor Stinner wrote:
> Hi,
> 
> I vaguely recall that PEP 617 "New PEG parser for CPython" was
> supposed to deprecate the lib2to3 module and the 2to3 program.
> 
> I see that lib2to3/__init__.py contains a PendingDeprecationWarning in
> Python 3.9. Can it be converted into a regular DeprecationWarning and
> document the deprecation in Doc/library/2to3.rst?
> 
> --
> 
> Do we still want to maintain lib2to3? People who didn't migrate to
> Python 3 can use 2to3 of Python 3.9 or Python 3.10. There are also
> other tools like 2to6, modernize, etc. (or my "sixer" project which I
> no longer maintains.)
> 
> I'm not sure sure if the lib2to3 module is used by 3rd party projects
> like modernize or black?

lib2to3 is used by distutils/util, so why deprecate it before distutils?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KZ37BBHWE7UG65J6APC6ONW2LGWFXNTT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Bumping minimum Sphinx version to 3.2 for cpython 3.10?

[Matthias Klose]

On 1/13/21 12:28 AM, Victor Stinner wrote:
> Since documentation changes are backported to 3.8 and 3.9 stable
> branches, if we increase the minimum required Sphinx version in
> master, I would prefer to also increase it in 3.8 and 3.9 branches.
> 
> I would prefer to not have to check manually if a doc backport PR is
> still compatible with Sphinx 2 or not. If we skip some doc backports,
> conflicts will become more likely with following doc backports.
> 
> --
> 
> I looked at Sphinx and Python versions of Debian, Ubuntu and Fedora:
> https://bugs.python.org/issue42843#msg384963
> 
> In my list, there is only Debian Buster (stable) which doesn't have
> Sphinx 3 yet. It uses Python 3.7 and so would not be affected by
> Python 3.8 changes.

That's not true. Ubuntu 20.04 LTS still sees updates to subminor Python 3.8
versions, having sphinx 1.8.5.

Matthias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/SN24BKRX3PIIYMGDGX563HHGZXTRCHUI/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

[Matthias Klose]

On 2/21/21 9:24 PM, Gregory P. Smith wrote:
> On Sun, Feb 21, 2021 at 10:15 AM Christian Heimes 
> wrote:
> 
>> On 21/02/2021 13.47, glaub...@debian.org wrote:
>>> Rust doesn't keep any user from building Rust for Tier 2 or Tier 3
>> platforms. There is no separate configure guard. All platforms that Rust
>> can build for, are always enabled by default. No one in Rust keeps anyone
>> from cross-compiling code for sparc64 or powerpcspe, for example.
>>>
>>> So if you want to copy Rust's mechanism, you should just leave it as is
>> and not claim that users are being confused because "m68k" shows up in
>> configure.ac.
>>
>> A --enable-unstable-platforms configure flag is my peace offer to meet
>> you half way. You get a simple way to enable builds on untested
>> platforms and we can clearly communicate that some OS and hardware
>> platforms are not supported.
>>
> 
> I personally wouldn't want to maintain such a check in autoconf, but it'll
> be an isolated thing on its own, that if you or someone else creates, will
> do its job and not bother the rest of us.
> 
> I think just publishing our list of (1) supported, (2) best-effort
> non-release-blocker quasi-supported, and (3) explicitly unsupported in a
> policy doc is sufficient.  But it's not like any of us are going to stop
> someone from codifying that in configure.ac to require a flag.

agreed with (1) and(2).  I don't like a negative list, as this will be
incomplete at any time for both *-linux and *-non{linux,win,mac}. Looking at
another project like GCC

  https://gcc.gnu.org/buildstat.html

used to collect information about successful builds on various platforms and
architectures.  But as you can see, the web page isn't updated in recent times.

GCC also doesn't keep an explicit list of the lesser supported platforms:
https://gcc.gnu.org/gcc-11/criteria.html

Matthias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/3BMMNW4NERXTORS4SZA7XWHAYEKUQFFU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

[Matthias Klose]

On 2/21/21 7:12 PM, Christian Heimes wrote:
> On 21/02/2021 13.47, glaub...@debian.org wrote:
>> Rust doesn't keep any user from building Rust for Tier 2 or Tier 3 
>> platforms. There is no separate configure guard. All platforms that Rust can 
>> build for, are always enabled by default. No one in Rust keeps anyone from 
>> cross-compiling code for sparc64 or powerpcspe, for example.
>>
>> So if you want to copy Rust's mechanism, you should just leave it as is and 
>> not claim that users are being confused because "m68k" shows up in 
>> configure.ac.
> 
> A --enable-unstable-platforms configure flag is my peace offer to meet
> you half way.

Making a "peace offer" is the confession of having started a war.   I don't know
why you are on this crusade, but I doubt it is the lonely "confused user" cited
in https://bugs.python.org/issue43179.

Others pointed out in this thread that it's not the first time of such
unfriendly behavior.  Even others might also see it as some kind of harassment.

Matthias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/6NIXUMZVGKCTBQCS4QFELSIVXYPL5JL2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Move support of legacy platforms/architectures outside Python

[Matthias Klose]

On 2/21/21 1:13 PM, Victor Stinner wrote:
> In short, I propose to move maintenance of the legacy platforms/archs
> outside Python: people are free to continue support them as patches.


> Concrete example: Christian Heimes proposed to drop support for 31-bit
> s390 Linux:
> https://bugs.python.org/issue43179

I'm not attached to s390. But I doubt that the proposed patch lowers any
"maintenance burden".
https://github.com/python/cpython/pull/24534.diff touches a few files

 - configure.ac: PLATFORM_TRIPLET is undefined with your patch,
   changing the extension suffix.  I don't think that maintaining
   such a delta outside the Python tree would be good practice.
   I am committed to maintain these definitions, and would like
   to ask not to remove *any* of these.  These definitions come
   from https://wiki.debian.org/Multiarch/Tuples.
 - Modules/_ctypes/libffi_osx/ffi.c: Removing a macro usage in a
   file specific to MacOS X.  Removal doesn't help, otherwise
   might create a patch conflict, if the MacOS X maintainers
   should ever decide to pull in a new libffi upstream version.
 - Lib/test/test_sysconfig.py: Doesn't simplify the code,
   just makes a bit more unfriendly.

I became curious about a s390 build, and checked it with a GCC multilib build as
available on at least Debian, OpenSuse, RHEL and Ubuntu.  Builds fine with
CC="gcc -m31" CXX="g++ -m31" ./configure && make && make test.  No test
failures, although I didn't build all extensions using externals libs.

Funny thing, your proposed patch doesn't make any difference in the test
results, so I'm not sure why you spent any volunteer time on that patch.

With the threat^Wannouncement in issue 43179 to do similar removals for alpha,
hppa and m68k, you'll likely step onto more toes, and again not removing any 
burden.

> The lack of clear definition on how a platform is supported or not
> confuses users who consider that their favorite platform/arch is
> supported, whereas core developers don't want to support it since it
> would be too much work.

If a platform stops working, fine.  Actively breaking a platform in some form:
Not ok from my point of view.  Python still is ubiquitous on almost all
platforms (linux and non-linux ones), and can be built with mostly generic
dependencies.   Maybe libffi is an exception, however even these maintainers
didn't remove s390 support (https://sourceware.org/libffi/).  And I was very
happy to build Python on a proprietary *nix platform five years ago, and having
a working Python.

> In fact, the PEP 11 has clear and explicit rules:
> https://www.python.org/dev/peps/pep-0011/#supporting-platforms

PEP 11 is not fit for that.  You're not trying to remove support for the Linux
platform, you're addressing a specific architecture.  Maybe PEP 11 should be
updated.

As seen with S390, there's not much architecture specific code.  If I see issues
on the Linux platform for different architectures, these usually are:

 (1) 32bit vs 64bit
 (2) big endian vs. little endian
 (3) architecture specific alignment requirements, sometimes
 resulting in degraded performance on unaligned accesses,
 sometimes in bus errors when running 32bit code on 64 kernels.

Looking at https://pythondev.readthedocs.io/platforms.html (yes, that gets the
platform/architecture distinction right),

 - (1) and (2) are covered as "well supported" platforms, although
   having just x86* might lead to some x86-isms.

 - (1), (2) and (3) are covered as "best effort support" with
   aarch64 (64/LE), powerpc64le (64/LE), s390x (64/BE), and
   arm-linux-gnueabi* as 32/LE having bus errors on unaligned
   accesses with 64bit kernels.

Unless CPython stops supporting these platforms above, I don't see little value
of removing that tiny amount of architecture specific code.

Document what is supported, be inclusive about anything else.  Don't make a
distinction yet between legacy and upcoming new architectures.


Apparently the cryptography project made the decision to rely on a build tool
which is not ubiquitously available anymore.   If CPython sees the need for such
restrictions in the future, I'd like to propose to use the same deprecation
process for modules/functions, e.g. if a new non-ubiquitous tool X is needed to
build/run, announce it for 3.10, and only make use of it in 3.11.

Matthias

> Example: 16-bit m68k
no, it's a 32bit platform with extra alignment requirements.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YWRUGIA7XGRP6NABB3JEV552OSL6O52G/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: NOTE: Python 3.9.3 contains an unintentional ABI incompatibility leading to crashes on 32-bit systems

[Matthias Klose]

On 4/4/21 4:44 AM, Terry Reedy wrote:
> On 4/3/2021 7:15 PM, Miro Hrončok wrote:
>> On 03. 04. 21 21:44, Łukasz Langa wrote:
>>> The memory layout of PyThreadState was unintentionally changed in the recent
>>> 3.9.3 bugfix release. This leads to crashes on 32-bit systems when importing
>>> binary extensions compiled for Python 3.9.0 - 3.9.2. This is a regression.
>>>
>>> We will be releasing a hotfix 3.9.4 around 24 hours from now to address this
>>> issue and restore ABI compatibility with C extensions built for Python 3.9.0
>>> - 3.9.2.
>>
>> Thanks for the hotifx.
>>
>> However, I need to ask: Would this also happen if there was a rc version of
>> 3.9.3?
> 
> Unless the mistake was just introduced, the mistake would have happened.  One
> this severe would likely have been caught within the week or two before a
> final.  But as Łukasz noted when announcing the change, .rcs are generally
> ignored.

Well, that's not true.  I think for at least the past 3.8 and current 3.9 cycle
I always tested the release candidates, and built them for various Linux
architectures.  And I'm filing issues marked as 'released-blocker' when I see
regressions introduced, it's up to the release managers to determine if such
changes are intended or not.

Looking at the failing CI tests triggered by these builds, yes I see that 32bit
archs have the ABI change.

So maybe it's worth to re-introduce these RC builds, or at least provide source
RC tarballs which can be tested.  If it's a matter of resources, maybe fall back
to quarterly subminor releases again (at least for the most current X.Y 
releases).

Matthias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/4TSX2OCZPOSDBB2VG26HJMUZPYZ56IL3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: NOTE: Python 3.9.3 contains an unintentional ABI incompatibility leading to crashes on 32-bit systems

[Matthias Klose]

On 4/4/21 2:49 PM, Stephen J. Turnbull wrote:
> Matthias Klose writes:
> 
>  > Looking at the failing CI tests triggered by these builds, yes I
>  > see that 32bit archs have the ABI change.
> 
> I'm not sure precisely what you mean by that, but if you mean that CI
> has caught the bug, then
> 
>  > So maybe it's worth to re-introduce these RC builds,
> 
> seems to be just makework.  It makes more sense to delay the schedule
> somewhat, but only so that the release engineer, or one among the many
> eyes, catch the CI's warning.

No, you can't see that with CPython's CI alone.  The Debian and Ubuntu build
machines trigger CI tests per architecture for around 3000 packages depending on
python3.9, using the just built python3.9, and without rebuilding these
packages.  That's where I see the 32bit failures.

How would delaying the release schedule have helped with the issue that we just 
saw?

Matthias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U6C6LLSQS26RC2XC2B5GWNSZQJ5PPYMF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] How to fix the incorrect shared library extension on linux for 3.2 and newer?

[Matthias Klose]

This is http://bugs.python.org/issue16754, affecting Linux systems only, and
only those which don't provide static libraries. PEP 3149 did change the SO
macro to include the ABI tag, although the SO macro is used to search for shared
system libraries too.  E.g. searching for the jpeg library search for a file
libjpeg.cpython3.3m.so, which is not found. If the static library libjpeg.a is
found, it is taken, and linked as -ljpeg, which then links with the shared 
library.

The patch in the issue now makes a distinction between EXT_SUFFIX and
SHLIB_SUFFIX, and restores the value of SO to SHLIB_SUFFIX.  Now this could
break users of sysconfig.get_config_var('SO'), however I don't see a better way
than to restore the original behaviour and advise people to use the new config
variables.

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] backporting the _sysconfigdata.py module to 2.7

[Matthias Klose]

I'd like to backport issue13150, the _sysconfigdata.py module to 2.7. My
motivation is not the improved startup time, but the ability to cross-build
extension modules using distutils/setuptools.  The basic idea is to use the
python interpreter of the build machine (the machine you build on), and the
_sysconfigdata.py for the host machine (the machine you build for).  This kind
of setup works fine as long as the setup.py for a third party package gets all
build related information from the sysconfig.py module, and not directly from os
or sys (e.g. sys.platform).

The patch for issue13150 doesn't change any API's, but only moves the
computation of the config vars from runtime to build time.  I'd like to avoid
backporting this to 3.2 as well, because the cross-build support is currently
only found in 2.7, 3.3 and the trunk.

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1

[Matthias Klose]

Am 25.03.2013 01:30, schrieb Benjamin Peterson:
> 2.7.4 will be the latest maintenance release in the Python 2.7 series.

I hope it's not (and in the IDLE thread you say so otherwise too).

  Matthias


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The end of 2.7

[Matthias Klose]

Am 07.04.2013 20:02, schrieb Guido van Rossum:
> But perhaps we could change the focus for 2.7 development a bit:
> instead of fixing bugs (or bickering about whether something is a bug
> fix or a new feature) we could limit changes to ensuring that it works
> on newer platforms. Martin mentioned that building 2.7 for Windows
> with the same toolchain that was used for the 2.7.0 release is getting
> more and more problematic. I'm not sure, but I could imagine similar
> problems for future versions of OS X and even Linux (though the Linux
> distributions typically take care of issues themselves).

I would like this new focus :-) Note that for 2.7.4 we did backport the bsddb
module to build with recent db-5.x versions, the embedded libffi library to
build on new platforms.  I would like to see a backport for #17536 too, a change
to support new web browsers in an updated runtime environment.

I would like to continue to backport cross build changes to 2.7.x, before all
these people with Raspberry Pi's get too much annoyed about slow native builds.

Support for new targets should be allowed after a review.

> There's not much of a point in fixing bugs that always existed in 2.7,
> since must 2.7 users are by now used to working around these. However,
> I do see a point in supporting builds targeting newer OS versions.

The upcoming Ubuntu 13.04 release uses mostly Python 3.3.1 for the desktop
images, but still ships with Python 2.7.4 too on the images.  For now most third
party modules and extensions still have to be available for both versions.  For
now these binary packages (in the sense of a package in a Linux distribution)
are built for Python 2 and 3 from the same source package, so keeping the build
procedures and support about the same way helps with this approach.

Of course I can patch things locally, but would prefer to push these changes
upstream.  At the language summit I was surprised to hear about a common subset
of backports for other vendor branches.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Mysterious Python pyc file corruption problems

[Matthias Klose]

Am 15.05.2013 22:58, schrieb Barry Warsaw:
> I am looking into a particularly vexing Python problem on Ubuntu that 
> manifests in several different ways.  I think the problem is the same one 
> described in http://bugs.python.org/issue13146 and I sent a message on the 
> subject to the ubuntu-devel list: 
> https://lists.ubuntu.com/archives/ubuntu-devel/2013-May/037129.html

please consider that Ubuntu does have some other upgrade issues, when files in
the dpkg database (/var/lib/dpkg/info/*) are corrupted, or just are files
having null bytes.  So these and the pyc issues share in common that these
files are written after a package is unpacked.  I'm not saying that the
problem might be with the pyc writing, but we do see other file corruption as
well.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Request for CPython 3.5.3 release

[Matthias Klose]

On 03.07.2016 06:09, Nick Coghlan wrote:
> On 2 July 2016 at 16:17, Ludovic Gasc  wrote:
>> Hi everybody,
>>
>> I fully understand that AsyncIO is a drop in the ocean of CPython, you're
>> working to prepare the entire 3.5.3 release for December, not yet ready.
>> However, you might create a 3.5.2.1 release with only this AsyncIO fix ?
> 
> That would be more work than just doing a 3.5.3 release, though - the
> problem isn't with the version number bump, it's with asking the
> release team to do additional work without clearly explaining the
> rationale for the request (more on that below). While some parts of
> the release process are automated, there's still a lot of steps to run
> through by a number of different people:
> https://www.python.org/dev/peps/pep-0101/.
> 
> The first key question to answer in this kind of situation is: "Is
> there code that will run correctly on 3.5.1 that will now fail on
> 3.5.2?" (i.e. it's a regression introduced by the asyncio and
> coroutine changes in the point release rather than something that was
> already broken in 3.5.0 and 3.5.1).

I don't know about 3.5.1 exactly, but at least things worked on the branch early
April, which were broken by 3.5.2 final.  I was trying to prepare for an update
to 3.5.2 for Ubuntu 16.04 LTS, and found some regressions, documented at
https://launchpad.net/bugs/1586673 (comment #6).

It looks like that at least the packages nuitka, python-websockets and urwid
fail to build with the 3.5.2 release. Still need to investigate.

Unless I'm missing things, there is unfortunately no issue in the Python bug
tracker, and there is no patch for the 3.5 branch either.  My understanding is
that it's not yet decided what to do about the issue.

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] release cadence

[Matthias Klose]

On 05.07.2016 21:11, Brett Cannon wrote:
> On Tue, 5 Jul 2016 at 10:45 Barry Warsaw  wrote:
> 
>> On Jul 04, 2016, at 10:31 AM, Nick Coghlan wrote:
>>
>>> While we liked the "consistent calendar cadence that is some multiple
>>> of 6 months" idea, several of us thought 12 months was way too short
>>> as it makes for too many entries in third party support matrices.
>>
>> 18 months for a major release cadence still seems right to me.  Downstreams
>> and third-parties often have to go through *a lot* of work to ensure
>> compatibility, and try as we might, every Python release breaks
>> *something*.
>> Major version releases trigger a huge cascade of other work for lots of
>> other
>> people, and I don't think shortening that would be for the overall
>> community
>> good.  It just feels like we'd always be playing catch up.
>>
> 
> Sticking w/ 18 months is also fine, but then I would like to discuss
> choosing what months we try to release to get into a date-based release
> cadence so we know that every e.g. December and June are when releases
> typically happen thanks to our 6 month bug-fix release cadence. This has
> the nice benefit of all of us being generally aware of when a bug-fix
> release is coming up instead of having to check the PEP or go through our
> mail archive to find out what month a bug-fix is going to get cut (and also
> something the community to basically count on).

I like the 18 months cycle, because it's a multiple of six, which fits the
Ubuntu release cadence (as as I understand, the Fedora cadence as well).
Sometimes it might be ambitious to update reverse dependencies in the distro
within two months until the distro freeze, and two more months during the freeze
leading to a distro release, but such is life, and it's then up to distro
maintainers of LTS releases to prepare the distro for a new version with only
four months left.

My hope with time based releases is that also upstreams will start testing with
new versions more early when they can anticipate the release date.

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] C99

[Matthias Klose]

On 06.08.2016 00:27, Brett Cannon wrote:
> On Fri, 5 Aug 2016 at 15:17 Guido van Rossum  wrote:
> 
>> I want it to list specific versions that are known to be good right
>> now, so we can point fingers appropriately when a regression happens.
>>
> 
> OK, then we could pin it to MSVC 2016, gcc 6.1, and clang 3.8.1 which are
> the latest releases (unless I'm missing a compiler we all feel like we
> should officially support).

please make that GCC 6.2 once released later this month (6.1 is the first 6.x
release).

GCC 5 and GCC 6 default to C11 (-std=gnu11), does the restriction to C99 mean
that -std=gnu99 should be passed explicitly?

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7.13 release dates

[Matthias Klose]

On 28.11.2016 20:38, Larry Hastings wrote:
> On 11/28/2016 10:42 AM, Guido van Rossum wrote:
>> Perhaps (un)related, is the release schedule for 3.5.3 set yet? (I'm holding
>> off putting a new version of typing.py on typeshed until that's out.)
> 
> Not set in stone, but I have a rough plan.  I want the dust to settle around 
> 3.6
> first.  So, have an RC1 in January, and then the final release two weeks 
> later. 
> I'm traveling the second week of January (#DSDad) so the RC would be either
> before or after that.

https://wiki.debian.org/DebianStretch shows the final Debian release freeze for
2016-02-05, which would be a bit close for a 3.5.3 release.  So anything with
some buffer to Feb 05 would be appreciated.

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.7.13 release dates

[Matthias Klose]

same here.
https://wiki.debian.org/DebianStretch shows the final Debian release freeze for
2016-02-05, which would be a bit close for a 2.7.13 release.

On 29.11.2016 15:12, Steve Dower wrote:
> I would much rather keep it in December, as I've already made scheduling 
> decisions around the planned release date and there are fixes in 2.7.13 that 
> I was expecting to be available by the end of the year. One month is highly 
> impactful for me.
> 
> Is this schedule change going to remove the month from 2.7.14? Or are we 
> slipping all the rest of the releases (apart from the very last one at the 
> end of 2019, presumably)? I too would like to know the intended use of the 
> extra time.
> 
> Top-posted from my Windows Phone
> 
> -Original Message-
> From: "Benjamin Peterson" 
> Sent: ‎11/‎29/‎2016 0:04
> To: "Raymond Hettinger" ; "Serhiy Storchaka" 
> 
> Cc: "Python-Dev@Python. Org" 
> Subject: Re: [Python-Dev] Python 2.7.13 release dates
> 
> Okay, by popular demand, 2.7.13 now happens in January.
> 
> I'm curious what people are planning to do to 2.7 with the extra 5
> weeks. The 2.7 branch is a place to put occasional conservative bug
> fixes, which we aggregate and release every 6 months. It shouldn't
> really need special attention or become less stable depending on the
> release stage of Python 3.
> 
> On Mon, Nov 28, 2016, at 20:50, Raymond Hettinger wrote:
>>
>>> On Nov 28, 2016, at 10:36 AM, Serhiy Storchaka  wrote:
>>>
>>> On 28.11.16 09:06, Benjamin Peterson wrote:
 I've have just updated PEP 373 to say that Python 2.7.13 release
 candidate 1 will be released on December 3. The final will follow two
 weeks later on December 17. If there are delays in the process, the
 final will likely to pushed into January.
>>>
>>> Could it be delayed until 3.6.0 released? I paused fixing non-critical and 
>>> non-documentation bugs while 3.6 in pre-release stage and this could 
>>> include bugs that affect 2.7.
>>>
>>> In additional, we always receive increased number of bug reports in the 
>>> first one or two weeks after releasing new Python version. Some of these 
>>> reports are about regressions introduced by bugfixes. If delay bugfix 
>>> releases after new version release, we could fix regressions caused by 
>>> backported bugfixes and make bugfix releases more reliable.
>>
>> +1 on delaying 2.7.13 for a bit.  As long as it doesn't muck up
>> Benjamin's schedule, the extra time would be helpful (Python 3.6.0 got
>> all the focus recently).
>>
>>
>> Raymond
>> ___
>> Python-Dev mailing list
>> Python-Dev@python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org
> 
> 
> 
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/doko%40ubuntu.com
> 

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython: Revert unintended merge

[Matthias Klose]

On 04.12.2016 00:19, Martin Panter wrote:
> On 3 December 2016 at 22:15, Steve Dower  wrote:
>> On 03Dec2016 1312, Serhiy Storchaka wrote:
>>>
>>> On 03.12.16 22:13, steve.dower wrote:

 https://hg.python.org/cpython/rev/a60767015bed
 changeset:   105436:a60767015bed
 user:Steve Dower 
 date:Sat Dec 03 12:12:23 2016 -0800
 summary:
   Revert unintended merge
>>>
>>>
>>> I suppose it should be reverted in the 3.6 branch too.
>>
>>
>> Maybe, but I didn't make the change in the 3.6 branch, so I have no idea
>> whether it is meant to be there or not. But it wasn't part of the change I
>> made, so I didn't want to merge it forward.
> 
> This change comes from Matthias (Doko), and was originally only in the
> 3.5 branch:
> https://hg.python.org/cpython/rev/14c80065c36e
> 
> I presume it was left unmerged until there is a branch for 3.6.1 to
> merge into. So Steve, when you did your first 3.5 → 3.6 merge, you
> merged Doko’s change into 3.6:
> 
> https://hg.python.org/cpython/rev/5171bd86a36f
> 
>> Someone who actually understands the implications of changing these files
>> (config.guess, config.sub) can figure it out :)

sorry about the confusion.  Noticed by other people as well.  The change should
be safe, only adding support for new targets which shouldn't affect current
architectures.  Now staying away from any other 3.5 updates until 3.6 is
released.  I'm usually trying to update these files before releases but maybe
was a bit late this time.

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [python-committers] Should I delay 3.5.3 and 3.4.6 by two weeks?

[Matthias Klose]

On 19.12.2016 06:26, Larry Hastings wrote:
> 
> 
> Python 3.6.0 final just slipped by two weeks.  I scheduled 3.5.3 and 3.4.6 to
> ship about a month after 3.6.0 did, to "let the dust settle" around the
> release.  I expect a flood of adoption of 3.6, and people switching will find
> bugs, and maybe those bugs are in 3.5 or 3.4.  So it just seemed sensible.
> 
> 3.6 just slipped by two weeks.  So now there's less than two weeks between 
> 3.6.0
> final shipping and tagging the release canddiates for 3.5.3 and 3.4.6.  This
> isn't as much time as I'd like.
> 
> If I had total freedom to do as I liked, I'd slip my releases by two weeks to
> match 3.6.  But there might be people planning around 3.5.3 and 3.4.6--like
> Guido was waiting for 3.5.3 for something iirc.
> 
> So, if you have an opinion, please vote for one of these three options:
> 
>  * Don't slip 3.5.3. and 3.4.6.
>  * Slip 3.5.3 and 3.4.6 by two weeks to match 3.6.0.
>  * Slip 3.5.3 and 3.4.6 by a whole month, to give 3.6.0 the ability to
>slip again without us having to change the release.

I would appreciate a 3.5.3 release which doesn't slip, or which only slips by a
week, to be available before the Debian freeze.  Neither Debian nor Ubuntu ship
the 3.4 branch anymore, so for 3.4 I'm fine with any solution.

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Help requested with Python 2.7 performance regression

[Matthias Klose]

On 01.03.2017 18:51, Antoine Pitrou wrote:
> As for the high level: what if the training set used for PGO in Xenial
> has become skewed or inadequate?

running the testsuite

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Re: Accepting PEP 602 -- Annual Release Cycle for Python

[Matthias Klose]

On 30.10.19 20:26, Brett Cannon wrote:

This was discussed on https://discuss.python.org


I appreciate that you are informing the python-dev ML. However this discussion 
was never announced on the ML.  I assume this is a kind of thing that makes the 
ML obsolete and forces everyone into discourse.



Matthias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TI6GFVMG7UH7LV4S7LUDL4K5KG5LIZNU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Accepting PEP 602 -- Annual Release Cycle for Python

[Matthias Klose]

On 30.10.19 22:22, Barry Warsaw wrote:

On Oct 30, 2019, at 12:50, Matthias Klose  wrote:


On 30.10.19 20:26, Brett Cannon wrote:

This was discussed on https://discuss.python.org


I appreciate that you are informing the python-dev ML. However this discussion 
was never announced on the ML.  I assume this is a kind of thing that makes the 
ML obsolete and forces everyone into discourse.


Doko, as to the substance of PEP 602, does the annual release cadence in 
October cause problems for Ubuntu?


It's around four months before the Ubuntu feature freeze [1], and six before the 
release in April (which every two years usually becomes a long term support 
release). Having a window of four months for the adoption, with new third party 
releases for 3.8/3.9 trickling in, might be a bit short.


The adoption of 3.8 during this time frame might be more dominated by the 
removal of Python2 packages [2] (3300 sources affected, 1900 still to do).  And 
having fun with third party packages without having 2.7 and 3.8 support in the 
same release, with the Py2 removal having a top to bottom order for 
dependencies, and the 3.8 addition a bottom to top order.


We will see in 2021, how the distros will do with that work load and 3.9.

An August or September release might help, but I think it's more the shortened 
release cycle which has a bigger effect.


Matthias

[1] https://wiki.ubuntu.com/FocalFossa/ReleaseSchedule
[2] 
http://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=py2removal;users=debian-pyt...@lists.debian.org

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZCQKOOH5O4YQH7HXXM4VU4MTCPOWFMZT/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: [bpo-22699] Cross-compiling and fixing sysconfig

[Matthias Klose]

distutils is pretty indifferent to whether it's building native or cross. The
idea is to run the host python with the target's sysconfig to provide the tools
needed for cross compilation.  Yes, there are problems if third-party modules
are using the sys module to reason about the target architecture.

On 3/19/20 9:34 PM, Ivan Pozdeev via Python-Dev wrote:
> Last time I checked, distutils didn't support compilation for anything but the
> running Python instance, nor was it intended to. Should it?
> If not, the efforts look misplaced, you should rather use a toolchain that 
> does...
> 
> On 19.03.2020 23:22, Steve Dower wrote:
>> So over on https://bugs.python.org/issue22699 I've been talking to myself as 
>> I
>> figure out all the ways that cross-compiling (on Ubuntu, targeting another
>> Linux distro via an SDK) fails.
>>
>> In short, it's either because sysconfig can't provide details about any
>> CPython install other than the currently running one, or it's because our
>> setup.py (for building the CPython extension modules) uses sysconfig.
>>
>> Either way, I'm not about to propose a rewrite to fix either of them without
>> finding those who are most involved in these areas. Please come join me on 
>> the
>> bug.
>>
>> Thanks,
>> Steve
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/6DBYMDCDLOS245XK57BD3E2GXGVDMBPX/
>>
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YOM5OZ5UGLP3WP6ZLNXH2SU7WJSELPBM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How to customize CPython to a minimal set

[Matthias Klose]

On 7/20/20 10:30 AM, Huang, Yang wrote:
> 
> Hi, all
> 
> There is a request to run python in a Linux-based embedded resource 
> constrained system with sqlite3 support.
> 
> So many features are not required, like posixmodule, signalmodule, hashtable 
> ...
> But seems there are some dependencies among the 
> Modules/Parser/Python/Objects/Programs...
> 
> Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less 
> syscalls the better) ? 
> Is it possible to do that?

CPython comes with the promise of all batteries included, however sometimes it
feels like a complete power plant.  For packaging purposes, most Linux distros
make the decision to ship CPython as a set of runtime packages, and a set of
packages used for development (and building C extensions). Breaking that down
for the Debian/Ubuntu packages comes down to these sizes (unpacked,
uncompressed, on x86_64):

minimal:
5516K   debian/libpython3.9-minimal
5856K   debian/python3.9-minimal

stdlib:
8528K   debian/libpython3.9-stdlib
624Kdebian/python3.9

development:
19468K  debian/libpython3.9-dev
25804K  debian/libpython3.9-testsuite
1232K   debian/python3-distutils
668Kdebian/python3-lib2to3
548Kdebian/python3.9-dev

extra modules:
1648K   debian/idle-python3.9
5208K   debian/python3.9-examples

132Kdebian/python3-gdbm
844Kdebian/python3-tk

What you don't see from the sizes, are the extra dependencies which add to the
size, e.g. the X stack for tk, or readline/curses/crypto for stdlib.

The "minimal" set may sound nice, however it's not used in practice, because the
set is not well defined, and it's difficult to keep the minimal set as a
self-contained set of stdlib modules, and the minimal usually keeps growing,
never shrinks.

It looks like this "minimal" set is even too much for your purposed, so you
would have to scale down this set even further.

Matthias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OZY63O24XP535PQYBAYNFXRF5S3PDV3E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 632: Deprecate distutils module

[Matthias Klose]

On 9/4/20 1:28 PM, Steve Dower wrote:
> Hi all.
> 
> setuptools has recently adopted the entire codebase of the distutils module, 
> so
> that they will be able to make improvements directly without having to rely on
> patching the standard library. As a result, we can now move forward with
> official deprecation (in 3.10) and removal (in 3.12) (noting that the 
> distutils
> docs already recommend switching to setuptools).

it would be nice to already announce that with the 3.9 release.

At the 2018 Language summit, I had a lightning talk to report about the
experience splitting out distutils into a separate binary package, showing some
unexpected usages:

Unexpected / Creative usages:

 - distutils.version
   Used “everywhere” ...

 - distutils.spawn: find_executable
   Replace with shutil.which

 - distutils.util: strtobool
   Rewrite, no equivalent in the stdlib?

 - distutils.sysconfig:
   Mostly replaced by sysconfig

It really would be nice to have recommended replacements, especially for the
version stuff (packaging?)

Matthias
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5L45GDUIALRJP6BJ5H6B3V4LFJJ2P3UU/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-Dev] PEP 382: Namespace Packages

[Matthias Klose]

Martin v. Löwis schrieb:
> I propose the following PEP for inclusion to Python 3.1.
> Please comment.
> 
> Regards,
> Martin
> 
> Abstract
> 
> 
> Namespace packages are a mechanism for splitting a single Python
> package across multiple directories on disk. In current Python
> versions, an algorithm to compute the packages __path__ must be
> formulated. With the enhancement proposed here, the import machinery
> itself will construct the list of directories that make up the
> package.

+1

speaking as a downstream packaging python for Debian/Ubuntu I welcome this
approach.  The current practice of shipping the very same file (__init__.py) in
different packages leads to conflicts for the installation of these packages
(this is not specific to dpkg, but is true for rpm packaging as well).

Current practice of packaging (for downstreams) so called "name space packages" 
is:

 - either to split out the namespace __init__.py into a separate
   (linux distribution) package (needing manual packaging effort for each
   name space package)

 - using downstream specific packaging techniques to handle conflicting files
   (diversions)

 - replicating the current behaviour of setuptools simply overwriting the
   file conflicts.

Following this proposal (downstream) packaging of namespace packages is made
possible independent of any manual downstream packaging decisions or any
downstream specific packaging decisions.

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Tools

[Matthias Klose]

Barry Warsaw schrieb:
> Someone (I'm sorry, I forgot who) asked me at Pycon about stripping out
> Demos and Tools.  I'm happy to remove the two I wrote - Tools/world and
> Tools/pynche - from the distribution and release them as separate
> projects (retaining the PSF license).   Should I remove them from both
> the Python 2.x and 3.x trunks?

+1, but please for 2.7 and 3.1 only.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [Distutils] request for comments - standardization of python's purelib and platlib

[Matthias Klose]

On 14.08.2009 10:02, Tarek Ziadé wrote:

On Thu, Aug 13, 2009 at 9:22 PM, Brett Cannon  wrote:



On Thu, Aug 13, 2009 at 11:23, Jan Matejek  wrote:


Hello,

I'm cross-posting this to distributi...@freedesktop and python-dev,
because the topic is relevant to both groups and should be solved in
cooperation.

The issue:

In Python's default configuration (on linux), both purelib (location for
pure python modules) and platlib (location for platform-dependent binary
extensions) point to $prefix/lib/pythonX.Y/site-packages.
That is no good for two main reasons.

One, python depends on the "lib" directory. (from distro's point of
view, prefix is /usr, so let's talk /usr/lib) Due to this, it's
impossible to install python under /usr/lib64 without heavy patching.
Repeated attempts to bring python developers to acknowledge and rectify
the situation have all failed (common argument here is "that would mean
redesign of distutils and huge parts of whatnot").


This is now Tarek's call, so this may or may not have changed in terms of
what the (now) distutils maintainer thinks.



I don't recall those repeated attempts , but I've been around for less
than two years.

You are very welcome to come in the Distutils-SIG ML to discuss these matters.
I'm moving the discussion there.

Among the proposals you have detailed, the sharedir way seems like the
most simple/interesting
one (depending on you answer to Brett's question )


The approach of splitting the installation into two different locations seems to 
be wrong, it changes the semantics for imports of python packages which are not 
installed in the same location. Simplest counter example is the use of relative 
imports, which will fail if the imported module/extension is not found in the 
same paths.


Other languages like Perl or Java don't have relative imports, or they map all 
components on the "path" into one logical path so you don't have this kind of 
problem.


I don't see an explict statement that code really has to live inside /usr/share, 
and even generated .py files differ depending on the architecture you build for 
(e.g. sip, qt bindings).


  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 3147, __pycache__ directories and umask

[Matthias Klose]

On 23.03.2010 02:28, Ben Finney wrote:

Antoine Pitrou  writes:


Barry Warsaw  python.org>  writes:


When Python is being installed, either by a from-source 'make
install' or by the distro packager, then you'd expect the umask not
to be insane. In the latter case, it's a bug and in the former case
you screwed up so you should be able to delete and reinstall, or at
the very least execute the same `find` command that Python's own
Makefile calls (in my branch) for 'make clean'.

When you're installing packages, again, I would expect that the
system installer, or you via `easy_install` or whatever, would not
have an insane umask.


Well, precisely. That's why I suggest that creating the __pycache__
directories be done *at install time* (or packaging time), and not via
the core import machinery (that is, not at import time). That is, when
you *know* you are the right user, with the right umask.


+1.

Taking advantage of caching directories that have already been set up
correctly in advance at install time is friendly. Littering the runtime
directory with new subdirectories by default is not so friendly.

Perhaps also of note is that the FHS recommends systems use
‘/var/cache/foo/’ for cached data from applications:

 /var/cache : Application cache data

 Purpose

 /var/cache is intended for cached data from applications. Such data is
 locally generated as a result of time-consuming I/O or calculation. The
 application must be able to regenerate or restore the data. Unlike
 /var/spool, the cached files can be deleted without data loss. The data
 must remain valid between invocations of the application and rebooting
 the system.

 
http://www.debian.org/doc/packaging-manuals/fhs/fhs-2.3.html#VARCACHEAPPLICATIONCACHEDATA>

This would suggest that Python could start using ‘/var/cache/python/’
for its cached bytecode tree on systems that implement the FHS.


it reads *data*, not code.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] autoconf update to 2.65 and cleanup

[Matthias Klose]

configure is still generated by 2.61; would it be possible to update to 2.65? 
The cr_lf issue mentioned in [1] seems to be resolved, ac_cr is now defined as


  ac_cr=`echo X | tr X '\015'`

Proposing to

 - fix some quoting in help strings and code snippets (#8509)
 - update to autoconf 2.65 (#8510)
 - convert obsolete macros (AC_HELP_STRING, AC_TRY_*, AC_AIX, ...)
   one by one (tracking these in separate reports).

Could this be done for both trunk and py3k branch, even if 2.7 already is in 
beta?


  Matthias

[1] http://mail.python.org/pipermail/python-dev/2008-November/083781.html
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] pybuildbot.identify?

[Matthias Klose]

On 19.05.2010 18:09, Bill Janssen wrote:

The PPC buildbots are running pretty well, now that I've opened a few
more ports, but I'd like to find this script "pybuildbot.identify" that
they keep complaining about, and install it.  I've poked around the
Python sources, but haven't found it.

Anyone know where to get it from?


it's just a custom script, currently very linux specific. attached here. maybe 
it's worth to check it in and extend it for other configurations.


  Matthias
#! /bin/sh

for arg in "$@"; do
case "$arg" in
*=*) eval $arg;;
esac
done

echo EXTRATESTOPTS: $EXTRATESTOPTS

echo  system information 
uname -a

if which getconf >/dev/null 2>&1; then
echo
echo "Processors: $(getconf _NPROCESSORS_ONLN)"
fi

if [ -f /proc/meminfo ]; then
cat /proc/meminfo | grep ^Mem
fi

if which df >/dev/null 2>&1; then
echo "Free disk (buildarea): $(df -h . | awk 'NR==2 {print $4}')"
echo "Free disk (/tmp): $(df -h /tmp | awk 'NR==2 {print $4}')"
fi

if which lsb_release >/dev/null 2>&1; then
echo
lsb_release -d -r
fi

if [ -n "$CC" ]; then
echo
$CC --version
fi

# Debian/Ubuntu specific
if which dpkg-architecture >/dev/null 2>&1; then
bds="libc*-dev gcc-?.? binutils"
bds="$bds libreadline*-dev libncurses*-dev tcl*-dev tk*-dev libdb*-dev"
bds="$bds zlib*-dev libgdbm-dev blt-dev libssl-dev libbz*-dev"
bds="$bds libbluetooth-dev libsqlite*-dev libffi*-dev libgpmg1"
dpkg-architecture -a | grep DEB_BUILD_GNU
echo
dpkg -l $bds | grep ^ii
fi

echo  end system information 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Matthias Klose]

On 25.06.2010 22:12, James Y Knight wrote:


On Jun 25, 2010, at 4:53 AM, Scott Dial wrote:


On 6/24/2010 8:23 PM, James Y Knight wrote:

On Jun 24, 2010, at 5:53 PM, Scott Dial wrote:

If the package has .so files that aren't compatible with other version
of python, then what is the motivation for placing that in a shared
location (since it can't actually be shared)


Because python looks for .so files in the same place it looks for the
.py files of the same package.


My suggestion was that a package that contains .so files should not be
shared (e.g., the entire lxml package should be placed in a
version-specific path). The motivation for this PEP was to simplify the
installation python packages for distros; it was not to reduce the
number of .py files on the disk.

Placing .so files together does not simplify that install process in any
way. You will still have to handle such packages in a special way.



This is a good point, but I think still falls short of a solution. For a
package like lxml, indeed you are correct. Since debian needs to build
it once per version, it could just put the entire package (.py files and
.so files) into a different per-python-version directory.


This is what is currently done.  This will increase the size of packages by 
duplicating the .py files, or you have to install the .py in a common location 
(irrelevant to sys.path), and provide (sym)links to the expected location.


A "different per-python-version directory" also has the disadvantage that file 
conflicts between (distribution) packages cannot be detected.



However, then you have to also consider python packages made up of
multiple distro packages -- like twisted or zope. Twisted includes some
C extensions in the core package. But then there are other twisted
modules (installed under a "twisted.foo" name) which do not include C
extensions. If the base twisted package is installed under a
version-specific directory, then all of the submodule packages need to
also be installed under the same version-specific directory (and thus
built for all versions).

In the past, it has proven somewhat tricky to coordinate which directory
the modules for package "foo" should be installed in, because you need
to know whether *any* of the related packages includes a native ".so"
file, not just the current package.

The converse situation, where a base package did *not* get installed
into a version-specific directory because it includes no native code,
but a submodule *does* include a ".so" file, is even trickier.


I don't think that installation into different locations based on the presence 
of extension will work.  Should a location really change if an extension is 
added as an optimization?  Splitting a (python) package into different 
installation locations should be avoided.


  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Matthias Klose]

On 26.06.2010 02:19, Nick Coghlan wrote:

On Sat, Jun 26, 2010 at 6:12 AM, James Y Knight  wrote:

However, then you have to also consider python packages made up of multiple
distro packages -- like twisted or zope. Twisted includes some C extensions
in the core package. But then there are other twisted modules (installed
under a "twisted.foo" name) which do not include C extensions. If the base
twisted package is installed under a version-specific directory, then all of
the submodule packages need to also be installed under the same
version-specific directory (and thus built for all versions).

In the past, it has proven somewhat tricky to coordinate which directory the
modules for package "foo" should be installed in, because you need to know
whether *any* of the related packages includes a native ".so" file, not just
the current package.

The converse situation, where a base package did *not* get installed into a
version-specific directory because it includes no native code, but a
submodule *does* include a ".so" file, is even trickier.


I think there are two major ways to tackle this:
- allow multiple versions of a .so file within a single directory (i.e
Barry's current suggestion)


we already do this, see the naming of the extensions of a python debug build on 
Windows.  Several distributions (Debian, Fedora, Ubuntu) do use this as well to 
provide extensions for python debug builds.



- enhanced namespace packages, allowing a single package to be spread
across multiple directories, some of which may be Python version
specific (i.e. modifications to PEP 382 to support references to
version-specific directories)


this is not what I want to use in a distribution.  package management systems 
like rpm and dpkg do handle conflicts and replacements of files pretty well, 
having the same file in potentially different locations in the file system 
doesn't help detecting conflicts and duplicate packages.


  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Matthias Klose]

On 24.06.2010 22:46, Barry Warsaw wrote:

On Jun 24, 2010, at 02:28 PM, Barry Warsaw wrote:


On Jun 24, 2010, at 01:00 PM, Benjamin Peterson wrote:


2010/6/24 Barry Warsaw:

On Jun 24, 2010, at 10:58 AM, Benjamin Peterson wrote:


2010/6/24 Barry Warsaw:

Please let me know what you think.  I'm happy to just commit this to the
py3k branch if there are no objections.  I don't think a new PEP is
in order, but an update to PEP 3147 might make sense.


How will this interact with PEP 384 if that is implemented?

I'm trying to come up with something that will work immediately while PEP 384
is being adopted.


But how will modules specify that they support multiple ABIs then?


I didn't understand, so asked Benjamin for clarification in IRC.

  barry: if python 3.3 will only load x.3.3.so, but x.3.2.so supports
   the stable abi, will it load it?  [14:25]
  gutworth: thanks, now i get it :)  [14:26]
  gutworth: i think it should, but it wouldn't under my scheme.  let me
think about it


So, we could say that PEP 384 compliant extension modules would get written
without a version specifier.  IOW, we'd treat foo.so as using the ABI.  It
would then be up to the Python runtime to throw ImportErrors if in fact we
were loading a legacy, non-PEP 384 compliant extension.


Is it realistic to never break the ABI?  I would think of having the ABI encoded 
in the file name as well, and only bump the ABI if it does change.  With the 
"versioned .so files" proposal an ABI bump is necessary with every python 
version, with PEP 384 the ABI bump will be decoupled from the python version.


  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FHS compliance of Python installation

[Matthias Klose]

On 25.06.2010 02:54, Ben Finney wrote:

James Y Knight  writes:


Really, python should store the .py files in /usr/share/python/, the
.so files in /usr/lib/x86_64- linux-gnu/python2.5-debug/, and the .pyc
files in /var/lib/python2.5- debug. But python doesn't work like that.


+1

So who's going to draft the “Filesystem Hierarchy Standard compliance”
PEP? :-)


This has nothing to do with the FHS.  The FHS talks about data, not code.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] FHS compliance of Python installation

[Matthias Klose]

On 26.06.2010 22:30, C. Titus Brown wrote:

On Sat, Jun 26, 2010 at 10:25:28PM +0200, Matthias Klose wrote:

On 25.06.2010 02:54, Ben Finney wrote:

James Y Knight   writes:


Really, python should store the .py files in /usr/share/python/, the
.so files in /usr/lib/x86_64- linux-gnu/python2.5-debug/, and the .pyc
files in /var/lib/python2.5- debug. But python doesn't work like that.


+1

So who's going to draft the ???Filesystem Hierarchy Standard compliance???
PEP? :-)


This has nothing to do with the FHS.  The FHS talks about data, not code.


Really?  It has some guidelines here for object files, etc., at least as
of 2004.

http://www.pathname.com/fhs/pub/fhs-2.3.html

A quick scan suggests /usr/lib is the right place to look:

http://www.pathname.com/fhs/pub/fhs-2.3.html#USRLIBLIBRARIESFORPROGRAMMINGANDPA


agreed for object files, but
http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA
explicitely states "The /usr/share hierarchy is for all read-only architecture 
independent *data* files".

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Matthias Klose]

On 25.06.2010 20:58, Brett Cannon wrote:

On Fri, Jun 25, 2010 at 01:53, Scott Dial

Placing .so files together does not simplify that install process in any
way. You will still have to handle such packages in a special way. You
must still compile the package multiple times for each relevant version
of python (with special tagging that I imagine distutils can take care
of) and, worse yet, you have created a more trick install than merely
having multiple search paths (e.g., installing/uninstalling lxml for
*one* version of python is actually more difficult in this scheme).


This is meant to be used by distros in a programmatic fashion, so my
response is "so what?" Their package management system is going to
maintain the directory, not a person. You and I are not going to be
using this for anything. This is purely meant for Linux OS vendors
(maybe OS X) to manage their installs through their package software.
I honestly do not expect human beings to be mucking around with these
installs (and I suspect Barry doesn't either).


Placing files for a distribution in a version-independent path does help 
distributions handling file conflicts, detecting duplicates and with moving 
files between different (distribution) packages.


Having non-conflicting extension names is a schema which already is used on some 
platforms (debug builds on Windows).  The question for me is, if just a renaming 
of the .so files is acceptable for upstream, or if distributors should implement 
this on their own, as something like:


  if ext_path.startswith('/usr/') and not ext_path.startswith('/usr/local/'):
load_ext('foo.2.6.so')
  else:
load_ext('foo.so')

I fear this will cause issues when e.g. virtualenv environments start copying 
parts from the system installation instead of symlinking it.


  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Matthias Klose]

On 07.07.2010 20:40, Barry Warsaw wrote:

Getting back to this after the US holiday.  Thanks for running these numbers
Scott.  I've opened a bug in the Python tracker and attached my latest patch:

http://bugs.python.org/issue9193

The one difference from previous versions of the patch is that the .so tag is
now settable via "./configure --with-so-abi-tag=foo".  This would generate
shared libs like _multiprocessing.foo.so.


 - imo, it's wrong to lookup _multiprocessing.so first, before looking
   up _multiprocessing.foo.so (at least for the use case to put the
   extensions for multiple python versions into one directory).

 - why is the flexibility of specifying the "foo" needed?  The
   naming for the __pycache__ files is fixed, why have it configurable
   for extensions?

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Matthias Klose]

On 15.07.2010 01:59, Barry Warsaw wrote:

PEP 384 describes a change to ``PyModule_Create()`` where ``3`` is
passed as the API version if the extension was complied with
``Py_LIMITED_API``.  This should be formalized into an official macro
called ``PYTHON_ABI_VERSION`` to mirror ``PYTHON_API_VERSION``.  If
and when the ABI changes in an incompatible way, this version number
would be bumped.  To facilitate sharing, Python would be extended to
search for extension modules with the ``PYTHON_ABI_VERSION`` number in
its name.  The prefix ``abi`` is reserved for Python's use.

Thus for example, an initial implementation of PEP 384, compiled with
`--with-so-abi-tag=cpython-xy` would search for the following file
names when extension module `foo` is imported (in this order)::

 foo.abi3.so
 foo.cpython-xy.so
 foo.so

The distutils [7]_ ``build_ext`` command would also have to be
extended to compile to shared library files with the ``abi3`` tag,
when the module author indicates that their extension supports that
version of the ABI.  This could be done in a backward compatible way
by adding a keyword argument to the ``Extension`` class, such as::

 Extension('foo', ['foo.c'], abi=3)


I like the proposal, but IMO it is too unspecific about the abi tag.  Assume 
that an extension is built with such a configured python and then tried to run 
with an abi compatible configured python, but with a slightly different version 
tag, the extension won't be found.  Differing file names per configuration 
should be avoided. Proposing


  1) Remove the configure option and use the new naming using the tag for
 all configurations unconditionally.  Everybody can expect the same
 name on every installation, and PEP 384 most likely will require
 using a tag too.
 As Amaury did point out, there is a use case in that PyPy can use this
 tag too to build extensions only usable by PyPy.

  2) As PEP 3147 defines a non-configurable name for .pyc files, this PEP
 should define a non-configurable way for the tag. The tag should
 include all information which currently makes an extension ABI
 incompatible:
  - the python implementation (cpython, PyPy, ...)
  - the python version (3,2, 3.3, ...)
  - unicode configure option (--with-wide-unicode, 16 or 32)
  - platform information (necessary?)
 If this list changes for coming python versions, then it can be
 extended.
 Barry pointed out on irc chat that people might want to build
 experimental ABI incompatible versions, which should have its
 own tag. If this is wanted, please provide a configure option
 which lets extend/append to the tag.

  3) In case that 1) is not acceptable, the --with-so-abi-tag option
 should be implemented in such a way that it isn't required to
 take any arguments, and in this case to default to the fixed
 naming schema described in 2).

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Matthias Klose]

On 16.07.2010 15:43, Daniel Stutzbach wrote:

On Fri, Jul 16, 2010 at 5:40 AM, Matthias Klose  wrote:


  2) As PEP 3147 defines a non-configurable name for .pyc files, this PEP
 should define a non-configurable way for the tag. The tag should
 include all information which currently makes an extension ABI
 incompatible:
  - the python implementation (cpython, PyPy, ...)
  - the python version (3,2, 3.3, ...)
  - unicode configure option (--with-wide-unicode, 16 or 32)
  - platform information (necessary?)



I'm not sure it's that easy to enumerate all of the ways to end up with an
incompatible ABI.  There are quite a lot of configure options listed with
"configure --help", and it's not always obvious if an option changes the
ABI.  On top of that, there are compilation flags that can change the ABI,
as Kristján discovered in the following thread:


right, I forgot about the debug builds, because it's already the standard on 
windows to build foo_d.so extensions, and I adopted it for Debian/Ubuntu too.



On the flip side, a fully enumerated ABI signature could be used to identify
(in)compatible binary eggs, which is basically impossible now.


indeed.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] versioned .so files for Python 3.2

[Matthias Klose]

On 26.07.2010 22:53, Ralf Schmitt wrote:

Barry Warsaw  writes:

That's fine, but it's not the way Debian/Ubuntu works today.  PEP 3149
adoption will definitely remove significant complication for deploying
multiple Python versions at the same time on those systems.


You're just moving that complication into python.


There is nothing which prevents you to still deploy/use python modules in 
separate directories, and if you see a python package as a directory, nothing 
will change for you with this PEP besides the naming of the extensions.


> I'd much prefer to have cleanly separated environments by having
> separate directories for my python modules.

That is your preference, but not what standards like the FHS talk about (i.e. 
having different locations for data, docs, headers).


> Sharing the source code and
> complicating things will not lead to increased robustness.

Not true. Package managers like dpkg/apt-get, rpm/yum and maybe others do this 
for ages. And yes, the added "complexity" of package managers does lead to 
increased robustness.


  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python-3 transition in Arch Linux

[Matthias Klose]

On 04.11.2010 21:12, "Martin v. Löwis" wrote:

To clarify (but I dont speak for the rest of #python, just myself), I
think the move was premature, but I don't use Arch and I don't know what
typical Arch users expect. The reason I think it's premature is that
'python2' just doesn't work everywhere, and I would have gone for a
transitionary period where '/usr/bin/python' is something that screams
loudly that it shouldn't be used before it executes 'python2'.


Iirc, it was an explicit decision made at the 2009 language summit not to 
introduce a python2 symlink, but using python3 for python3.x instead. 
Debian/Ubuntu don't ship a python2 symlink by intent.


Did the plans change, i.e. are there plans to provide a python symlink for 
python 3.x altinstall in a future release, e.g in 3.4 or 3.5?


  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Question about GDB bindings and 32/64 bits

[Matthias Klose]

On 26.11.2010 05:11, Jesus Cea wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

I have installed GDB 7.2 32 bits and 32 bits buildslaves are green.
Nevertheless 64 bits buildslaves are failing test_gdb.

Is there any expectation that a 32 bits GDB be able to debug a 64 bits
python?. If not, gdb test should compare "platform.architecture()" (for
python and gdb in the system) and run only when they are the same.


that would be too restrictive, as an 64bit gdb is able to handle 32bit binaries 
too.


If
this should work, I would open a bug and maybe spend some time with it.

But before thinking about investing time, I would like to know if this
mix is actually expected or not to work.

If not, I would consider to install a 64 bits GDB too and do some tricks
(like using an "/usr/local/bin/gdb" script wrapper to choose 32/64
"real" gdb version) to actually execute "test_gdb" in both buildslaves
(they are running in the same physical machine).


yes, and then you should be able to use this gdb for both 32 and 64bit builds. 
No need for a wrapper (Such a gdb is available in the gdb64 package on 
Debian/Ubuntu).


  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 384 final review

[Matthias Klose]

On 29.11.2010 00:40, "Martin v. Löwis" wrote:

I have now completed

http://www.python.org/dev/peps/pep-0384/

Benjamin has volunteered to rule on this PEP.

Please comment with any changes you want to see, or speak in
favor or against this PEP.


I looked at a diff with r84330 from the py3k branch.

Extensions built with Py_LIMITED_API have the python version encoded in it's 
name.  Which abi name should be used for these extensions?


 - The m and u modifiers in the abi name are complimentary (?)
 - debug builds and Py_LIMITED_API are incompatible (?) and therefore
   the current name should be used?
 - For posix systems the implementation is currently part of the abi name,
   are Py_LIMITED_API extensions supposed to be compatible with e.g. PyPy?
   Should the LIMITED_API abi name include the implementation string?
 - Should the distutils support for LIMITED_API be part of the pep, or
   be implemented later?

In favour of the pep.

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 384 final review

[Matthias Klose]

On 29.11.2010 14:14, Éric Araujo wrote:

Hello,


Please comment with any changes you want to see, or speak in
favor or against this PEP.


How to get a diff between py3k and this branch?


I used
svn diff svn://svn.python.org/python/branches/p...@84330 
svn://svn.python.org/python/branches/pep-0384

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 384 accepted

[Matthias Klose]

On 03.12.2010 00:25, Tarek Ziadé wrote:

2010/12/2 "Martin v. Löwis":

No, only the ones that didn't cause backwards incompatibilities,
and broke existing packages.


This is impossible. I can point you to some third party project that
can break if you touch some distutils internals, like setuptools.
Setuptools also uses some privates global variables in some other
modules in the stdlib FYI.


So what would break if Extension accepted an abi= keyword parameter?


I suppose you have code behind this, that will be in build_ext and in
the compilers. So you will need to try out ALL projects out there that
customize build_ext, like numpy or setuptools, etc, But you won't be
able to try out all projects because they are not listed somewhere.


is this necessary?  are all these projects known to work with 3.2, without 
having changes compared to 3.1 *without* this pep?  hardly ...


how many extensions will use this restricted api at all?  Is it a legitimate 
solution to back up building an extension in the "default" mode?


even without having any changes in distutils it would make sense to know if an 
extension can be built with the restricted ABI, so maybe it is better to defer 
any changes to the extension soname, and provide a check for an extension if it 
conforms to the restricted ABI, even if the extension still uses the python 
version specific soname.


I did not mean to block this pep by choosing any installation names.

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream

[Matthias Klose]

On 02.03.2011 16:54, Nick Coghlan wrote:
> /tangent
> 
> Does this discussion remind anyone else of the bash/dash switch for
> /usr/bin/sh in Ubuntu?
> 
> The distro itself coped fine, but 3rd party shell scripts that used
> bash extensions were a whole different story.
> 
> (No, I'm not sure what lessons, if any, we can draw from that. It just
> struck me as an interesting parallel worth mentioning)

I think that comparison is wrong.  /bin/sh is defined to point to a Bourne
compatible shell.  If you use bash features, and you are not using /bin/bash,
then you are calling for trouble.  With /bin/sh, there is a standard what to
expect, and you can write /bin/sh scripts which conform to all /bin/sh
implementations.

Compare that to /usr/bin/python and try to define how to write a script that
works with reasonable new 2.x and 3.x python versions.  Maybe there is a small
subset of scripts (not using any third party python package), but how would such
a "standard" be useful?  Then even better point /usr/bin/python to
/usr/bin/perl6, then at least all scripts break consistently ;)

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream

[Matthias Klose]

On 04.03.2011 08:44, Kerrick Staley wrote:

[looking at version 88755 of the draft]

+1 on anything what is said about python2 (still remembering the unsuccessful
proposal from one of the Chicago language summits).

I do not like the vagueness about the python link.  Sounds like "It may point to
this or that, but it might change, and it might break, maybe we'll change our
position later, in some years".

This recommendation is imo wrong:

  For the time being, it is recommended that python should refer to python2,
  except on distributions which include only python3 in their base install, or
  those that wish to push strongly for migration of user scripts to Python 3.

Maybe some distributions already did decide on this, but it's definitely not
something which will help compatibility across distributions.

Further down in a note,

  It is suggested that even distribution-specific packages follow the
  python2/python3 convention [...]

The note should be a recommendation, and the recommendation a note.  Cross
platform compatibility should be recommended.

The rationale of the proposal only gives a rationale for the python2 link, not
for the vagueness of the python link, and when to use it, e.g. to use the most
recent interpreter interactively (as suggested by Martin v. Loewis), or to only
use it for 2.x and 3.x compatible scripts.

>From a Debian/Ubuntu perspective I see python2.7 in the distribution for at
least two more (LTS) releases (i.e. the next 4-5 years) to run old code or code
abandoned by upstreams.  I only see extra work with this code, when changing
python to point to python3, while I do not see a disadvantage to use python3
(which was part of python3.x from the beginning) for the packaging inside a
distribution.

so -1 on the python link bits.

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns

[Matthias Klose]

Jeff Rush writes:
> I was in a Packaging BoF yesterday and, although not very relevant to the 
> packager bootstrap thread, Guido has asked me to post some of the concerns.

We did address many topics on both days, I added the following topics
which were addressed on the Friday BoF only, see
http://wiki.python.org/moin/PackagingBOF

- Linux distributions try to ship only one version of a
  package/egg/module in one release, only shipping more than one
  version if necessary. eggs (as least as shipped with Debian, Fedora,
  Ubuntu) are all built using --single-version-externally-managed.

   - import foo should work wether installed as an egg or installed
 with distutils, and without using pkg_resources.require
   - pkg_resources should handle the situation of one egg version
 installed as --single-version-externally-managed (default version) and
 one or more eggs installed not using
 --single-version-externally-managed. Currently these additional
 versions cannot be imported. 

- It would be useful if setuptools could handle separate build and
  install steps like most configure/make/make install systems do. Access
  to external resources should optionally be disabled during a build.

- The idea was brought up to use a to-be-defined api-version to
  describe dependencies between eggs. Version numbers are generally used
  for more than api changes; the idea follows existing practice for
  shared object names, only changing when the API is changed.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Capsule Summary of Some Packaging/Deployment Technology Concerns

[Matthias Klose]

Phillip J. Eby writes:
> >7. Many wanted to ability to install files anywhere in the install tree and
> > not just under the Python package.  Under distutils this was possible 
> > but
> > it was removed in setuptools for security reasons.
> 
> It wasn't security, it was manageability.  Egg-based installation 
> means containment, (analagous to GNU stow) and therefore portability 
> and disposability of plugins.  (Which again is what eggs were really 
> developed for in the first place.)

defining containment this way doesn't help when preparing eggs for
inclusion in a linux distribution.  E.g. users on these distributions
are used to find log files in /var/log (maybe in a subdir),
documentation in /usr/share/doc/.  You probably will get
different views about manageability depending on your background (used
to linux distribution standards or used to standards set by
setuptools/cheeseshop).  Packagers currently move these files manually
to the standard locations and often have to keep symlinks in the egg
dirs to these locations.  Installation on linux distributions is
handled by existing package tools which is unlikely to change.  So it
would be nice to find a common layer which can be used for both
distribution methods, optionally enabling this with some kind of
option like --install-files-in-places-not-handled-by-setuptools ;)

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] what version of bsddb to use/accept for 2.6/3.0

[Matthias Klose]

[CCing Clint]

4.7.25 generally looks fine, except for sparc (afaics); I asked the Debian bsddb
maintainer to have a look at the sparc problems. At least on sparc, there's no
difference between 4.6 and 4.7. I usually try to run the buildbots with the
recent software in the distribution, as this is what users will see with the
next release.

  Matthias

PS: currently travelling, won't be able to answer in time.

Neal Norwitz schrieb:
> Many buildbots are running bsddb 4.7, particularly the debian/ubuntu
> ones (4.7.25 which seems to be the latest).  Some of them are
> crashing, others are not.  The max version we support in both 2.6 and
> 3.0 is 4.7.  Should we allow this version or should we use a lower
> maximum that is more likely to work (ie, not crash)?
> 
> It looks like the WIndows buildbots use 4.4.20.  Unfortunately, the
> Windows bots aren't in great shape either.
> 
> Additionally, there are reference leaks in both 2.6 and 3.0:
>   test_bsddb3 leaked [80, 80] references, sum=160 (2.6)
>   test_bsddb3 leaked [63, 63] references, sum=126 (3.0)
> 
> It would be nice to get as many of these things fixed up before
> release.  Jesus, Greg, Trent, can you make some time over the next
> week to stabilize bsddb support?
> 
> Thanks,
> n
> PS.  To change the max version of bsddb supported in Unix, modify
> max_db_ver in setup.py.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.5.3: call for patches

[Matthias Klose]

Martin v. Löwis schrieb:
> Within a few weeks, we will release Python 2.5.3. This will be the last
> bug fix release of Python 2.5, afterwards, future releases of 2.5 will
> only include security fixes, and no binaries (for Windows or OSX) will
> be provided anymore (from python.org).
> 
> In principle, the release will include all changes that are already on
> the release25-maint branch in subversion [1]. If you think that specific
> changes should be considered, please create an issue in the bug tracker
> [2], and label it with the 2.5.3 version. Backports of changes that
> are already released in Python 2.6 but may apply to 2.5 are of
> particular interest.

I would like to apply fixes for some CVE's which are addressed in 2.5 but not
yet in 2.4. this would include

CVE-2007-4965
CVE-2008-1679
CVE-2008-1721
CVE-2008-2315
CVE-2008-3144
CVE-2008-1887
CVE-2008-4864

Matthias

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python 2.5.3: call for patches

[Matthias Klose]

Martin v. Löwis schrieb:
>> I would like to apply fixes for some CVE's which are addressed in 2.5 but not
>> yet in 2.4. this would include
>>
>> CVE-2007-4965
>> CVE-2008-1679
>> CVE-2008-1721
>> CVE-2008-2315
>> CVE-2008-3144
>> CVE-2008-1887
>> CVE-2008-4864
> 
> Can you identify the revisions that would need backporting?
> 
> I could only find (trunk revisions)
>   CVE-2007-4965: r65880
>   CVE-2008-1721: r62235, issue2586
>   CVE-2008-3144: issue2588, issue2589, r63734, r63728.
>   CVE-2008-1887: issue2587, r62261, r62271
>   CVE-2008-4864: r66689
> 
> So what about
> 
>   CVE-2008-1679: claimed to be issue1179 in the CVE, but
>  that says it fixes CVE-2007-4965 only?

the original fix for CVE-2007-4965 did miss two chunks, which are included in
r65878 on the 2.5 branch.

>   CVE-2008-2315

this is r65334 on the 2.5 branch and r65335 on the trunk:
Security patches from Apple:  prevent int overflow when allocating memory
this was already checked in, with an added NEWS item in 2.4.5. Moved this
to 2.4.6.

> In principle, this is fine with me, so go ahead.

Done.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Module/Setup.dist maintainance

[Matthias Klose]

I was experimenting with profiled builds and building more extensions as
builtins instead of loadable modules, noticing that Module/Setup.dist is not
kept updated. Currently only the section above the *shared* line is really used
for the build, but everything else doesn't seem to be updated (extensions are
missing or out of date). Is there another way to build extensions as builtins,
or should this file kept be up to date?

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (3.1): - rename configure.in to configure.ac

[Matthias Klose]

On 15.03.2012 11:31, Antoine Pitrou wrote:

On Wed, 14 Mar 2012 23:27:24 +0100
matthias.klose  wrote:

http://hg.python.org/cpython/rev/55ab7a272f0a
changeset:   75659:55ab7a272f0a
branch:  3.1
parent:  75199:df3b2b5db900
user:Matthias Klose
date:Wed Mar 14 23:10:15 2012 +0100
summary:
   - rename configure.in to configure.ac
- change references from configure.in to configure.ac


What's the rationale for this change? There doesn't seem to be an issue
number.


autoconf files up to 2.13 have the .in extension, autoconf files for 2.50 and 
newer the .ac extension.  This is a no change, except for having autoconf2.13 
installed, and the autoconf then failing.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Debian wheezy, amd64: make not finding files for bz2 and other packages

[Matthias Klose]

On 05.05.2012 16:13, Antoine Pitrou wrote:
> On Sat, 05 May 2012 16:04:40 +0200
> Christian Heimes  wrote:
>> Am 05.05.2012 15:39, schrieb Antoine Pitrou:
>>> On Sat, 05 May 2012 15:31:24 +0200
>>> Christian Heimes  wrote:
 Am 05.05.2012 12:36, schrieb Antoine Pitrou:
>
> Hello,
>
> On Fri, 04 May 2012 14:07:28 -0400
> "Edward C. Jones"  wrote:
>> Filelist of package libbz2-dev in wheezy of architecture amd64
>>
>> /usr/include/bzlib.h
>> /usr/lib/x86_64-linux-gnu/libbz2.a
>> /usr/lib/x86_64-linux-gnu/libbz2.so
>> /usr/share/doc/libbz2-dev
>
> setup.py probably doesn't search in the right paths for libbz2.so. I
> suggest you open a bug at http://bugs.python.org

 The issue might be caused by Debian's new multiarch libraries. In recent
 versions of Debian (and Ubuntu), 64bit and 32bit libraries can coexist
 on the same system.
>>>
>>> It probably is, but I thought Barry had tackled that in setup.py :-)
>>
>> The fix needs the dpkg-architecture program. As Tshepang pointed out it
>> may not be available on Edward's box. I always install build-essential
>> on all development boxes as it includes GCC, make and dpkg-dev.
> 
> Perhaps setup.py should detect that? It shouldn't be too hard to
> parse /etc/debian_version in order to know whether the system is
> multiarch-enabled. That would avoid confusing build failures.

IMO, the correct fix would be not to hard-code the system include and library
directories, but get them from gcc directly (if CC is gcc), and not relying on
dpkg-architecture.

$ gcc -v -E -  search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/4.6/include
 /usr/local/include
 /usr/lib/gcc/x86_64-linux-gnu/4.6/include-fixed
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
[...]
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/4.6/:/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../:/lib/:/usr/lib/

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cross-compiling patches

[Matthias Klose]

On 30.06.2012 23:17, Antoine Pitrou wrote:
> 
> Hello,
> 
> I think these patches are premature (they break compilation on OS X,
> and they break ctypes configure on my Linux box).

that was unrelated. fixed last night.

> Furthermore, they
> were committed post-beta, which means they should probably have waited
> for after the 3.3 release. So I propose for these commits to be
> reverted.

somebody (?) asked on irc that either Martin or I should update the internal
copies, and someone suggested to do the same with the expat sources. And I asked
our release manager if he was ok with this post beta change.

> (to be clear, I'm talking about all configure / Makefile / setup.py /
> libffi changes since and including
> http://hg.python.org/cpython/rev/e6e99d449bdc876fa57111e7e534c44ecbc3bcbd
> )

the first cross-build fixes went in in April, please consider these fixes for
the then incomplete cros-build fixes. The build issues you did see last night,
were fixed for the OS X build, and I reverted the update for the compiler search
paths for the native build. So please lets keep this cross-build support for
3.3. It's working, tested with a arm-linux-gnueabi build on x86_64-linux-gnu.

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What's New in Python 3.3

[Matthias Klose]

On 09.08.2012 01:04, Victor Stinner wrote:
> Does Python 3.3 support cross-compilation? There are two new option
> for configure: --host and --build, but it's not mentioned in What's
> New in Python 3.3.

it does work, but it is only tested for the linux -> linux case. the mingw and
macosx cross builds did require changes, which didn't go into 3.3 before the
first beta release. what is completely missing is the cross build infrastructure
to build third party extensions. so maybe it's a bit early to announce it in the
release notes.

  Matthias


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Stdlib and timezones, again

[Matthias Klose]

On 30.09.2012 20:18, Gregory P. Smith wrote:
> On Sun, Sep 30, 2012 at 8:33 AM, Benjamin Peterson wrote:
> 
>> 2012/9/30 Xavier Morel :
>>> But at worst, an outdated unicode database will be missing data right?
>>>
>>> Doesn't an outdated timezone db have the risk of returning *incorrect*
>> data?
>>
>> Unicode updates also include corrections; however, it seems there are
>> not significant enough or about obscure enough scripts that not many
>> notice. :)
>>
> 
> We never hear anyone complain because the corrections are not for English
> or other "western" languages that the majority of us speak.  ;)
> 
> Regardless, I think including a version of the database on windows releases
> makes sense.  Update it on a best effort basis before each .x minor
> release.  The documentation can make it clear that it is a changing
> database how to use an updated version with your application.
> 
> One additional thing I'd like to see: Don't let the successful importing of
> a 'pytzdata' module be the only way to get an updated tz database.  Create
> an API for someone to supply one themselves at runtime that is cleaner than
> shoving something into sys.modules['pytzdata'].  And define in which order
> they'll be used:
> 
> priority:
>   1) api call supplying tz data to the process.
>   2) pytzdata module if it exists
>   3) tz data from the underlying operating system
>   4) error.

I disagree on this order, at least for Linux systems. the tzdata database is
well managed on major Linux distributions and should be used for this reason.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Stdlib and timezones, again

[Matthias Klose]

On 01.10.2012 00:50, Gregory P. Smith wrote:
> On Sun, Sep 30, 2012 at 3:17 PM, Matthias Klose  wrote:
> 
>> On 30.09.2012 20:18, Gregory P. Smith wrote:
>>> On Sun, Sep 30, 2012 at 8:33 AM, Benjamin Peterson >> wrote:
>>>
>>>> 2012/9/30 Xavier Morel :
>>>>> But at worst, an outdated unicode database will be missing data right?
>>>>>
>>>>> Doesn't an outdated timezone db have the risk of returning *incorrect*
>>>> data?
>>>>
>>>> Unicode updates also include corrections; however, it seems there are
>>>> not significant enough or about obscure enough scripts that not many
>>>> notice. :)
>>>>
>>>
>>> We never hear anyone complain because the corrections are not for English
>>> or other "western" languages that the majority of us speak.  ;)
>>>
>>> Regardless, I think including a version of the database on windows
>> releases
>>> makes sense.  Update it on a best effort basis before each .x minor
>>> release.  The documentation can make it clear that it is a changing
>>> database how to use an updated version with your application.
>>>
>>> One additional thing I'd like to see: Don't let the successful importing
>> of
>>> a 'pytzdata' module be the only way to get an updated tz database.
>>  Create
>>> an API for someone to supply one themselves at runtime that is cleaner
>> than
>>> shoving something into sys.modules['pytzdata'].  And define in which
>> order
>>> they'll be used:
>>>
>>> priority:
>>>   1) api call supplying tz data to the process.
>>>   2) pytzdata module if it exists
>>>   3) tz data from the underlying operating system
>>>   4) error.
>>
>> I disagree on this order, at least for Linux systems. the tzdata database
>> is
>> well managed on major Linux distributions and should be used for this
>> reason.
>>
> 
> Agreed, but that is why anyone writing code for Linux will simply never use
> 1) or 2).  Those exist primarily to handle the Windows and people with
> software running on non well managed systems.
> 
> I'd add a 3.5) to the above list: tz data bundled with the Python
> distribution.  Bundled tz data would likely simply not even be included in
> a Python package on a Linux system.

I would add

 - a 2.5) if pytzdata does exist and uses the tz data from the underlying
operating system.
 - a 3.5) if pytzdata does exist and doesn't use the tz data from the underlying
operating system.

however I see our patch to pytzdata to use the os data is a local patch, and not
found upstream.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Stdlib and timezones, again

[Matthias Klose]

On 01.10.2012 00:51, Lennart Regebro wrote:
> On Mon, Oct 1, 2012 at 12:17 AM, Matthias Klose  wrote:
> 
>>> priority:
>>>   1) api call supplying tz data to the process.
>>>   2) pytzdata module if it exists
>>>   3) tz data from the underlying operating system
>>>   4) error.
>>
>> I disagree on this order, at least for Linux systems. the tzdata database
>> is
>> well managed on major Linux distributions and should be used for this
>> reason.
>>
> 
> Sure, but in the cases where it is not, overriding should be possible. The
> OS tz data is last because it is the default.

but not overriding with an outdated database.

the new venv stuff makes the distinction to just use the core python
environment, or use the core and site/dist-packages. I don't want to see an
exception to not use something which is not in the core.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Stdlib and timezones, again

[Matthias Klose]

On 01.10.2012 01:07, Chris Angelico wrote:
> On Mon, Oct 1, 2012 at 8:17 AM, Matthias Klose  wrote:
>> On 30.09.2012 20:18, Gregory P. Smith wrote:
>>> priority:
>>>   1) api call supplying tz data to the process.
>>>   2) pytzdata module if it exists
>>>   3) tz data from the underlying operating system
>>>   4) error.
>>
>> I disagree on this order, at least for Linux systems. the tzdata database is
>> well managed on major Linux distributions and should be used for this reason.
> 
> There's no guarantee that an individual sysadmin will have OS updates
> up-to-date. If, on Linux, the pytzdata module is not installed unless
> explicitly called for, that would define pytzdata as high specificity,
> ergo it should override the lower specificity of the OS-provided data.
> 
> The normal case on Linux will happily use the well-managed and
> frequently-updated tzdata.

I don't care as long as pytzdata is a third party module. I do care if it
belongs to the standard library.

I don't want to see a python and perl/java/foo application seeing different
local times.

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Bumping autoconf from 2.68 to 2.69

[Matthias Klose]

On 16.10.2012 17:58, David Malcolm wrote:
> On Tue, 2012-10-16 at 10:59 +0200, Stefan Krah wrote:
>> Charles-François Natali  wrote:
>>> Well, so I guess all committers will have to use the same
>>> Linux/FreeBSD/whatever distribution then?
>>> AFAICT there's no requirement regarding the mercurial version used by
>>> committers either.
>>
>> It should be sufficient to install autoconf-x.y into /home/user/bin or
>> something similar. Installing autoconf from source really takes about
>> 3 minutes.
> FWIW this was also documented in the devguide, as of:
> http://hg.python.org/devguide/rev/5432be4d4e1a
> 
> though that description was then removed in:
> http://hg.python.org/devguide/rev/27be97280cf

so maybe this could be restored.

the python configure.ac doesn't use that internal and specific autoconf
features, so it might be ok to live with the diffs.  I didn't see any issues
using the system supplied autoconf with any python version on not that recent
distro releases.

However the effort to use a specific autoconf version for each branch should be
minimal (other projects do require this).

  Matthias

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handling support for newer OS features at run time

[Matthias Klose]

Am 27.11.2012 23:49, schrieb Trent Nelson:
> I don't think we've currently got the ability to do this, right?
> Is there a precedent set anywhere else?  I suspect it's not as
> much of an issue on *NIX platforms as you'll typically compile
> from source.  Windows, not so much.
> 
> Thoughts?  A single binary that dynamically loads applicable
> modules seems cleaner but will obviously take more work.  Other
> approach would be to start distributing multiple versions of
> Python based on the underlying Windows version.  Not the nicest
> approach.

Usually I have to build a python package on a build daemon running the kernel of
the latest stable (or latest stable long term support) release.  Thus if a
configure test checks the current kernel, then you may get an unexpected answer
and a missing feature. It is somehow different that I already build different
binaries (per release), but the hard-coding of kernel features during configure
time looks like the same issue. Currently working around it by patching
configure to remove the test and hard-code it for a particular build. Another
solution maybe would to have something like --enable-kernel= (as found
in glibc), and hope that you can deduce the features from the kernel version.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Handling support for newer OS features at run time

[Matthias Klose]

Am 28.11.2012 06:37, schrieb Gregory P. Smith:
> On Tue, Nov 27, 2012 at 3:19 PM, Trent Nelson  wrote:
> 
>> On Tue, Nov 27, 2012 at 03:09:12PM -0800, Matthias Klose wrote:
>>> Am 27.11.2012 23:49, schrieb Trent Nelson:
>>>> I don't think we've currently got the ability to do this, right?
>>>> Is there a precedent set anywhere else?  I suspect it's not as
>>>> much of an issue on *NIX platforms as you'll typically compile
>>>> from source.  Windows, not so much.
>>>>
>>>> Thoughts?  A single binary that dynamically loads applicable
>>>> modules seems cleaner but will obviously take more work.  Other
>>>> approach would be to start distributing multiple versions of
>>>> Python based on the underlying Windows version.  Not the nicest
>>>> approach.
>>>
>>> Usually I have to build a python package on a build daemon running the
>>> kernel of the latest stable (or latest stable long term support)
>>> release.  Thus if a configure test checks the current kernel, then you
>>> may get an unexpected answer and a missing feature. It is somehow
>>> different that I already build different binaries (per release), but
>>> the hard-coding of kernel features during configure time looks like
>>> the same issue. Currently working around it by patching configure to
>>> remove the test and hard-code it for a particular build. Another
>>> solution maybe would to have something like --enable-kernel=
>>> (as found in glibc), and hope that you can deduce the features from
>>> the kernel version.
>>
>> Hmmm.  How often do Linux kernel versions expose new features that
>> we can make use of in Python?  i.e. do you run into this problem
>> often?  Can you cite some recent examples?
>>
> 
> Here's an example of using the pipe2() system call. The code is only
> compiled in if the C library supports it. If the C library supports it, the
> system call can still return an error because the running kernel doesn't
> support it (ENOSYS). In that case it falls back to the legacy code:
> 
> 
> http://hg.python.org/cpython/file/618ea5612e83/Modules/_posixsubprocess.c#l738

another one is the check for working semaphores for the multiprocessing module,
seen at https://launchpad.net/bugs/630511

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] mingw32 port

[Matthias Klose]

[No, I'm not interested in the port myself]

patches for a mingw32 port are floating around on the web and the python bug
tracker, although most of them as a set of patches in one issue addressing
several things, and which maybe outdated for the trunk. at least for me
re-reading a big patch in a new version is more work than having the patches in
different issues. So proposing to break down the patches in independent ones,
dealing with:

 - mingw32 support (excluding any cross-build support). tested with
   a native build with srcdir == builddir. The changes for distutils
   mingw32 support should be a separate patch. Who could review these?
   Asked Martin, but didn't get a reply yet.

 - patches to cross-build for mingw32.

 - patches to deal with a srcdir != builddir configuration, where the
   srcdir is read-only (please don't mix with the cross-build support).

All work should be done on the tip/trunk.

So ok to close issue16526, issue3871, issue3754 and suggest in the reports to
start over with more granular changes?

  Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build

[Matthias Klose]

Am 03.02.2013 22:20, schrieb Gregory P. Smith:
> On Thu, Jan 31, 2013 at 11:14 PM, Antoine Pitrou wrote:
> 
>> On Fri, 1 Feb 2013 11:00:24 +1000
>> Nick Coghlan  wrote:
>>> On Fri, Feb 1, 2013 at 9:50 AM, Antoine Pitrou 
>> wrote:
 On Thu, 31 Jan 2013 23:52:27 +0100 (CET)
 matthias.klose  wrote:
> http://hg.python.org/cpython/rev/8ee6d96a1019
> changeset:   81859:8ee6d96a1019
> branch:  2.7
> parent:  81855:df9f8feb7444
> user:d...@python.org
> date:Thu Jan 31 23:52:03 2013 +0100
> summary:
>   - Issue #17086: Backport the patches from the 3.3 branch to
>> cross-build
>   the package.

 You aren't supposed to add new features to bugfix branches. Did you
 have a specific reason to do this?
>>>
>>> One of the reasons for the long maintenance period on 2.7 is to keep
>>> it building as the underlying platforms change. With the rise of ARM
>>> systems, being able to reliably cross-build Python 2.7 for ARM from an
>>> x86_64 system is fairly important.
>>
>> I would like to see a better argument for this. The rise of ARM systems
>> is the rise of ARM systems powerful enough to build Python without
>> cross-compiling (which is why we *now* have ARM buildbots).
>> The very small ARM systems which need cross-compiling have existed for
>> decades.
>>
> 
> It is quite common for developers to build a single code base on a single
> workstation targeting a plethora of platforms all at once. Requiring native
> systems with self hosting tool-chains for builds is a non-starter as those
> often don't exist. Making Python 2.7's configure+makefiles easier to cross
> compile out of the box is a good thing.
> 
> Side note: we really need a cross compiling build-bot host + target system
> or we'll inevitably break this.
> 
> 
>>> That said, as a procedural point for build related new features in
>>> 2.7, I agree they should be proposed, with an explicit rationale, on
>>> python-dev before proceeding with the commit.
>>
>> I think this huge changeset should be reverted. It's a complete failure
>> in terms of procedure and following the rules. "Just because it can be
>> useful" is not a good enough reason to violate our release model
>> without even asking.
>>
> 
> That's up to the 2.7 release manager.  Yes, this could have been done
> better by asking first. But IMNSHO I'd prefer to see it stay in.

I filed Issue #17086, and got feedback from the release manager, which I maybe
interpreted too easily as not objecting.  And it looked like a nice opportunity
to get this into a release (hoping not to listen to another PyCon talk how to
hack cross-builds).

Even for the trunk, getting feed-back for cross-build related issues is
difficult.  Maybe reviewers are turned off by impolite comments in some of the
cross-build related issues in the bug tracker, but that doesn't explain that you
don't get feedback for most of the cross-build issues.

So what I do understand, build-related issues like an arm64 or mingw32 port are
ok for 2.7, if they are stable on the trunk, and communicated on python-dev?

I'll see that I setup a cross buildd building in a cross-build environment and
testing in a chroot with qemu installed. I hope that the buildd setup is able to
support this.

Talking about the release model, yes I think there are some issues:

 - the 2.7 branch is the only branch which doesn't have expected release
   dates on the calendar. And from a distributor/vendor point of view, I
   think yearly releases are too seldom. Such a release could end up
   only up to 24 months later in a (linux) distribution.

 - there were way too may regressions checked in on at least the 2.7
   branch.  Is it just our vcs merge model that we first have to check in
   on the branches, and then merge to the trunk? Afaics python is the
   only project to have such an approach. Others have trunk first, maybe
   with immediate backport, maybe with later backport.

Matthias

PS: Antoine: Please CC the committer for such emails.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] cpython (2.7): - Issue #17086: Backport the patches from the 3.3 branch to cross-build

[Matthias Klose]

Am 05.02.2013 07:13, schrieb Terry Reedy:
> On 2/4/2013 3:04 PM, Matthias Klose wrote:
> 
>>   - the 2.7 branch is the only branch which doesn't have expected release
>> dates on the calendar.
> 
> Which calendar? I see 2.7.4, 3.2.4 (its final release), and 3.3.1 on the 
> Release
> Schecule at python.org.

maybe these are now updated. until recently 3.3.1 was targeted for Nov/Dec 2012,
and 2.7.4 wasn't found on the calendar.

>>   - there were way too may regressions checked in on at least the 2.7
>> branch.
> 
> Regressions? That normally means adding a bug as part of a patch, especially 
> one
> that was previously fixed. We continually add tests to try to prevent that. 
> What
> period of time do you mean 'were' to refer to?

 - http://bugs.python.org/issue9374
 - http://bugs.python.org/issue15906
 - http://bugs.python.org/issue16012
 - http://bugs.python.org/issue10182
 - http://bugs.python.org/issue16828

not a complete list, all these are past the 2.7.3 release.

>>  Is it just our vcs merge model that we first have to check in
>> on the branches, and then merge to the trunk?
> 
> Mercurial works best if merges are done in the forward direction. However, 
> this
> is not relevant to 2.7 patches as they are pushed to tip but *not* merged. The
> repository is a two-headed monster ;=).

so it should be possible to delay patches for 2.7 until they don't show issues
in 3.2 or 3.3.

>>  Afaics python is the
>> only project to have such an approach. Others have trunk first, maybe
>> with immediate backport, maybe with later backport.
> 
> If a patch is first commited to tip (for 3.4) and then backported to 3.3, then
> when the backport is pushed, a null merge is needed to avoid making a third
> head, and the dag looks messier. (I believe 'null merge' is the right term, 
> but
> I have never done this.) My impression is that most 3.x bugfix patches merge
> forward with little or no change.

so is it only this technical limitation, why the bugfix process is defined this 
way?

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] license issues with profiler.py and md5.h/md5c.c

[Matthias Klose]

A Debian user pointed out (http://bugs.debian.org/293932), that the
current license for the Python profiler is not conforming to the DFSG
(Debian free software guidelines).

http://www.python.org/doc/current/lib/node829.html states

  "This permission is explicitly restricted to the copying and
  modification of the software to remain in Python, compiled Python,
  or other languages (such as C) wherein the modified or derived code
  is exclusively imported into a Python module."

The DFSG, http://www.debian.org/doc/debian-policy/ch-archive.html#s-dfsg,
third paragraph state:

  "Derived Works
The license must allow modifications and derived works, and must
allow them to be distributed under the same terms as the license
of the original software."

- Does somebody knows about the history of this license, why it is
  more restricted than the Python license?
- Is there a chance to change the license for these two modules
  (profile.py, pstats.py)?


The md5.h/md5c.c files allow "copy and use", but no modification of
the files. There are some alternative implementations, i.e. in glibc,
openssl, so a replacement should be sage. Any other requirements when
considering a replacement?

Matthias
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com