Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
> Here is a draft PEP (forgive me if it's incorrectly formatted; I've > never done this before). LGTM. Please specify what /usr/bin/python is supposed to be also (rather: the "python" utility). I'd like it ruled out that installations *only* provide python2 and python3 - "python" could be either one, but should be present "normally" (i.e. SHOULD in the RFC 2119 sense). Nitpickingly, I'd add that scripts can, of course, also specify python2.7 (or some such). Actually, scripts can do whatever they want - it's more about what they then can expect to happen. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
[Guido van Rossum, 2011-03-02] > On Wed, Mar 2, 2011 at 4:56 AM, Piotr Ożarowski wrote: > > [Sandro Tosi, 2011-03-02] > >> On Wed, Mar 2, 2011 at 10:01, Piotr Ożarowski wrote: > >> > I co-maintain with Matthias a package that provides /usr/bin/python > >> > symlink in Debian and I can confirm that it will always point to Python > >> > 2.X. We also do not plan to add /usr/bin/python2 symlink (and I guess > >> > only accepted PEP can change that) > >> > >> Can you please explain why you NACK this proposed change? > > > > it encourages people to change /usr/bin/python symlink to point to > > python3.X which I'm strongly against (how can I tell that upstream > > author meant python3.X and not python2.X without checking the code?) > > But the same is already true for python2.X vs. python2.Y. Explicit is > better than implicit etc. Plus, 5 years from now everybody is going to > be annoyed that "python" still refers to some ancient unused version > of Python. I don't really mind adding /usr/bin/python2 symlink just to clean Arch mess, but I do mind changing /usr/bin/python to point to python3 (and I can use the same argument - "Explicit is better than implicit" - if you need Python 3, say so in the shebang, right?). What I'm afraid of is when we'll add /usr/bin/python2, we'll start getting a lot of scripts that will have to be checked manually every time new upstream version is released because we cannot assume what upstream author is using at given point. If /usr/bin/python will be disallowed in shebangs on the other hand (and all scripts will use /usr/bin/python2, /usr/bin/python3, /usr/bin/python4 or /usr/bin/python2.6 etc.) I don't see a problem with letting administrators choose /usr/bin/python (right now not only changing it from python2.X to python3.X will break the system but also changing it from /usr/bin/pytohn2.X to /usr/bin/python2.Y will break it, and believe me, I know what I'm talking about (one of the guys at work did something like this once)) [all IMHO, dunno if other Debian's python-defaults maintainers agree with me] -- Piotr Ożarowski Debian GNU/Linux Developer www.ozarowski.pl www.griffith.cc www.debian.org GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Improvements for Porting C Extension from 2 to 3
Hi, While porting a C extension from 2 to 3, I realized that there are some general cases which can be automated. For example, for my specific application (yappi - http://code.google.com/p/yappi/), all I need to do is following things: 1) define PyModuleDef 2) change PyString_AS_STRING calls to _PyUnicode_AsString 3) change module init code a little. It occurred to me all these kind of standart changes can be automated via a script. Not sure on the usability of this however, because of my limited knowledge on the area. Does such a tool worth being implemented? -- Sumer Cip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improvements for Porting C Extension from 2 to 3
Sümer Cip wrote: > Hi, > > While porting a C extension from 2 to 3, I realized that there are some > general cases which can be automated. For example, for my specific > application (yappi - http://code.google.com/p/yappi/), all I need to do is > following things: > > 1) define PyModuleDef > 2) change PyString_AS_STRING calls to _PyUnicode_AsString Aside: Please don't use private APIs in Python extensions. Esp. the above Unicode API is likely going to be phased out. You're better off, using PyUnicode_AsUTF8String() instead and then leaving the PyString_AS_STRING() macro in place. > 3) change module init code a little. > > It occurred to me all these kind of standart changes can be automated via a > script. > > Not sure on the usability of this however, because of my limited knowledge > on the area. > > Does such a tool worth being implemented? I'm not sure whether you can really automate this: The change from 8-bit strings to Unicode support usually requires reconsidering whether you're dealing with plain text, encoded text data or binary data. However, a guide of what to replace and how to change the code would probably help a lot. Please share your thoughts on the python-porting mailint list and/or add to these wiki pages: http://wiki.python.org/moin/PortingToPy3k http://wiki.python.org/moin/PortingExtensionModulesToPy3k Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Mar 03 2011) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improvements for Porting C Extension from 2 to 3
Le jeudi 03 mars 2011 à 11:06 +0200, Sümer Cip a écrit : > While porting a C extension from 2 to 3, I realized that there are some > general cases which can be automated. For example, for my specific > application (yappi - http://code.google.com/p/yappi/), all I need to do is > following things: See also 2to3c project based on Coccinelle: https://fedorahosted.org/2to3c/ http://coccinelle.lip6.fr/ Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Official Roadmap (Re: Let's get PEP 380 into Python 3.3)
On Sat, Feb 26, 2011 at 12:43 AM, Guido van Rossum wrote: > Now that the language moratorium is lifted, let's make sure to get PEP > 380 implemented for Python 3.3. How about official RoadMap? There is no visibility into what's going on in Python development. New people can' t jump in and help do bring some features faster. http://dungeonhack.sourceforge.net/Roadmap "yield from" seems useful. -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improvements for Porting C Extension from 2 to 3
> > > > 1) define PyModuleDef > > 2) change PyString_AS_STRING calls to _PyUnicode_AsString > > Aside: Please don't use private APIs in Python extensions. Esp. > the above Unicode API is likely going to be phased out. > > You're better off, using PyUnicode_AsUTF8String() instead and > then leaving the PyString_AS_STRING() macro in place. > In the standart Python 3.2 source tree, Modules/_lsprof.c uses that internal function _PyUnicode_AsString. Internal means internal to the whole distribution here I think?. But IMHO, this should not be the case, C API modules in the standart dist. should not use internal functions of other areas. Like in the example: cProfile code has nothing to do with the Unicode internals. New developers like me, are in need a consistent examples of usage of Python C API, especially on Python 3.2. Thanks, -- Sumer Cip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improvements for Porting C Extension from 2 to 3
2011/3/3 Sümer Cip : > >> > >> > 1) define PyModuleDef >> > 2) change PyString_AS_STRING calls to _PyUnicode_AsString >> >> Aside: Please don't use private APIs in Python extensions. Esp. >> the above Unicode API is likely going to be phased out. >> >> You're better off, using PyUnicode_AsUTF8String() instead and >> then leaving the PyString_AS_STRING() macro in place. > > In the standart Python 3.2 source tree, Modules/_lsprof.c uses that internal > function _PyUnicode_AsString. Internal means internal to the whole > distribution here I think?. But IMHO, this should not be the case, C API > modules in the standart dist. should not use internal functions of other > areas. Like in the example: cProfile code has nothing to do with the Unicode > internals. New developers like me, are in need a consistent examples of > usage of Python C API, especially on Python 3.2. I'm not sure why what C-API is used in Python's extension modules needs to be anyway to you. -- Regards, Benjamin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improvements for Porting C Extension from 2 to 3
> > I'm not sure why what C-API is used in Python's extension modules > needs to be anyway to you. > I just tought it is a better programming practice to decrease/narrow inter modular usage of functions inside the whole dist. and also a good example for the other 3rd party C Extension developers. I may be wrong. Anyway thanks for the answers. This is just a suggestion. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improvements for Porting C Extension from 2 to 3
On 03/03/2011 13:45, Benjamin Peterson wrote: 2011/3/3 Sümer Cip: 1) define PyModuleDef 2) change PyString_AS_STRING calls to _PyUnicode_AsString Aside: Please don't use private APIs in Python extensions. Esp. the above Unicode API is likely going to be phased out. You're better off, using PyUnicode_AsUTF8String() instead and then leaving the PyString_AS_STRING() macro in place. In the standart Python 3.2 source tree, Modules/_lsprof.c uses that internal function _PyUnicode_AsString. Internal means internal to the whole distribution here I think?. But IMHO, this should not be the case, C API modules in the standart dist. should not use internal functions of other areas. Like in the example: cProfile code has nothing to do with the Unicode internals. New developers like me, are in need a consistent examples of usage of Python C API, especially on Python 3.2. I'm not sure why what C-API is used in Python's extension modules needs to be anyway to you. I think its fair enough to point out the inconsistency in python-dev declaring that authors *should not* use certain parts of the C-API in extensions whilst using them in the extensions python-dev is responsible for... Michael -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official Roadmap (Re: Let's get PEP 380 into Python 3.3)
On Thu, Mar 3, 2011 at 06:40, anatoly techtonik wrote: > On Sat, Feb 26, 2011 at 12:43 AM, Guido van Rossum > wrote: > > Now that the language moratorium is lifted, let's make sure to get PEP > > 380 implemented for Python 3.3. > > How about official RoadMap? There is no visibility into what's going > on in Python development. New people can' t jump in and help do bring > some features faster. http://dungeonhack.sourceforge.net/Roadmap You could put together something fancy out of a query for all 3.3 feature requests [0] if you want this. I'm also not sure if that would be entirely useful. Python development is fairly relaxed in that you work on what you want to work on, when you want to do it. If you fill up some roadmap with all 180 feature requests on that page, they are not all going to get done, and it's not a condition of doing the release. [0] http://goo.gl/4RMp8 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official Roadmap (Re: Let's get PEP 380 into Python 3.3)
A project roadmap is usually something that a team commits to, with some caveats, in order to give the customers (the "outside world") some view in development that is already planned or under way or on some other way committed to. But a roadmap itself takes effort to create and maintain. Also, often projects are undertaken on the spur of the moment that aren't on the roadmap, and changes in personnel or priorities may cause items to linger on the "official" roadmap without making much progress. I think that if someone is interested in contributing to Python but they aren't into coding, managing a roadmap could be a useful role to perform, especially for someone with some project management skills and the tenacity to stick around for a long time. (You don't publish a roadmap and then leave; it needs regular updates, say every month or quarter at least.) At the same time the roadmap manager ought to recall that they are not the project manager and they will have to be careful not to use the roadmap to force core developers (who all have their own agendas) to do (or refrain from) specific things. The kind of items that would make sense to have on a roadmap are probably usually PEP-sized features; a list of feature extracted from the tracker would be too overwhelming for the external consumers of the roadmap. A note about the example from dungeonhack: that format is probably too detailed for Python; I don't think we need to have dates and assigned developers in the roadmap. An example entry could say "yield from (PEP 380)". There is no expectation for roadmaps to be complete or always uptodate. Anyone feel compelled to give it a try? --Guido On Thu, Mar 3, 2011 at 6:12 AM, Brian Curtin wrote: > On Thu, Mar 3, 2011 at 06:40, anatoly techtonik wrote: >> >> On Sat, Feb 26, 2011 at 12:43 AM, Guido van Rossum >> wrote: >> > Now that the language moratorium is lifted, let's make sure to get PEP >> > 380 implemented for Python 3.3. >> >> How about official RoadMap? There is no visibility into what's going >> on in Python development. New people can' t jump in and help do bring >> some features faster. http://dungeonhack.sourceforge.net/Roadmap > > You could put together something fancy out of a query for all 3.3 feature > requests [0] if you want this. > I'm also not sure if that would be entirely useful. Python development is > fairly relaxed in that you work on what you want to work on, when you want > to do it. If you fill up some roadmap with all 180 feature requests on that > page, they are not all going to get done, and it's not a condition of doing > the release. > > [0] http://goo.gl/4RMp8 > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Improvements for Porting C Extension from 2 to 3
On Thu, Mar 3, 2011 at 9:05 AM, Michael Foord wrote: .. > I think its fair enough to point out the inconsistency in python-dev > declaring that authors *should not* use certain parts of the C-API in > extensions whilst using them in the extensions python-dev is responsible > for... This is not an inconsistency, but a balancing act. By using APIs that are likely to change or be removed, we assume the responsibility to clean up code that is under our control at some point in the future. Users or 3rd party developers can do the same balancing, but the assumption is that the amount of code in their libraries is much greater than that in stdlib and they do not follow Python changes as closely as core developers. These considerations usually tip the scale towards not using private APIs. (For example, they don't and should not run buildbots testing their code with the latest VC checkout.) On the other hand, if you can ship your application sooner using a hack like #define PyString_AS_STRING _PyUnicode_AsString more power to you, but you have been warned - don't file a bug report when you code stops working with Python 3.5. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
On Thu, Mar 03, 2011 at 09:55:25AM +0100, Piotr Ożarowski wrote:
> [Guido van Rossum, 2011-03-02]
> > On Wed, Mar 2, 2011 at 4:56 AM, Piotr Ożarowski wrote:
> > > [Sandro Tosi, 2011-03-02]
> > >> On Wed, Mar 2, 2011 at 10:01, Piotr Ożarowski wrote:
> > >> > I co-maintain with Matthias a package that provides /usr/bin/python
> > >> > symlink in Debian and I can confirm that it will always point to Python
> > >> > 2.X. We also do not plan to add /usr/bin/python2 symlink (and I guess
> > >> > only accepted PEP can change that)
> > >>
> > >> Can you please explain why you NACK this proposed change?
> > >
> > > it encourages people to change /usr/bin/python symlink to point to
> > > python3.X which I'm strongly against (how can I tell that upstream
> > > author meant python3.X and not python2.X without checking the code?)
> >
> > But the same is already true for python2.X vs. python2.Y. Explicit is
> > better than implicit etc. Plus, 5 years from now everybody is going to
> > be annoyed that "python" still refers to some ancient unused version
> > of Python.
>
> I don't really mind adding /usr/bin/python2 symlink just to clean Arch
> mess, but I do mind changing /usr/bin/python to point to python3 (and I
> can use the same argument - "Explicit is better than implicit" - if you
> need Python 3, say so in the shebang, right?). What I'm afraid of is
> when we'll add /usr/bin/python2, we'll start getting a lot of scripts
> that will have to be checked manually every time new upstream version is
> released because we cannot assume what upstream author is using at given
> point.
>
> If /usr/bin/python will be disallowed in shebangs on the other hand
> (and all scripts will use /usr/bin/python2, /usr/bin/python3,
> /usr/bin/python4 or /usr/bin/python2.6 etc.) I don't see a problem with
> letting administrators choose /usr/bin/python (right now not only
> changing it from python2.X to python3.X will break the system but also
> changing it from /usr/bin/pytohn2.X to /usr/bin/python2.Y will break it,
> and believe me, I know what I'm talking about (one of the guys at work
> did something like this once))
>
> [all IMHO, dunno if other Debian's python-defaults maintainers agree
> with me]
>
Thinking outside of the box, I can think of something that would satisfy
your requirements but I don't know how appropriate it is for upstream python
to ship with. Stop shipping /usr/bin/python. Ship "python" in an alternate
location like $LIBEXECDIR/python2.7/bin (I think this would be
/usr/lib/python2.7/bin on Debian and /usr/libexec/python2.7/bin on Fedora
which would both be appropriate) then configure which python version is
invoked by the user typing "python" by configuring PATH (a shell alias might
also work). You could configure this with environment-modules[1]_ if Debian
supports using that in packaging.
Coupled with a PEP that recommends against using /usr/bin/python in scripts
and instead using /usr/bin/python$MAJOR, this might be sufficient. OTOH, my
cynical side doubts that script authors read PEPs so it'll take either
upstream python shipping without /usr/bin/python or consensus among the
distros to ship without /usr/bin/python to reach the point where script
authors realize that they need to use /usr/bin/python{2,3} instead of
/usr/bin/python.
.. _[1]: http://modules.sourceforge.net/
-Toshio
pgp97oSsV2cOw.pgp
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official Roadmap (Re: Let's get PEP 380 into Python 3.3)
On Mar 3, 2011, at 4:40 AM, anatoly techtonik wrote: > How about official RoadMap? There is no visibility into what's going > on in Python development. New people can' t jump in and help do bring > some features faster. http://dungeonhack.sourceforge.net/Roadmap Thanks for the link. Their roadmap made my day: High priority: code to spawn spiders in the forest Medium priority: write pet rat fetch quest And best of all, the "animate rat" task is assigned to "Musk of Ephesus". Raymond ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official Roadmap (Re: Let's get PEP 380 into Python 3.3)
On Thu, Mar 3, 2011 at 4:12 PM, Brian Curtin wrote: > On Thu, Mar 3, 2011 at 06:40, anatoly techtonik wrote: >> >> On Sat, Feb 26, 2011 at 12:43 AM, Guido van Rossum >> wrote: >> > Now that the language moratorium is lifted, let's make sure to get PEP >> > 380 implemented for Python 3.3. >> >> How about official RoadMap? There is no visibility into what's going >> on in Python development. New people can' t jump in and help do bring >> some features faster. http://dungeonhack.sourceforge.net/Roadmap > > You could put together something fancy out of a query for all 3.3 feature > requests [0] if you want this. > [0] http://goo.gl/4RMp8 I've got error - "You do not have permission to store queries". That should mean something. > I'm also not sure if that would be entirely useful. Python development is > fairly relaxed in that you work on what you want to work on, when you want > to do it. If you fill up some roadmap with all 180 feature requests on that > page, they are not all going to get done, and it's not a condition of doing > the release. Roadmap is not a strict plan for release. It helps people *self-organize into teams* around specific feature or bugs. One of the features of Roadmap is that *if you add an item to it, you're willing to support development* of this item. Roadmap is not needed for those core developers, who prefer to work in isolation, but even they can find it *useful to see who works on what, and what users actually want*. Yes, there can (or even should) be a way for everybody with Python account to vote on items in the Roadmap and subscribe to updates (like commits, messages, code reviews, tweets and other stuff related to one item). This will give Jesse and teams a better picture, what sprints need funding. Tracker doesn't allow this - even though users can subscribe to Roundup issues, it just doesn't work for them. And the last distinction between Roadmap and Tracker is that the former gives you a quick overview of each item without going into the gory details of lengthy discussions. I'd say this is a good project for Google Summer of Code, which deadline for submission is next Friday, BTW. -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange occasional marshal error
On 03/03/2011 15:09, Graham Stratton wrote:
On Mar 2, 3:01 pm, Graham Stratton wrote:
We are using marshal for serialising objects before distributing them
around the cluster, and extremely occasionally a corrupted marshal is
produced. The current workaround is to serialise everything twice and
check that the serialisations are the same. On the rare occasions that
they are not, I have dumped the files for comparison. It turns out
that there are a few positions within the serialisation where
corruption tends to occur (these positions seem to be independent of
the data of the size of the complete serialisation). These are:
4 bytes starting at 548867 (0x86003)
4 bytes starting at 4398083 (0x431c03)
4 bytes starting at 17595395 (0x10c7c03)
4 bytes starting at 19794819 (0x12e0b83)
4 bytes starting at 22269171 (0x153ccf3)
2 bytes starting at 25052819 (0x17e4693)
3 bytes starting at 28184419 (0x1ae0f63)
I modified marshal.c to print when it extends the string used to write
the marshal to. This gave me these results:
s = marshal.dumps(list((i, str(i)) for i in range(140)))
Resizing string from 50 to 1124 bytes
Resizing string from 1124 to 3272 bytes
Resizing string from 3272 to 7568 bytes
Resizing string from 7568 to 16160 bytes
Resizing string from 16160 to 33344 bytes
Resizing string from 33344 to 67712 bytes
Resizing string from 67712 to 136448 bytes
Resizing string from 136448 to 273920 bytes
Resizing string from 273920 to 548864 bytes
Resizing string from 548864 to 1098752 bytes
Resizing string from 1098752 to 2198528 bytes
Resizing string from 2198528 to 4398080 bytes
Resizing string from 4398080 to 8797184 bytes
Resizing string from 8797184 to 17595392 bytes
Resizing string from 17595392 to 19794816 bytes
Resizing string from 19794816 to 22269168 bytes
Resizing string from 22269168 to 25052814 bytes
Resizing string from 25052814 to 28184415 bytes
Resizing string from 28184415 to 31707466 bytes
Every corruption point occurs exactly three bytes above an extension
point (rounded to the nearest word for the last two). This clearly
isn't a coincidence, but I can't see where there could be a problem.
I'd be grateful for any pointers.
I haven't found the cause, but I have found something else I'm
suspicious of in the source for Python 3.2.
In marshal.c there's a function "w_object", and within that function is
this:
else if (PyAnySet_CheckExact(v)) {
PyObject *value, *it;
if (PyObject_TypeCheck(v, &PySet_Type))
w_byte(TYPE_SET, p);
else
w_byte(TYPE_FROZENSET, p);
"w_byte" is a macro which includes an if-statement, not a function.
Doesn't it need some braces? (There's are braces in the other places
they're needed.)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Strange occasional marshal error
On Thu, Mar 3, 2011 at 9:40 AM, MRAB wrote:
> On 03/03/2011 15:09, Graham Stratton wrote:
>>
>> On Mar 2, 3:01 pm, Graham Stratton wrote:
>>>
>>> We are using marshal for serialising objects before distributing them
>>> around the cluster, and extremely occasionally a corrupted marshal is
>>> produced. The current workaround is to serialise everything twice and
>>> check that the serialisations are the same. On the rare occasions that
>>> they are not, I have dumped the files for comparison. It turns out
>>> that there are a few positions within the serialisation where
>>> corruption tends to occur (these positions seem to be independent of
>>> the data of the size of the complete serialisation). These are:
>>>
>>> 4 bytes starting at 548867 (0x86003)
>>> 4 bytes starting at 4398083 (0x431c03)
>>> 4 bytes starting at 17595395 (0x10c7c03)
>>> 4 bytes starting at 19794819 (0x12e0b83)
>>> 4 bytes starting at 22269171 (0x153ccf3)
>>> 2 bytes starting at 25052819 (0x17e4693)
>>> 3 bytes starting at 28184419 (0x1ae0f63)
>>
>> I modified marshal.c to print when it extends the string used to write
>> the marshal to. This gave me these results:
>>
> s = marshal.dumps(list((i, str(i)) for i in range(140)))
>>
>> Resizing string from 50 to 1124 bytes
>> Resizing string from 1124 to 3272 bytes
>> Resizing string from 3272 to 7568 bytes
>> Resizing string from 7568 to 16160 bytes
>> Resizing string from 16160 to 33344 bytes
>> Resizing string from 33344 to 67712 bytes
>> Resizing string from 67712 to 136448 bytes
>> Resizing string from 136448 to 273920 bytes
>> Resizing string from 273920 to 548864 bytes
>> Resizing string from 548864 to 1098752 bytes
>> Resizing string from 1098752 to 2198528 bytes
>> Resizing string from 2198528 to 4398080 bytes
>> Resizing string from 4398080 to 8797184 bytes
>> Resizing string from 8797184 to 17595392 bytes
>> Resizing string from 17595392 to 19794816 bytes
>> Resizing string from 19794816 to 22269168 bytes
>> Resizing string from 22269168 to 25052814 bytes
>> Resizing string from 25052814 to 28184415 bytes
>> Resizing string from 28184415 to 31707466 bytes
>>
>> Every corruption point occurs exactly three bytes above an extension
>> point (rounded to the nearest word for the last two). This clearly
>> isn't a coincidence, but I can't see where there could be a problem.
>> I'd be grateful for any pointers.
This bug report doesn't mention the Python version nor the platform --
it could in theory be a bug in the platform compiler or memory
allocator. It would also be nice to provide the test program that
reproduces the issue. It would also be useful to start tracking it in
the issue tracker at bugs.python.org
Assuming it's 3.2, I would audit _PyBytes_Resize() and whatever it
uses -- if your hunch is right and there is a problem with resizing
that's where it's done.
> I haven't found the cause, but I have found something else I'm
> suspicious of in the source for Python 3.2.
>
> In marshal.c there's a function "w_object", and within that function is
> this:
>
> else if (PyAnySet_CheckExact(v)) {
> PyObject *value, *it;
>
> if (PyObject_TypeCheck(v, &PySet_Type))
> w_byte(TYPE_SET, p);
> else
> w_byte(TYPE_FROZENSET, p);
>
> "w_byte" is a macro which includes an if-statement, not a function.
> Doesn't it need some braces? (There's are braces in the other places
> they're needed.)
That macro looks fine to me; looking at the definition of w_byte() it
has matched if/else clauses:
#define w_byte(c, p) if (((p)->fp)) putc((c), (p)->fp); \
else if ((p)->ptr != (p)->end) *(p)->ptr++ = (c); \
else w_more(c, p)
Although traditionally, just to be sure, we've enclosed similar macros
inside do { ... } while (0). Also it would be nice to call out its
macro-status by renaming it to W_BYTE -- I suppose at one point in the
past it was a plain function...
--
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
On Wed, 2011-03-02 at 01:14 +0100, "Martin v. Löwis" wrote: > > I think a PEP would help, but in this case I would request that before > > the PEP gets written (it can be a really short one!) somebody actually > > go out and get consensus from a number of important distros. Besides > > Barry, do we have any representatives of distros here? > > Matthias Klose represents Debian, Dave Malcolm represents Redhat, > and Dirkjan Ochtman represents Gentoo. Current status within RHEL and Fedora: The "python" rpm package has: - a "/usr/bin/python", which is the "system" build of Python 2 - hardlinked with "/usr/bin/python2.N" (where N is the appropriate minor release number; currently 2.7 for Fedora 14 onwards) - a symlink "/usr/bin/python2", pointing at "/usr/bin/python" There are a number of other rpm packages with names matching "*py*", which use the system build of Python 3 There is a "python3" package on Fedora 13 onwards with: - a "/usr/bin/python3", which is the "system" build of Python 3 - hardlinked with "/usr/bin/python3.N" (where N is the appropriate minor release number; will be 3.2 as of Fedora 15) There are number of add-on rpm packages containing 3rd-party Python 3 code with names of the form "python3-*". Some more status on our pre-packaged Python 3 stack can be seen here: https://fedoraproject.org/wiki/Python3 I've also added "python-debug" and "python3-debug" binaries, containing --with-pydebug builds of the same code. On a related note, we have a number of scripts packaged across the distributions with a shebang line that reads: #!/usr/bin/env python which AIUI follows upstream recommendations. There was a proposal to change these when packaging them to hardcode the specific python binary: https://fedoraproject.org/wiki/Features/SystemPythonExecutablesUseSystemPython on the grounds that a packaged system script is expecting (and has been tested against) a specific python build. That proposal has not yet been carried out. Ideally if we did this, we'd implement it as a postprocessing phase within "rpmbuild", rather than manually patching hundreds of files. Note that this would only cover shebang lines at the tops of scripts. If a 3rd-party program launches python directly, that could fail, and I don't see a convenient way of fixing every reference in all code in all packages (without, say, running a SystemTap script to monitor for programs exec-ing "/usr/bin/python") For example, I wonder what the automake macro for detecting python would make of a /usr/bin/python that's python 3: http://www.gnu.org/software/hello/manual/automake/Python.html I've seen a few hand-coded makefiles for Python extension modules that were broken by the SOABI changes in PEP 3149. To be fair, thouse makefiles were badly written, but I think that changing the meaning of /usr/bin/python would break a lot of things. FWIW, I don't see the harm in providing a /usr/bin/python2 symlink, but I don't plan to change /usr/bin/python at this time. Hope this is helpful Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
On Thu, 2011-03-03 at 14:17 -0500, David Malcolm wrote: > On Wed, 2011-03-02 at 01:14 +0100, "Martin v. Löwis" wrote: > There are a number of other rpm packages with names matching "*py*", > which use the system build of Python 3 Gah; I meant Python 2 here. (Must proofread my screeds before posting them) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] unittest.main() --catch parameter
I am looking at --help of test runner and asking the question: what is the use case for -c, --catch option? It doesn't look like it should be present in generic runner. I also can't find reasons to waste short option for it. There will be big problems with people complaining about BC break even if this option is not used by anyone. Usage: tests.py [options] [test] [...] Options: -h, --help Show this message -v, --verboseVerbose output -q, --quiet Minimal output -f, --failfast Stop on first failure -c, --catch Catch control-C and display results -b, --buffer Buffer stdout and stderr during test runs -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest.main() --catch parameter
On 03/03/2011 20:31, anatoly techtonik wrote: I am looking at --help of test runner and asking the question: what is the use case for -c, --catch option? It catches keyboard interrupt and instead of just bombing out of the test run it reports all the results collected so far. Without this option interrupting a test run with a ctrl-c kills the run and reports nothing. Seeing an unexpected failure or error during a long test run and having to wait to the end of the test run to see the traceback can be annoying, this feature solves that problem. It doesn't look like it should be present in generic runner. I also can't find reasons to waste short option for it. Nose, django and other test runners provide this option, so it is functionality that people seem to value. There will be big problems with people complaining about BC break even if this option is not used by anyone. I don't understand this sentence, sorry. All the best, Michael Foord Usage: tests.py [options] [test] [...] Options: -h, --help Show this message -v, --verboseVerbose output -q, --quiet Minimal output -f, --failfast Stop on first failure -c, --catch Catch control-C and display results -b, --buffer Buffer stdout and stderr during test runs -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Official Roadmap (Re: Let's get PEP 380 into Python 3.3)
On Thu, 3 Mar 2011 09:24:29 -0800 Raymond Hettinger wrote: > > On Mar 3, 2011, at 4:40 AM, anatoly techtonik wrote: > > > How about official RoadMap? There is no visibility into what's going > > on in Python development. New people can' t jump in and help do bring > > some features faster. http://dungeonhack.sourceforge.net/Roadmap > > Thanks for the link. Their roadmap made my day: > >High priority: code to spawn spiders in the forest >Medium priority: write pet rat fetch quest > > And best of all, the "animate rat" task is assigned to "Musk of Ephesus". You need quite some musk to animate a rat. PS: this isn't meant to be a statement about distutils ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest.main() --catch parameter
On Thu, Mar 3, 2011 at 10:44 PM, Michael Foord wrote: > On 03/03/2011 20:31, anatoly techtonik wrote: >> >> I am looking at --help of test runner and asking the question: what is >> the use case for -c, --catch option? > > It catches keyboard interrupt and instead of just bombing out of the test > run it reports all the results collected so far. > > Without this option interrupting a test run with a ctrl-c kills the run and > reports nothing. Seeing an unexpected failure or error during a long test > run and having to wait to the end of the test run to see the traceback can > be annoying, this feature solves that problem. Why not just leave this behavior by default and just return -1 if the Ctrl-C was pressed? >> It doesn't look like it should be >> present in generic runner. I also can't find reasons to waste short >> option for it. > > Nose, django and other test runners provide this option, so it is > functionality that people seem to value. > >> There will be big problems with people complaining >> about BC break even if this option is not used by anyone. >> > > I don't understand this sentence, sorry. If the option is useless, people won't allow to remove it, because it will break "backward compatibility" (BC), even if they don't use this option themselves. > All the best, > > Michael Foord >> >> Usage: tests.py [options] [test] [...] >> >> Options: >> -h, --help Show this message >> -v, --verbose Verbose output >> -q, --quiet Minimal output >> -f, --failfast Stop on first failure >> -c, --catch Catch control-C and display results >> -b, --buffer Buffer stdout and stderr during test runs >> >> >> -- >> anatoly t. >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > -- > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > May you share freely, never taking more than you give. > -- the sqlite blessing http://www.sqlite.org/different.html > > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
On Mar 3, 2011, at 3:55 AM, Piotr Ożarowski wrote: > I don't really mind adding /usr/bin/python2 symlink just to clean Arch mess Is there any chance you would add the symlink in the next Debian stable point release? If both Debian and Python upstream added the python2 symlink in the next stable update rather than waiting for the next major release (which for Python itself is of course "never"), that could reduce the problem of systems not having the symlink installed quite significantly. James ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest.main() --catch parameter
On 03/03/2011 21:54, anatoly techtonik wrote: On Thu, Mar 3, 2011 at 10:44 PM, Michael Foord wrote: On 03/03/2011 20:31, anatoly techtonik wrote: I am looking at --help of test runner and asking the question: what is the use case for -c, --catch option? It catches keyboard interrupt and instead of just bombing out of the test run it reports all the results collected so far. Without this option interrupting a test run with a ctrl-c kills the run and reports nothing. Seeing an unexpected failure or error during a long test run and having to wait to the end of the test run to see the traceback can be annoying, this feature solves that problem. Why not just leave this behavior by default and just return -1 if the Ctrl-C was pressed? Because it means installing a signal handler which is not necessarily appropriate for all test systems. We *could* make it the default in the future (for the test runner only - not when unittest is used via the api). It doesn't look like it should be present in generic runner. I also can't find reasons to waste short option for it. Nose, django and other test runners provide this option, so it is functionality that people seem to value. There will be big problems with people complaining about BC break even if this option is not used by anyone. I don't understand this sentence, sorry. If the option is useless, people won't allow to remove it, because it will break "backward compatibility" (BC), even if they don't use this option themselves. So you want it on by default but are also worried about the backwards compatibility issues of it even existing as an option? Anyway, your assertion that the option is or may be useless is unfounded. Don't worry about it. All the best, Michael Foord All the best, Michael Foord Usage: tests.py [options] [test] [...] Options: -h, --help Show this message -v, --verboseVerbose output -q, --quiet Minimal output -f, --failfast Stop on first failure -c, --catch Catch control-C and display results -b, --buffer Buffer stdout and stderr during test runs -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] contributors survey?
On Wed, Mar 2, 2011 at 4:22 AM, Stephen J. Turnbull wrote: > Antoine Pitrou writes: > > > Following the example given in the original article, I was considering > > a single freeform question: "why did you stop contributing after your > > last patch to CPython?" (of course, that text should be decorated with > > a greeting and an introduction and the wording can be improved). > > Does anybody ever stop contributing? Occasionally, yes, but in most > cases it's just that this interval between explicit contributions > (usually patches, but also reviews, PEPs, mailing list posting, bug > reports, ...) is longer than the period since the last one. :-) > > How about > > Hello. We [the PSF?] would like to thank you for your past > patches to CPython, and take this opportunity to learn something > about how to improve our workflow. We would appreciate your > cooperation in answering the following question. > > It has been more than one year since your last patch to CPython. > We would like to understand why it's been so long [, and if there > is anything we could do to help you contribute patches more > frequently]. > > The clause in brackets is outside the scope of Antoine's wording, but > I assume that's where we're going with this. Unpythonic, without human touch, dull and boring, sounds like a letter from some executive, who was pushed for bad performance. Such letters are bad for digestion and not tasty to reply. How about making it more personal? Why the distinction of "we" and "you"? How about s/we/BDFL/ and take the tone appropriate? Hi there. BDFL was watching you, and now wants to know why are you not contributing patches anymore? If there is something for you to say, please tell everyone, or else we all may be in trouble! Thanks for being the part of Python Community. Sigh. Your English may vary. -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] contributors survey?
On Wed, Mar 2, 2011 at 5:25 AM, Westley Martínez wrote: > If I got a message like that in my mailbox I would be rather annoyed, > mark it as spam, and be less likely to contribute again. > > Just my point of view. +1 -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] contributors survey?
On Wed, Mar 2, 2011 at 2:54 PM, Andrew Svetlov wrote: >> >> Will Mercurial make things more attractive? >> > Definitely yes! I welcome upcoming migration. > >>> And, of course, very long lifecycle of the most issues greatly reduces >>> enthusisasm. >> >> True. I believe we are improving that, but perhaps that's a >> misperception on my part. >> > > I understand reasons for that situation and really doubt if process > can be significantly accelerated. > But it just very unconvenient. Actually, it can. Many people are actually interested in helping specific parts of Python, but the only way they currently have in participating in the process is to monitor the whole flow and filter interesting parts manually. No one has time for that. But it is not impossible to do - I've created an issue in case somebody would like to work with me on this http://code.google.com/p/pydotorg/issues/detail?id=8 -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Integrate the faulthandler module into Python 3.3?
Hi, Some months ago, I proposed a patch to display the Python backtrace on a segfault. The API was not stable, it was too for Python 3.2, and it was enabled by default. I wrote a module instead of a patch so it can be used on any version of Python, and it has a better API. And now I would like to include the module into Python directly to use it more easily. See http://bugs.python.org/issue11393 for the details. The module: https://github.com/haypo/faulthandler It is now possible to dump the backtrace on an user signal (eg. SIGUSR1) or after a delay in seconds (it may be useful for buildbot hangs without user interaction). You can choose to display the current thread or all threads, and in which file the trace is written. So, what do you think? Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unittest.main() --catch parameter
On Thu, Mar 3, 2011 at 11:58 PM, Michael Foord wrote: >>> >>> Without this option interrupting a test run with a ctrl-c kills the run >>> and >>> reports nothing. Seeing an unexpected failure or error during a long test >>> run and having to wait to the end of the test run to see the traceback >>> can >>> be annoying, this feature solves that problem. >> >> Why not just leave this behavior by default and just return -1 if the >> Ctrl-C was pressed? >> > > Because it means installing a signal handler which is not necessarily > appropriate for all test systems. We *could* make it the default in the > future (for the test runner only - not when unittest is used via the api). What does it mean - "not approriate"? Do signal handlers crash or kill people on some test systems? If tests are run with output buffering then it's reasonable to turn off handling of Ctrl-C. If they are executed by buildbots, I don't see how it hurts the test process. In any case there should be a way to turn Ctrl-C handing using some internal flag, but moving it into user command doesn't seem like a good idea to me. By the way, is calling unittest.main() is using unittest via api? > So you want it on by default but are also worried about the backwards > compatibility issues of it even existing as an option? Anyway, your > assertion that the option is or may be useless is unfounded. Don't worry > about it. I am worried that I won't have space to add more useful options to the runner or they will be lost for users in the abundance of highly technical parameters that runner provides. I am concerned that users will never ever understand the true Awesomeness of the New Runner (tm). ;) -- anatoly t. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
On Mar 03, 2011, at 09:55 AM, Piotr Ożarowski wrote: >I don't really mind adding /usr/bin/python2 symlink just to clean Arch >mess, but I do mind changing /usr/bin/python to point to python3 (and I >can use the same argument - "Explicit is better than implicit" - if you >need Python 3, say so in the shebang, right?). What I'm afraid of is >when we'll add /usr/bin/python2, we'll start getting a lot of scripts >that will have to be checked manually every time new upstream version is >released because we cannot assume what upstream author is using at given >point. > >If /usr/bin/python will be disallowed in shebangs on the other hand >(and all scripts will use /usr/bin/python2, /usr/bin/python3, >/usr/bin/python4 or /usr/bin/python2.6 etc.) I don't see a problem with >letting administrators choose /usr/bin/python (right now not only >changing it from python2.X to python3.X will break the system but also >changing it from /usr/bin/pytohn2.X to /usr/bin/python2.Y will break it, >and believe me, I know what I'm talking about (one of the guys at work >did something like this once)) > >[all IMHO, dunno if other Debian's python-defaults maintainers agree >with me] This all seems reasonable to me, except that I think it would be good at some point -- which might be in several years -- to point /usr/bin/python to python3. We are not there now, but I do think we will be there one day. I also don't think we have to worry about a Python 4. I'm skeptical it will ever happen, because really, how many big warts in Python 3 do you think we'll need to break in a backward incompatible way? If it does, and history holds true, it'll be 15 years from now. Then all you whippersnappers can do whatever you like. :) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
On Mar 03, 2011, at 02:17 PM, David Malcolm wrote: >On a related note, we have a number of scripts packaged across the >distributions with a shebang line that reads: > #!/usr/bin/env python >which AIUI follows upstream recommendations. Actually, I think this is *not* a good idea for distro provided scripts. For any Python scripts released by the distro, you know exactly which Python it should run on, so it's better to hard code it. That way, if someone installs Python from source, or installs an experimental version of a new distro Python, it won't break their system. Yes, this has happened to me. Also, note that distutils/setuptools/distribute rewrite the shebang line when they install scripts. >There was a proposal to change these when packaging them to hardcode the >specific python binary: > >https://fedoraproject.org/wiki/Features/SystemPythonExecutablesUseSystemPython >on the grounds that a packaged system script is expecting (and has been >tested against) a specific python build. > >That proposal has not yet been carried out. Ideally if we did this, >we'd implement it as a postprocessing phase within "rpmbuild", rather >than manually patching hundreds of files. > >Note that this would only cover shebang lines at the tops of scripts. JFDI! FWIW, a quick grep reveals about two dozen such scripts in /usr/bin on Ubuntu. We should fix these. ;) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
On Mar 03, 2011, at 09:08 AM, Toshio Kuratomi wrote: >Thinking outside of the box, I can think of something that would satisfy >your requirements but I don't know how appropriate it is for upstream python >to ship with. Stop shipping /usr/bin/python. Ship "python" in an alternate >location like $LIBEXECDIR/python2.7/bin (I think this would be >/usr/lib/python2.7/bin on Debian and /usr/libexec/python2.7/bin on Fedora >which would both be appropriate) then configure which python version is >invoked by the user typing "python" by configuring PATH (a shell alias might >also work). You could configure this with environment-modules[1]_ if Debian >supports using that in packaging. I wonder if Debian's alternatives system would be appropriate for this? http://wiki.debian.org/DebianAlternatives -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
On Thu, Mar 03, 2011 at 09:11:40PM -0500, Barry Warsaw wrote: > On Mar 03, 2011, at 02:17 PM, David Malcolm wrote: > > >On a related note, we have a number of scripts packaged across the > >distributions with a shebang line that reads: > > #!/usr/bin/env python > >which AIUI follows upstream recommendations. > > Actually, I think this is *not* a good idea for distro provided scripts. For > any Python scripts released by the distro, you know exactly which Python it > should run on, so it's better to hard code it. That way, if someone installs > Python from source, or installs an experimental version of a new distro > Python, it won't break their system. Yes, this has happened to me. Also, > note that distutils/setuptools/distribute rewrite the shebang line when they > install scripts. > > >There was a proposal to change these when packaging them to hardcode the > >specific python binary: > > > >https://fedoraproject.org/wiki/Features/SystemPythonExecutablesUseSystemPython > >on the grounds that a packaged system script is expecting (and has been > >tested against) a specific python build. > > > >That proposal has not yet been carried out. Ideally if we did this, > >we'd implement it as a postprocessing phase within "rpmbuild", rather > >than manually patching hundreds of files. > > > >Note that this would only cover shebang lines at the tops of scripts. > > JFDI! > > FWIW, a quick grep reveals about two dozen such scripts in /usr/bin on > Ubuntu. We should fix these. ;) > Note, we were unable to pass Guideline changes to do this in Fedora. Gory details of the FPC meeting are at 16:15:03 (abadger1999 == me): http://meetbot.fedoraproject.org/fedora-meeting/2009-08-19/fedora-meeting.2009-08-19-16.01.log.html The mailing list thread where this was discussed is here: http://lists.fedoraproject.org/pipermail/packaging/2009-July/006248.html Note to dmalcolm: IIRC, that also means that the Feature page you point to isn't going to happen either. Barry -- if other distros adopted stronger policies, then that might justify me taking this back to the Packaging Committee. -Toshio pgpeLOL8uwMOh.pgp Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
On Thu, Mar 03, 2011 at 09:46:23PM -0500, Barry Warsaw wrote: > On Mar 03, 2011, at 09:08 AM, Toshio Kuratomi wrote: > > >Thinking outside of the box, I can think of something that would satisfy > >your requirements but I don't know how appropriate it is for upstream python > >to ship with. Stop shipping /usr/bin/python. Ship "python" in an alternate > >location like $LIBEXECDIR/python2.7/bin (I think this would be > >/usr/lib/python2.7/bin on Debian and /usr/libexec/python2.7/bin on Fedora > >which would both be appropriate) then configure which python version is > >invoked by the user typing "python" by configuring PATH (a shell alias might > >also work). You could configure this with environment-modules[1]_ if Debian > >supports using that in packaging. > > I wonder if Debian's alternatives system would be appropriate for this? > > http://wiki.debian.org/DebianAlternatives > No, alternatives is really only useful for a very small class of problems [1]_ and [2]_. For this discussion there's an additional problem which is that alternatives works by creating symlinks. Piotr Ożarowski wants to make /usr/bin/python not exist so that scripts would have to use either /usr/bin/python3 or /usr/bin/python2. If alternatives places a symlink there, it defeats the purpose of avoiding that path in the package itself. I will note, though that scripts that have /usr/bin/env and take the route of setting the PATH would still fall victim to this. I think that environment-modules can also set up aliases. If so, that wouldbe better than setting PATH for finding and removing "python" without a version in scripts. One further note on this since one of the other messages here had a reference to this that kinda rains on this parade: http://refspecs.linux-foundation.org/LSB_4.1.0/LSB-Languages/LSB-Languages/pylocation.html The LSB is a standard that Linux distributions may or may not follow -- unlike the FHS, the LSB goes beyond encoding what most distros already do to things that they think people should do. For instance, Debian derivatives might find the software installation section of LSB[3]_ to be a bit... hard to swallow. Fedora provides a package which aims to make a fedora system lsb compliant but doesn't install it by default since it drags in gobs of packages that are otherwise not necessary on many systems. However, it does specify /usr/bin/python so getting rid of /usr/bin/python at the Linux distribution level might not reach universal aclaim. A united front from upstream python through the python package maintainers on the Linux distros would probably be needed to get people thinking about making this change... and we still would likely have the ability to add /usr/bin/python back onto a system (for instance, as part of that lsb package I mentioned earlier.) .. [1]: https://fedoraproject.org/wiki/Packaging:EnvironmentModules#Introduction .. [2]: http://fedoraproject.org/wiki/Packaging:Alternatives#Recommended_usage .. [3]: http://refspecs.linux-foundation.org/LSB_4.1.0/LSB-Core-generic/LSB-Core-generic/swinstall.html -Toshio pgpRUO8y9NO0L.pgp Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
> LGTM. Please specify what /usr/bin/python is supposed to be also
> (rather: the "python" utility). I'd like it ruled out that
> installations *only* provide python2 and python3 - "python" could
> be either one, but should be present "normally" (i.e. SHOULD
> in the RFC 2119 sense).
>
> Nitpickingly, I'd add that scripts can, of course, also specify
> python2.7 (or some such). Actually, scripts can do whatever they
> want - it's more about what they then can expect to happen.
Good points--I've taken them into account in the revised PEP, which I will
send to this list shortly.
> OTOH, my
> cynical side doubts that script authors read PEPs so it'll take either
> upstream python shipping without /usr/bin/python or consensus among the
> distros to ship without /usr/bin/python to reach the point where script
> authors realize that they need to use /usr/bin/python{2,3} instead of
> /usr/bin/python.
I don't see this as an issue: all it will take is for one user with a
different "python" to contact a developer once, referring the developer to
the PEP, and then every program that that developer ever writes in the
future will be fixed (and, if the developer's worth anything, every program
he's already written, as well). Word of mouth tends to pass these things
around quickly.
> Actually, I think this is *not* a good idea for distro provided scripts.
For
> any Python scripts released by the distro, you know exactly which Python
it
> should run on, so it's better to hard code it. That way, if someone
installs
> Python from source, or installs an experimental version of a new distro
> Python, it won't break their system.
I agree. I would personally like it if distributions would provide a python
utility that can be changed by the sysadmin without breaking anything, but
implementing such a thing would be too much to ask from many distros, so I
left it out of the first draft of the PEP. I'll put into the Notes in the
second draft, though.
Again, I think that the nature of the "python" utility is a decision that
belongs squarely to the distributions (it'll likely evolve into an "everyone
else is doing it" sort of issue for distros like Debian that don't like to
change without good reason). On the other hand, ensuring that cross-platform
support exists is an issue that should be handled by a central authority.
I almost added provisions to the second draft of the PEP for things like
"python2.6", since the same issue basically exists with them, so it would be
logical to also address their support. However, since there are far fewer
cases where minor version changes break code, and far more cases where a
specific minor version of the Python interpreter isn't and doesn't need to
be installed, I imagine a problem will occur if we make "pythonX.X"
standard: developers will use the "pythonX.X" invocations instead of
ensuring their code works on all (recent) versions of the interpreter, and
much larger issues will arise when users are forced to repeatedly install
different minor versions of the interpreter. I instead addressed this topic
in the "Notes" section.
-Kerrick Staley
On Thu, Mar 3, 2011 at 10:09 PM, Toshio Kuratomi wrote:
> On Thu, Mar 03, 2011 at 09:11:40PM -0500, Barry Warsaw wrote:
> > On Mar 03, 2011, at 02:17 PM, David Malcolm wrote:
> >
> > >On a related note, we have a number of scripts packaged across the
> > >distributions with a shebang line that reads:
> > > #!/usr/bin/env python
> > >which AIUI follows upstream recommendations.
> >
> > Actually, I think this is *not* a good idea for distro provided scripts.
> For
> > any Python scripts released by the distro, you know exactly which Python
> it
> > should run on, so it's better to hard code it. That way, if someone
> installs
> > Python from source, or installs an experimental version of a new distro
> > Python, it won't break their system. Yes, this has happened to me.
> Also,
> > note that distutils/setuptools/distribute rewrite the shebang line when
> they
> > install scripts.
> >
> > >There was a proposal to change these when packaging them to hardcode the
> > >specific python binary:
> > >
> > >
> https://fedoraproject.org/wiki/Features/SystemPythonExecutablesUseSystemPython
> > >on the grounds that a packaged system script is expecting (and has been
> > >tested against) a specific python build.
> > >
> > >That proposal has not yet been carried out. Ideally if we did this,
> > >we'd implement it as a postprocessing phase within "rpmbuild", rather
> > >than manually patching hundreds of files.
> > >
> > >Note that this would only cover shebang lines at the tops of scripts.
> >
> > JFDI!
> >
> > FWIW, a quick grep reveals about two dozen such scripts in /usr/bin on
> > Ubuntu. We should fix these. ;)
> >
> Note, we were unable to pass Guideline changes to do this in Fedora. Gory
> details of the FPC meeting are at 16:15:03 (abadger1999 == me):
>
> http://meetbot.fedoraproject.org/fedora-meeting/2009-08-19/fedora-meeting.2009-08-19-16.01.lo
Re: [Python-Dev] Support the /usr/bin/python2 symlink upstream
PEP: ??? Title: The python Utility on Unix-Like Systems Version: ??? Last-Modified: ??? Author: Kerrick Staley Status: Draft Type: Informational Content-Type: text/x-rst Created: 02-Mar-2011 Post-History: ??? Abstract == This PEP provides a convention to ensure that Python scripts can continue to be portable across *nix systems, regardless of the default version of the Python interpreter (i.e. the version invoked by the "python" utility). Recommendation * ``*nix`` software distributions should install the "python2" utility into the default path whenever a version of the Python 2 interpreter is installed, and the same for "python3" and the Python 3 interpreter. When invoked, "python2" should run some version of the Python 2 interpreter, and "python3" should run some version of the Python 3 interpreter. The same applies for the more general "python" command, which should should be installed whenever any version of Python is installed and should invoke some Python interpreter. * All new code that needs to invoke the Python interpreter should not specify "python", but rather should specify either "python2" or "python3" (or the more specific "python2.X" and "python3.X" versions; see the Notes). This distinction should be made in shebangs, when invoking from a shell script, when invoking via the system() call, or when invoking in any other context. Rationale === This is needed because some distributions alias the "python" command to Python 3, while others alias it to Python 2. Some of the latter also do not provide a "python2" command; hence, there is no way for Python 2 code (or any code that invokes the Python 2 interpreter) to reliably run on all systems without modification, because both the "python" and the "python2" commands will fail on some systems. The recommendations in this PEP provide a very simple mechanism to restore cross-platform support, with very little additional work required on the part of distribution maintainers. Notes === * Distributions can alias the "python" command to whichever version of the Python interpreter they choose. * The "pythonX.X" (e.g. "python2.6") utilities exist on some systems, on which they invoke specific minor versions of the Python interpreter. It would be wise for distribution-specific packages to take advantage of these utilities if they exist, since it will prevent code breakage if the default minor version of a given major version is changed. However, scripts intending to be cross-platform should not rely on the presence of these utilities, but rather should be tested on several recent minor versions of the target major version, compensating, if necessary, for the small differences that exist between minor versions. This prevents the need for sysadmins to install many very similar versions of the interpreter. * It would be wise for distribution-specific packages to always follow the "python2"/"python3" convention, even in code that is not intended to operate on other distributions. This will prevent problems if the distribution decides to upgrade the version of the Python interpreter that the "python" command invokes, or if the sysadmin installs a custom "python" utility with a different major version than the distribution default. Distributions can test whether they are fully following this convention by changing the "python" interpreter on a test box and checking to see if anything breaks. * If the above point is adhered to and sysadmins are permitted to change the "python" utility, then the "python" utility should always be implemented as link to the interpreter binary (or a link to a link) and not vice versa. That way, if the sysadmin does decide to replace the installed "python" file, he can do so without inadvertently deleting the previously installed binary. * The first recommendation can be ignored for systems on which the "python" command itself has traditionally been left undefined and users have always had the responsibility of linking the "python" command to the Python interpreter. * If the Python 2 interpreter becomes uncommon, scripts should nevertheless continue to use the "python3" convention rather that just "python". This will ease transition in the event that yet another major version of Python is released. * If these conventions are adhered to, it will be the case that the "python" utility is only executed in an interactive manner. Backwards Compatibility = A potential problem can arise if a script adhering to the "python2"/"python3" convention is executed on a system not supporting these commands. This is mostly a non-issue, since the sysadmin can simply create these symbolic links and avoid further problems. Copyright === This document has been placed in the public domain. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/opti
