[Python-Dev] Mark old distutils as deprecated
Hello To avoid confusion, as suggested by Akira who works on cleaning the Distutils pages on the python.org website, I would like to move http://svn.python.org/view/distutils/trunk into a branch and add a README.txt in an empty trunk to explain the current status of the package. Any objection ? Regards, Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tracker archeology
On Tue, Feb 10, 2009 at 2:23 PM, Daniel (ajax) Diniz wrote: > > If anyone is interested in being added as nosy for any category of > bugs, let me know and I'll do that as I scan the tracker. I'll take Distutils related issues, Thank you Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tracker archeology
On Wed, Feb 11, 2009 at 12:46 AM, Stephen Thorne wrote: > On 2009-02-10, Tarek Ziadé wrote: >> On Tue, Feb 10, 2009 at 2:23 PM, Daniel (ajax) Diniz >> wrote: >> >> > >> > If anyone is interested in being added as nosy for any category of >> > bugs, let me know and I'll do that as I scan the tracker. >> >> I'll take Distutils related issues, > > If you could look at a solution for http://bugs.python.org/issue1533164 > I would be eternally grateful. ok, that would be the next one I am working on in that case, Regards Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Mark old distutils as deprecated
2009/2/9 M.-A. Lemburg : > On 2009-02-08 11:15, Tarek Ziadé wrote: >> Hello >> >> To avoid confusion, as suggested by Akira who works on cleaning the >> Distutils pages on the python.org website, >> I would like to move http://svn.python.org/view/distutils/trunk into a >> branch and add a README.txt in an empty trunk >> to explain the current status of the package. >> >> Any objection ? > > No. > > It be worthwhile keeping just the setup code and adjust that > to take the distutils package from the python/ dir in order to > build separate releases of the code for upload to PyPI (ones > that basically provide the code as released in Python 2.x as > separate download for 2.(x-1) and 2.(x-2)). Indeed, and for any Python version in fact, to get early feedbacks between two Python releases. I will make releases as you mentioned, and also a development release using the svn revision, so people can try out the current trunk. >>> '%sdev-r69676' % sys.version.split()[0] '2.7a0dev-r69676' Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To 3.0.2 or not to 3.0.2?
2009/2/17 Georg Brandl : > Benjamin Peterson schrieb: >> On Tue, Feb 17, 2009 at 5:25 AM, Samuele Pedroni wrote: >>> Didn't a test fail because of this? seems the underlying issue is that this >>> part of the stdlib didn't have enough test coverage. It seems that having >>> very good/improving test coverage like is recommended for 3rd-party project >>> wanting to switch would be a good goal for 3.0 evolution too. We know from >>> PyPy experience that while always improving the test suite coverage is quite >>> spotty at times. >> >> No, a test didn't fail. Our new distutils maintainer, Tarek Ziade, >> though, has been increasing the distutils test coverage greatly. I'll add one in that area. Note that I am also planning to: - remove in the current trunk things like cmp() so the code looks similar in trunk and py3k -> so if you change something in py3k branch in distutils, if you have time please backport it to the trunk right away when appliable - release Distutils at PyPI on its own, (stable releases, and dev releases) following Marc-André suggestion. this will use externals, (see http://svn.python.org/projects/distutils/trunk/) So it should be simpler to work things out between two Python releases Regards Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] To 3.0.2 or not to 3.0.2?
2009/2/18 "Martin v. Löwis" : >> this will use externals, (see >> http://svn.python.org/projects/distutils/trunk/) > > This I don't understand. There is file named EXTERNALS.txt, but I don't > understand its purpose. This is how I work with externals. This file is used to store the svn:externals property and have it in a clear human readable text filethat can be seen in any svn viewer. If I need to change the externals I change this file and do: $ svn propset svn:externals -F EXTERNALS.txt $ svn ci . EXTERNALS.txt -m "comment" then, if you do a checkout of http://svn.python.org/projects/distutils/trunk it will grab Python's Lib/distutils. Let me know if this is not wanted. I can drop it it's no big deal. Regards Tarek > > Regards, > Martin > -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Greg Ward email
Hello, I am trying to reach Greg Ward to get a maintainer access to Distutils at PyPI, but his email address at python.net (and some other) doesn't work anymore. Anyone knows how to reach him ? Thanks Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Greg Ward email
On Mon, Feb 23, 2009 at 6:43 PM, A.M. Kuchling wrote: > On Mon, Feb 23, 2009 at 02:16:17PM +0100, Tarek Ziadé wrote: >> I am trying to reach Greg Ward to get a maintainer access to Distutils >> at PyPI, but his email address at python.net (and some other) doesn't >> work anymore. > > Greg's website at www.gerg.ca (not a typo!) has e-mail addresses. Yes that's were I tried at first > > However, the package index entry at > http://pypi.python.org/pypi/Distutils/1.0 lists Greg as the author, > but it's not actually owned by him; it looks like some random person > registered it. So we should probably just reassign the package to Tarek. That would be great, so i can start to release the standalone versions, Thanks Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Distutils in PEP 291
Hello, If no one objects, I would like to: - put Distutils back into PEP 291 for Python 2.3 compatibility - fix the two issues that prevent the current trunk to run with Python 2.3 to 2.5 (see http://bugs.python.org/issue5052 and the patch I worked on) Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
On Wed, Mar 25, 2009 at 1:25 PM, Antoine Pitrou wrote: > Paul Moore gmail.com> writes: >> >> 3. Setuptools, unfortunately, has divided the Python distribution >> community quite badly. > > Wait a little bit, and it's gonna be even worse, now that buildout and pip > seem > to become popular. For example, the TurboGears people are considering > switching > from setuptools to pip... > > Tarek is now doing a lot of very useful work on distutils (thanks Tarek!), but > I'm not sure it'll be enough to stop people from converting to whatever of the > many build/packaging systems which have been popping up recently. Combined > with > the current trend that everything must be exploded into lots of interdependent > but separately packaged libraries (the paste/pylons syndrome... try pulling > something like TurboGears2 and see how many third-party packages it > installs), I > fear this is going to generate a very painful user/developer experience :-( > I think we are in a point in Python development where we need to clearly define how dependencies work. And this has to be defined in Python (in Distutils) for the sake of standardization. I think the Language Summit tomorrow is a good place to discuss about these problems, and to make sure pip, setuptools and zc.buildout rely on the same standards and pieces. PEP 376 is my first attempt to make it happen, and my goal is to see another pre-PEP coming out of thea language summit, adressing the dependencies problem. I can't hear that setuptools has divided the Python community. It has provided solutions to real problems we had in web development. It's unperfect, and it has to be fixed and integrated into Python. But it should not be done outside Python imho. If you had worked with Zope 5 years ago, you would understand what setuptools and zc.buildout brought to these communities. And today's experience is a whole less painful trust me. But I agree that the sizes of the packages are too small now, and it has gone to far. Installing a web app like Plone is scary (+100 packages) But this is the responsability of Zope, TG, etc to distribute their packages in bigger pieces I guess. (I can't wait to be at the summit ! :)) Cheers Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
On Wed, Mar 25, 2009 at 2:45 PM, Antoine Pitrou wrote: > Tarek Ziadé gmail.com> writes: >> >> But I agree that the sizes of the packages are too small now, and it has gone >> to far. Installing a web app like Plone is scary (+100 packages) > > I am working on a TurboGears2-based app and I just did a count of the .egg > packages in the virtualenv. There are 45 of them > > People should really stop splitting their work into micro-libraries (with such > ludicrous names as "AddOns" or "Extremes", I might add (*)), and myriads of > separately-packaged plugins (the repoze stuff). The Twisted approach is much > saner, where you have a cohesive whole in a single package. Yes but this means that you have to wait for the next version of the "big" package when a bug is corrected or a feature added, or you need to patch it. (or maybe use the namespace trick to override it) Having them as separated package allow different release cycles, which makes it more "agile". I think there should be a right middle between a big package and a myriad of small ones. In Zope for instance, we could probably reunite a lot of package in a single package, because they don't evolve much anymore. (things like zope.interface) Regards Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
On Wed, Mar 25, 2009 at 3:08 PM, Paul Moore wrote: > > Hence my comment about "dividing the community". From my limited > perspective, it's about no longer having a standard Windows binary > distribution format used by all, not about some sort of ideological > battles. Sorry for being unclear. Are you coming to the Summit tomorrow ? If not, I'll bring up this point. I whish we will be able in the near future to build a team of people from various area of expertise, to work on these problems. Tarek > > Paul. > -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
Yes, I'll try to blog today the initial list of topics that will be discussed during the Summit, Regards Tarek 2009/3/25 Jesse Noller : > Yes it's on the agenda > > On Mar 25, 2009, at 12:46 PM, s...@pobox.com wrote: > >> Is setuptools/distutils/whatever on the agenda for the tomorrow's language >> summit? Or is there some other get-together at PyCon for this? >> >> Skip >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/jnoller%40gmail.com > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
On Thu, Mar 26, 2009 at 3:26 AM, Stephen J. Turnbull wrote: > Tennessee Leeuwenburg writes: > > GSOC? > > No. This is territory that nobody knows how to mentor yet, ya know? > Try it now, and the poor student is likely to find him or herself at > the center of a firestorm! No, we wil > > Maybe next year > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
On Thu, Mar 26, 2009 at 3:29 AM, Tarek Ziadé wrote: > On Thu, Mar 26, 2009 at 3:26 AM, Stephen J. Turnbull > wrote: >> Tennessee Leeuwenburg writes: >> > GSOC? >> >> No. This is territory that nobody knows how to mentor yet, ya know? >> Try it now, and the poor student is likely to find him or herself at >> the center of a firestorm! > > No, we wil oups, sent to early sorry -> No, we will hopefully have great PEPs after the language summit tomorrow so the students will be OK ;) There's already a GSOC topic for Disutils about building a distributed test system using PyPI, and I'm hoping to add more topics on packaging after the Summit. Cheers, Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Packaging Survey first results + Summit schedule
Hi, Sorry for the cross-post, but it seemed appropriate since packaging is being discussed in python-dev tonight, - Here are the first results for the packaging survey: http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first-results/ - And tomorrow's Summit schedule for the packaging part : http://tarekziade.wordpress.com/2009/03/26/pycon-language-summit-is-tomorrow/ Please comment (in the appropriate list or in my blog) if you have something you would like to say or see addressed during the Summit and you are unable to be present. (I am already trying to summarize what has been said in python-dev today but I am not sure I'll be able to read everything before tomorrow) Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
On Thu, Mar 26, 2009 at 5:19 AM, David Cournapeau wrote: > On Thu, Mar 26, 2009 at 12:02 PM, Nick Coghlan wrote: > >> If that perception is accurate, then any changes likely need to focus on >> the *opposite* end of the toolchain: the part between the "> packaging spec>" and the end users. > > Yes - but is this part the job of python ? > >> In other words: Given an egg, how easy is it for a packager/distributor >> to create a platform specific package that places the files in the >> correct locations for that particular platform (regardless of how >> arbitrary those rules may appear to the original developers)? I think Distutils (and therefore Setuptools) should provide some APIs to play with special files (like resources) and to mark them as being special, no matter where they end up in the target system. So the code inside the package can use these files seamessly no matter what the system is and no matter where the files have been placed by the packager. This has been discussed already but not clearly defined. > > Why coming from eggs and not from the build tool provided by python > itself (distutils) ? I don't see what eggs brings - specially since > the format is not even standardized. I don't think the "egg as a format" part has anything to do with this discussion. It's just a zip file with a specific layout. What's important is what we store in the .egg-info directory (or file) which has two standards today (one in Distutils, one in Setuptools) but wich should get a common standard soon (PEP 376) Cheers Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Integrate BeautifulSoup into stdlib?
On Thu, Mar 26, 2009 at 5:32 AM, David Cournapeau wrote: > If distutils was split into different modules (one for the build, one > for the compiler/platform configuration, one for the installation), > which could be extended, tweaked, it would be much better. But the > distutils design makes this inherently very difficult (commands). I am not sur why the command design is a problem here. And I think Distutils features are not far from what you need, if you look at things like customize_compiler, which is called by build_clib. I'm ready to discuss your use case in Distutils-SIG, if you have a complete example etc. Cheers Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
On Thu, Mar 26, 2009 at 2:37 PM, Barry Warsaw wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On Mar 26, 2009, at 2:31 PM, Guido van Rossum wrote: > >>> One thing that /would/ be helpful though is the ability to list all the >>> resources under a specific package path. This is (I think) one use case >>> that pkg_resource fails to support and it's the one place that I've had >>> to >>> drop down to file system introspection. >>> >>> Think: the package-y flavor of >>> os.listdir(os.path.dirname(package.__file__)) >> >> Good idea. Can I suggest that API this takes a glob-style pattern? (Or >> to be fully general, a list of patterns and a list of exclusion >> patterns.) > > Good idea. Thinking back on my typical use cases, a glob style pattern > would be enough. Usually I'm asking "give me all the .py files in this > package" or "I'm looking for the .txt files in this path". I think shutil.copytree new ignore mechanism handles this use case pretty well (see the ignore_patterns factory in http://docs.python.org/library/shutil.html) Maybe we could use the same pattern. Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
On Thu, Mar 26, 2009 at 2:36 PM, Tres Seaver wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Barry Warsaw wrote: >> On Mar 25, 2009, at 6:06 PM, Tennessee Leeuwenburg wrote: >> >>> For case one, where I want to install additional functionality into my >>> system Python interpreter "forever", it would be great to have my >>> system >>> manage this. >> >> In fact, I think it /has/ to. I'll go further and say that I'm very >> wary of using easy_install and the like to install non-distro provided >> packages into the system Python. Many Linux distros require a >> functioning system Python to operate and the distros (should) take >> great care to make sure that if you install package X for application >> Y, you won't break essential system service Z. Once you start >> installing your own stuff in the system Python's site-packages, you >> break the warranty. > > Exactly: I never use easy_isntall to put packages into the system > python. in fact, I only use it inside a virtalenv-generated isolated > environment. But how each application use 'Python the interpreter' ? application = an interpreter + a list of paths + a script to run your application. Isolating everything with virtualenv or zc.buildout is a way to go, but what is really important is not where each package you use is located but how Python see and load them. So if we, for once, forget about the central site-packages and define some kind of configuration process that is run when every script is launched to decide what packages should be loaded, we could seperate "python the interpreter" from "python the pile of packages" If some kind of API + configuration file could adress this need, maybe things could be simpler. You wouldn't fight against a central site-packages or a per-user site-packages. You would have to explicitely provide it Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] My summit notes (packaging)
Here's a wrapup of my notes for the Distutils part (with Jim's help).
The PyPI part will come later.
I have presented the various problems developers have with packaging
today, wether they are using Distutils, Setuptools or any other third
party tools out there.
Here's the short-list:
- Distutils code is still undertested, with over 100 issues to
take care of in the tracker.
- Setuptools is the de-facto extension of Distutils. It brings
a lot of good features, but some features are controversial
amongst the community.
- We have now two tools that install a package and look for its
dependencies at PyPI and install them: pip and easy_install.
The latter is included in setuptools. They both aim at the
same goal besides a few differences, and they both rely
on a new metadata introduced by setuptools, wich is.
"install_requires". This new metadata extends the metadata.
described in PEP 314 but is slightly different from.
what is descibred in the Draft PEP 345 ("Requires").
..
PEP 345 introduces "Requires" and "Provides" wich are
are implemented in Distutils and PyP, but are not
widely used. 40 out of +4000 if I remember correctly. Martin will
correct me here if I am wrong.
- pip, easy_install and distutils use different formats for installed
metadata. For instance, Distutils installs a PKG-INFO file whereas
setuptools creates a directory with the PKG-INFO file in it.
- There's no way to uninstall a package. The only way to do it
is to remove manually the directory located in your site-packages
and remove the line located in the .pth file that might point to it.
- There's no standard way to use resource files, for instance
to be able to read or write a resource file that might be installed
in a specific place on the target system by the system packager.
Most of the time people are just working with files located.
in the package directory.
Ian Bicking and Jim Fulton gave short presentations of virtualenv
and zc.buildout. These tools will let you build an isolated environment,
where you can install some packages of certain versions, that play well
together to run an application, without having to put anything into your
site-packages. Many agreed that adding packages into the system's
site-packages for an application to run is not a good idea.
Also, many agreed that these third party tools are doing their.
job and that we shouldn't try to include some of the features they.
provide in any way in the standard library. Such as supporting multiple.
versions for the same package for example.
Instead, the idea is to try to reduce the scope of Distutils and to
provide some better API and conventions for people to access to the package
metadata (wether it's installed or not) because people have different
needs and purposes using these data.
*What's next ?*
Guido provided a high level plan for the work to be done in packaging.
Let me quote him:
- standardize more metadata, especially including dependencies,.
and APIs for accessing the metadata and dependencies.
- there should be an API to get metadata for a package without
actually executing any of the package's installation script.
- this will go into the stdlib for Python 2.7 and 3.1
- it will be provided as separately downloadable backports for prior
versions (maybe using a bootstrap not unlike ez_install)
- keep distutils, but start deprecating certain higher-level
functionality in it (e.g. bdist_rpm)
- don't try to provide higher-level functionality in the stdlib, but
instead let third party tools built on top of these core APIs compete
So we are currently working this week to flesh out this plan, and we
will send a mail to Distutils-SIG with the ouput asap.
Cheers
Tarek
--
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My summit notes (packaging)
2009/3/27 P.J. Eby : > Also, it's quite likely that platform-specific dependencies may exist as > well. It might be possible to accommodate these things with a sufficiently > flexible format, but currently, the only way to handle them with > distutils/setuptools is in the setup script. > Yes, we are working on these particular issues when some metadata require some code to run. But most of the time, a package is able to express its dependencies in a static file. Regards Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] "setuptools has divided the Python community"
On Fri, Mar 27, 2009 at 3:48 PM, M.-A. Lemburg wrote: > More importantly: > > Why is the non-use of a command by a single Python developer enough > motivation to remove a useful feature of distutils that's been in > use by many others for years ? >From the discussions I had with RPM packagers, bdist_rpm is hard to use to comply with all the different RPM-based systems out there, Fedora, Red Hat, etc.. I think that each OS community should maintain its own tool, that complies to the OS standard (wich has its own evolution cycle) Of course this will be possible as long as Distutils let the system packager find/change the metadata in an easy way. I think this is the same rationale for debian packages. Right now people tend to use external tools like stdeb and they are OK with it (but still gets problems extracting stuff out of Python packages at this point) Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] My summit notes (packaging)
On Fri, Mar 27, 2009 at 9:54 PM, Kevin Teague wrote: > > > Tarek, was there any further discussion on "Requires" vs "install_requires" > or any decisions made on what to do about this? > > (I've got a +1 ready for including 'install_requires' in the standard > package metadata and marking 'Requires' as deprecated if that was put forth > :P) Yes that is what we were thinking of doing. (deprectating Requires and Provides and and install_requires) We are working on it. We'll try to organize our work in the wiki in the comng days, so people can participate. Regards Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Package Management - thoughts from the peanut gallery
Guys, I have taken the commitment to lead these tasks and synchronize the people that are willing to help on this. We are working on several tasks and PEPS to make things happen since the summit. There's no public roadmap yet on when things will be done (because there's no 100% certitude yet on what shall be done). But that it will probably be too late to see it happen in 3.1. Python 2.7 will be our target. The tasks discussed so far are: - version definition (http://wiki.python.org/moin/DistutilsVersionFight) - egg.info standardification (PEP 376) - metadata enhancement (rewrite PEP 345) - static metadata definition work (*) - creation of a network of OS packager people - PyPI mirroring (PEP 381) Each one of this task has a leader, except the one with (*). I just got back from travelling, and I will reorganize http://wiki.python.org/moin/Distutils asap to it is up-to-date. If you want to work on one of this task or feel there's a new task you can start, please, join Distutils SIG or contact me, Regards Tarek On Fri, Apr 3, 2009 at 6:55 AM, Stephen J. Turnbull wrote: > Chris Withers writes: > > > Personally I feel all of the above are perfectly possible, and can't see > > anyone being left unhappy by them. I'm sure I've missed something then, > > otherwise why not make it happen? > > Labor shortage. > > We will need a PEP, the PEP will need a sample implementation, and > a proponent. Who's gonna bell the cat? > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] UnicodeDecodeError bug in distutils
On Fri, Apr 3, 2009 at 2:25 AM, Ben Finney wrote: > "Phillip J. Eby" writes: > >> However, there's currently no standard, as far as I know, for what >> encoding the PKG-INFO file should use. > > Who would define such a standard? PEP 376 where we can explain that all files in egg-info should be in a specific encoding > My vote goes for “default is UTF-8”. +1 > >> Meanwhile, the 'register' command accepts Unicode, but is broken in >> handling it. […] how so ? Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Package Management - thoughts from the peanut gallery
On Fri, Apr 3, 2009 at 2:56 PM, Olemis Lang wrote: > > BTW ... I see nothing about removing dist_* commands from distutils ... > > Q: Am I wrong or it seems they will remain in stdlib ? Ok, beware that what I am writing here is for the long term. There are no plans yet to remove things right now. Maybe some things for 3.1, as long as it is clearly defined and non-controversial. And this is not the most urgent thing to take care of. So, Some commands are not really used by the OS packagers, whether because these commands don't provide what packagers need, whether because they are unable to let the packagers configure them the way they would like to. Packagers still need to tell us why and how to make things better. Some people like Toshio or Matthias are already helping a lot on this. We are making a lot of progress since the summit to share our point of views. So I'd put this task under "creation of a network of OS packager people" (them+others) And in detail : 1/ define with them the precise usage of Distutils commands in each OS community 2/ define if there's a leading project that could take care of building OS-dependant packages, using packages built by/with Distutils 4/ see what needs to be done in Distutils to let these projects play with Python packages whithout pain. 5/ finally, see what could be externalized/removed from Distutils in favor of these third-party projects. This is roughly what Guido was talking about when he said we would remove things like bdist_rpm from the stdlib : it's too OS-specific for the stdlib to do a good job in this area. To discuss this plan in details, let's move to Distutils-SIG Cheers Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Package Management - thoughts from the peanut gallery
On Fri, Apr 3, 2009 at 6:20 PM, Chris Withers wrote: > Tarek Ziadé wrote: >> >> I have taken the commitment to lead these tasks and synchronize the people >> that are willing to help on this. > > Good, I'm one of those people, Great ! > sadly my only help may be to ask "how is this > bit going to be done?". I'll work on the wiki this week end for that > >> The tasks discussed so far are: >> >> - version definition (http://wiki.python.org/moin/DistutilsVersionFight) >> - egg.info standardification (PEP 376) >> - metadata enhancement (rewrite PEP 345) >> - static metadata definition work (*) > > These all seem to be a subset of the last one, right? Sorry I used "task" I should have used "topics". We are trying to have a list of well-defined, isolated tasks. Theses tasks are built upon the discussions we have in these topics. The last topic (static metadata) might generate new tasks and/or complete existing tasks. >> - PyPI mirroring (PEP 381) > > I don't see why PyPI isn't just ported to GAE with an S3 data storage bit > and be done with it... Offline mirrors for people behind firewalls already > have solutions out there... GAE+S3 is just an implementation imho. We still need a mirroring protocol ala CPAN and features in client softwares to use them. (as defined in 381) > >> Each one of this task has a leader, except the one with (*). I just got >> back >> from travelling, and I will reorganize >> http://wiki.python.org/moin/Distutils asap to it is up-to-date. > > Cool, is this the focal point to track your activities? Exactly. And Distutils-SIG is the mailing list to discuss in ;) > >> If you want to work on one of this task or feel there's a new task you can >> start, please, join Distutils SIG or contact me, > > Well, I think my "big list" breaks down roughly as tasks, of which I think > the stuff you're already doing will hopefully take care of the first 2, but > what about the rest. If labour shortage is all that's stopping this, then > let me know ;-) > Please discuss these new points in Distutils-SIG Cheers Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Google Summer of Code/core Python projects - RFC
> -> I'm also skeptical that this is a good SoC project in the first place. What is a good SoC project from your point of view ? > -> Coming up with a wrapper for, say, Apple Keychain, could be a good > -> project. Coming up with a unifying API for all keychains is out of > -> scope, IMO; various past attempts at unifying APIs have demonstrated > -> that creating them is difficult, and might require writing a PEP > -> (whose acceptance then might not happen within a summer). > > Well, that's a more unassailable argument and one I agree with ;). For this case, the student work is not a "dumb" work consisting of writing code on an already-thaught PEP... Part of the work will consist of working on a PEP-like document, and on building APIs for various keychains and see if we can have an unified one. I doubt the PEP-like document can be written before writing prototypes APIs for various keychains has been done. At the end of the summer, if we come up with a nice unified API, I'd like to include it to Distutils for the "register" command, and maybe write a PEP to have it as part of the standard library because it makes sense to have this kind of feature imho. Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Google Summer of Code/core Python projects - RFC
Ok what about this then: I am changing the scope a little bit, and I think the students will be fine with this change since it's the same work. "The project will consist of creating a plugin system into Distutils to be able to store and retrieve the username/password used by some commands, without having to store it in *clear text* in the .pypirc file anymore. The student will also provide some plugins for a maximum number of existing keyring systems. Some of these plugins might be included in Distutils, and some of them in a third-party package. " Regards Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 382: Namespace Packages
On Wed, Apr 15, 2009 at 9:22 PM, P.J. Eby wrote: > At 02:52 PM 4/15/2009 -0400, A.M. Kuchling wrote: >> >> On Wed, Apr 15, 2009 at 01:59:34PM -0400, P.J. Eby wrote: >> > Please see the large number of Zope and PEAK distributions on PyPI as >> > minimal examples that disprove this being the common use case. I expect >> > you will find a fair number of others, as well. >> ... >> > In other words, the "base package" scenario is the exception these days, >> > not the rule. I actually know specifically of only one other such >> > package besides your mx.* case, the logilab ll.* package. >> >> Isn't that pretty even, then? zope.* and PEAK are two examples of one >> approach; and mx.* and ll.* are two examples that use the base package >> approach. Neither approach seems to be the more common one, and both >> are pretty rare. > > If you view the package listings on PyPI, you'll see that the "pure" > namespaces currently in use include: > > alchemist.* > amplecode.* > atomisator.* > bda.* > benri.* > beyondskins.* > bliptv.* > bopen.* > borg.* > bud.* > ... > > This is just going down to the 'b's, looking only at packages whose PyPI > project name reflects a nested package name, and only including those with > entries that: > > 1. use setuptools, > 2. declare one or more namespace packages, and > 3. do not depend on some sort of "base" or "core" package. > > Technically, setuptools doesn't support base packages anyway, but if the > organization appeared to be based on a "core+plugins/addons" model (as > opposed to "collection of packages grouped in a namespace") I didn't include > it in the list above -- i.e., I'm bending over backwards to be fair in the > count. > > If somebody wants to do a formal count of base vs. pure, it might provide > interesting stats. I initially only mentioned Zope and PEAK because I have > direct knowledge of the developers' intent regarding their namespace > packages. > > However, now that I've actually looked at a tiny sample of PyPI, it's clear > that the actual field use of pure namespace packages has positively exploded > since setuptools made it practical to use them. > > It's unclear, however, who is using base packages besides mx.* and ll.*, > although I'd guess from the PyPI listings that perhaps Django is. (It seems > that "base" packages are more likely to use a 'base-extension' naming > pattern, vs. the 'namespace.project' pattern used by "pure" packages.) > > Of course, I am certainly not opposed to supporting base packages, and > Martin's version of PEP 382 is a plus for setuptools because it would allow > setuptools to better support the "base" scenario. > > But pure packages are definitely not a minority; in fact, a superficial > observation of the full PyPI list suggests that there may be almost as many > projects using pure-namespace packages, as there are non-namespaced > projects! > In the survey I have done on packaging, 34% of the people that answered are using setuptools namespace feature, which currently makes it impossible to use the namespace for the base package. Now for the "base" or "core" package, what peoplethat uses setuptools do most of the time: 1- they use zc.buildout so they don't need a base package : they list in a configuration files all packages needed to build the application, and one of these package happen to have the scripts to launch the application. 2 - they have a "main" package that doesn't use the same namespace, but uses setuptools instal_requires metadata to include namespaced packages. It acts like zc.buildout in some ways. For example, you mentioned atomisator.* in your example, this app has a main package called "Atomisator" (notice the upper A) that uses strategy #2 But frankly, the "base package" scenario is not spread these days simply because it's not obvious to do it without depending on a OS that has its own strategy to install packages. For example, if you are not under debian, it's a pain to use logilab packages because they use this common namespace for several packages and a plain python installation of the various packages won't work out of the box under other systems like windows. (and for pylint, I ended up creating my own distribution for windows...) So : - having namespaces natively in Python is a big win (Namespaces are one honking great idea -- let's do more of those!) - being able to still write some code under the primary namespace is something I (and lots of people) wish we could do with setuptools, so it's a big win too. Regards, Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Help on issue 5941
Hello, I need some help on http://bugs.python.org/issue5941 The bug is quite simple: the Distutils unixcompiler used to set the archiver command to "ar -rc". For quite a while now, this behavior has changed in order to be able to customize the compiler behavior from the environment. That introduced a regression because the mechanism in Distutils that looks for the AR variable in the environment also looks into the Makefile of Python. (in the Makefile then is os.environ) And as a matter of fact, AR is set to "ar" in there, so the -cr option is not set anymore. So my question is : should I make a change into the Makefile by adding for example a variable called AR_OPTIONS then build the ar command with AR + AR_OPTIONS *or* that doesn't make sense and I just need to change the behavior so it doesn't look for AR into the Makefile. (just in os.environ) Thanks Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Help on issue 5941
On Thu, May 7, 2009 at 11:50 AM, David Cournapeau wrote: > Then, in the customize_compiler function, set archiver to $AR + > $ARFLAGS. IOW, just copying the logic used for e.g. ldshared, > > I can prepare a patch if you want, I am ok on Distutils side, but I wouldn't mind some help on the makefile/configure side Even if I could mimic what's in there, I am not confident enough yet. Please do so, by attaching your patch in the issue, Thanks Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Help on issue 5941
On Thu, May 7, 2009 at 1:37 PM, David Cournapeau wrote: > On Thu, May 7, 2009 at 7:07 PM, Tarek Ziadé wrote: >> On Thu, May 7, 2009 at 11:50 AM, David Cournapeau wrote: >>> Then, in the customize_compiler function, set archiver to $AR + >>> $ARFLAGS. IOW, just copying the logic used for e.g. ldshared, >>> >>> I can prepare a patch if you want, >> >> I am ok on Distutils side, but I wouldn't mind some help on the >> makefile/configure side > > Ok, I ended up making a patch for everything. I tested it on Linux, > where it fixed the issue while keeping the customization (both AR and > ARFLAGS can be customized through environment variables). > > numpy now builds under python 2.7, > > cheers, > > David > ok thanks David, I'll complete your patch with the test I have written for this issue and commit it so it's included in 2.7/3.1. Notice that from the beginning, the unixcompiler class options are never used if the option has been customized in distutils.sysconfig and present in the Makefile, so we need to clean this behavior as well at some point, and document the customization features. By the way, do you happen to have a buildbot or something that builds numpy ? If not it'll be very interesting: I wouldn't mind having one numpy track running on the Python trunk and receiving mails if something is broken. Regards Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Help on issue 5941
On Thu, May 7, 2009 at 2:11 PM, David Cournapeau wrote: > But I don't know if that's easy to set up such as both python and > numpy are built from sources. I don't know about the numpy part, but the PyBots project code could be a source of inspiration for the Python part http://code.google.com/p/pybots/source/browse/trunk/master/community.cfg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] py3k build broken
On Thu, May 7, 2009 at 11:36 PM, Eric Smith wrote: > Tarek: > > With you ARFLAGS change, I now get the following error on a 32 bit Fedora 6 > box. I've done "make distclean" and "./configure": Sorry yes, I am on it now, the produced Makefile is broken, until then you can change it <<< line 71 arflags=...@arflags@ <<< ARFLAGS= cr <<< Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] py3k build broken
On Thu, May 7, 2009 at 11:51 PM, Eric Smith wrote: > Tarek Ziadé wrote: >> >> On Thu, May 7, 2009 at 11:36 PM, Eric Smith wrote: >>> >>> With you ARFLAGS change, I now get the following error on a 32 bit Fedora >>> 6 >>> box. I've done "make distclean" and "./configure": >> >> Sorry yes, I am on it now, the produced Makefile is broken, until then >> you can change it > > ... > > No problem. I'll wait. I have fixed configure by runing autoconf, everything should be fine now Sorry for the inconvenience. Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Adding a "sysconfig" module in the stdlib
Hello, I am trying to refactor distutils.log in order to use logging but I have been bugged by the fact that site.py uses distutils.util.get_platform() in "addbuilddir". The problem is the order of imports at initialization time : importing "logging" into distutils will make the initialization/build fail because site.py wil break when trying to import "logging", then "time". Anyways, So why site.py looks into distutils ? because distutils has a few functions to get some info about the platform and about the Makefile and some other header files like pyconfig.h etc. But I don't think it's the best place for this, and I have a proposal : let's create a dedicated "sysconfig" module in the standard library that will provide all the (refactored) functions located in distutils.sysconfig (but not customize_compiler) and disutils.util.get_platform. This module can be used by site.py, by distutils, and others, and will focus on this role. Regards Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 376 : Changing the .egg-info structure
Hello I'm proposing this PEP, which has been discussed in Distutils-SIG, for inclusion in Python 2.7 and 3.2 http://www.python.org/dev/peps/pep-0376/ Please comment ! Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
2009/5/15 P.J. Eby : > At 12:21 AM 5/15/2009 +0200, Tarek Ziadé wrote: >> >> Hello >> >> I'm proposing this PEP, which has been discussed in Distutils-SIG, for >> inclusion in Python 2.7 and 3.2 >> >> http://www.python.org/dev/peps/pep-0376/ >> >> Please comment ! > > I'd like to reiterate my suggestion that the uninstall record include size > and checksum information, ala PEP 262's "FILES" section. This would allow > the uninstall function to validate whether a file has been modified, and > thus prevent uninstalling a locally-modified file, or a file installed in > some other way. good point, I'll re-work that part > > It may also be that providing an uninstall API that simply yields files to > be uninstalled, with data about their existence/modification status, would > be more useful than a blind uninstall operation with a filter function. Sure we could have it in that shape, I'll work on this as well. > > Also, the PEP doesn't document what happens if a single file was installed > by more than one package. It does: "...as long as they are not mentioned in another RECORD file..." > Ideally, a file with identical size/checksum that > belongs to more than one project should be silently left alone, and a file > installed by more than one project with *different* size/checksum should be > warned about and left alone. I think the path is the info that should be looked at. And a warning could be raised like you said if a file was manually modified. But I don't think you want to leave alone a file with identical size/checksum that belongs to more than one project when it's not the same absolute path. Here's an example why : if two different packages includes the "feedparser.py" module (from the FeedParser project) for conveniency, and if you remove one package, you *do* want to remove its "feeparser.py" module even if it exists in the other project. So it's rather changing the PEP text like this: "...as long as they are not mentioned in another RECORD file, with the same size/checksum..." > > Next, the doc for the metadata API functions seems quite sparse. ISTR that > I've previously commented on such issues as case- and > punctuation-insensitivity of project names, and '/' separation in egg_info > subpaths, but these don't seem to have been incorporated into the current > version of the PEP. > > These are important considerations in general, btw, because project name and > version canonicalization and escaping are an important part of both > generating and parsing .egg-info filenemaes. At minimum, the relevant > setuptools docs that define these standards should be cited. I'll add more info on that part accordingly then, > > Finally, the "Definitions" section also claims that a project installs one > or more packages, but a project may not contain *any* packages; it may have > a standalone module, or just a script, data, or metadata. > > ok Thanks for the feedbacks -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
Ok I've changed the PEP with all the points you mentioned, if you want to take a look. 2009/5/15 P.J. Eby : > Next, the doc for the metadata API functions seems quite sparse. ISTR that > I've previously commented on such issues as case- and > punctuation-insensitivity of project names, and '/' separation in egg_info > subpaths, but these don't seem to have been incorporated into the current > version of the PEP. > > These are important considerations in general, btw, because project name and > version canonicalization and escaping are an important part of both > generating and parsing .egg-info filenemaes. At minimum, the relevant > setuptools docs that define these standards should be cited. I need to find back your comments for this part, I must have missed them. That's the last part I didn't work out yet on the current PEP revision. Regards Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
Yes, I don't think it's relevant to optimize install/uninstall code in Python. In the whole PEP 376 proposal, the only part that will need care will be the code that browses sys.path. On Fri, May 15, 2009 at 9:50 AM, Dirkjan Ochtman wrote: > On Fri, May 15, 2009 at 8:32 AM, Jeroen Ruigrok van der Werven > wrote: >> Agreed. Within FreeBSD's ports the installed package registration gets a MD5 >> hash per file recorded. Size is less interesting though, since essentially >> this information is encapsulated within the hash. Remove one byte from the >> file and your hash is already different. And the case of a collision for >> this kind of registration is sufficiently small to need the size >> information. > > Size is nice because it's much cheaper to check. I don't know if mass > uninstalls will be so common that this is actually something we have > to worry about, though. > > Cheers, > > Dirkjan > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] LZW support in tarfile ?
Hello, I want to remove the usage of the "tar" command in Distutils in favor or the "tarfile" module. But, there's an option in Distutils.make_archive to create a tarball using the "compress" [1] program rather than gzip or bzip2. Using tar -Z, it will pipe it to the compress program if present. This program implements the LZW algorithm [2]. The LZW used to be patented but this patent seem to be expired in every country now [3]. On Distutils side I can work things out so the tar archive created can be piped to an arbitraty compression program when it is not compressed using bzip2 or gzip; But I was wondering if we should we add a LZW support in tarinfo, besides gzip and bzip2 ? Although this compression standard doesn't seem very used these days, Regards Tarek [1] http://en.wikipedia.org/wiki/Compress [2] http://en.wikipedia.org/wiki/LZW [3] http://www.unisys.com/about__unisys/lzw -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] LZW support in tarfile ?
Ok thanks for all the feedback, I'll remove compress support Tarek On Mon, May 18, 2009 at 10:06 AM, Nick Craig-Wood wrote: > Michael Foord wrote: >> Antoine Pitrou wrote: >> > Tarek Ziadé gmail.com> writes: >> > >> >> But I was wondering if we should we add a LZW support in tarinfo, >> >> besides gzip and bzip2 ? >> >> >> >> Although this compression standard doesn't seem very used these days, >> >> >> > >> > It would be more useful to add LZMA / xz support. >> > I don't think compress is used anymore, except perhaps on old legacy >> > systems. >> > On my Linux system, I have lots of .gz, .bz2 and .lzma files, but >> > absolutely no >> > .Z file. >> >> I've seen the occasional .Z file in recent years, but never that I >> recall for a Python package. > > On my unix filesystem (which has files stretching back over 20 years) > I find only two .Z files, one dated 1989 and one 2002. I think you > can safely say that compress is gone! > > The worst you are doing by removing compress support is getting the > user of some ancient platform to download one of the binaries here > first. > > http://www.gzip.org/#exe > >> As plugging in external compression tools is less likely to work >> cross-platform wouldn't it be both easier and better to deprecate (and >> not replace) the compress support. > > Agreed. > > -- > Nick Craig-Wood -- http://www.craig-wood.com/nick > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
On Sat, May 16, 2009 at 6:55 PM, P.J. Eby wrote: > > 1. Why ';' separation, instead of tabs as in PEP 262? Aren't semicolons a > valid character in filenames? I am changing this into a . for now. What about Antoine's idea about doing a quote() on the names ? >From my point of view seems more simple to deal with, if 3rd-party tools want to work on these files without using pkgutil or Python. > > 4. There should probably be a way to iterate over the projects in a > directory, since it's otherwise impossible for an installation tool to find > out what project(s) "own" a file that conflicts with something being > installed. Alternatively, reshaping the file API to allow querying by path > as well as by project might work. I am adding a "get_projects" api: get_projects() -> iterator Provides an iterator that will return (name, path) tuples, where `name` is the name of a registered project and `path` the path to its `egg-info` directory. But for the use case you are mentioning, what about an explicit API: get_owners(paths) -> sequence of project names returns a sequence of tuple. For each path in the "paths" list, a tuple of project names is returned > > 5. If any cache mechanisms are to be used by the API, the API *must* make it > possible to bypass or explicitly manage that cache, as otherwise > installation tools and tools that manipulate sys.path at runtime may end up > using incorrect data. work in progress - (I am afraid I have to write an advanced prototype to be able to know exaclty how the cache might work, and so, what API we should have) > > 6. get_files() doesn't document whether the yielded paths are absolute or > relative, local or cross-platform, etc. I am fixing this as well >> I need to find back your comments for this part, I must have missed >> them. That's >> the last part I didn't work out yet on the current PEP revision. > > Well, if you can't find them, the EggFormats doc explains how these file/dir > structures are currently laid out by setuptools, easy_install, pip, etc., > and the PEP should probably reference that. work in progress Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
On Tue, May 19, 2009 at 4:03 PM, Antoine Pitrou wrote: > Ronald Oussoren mac.com> writes: >> >> Wouldn't it be possible to use a CSV file for this? That way we >> wouldn't have to invent yet another escaping mechanism and there's >> already good suppport for reading and writing CSV files in the >> standard library. > > +1 > > We can even customize the delimiter if you want to make it more readable (or > if > there's a shortage of bikeshed material ;-)). +1 and the default csv delimiter "," makes it perfectly readable ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
On Tue, May 19, 2009 at 10:36 PM, P.J. Eby wrote: > > Now, about the APIs... > > I think it would be simpler to have explicit object types representing > things like a directory, a collection of directories, and individual > projects, and these object types should be part of the API. > > Any function-oriented API should just be exposed as the methods of a default > singleton. Other Python modules follow this pattern -- and it's what I > copied for the pkg_resources design. It gives a nice tradeoff between > keeping the simple things simple, and complex things possible, as well as > keeping mechanism and policy separate. > > Right now, the API design you're trying to do is being burdened by using > strings and tuples to represent things that could just as easily be objects > with their own methods, instead of things you have to pass back into other > APIs. This also makes caching more complex, because you can't just have one > main object with stuff hanging off; you've got to have a bunch of > dictionaries, tuples, lists, sets, etc. I don't know how other people work on building APIs in PEPs, but at this stage I am unable to work them on the paper, without having a prototype to try things out. So I guess I'll start this prototype in bitbucket and come back with it for feedback in Distutils-SIG, for a new PEP 376 round. Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] distutils.build_ext path comparison - python 2.5.2
Hi Sven can you add an issue with your patch in http://bugs.python.org/ Thanks in advance Tarek On Wed, May 20, 2009 at 2:31 PM, Sven Schrader wrote: > Hi, > > since our python installation is located on a symlink'ed directory, > our variables "sys.exec_prefix" and "sys.executable" can have different > paths. Therefore, the respective test in build_ext.py fails (line 202) > and a wrong > library directory is obtained. > > To fix this issue, I have attached a patch that uses "os.path.samefile" > instead, > to see whether two files are identical irrespective of its path. > > > Greetings > > Sven Schrader > > > > ps: please CC answers to me, I'm not on the list :-) > pps: I hope the attachment isn't inline... > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 : Changing the .egg-info structure
On Wed, May 20, 2009 at 11:48 AM, Tarek Ziadé wrote: > So I guess I'll start this prototype in bitbucket and come back with it for > feedback > in Distutils-SIG, for a new PEP 376 round. Ok so FYI, I moved the discussion here: http://mail.python.org/pipermail/distutils-sig/2009-May/011933.html Regards Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [buildbot] some build slaves in bad shape
Hello, I've noticed some problems since this morning with the trunk and 3.x stable buildbots: - x86 XP-4 (trunk and 3x) is throwing an "no space left on device" error when it compiles the sqlite module in its temp dir - amd64 gentoo 3.x and ia64 Ubuntu 3.x buildbot versions seem to be too old to run, they should be upgraded - ppc Debian unstable trunk keeps on failing to connect to svn.python.org Regards Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [buildbot] some build slaves in bad shape
On Fri, May 29, 2009 at 12:39 AM, David Bolen wrote: > David Bolen writes: > >> Ooops, that's mine. Geez - it's a VM, but has a 10GB C: drive, and >> the actual build slave has its working directory on a separate virtual >> drive. Wonder what the heck has filled up the system drive. I'm >> working on it now though. > > Well, looks like it was 5+GB of temporary files of some sort. It's > cleaned up now and back online. Thanks that's great ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 376
Hello, We have polished out PEP 376 and its code prototype at Distutils-SIG. It seems to fullfill now all the requirements, so I am mailing it here again, for a new round of feedback, if needed. - the pep : http://svn.python.org/projects/peps/trunk/pep-0376.txt - the code prototype : http://bitbucket.org/tarek/pep376/src/tip/pkgutil.py Notice that if the PEP is accepted at this point, I will : - focus on making the code work as fast as possible, for directories browsing - work on the backport and the required patches for setuptools and pip at the same time, and see if I can get some beta-testers that are willing to switch to this new version to test it extensively before 2.7/3.2 are out. Regards Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Mon, Jun 22, 2009 at 4:59 PM, Antoine Pitrou wrote: > - the **MD5** hash of the file, encoded in hex. Notice that `pyc` and > `pyo` >generated files will not have a hash. > > Why the exception for pyc and pyo files? As in PEP 262, since they are produced automatically from a py file, checking the py file is enough to decide if the file has changed. > > - `zlib` and `zlib-2.5.2.egg-info` are located in `site-packages` so the > file >paths are relative to it. > > Is it a general rule? That is, the paths in RECORD are always relative to > its > grandparent directory? no, they can be located anywhere on the system. But when the paths are located in the same directory where the .egg-info directory is located, a relative path is used. (see the section before this example) I'll add an example that contains files located elswhere in the system (like a script and a data file) > > > The RECORD format > - > > The `RECORD` file is a CSV file, composed of records, one line per > installed file. The ``csv`` module is used to read the file, with > the `excel` dialect, which uses these options to read the file: > > - field delimiter : `,` > - quoting char : `"`. > - line terminator : `\r\n` > > Wouldn't it be better to use the native line terminator on the current > platform? (someone might want to edit or at least view the file) Good idea, I'll change that, > > > What is the character encoding for non-ASCII filenames? UTF-8? > Yes, there's a constant in Distutils, called PKG_INFO_ENCODING that will be used for the file generation. > > Are the RECORD file's contents somehow part of the DistributionMetadata? The DistributionMetadata correspond to the fields defined in PEP 345, e.g. written in the PKG-INFO file, which is mentioned in the RECORD file. We are reworking PEP 345 as well, to add some fields. What did you have in mind ? > > > - ``DistributionDirectories``: manages ``EggInfoDirectory`` instances. > > What is an EggInfoDirectory ? Typo (old name), fixing this.. > > > A plural class name looks strange (I think it's the first time I see one in > the CPython codebase). How about another name? (DistributionPool, > DistributionMap, WorkingSet etc.). Sure, WorkingSet is nice, it's the name used in setuptools, > > > - ``get_egginfo_file(path, binary=False)`` -> file object > > Returns a file located under the `.egg-info` directory. > > Returns a ``file`` instance for the file pointed by ``path``. > > Is it always a file instance or just a file-like object? (zipped > distributions > come to mind). Is it opened read-only? It's in read-only mode, either "r" either "rb" and in case of a zip file, it returns a file-like object using zipfile.ZipFile.open. > > - ``owner(path)`` -> ``Distribution`` instance or None > >If ``path`` is used by only one ``Distribution`` instance, returns it. >Otherwise returns None. > > This is a bit confusing. If it returns None, it doesn't distinguish between > the > case where several Distributions refer to the path, and the case where no > distributions refer to it, does it? > The idea of this API is to find out of a distribution "owns" a file, e.g. is the only distribution that uses it, so it can be safely removed. > Is there any reason to have this method while file_users(path) already > exists? > Its just a helper for uninstallers, but file_users() could probably be enough, I can remove "owns" if people find it confusing, > A new class called ``DistributionDirectories`` is created. It's a > collection of > ``DistributionDirectory`` and ``ZippedDistributionDirectory`` instances. > The constructor takes one optional argument ``use_cache`` set to ``True`` > by > default. > > You forgot to describe the constructor's signature and what it does > exactly. > I'll add that, > > ``EggInfoDirectories`` also provides the following methods besides the > ones > from ``dict``:: > > What is EggInfoDirectories? This is a DistributionDirectories, one more instance I forgot to rename, I'll fix that > > > - ``append(path)`` > >Creates an ``DistributionDirectory`` (or > ``ZippedDistributionDirectory``) >instance for ``path`` and adds it in the mapping. > > - ``load(paths)`` > >Creates and adds ``DistributionDirectory`` (or >``ZippedDistributionDirectory``) instances corresponding to ``paths``. > > Why are these methods named completely differently although they do almost > the > same thing? Besides, append() makes it look like ordering matters. Does it? > (for a naming suggestion, e.g. load(path) and load_all(paths). Or, > even simpler, load(*paths) or load(paths)) > Right, I'll fix that, > > - ``get_file_users(path)`` -> Iterator of ``Distribution`` (or >``ZippedDistribution``) instances. > > This method is named file_users in another class. Perhaps the naming should > be > consistent? Right, it used to be get_*, that's a typo. I'll fix it, > > > All these function
Re: [Python-Dev] PEP 376
2009/6/22 P.J. Eby : > At 05:42 PM 6/22/2009 +0200, Tarek Ziadé wrote: >> >> Wouldn't it be better to use the native line terminator on the current >> platform? (someone might want to edit or at least view the file) >> >> >> Good idea, I'll change that, >> > > As long as the file is always *read* with "U" mode, so that you can't mess > it up, especially if the install is to a directory shared between platforms. Adding that too, thanks; > > >> The idea of this API is to find out of a distribution "owns" a file, e.g. >> is the only distribution >> that uses it, so it can be safely removed. > > This could equally well be done by ``owners(path)``, returning a sequence of > zero or more items. Any length <> 1 means the file can't be safely removed. > Meanwhile, having the data about all the owners of a file would also be > useful for tools that just want to inspect a directory's contents, for > example, or to detect conflicts and overwrites. that's basically what "get_file_users" does except it's an iterator, roughly that would be : def get_file_owners(path): return list(get_file_users(paths)) So I am wondering if it worth having it -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Tue, Jun 23, 2009 at 3:41 AM, Kevin Teague wrote: >> >> >> A plural class name looks strange (I think it's the first time I see one >> in >> the CPython codebase). How about another name? (DistributionPool, >> DistributionMap, WorkingSet etc.). >> >> Sure, WorkingSet is nice, it's the name used in setuptools, >> > > A WorkingSet and a DistributionDirectories (or whatever it gets named to) > are different things though, no? > > A WorkingSet is "a collection of active distributions", where each > distribution might come from different distribution directories: > > http://peak.telecommunity.com/DevCenter/PkgResources#workingset-objects > > Where as DistributionDirectories is a dictionary of locations where > distributions are installed. The WorkingSet may be comprised of > distributions from several different locations, and each location may > contain the same or different versions of the same distribution. DistributionDirectories can contain directories that are not located in the same parent directory, so I find it rather similar besides the "active" feature in Python doesn't exist (yet) In any case, maybe picking up a name that is not from setuptools will be less confusing for people that uses WorkingSet classes nowadays. What about using the same names used in Python's site module: "sitedir" is the name used for a directory we named DistributionDirectory. So what about : DistributionDirectory -> SiteDir DistributionDirectories -> SiteDirMap ++ Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Wed, Jun 24, 2009 at 11:18 AM, Antoine Pitrou wrote: > Sridhar Ratnakumar activestate.com> writes: >> >> On 09-06-23 02:57 AM, Nick Coghlan wrote: >> > Something like DistributionDirectoryMap should cover it. >> > >> > You could probably get away with shortening "Directory" to "Dir" in the >> > class names though: >> > >> > - Distribution >> > - ZippedDistribution >> > - DistributionDir >> > - ZippedDistributionDir >> > - DistributionDirMap >> >> 'Map' reminds me of an associated hash (or is it the actual map?). > > Good thing, because it is a dict subclass if you read the PEP :) > > +1 with Nick's proposal. > I've changed using these names. Any other feedback with this PEP or we're good to go ? :) Regards Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 376
Hello, If no one objects, I'd like to push PEP 376 in the "accepted" status and go ahead with its implementation, with continuous feedback at Distutils-SIG as we did to build it. The next PEPs that are being discussed at Distutils-SIG are : - the new version of PEP 345 for the inclusion of fields like "installed_requires" (dependencies) - PEP 386, that talks about distribution version numbers (because it's needed to describe dependencies) While they are still under heavy discussion, I have good hope that they will both make it for 2.7 and 3.2. Regards Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Tue, Jun 30, 2009 at 2:32 PM, Nick Coghlan wrote: > Martin v. Löwis wrote: >>> If no one objects, I'd like to push PEP 376 in the "accepted" status >>> and go ahead with its implementation, >>> with continuous feedback at Distutils-SIG as we did to build it. >> >> I think this isn't quite the process. In the past, every PEP required >> BDFL pronouncement, which you should now seek. > > Agreed. While Guido is highly likely to just accept the distutils-sig > consensus on something like this, that doesn't eliminate the need for > him to actually *say* that he is approving the PEP on that basis. Sure, I didn't want to bypass the process, I was not really sure about the right move on this PEP since it was based on the summit decisions, I'll wait for Guido decision, thanks. Cheers Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
2009/6/30 Guido van Rossum : > On Tue, Jun 30, 2009 at 5:32 AM, Nick Coghlan wrote: >> Martin v. Löwis wrote: >>>> If no one objects, I'd like to push PEP 376 in the "accepted" status >>>> and go ahead with its implementation, >>>> with continuous feedback at Distutils-SIG as we did to build it. >>> >>> I think this isn't quite the process. In the past, every PEP required >>> BDFL pronouncement, which you should now seek. >> >> Agreed. While Guido is highly likely to just accept the distutils-sig >> consensus on something like this, that doesn't eliminate the need for >> him to actually *say* that he is approving the PEP on that basis. > > Sure. :-) > > So what *is* the distutils-sig consensus? The consensus is to have one and only one way to install distributions in Python, and a way to retrieve information on installed distributions, and their files. > > And is there consensus outside of it? (Remember the ipaddr debacle. > It's easy for people to miss an important PEP.) I am not sure who you are thinking about, I am blogging a lot on the topic and I am trying to get key players involved in this, like os packagers etc. We've built this PEP in respect with existing tools like setuptools, etc, and I am sending mails at python-dev to make sure evereyone involved in python development has a chance to provide some feedback, > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore wrote: > - The terminology and focus feels setuptools-inspired (my apologies if > that's not the case). Expect pushback from setuptools haters... setuptools implemented *needed* features, like a way for developers to browse installed packages. We said during the summit at Pycon that we wanted this feature in Distutils, (Guido said so) So I worked in PEP 376 to introduce them. Now if the fact that we want to introduce the good ideas of setuptools into distutils, (problems Phillip resolved) will make people push it back *even* they are good idead, needed features, is something we need to fight against. > - It's quite dense for the casual reader not familiar with the > terminology. I've never managed to read the whole thing through, > personally. I'll try to add a definitions section, > > I'd suggest two things: > - Add a section to the PEP describing the purely end user impact of the > changes > - Post that to python-list, with a note pointing to the PEP for people > who care about distutils details > Ok. > If that gets no feedback, you've done as much as you can. I hope it'll make it one day. If not, I don't understand the goal of the "Python Language Summit' > > Paul. > > [1] I'd actually like it if the PEP defined an uninstall command - > something like "python -m distutils.uninstall packagename". It can be > as minimalist as you like, but I'd like to see it present. it's already there: http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function Regards Tarek -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
2009/6/30 P.J. Eby : > At 07:57 PM 6/29/2009 +0200, Tarek Ziadé wrote: >> >> Hello, >> >> If no one objects, I'd like to push PEP 376 in the "accepted" status >> and go ahead with its implementation, >> with continuous feedback at Distutils-SIG as we did to build it. > > I do have a question about the current draft... Do zipped distributions use > EGG-INFO or a project-version.egg-info? This isn't spelled out in the PEP, > although I get the general idea that the EGG-INFO format isn't supported, > and thus the PEP and API do not support existing .egg files. This should > probably be made clear, if that's the intention. The intention is to have only one standard for the egg-info, I have explained it in this section: http://www.python.org/dev/peps/pep-0376/#egg-info-becomes-a-directory previous formats will not be supported but that won't break anything of course since the new APIs will work only over the distribution installed with the new version of distutils. > > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano wrote: > On Wed, 1 Jul 2009 05:19:07 am Tarek Ziadé wrote: >> 2009/6/30 Guido van Rossum : > ... >> > So what *is* the distutils-sig consensus? >> >> The consensus is to have one and only one way to install >> distributions in Python, > > "One and only one way"? Shouldn't that be "one obvious way"? > > There is a big difference -- the first implies that Python should > actively prohibit (somehow!) or at least inhibit any other install > mechanisms, while the second implies that there should be a preferred, > obvious, but not compulsory, mechanism. > > If you say "one and only one", and I take you at your word, then I can > only assume that you want to actively prevent me from manually > dropping .py files into my PYTHONPATH and having them work. I would > guess that's probably not what you mean, but that's what it sounds like > you're proposing. I don't like guessing -- would you please clarify > what you mean? one and only one way to install a distribution using its setup.py script, among the ways provided by easy_install, distutils and pip. e.g. avoiding having different format/location for the generated .egg.info; > > > Some questions and comments regarding the PEP: > > Rationale: > http://www.python.org/dev/peps/pep-0376/#id13 > > "There are too many ways to do it." > > Why is having multiple ways of adding distributions a problem to be > solved? On my Linux system, I can add packages with rpm/yum, or I can > compile them from source and manage them myself. I possibly even could > (but never have!) install apt-get and use it to manage packages. If you choose not to manually manage them, the packaging system you use is installing files in standard places where you expect to find them, and provides tools to know what is installed. Right now we don't have this standard in Python stdlib to interact with the distributions in a Python installation. And since packaging systems that are used out there could use the same standard, since we would like to make distutils a reference package for this kind of need, it seems like a good idea to define this standard in here. then each packaging system is free to implement its features on the top of it, ala wsgiref. > > Another rationale which should be added: > > There is no standard way of uninstalling distributions. > > > How distributions are installed: > http://www.python.org/dev/peps/pep-0376/#id14 > > "The problem is that many people use easy_install (from the setuptools > project [4]) or pip [5] to install their packages, and these > third-party tools do not install packages in the same way that > Distutils does" > > Why is that a problem to be solved? Being able to retrieve the metadata of an installed distribution, or the list of its installed files, no matter which tool installed it. > > > Uninstall information: > http://www.python.org/dev/peps/pep-0376/#id15 > > "Under some circumstances, you might not be able to know for sure that > you have removed everything, or that you didn't break another > distribution by removing a file that is shared among several > distributions." > > I don't see how this proposal will help in the second case. If you > install distribution Spam, containing file spam.py, and then install > distribution Ham, which requires spam.py, what is to prevent you from > removing Spam and breaking Ham? > > If you don't propose a solution for the dependency problem, you should > say so. This problem is solved as described later in the PEP, with the API that allows you to get the list of the distributions that use a given file. (thanks to the RECORD files) If Spam and Ham use smap.py, and if you uninstall Spam, this file will not be removed because the API will tell you its used in both distributions. > > > The RECORD format: > http://www.python.org/dev/peps/pep-0376/#id19 > > "Each record is composed of three elements. > ... > the MD5 hash of the file, encoded in hex. Notice that pyc and pyo > generated files will not have a hash because they are automatically > produced from py files." > > What if your distribution is not a source distribution and only provides > pyc and pyo files? > Good question, I have never created such distribution. Aren't they read-only files ? What happens if someone change them, do we really want to keep them since it breaks the distribution ? How they can be changed in the first place, without the source ? Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Tue, Jun 30, 2009 at 10:06 PM, Scott David Daniels wrote: > Tarek Ziadé wrote: >> >> On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore wrote: >>> >>> [1] I'd actually like it if the PEP defined an uninstall command - >>> something like "python -m distutils.uninstall packagename". It can be >>> as minimalist as you like, but I'd like to see it present. >> >> it's already there: >> >> http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function > > That (at least as I read it) is a function, not a command. > If it is a command, give an example of its use from the command line > for us poor "don't want to research" people. If the following works: > > $ python setup.py uninstall some_package > > Then explicitly say so for us poor schlubs. > Right, I'll add that. Although it will be a reference implementation only. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Tue, Jun 30, 2009 at 11:11 PM, Nick Coghlan wrote: >> >> >> previous formats will not be supported but that won't break anything >> of course since the new APIs will work only over the distribution >> installed with the new version of distutils. > > To address PJE's question in the PEP, it may be worth expanding on this > in the backwards compatibility section explaining how the new distutils > metadata system avoids getting confused by the old pre-standardisation > installation formats (e.g. it may be that the directory names and/or > filenames all deliberately differ from current approaches precisely so > they can coexist without interfering with each other) > I'll work on making it clearer, > Also, I find the following paragraph (near the start of the section > linked above) confusing: > > [PEP 376] >> Notice that this change is based on the standard proposed by >> EggFormats, although this standard proposes two ways to install >> files: >> >> * A self-contained directory that can be zipped or left unzipped and >> contains the distribution files and the .egg-info directory. >> * A distinct .egg-info directory located in the site-packages directory. > > This is unclear as to what "this standard" is referring to (since the > PEP itself is proposing a standard, but the sentence is also referring > to the existing EggFormats de facto standard). If the PEP only supports > the latter of the two options (which appears to be the case) it should > say so explicitly. ok > > Another minor nit from the same section: > > [PEP 376] >> Any '-' characters are currently replaced with '_'. > > This should say something like "Any '-' characters other than the one in > 'egg-info' and the one separating the name from the version number are > included in the replacement of non-alphanumeric characters with '_'" ok > > Other questions/comments: > > - What is a "local absolute path"? Absolute path I understand, relative > path I understand, but "local absolute" is a novel term to me. local means that the "/" separator that is used in the RECORD file for example, no matter what platform you are on, is translated using the local separator (/ or \) I'll make it clearer, Regards Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Tue, Jun 30, 2009 at 10:11 PM, Paul Moore wrote: >> We said during the summit at Pycon that we wanted this feature in >> Distutils, (Guido said so) > > "We" in this context denotes the people at the summit. Please remember > that people who weren't there still have an opinion - and it may well > differ. I'm not saying that it *does* differ, just that the view of > the summit should not be viewed as conclusive (unless the way Python > is developed is very different from how I understand it). Sorry I didn't mean it this way. The summit was just the kickoff for this PEP, but with the strong desire to have a standard + some API for it. Also, notice that I put in "We" also the people that answered the survey I did before the summit. (http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first-results/) I'll make other fixes in the PEP with the feedback you gave, thanks Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
2009/7/1 P.J. Eby : > At 01:34 AM 7/1/2009 +0200, Tarek Ziadé wrote: >> >> On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano >> wrote: >> > I don't see how this proposal will help in the second case. If you >> > install distribution Spam, containing file spam.py, and then install >> > distribution Ham, which requires spam.py, what is to prevent you from >> > removing Spam and breaking Ham? >> > >> > If you don't propose a solution for the dependency problem, you should >> > say so. >> >> This problem is solved as described later in the PEP, with the API >> that allows you to get the >> list of the distributions that use a given file. (thanks to the RECORD >> files) >> >> If Spam and Ham use smap.py, and if you uninstall Spam, this file will >> not be removed >> because the API will tell you its used in both distributions. > > That's not the scenario he's talking about. He's talking about the case > where Ham has an 'install_requires' of Spam. That is, a runtime dependency, > not a shared file. Ah, right sorry I misunderstood... They are no plans to handle dependency installation / uninstallation / managment at distutils level. so if you remove Ham, it will not check what distributions use it. So yes, I'll add a note on this, That said, the APIs will be powerfull enough for a third-party package managers to handle this case by throwing for example a warning or an exception. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Wed, Jul 1, 2009 at 2:27 PM, Nick Coghlan wrote: > However, having uninstall as a command supported by setup.py also makes > a certain amount of sense. > Yes, that could be an alias that just calls the uninstall global function using the distribution name. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Wed, Jul 1, 2009 at 10:35 AM, Paul Moore wrote: > 2009/7/1 Tarek Ziadé : >>> That (at least as I read it) is a function, not a command. >>> If it is a command, give an example of its use from the command line >>> for us poor "don't want to research" people. If the following works: >>> >>> $ python setup.py uninstall some_package >>> >>> Then explicitly say so for us poor schlubs. >>> >> >> Right, I'll add that. Although it will be a reference implementation only. > > -1. Where does the setup.py file come from? If I have docutils > installed, and want to remove it, must I download the source again so > that I can get the setup.py, so I can run the uninstall? This is daft > - particularly given that the point of PEP 376 is to ensure that all > of the required information is available from the installed package! > > As I suggested before: > > python -m distutils.uninstall packagename yes sorry if it was unclear, I was not thinking about adding something based on setup.py, but just saying that I was going to add this feature in the PEP. and it will be of the form: python -m distutils.uninstall packagename > > Calling it a "reference implementation" should not imply that it's not > built to be usable and complete. If it's there,m people should be able > to use it. It will be usable and complete, but very limited. As someone mentioned, it will not take care of dependencies and prevent you from removing a distribution that is mentioned in another distrubution in a setuptools' install_requires metadata. That said, when PEP 345 evolves like we have planned to (adding install_requires in the metadata) Then we should be able to provide this kind of warning with no pain. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
Right, the API part is almost empty, http://docs.python.org/distutils/apiref.html I'll complete it for the relevant part. On Wed, Jul 1, 2009 at 2:37 PM, Paul Moore wrote: > 2009/7/1 Paul Moore : >> One other thought. You haven't documented the DistributionMetaData class. The > > Just noticed that it's defined in distutils. But it's not documented there :-( > > Maybe just add a bit to the PEP saying that the class exists in > distutils, give a brief summary, and say that as part of implimenting > the PEP it will be formally documented. (Even though distutils is a > mess of undocumented functionality, I don't think that you can base > new work on undocumented internals - you should document them first, > or things will never get any better :-() > > Paul. > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore wrote: > "The problem is that many people use easy_install or pip to install > their packages..." > > As already noted by others, it's not clear why this is a problem. But > worse is the fact that the paragraph reads to me as saying that > easy_install/pip/etc don't follow the *current* standard structure. > Given this fact (which *is* a problem!) I fail to see why I should > expect them to follow any *new* standard - so how does this PEP > actually address the issue? Rather, it seems to say to me that the > existing tools have a history of ignoring the standard approach, so > this PEP is going to be useless :-( > > [I expect that isn't what you're trying to say, so you may just need > to clarify your meaning. But I do think it's important to address the > question of how this PEP is going to ensure that existing tools adapt. > In particular, setuptools seems to have completely stagnated, so I see > very little likelihood that setuptools is going to change to conform > to the PEP - how does that affect things?] Right it's unclear, I'll work on this part. To resume : - Phase 1 : introduction of the egg-info file in distutils Philipp introduced the creation of a file named xxx.egg-info file in 2006 (see http://bugs.python.org/issue1459476) alongside distutils-installed package, that contains the metadata of the distribution. - Phase 2: two new formats in the setuptools project Then he created two new formats in the setuptools project: 1. an .egg-info directory containing a PKG-INFO file, which is similar to the previous .egg-info file. This happened because other files were added in that directory. 2. an .egg directory, possibly zipped, that is a self-contained version of the distribution, containing the metadata and all other files. Setuptools uses 2. by default. There's an option when you use setuptools to force 1. (–single-version-externally-managed *or* --root). - Phase 3: adopting a unified standard. It occurs that having a .egg-info directory (1.) is way better than a single file because it's a place-holder for other files. It is also adopted by the community when it comes to install setuptools-based package.: - This is what "pip" uses to install packages in a more flat manner. - It's also the policy under debian (http://wiki.debian.org/DebianPython/NewPolicy) - and under Fedora (http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools) The .egg directory (2.) is more controversial because it a self-contained directory that doesn't install packages so it makes two different ways to work with distributions for packagers. So I have proposed in the PEP to adopt the standalone .egg-info directory as the new distutils unified standard. This means that all the third-party tools out there already conform to that standard, and that packages installed in other formats will not benefit from the new APIs. which means that people that want to work with distributions installed as .egg directories will have to use setuptools APIs. Which makes sense. > > ".egg-info becomes a directory" > > Don't refer to the setuptools documentation (EggFormats)! It is > obscure and user-hostile, as well as not actually being the same as > the PEP's proposal. Rather, just document the structure of .egg-info. > If you want, you can then add a cross-reference note, saying something > like "The setuptools structure, as proposed in the EggFormats > documentation for that package [ref], is a subset of this standard. In > order to conform to this PEP, setuptools will have to be amended to > only install .egg-info directories in the format defined by this PEP". I'll work that way. > > "However, it will impact the setuptools and pip projects, but given > the fact that..." > > Confusing. Will these tools need to change (I believe so) or not? If > they will need to change, that hardly counts as "no deep consequences" > - there's the whole backward compatibility issue for them to handle. I'll add this in a backward compatibility section, as suggested earlier by someone. > > "Adding a RECORD file" > > "The RECORD format" > > "Adding an INSTALLER file in the .egg-info directory" These section will have more details about the way they interact with bdist_* for the rest of the mail, I'll clarify the rest of the PEP using your feedback; > "New APIs in pkgutil" > > You say "the best place to put these APIS seems to be pkgutil". You > should be more definite - "these APIs will be added to the pkgutil > module". ok > General rule - don't document (and commit yourself to) any internal > details that the user can't access from the public API. It just makes > backward compatibility harder to maintain. The purpose is to provide this documentation to third-party projects that want to implement a packaging system on the top of these classes. Maybe this should be removed from the PEP and but into another document targeted to developers ? > > "Adding an unins
Re: [Python-Dev] PEP 376
On Wed, Jul 1, 2009 at 10:20 AM, Michael Foord wrote: >> >> Uninstall as a command feels a little weird. Since "python setup.py >> [some-command]" implies that the setup.py contains information about the >> distribution that the command is being applied to. So instead of: >> >> $ python setup.py uninstall some_package > > It could be: > > $ python -m distutils uninstall some_package > > Asymmetrical with the install of course. > Yes exactly, I was going to add: $ python -m distutils.uninstall some_package The whole point of the RECORD file is to be able to uninstall without depending on the original archive used to install ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Thu, Jul 2, 2009 at 4:41 AM, P.J. Eby wrote: > Yes and no. Not providing uninstall support is reasonable, but the PEP also > has features to query packages in general. > > (There's also no technical reason why comparable manifest and uninstall > support can't be provided for .egg files and directoriees, since they > already have an implicit manifest: their contents. However, since I'm not > currently possessed of the time to provide a patch myself, I'm not going to > lobby for this as a feature.) > Yes that's not technical : I am in favor of keeping a single format. I don't see but confusion in keeping several formats, ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull wrote: > That's a judgment you must make. However, Paul's opinion seems to be > that it is internal, and not needed by third-parties who are working > "on the top" of these classes. If upon consideration you agree, you > should take those "details" out of the PEP proper. If you disagree, > you should promote them to the "official"/public API. > > The point of a PEP is not to construct a duck; it is to explain what > "quack" means. Yes, while the APIs I have written in the prototype+PEP helped us claryfing what we wanted, I agree they would be better in a second document. They are two target audience, the users of distutils and the builders of package managers, so removing this details from the PEP will also make it easier to read for the first crowd. > > Another general principle: even in the draft PEP, say "is", not "will > be". Ok I'll fix that. That's a French stuff : in french, "will be" isn't speculative at all. Thx ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Thu, Jul 2, 2009 at 1:32 PM, Nick Coghlan wrote: >> >> Yes exactly, I was going to add: >> >> $ python -m distutils.uninstall some_package > > A directly executable submodule is an even better idea than making > distutils itself executable. Definitely worth mentioning in the PEP in > the section on uninstallation support. > The PEP is updated for this, and takes into account several feedback from this thread, (like removing implementation details) I am working now on describing the behavior with the bdist_* commands ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore wrote: > "Adding a RECORD file" > > You say "at installation time" - please clarify. Do you only mean > setup.py install? What about bdist_wininst and bdist_msi? What about > third party bdist style tools? How will they ensure they get a RECORD > file? > > "The RECORD format" > > The line separator shouldn't be os-dependent. What value is used for a > pure-python (ie, platform independent) package? Unless the file is > generated when the install is done, in which case see the previous > point... > > Absolute file paths - this implies that RECORD has to be generated by > the installer (which is the only place that knows absolute paths) > which means that every bdist_xxx installer has to write its own RECORD > file. Does the PEP offer no support for this? In any case, the > bdist_wininst and bdist_msi code (which is core distutils) will need > to be amended to manage RECORD files appropriately. Since all bdist_* commands are using the install command in a temporary build directory to create a binary distribution, the RECORD file (and the INSTALLER) will be pre-generated then installed alongside the packages/modules. For absolute paths now that gets installed, what would be the difference between the pre-generated RECORD file and the RECORD file installed on the win32 target system, if any ? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Thu, Jul 2, 2009 at 2:39 PM, Paul Moore wrote:
> 2009/7/2 Tarek Ziadé :
>> For absolute paths now that gets installed, what would be the
>> difference between the pre-generated
>> RECORD file and the RECORD file installed on the win32 target system, if any
>> ?
>
> When is an absolute path generated? If you can give me a small sample
> of a distribution that installs a file in an absolute path, I'll do
> some testing on Windows.
Try this setup.py file:
"""
from distutils.core import setup
setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])])
"""
with this MANIFEST.in file:
""
include data
"""
and add a 'data' file alongside
If you install it, data is copied in /tmp. If you create a bdist
distribution it will be created in the root of the dumb directory
which is used to generate the binary distro.
Of course you'll have to change '/tmp' to 'c:\tmp'
(which makes me realizes that there's no way to force the installation
data_files in another drive under windows:
the binary distribution will be the same, no matter what drive you
use in the absolute path used in data_files
I'll add an issue on this)
>
> But the immediate answer is that there are *no* reliable "absolute"
> paths on Windows, so you're not looking at the likes of /usr/doc, but
> rather paths that are relative to sys.prefix, but not to the package
> directory. In that case, the key point is that if the installer is
> built on a system where Python is installed in a different directory
> than the system where the installer is run, paths need to be
> relocated. (E.g., C:\Python27 vs D:\Apps\Python27).
I get the point: they are three levels we should handle in the RECORD file
1. absolute paths
2. paths relative to sys.prefix or sys.exec_prefix
3. paths relative to the directory where the .egg-info directory is located
the RECORD file doesn't handle 2. indeed. For instance you can add a script:
setup(
..
scripts=['myscript.py']
..)
that will get installed in : sys.prefix + 'Scripts/myscript.py' under
win32 for instance
So a possible solution is to add 2. in the RECORD files by using a
notation such as
"$PREFIX" "$EXEC_PREFIX" when we detect that the file is under on of
these paths.
The query functions will then be able to use sys.prefix and
sys.exec_prefix for recompose
the absolute pat on the target system
>
> Paul.
>
--
Tarek Ziadé | http://ziade.org
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Thu, Jul 2, 2009 at 4:35 PM, Paul Moore wrote: > Ta. I'll give it a go tonight. But haven't you made the point yourself > by saying I'll need to change the directory? This will fail for me as > I don't have a "/tmp" directory. So I'd expect a bdist installer > (*any* bdist installer) to fail, as it's got no way to tell where to > put that file. I can only imagine this type of thing being done by > someone packaging for Unix, with no interest in cross-platform > portability (I suspect MAC OS directory names differ from Unix as > well). It will fail if your path starts with "/" and you are under win32 because there's a converter that will raise an error in that case at installation time. But you can use some code in your setup.py to provide the right directory name when install is called. I don't see it as a problem. I'm just afraid it's impossible to use efficiently under win32 because of the drive letter. But that's rather a bug. > > So maybe absolute filenames should be banned in distutils? (OK, less > radically, maybe someone should ask where that functionality is used > in practice, and based on the answers come up with a more portable > approach). > I'll try to see if I can collect that info out of PyPI with a script, to list the real-world usages. Maybe Martin you have a simple way to do this on PyPI ? Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
2009/7/3 Paul Moore : > This is a good point. Distutils only installs files in the filesystem > - it has no facilities for installing packages based on any other sort > of PEP 302 based importers. Hence, PEP 376 in principle should only > relate to filesystem-based distributions. But it also mentions > zipfile-based distributions: "Notice that the API is organized in five > classes that work with directories and Zip files (so it works with > files included in Zip files, see PEP 273 for more details [8])." > > This is wrong. The PEP should either (1) restrict itself to filesystem > based implementations (leaving the problem of other PEP 302 loaders to > systems that manage these) or (2) be defined in a sufficiently general > way that it can be implemented for *any* PEP 302 based loader - which > probably means extending the PEP 302 protocols - and supplying zipfile > functions as an example of how this is used. > > I believe that (1) is unlikely to be sufficient for real world use. > Zip files (eggs, py2exe embedded modules, etc) are far too important a > real world use case to ignore. The problem with (2) is that it > requires significant extra work. But special-casing zip files (as the > present implementation appears to do) will break as soon as any other > PEP 302 compliant format becomes popular. >> Moreover, the proposed ``egginfo_dirname()`` routine is a step-back from the >> ``pkg_resources`` approach where we don't enforce resources to reside on a >> traditional filesystem. > > On the other hand, pkgutil.get_data is the standard library means of > reading resources from a package. This is PEP 302 compliant now. This > new PEP doesn't affect that. > Right. While it would be feasible to make pgutil works with PEP 302 loaders, we would still need to define in a generic way the content of the RECORD files. Right now it works for directory and zipped files since it's expressed with '/' separated paths. And if I understand PEP 302 right, any backend would be able to handle these paths no matter how they are stored, as long as the implement APIs like get_data() > What PEP 302 doesn't provide is package management. But Python itself > doesn't provide package management, except in the form of distutils > setup.py install - which is solely filesystem based. > > Maybe there's a case for extending PEP 302 and distutils to allow > integrated management of other forms of importer format, but that's a > huge other project, which no-one to my knowledge is even looking at. Sounds like a fully-featured packaging managment system, which is imho, out of scope. And I don't see PEP 376 making it impossible for someone to build such a packaging system on the top of distutils. I've started one myself for the sake of experimentation, with built-in multiversion support, for a full replacement of site-packages. > Eggs are fundamentally a PEP 302 zip file format. There are some extra > bits of metadata for setuptools/easy_install in there (as I understand > things) but essentially they are zip files. When you say "decoupling > the egg format", I assume you mean "decoupling the egg metadata" - > which is fine, but to properly decouple, you need API level access to > the metadata. PEP 376 offers read-only access, but as you rightly > point out, it is only for filesystem data (and some form of zip file, > which appears to be limited in some way, as it isn't PEP 302 based, > and the actual format isn't defined anywhere). And also PEP 376 goal is to define a single standard location of egg-info files for filesystem data. The zip form was built so it could work with zipped site-packages directories, like what the py2app project uses. > > The basic point here is that PEP 376 needs to define precisely how > pkgutil.get_distributions() scans sys.path looking for ".egg-info > directories". What does it do for sys.path entries that don't > correspond to filesystem directories? (Note - these may or may not be > zip files. Even if they are zip files, an earlier entry on > sys.path_hooks could have taken precedence. At the very least, you > should only process path entries as zip files if their importer - in > sys.path_importer_cache or via an explicit path hook scan - is a > zipimporter object.). I'll add more details on that part. right now it visits directories and zip files. Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Fri, Jul 3, 2009 at 4:28 PM, Nick Coghlan wrote: > I will note (and I believe this is also the main point that Lukasz was > making) that having the distribution metadata outside the distribution > as currently proposed in PEP 376 is going to make any eventual PEP 302 > integration much harder - 302 importers only provide a mechanism for > accessing files inside the distribution, not "adjacent" to them, so the > mechanism in the PEP doesn't generalise properly. > > I suspect this limitation of the PEP 302 APIs is the origin of the > setuptools format that embeds the metadata inside the distribution - it > lets you get at the metadata without having to assume that it exists > directly on the filesystem anywhere. Maybe we could rework the pkgutil classes for PEP 376 so they look like an implementation of the PEP 302 protocol for directories and zip files, with an extra "get_metadata()" API and state that it could be an extension for PEP 302 later. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/3 Paul Moore : > Does this sound sensible? Tarek, would you be OK with me attempting to > modify your prototype to support this protocol? Are there any tests > for PEP 376, so that I can confirm I haven't completely broken > something? If I can, I'll knock up some simple prototype importers for > non-standard examples, and see how they work with all this. Yes that's exactly what I was thinking about after the discussion we had in the other thread. This change would allow much more flexibility. Pleas go ahead, it's located here : http://bitbucket.org/tarek/pep376/ You can give me a bitbucket account so I can give you write access to the repo, There are tests as long as you install Nose. Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376
On Fri, Jul 3, 2009 at 7:02 PM, P.J. Eby wrote: > At 02:54 PM 7/3/2009 +0100, Paul Moore wrote: >> >> Eggs are fundamentally a PEP 302 zip file format. There are some extra >> bits of metadata for setuptools/easy_install in there (as I understand >> things) but essentially they are zip files. When you say "decoupling the egg >> format", I assume you mean "decoupling the egg metadata" - which is fine, >> but to properly decouple, you need API level access to the metadata. PEP 376 >> offers read-only access, but as you rightly point out, it is only for >> filesystem data (and some form of zip file, which appears to be limited in >> some way, as it isn't PEP 302 based, and the actual format isn't defined >> anywhere). The basic point here is that PEP 376 needs to define precisely >> how pkgutil.get_distributions() scans sys.path looking for ".egg-info >> directories". What does it do for sys.path entries that don't correspond to >> filesystem directories? (Note - these may or may not be zip files. Even if >> they are zip files, an earlier entry on sys.path_hooks could have taken >> precedence. At the very least, you should only process path entries as zip >> files if their importer - in sys.path_importer_cache or via an explicit path >> hook scan - is a zipimporter object.). To be honest, this is a major can of >> worms. But if PEP 376 is not going to support PEP 302, then it must state >> that fact explicitly, to avoid giving people false expectations - >> particularly with Brett's importlib in Python 3.1, which will make it far >> easier for people to experiment with new packaging formats such as the ones >> Lukasz mentions above. And it MUST fail gracefully in the face of >> unsupported importer types. > > Well, we could always resurrect PEP 365, since pkg_resources already has > documented extensible support for arbitrary importers. That solves backward > *and* forward compatibility. Then PEP 376's uninstall facilities could be > implemented using pkg_resources' existing metadata query features. > > The primary downside to that, of course, is that it brings in the matter of > version specifications and dependencies... which appear to be a contentious > topic. (Note that Tarek is proposing to drop the PEP 386 proposal to > standardize a much more restrictive scheme than seutptools' version parser, > precisely because of the controversy.) I don't think we can add pkg_resources as-is because it does much more than providing the query features, which is beyond distutils scope. I think that part of the PEP 376 code at the end will be very similar to the subpart of pkg_resources that has the query features (as it is now with the current prototype) And what is nice with Paul's proposal on changing PEP 302 is that setuptools will be able to propose its EggFormats on the top of the pkgutils/distutils changes and make its pkg_resource code lighter. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/4 Paul Moore : > 2009/7/4 Paul Moore : >> 2009/7/3 Tarek Ziadé : >>> You can give me a bitbucket account so I can give you write access to the >>> repo, >>> There are tests as long as you install Nose. >> >> How do I get the tests to work? Just running nosetests gives an error >> (probably because pkgutil is being imported from the stdlib, rather >> than from this directory). >> I just run them from within the directory >> If I set PYTHONPATH=. then I get errors. I suspect path normalisation >> (for backslashes) in the zipfile handling. > > Actually, the test > > assert_equals(list(dist.get_egginfo_files(local=True)), > [os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/PKG_INFO'), > os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/RECORD')]) > > is broken, because the expected value uses slashes, which are *not* > the local separator on win32. > > I've attached a patch. Applied, thanks (I didn't run them under win32 yet) > > But there's 2 comments I'd make (one minor, one major) > > Minor one: The tests often seem to be exercising the internal classes, > not so much the public API, so many of them will probably not be of > much use to me :-( I'll add some more tests then, or even user stories. > I think you need some real-world use cases, with actual sample > (pseudo-)code, to validate the design here. As things stand, it's both > confusing and (I suspect) unusable in practice. Sorry, I know that > sounds negative, but if this isn't to be a source of subtle bugs for > years to come, it needs to be clarified now. PEP 302 is still hitting > this type of issue - runpy and importlib have brought out errors and > holes in the protocol quite recently - even though Just and I went to > great lengths to try to tease out hidden assumptions up front. Agreed, the zip case was added afterwards, but in practice, the APIs are still dealing with the files are *filesystem files* located in a container (eg a directory or a zip file) located somewhere on the filesystem. "local" in that case is a flag that means "translate a file path expressed in the local filesystem" which make no sense anymore with zip files. But the goal really, is to be able to point out that two distributions are using the very same file. Right now PEP 376 and the prototype code handle these two real world use cases: - browsing regular site-packages-like directories - browsing site-packages-like directories, that are zipped. For example: - I have a "packages.zip" file in /var/, wich is also in my sys.path. It contains a distribution "foo-1.0" that has the "roman.py" file in its root. So the RECORD file located in "foo-1.0.egg-info" has a line starting with "roman.py,..." - Then if I install docutils 0.5 as a regular filesystem distribution, "roman.py" will be added in Python's site-packages. and docutils-0.5.egg-info/RECORD will contain "roman.py,..." with the same hash. The local flag will return these paths: - /var/packages.zip/roman.py <--- not a "real" path - /usr/local/lib/python2.6/site-packages/roman.py So removing the docutils distribution will be doable, because these paths are different. > > Concrete proposal: > > get_metadata_files() - returns slash-separated names, relative to the > egginfo dir > get_metadata_file(path) - path must be slash-separated, relative to > the egginfo dir > > get_installed_files - returns the contents of RECORD unaltered > uses(path) - checks if path is in RECORD > > The latter 2 are not very useful in practice - you can't say anything > about entries in different RECORD files, which is likely the real use > case you want. Maybe RECORD could have an extra "Location" entry, > which determines where it exists globally (this would be the directory > to which the filenames were relative, in the case of filesystem-based > distributions) and RECORD entries are comparable if the Location > values in the 2 RECORD files match. That's a lot more complex - but > depending on what use people expect to make of these 2 APIs, it may be > justified. Yes, In practice, if you look at my previous example, even if "/var/packages.zip/roman.py" isn't a real path, it's enough to compare RECORD entries globally. The "Location" entry you are proposing in that case, would be "/var/packages.zip". But do we really need to store it the RECORD ? Or can't we define an API that returns two elements : - the path to the location (in the example: /var/packages.zip or /usr/local/lib/python2.6/site-packages) - the path within the
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
On Sat, Jul 4, 2009 at 3:04 PM, Paul Moore wrote: > 2009/7/3 Tarek Ziadé : >> 2009/7/3 Paul Moore : >>> Does this sound sensible? Tarek, would you be OK with me attempting to >>> modify your prototype to support this protocol? Are there any tests >>> for PEP 376, so that I can confirm I haven't completely broken >>> something? If I can, I'll knock up some simple prototype importers for >>> non-standard examples, and see how they work with all this. >> >> Yes that's exactly what I was thinking about after the discussion we >> had in the other thread. This change would allow much more flexibility. > > One important note - I plan on using the fact that DistributionDirMap > is not public, and hacking it about drastically, or possibly even > removing it. (After all, the likes of the load method don't make sense > when you've got sys.meta_path, sys.path_importer_cache and the like to > consider). > > If the loss of any of the "internal" classes is an issue, say so now! Go for it please, and let me know if you set a bitbucket account, so you can push your commits in there directly > > Paul. > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/4 Brett Cannon : >> >> P.S. +lots on using 'metadata' in the PEP 376 method names rather than >> the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly >> gratuitous in this case. > > Ditto from here. Plus I have an aversion to terminology that goes down the > reptile route instead of the Monty Python route. If it turns out that we use PEP 302-like loaders, I am also suggesting that the default metadata directory name used in Distutils is changed to "DIST_NAME.metadata". The loader would still work with "DIST_NAME.egg-info" directories for compatibility with existing format in the query APIs, but the Distutils install command would rather create "DIST_NAME.metadata" ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
On Sun, Jul 5, 2009 at 3:13 PM, Paul Moore wrote: > 2009/7/5 Tarek Ziadé : >> Go for it please, and let me know if you set a bitbucket account, so >> you can push your commits in there directly > > My bitbucket account is 'pmoore'. You're set. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
On Sun, Jul 5, 2009 at 3:27 PM, Paul Moore wrote: >> >> A concrete proposal would be to take back your proposal, but return >> tuples with the location as the first member. >> e.g. "(location, relative path[s])" > > That sounds reasonable. So we can forget the "local" parameter, and > return a tuple: > > - absolute location of the container (directory, zipfile or whatever > containing the egginfo file) as a filesystem path in canonical native > form (where it's filesystem based) or as an opaque token for the odd > cases (frozen modules, for example) where a filesystem location isn't > available. > - entry from the RECORD file, as a slash-separated filename relative > to the root of the container. exactly, > > But if you're using get_installed_files(), why would you then want to > read the files? What exactly would you *use* get_installed_files for > which would then leave you needing to read the files? If it's to check > they haven't changed (by comparing md5 values) you're doing that to > uninstall, so that's the responsibility of the uninstall function. > > Again, it's a question of what is a public API, and what is the use > case it's designed for. Right. These APIs were created for third-party package managers. One use case of a package manager is the uninstallation, but I have no other use case in mind. > > I'm currently writing a SQLite importer, which will allow me to store > "files" in any sort of database tables I want, so I can build in some > nice pathological behaviour. That should tease out some awkward corner > cases :-) Sounds good. Semi-related: even if the files themselves are in the filesystem, having a sqlite db to index the list of installed distributions makes a good cache solution to reduce the disk I/O and speed up the query functions. So maybe we could use a disk-based cache for site-packages-like directories in the form of a sqlite db. That's what I am experimenting on my side. > > Paul > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - get_egginfo_files
2009/7/5 Paul Moore : > The PEP says: > > """ > get_egginfo_files(local=False) -> iterator of paths > > Iterates over the RECORD entries and return paths for each line if the > path is pointing a file located in the .egg-info directory or one of > its subdirectory. > """ > > Should this method really only return filenames noted in the RECORD > file? Would it not be better for it to iterate over *all* files in the > .egg-info directory? > I understand that there shouldn't, in practice, > be any files in that directory *not* mentioned in the RECORD file, but > given that we already have get_installed_files to read the RECORD > file, I would imagine it's better for this file to so something more > than filter the return values from get_installed_files. I don't see a use case for having more out of get_egginfo_files. I still find it useful because to iterate over metadata files. Maybe we could remove it and add a filter option for get_installed_files. A callable that gets each visited file and returns True or False to filter them out: get_installed_files(path, filter=callable) And then provide a "egginfo_files" callable to get what we have with get_egginfo_files : get_installed_files(path, filter=egginfo_files) > > Actually, on that note, consider the pkgutil functions: > > def get_distribution(name): > for d in get_distributions(): > if d.name == name: > return d > return None > > def get_file_users(path): > for d in get_distributions(): > if d.uses(path): > yield d > > These don't actually add much to the API. While I can see the > advantage of having them as convenience methods, it might be worth > pointing out in the PEP that that's all they are. Sure, > > Similarly, how valuable is Distribution.name, given that it's the same > as Distribution.metadata.name? I'm probably just going to make it a > property - It's just for conveniency, since this metadata field is also the identifier of the distribution. > > @property > def name(self): > return self.metadata.name I don't think this adds any value, since self.metadata is a read-only instance, that gets loaded once when the Distribution object is created. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - get_egginfo_files
2009/7/5 P.J. Eby : > At 05:26 PM 7/5/2009 +0100, Paul Moore wrote: >> >> def get_distribution(name): >> for d in get_distributions(): >> if d.name == name: >> return d >> return None > > Btw, this is broken code anyway, because it's not handling > case-insensitivity or name canonicalization. (I've mentioned these issue > previously on the distutils-sig.) Yes thanks, we need to fix that, the case-insensitivity or name canonicalization functions are present, just to be used in that function too > > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/5 P.J. Eby : > At 03:13 PM 7/5/2009 +0200, Tarek Ziadé wrote: >> >> The loader would still work with "DIST_NAME.egg-info" directories for >> compatibility with >> existing format in the query APIs, but the Distutils install command >> would rather create "DIST_NAME.metadata" > > Note that this would then break setuptools without adding any benefit; > ".metadata" is less precise and less unique than '.egg-info'. But if it's based on PEP 302 protocols and if the pkgutil code works with the sys.meta_path hook, setuptools could then provide its loader, based on its EggFormats and act as a provider without being broken. In that case, Distutils could provide a standard loader, with the change I mentioned. > If you want a > clearer name, '.pydist' or some such would at least be reasonably specific. > (It'd still have a backward compatibility problem, but at least then > there'd be some benefit to the name change.) I do find "DIST_NAME.metadata" well-named and specific. But I guess that's just bikeshedding :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
But as I said earlier, if we use a PEP 302-like loader, distutils will be able to consume several loaders, so setuptools will be able to provide its storage strategy (naming and egg dir locations) So I don't understand why you are saying that it will be incompatible or get confused. 2009/7/6 P.J. Eby : > At 07:10 AM 7/6/2009 +1000, Nick Coghlan wrote: >> >> By using a new name for the >> directory we *guarantee* that old packaging utilities won't get confused >> by the new format (they simply won't acknowledge its existence). > > This is incorrect; they will get confused because they will think that the > relevant package is *not* installed, and proceed to install a duplicate. > That's why .egg-info was added to the stdlib in the first place. > > -- Tarek Ziadé | http://ziade.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/6 Paul Moore : > 2009/7/6 P.J. Eby : >> At 08:43 PM 7/5/2009 +0200, Tarek Ziadé wrote: >>> >>> But if it's based on PEP 302 protocols and if the pkgutil code works >>> with the sys.meta_path hook, >>> setuptools could then provide its loader, based on its EggFormats and >>> act as a provider without being broken. >> >> You misunderstand me. The whole point of putting .egg-info in distutils in >> the first place was to enable setuptools to detect the presence of >> disutils-installed packages. That's what's broken by changing the name. > Forget about my previous mail, I didn't see that answer, > This implies that there is no possibility that setuptools will be > changed to support the new standard. That's patently false. Whether > you *want* to change setuptools in such a way is up to you. And it's > worth a note in the PEP if this change is made, that it will require a > change to setuptools if that package is still to recognise > distutils-installed packages. +1 setuptools is built on the top of distutils, not the contrary. And if setuptools wants to query installed distribution, it will have to be changed to use the query functions added in pkgutil. Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
2009/7/6 Ronald Oussoren : > I'm -1 on changing the name. For better or worse setuptools is the elephant > in the room w.r.t. package management and it would IMHO be better to stay > compatible (even if the stdlib only implements a subset of > setuptools/pkg_resources) > I'd rather see the elephant evolves. I don't see why we should bend a standard we want to introduce in the stdlib, for a third-party package that is able to evolve to stick to a new standard without any problem. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
On Mon, Jul 6, 2009 at 5:14 PM, Paul Moore wrote: > [...] > And yet, given that PEP 376 is explicitly being designed with a goal > being to act as the common standard that *all* package management > formats can use, is it not the whole point that once it's defined and > we have achieved consensus, existing package managers will switch to > using it rather than retaining a range of custom formats? That's the goal from the very beginning. > > I honestly fail to see why we are catering to this "setuptools cannot > change" view. Usually, it's the standard library that cannot change, > because of the longer release cycle of the core. If a standard is > agreed, and setuptools won't conform to it, maybe it's time for > someone to fork setuptools (unilaterally, if Phillip can't agree to an > amicable transfer). Otherwise setuptools will remain a roadblock for > any development of distutils. > > Frustrated-ly y'rs +1 why can't we just go ahead and continue the work as we started with PEP 376, introducing your work on PEP 302-like behavior. Then if we get a consensus on this PEP and introduce it in 2.7/3.2, setuptools will have to follow this consensus. FWIW I am also frustrated because the setuptools development has been stopped 9 months ago, and *doesn't work anymore with the current distutils trunk*. There are a lot of patches waiting in the setuptools tracker people wrote. I hate to say this but this project needs an active maintainer or it's going to die. So I've said in distutils-sig that I would start my own branch of setuptools before 2.7/3.2 is out for the people in the community that relies on this package. Not to maintain the project because I have a lot to do already on distutils side. But at least to make sure everyone can work with the upcoming releases of Python 2.x and 3.X without having their setup.py broken (and switch eventually to a plain distutils solution maybe) Tarek ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
On Mon, Jul 6, 2009 at 5:53 PM, Paul Moore wrote: > 2009/7/6 Tarek Ziadé : >> why can't we just go ahead and continue the work as we started with PEP 376, >> introducing your work on PEP 302-like behavior. >> >> Then if we get a consensus on this PEP and introduce it in 2.7/3.2, >> setuptools will have to follow this consensus. > > Essentially, because when I ask questions, responses along the lines > of "you have to do it like X because setuptools does that" come back, > and (not being a setuptools user) I can't tell whether there's a valid > reason in there. Notice that we created PEP 376 in distutils-SIG with most of the valid reasons/use cases setuptools had, with the help of Phillip, before I brought it up again on python-dev. > > I'm uncomfortable assuming that setuptools experience is irrelevant, > but I can't distinguish between valid arguments and "setuptools can't > change" arguments. > > I need to write another email with a list of outstanding open issues. > If we can thrash them out *without* getting bogged down in setuptools > compatibility questions, then I think we can move forward again. I'll > do that this evening. Ok, I'll wait for your work to work on the PEP again then, and your push in the hg repo as well. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
On Mon, Jul 6, 2009 at 6:58 PM, P.J. Eby wrote: >> My site-packages has a confusing mix of egginfo directories and files. >> Note that I NEVER use setuptools other than where an existing >> package's setup.py requires it. In that case, I still only do python >> setup.py bdist_wininst and install the generated installer. If you end up with confusing mix of egginfo directories and files, that's because for the packages that uses setuptools in their setup.py, it patches distutils and to creates its own egg-info format. >> So is PEP 376 going to be able to cope with the stuff I have installed >> at the moment? If not, what's the point??? > > If I understand Tarek's proposal correctly, then no, it will not cope. Why that ? Can you detail ? On a system that uses only plain distutils distributions, it'll work. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
On Tue, Jul 7, 2009 at 12:16 AM, Paul Moore wrote: >> >> I believe the idea of having different names in 2.x and 3.x would likely >> cause too many problems for simple bdist_* installers of modules that >> use only the common 2.x/3.x language subset, so I'm also -1 on that idea. > > That suits me too. I'm happy for the name to stay as it is. While I'd rather like having an "egg"-free standard, it's a detail compared to the rest, so I'm 0+ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/7 Paul Moore : > 2009/7/6 P.J. Eby : >> At 07:38 PM 7/6/2009 +0100, Paul Moore wrote: >>> >>> As promised, here are some open questions on PEP 376. >>> >>> - Will the public API names be changed from *egginfo* to *metadata*? >> >> +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff >> as.) > > Thanks. I think this one is pretty non-controversial. +1, I'll change the PEP accordingly; >>> - How will bdist_wininst/bdist_msi/bdist_rpm be updated? >> >> bdist_wininst, bdist_dumb, and various others use 'install --root' pattern >> to generate files for installation, which means that they would >> transparently end up writing a correct RECORD file, except for the inclusion >> of incorrect absolute paths for non-libdir-relative files. However, if we >> used the "all relative in RECORD, with a base in INSTALLER", these cases >> could transparently be treated as another instance of install directory >> relocation. >> >> I don't know if bdist_msi does a --root install before generating the .msi; >> if it does, then it should work the same way. > > Yes, that sounds like a plausible approach. As I said earlier in the previous thread, all these commands rely on a call to the install command, so bdist_msi does too. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
2009/7/7 Paul Moore :
> 2009/7/6 Nick Coghlan :
>> I'd add one more question to the list: is allowing backslash separated
>> names in the RECORD file actually a good idea, or would it be better to
>> always use forward slashes?
>
> They do always use forward slashes.
>
>> For the other questions, I don't have anything much to add to PJE's
>> comments, except that the "all relative" paths idea won't work due to
>> the Windows drive letter issue (i.e. if an installer puts files in
>> C:\Program Files, there is no guarantee that a relative path between
>> site-packages and Program Files even exists if Python is installed on a
>> different drive).
>
> The big question, though, is can an installer actually *do* that in
> practical terms?
>
> - There are *no* guaranteed absolute locations on Windows, so any such
> oddly-located file would require user interaction to work. Certainly
> bdist_wininst and bdist_msi don't do that.
> - My experiments indicate that bdist_{wininst,msi} are broken with
> respect to absolute paths anyway: they do a --root install to a
> temporary directory (and the absolute paths don't end up in there) and
> then package up that temporary directory.
yes that's unfortunately the case for all windows-based installation.
wether it's a bdist call to the install command, to create a binary package,
wether it's an installation.
c:\something or d:\something will be installed in sys.prefix\something.
I will add an issue for distutils for this, probably ending up raising
an exception when we hit this case, because I don't see how these
paths can work.
Unless we define a "drive that contains the python installation" maybe, or
the "Program Files" directory
would that make sense from a win32 point of view ?
>
> I still want to see a real life example that demonstrates that there
> is a genuine issue here. We're spending a lot of energy and complexity
> trying to design a solution to a problem that actually doesn't appear
> to exist in practice...
>
> (To be honest, I'd be fairly confident in saying that absolute paths
> can be ignored on Windows, subject to some corner cases that I haven't
> thought through yet. My worry is that I don't know what Unix and Mac
> users might do, so I can't just wish away the issue because it can't
> arise on Windows. Can a Unix/Mac user offer a real-world example on
> their own system?)
>
I know some people are writing to /etc to add their configuration file
on the system,
So a real-world example under linux would be:
setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)
That is basically how the examples are shown at:
http://docs.python.org/distutils/setupscript.html#installing-additional-files
But this is already os-specific, and exists because distutils doesn't
have a way (yet)
to express systems locations independantly from their physical location,
like what the RPM system does with %VARIABLES.
So another way to handle this maybe, like I have added with $PREFIX
and $EXEC_PREFIX
would be to nominate a list of variables that every python environment
has (querying
modules like sys) and let the developers use them as root locations
for some files.
- sys.prefix
- sys.exec_prefix
- some elements returned by distutils.sysconfig.get_config_vars()
(distutils.sysconfig queries the python Makefile amongst others)
- ...
Semi-related: distutils.sysconfig should be removed from distutils,
and be a standalone module in the sdtlib for example.
Regards
Tarek
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 - Open questions
On Tue, Jul 7, 2009 at 10:33 AM, Paul Moore wrote:
> 2009/7/7 Tarek Ziadé :
>> Unless we define a "drive that contains the python installation" maybe, or
>> the "Program Files" directory
>>
>> would that make sense from a win32 point of view ?
>
> I can't imagine that it would be useful in practice.
>
>> I know some people are writing to /etc to add their configuration file
>> on the system,
>>
>> So a real-world example under linux would be:
>>
>> setup(..., data_files=[('/etc', ['myconf.cfg'])], ...)
>>
>>
>> That is basically how the examples are shown at:
>>
>> http://docs.python.org/distutils/setupscript.html#installing-additional-files
>
> Thanks. Yes, that makes for a good example.
>
>> But this is already os-specific, and exists because distutils doesn't
>> have a way (yet)
>> to express systems locations independantly from their physical location,
>> like what the RPM system does with %VARIABLES.
>>
>> So another way to handle this maybe, like I have added with $PREFIX
>> and $EXEC_PREFIX would be to nominate a list of variables that every
>> python environment has (querying modules like sys) and let the developers
>> use them as root locations for some files.
>>
>> - sys.prefix
>> - sys.exec_prefix
>> - some elements returned by distutils.sysconfig.get_config_vars()
>> (distutils.sysconfig queries the python Makefile amongst others)
>> - ...
>
> This adds extra complexity to the RECORD format, for little practical
> benefit that I can see.
>
> From the POV of core distutils:
> - Windows users use bdist_wininst and/or bdist_msi, and absolute
> locations are a lost cause.
> - Presumably, some people use bdist_rpm (I don't know if there are
> other ways of creating RPMs).
They are othe ways to generate RPMs.
There are also debian packaging scripts
> - Everyone else uses setup.py install or a 3rd party tool.
> - PEP 302 style loaders aren't relevant as the core only uses the
> filesystem (not even zip files).
> - Only 3rd party tools will consume this data.
>
> So, we need input from developers of 3rd party tools here. Phillip has
> stated the case for setuptools, from his POV having everything
> relative to the "install location" which is stored elsewhere (in the
> installer file) is fine. I'd like to know whether he needs
> "upwards-pointing" relative paths like ../../../../xx.py, but that's a
> small detail.
>
> So - do any other potential users of the PEP 376 metadata want to speak up?
>
> At the moment, it feels like we're designing things more or less in a vacuum.
I am CC'ing the people that worked with us for the versionning matters,
they can speak from a Fedora and Ubuntu POV, I am also adding Jim for
zc.buildout,
he can provide precious input.
(I am sorry if some people get the mail twice)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata
On Tue, Jul 7, 2009 at 11:28 AM, M.-A. Lemburg wrote: > > Writing an uninstall command really isn't all that hard, provided > you stick with the standard distutils "python setup.py install" > dance. The whole packaging approach only complicates things, IMHO. You can't expect people to keep the distribution they used to install in a corner of their hard drive, to make sure they will be able to uninstall it (or to ask them to download it again) That's all about the RECORD files and the asymetric uninstall script described in PEP 376. having an uninstall command on the top of the uninstall script is a nice shortcut though. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
