Re: [Python-Dev] [Distutils] Capsule Summary of Some Packaging/Deployment Technology Concerns
I've added your comments to a wiki page (http://wiki.python.org/moin/PackagingBOF) I was working on to summarize some of what went on during these BoF meeting, at least from the Enthought point-of-view. Unfortunately, I wasn't at the first night's event and don't yet have Travis Oliphant's notes on it here in front of me (he's still sprinting) so I only added some more detail to your comments, and also noted some previous issues we'd briefly discussed via e-mail with Phillip. It was great to see so many people interested in sharing their experiences and wanting to help things get better! As you can probably guess as a result of this being a two-night meeting, there wasn't enough time to discuss everything that needed to be brought up. It was suggested that a wiki page be created (see above) and that a new mailing list be setup for those who wanted to discuss further. (Some didn't feel the existing distutils-sig was appropriate.) I'll try to get the latter done shortly. -- Dave Jeff Rush wrote: > I was in a Packaging BoF yesterday and, although not very relevant to the > packager bootstrap thread, Guido has asked me to post some of the concerns. > > The BoF drew about 15 people... > ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby wrote: At 05:10 PM 3/17/2008 -0500, Jeff Rush wrote: 1. Many felt the existing dependency resolver was not correct. They wanted a full tree traversal resulting in an intersection of all restrictions, instead of a first-acceptable-solution approach taking now, which can result in top-level dependencies not being enforced upon lower levels. The latter is faster however. One solution would be to make the resolver pluggable. Patches welcome, on both counts. Personally, Bob and I originally wanted a full-tree intersection, too, but it turned out to be hairier to implement than it seems at first. My guess is that none of the people who want it, have actually tried to implement it without a factorial or exponential O(). But that doesn't mean I'll be unhappy if somebody succeeds. :) I think we'd make significant progress by just intersecting the dependencies we know about as we progress through the dependency tree. For example, if A requires B==2 and C==3, and if B requires C>=2,<=4, then at the time we install A we'd pick C==3 and also at the time we install B we'd pick C==3. As opposed to the current scheme that would choose C==4 for the latter case. This would allow dependent projects (think applications here) to better control the versions of the full set of libraries they use. Things would still fail (like they do now) if you ran across dependencies that had no intersection or if you encountered a new requirement after the target projected was already installed. If you really wanted to do a full-tree intersection, it seems to me that the problem is detecting all the dependencies without having to spend significant time downloading/building in order to find them out. This could be solved by simply extending the cheeseshop interface to export the set of requirements outside of the egg / tarball / etc. We've done this for our own egg repository by extracting the appropriate meta-data files out of EGG-INFO and putting it into a separate file. This info is also useful for users as it gives them an idea of how much *new* stuff is going to be installed (a la yum, apt-get, etc.) In other words, we attempt to achieve heuristically what's being proposed to do algorithmically. And my guess is that whatever cases the heuristic is failing at, would probably not be helped by an algorithmic approach either. But I would welcome some actual data, either way. With our ETS projects, we've run into problems with the current heuristic. Perhaps we just don't know how to make it work like we want? We have a set of projects that we want to be individually installable (to the extent that we limit cross-project dependencies) but we also want to make it easy to install the complete set. We use a meta-egg for the latter. It's purpose is only to specify the exact versions of each project that have been explicitly tested to work together -- you could almost think of it as a source control system tag. Whereas on the individual projects, we explicitly want to ensure that people get the latest possible release of each required API so the version requirements are wider here. This setup causes problems whenever we release new versions of projects because it seems easy_install ignores the meta-egg exact versions when it gets down into a project and comes across a wider cross-project dependency. We ended up having to give up on the ranges in the cross-project dependencies and synchronize them to the same values in the meta-egg dependencies. There are numerous side-effects of this that we don't like but we haven't found a way around it. Again, though, patches are welcome. :) (Specifically, for the trunk; I don't see a resolver overhaul as being suitable for the 0.6 stable branch.) We're planning to pursue this (for the above mentioned strategy) as soon as we work ourselves out of a bit of a backlog of other things to do. 2. People want a solution for the handling of documentation. The distutils module has had commented out sections related to this for several years. As with so many other things, this gets tossed around the distutils-sig every now and then. A couple of times I've thrown out some options for how this might be done, but then the conversation peters out around the time anybody would have to actually do some work on it. (Me included, since I don't have an itch that needs scratching in this area.) In particular, if somebody wants to come up with a metadata standard for including documentation in eggs, we've got a boatload of hooks by which it could be done. Nothing's stopping anybody from proposing a standard and building a tool, here. (e.g. using the setuptools command hook, .egg-info writer hook, etc.) Enthought has started an effort (it's currently one of two things in our ETSProjectTools project at https://svn.enthought.com/svn/enthought/ETSProjec
Re: [Python-Dev] [Distutils] Capsule Summary of Some Packaging/Deployment Technology Concerns
Phillip J. Eby wrote: At 03:57 AM 3/19/2008 -0500, Jeff Rush wrote: I'd be willing to help out, and keep a carefully balanced hand in what is accepted. I'm not sure exactly how to go about such a handoff though. My guess is that we need a bug/patch tracker, and a few people to review, test, and apply. Maybe a transitional period during which I just say yea or nay and let others do the test and apply, before opening it up entirely. That way, we can perhaps solidify a few principles that I'd like to have stay in place. (Like no arbitrary post-install code hooks.) +1 to blessing more people to commit. +1 to the transition period idea. These two ought to enable things to move a bit quicker than taking a year to accept a patch. :-) In addition to a bug tracker and patch manager, seems like perhaps a wiki to help document some of these solidified principles and other notes would be a good thing. (Like a patch should almost always include at least one test, possibly more.) Given that the source for setuptools is in the python.org svn, couldn't we just use the python.org roundup and wiki for these facilities? Though looking at the list of components, it seems that things in the sandbox generally aren't tracked in this infrastructure. In which case, I'm sure we could use sf, launchpad, or some such external provider. Enthought could even host this stuff. Like Jeff Rush, I'm also willing to help out as both a writer and reviewer of patches. As you can see from my earlier posts there are a number of things (besides running an arbitrary post-install script) that we'd like to be able to get into the codebase. -- Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The Breaking of distutils and PyPI for Python 3000?
Martin v. Löwis wrote: 1. What is the plan for PyPI when Python 3.0 comes out and dependencies start getting satisfied from distribution across the great divide, e.g. a 3.0-specific package pulls from PyPI a 2.x-specific package to meet some need? Are there plans to fork PyPI, apply special tags to uploads or what? I don't see the need to for PyPI. For packages (or "distributions", to avoid confusion with Python packages), I see two options: a) provide a single release that supports both 2.x and 3.x. The precise strategy to do so might vary. If one is going for a single source version, have setup.py run 2to3 (or perhaps 3to2). For dual-source packages, have setup.py just install the one for the right version; setup.py itself needs to be written so it runs on both versions (which is easy to do). b) switch to Python 3 at some point (i.e. burn your bridges). You seem to be implying that some projects may release separate source distributions. I cannot imagine why somebody would want to do that. While not quite to the same scale as the 2 to 3 transition, this problem seems like one that would generally already exist. If one writes code that uses newer 2.4/2.5 features (say decorators for example,) it won't build/run on 2.3 or earlier installs. How have people been handling that sort of situation? Is it only by not using the newer features or is there some situation I'm just not seeing in a brief review of a few projects pages on PyPI where there is only one source tarball? -- Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you
Phillip J. Eby wrote: Applying LSB and FHS to the innards of Python packages makes as much sense as applying them to the contents of Java .jar files -- i.e., none. If it's unchanging data that's part of a program or library, then it's a program or library, just like static data declared in a C program or library. Whether the file extension is .py, .so, or even .png is irrelevant. The FHS defines places to put specific kinds of files, such as command scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation (/usr/share/doc/package-name), and configuration files (/etc). There are several kinds of files identified and places defined to put them. Distribution by eggs has a tendency to scoop up all of those files and put them in /usr/lib/python/site-packages, regardless of where they belong. Eggs don't include documentation or configuration files, and they install scripts in script directories, so I don't get what you're talking about here. For any other data that a package accesses at runtime, my earlier comments apply. We've talked a bit about this before, and IIRC, at that time you (Phillip) were willing to consider patches to setuptools that allowed for the inclusion of documentation in eggs such that it was placed into an LSB/FHS appropriate directory (or some standard dir for non-LSB systems) during the install process. I'm assuming that something similar for config files wouldn't be a problem either? Or is this whole idea out the window given the way the discussion has trended and the reiteration above that eggs are meant to be similar in principal to jars? Not that I have a patch yet, but we've been working on it in our "spare time" over here at Enthought. I'm now wondering if we're wasting our time. :-) I think the biggest use-case confusion in the current discussion is whether we're talking about applications or libraries? If we're talking about libraries, then clearly distribution of only executables is sufficient because anything else should be handled by the application distribution when that library is used in an app. Whereas if we're talking about applications, you probably *do* want to include documentation, configuration info, etc. in your distribution. I think I can sum up any further points by simply asking: "Should it be safe to assume I can distribute my application via eggs / easy_install just because it is written in Python?" -- Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] How does one build Python25.chm on Windows?
I'm trying to build a Python25.chm file using the source in the Python 2.5.2 tarball. I'm not really a Windows guru so I'm at a loss of even where to begin as the Doc\README file doesn't mention how to build .chm files at all. I've done a number of web searches with things like "how to build python25.chm" but am not finding any useful hits. (Most hits tell people how to use the Python25.chm for various purposes but none on actually building it that I've seen.) Any pointers would be appreciated! -- Dave ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
