[Python-Dev] Path object design
Path("ABC") # Works the same for files/directories on any platform.
Nevertheless, I'm open to seeing a multi-class API, though hopefully
less verbose than Talin's preliminary one (Oct 26). Is it necessary
to support path.parent(), pathobj.parent(), io.dir.listdir(), *and*
io.dir.Directory(). That's four different namespaces to memorize
which function/method is where, and if a function/method belongs to
multiple ones it'll be duplicated, and you'll have to remember that
some methods are duplicated and others aren't... Plus, constructors
like io.dir.Directory() look too verbose. io.Directory() might be
acceptable, with the functions as class methods.
I agree that supporting non-filesystem directories (zip files,
CSV/Subversion sandboxes, URLs) would be nice, but we already have a
big enough project without that. What constraints should a Path
object keep in mind in order to be forward-compatible with this?
If anyone has design ideas/concerns about a new Path class(es), please
post them. If anyone would like to work on a directory-based
spec/implementation, please email me.
--
Mike Orr <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
;
> > 2) Make selected "shutil" operations available via the "os" name-
> > space; the old POSIX API vs. POSIX SHELL distinction is pretty
> > irrelevant. Also make the os.path predicates available via the
> > "os" namespace.
> >
> > This gives a very simple conceptual model for the user; to manipulate
> > path *names*, use "os.path.(string)" functions or the ""
> > wrapper. To manipulate *objects* identified by a path, given either as
> > a string or a path wrapper, use "os.(path)". This can be taught in
> > less than a minute.
Making this more concrete, I think Fredrik is suggesting:
- Make (os.path) abspath, basename, commonprefix, dirname,
expanduser, expandvars, isabs, join, normcase, normpath, split,
splitdrive, splitext, splitunc methods of a Path object.
- Copy functions into os: (os.path) exists, lexists,
get{atime,mtime,ctime,size}, is{file,dir,link,mount}, realpath,
samefile, sameopenfile, samestat, (shutil) copy, copy2,
copy{file,fileobj,mode,stat,tree}, rmtree, move.
- Deprecate the old functions to remove in 3.0.
- Abandon os.path.walk because os.walk is better.
This is worth considering as a start. It does mean moving a lot of
functions that may be moved again at some point in the future.
If we do move shutil functions into os, I'd at least like to make some
tiny improvements in them. Adding four lines to the beginning of
rmtree would make it behave like my purge() function without
detracting from its existing use:
if not os.exists(p):
return
if not os.isdir(p):
p.remove()
Also, do we really need six copy methods? copy2 can be handled by a
third argument, etc.
--
Mike Orr <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
On 11/1/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> On 06:14 pm, [EMAIL PROTECTED] wrote:
> >[EMAIL PROTECTED] wrote:
> >
> >> I assert that it needs a better[1] interface because the current
> >> interface can lead to a variety of bugs through idiomatic, apparently
> >> correct usage. All the more because many of those bugs are related to
> >> critical errors such as security and data integrity.
>
> >instead of referring to some esoteric knowledge about file systems that
> >us non-twisted-using mere mortals may not be evolved enough to under-
> >stand,
>
> On the contrary, twisted users understand even less, because (A) we've been
> demonstrated to get it wrong on numerous occasions in highly public and
> embarrassing ways and (B) we already have this class that does it all for us
> and we can't remember how it works :-).
This is ironic coming from one of Python's celebrity geniuses. "We
made this class but we don't know how it works." Actually, it's
downright alarming coming from someone who knows Twisted inside and
out yet still can't make sense of path patform oddities.
> * This is confusing as heck:
>>>> os.path.join("hello", "/world")
>'/world'
That's in the documentation. I'm not sure it's "wrong". What should
it do in this situation? Pretend the slash isn't there?
This came up in the directory-tuple proposal. I said there was no
reason to change the existing behavior of join. Noam favored an
exception.
>>>> os.path.join("hello", "slash/world")
>'hello/slash/world'
That has always been a loophole in the function, and many programs
depend on it. Again, is it "wrong"? Should an embedded separator in
an argument be an error? Obviously this depends on the user's
knowledge that the separator happens to be slash.
>>>> os.path.join("hello", "slash//world")
>'hello/slash//world'
Again a case of what "should" it do? The filesystem treats it as a
single slash. The user didn't call normpath, so should we normalize
it anyway?
> * Sometimes a path isn't a path; the zip "paths" in sys.path are a good
> example. This is why I'm a big fan of including a polymorphic interface of
> some kind: this information is *already* being persisted in an ad-hoc and
> broken way now, so it needs to be represented; it would be good if it were
> actually represented properly. URL
> manipulation-as-path-manipulation is another; the recent
> perforce use-case mentioned here is a special case of that, I think.
Good point, but exactly what functionality do you want to see for zip
files and URLs? Just pathname manipulation? Or the ability to see
whether a file exists and extract it, copy it, etc?
> * you have to care about unicode sometimes. rarely enough that none of
> your tests will ever account for it, but often enough that _some_ users will
> notice breakage if your code is ever widely distributed.
This is a Python-wide problem. The move to universal unicode will
lessen this, or at least move the problem to *one* place (creating the
unicode object), where every Python programmer will get bitten by it
and we'll develop a few standard strategies to deal with it.
(The problem is that if str and unicode are mixed in expressions,
Python will promote the str to unicode and you'll get a
UnicodeDecodeError if it contains non-ASCII characters. Figuring out
all the ways such strings can slip into a program is difficult if
you're dealing with user strings from an unknown charset, or your
MySQL server is configured differently than you thought it was, or the
string contains Windows curly quotes et al which are undefined in
Latin-1.)
> * the documentation really can't emphasize enough how bad using
> 'os.path.exists/isfile/isdir', and then assuming the file continues to exist
> when it is a contended resource, is. It can be handy, but it is _always_ a
> race condition.
What else can you do? It's either os.path.exists()/os.remove() or "do
it anyway and catch the exception". And sometimes you have to check
the filetype in order to determine *what* to do.
--
Mike Orr <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
printer port (normally LPT1) NUL bit bucket AUX game port? COPY CON FILENAME.TXT # Unix: "cat >filename.txt". COPY FILENAME.TXT PRN # Unix: "lp filename.txt" or "cat filename.txt | lp". TYPE FILENAME.TXT # Unix: "cat filename.txt". > >Where have all the proponents of non-OO or limited-OO strategies been? > > This continuum doesn't make any sense to me. Where would you place > Twisted's solution on it? In the "let's create a brilliant library and put a dark box around it so nobody knows it's there" position. Although you say you've been trying to spread the word about it. For whatever reason, I haven't heard about it till now. Not sure what this means. But what I meant is, we OO proponents have been trying to promote path.py and/or get a similar module into the stdlib for years, and all we got was... not even hostility... just indifference and silence. People like to complain about os.path but not do anything about fixing it, or even to say which approach they *would* support. Talin started a great thread on the python-3000 list, going back to the beginning and saying "What is wrong with os.path, how much does it need fixing, and is consensus on an API possible?" Maybe he did what the rest of us (including me) should have done long ago. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
On 11/1/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> On 01:46 am, [EMAIL PROTECTED] wrote:
> >On 11/1/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
>
> >This is ironic coming from one of Python's celebrity geniuses. "We
> >made this class but we don't know how it works." Actually, it's
> >downright alarming coming from someone who knows Twisted inside and
> >out yet still can't make sense of path patform oddities.
>
> Man, it is going to be hard being ironically self-deprecating if people keep
> going around calling me a "celebrity genius". My ego doesn't need any help,
> you know? :)
I respect Twisted in the same way I respect a loaded gun. It's
powerful, but approach with caution.
> If you ever think I'm suggesting breaking something in Python, you're
> misinterpreting me ;). I am as cagey as they come about this. No matter
> what else happens, the behavior of os.path should not really change.
The point is, what *should* a join-like method do in a future improved
path module? os.path.join should not change because too many programs
depend on its current behavior, in ways we can't necessarily predict.
But a new function/method is not bound by these constraints, as long
as the boundary cases are well documented. All the os.path and
file-related os/shutil functions need to be reexamined in this
context. Maybe the existing behavior is best, maybe we'll keep it
even if it's sub-optimal, but we should document why we're making
these choices.
> >The user didn't call normpath, so should we normalize it anyway?
>
> That's really the main point here.
>
> What is a path that hasn't been "normalized"? Is it a path at all, or is it
> some random garbage with slashes (or maybe other things) in it? os.path
> performs correct path algebra on correct inputs, and it's correct (as far as
> one can be correct) on inputs that have weird junk in them.
I'm tempted to say Path("/a/b").join("c", "d") should do the same
thing your .child method does, but allow multiple levels in one step.
But on the other hand, there will always be people with prebuilt
"path/fragments" to join to other fragments, and I'm not sure we
should force them to split the fragment just to rejoin it again.
Maybe we need a .join_unsafe method for this, haha.
--
Mike Orr <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Mini Path object
Posted to python-dev and python-3000. Follow-ups to python-dev only please.
On 10/31/06, Fredrik Lundh <[EMAIL PROTECTED]> wrote:
> here's mine; it's fully backwards compatible, can go right into 2.6,
> and can be incrementally improved in future releases:
>
> 1) add a pathname wrapper to "os.path", which lets you do basic
>path "algebra". this should probably be a subclass of unicode,
>and should *only* contain operations on names.
>
> 2) make selected "shutil" operations available via the "os" name-
>space; the old POSIX API vs. POSIX SHELL distinction is pretty
>irrelevant. also make the os.path predicates available via the
>"os" namespace.
>
> this gives a very simple conceptual model for the user; to manipulate
> path *names*, use "os.path.(string)" functions or the ""
> wrapper. to manipulate *objects* identified by a path, given either as
> a string or a path wrapper, use "os.(path)". this can be taught in
> less than a minute.
Given the widely-diverging views on what, if anything, should be done
to os.path, how about we make a PEP and a standalone implementation of
(1) for now, and leave (2) and everything else for a later PEP. This
will make people who want a reasonably forward-compatable object NOW
for their Python 2.4/2.5 programs happy, provide a common seed for
more elaborate libraries that may be proposed for the standard library
later (and eliminate the possibility of moving the other functions and
later deprecating them), and provide a module that will be well tested
by the time 2.6 is ready for finalization.
There's already a reference implementation in PEP 355, we'd just have
to strip out the non-pathname features. There's a copy here
(http://wiki.python.org/moin/PathModule) that looks reasonably recent
(constructors are self.__class__() to make it subclassable), although
I wonder why the class is called path instead of Path. There was
another copy in the Python CVS although I can't find it now; was it
deleted in the move to Subversion? (I thought it was in
/sandbox/trunk/: http://svn.python.org/view/sandbox/trunk/).
So, let's say we strip this Path class to:
class Path(unicode):
Path("foo")
Path( Path("directory"), "subdirectory", "file")# Replaces
.joinpath().
Path()
Path.cwd()
Path("ab") + "c" => Path("abc")
.abspath()
.normcase()
.normpath()
.realpath()
.expanduser()
.expandvars()
.expand()
.parent
.name # Full filename without path
.namebase# Filename without extension
.ext
.drive
.splitpath()
.stripext()
.splitunc()
.uncshare
.splitall()
.relpath()
.relpathto()
Would this offend anyone? Are there any attribute renames or method
enhancements people just can't live without? 'namebase' is the only
name I hate but I could live with it.
The multi-argument constructor is a replacement for joining paths.
(The PEP says .joinpath was "problematic" without saying why.)This
could theoretically go either way, doing either the same thing as
os.path.join, getting a little smarter, or doing "safe" joins by
disallowing "/" embedded in string arguments.
I would say that a directory-tuple Path object with these features
could be maintained in parallel, but since the remaining functions
require string arguments you'd have to use unicode() a lot.
--
Mike Orr <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Path object design
On 11/5/06, Andrew Dalke <[EMAIL PROTECTED]> wrote:
>
>I agree that supporting non-filesystem directories (zip files,
>CSV/Subversion sandboxes, URLs) would be nice, but we already have a
>big enough project without that. What constraints should a Path
>object keep in mind in order to be forward-compatible with this?
>
> Is the answer therefore that URLs and URI behaviour should not
> place constraints on a Path object becuse they are sufficiently
> dissimilar from file-system paths? Do these other non-FS hierarchical
> structures have similar differences causing a semantic mismatch?
This discussion has renforced my belief that os.path.join's behavior
is correct with non-initial absolute args:
os.path.join('/usr/bin', '/usr/local/bin/python')
I've used that in applications and haven't found it a burden.
Its behavior with '..' seems justifiable too, and Talin's trick of
wrapping everything in os.path.normpath is a great one.
I do think join should take more care to avoid multiple slashes
together in the middle of a path, although this is really the
responsibility of the platform library, not a generic function/method.
Join is true to its documentation of only adding separators and never
than deleting them, but that seems like a bit of sloppiness. On the
other hand, the filesystems don't care; I don't think anybody has
mentioned a case where it actually creates a path the filesystem can't
handle.
urljoin clearly has a different job. When we talked about extending
path to URLs, I was thinking more in terms of opening files, fetching
resources, deleting, renaming, etc. rather than split-modify-rejoin.
A hypothetical urlpath module would clearly have to follow the URL
rules. I don't see a contradition in supporting both URL joining
rules and having a non-initial absolute argument, just to avoid
cross-"platform" surprises. But urlpath would also need methods to
parse the scheme and host on demand, query strings, #fragments, a
class method for building a URL from the smallest parts, etc.
As for supporting path fragments and '..' in join arguments (for
filesystem paths), it's clearly too widely used to eliminate. Users
can voluntarily refrain from passing arguments containing separators.
For cases involving a user-supplied -- possibly hostile -- path,
either a separate method (safe_join, child) could achieve this, or a
subclass implemetation that allows only safe arguments.
Regarding pathname-manipulation methods and filesystem-access methods,
I'm not sure how workable it is to have separate objects for them.
os.mkdir( Path("/usr/local/lib/python/Cheetah/Template.py").parent )
Path("/usr/local/lib/python/Cheetah/Template.py").parent.mkdir()
FileAccess(
Path("/usr/local/lib/python/Cheetah/Template.py").parent ).mkdir()
The first two are reasonable. The third... who would want to do this
for every path? How often would you reuse the FileAccess object? I
typically create Path objects from configuration values and keep them
around for the entire application; e.g., data_dir. Then I create
derived paths as necessary. I suppose if the FileAccess object has a
.path attribute, it could do double-duty so you wouldn't have to store
the path separately. Is this what the advocates of two classes have
in mind? With usage like this?
my_file = FileAccess( file_access_obj.path.joinpath("my_file") )
my_file = FileAccess( Path(file_access_obj,path, "my_file") )
Working on my Path implementation. (Yes it's necessary, Glyph, at
least to me.) It's going slow because I just got a Macintosh laptop
and am still rounding up packages to install.
--
Mike Orr <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Distribution tools: What I would like to see
On 11/26/06, Talin <[EMAIL PROTECTED]> wrote: > I've been looking once again over the docs for distutils and setuptools, > and thinking to myself "this seems a lot more complicated than it ought > to be". > > Before I get into detail, however, I want to explain carefully the scope > of my critique - in particular, why I am talking about setuptools on the > python-dev list. You see, in my mind, the process of assembling, > distributing, and downloading a package is, or at least ought to be, a > unified process. It ought to be a fundamental part of the system, and > not split into separate tools with separate docs that have to be > mentally assembled in order to understand it. > > Moreover, setuptools is the defacto standard these days - a novice > programmer who googles for 'python install tools' will encounter > setuptools long before they learn about distutils; and if you read the > various mailing lists and blogs, you'll sense a subtle aura of > deprecation and decay that surrounds distutils. Look at the current situation as more of an evoluntionary point than a finished product. There's widespread support for integrating setuptools into Python as you suggest. I've heard it discussed at Pycon the past two years. The reason it hasn't been done yet is technical, from what I've heard. Distutils is apparently difficult to patch correctly and could stand a rewrite. I'm currently studying the Pylons implementation and thus having to learn more about entry points, resources, ini files used by eggs, etc. This requires studying three different pages on the peak.telecommunity.com site -- exactly the problem you're describing. A comprehensive third-party manual that integrates the documentation would be a good place to start. Even the outline of such a manual would be a good. That would give a common baseline of understanding for package users, package developers, and core developers. I wonder if one of the Python books already has this written down somewhere. >From the manual one could then distill a spec for "what's needed in a package manager, what features a distutils upgrade would provide, and what a package should/may contain". That would be a basis for one or more PEPs. The "diff" approach is understandable at the beginning, because that's how the developers think of it, and how most users will approach it initially. We also needed real-world experience to see if the setuptools approach was even feasable large-scale or whether it needed major changes. Now we have more experience, and more Pythoneers are appearing who are unfamiliar with the "distutils-only" approach. So requests like Talin's will become more frequent. It's such a big job and Python 2.6 is slated as "minimal features" release, so it may be better to target this for Python 3 and backport it if possible. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Distribution tools: What I would like to see
On 11/26/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > I have noticed, however, that a signficant number of help requests for > setuptools can be answered by internal links to one of its manuals -- and > when a topic comes up that isn't in the manual, I usually add it. Hmm, I may have a couple topics for you after I check my notes. > The "diff" issue is certainly there, of course, as is the fact that there > are multiple manuals. However, I don't think the answer is fewer manuals, > in fact it's likely to be having *more*. What exists right now is a > developer's guide and reference for setuptools, a reference for the > pkg_resources API, and an all-purpose handbook for easy_install. Each of > these could use beginner's introductions or tutorials that are deliberately > short on details, but which provide links to the relevant sections of the > comprehensive manuals. I could see a comprehensive manual running forty pages, and most readers only caring about a small fraction of it. So you have a point. Maybe more impotant than one book is having "one place to go", a TOC of articles that are all independent yet written to complement each other. But Talin's point is still valid. Users have questions like, "How do I structure my package so it takes advantage of all the gee-whiz cheeseshop features? Where do I put my tests? Should I use unittest, py.test, or nose? How will users see my README and my docs if they easy_install my package? What are all those files in the EGG-INFO directory? What's that word 'distribution' in some of the function signatures? How do I use entry points, they look pretty complicated?" Some of these questions are multi-tool or are outside the scope of setuptools; some span both the Peak docs and the Python docs. People need an answer that starts with their question, rather than an answer that's a section in a manual describing a particular tool. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Distribution tools: What I would like to see
On 11/27/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Talin schrieb: > > As far as rewriting it goes - I can only rewrite things that I understand. > > So if you want this to change, you obviously need to understand the > entire distutils. It's possible to do that; some people have done > it (the "understanding" part) - just go ahead and start reading source > code. You (and Fredrik) are being a little harsh on Talin. I understand the need to encourage people to fix things themselves rather than just complaining about stuff they don't like. But people don't have an unlimited amount of time and expertise to work on several Python projects simultaneously. Nevertheless, they should be able to offer an "It would be good if..." suggestion without being stomped on. The suggestion itself can be a contribution if it focuses people's attention on a problem and a potential solution. Just because somebody can't learn a big subsystem and write code or docs for it *at this moment* doesn't mean they never will. And even if they don't, it's possible to make contributions in one area of Python and suggestions in another... or does the karma account not work that way? I don't see Talin saying, "You should fix this for me." He's saying, "I'd like this improved and I'm working on it, but it's a big job and I need help, ideally from someone with more expertise in distutils." Ultimately for Python the question isn't, "Does Talin want this done?" but, "Does this dovetail with the direction Python generally wants to go?" From what I've seen of setuptools/distutils evolution, yes, it's consistent with what many people want for Python. So instead of saying, "You (Talin) should take on this task alone because you want it" as if nobody else did, it would be better to say, "Thank you, Talin, for moving this important Python issue along." I've privately offered Talin some (unfinished) material I've been working on anyway that relates to his vision. When I get some other projects cleared away I'd like to put together that TOC of links I mentioned and perhaps collaborate on a Guide with whoever wants to. But I also need to learn more about setuptools before I can do that. As it happens I need the information anyway because I'm about to package an egg -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python and the Linux Standard Base (LSB)
On 11/28/06, Barry Warsaw <[EMAIL PROTECTED]> wrote: > For distros like Gentoo or Ubuntu that rely heavily on their > own system Python for the OS to work properly, I'm quite loathe to > install Cheeseshop packages into the system site-packages. I've had > Gentoo break occasionally when I did this for example (though I don't > remember the details now), so I always end up installing my own /usr/ > local/bin/python and installing my 3rd party packages into there. > Even though site-packages is last on sys.path, installing 3rd party > packages can still break the OS if the system itself installs > incompatible versions of such packages into its site-packages. One wishes distro vendors would install a separate copy of Python for their internal OS stuff so that broken-library or version issues wouldn't affect the system. That would be worth putting into the standard. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python and the Linux Standard Base (LSB)
On 11/29/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > The major advantage ~/.local has for *nix systems is the ability to have a > parallel *bin* directory, which provides the user one location to set their > $PATH to, so that installed scripts work as expected, rather than having to > edit a bunch of .foorc files to add to your environment with each additional > package. After all, what's the point of a per-user "install" if the > software isn't actually installed in any meaningful way, and you have to > manually edit your shell startup scripts, log out and log in again anyway? > Another nice feature there is that it uses a pre-existing layout convention > (bin lib share etc ...) rather than attempting to build a new one, so the > only thing that has to change about the package installation is the root. Putting programs and libraries in a hidden directory? Things the user intends to run or inspect? Putting a hidden directory on $PATH? I'm... stunned. It sounds like a very bad idea. Dotfiles are for a program's internal state: "black box" stuff. Not programs the user will run, and not Python modules he may want to inspect or subclass. ~/bin and ~/lib already work well with both Virtual Python and ./configure, and it's what many users are already doing. On the other hand, the freedesktop link says ~/.local can be overridden with environment variables. That may be an acceptable compromise between the two. Speaking of Virtual Python [1], I've heard some people recommending it as a general solution to the "this library breaks that other application" problem and "this app needs a different version of X library than that other app does". I've started using it off and on but haven't come to any general conclusion on it. Is it becoming pretty widespread among Python users. Would it be worth mentioning in the LSB/FHS? It only works on *nix systems currently, but Linux is a *nix system anyway. [1] http://peak.telecommunity.com/dist/virtual-python.py (It installs a pseudo copy of Python symlinked to the system one, so that you have your own site--packages directory independent of others ) -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Virtual Python (was Re: Python and the Linux Standard Base (LSB))
This may be a bit too FAQ-ish for some people but I'll post it in case anybody benefits from the answer. On 11/30/06, Phillip J. Eby <[EMAIL PROTECTED]> wrote: > At 02:46 PM 11/30/2006 -0800, Mike Orr wrote: > >Speaking of Virtual Python [1], I've heard some people recommending it > >as a general solution to the "this library breaks that other > >application" problem and "this app needs a different version of X > >library than that other app does". > > It was actually created to help people who don't have root access (e.g. in > shared web hosting accounts) get their "own" Python. It does work for > other things, but I haven't been especially recommending it for anything > besides that, since eggs take care of multi-version app/library support > quite nicely. > > Indeeed, the virtual-python approach is itself unnecessary now, as > easy_install has much better PYTHONPATH support now, than when > virtual-python was created. What does this last part mean? Virtual Python is still one of the four recommended ways to set up a custom install location in the EasyInstall docs. http://peak.telecommunity.com/DevCenter/EasyInstall#custom-installation-locations The other approaches work fine for giving each user a private install dir, but don't address the case of the same user wanting different install dirs for different projects. For instance, when exploring Pylons or TurboGears which install a lot of packages I may not otherwise want, I create a Virtual Python for each of them. If I'm developing an application under Virtual Python, I can see at a glance which packages my project needs installed. I can't think of any other way except Virtual Python to do this. Another point. Setuptools seems to have Two Ways To Do Things regarding package activiation. easy_install puts the latest-installed egg version in its .pth file so it's automatically available via a naive "import". This happens to clutters sys.path with a more entries than some people desire. Meanwhile, programs can use pkg_resources to activate a package or version that may not already be in sys.path. Is this the Way Of The Future? Should people start using pkg_resources for all packages they import? (That would also allow one to remove little-used packages from easy-install.pth.) Finally, one can use ~/.pydistutils.cfg to specify an install location, but that again allows only one location per user. And in the case of root, it can't distinguish between manually-installed packages and OS-installed packages, which the admin might want in different directories (/usr/local/lib/python vs /usr/lib/pythonVERSION). This hasn't been an issue because OSes haven't been using easy_install or eggs, but it will be when they start doing so. Do the PYTHONPATH improvements make it possible to just put a directory on your PYTHONPATH and have Python process .pth files in it without using the site.addsitedir() hack? That would probably be my biggest wishlist item. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Warning for 2.6 and greater
On 1/11/07, James Y Knight <[EMAIL PROTECTED]> wrote: > If the goal is really to have Py 3.0 be released later this year, 3.0 alpha is scheduled for this year. 3.0 final is not scheduled till next year, and of course another level of tweaks will have to be made after it's been in the Real World for a few months, so it won't fully stabilize until 3.1 or 3.2. That gives 1 1/2 - 2 years if you want to target "production-level" 3.1. Also, major parts of the 3.0 core have not been written yet, so it remains to be seen how realistic the schedule is (or how radical the changes will be if the schedule is kept). There will certainly be demand for an asynchronous server in 3.0, along with the large set of protocols support in Twisted. So either Twisted will be ported or a new package will be written, possibly by some of the Twisted developers, and very likely modelled after Twisted even if the literal code can't be copied. > Due to a lack of manpower, it is also not in all probability a > realistic option to maintain two parallel almost-identical > development branches, So the two projects will operate independently, and the 3.0 one may be smaller and less ambitious than Twisted. But if the need is there it will be written. > Just, please don't gratuitously remove an old > API to make Python a tiny bit cleaner, when there's not already > something I can reasonably do *today* to replace it. How did Perl 4 and Perl 5 handle the situation? I basically waited 2-3 years after Perl 5 came out, then started programming the new way. If it mattered (it didn't), I would have tied my applications specifically to Perl 4. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] The bytes type
On 1/12/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: > On 06:49 pm, [EMAIL PROTECTED] wrote: > > >I think we should draw a line in the sand and resolve not to garbage-up > Py2.6. > >The whole Py3.0 project is about eliminating cruft and being free of the > >bonds of backwards compatibility. Adding non-essential cruft to Py2.6 > >goes against that philosophy. > > Emotionally charged like "cruft" and "garbage" are obscuring the issue. > > Let's replace them with equivalents charged in the opposite direction: > > "I think we should draw a line in the sand and resolve not to > compatibility-up Py2.6. The whole Py3.0 project is about eliminating useful > libraries and being free of the bonds of working software. Adding > non-essential forward-compatibility to Py2.6 goes against that philosophy." Well put. Not that I agree. > The benefit (to me, and to many others) of 3.x over 2.x is the promise of > more future maintenance, not the lack of cruft. The benefit (to me, and to many others) of 3.x over 2.x is the promise of getting rid of cruft. If we're going to re-add cruft for the sake of temporary compatibility, we may as well just stick with 2.x. You have to take a quantum leap sometimes or you end up working around the same old mistakes. For instance, getting rid of the str/unicode dichotomy would be a wonderful thing. Then if I make a subclass of string I don't have to think "str or unicode? or two parallel classes?" It'll also eliminate the possibility of a stray 8-bit string containing non-ASCII characters from making your program go kablooie UnicodeDecodeError when it's in a "%" or ".join()", or at least localize the problem to ONE place (the string constructor) where it'll be in every programmer's face and they'll all hopefully learn to handle it consistently and not pass bad strings to other packages. Getting rid of old-style classes would be nice too. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Unipath package
I finally finished my path package (Unipath) and put it in the Cheeseshop.
http://sluggo.scrapping.cc/python/unipath/
There's a Path class for pathname calculations, and a FSPath subclass
for filesystem calls. I'm hoping Path -- or something resembling it
-- will find its way into os.path in Python 2.6 or 3.0. FSPath is
full of convenience methods so it may not be everybody's cup of tea,
but perhaps something similar can go into Python in the farther future
Unipath is an early alpha release so the API may change as it gets
more real-world use. There's an extensive unittest suite, which
passes on Python 2.5 and 2.4.4 on Linux. Windows and Macintosh
testers are needed.
Following are highlights from the python-3000 discussion and deviations from it:
- Path subclasses unicode, or str if the platform can't handle
Unicode pathnames.
- FSPath subclasses Path. This allows you to do "from unipath import
FSPath as Path" and pretend there's only one class. I find this much
more convenient in applications, so you don't have to cast objects
back and forth between the classes. Also, it just became infeasable
not to let them inherit, because so many methods call other methods.
- Nevertheless, you can use Path alone and rest assured it will never
touch the filesystem. You can even use Path objects with os.*
functions if you are so heretically inclined. If Path is accepted
into the stdlib, FSPath will inherit it from there.
- I tried splitting FSPath into several mixins according to type of
operation, but the need for methods to call other methods in a
different category sabotaged that too. So FSPath proudly has about
fifty methods. (Path has 10 public methods and 4 properties.)
- The dirname property is called .parent. The basename property is
.name. The extension property is .ext. The name without extension is
.stem.
- .components() returns a list of directory components. The first
component is "/", a Windows drive root, a UNC share, or "" for a
relative path. .split_root() returns the root and the rest.
- PosixPath, NTPath, and MacPath are Path subclasses using a specific
path library. This allows you to express non-native paths and convert
paths. Passing a relative foreign path to a *Path constructor
converts it to the destination type. Passing an absolute foreign path
is an error, because there's no sane way to interpret "C:\\" on Posix
or "/" on Windows. I'm skeptical whether this non-native support is
really worth it, because .norm() and .norm_case() already convert
slashes to backslashes (which is what Talin really wanted to do --
have Posix paths in a config file and automatically convert them to
the native format). And if Python is going to drop Mac OS 9 soon then
MacPath is obsolete. So maybe this non-native path code will prove
less than useful and will be deleted. On the other hand, if someone
is burning to write a zippath or ftppath library, you can use it with
this.
- Setting Path.auto_norm to true will automatically normalize all
paths on construction. I'm not sure if this should be the default,
because normalizing may produce the wrong path (e.g., if it contains a
symlink). You can always pass norm=True or norm=False to the
constructor to enable/disable it.
- p.child("subdir", "grandkid") is Glyph's favorite "safe join" method
that prevents creating a path pointing above 'p'. You can use it as a
.joinpath if you don't mind this restriction. The constructor allows
multiple positional arguments, which are joined using os.path.join().
- Listing is p.listdir(pattern=None, filter=None, names_only=False).
This returns a non-recursive list of paths. If 'names_only' is true
it returns the same as os.listdir(). p.walk(pattern=None,
filter=None, top_down=True) is the recursive counterpart, yielding
paths. In both cases 'pattern' is a glob pattern, and 'filter' is one
of the constants (FILES, DIRS, LINKS, FILES_NO_LINKS, DIRS_NO_LINKS,
DEAD_LINKS) or a custom boolean function. I spent a lot of time going
back and forth between Orendorff's six listing methods and Raphael's
one, and finally decided on this, partly because I'm not satisfied
with how either Orendorff's class or Raphael's or os.walk() handle
symlinks -- sometimes you want to ignore the links and then iterate
them separately.
- .read_file(mode) and .write_file(content, mode) are a compromise
between Orendorff's seven methods and purists' desire for zero
methods.
- .mkdir(), .rmdir(), .rmtree(), .copy(), .copy_stat(), .copy_tree(),
and .move() are more fancy than their stdlib/Orendorff counterparts.
They silently succeed if the operation is already done, and have
arguments to smartly create/delete intermediate directories, etc.
- Two extra functions are in 'unipath.tools'. 'dict2d
Re: [Python-Dev] Unipath package
On 1/28/07, Tobias Ivarsson <[EMAIL PROTECTED]> wrote: > I am a MSc of Computer Engineering student from Sweden. A frequent reader of > python-3000, and an occasional reader of python-dev for about a year now. I > have wanted to get involved in the development of python for a while but > haven't found the time before. Now, when I'm in the process of trying to > figure out what project to do for my thesis, I have actually found myself > having more time than before. Therefore I thought that now is a better time > than never to get involved. After that compulsory first-post-introduction of > myself, lets get to the reason I choose to send the post at all. Hi Tobias! You've got the first step in Python development -- reading python-dev. Now you just have to pick an interesting task from the dozens hinted on the list and do it. Of course, there are many tasks not on python-dev that are also worthwhile. Bugfixes in the Python core, adopting a package that lost its maintainer, being active in the SIGs, testing code (as you've just done) -- these are just a few things off the top of my head. Working on third-party libraries is just as legitimate as working on the core; both are needed. There are also non-technical needs like promoting/marketing Python. If you can make it to PyCon or EuroPython, you'll meet a lot of contacts you can potentially collaborate with. Laura Creighton lives in Sweden and is highly involved in Python activities; she would no doubt have some suggestions. If you can't locate her I can dig up her address. If you'd like to write me privately about your specific interests and talents in Python, I can try to steer you to a group that is working on that. > I noticed that you wanted Mac OS tests, here are the test results from my > Mac OS X (10.4), running on an Intel Core Duo MacBook, using python 2.5: Thanks for the results. The details of Unipath's code are outside the scope of python-dev so I'll respond privately. The issue here is what kind of path object would be suitable for the standard library in 2.x. Guido has said he's not convinced that any OO-based path class is necessarily superior to the existing functions, but left the door open to later consideration. The existing proposals were rejected for reasons the Guido didn't personally elaborate, but others channeling him speculated they were too monolithic: too many methods in one class, too many diverse kinds of methods in one class, and especially mixing pathname-calculation and filesystem-access methods undifferentiated in one class. There is general agreement that a pathname-calculation class is more needed than a filesystem-access class due to the nesting of pathname calculations in expressions, so a pathname-calculation class has a greater chance of acceptance. There's also the question of what can be put in Python 2.x vs 3.0. There's a parallel proposal to move the filesystem-access functions in os.path into os, and to move everything in shlib into os, possibly with changes. This has the greatest chance of acceptance in 2.x. A small os.path.Path class has a middling chance. The discussion has been hampered by the lack of released code. Only Orendorff's class has been widely used in production systems. The others have either never been used or only by their authors; they haven't made it to the Cheeseshop. Unipath is merely to say "Here's another way to do it; see if it works for you." Certainly the Path methods need more testing and use in the real world before they'd be ready for the stdlib. The FSPath methods are more experimental so I'd say they need a year of use before they can be considered sufficiently stable. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unipath package
On 1/28/07, Steven Bethard <[EMAIL PROTECTED]> wrote: > On 1/28/07, Michael Foord <[EMAIL PROTECTED]> wrote: > > Mike Orr wrote: > > > I finally finished my path package (Unipath) and put it in the Cheeseshop. > > > http://sluggo.scrapping.cc/python/unipath/ > > > > > Hello Mike, > > > > Looking through the docs it looks like a great package. However, didn't > > Guido (on this list anyway) rule that he wouldn't accept a solution > > which subclassed a string type? > > To be easily useful as a third-party module, it needs to subclass > str/unicode so you can pass it directly to something like open(). > That inheritance could be dropped if the module was included in the > stdlib, as long as the inclusion was also accompanied by appropriate > modifications to open()/file() to allow Path objects. Character slicing and some other string methods are also immensely useful. Subclassing string allows you to get all of them at once, rather than leaving out the one that was not foreseen to be needed. I was formerly a fan of the tuple-component approach but somebody said it splits the path too early, makes too many platform-specific assumptions or something like that. In practice, you really do want a path compatible with string so you can pass it to any function (stdlib or third-party) that accepts a string path, without getting into cases where you have to cast it (str()). There's also the string/unicode problem in Python 2.x: if you call str() on a string containing non-ASCII characters, you get an exception, and it's not always possible to guarantee the string you got from some other code is safe. If the BDFL does not accept a string-based path class, it will still have a long happy life as a third-party library. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Unipath package
On 1/28/07, Larry Hastings <[EMAIL PROTECTED]> wrote:
> I dropped the Cc: of Python-3000, because I don't think this discussion
> falls under that mailing list's charter. As I understand it, the
> Python-3000 mailing list is for discussing the details of implementing
> Python 3000. "Stuff I'd like to see in Python 3000" doesn't go there, it
> goes to "Python-Ideas".
python-ideas didn't exist when this thread was started. :) I was just
giving a status report on existing issues we'd discussed on
python-3000. You're right that any future work in this area will
probably involve python-ideas. I'm waiting for Unipath to get some
use before re-raising the question of what direction we want the
stdlib to go. I also support the other proposal to move some
functions from os.path/shutil to os, which is being worked on.
> One part of your message caught my eye:
>
> Passing an absolute foreign path is an error, because there's no sane way
> to interpret "C:\\" on Posix or "/" on Windows.
> There is in fact a very sane way to interpret "/" on Windows: the root
> directory of the "current" drive. It's equivalent to "\".
That's one way to do it, but whether Unipath should presume this is
what the programmer wants is another issue. One can make an argument
either way. I found it appealing to categorically refuse to translate
absolute paths to a foreign system, and force the user to make the
path relative and explicitly attach it to a new root (or
subdirectory). "Explicit is better than implicit", as the Python
mantra goes. If this turns out to be a hassle for programmers, we can
always be more lenient later.
> I guess it isn't
> widely known, but nearly all Windows APIs are agnostic about whether you use
> "\" or "/" as a directory separator. (The only exceptions I recall are the
> common dialog open / save file functions.) I'm a Windows programmer, and I
> frequently use "/".
I've heard contradictory things about this. Some people say just use
"/" and be happy; others say you get into trouble sometimes because
it's not the Windows kernel that does this but the application or
function, so some applications support it and some don't. In
particular I've heard that "/" on the command line is handled by the
application, which may impose the DOS semantics ("option flag") rather
than "directory separator".
> The fact that you don't understand this about how Windows paths work makes
> me nervous about your path library. For instance, did you correctly support
> local paths on explicit drives (e.g. "d:../foo")?
I did use Windows regularly until 1998. I do remember drive-relative
paths. My question is, do people seriously use them anymore? In a
Python program? Specifying a drive-relative path or chdir'ing to it
should work. The one thing Unipath does is consider it absolute:
isabsolute() returns True, and .components() and .split_root() return
"d:" as the root of the filesystem -- which it is. Joining the
components produces the original path, which Windows should correctly
interpret. So I don't see the harm as treating it as absolute in my
model. Maybe it does create some hardship for Windows users, and
maybe I'll have to consider. But drive-relative paths are really an
intermediate level between absolute and relative, and the model has no
place for the relative drive specification. E.g.,
Path("../foo").components() => ["", "..", "foo"]
Path("C:\\foo").components() => ["C:\\", "foo"]
Path("C:../foo").components() => ["C:", "..", "foo"]
Strictly speaking, the last path is relative so the first element
should be "". But that would leave no place for the "C:" specifier,
which would either belongs at index 0.5 or in a special attribute.
Accommodating either would require changing the entire model to
accommodate an obscure Windows-only feature, even though the model
works quite well otherwise.
It may be that Unipath just isn't suitable for manipulating Windows
drive-relative paths, and making it do so would would mess up Unipath
too much. The former is for Windows users to decide; the latter is
for me to decide based on their input.
I have also dropped .drive, .unc, .splitdrive(), and .splitunc(). I
think .split_root() is robust enough for what people want to do. But
I may be wrong. If there's necessity for those methods/properties
I'll add them, but I don't want to add a bunch of legacy methods that
will rarely be called.
--
Mike Orr <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-3000] Unipath package
On 1/28/07, Mike Orr <[EMAIL PROTECTED]> wrote:
> > Passing an absolute foreign path is an error, because there's no sane way
> > to interpret "C:\\" on Posix or "/" on Windows.
> > There is in fact a very sane way to interpret "/" on Windows: the root
> > directory of the "current" drive. It's equivalent to "\".
>
> That's one way to do it, but whether Unipath should presume this is
> what the programmer wants is another issue. One can make an argument
> either way.
The issue is that *maybe* the programmer wants to copy "/etc/mailcap"
on his Posix filesystem to "\etc\mailcap" on his Windows drive, but
should we assume this since \etc does not have the special status on
Windows that /etc does on Unix? Maybe we should raise an exception
because the programmer intended to copy it somewhere else and forgot
something. Nothing wrong with forcing him to say, "Yes, I really mean
'\'."
As for Posix-format paths in config files that should be translated to
the native NT, that sounds straightforward. These paths should be
relative to some platform-specific root, because "C:/etc/mailcap" is
certainly not the path you intend to use on all platforms. (Unix
would put it in the current directory, wherever that might be.) You
can create a Path("/foo/bar") on NT and use it. The only thing you
can't do is:
p = Path( PosixPath("/foo/bar") ) # Illegal for absolute paths
if Path != PosixPath.
--
Mike Orr <[EMAIL PROTECTED]>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Unipath package
Thanks everyone who sent Windows and Mac unittest reports. I've got enough of them now. 95% of the Windows errors are because I forgot to remove symlinks from the control values. The repr() problem is more mysterious: somehow one path is turning into unicode while the other isn't. The chmod error probably just needs some bits excluded from the comparision. The Mac errors have to do with the general temp directory being a symlink, and some paths being resolved while others aren't. I'll have to take that up with my Mac-loving colleagues. 0.1.1 will be in the Cheeseshop soon with fixed tests. -- Mike Orr <[EMAIL PROTECTED]> ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] More Path comments (PEP 355)
I just read over the changes to the proposed Path class since the discussion last summer. A big thanks to Bjorn Lindqvist for writing a PEP, Jason Orendorff for the original path.py and his suggestions on how the Path class should be different, and the writers of the Python-Dev Summary for bringing the discussion to my attention. I've been testing/using the interim Path class in the Python subversion (/sandbox/trunk/path, last modified in September), and have a few comments about PEP 355: - .walk*() return a list rather than an iterator. Was this an intentional change or a typo? Most typical uses yield thousands of paths which do not need to be in memory simultaneously. - An equivalent to os.listdir() is frequently useful in applications. This would return a list of filenames (strings) without the parent info. Path.listdir() calls os.listdir() and wraps all the items into Paths, and then I have to unwrap them again, which seems like a waste. I end up calling os.listdir(my_path) instead. If we decide not to subsume many os.* functions into Path, that's fine, but if we deprecate os.listdir(), it's not. - -1 on removing .joinpath(), whatever it's called. Path(basepath, *args) is good but not the same. (1) it's less intuitive: I expect this to be a method on a directory. (2) the class name is hardcoded: do I really have to do self.__class__(self, *args) to make my code forward compatible with whatever nifty subclasses might appear? - +1 on renaming .directory back to .parent. - -1 on losing a 1-liner to read/iterate a file's contents. This is a frequent operation, and having to write a 2-liner or a custom function is a pain. - +1 on consolidating mkdir/makedirs and rmdir/rmdirs. I'd also suggest not raising an error if the operation is already done, and a .purge() method that deletes recursively no matter what it is. This was suggested last summer as a rename for my .delete_dammit() proposal. Unsure what to do if permission errors prevent the operation; I guess propagating the exception is best. This would make .rmtree() redundant, which chokes if the item is a file. - +1 for rationalizing .copy*(). - +1 for .chdir(). This is a frequent operation, and it makes no sense not to include it. -- Mike Orr <[EMAIL PROTECTED]> ([EMAIL PROTECTED] address is semi-reliable) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Path.ancestor()
This is a potentially long discussion so I'm putting it in a separate thread. When finding one file relative to another, it's difficult to read multiple ".parent" attributes stacked together. Worse, if you have the wrong number, you end up at the wrong directory level, potentially causing destructive damage. Doubly worse, the number of ".parent" is non-intuitive for those used to the near-universal "." and ".." conventions. Say I'm in apps/myapp/bin/myprogram.py and want to add apps/myapp/lib and apps/shared/lib to sys.path in a portable way. app_root = Path(__file__).realpath().abspath().parent.parent assert app_root.parent.name == 'apps' sys.path.insert(0, app_root.parent / 'shared/lib') sys.path.insert(0, app_root / 'lib') Yikes! At least it's better than: lib = os.path.join(os.path.dirname(os.path.dirname(x)), "lib") which is completely unreadable. (Silence to those who say __path__ is obsolete now that setuptools has a function for finding a file in an egg. (1) I don't understand that part of the setuptools docs. (2) It will be many months before most Python programmers are ready to switch to it.) The tricky thing with "." and ".." is they have a different meaning depending on whether the original path is a file or directory. With a directory there's one less ".parent". I've played a bit with the argument and come up with this: # N is number of ".."; None (default arg) is special case for ".". .ancestor() => "." => p.parent or d .ancestor(0) => ValueError .ancestor(1) => ".." => p.parent.parent or d.parent .ancestor(2) => "../.." => p.parent.parent.parent or d.parent.parent The simplest alternative is making N the number of ".parent". This has some merit, and would solve the original problem of too many ".parent" stacking up. But it means Path wouldn't have any equivalent to "." and ".." behavior. Another alternative is to make .ancestor(0) mean ".". I don't like this because "." is a special case, and this should be shown in the syntax. Another alternative is to move every number down by 1, so .ancestor(0) is equivalent to "..". The tidiness of this is outweighed by the difficulty of remembering that N is not the number of "..". -- Mike Orr <[EMAIL PROTECTED]> ([EMAIL PROTECTED] address is semi-reliable) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Alternative path suggestion
reduced. The most obvious
> example are the copy functions. In the current proposal:
>
> {{{
> def copyfile(self, dst): ...
> def copymode(self, dst): ...
> def copystat(self, dst): ...
> def copy(self, dst): ...
> def copy2(self, dst): ...
> }}}
>
> In my proposal:
>
> {{{
> def copy(self, dst, copystat=False): ...
> }}}
>
> It's just that I think that copyfile, copymode and copystat aren't
> usually useful, and there's no reason not to unite copy and copy2.
Sounds good.
> = Other Changes =
>
> Here is a list of the smaller things I've changed in my proposal.
>
> The current normpath removes '..' with the name before them. I didn't
> do that, because it doesn't return an equivalent path if the path
> before the '..' is a symbolic link.
I was wondering what the fallout would be of normalizing "a/../b" and
"a/./b" and "a//b", but it sounds like you're thinking about it.
> I removed the methods associated with file extensions. I don't recall
> using them, and since they're purely textual and not OS-dependent, I
> think that you can always do p[-1].rsplit('.', 1).
No, .ext and .namebase are important! I use them all the time; e.g.,
to force the extension to lowercase, to insert/update a suffix to the
basename and re-add the extension, etc. I can see cutting the other
properties but not these.
.namebase is an obnoxious name though. I wish we could come up with
something better.
> I removed unlink. It's an alias to remove, as far as I know.
It was added because .unlink() is cryptic to non-Unix users. Although
I'd argue that .delete() is more universally understood than either.
> I removed expand. There's no need to use normpath, so it's equivalent
> to .expanduser().expandvars(), and I think that the explicit form is
> better.
Expand is useful though, so you don't forget one or the other.
> copytree - I removed it. In shutil it's documented as being mostly a
> demonstration, and I'm not sure if it's really useful.
Er, not sure I've used it, but it seems useful. Why force people to
reinvent the wheel with their own recursive loops that they may get
wrong?
> symlink - Instead of a function like copy, with the destination as the
> second (actually, the only) argument, I wrote "writelink", which gets
> a string and creates a symbolic link with that value. The reason is
> that symbolic links can be any string, not necessarily a legal path.
Does that mean you have to build a relative link yourself if you're
going from one directory to another?
> p.ext -> ''.join(p[-1].rsplit('.', 1)[1:])
People shouldn't have to do this.
> Unicode - I have no idea about unicode paths. My current
> implementation simply uses str. This should be changed, I guess.
No idea about that either.
You've got two issues here. One is to go to a tuple base and replace
several properties with slicing. The other is all your other proposed
changes. Ideally the PEP would be written in a way that these other
changes can be propagated back and forth between the PEPs as consensus
builds.
--
Mike Orr <[EMAIL PROTECTED]>
([EMAIL PROTECTED] address is semi-reliable)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Alternative path suggestion
I've updated the wiki with a second proposal based on this thread, and also summarized the Python-dev discussions. Please make sure your favorite feature or pet peeve is adequately represented. http://wiki.python.org/moin/AlternativePathClass -- Mike Orr <[EMAIL PROTECTED]> ([EMAIL PROTECTED] address is semi-reliable) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Alternative path suggestion
On 5/4/06, Nick Coghlan <[EMAIL PROTECTED]> wrote:
> Mike Orr wrote:
> >> == a tuple instead of a string ==
> >>
> >> The biggest conceptual change is that my path object is a subclass of
> >> ''tuple'', not a subclass of str.
>
> Why subclass anything? The path should internally represent the filesystem
> path as a list or tuple, but it shouldn't *be* a tuple.
Good point.
> Also, was it a deliberate design decision to make the path objects immutable,
> or was that simply copied from the fact that strings are immutable?
>
> Given that immutability allows the string representation to be calculated once
> and then cached, I'll go with that interpretation.
Immutability also allows them to be dictionary keys. This is
convenient in many applications.
> Similarly, I would separate out the extension to a distinct attribute, as it
> too uses a different separator from the normal path elements ('.' most places,
> but '/' on RISC OS, for example)
That would be too surprising to users. p[-1] should be the full
filename. We can use p.name, p.extsep, and p.ext for the parts.
> Alternatively, if the path elements are stored in separate attributes, there's
> nothing stopping the main object from inheriting from str or unicode the way
> the PEP 355 path object does.
Do the string methods call .__str__(), or how do they get the string value?
> I wouldn't expose stat() - as you say, it's a Unixism. Instead, I'd provide a
> subclass of Path that used lstat instead of stat for symbolic links.
>
> So if I want symbolic links followed, I use the normal Path class. This class
> just generally treat symbolic links as if they were the file pointed to
> (except for the whole not recursing into symlinked subdirectories thing).
>
> The SymbolicPath subclass would treat normal files as usual, but *wouldn't*
> follow symbolic links when stat'ting files (instead, it would stat the
> symlink).
Normally I'd be processing the subdirectories, then the symlinks, off
the same path. I'd rather just call a different method rather than
having to construct another object.
> >> == One Method for Finding Files ==
> >>
> >> (They're actually two, but with exactly the same interface). The
> >> original path object has these methods for finding files:
> >>
> >> {{{
> >> def listdir(self, pattern = None): ...
> >> def dirs(self, pattern = None): ...
> >> def files(self, pattern = None): ...
> >> def walk(self, pattern = None): ...
> >> def walkdirs(self, pattern = None): ...
> >> def walkfiles(self, pattern = None): ...
> >> def glob(self, pattern):
> >> }}}
> >>
> >> I suggest one method that replaces all those:
> >> {{{
> >> def glob(self, pattern='*', topdown=True, onlydirs=False,
> >> onlyfiles=False): ...
> >> }}}
>
> Swiss army methods are even more evil than wide APIs.
I don't see the point in combining these either. There's the problem
of walking special files, but these are so rare and diverse that we
either have to provide methods for all of them, or an argument, or
force the user to use .walk(). Since they are rare, the latter is OK.
> Now, what might potentially be genuinely useful is paired walk methods that
> allowed the following:
>
># Do path.walk over this directory, and also return the corresponding
># information for a destination directory (so the dest dir information
># probably *won't* match that file system
>for src_info, dest_info in src_path.pairedwalk(dest_path):
>src_dirpath, src_subdirs, src_files = src_info
>dest_dirpath, dest_subdirs, dest_files = dest_info
> # Do something useful
>
># Ditto for path.walkdirs
>for src_dirpath, dest_dirpath in src_path.pairedwalkdirs(dest_path):
># Do something useful
>
># Ditto for path.walkfiles
>for src_path, dest_path in src_path.pairedwalkfiles(dest_path):
>src_path.copy_to(dest_path)
These look like os.walk() derivatives. I've never found that as
useful as getting a flat iteration of paths. But if enough people
want it...
--
Mike Orr <[EMAIL PROTECTED]>
([EMAIL PROTECTED] address is semi-reliable)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Alternative path suggestion
On 5/4/06, Paul Moore <[EMAIL PROTECTED]> wrote: > On 5/4/06, Nick Coghlan <[EMAIL PROTECTED]> wrote: > > My inclination was to have a PlatformPath subclass that accepted 'os', 'sep' > > and 'extsep' keyword arguments to the constructor, and provided the > > appropriate 'sep' and 'extsep' attributes (supplying 'os' would just be a > > shortcut to avoid specifying the separators explicitly). > > > > That way the main class can avoid being complicated by the relatively rare > > need to operate on another platform's paths, while still supporting the > > ability. > > You ought to have predefined classes for the standard OSes. Expecting > people to know the values for sep and extsep seems unhelpful. > > In fact, unless you expect to support the os.path interface forever, > as well as the new interface, I'd assume there would have to be > internal WindowsPath and PosixPath classes anyway - much like the > current ntpath and posixpath modules. So keeping that structure, and > simply having > > if os.name == 'posix': > Path = PosixPath > elif os.name == 'nt': > Path = WindowsPath > ... etc > > at the end, would seem simplest. Why not just put a platform-specific Path class inside posixpath, macpath, etc. Python already chooses os.path for the actual platform, and you can import a foreign module directly if you want to. > (But all the current proposals seem to build on os.path, so maybe I > should assume otherwise, that os.path will remain indefinitely...) They build on os.path because that's what we're familiar with using. There's no reason to write the platform-specific classes until we agree on an API; that would just be work down the drain. When the new classes are in the library, we can: (one or more) - Leave os.path.foo() alone because it works and most existing programs need it. - Discourage os.path.foo() in the documentation but continue to support it. - Rewrite os.path.foo() to use Path.foo(). A lot of useless work if we... - Remove os.path.foo() in Python 3.0. -- Mike Orr <[EMAIL PROTECTED]> ([EMAIL PROTECTED] address is semi-reliable) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Alternative path suggestion
On 5/5/06, Nick Coghlan <[EMAIL PROTECTED]> wrote:
> Mike Orr wrote:
> > On 5/4/06, Paul Moore <[EMAIL PROTECTED]> wrote:
> >> (But all the current proposals seem to build on os.path, so maybe I
> >> should assume otherwise, that os.path will remain indefinitely...)
> >
> > They build on os.path because that's what we're familiar with using.
> > There's no reason to write the platform-specific classes until we
> > agree on an API; that would just be work down the drain. When the new
> > classes are in the library, we can: (one or more)
> >
> > - Leave os.path.foo() alone because it works and most existing programs
> > need it.
>
> The threading module doesn't really obsolete the thread module, it just
> provides a higher level, more convenient API.
>
> Similarly, I don't believe it's a given that a nice path object will obsolete
> the low level operations. When translating a shell script to Python (or vice
> versa), having access to the comparable low level operations would be of
> benefit.
I think you meant to say Perl in that sentence. In Python there
should be one way to do it, and beautiful is better than ugly. The
os.path functions are bad enough, but shutil.copy2 is just plain evil.
Is it that much of a step to translate:
Y="$(dirname $(dirname $X))/lib"
as
y = Path(x).parent.parent + "lib"
y = Path(x).parent.parent.join("lib")
or whatever the syntax do jour is
rather than
y = os.path.join(os.path.dirname(os.path.dirname(x)), "lib")
?
--
Mike Orr <[EMAIL PROTECTED]>
([EMAIL PROTECTED] address is semi-reliable)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Alternative path suggestion
There's something that bothers me about putting the path in an
attribute rather than subclassing tuple. I prefer it that way but I
don't see how you'd do directory slicing and joining. If the path is
a tuple it's easy:
p = Path("/a/b/c")
p[:-1] # Path("/a/b")
If the directory components are on an attribute
p.path[:-1] # ("a", "b")
How do you do slicing and joining? If Path subclasses object, it
could be done there like in the first example. But if Path subclasses
string, that API is taken:
p[:-1]# "/a/b"
--
Mike Orr <[EMAIL PROTECTED]>
([EMAIL PROTECTED] address is semi-reliable)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Alternative path suggestion
On 5/6/06, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Greg Ewing wrote: > > Nick Coghlan wrote: > > > >> So I suggest splitting the internal data into 'path elements separated > >> by os.sep', 'name elements separated by os.extsep' > > > > What bothers me about that is that in many systems > > there isn't any formal notion of an "extension", > > just a convention used by some applications. > > > > Just because I have a "." in my filename doesn't > > necessarily mean I intend what follows to be > > treated as an extension. Good points, Nick and Greg. Splitting directories is unambiguous and makes the API much simpler. But filename extensions are just a convention. The user may want all, some, or no extensions split. With os.splitext() the user is explicitly saying, "Split this number of extensions on this particular file." We we should not presume to automate this. How about: .splitext(max_exts=1)=> (filename, list_of_extensions) Another good idea yesterday: p.parent(2) # Fulfills my .ancestor() proposal. > This is an interesting point - I've been tinkering with this a bit (despite my > best intentions about not getting distracted by the topic ;), and it's > starting to look like treating directory paths and file paths identically > poses significant problems for an object-oriented API. > > For example, appending a filename or a relative directory path to an existing > directory path is fine (giving a filename or directory path as the result), > but you can't do either to a filename. > > Listing the entries or walking a file doesn't make any sense either. > > And extension information is typically significant only for filenames, not for > directories. > > When paths are just strings, the distinction doesn't matter because the > instances have no inherent behaviour - it is the application that carries the > state regarding whether something is meant to be a directory name or a file > name. > > So it makes a lot more sense to have separate Dirpath and Filepath (and > possibly Linkpath) objects - from an OO point of view, the methods provided > for each of these kinds of filesystem entities will be significantly > different. A common parent Path object may still make sense if there's > sufficient overlap, though (e.g. retrieving status related information). Very interesting. The user knows whether he wants a file or directory, so why not force him to specify? Then we can have a standard exception if the actual filesystem object is inconsistent with the spec. -- Mike Orr <[EMAIL PROTECTED]> ([EMAIL PROTECTED] address is semi-reliable) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3102: Keyword-only arguments
On 5/7/06, Steven Bethard <[EMAIL PROTECTED]> wrote: > The most obvious one to me is the optparse module, where add_option > takes all kinds of different keyword arguments, and there's really no > intention of these ever being specified as positional arguments: > http://docs.python.org/lib/module-optparse.html Or MySQLdb, which specifically recommends keyword arguments for the constructor. Mainly because the underlying library is controlled by a third party, so it's unpredictable when and where new arguments will be added. (Not that I'm in favor of keyword-only arguments, for the reasons Terry Reedy has mentioned.) -- Mike Orr <[EMAIL PROTECTED]> ([EMAIL PROTECTED] address is semi-reliable) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
