Re: [Python-Dev] Update to Python Documentation Website Request
David Lyon wrote: > There's an awful lot to take in, and there must be 20,000 lines of > emails for every 1 line of python code that is required to fix this > thing. Yep, which goes way back to one of my first emails in this thread: compared to the social aspects, the technical aspects of packaging tools are relatively straightforward :) It's great to see some energy directed towards solving these issues, but I think it's important for those trying to contribute to realise what they're dealing with. I just hope I don't kill their enthusiasm in the process :P Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update to Python Documentation Website Request
On Wed, Jul 29, 2009 at 2:36 AM, David Lyon wrote: > On Wed, 29 Jul 2009 10:56:20 +1000, Nick Coghlan > wrote: >> The words "eggs" brings with it a whole lot more baggage than just the >> sum of the technical parts in the language core that support them >> (primarily distutils and zipimport). > > Well, in this case, (talking metaphorically) we have the ability > to roll the eggs, crack a whole in them and suck out the contents > (deal with a package in a zipped egg) > > So metaphorically we could say that python can work with egg shells.. > > As for what's in the egg... well lets just say that it's a bit floaty.. > > And perphaps that is the best way for things to be for a while. > >> I find it unfortunate that the name >> for the distutils metadata format contains the word "egg" because it >> implies much greater consensus around the philosophy behind eggs than >> actually exists. > > I see it a different way. I thinks eggs are simple - as in like a java > bean. A simple way to move a package from one place to another. > > What seems to have gone wrong is that there is a lot of pioneering > and interconnected and interdependent technology within them. They're > an egg.. but in the past they've had little monsters that bite your > fingers when you try to put them in the pan... :-) > > I think "Egg" term is very good. But maybe we are best not trying > to over-specify their contents. > > Maybe we should say it has certain things (EGG_INFO, PACKAGE DATA) > as in (YOLK, WHITE) and pretty much leave it at that. If more detail > is required - rtm. > >> A lot of the baggage associated with the "eggs" concept is related to >> the inherent conflict between different approaches to dependency >> management: > > That's not an egg problem. That's a packaging/metadata problem. > > It's the package dependency issue that's the problem. That's a distutils > problem. > >> 1. Use the system package management system for everything (preferred by >> many, perhaps even most, *nix sysadmins, but not an option on Windows) > > Yes, because the package dependency issues are just so great. > >> 2. Create a Python specific package management system independent of the >> system package manager (an area dominated by setuptools, including both >> eggs and non-zipped package distributions) > > More work needs to go into distutils. Not taking away from any existing > work - but setuptools has relied on the deficiencies of distutils to > gain a foothold. > > Fix distutils (give me time to think..) and the need for setuptools and > any further fork is probably negated. > >> 3. Bundle everything into a monolithic application or framework (the >> typical approach on Windows with py2exe, but also the philosophy behind >> tools like virtualenv) > > Windows users are using py2exe and other products over distutils. To a > normal windows developer, distutils makes imho no sense in the way that > it goes about things now. > > For example, very simple concepts like "program directories", (where > you stick the program) isn't an available option in distutils. But > it is the first thing that you need to know in a windows program. > > So it's very hard to get to step 1... > >> Your comments about your >> package management system suggest that it is just yet another entrant in >> category 2 and you have said nothing to allay the concerns of those that >> despise that philosophy with a passion because of the problems it causes >> them when trying to manage their systems using the first philosophy. > > I'm a windows user.. I can't be in category #2.. > > I'm in category #1, have nothing... (except now my own tool) > >> Since the Python constituency includes developers and system >> administrators that favour all 3 philosophies (and probably other >> variants that I haven't thought to describe), anything that makes it >> into the standard library will need to adequately balance the interests >> of all parties (e.g. as has occurred in the PEP 376 metadata enhancement >> discussions). > > Well at least you are saying that there is some way of reconciling > all the different options... > > There's an awful lot to take in, and there must be 20,000 lines of > emails for every 1 line of python code that is required to fix this > thing. > > Take care > > David > I really do think this mail thread needs to move to disutils-sig or python-ideas. jesse ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Update to Python Documentation Website Request
Jesse Noller gmail.com> writes: > > I really do think this mail thread needs to move to disutils-sig or > python-ideas. +1 ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded
On Sun, Jul 26, 2009 at 11:01 AM, Joshua Haberman wrote:
> I'm writing a C Python extension that needs to generate PyTypeObjects
> dynamically. Unfortunately, the Py_TPFLAGS_HEAPTYPE flag is overloaded
> in a way that makes it difficult to achieve this goal.
>
> The documentation for Pt_TPFLAGS_HEAPTYPE says:
>
> Py_TPFLAGS_HEAPTYPE
>
> This bit is set when the type object itself is allocated
> on the heap. In this case, the ob_type field of its
> instances is considered a reference to the type, and the
> type object is INCREF’ed when a new instance is created,
> and DECREF’ed when an instance is destroyed (this does not
> apply to instances of subtypes; only the type referenced
> by the instance’s ob_type gets INCREF’ed or DECREF’ed).
>
> This sounds like exactly what I want. I want my type object INCREF'd
> and DECREF'd by its instances so it doesn't leak or get deleted
> prematurely. If this were all that Py_TPFLAGS_HEAPTYPE did, it would
> work great for me.
>
> Unfortunately, Py_TPFLAGS_HEAPTYPE is also overloaded to mean
> "user-defined type" (as opposed to a built-in type). It controls
> numerous subtle behaviors such as:
>
> - whether the type's name is module.type or just type.
> - whether you're allowed to set __name__, __module__, or __bases__ on the
> type.
> - whether you're allowed to set __class__ on instances of this type.
> - whether the module name comes from the type name or the __module__
> attribute.
> - whether it will use type->tp_doc as the docstring
> - whether its repr() calls it a "class" or a "type".
> - whether you can set attributes of the type.
> - whether someone is attempting the Carlo Verre hack.
>
> So I'm stuck with an unenviable choice. I think the lesser of two evils
> is to *not* specify Py_TPFLAGS_HEAPTYPE, because the worst that will
> happen is that my types will leak. This is not as bad as having someone
> set __class__ on one of my instances, or set attributes on my type, etc.
>
> Ideally the interpreter would have a separate flag like
> Py_TPFLAGS_BUILTIN that would trigger all of the above behaviors, but
> still make it possible to have dynamically generated built-in types get
> garbage collected appropriately.
>
> At the very least, the documentation I cited above should make it clear
> that Py_TPFLAGS_HEAPTYPE controls more than just whether the type gets
> INCREF'd and DECREF'd. Based on the list of behaviors I discovered
> above, it is almost certainly not correct for a C exension type to be
> declared with Py_TPFLAGS_HEAPTYPE.
>
> Josh
Hi Joshua, recently I also needed to dynamically make subtypes from C,
I tried 2 ways of doing this, one is to do the C equivalent of calling
type("name",(bases,...), dict) and the other is to malloc()
PyTypeObject's, fill in the slots and run PyType_Ready on them to
initialize them.
It seems the first is the expected way to make your own types so I
assume thats what your doing?, Just wondering because if you do it the
second way I think youll have more control and the types will be more
limited (like internal types).
I'm not expert enough in this area to know if malloc'ing PyTypeObject
and initializing has some other problems.
- Campbell
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing File Modes
My motivation came from an instance when I was using subprocess.Popen for a
Linux / Windows cross platform program. In part of the program, I was
writing and reading to a cron like object. On Windows, it was a text file
and on Linux it would be the crontab executable. Had I been able to
substitute the "open()" function with my wrapper, it would have been the
only change I had to make for cross platform compatibility; instead of
having to change numerous lines because Linux would need Popen and Windows
would need a regular file open(), I could simply make it so that if the
platform was Linux, my wrapper is used in place of that. Just another
example would be having an external program decrypt a file that can be in
plain text or encrypted that might go something like this:
if encryptionEnabled:
fileobj = subprocess.ProcessIOWrapper("gpg --decrypt
supersecret.html.gpg")
else:
fileobj = open("notsosecret.html")
>From there, the functions would not have to be modified despite using a
running process versus a file object.
On Tue, Jul 28, 2009 at 01:53, Gregory P. Smith wrote:
> On Mon, Jul 27, 2009 at 5:32 PM, Glyph Lefkowitz
> wrote:
> > On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore wrote:
> >>
> >> I like MRAB's idea of using a (non-standard) "e" flag to include
> >> stderr. So "r" reads from stdout, "re" reads from stdout+stderr.
> >>
> >> Anything more complicated probably should just use "raw" Popen
> >> objects. Don't overcomplicate the interface.
> >
> > In my opinion, mangling stderr and stdout together is already an
> > overcomplication. It shouldn't be implemented.
> >
> > It seems like a good idea, until you realize that subtle changes to your
> OS,
> > environment, or buffering behavior may result in arbitrary, unparseable
> > output.
>
> Agreed. Leave stderr support out of this. People who need stderr
> should use the full subprocess.Popen interface. Quick hack unixy
> types will just run their process using a shell (which already seems
> to be the default based on the "ls -l" example) and add 2>&1. This
> functionality is basically the equivalent of adding the | symbol on
> either or both ends of a filename given to open() in perl. (but
> without being so gross).
>
> I do wonder why you're trying to make it behave exactly like open()
> including the mode= syntax.
>
> Why not just define several names based on the behavior?
>
> ProcessReadWrapper()
> ProcessWriteWrapper()
> ProcessReadWriteWrapper()
>
> -gps
>
> >
> > For example, let's say you've got a program whose output is a list of
> lines,
> > each one containing a number. Sometimes it tries to import gtk, and
> fails
> > to open its display.
> >
> > That's fine, and you can still deal with it, as long as the interleaved
> > output looks like this:
> >
> > 100
> > 200
> > Gtk-WARNING **: cannot open display:
> > 300
> > 400
> >
> > but of course the output might (although unlikely with such small chunks
> of
> > output) end up looking like this, instead:
> >
> > 100
> > 2Gtk-WAR0NING0 **:
> > can30not 0open display:
> >
> > 400
> >
> > this is the sort of thing which is much more likely to happen once you
> start
> > dealing with large volumes of data, where there are more page-boundaries
> for
> > your buffers to get confused around, and you are playing with buffering
> > options to improve performance. In other words, it's something that
> fails
> > only at scale or under load, and is therefore extremely difficult to
> debug.
> >
> > This option might be okay if it were allowed only on subprocesses opened
> in
> > a text mode, and if the buffering logic involved forced stderr and stdout
> to
> > be line-delimited, and interleave only lines, rather than arbitrary
> chunks
> > of bytes. Of course then if you use this flag with a program that
> outputs
> > binary data with no newlines it will buffer forever and crash your
> program
> > with a MemoryError, but at least that's easy to debug when it happens.
> >
> > ___
> > Python-Dev mailing list
> > Python-Dev@python.org
> > http://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
> >
> >
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing File Modes
Hmm... can't you do this?
if encryptionEnabled:
p = subprocess.Popen(["gpg", "--decrypt", "supersecret.html.gpg"],
stdin = subprocess.PIPE)
fileobj = p.stdin
else:
fileobj = open("notsosecret.html")
I think that works. Is there something this way won't work for? You
can also do the same thing to get stdout and stderr file objects. I
guess a wrapper would simplify this process.
-Devin
On Wed, Jul 29, 2009 at 7:41 PM, Eric Pruitt wrote:
> My motivation came from an instance when I was using subprocess.Popen for a
> Linux / Windows cross platform program. In part of the program, I was
> writing and reading to a cron like object. On Windows, it was a text file
> and on Linux it would be the crontab executable. Had I been able to
> substitute the "open()" function with my wrapper, it would have been the
> only change I had to make for cross platform compatibility; instead of
> having to change numerous lines because Linux would need Popen and Windows
> would need a regular file open(), I could simply make it so that if the
> platform was Linux, my wrapper is used in place of that. Just another
> example would be having an external program decrypt a file that can be in
> plain text or encrypted that might go something like this:
>
> if encryptionEnabled:
> fileobj = subprocess.ProcessIOWrapper("gpg --decrypt
> supersecret.html.gpg")
> else:
> fileobj = open("notsosecret.html")
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Implementing File Modes
Well, with a few changes to your code, that would indeed work (you are using
stdin as your pipe. Correct me if I'm wrong but if you intend to read from
it, you need to change it to "stdout = subprocess.PIPE" and the other lines
as well to reflect this change). My Google Summer of Code modifications to
subprocess.Popen provide non-blocking, asynchronous I/O support and my file
wrapper is built upon that augmented functionality. If I remember correctly,
when I was working on the program where I first thought a file wrapper for
subprocess.Popen would be rather handy, I also ran into blocking I/O as
well.
On Wed, Jul 29, 2009 at 20:20, Devin Cook wrote:
> Hmm... can't you do this?
>
> if encryptionEnabled:
>p = subprocess.Popen(["gpg", "--decrypt", "supersecret.html.gpg"],
> stdin = subprocess.PIPE)
>fileobj = p.stdin
> else:
>fileobj = open("notsosecret.html")
>
> I think that works. Is there something this way won't work for? You
> can also do the same thing to get stdout and stderr file objects. I
> guess a wrapper would simplify this process.
>
> -Devin
>
> On Wed, Jul 29, 2009 at 7:41 PM, Eric Pruitt wrote:
> > My motivation came from an instance when I was using subprocess.Popen for
> a
> > Linux / Windows cross platform program. In part of the program, I was
> > writing and reading to a cron like object. On Windows, it was a text file
> > and on Linux it would be the crontab executable. Had I been able to
> > substitute the "open()" function with my wrapper, it would have been the
> > only change I had to make for cross platform compatibility; instead of
> > having to change numerous lines because Linux would need Popen and
> Windows
> > would need a regular file open(), I could simply make it so that if the
> > platform was Linux, my wrapper is used in place of that. Just another
> > example would be having an external program decrypt a file that can be in
> > plain text or encrypted that might go something like this:
> >
> > if encryptionEnabled:
> > fileobj = subprocess.ProcessIOWrapper("gpg --decrypt
> > supersecret.html.gpg")
> > else:
> > fileobj = open("notsosecret.html")
> >
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
