[Python-Dev] Extended Buffer Protocol - simple use examples
Hi, I'll admit right off that I haven't followed all of the extended buffer protocol discussions - I have no real need for anything much beyond the existing "here's a blob of memory" level of functionality. I have skimmed (briefly, I'll admit!) the pre-PEP, but I've found it extremely difficult to find a simple example of the basic (in my view) use case of an undifferentiated block of bytes. Can I suggest that the following uses would be the significant majority of buffer protocol uses - they are certainly going to be the basic transitional case from the existing protocol - and deserve to be singled out in the PEP with some description or (better) pseudo-code examples. If they are already there, I apologise, but I did look, and got bogged down in the more sophisticated stuff - so they probably need making more visible. 1. (Producer) I have a block of memory in my C extension and I want to expose it as a simple contiguous block of bytes to Python. 2. (Consumer) I want to get at a block of memory exposed as a buffer. I am only interested in, and only support, viewing a buffer as a block of contiguous bytes. I expect most if not all extensions to be able to provide such a view. I'd hope these 2 cases are pretty simple to code, but I'd like to see that made explicit in the PEP. unsophisticated-ly y'rs Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Extended Buffer Protocol - simple use examples
Paul Moore wrote: > Hi, > I'll admit right off that I haven't followed all of the extended > buffer protocol discussions - I have no real need for anything much > beyond the existing "here's a blob of memory" level of functionality. > > I have skimmed (briefly, I'll admit!) the pre-PEP, but I've found it > extremely difficult to find a simple example of the basic (in my view) > use case of an undifferentiated block of bytes. > This is a great suggestion and it was on my to-do list. I've included some examples of this use-case in the new PEP. > > 1. (Producer) I have a block of memory in my C extension and I want to > expose it as a simple contiguous block of bytes to Python. This is now Ex. 2 in the PEP. > > 2. (Consumer) I want to get at a block of memory exposed as a buffer. > I am only interested in, and only support, viewing a buffer as a block > of contiguous bytes. I expect most if not all extensions to be able to > provide such a view. > This is now Ex. 3 Thanks for the suggestions. -Travis ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 3118: Extended buffer protocol (new version)
Changes: * added the "flags" variable to allow simpler calling for getbuffer. * added some explanation of ideas that were discussed and abandoned. * added examples for simple use cases. * added more C-API calls to allow easier usage. Thanks for all feedback. -Travis PEP: 3118 Title: Revising the buffer protocol Version: $Revision$ Last-Modified: $Date$ Authors: Travis Oliphant <[EMAIL PROTECTED]>, Carl Banks <[EMAIL PROTECTED]> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Aug-2006 Python-Version: 3000 Abstract This PEP proposes re-designing the buffer interface (PyBufferProcs function pointers) to improve the way Python allows memory sharing in Python 3.0 In particular, it is proposed that the character buffer portion of the API be elminated and the multiple-segment portion be re-designed in conjunction with allowing for strided memory to be shared. In addition, the new buffer interface will allow the sharing of any multi-dimensional nature of the memory and what data-format the memory contains. This interface will allow any extension module to either create objects that share memory or create algorithms that use and manipulate raw memory from arbitrary objects that export the interface. Rationale = The Python 2.X buffer protocol allows different Python types to exchange a pointer to a sequence of internal buffers. This functionality is *extremely* useful for sharing large segments of memory between different high-level objects, but it is too limited and has issues: 1. There is the little used "sequence-of-segments" option (bf_getsegcount) that is not well motivated. 2. There is the apparently redundant character-buffer option (bf_getcharbuffer) 3. There is no way for a consumer to tell the buffer-API-exporting object it is "finished" with its view of the memory and therefore no way for the exporting object to be sure that it is safe to reallocate the pointer to the memory that it owns (for example, the array object reallocating its memory after sharing it with the buffer object which held the original pointer led to the infamous buffer-object problem). 4. Memory is just a pointer with a length. There is no way to describe what is "in" the memory (float, int, C-structure, etc.) 5. There is no shape information provided for the memory. But, several array-like Python types could make use of a standard way to describe the shape-interpretation of the memory (wxPython, GTK, pyQT, CVXOPT, PyVox, Audio and Video Libraries, ctypes, NumPy, data-base interfaces, etc.) 6. There is no way to share discontiguous memory (except through the sequence of segments notion). There are two widely used libraries that use the concept of discontiguous memory: PIL and NumPy. Their view of discontiguous arrays is different, though. The proposed buffer interface allows sharing of either memory model. Exporters will use only one approach and consumers may choose to support discontiguous arrays of each type however they choose. NumPy uses the notion of constant striding in each dimension as its basic concept of an array. With this concept, a simple sub-region of a larger array can be described without copying the data. T Thus, stride information is the additional information that must be shared. The PIL uses a more opaque memory representation. Sometimes an image is contained in a contiguous segment of memory, but sometimes it is contained in an array of pointers to the contiguous segments (usually lines) of the image. The PIL is where the idea of multiple buffer segments in the original buffer interface came from. NumPy's strided memory model is used more often in computational libraries and because it is so simple it makes sense to support memory sharing using this model. The PIL memory model is sometimes used in C-code where a 2-d array can be then accessed using double pointer indirection: e.g. image[i][j]. The buffer interface should allow the object to export either of these memory models. Consumers are free to either require contiguous memory or write code to handle one or both of these memory models. Proposal Overview = * Eliminate the char-buffer and multiple-segment sections of the buffer-protocol. * Unify the read/write versions of getting the buffer. * Add a new function to the interface that should be called when the consumer object is "done" with the memory area. * Add a new variable to allow the interface to describe what is in memory (unifying what is currently done now in struct and array) * Add a new variable to allow the protocol to share shape information * Add a new variable for sharing stride information * Add a new mechanism for sharing arrays that must be accessed using pointer indirection. * Fix all objects in the core and the standard librar
Re: [Python-Dev] PEP 3118: Extended buffer protocol (new version)
Travis Oliphant: > PEP: 3118 > ... I'd like to see the PEP include discussion of what to do when an incompatible request is received while locked. Should there be a standard "Can't do that: my buffer has been got" exception? Neil ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] function for counting items in a sequence
Greg Ewing wrote: > Steven Bethard wrote: > >> * Greg Ewing - counteach(), countall() >> * Guido - counts() is fine > > I'm happy with counts() too -- I only suggested the > others in case counts() wasn't acceptable for some > reason. If Guido likes it, that's good enough for > me. I think I'd prefer "tally" as less ambiguous, but "counts" works too. Tim Delaney ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3118: Extended buffer protocol (new version)
Travis Oliphant wrote: > Py_BUF_READONLY >The returned buffer must be readonly and the underlying object should make >its memory readonly if that is possible. I don't like the "if possible" thing. If it makes no guarantees, it pretty much useless over Py_BUF_SIMPLE. > Py_BUF_FORMAT >The consumer will be using the format string information so make sure that >member is filled correctly. Is the idea to throw an exception if there's some other data format besides "b", and this flag isn't set? It seems superfluous otherwise. > Py_BUF_SHAPE >The consumer can (and might) make use of using the ndims and shape members > of the structure >so make sure they are filled in correctly. > > Py_BUF_STRIDES (implies SHAPE) >The consumer can (and might) make use of the strides member of the > structure (as well >as ndims and shape) Is there any reasonable benefit for allowing Py_BUF_SHAPE without Py_BUF_STRIDES? Would the array be C- or Fortran-like? Another little mistake I made: looking at the Python source, it seems that most C defines do not use the Py_ prefix, so probably we shouldn't here. Sorry. Carl ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 3118: Extended buffer protocol (new version)
Carl Banks wrote: > > > Travis Oliphant wrote: > > Py_BUF_READONLY > >The returned buffer must be readonly and the underlying object > should make > >its memory readonly if that is possible. > > I don't like the "if possible" thing. If it makes no guarantees, it > pretty much useless over Py_BUF_SIMPLE. O.K. Let's make it raise an error if it can't set it read-only. >> Py_BUF_FORMAT >>The consumer will be using the format string information so make >> sure thatmember is filled correctly. > > Is the idea to throw an exception if there's some other data format > besides "b", and this flag isn't set? It seems superfluous otherwise. The idea is that a consumer may not care about the format and the exporter may want to know that to simplify the interface.In other words the flag is a way for the consumer to communicate that it wants format information (or not). If the exporter wants to raise an exception if the format is not requested is up to the exporter. >> Py_BUF_SHAPE >>The consumer can (and might) make use of using the ndims and shape >> members of the structure >>so make sure they are filled in correctly.Py_BUF_STRIDES >> (implies SHAPE) >>The consumer can (and might) make use of the strides member of the >> structure (as well >>as ndims and shape) > > Is there any reasonable benefit for allowing Py_BUF_SHAPE without > Py_BUF_STRIDES? Would the array be C- or Fortran-like? Yes, I could see a consumer not being able to handle simple striding but could handle shape information. Many users of NumPy arrays like to think of the array as an N-d array but want to ignore striding. I've made the changes in numpy's SVN. Hopefully they will get mirrored over to the python PEP directory eventually. -Travis ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] deprecating BaseException.message
During the PyCon sprint I tried to make BaseException accept only a single argument and bind it to BaseException.message . I was successful (see the p3yk_no_args_on_exc branch), but it was very painful to pull off as anyone who sat around me the last three days of the sprint will tell you as they had to listen to me curse incessantly. Because of the pain that I went through in the transition and thus the lessons learned, Guido and I discussed it and we think it would be best to give up on forcing BaseException to accept only a single argument. I think it is still doable, but requires a multi-release transition period and not the one that 2.6 -> 3.0 is offering. And so Guido and I plan on deprecating BaseException.message as its entire point in existence was to help transition to what we are not going to have happen. =) Now that means BaseException.message might hold the record for shortest lived feature as it was only introduced in 2.5 and is now to be deprecated in 2.6 and removed in 2.7/3.0. =) Below is PEP 352, revised to reflect the removal of BaseException.messageand for letting the 'args' attribute stay (along with suggesting one should only pass a single argument to BaseException). Basically the interface for exceptions doesn't really change in 3.0 except for the removal of __getitem__. -- PEP: 352 Title: Required Superclass for Exceptions Version: $Revision: 53592 $ Last-Modified: $Date: 2007-01-28 21:54:11 -0800 (Sun, 28 Jan 2007) $ Author: Brett Cannon <[EMAIL PROTECTED]> Guido van Rossum <[EMAIL PROTECTED]> Status: Final Type: Standards Track Content-Type: text/x-rst Created: 27-Oct-2005 Post-History: Abstract In Python 2.4 and before, any (classic) class can be raised as an exception. The plan for 2.5 was to allow new-style classes, but this makes the problem worse -- it would mean *any* class (or instance) can be raised! This is a problem as it prevents any guarantees from being made about the interface of exceptions. This PEP proposes introducing a new superclass that all raised objects must inherit from. Imposing the restriction will allow a standard interface for exceptions to exist that can be relied upon. It also leads to a known hierarchy for all exceptions to adhere to. One might counter that requiring a specific base class for a particular interface is unPythonic. However, in the specific case of exceptions there's a good reason (which has generally been agreed to on python-dev): requiring hierarchy helps code that wants to *catch* exceptions by making it possible to catch *all* exceptions explicitly by writing ``except BaseException:`` instead of ``except *:``. [#hierarchy-good]_ Introducing a new superclass for exceptions also gives us the chance to rearrange the exception hierarchy slightly for the better. As it currently stands, all exceptions in the built-in namespace inherit from Exception. This is a problem since this includes two exceptions (KeyboardInterrupt and SystemExit) that often need to be excepted from the application's exception handling: the default behavior of shutting the interpreter down without a traceback is usually more desirable than whatever the application might do (with the possible exception of applications that emulate Python's interactive command loop with ``>>>`` prompt). Changing it so that these two exceptions inherit from the common superclass instead of Exception will make it easy for people to write ``except`` clauses that are not overreaching and not catch exceptions that should propagate up. This PEP is based on previous work done for PEP 348 [#pep348]_. Requiring a Common Superclass = This PEP proposes introducing a new exception named BaseException that is a new-style class and has a single attribute, ``args``. Below is the code as the exception will work in Python 3.0 (how it will work in Python 2.x is covered in the `Transition Plan`_ section):: class BaseException(object): """Superclass representing the base of the exception hierarchy. Provides a 'message' attribute that contains either the single argument to the constructor or the empty string. This attribute is used in the string representation for the exception. This is so that it provides the extra details in the traceback. """ def __init__(self, *args): """Set the 'message' attribute'""" self.args = args def __str__(self): """Return the str of 'message'""" if len(self.args) == 1: return str(self.args[0]) else: return str(self.args) def __repr__(self): return "%s(*%s)" % (self.__class__.__name__, repr(self.args)) No restriction is placed upon what may be passed in for ``args`` for backwards-compatibility reasons. In practice, though, only a single string argument should be used. This keeps the string repr
