Re: [Cython] [cython] Stopgap solution for NumPy 1.7 API changes (#99)
Dag Sverre Seljebotn, 21.03.2012 22:36: > NumPy is starting to seriously deprecating access to the member fields > in an ndarray (it was always frowned upon, but now it is starting to > become enforced). To support the large body of Cython code out there > accessing these fields (arr.shape[0] and so on), we special-case > PyArrayObject in Cython, with special knowledge of the NumPy API. > > Ideally, we may introduce features in Cython in the future that allows > specifying this kind of magic with syntax in pxd files, and then we can > move away from special-casing NumPy. Given the comments in this change, wouldn't it make sense to emit a visible "this will eventually stop working" warning when user code runs into the special casing hacks? Just because the fields are convenient doesn't mean code should rely on them being there when the NumPy folks say they shouldn't be accessible. As I understand it, it's encouraged to use the accessor functions instead. If that's what the NumPy developers want, users should just do that, should they not? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Gsoc project
Dag Sverre Seljebotn, 28.03.2012 05:05: > - Very often one is not interested in the full header file. One really > wants "the API", not a translation of the C header. This probably requires > a) some heuristics, and b) the possibility for, as easily as possible, > write some selectors/configuration for what should be included and not. > Making that end-user-friendly is perhaps a challenge, I'm not sure. It's usually not a problem when there's too much declared in a .pxd, but users may want to leave specific things out. So I think a simple blacklist configuration file would work just fine. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Gsoc project
Stefan Behnel, 28.03.2012 09:20: > Dag Sverre Seljebotn, 28.03.2012 05:05: >> - Very often one is not interested in the full header file. One really >> wants "the API", not a translation of the C header. This probably requires >> a) some heuristics, and b) the possibility for, as easily as possible, >> write some selectors/configuration for what should be included and not. >> Making that end-user-friendly is perhaps a challenge, I'm not sure. > > It's usually not a problem when there's too much declared in a .pxd, but > users may want to leave specific things out. So I think a simple blacklist > configuration file would work just fine. Adding a bit to this: The other side of the use case, that you want to override a specific definition that was auto-generated into a .pxd file, is trivial in the sense that you can always add a hand-written .pxd file next to it and use that instead. And for declarations that you need to override but that are used in the .pxd file itself, a blacklist entry plus a cimport from a hand-written .pxd file would do the trick. So, basically, the generator would start by creating two .pxd files: one to be manually edited and one that contains the generated declarations and that cimports (or, maybe even better, includes) the editable file at the top. From that point on, it only changes the generated file and uses a blacklist to keep out certain declarations. It's then up to the user to provide them in the hand written file (or not at all). Sounds easy enough. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython] Stopgap solution for NumPy 1.7 API changes (#99)
The idea really was that it would never stop working, but could in time be replaced by other features that would make this implementable in the pxd (like inlineable properties on extension types, typed tuples or lists). Consider it speeding up the ndarray Python API, not providing any C API. Note that even before this, the 'dims' field was renamed 'shape', because it is shape in the Python API. Dag -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Stefan Behnel wrote: Dag Sverre Seljebotn, 21.03.2012 22:36: > NumPy is starting to seriously deprecating access to the member fields > in an ndarray (it was always frowned upon, but now it is starting to > become enforced). To support the large body of Cython code out there > accessing these fields (arr.shape[0] and so on), we special-case > PyArrayObject in Cython, with special knowledge of the NumPy API. > > Ideally, we may introduce features in Cython in the future that allows > specifying this kind of magic with syntax in pxd files, and then we can > move away from special-casing NumPy. Given the comments in this change, wouldn't it make sense to emit a visible "this will eventually stop working" warning when user code runs into the special casing hacks? Just because the fields are convenient doesn't mean code should rely on them being there when the NumPy folks say they shouldn't be accessible. As I understand it, it's encouraged to use the accessor functions instead. If that's what the NumPy developers want, users should just do that, should they not? Stefan _ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] test
___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Gsoc project
On Wed, Mar 28, 2012 at 12:52 AM, Stefan Behnel wrote: > Stefan Behnel, 28.03.2012 09:20: >> Dag Sverre Seljebotn, 28.03.2012 05:05: >>> - Very often one is not interested in the full header file. One really >>> wants "the API", not a translation of the C header. This probably requires >>> a) some heuristics, and b) the possibility for, as easily as possible, >>> write some selectors/configuration for what should be included and not. >>> Making that end-user-friendly is perhaps a challenge, I'm not sure. >> >> It's usually not a problem when there's too much declared in a .pxd, but >> users may want to leave specific things out. So I think a simple blacklist >> configuration file would work just fine. > > Adding a bit to this: The other side of the use case, that you want to > override a specific definition that was auto-generated into a .pxd file, is > trivial in the sense that you can always add a hand-written .pxd file next > to it and use that instead. And for declarations that you need to override > but that are used in the .pxd file itself, a blacklist entry plus a cimport > from a hand-written .pxd file would do the trick. > > So, basically, the generator would start by creating two .pxd files: one to > be manually edited and one that contains the generated declarations and > that cimports (or, maybe even better, includes) the editable file at the > top. From that point on, it only changes the generated file and uses a > blacklist to keep out certain declarations. It's then up to the user to > provide them in the hand written file (or not at all). > > Sounds easy enough. Perhaps the blacklist could be (in part) based on the hand-generated .pxd file, and the generated .pxd file would simply not emit re-declarations. - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Gsoc project
On Tue, Mar 27, 2012 at 8:05 PM, Dag Sverre Seljebotn
wrote:
> On 03/27/2012 02:17 PM, Philip Herron wrote:
>>
>> Hey
>>
>> I got linked to your idea
>>
>> http://groups.google.com/group/cython-users/browse_thread/thread/cb8aa58083173b97/cac3cf12d438b122?show_docid=cac3cf12d438b122&pli=1
>> by David Malcolm on his plugin mailing list.
>>
>> I am looking to apply to Gsoc once again this year i have done gsoc
>> 2010 and 2011 on GCC implementing my own GCC front-end for python
>> which is still in very early stages since its a huge task. But i am
>> tempted to apply to this project to implement a more self contained
>> project to give back to the community more promptly while that hacking
>> on my own front-end on my own timer. And i think it would benefit me
>> to get to understand in more detail different aspects of python which
>> is what i need and would gain very much experience from.
>
>
> Excellent! After talking to lots of people at PyCon about Cython, it is
> obvious that auto-generation of pxd files is *the* most missed feature in
> Cython today. If you do this, lots of Cython users will be very grateful.
+1, this idea has been floated many times before, and I think it would
make a great GSoC project.
>> I was wondering if you could give me some more details on how this
>> could all work i am not 100% familiar with cython but i think i
>> understand it to a good extend from playing with it for most of my
>> evening. I just want to make sure i understand the basic use case of
>> this fully, When a user could have something like:
>>
>> -header foo.h
>>
>> extern int add (int, int);
>>
>> -source foo.c
>>
>> #include "foo.h"
>>
>> int add (int x, int y)
>> {
>> return x+y;
>> }
>>
>> We use the plugin to go over the decls created and create a pxd file like:
>>
>> cdef int add (int a, int b):
>> return a + b
>>
>> Although this is a really basic example i just want to make sure i
>> understand whats going on. Maybe some more of you have input? I guess
>> this would be best suited as a proposal for Python rather than GCC?
>
>
> This isn't quite what should be done. Cython generates C code that includes
> C header files; what the pxd files are needed for is to provide declarations
> for Cython about what is available on the C side (during the Cython->C
> translation/compilation).
>
> So: "foo.c" is irrelevant to Cython. And, foo.h should turn into foo.pxd
> like this:
>
> cdef extern from "foo.h":
> int add(int, int)
>
> Let us know if you have any question; you may want to look at examples for
> using Cython to wrap C code, such as
>
> https://github.com/zeromq/pyzmq/blob/master/zmq/core/libzmq.pxd
>
> and the rest of the pyzmq code.
>
> Moving over to the idea of making this a GSoC:
>
> First, we have a policy of requiring patches from prospective students in
> addition to their application. Often, this has been to fix a bug or two in
> Cython. However, given that pxd generation can be done without much digging
> into Cython itself, I think that something like a crude prototype of the pxd
> generator (supporting only a subset of C) would be a better fit (other devs,
> what do you think?)
Yep, that would be sufficient for me.
> The project should contain at least:
>
> - The wrapper generator itself
> - Tests for it (including the task of figuring out how to test this,
> possibly both unit tests and integration tests)
> - A strategy for testing it for all relevant versions of gcc; one should
> probably set up Jenkins jobs for it
>
> Even then, I feel that this is rather small for a full GSoC, even when
> supporting the subset of C++ supported by Cython, I would estimate a month
> or so (and GSoC is two months). So it should be extended in one direction or
> another. Some ideas:
>
> - Very often one is not interested in the full header file. One really
> wants "the API", not a translation of the C header. This probably requires
> a) some heuristics, and b) the possibility for, as easily as possible, write
> some selectors/configuration for what should be included and not. Making
> that end-user-friendly is perhaps a challenge, I'm not sure.
>
> One idea here is to make possible an interplay where you look at the pyx
> file what needs to be wrapped. I.e. you first try to use a function in the
> pyx file as if it had already been declared, then run the pxd generator
> feeding in the pyx files (and .h files), and out comes the required pxd file
> bridging the two (containing only the used subset).
>
> - Support using clang to parse C code in addition
>
> - There's a problem in that an often-used Cython approach is:
>
> 1) Generate C file from pyx and pxd files
> 2) Ship to other computers
> 3) Compile C file
>
> However, this is fragile when combined with auto-generated pxd files,
> because the resulting pxd may be different depending on whether -DFOO is
> given to gcc or not.
>
> The above 3 steps are possible because Cython often does not care about the
> exact type of something, jus
Re: [Cython] Question on "numpy_common.pxi" in NumPy tests
On Tue, Mar 27, 2012 at 11:49 PM, Stefan Behnel wrote: > Lisandro Dalcin, 27.03.2012 21:34: >> On 27 March 2012 17:20, Stefan Behnel wrote: >>> the NumPy related tests use a file "numpy_common.pxi" that contains this >>> useless code: >>> >>> """ >>> cdef extern from *: >>> bint FALSE "0" >>> void import_array() >>> void import_umath() >>> >>> if FALSE: >>> import_array() >>> import_umath() >>> """ >>> >>> Does this serve any purpose? It currently leads to build failures of the >>> tests in C++ mode because the "import_umath()" call seems to inline code >>> with a bare "return" statement, which is not allowed in the module init >>> function (which returns a reference to the module). >>> >>> Is there any reason why this can't just die? >> >> Oh! long long ago Cython testsuite was able to run without a single C >> compiler warning... That numpy common file was a hack to silent >> "defined but not used" warnings about the import array/umath functions >> defined in NumPy headers. > > Ok - in that case, it should be enough to move the import calls into a cdef > function body and "avoiding" to call that. And then gcc helpfully spits out an "unused function" warning, which is what we're trying to avoid in the first place... > And a comment in the file would be helpful ... Done. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Gsoc project
Hey all
I am implemented a very crude and simplistic and very badly programmed
version of a pxd generator i think i understand what were after now
but i would appreciate if you look over what i did to make sure i have
grasped the basic idea for now:
So if i have:
#include "test.h"
int add (int x, int y)
{
return x + y;
}
#ifndef __TEST_H__
#define __TEST_H__
extern int add (int, int);
struct foobar {
int a;
unsigned char * b;
};
#endif //__TEST_H_
We run gcc -fplugin=./python.so -fplugin-arg-python-script=walk.py test.c
And i output out.pxd:
cdef extern from "test.h":
int add (int, int)
ctypedef struct foobar:
int a
unsigned char * b
So far in a very crude way it understands structs and functions but
nothing else i can see how it could all work but i see now why you see
it could be a very small project David's plugin system has mostly
everything done for you but i would like to add to his plugin for some
tree access code etc...
Adding a config file for telling the plugin to not care about certain
things would be a nice addition. It seems interesting the clang idea,
it could be interesting porting this to other front-ends of gcc like
gccgo.
--Phil
import gcc
import re
from pprint import pformat
from gccutils import get_src_for_loc, cfg_to_dot, invoke_dot
decls = []
def decl_location_get_file (decl):
repr = ('%r' % decl.location)
idx = repr.index ('file=')
idy = repr.index ('line=')
file_string = ""
for i in range (idx+6, idy-3):
file_string += repr[i]
return file_string
def generate_pxd_decl_function (fd, decl):
fmt = "%s" % decl.type
ident = '%s' % decl
if re.search ("<.*>", fmt):
func = re.sub ("<.*>", ident, fmt)
fd.write ("cdef extern from \"%s\":\n" % decl_location_get_file (decls[0]))
fd.write ('\t%s\n' % func)
def decl_identifier_node_to_string (node):
repr = ('%r' % node)
idx = repr.index ('name=')
idy = repr.index (')')
retval = ""
for i in range (idx+6, idy-1):
retval += repr [i]
return retval
def generate_pxd_decl_type (fd, decl):
if ('%s' % type (decl.type)) == "":
ident = decl_identifier_node_to_string (decl.type.name)
fd.write ("\tctypedef struct %s:\n" % ident)
for f in decl.type.fields:
fd.write ("\t\t%s %s\n" % (f.type, f.name))
def generate_pxd_data (T, fd, decl):
if T == "":
return generate_pxd_decl_function (fd, decl)
elif T == "":
return generate_pxd_decl_type (fd, decl)
else:
print "unhandled type <%s>!\n" % T
def walk_generate_pxd_decls ():
if len (decls) > 0:
fd = open ('out.pxd', 'w')
for decl in decls:
print('%r:%s:%s' % (decl.location, decl, decl.type))
decl_type = '%s' % type (decl)
generate_pxd_data (decl_type, fd, decl)
fd.write ("\n")
fd.close ()
def on_pass_execution (p, fn):
if p.name == '*free_lang_data':
for u in gcc.get_translation_units ():
for decl in u.block.vars:
f = decl_location_get_file (decl)
if f != '':
decls.append (decl)
walk_generate_pxd_decls ()
gcc.register_callback (gcc.PLUGIN_PASS_EXECUTION, on_pass_execution)
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Gsoc project
On 03/28/2012 07:58 PM, Philip Herron wrote:
Hey all
I am implemented a very crude and simplistic and very badly programmed
version of a pxd generator i think i understand what were after now
but i would appreciate if you look over what i did to make sure i have
grasped the basic idea for now:
So if i have:
#include "test.h"
int add (int x, int y)
{
return x + y;
}
#ifndef __TEST_H__
#define __TEST_H__
extern int add (int, int);
struct foobar {
int a;
unsigned char * b;
};
#endif //__TEST_H_
We run gcc -fplugin=./python.so -fplugin-arg-python-script=walk.py test.c
And i output out.pxd:
cdef extern from "test.h":
int add (int, int)
ctypedef struct foobar:
int a
unsigned char * b
Nice. But please use 4 spaces (see PEP 8) :-)
More ideas for project proposal:
Another slight complication is that you should ideally turn
#define FOO 3
#define BAR 4
into
cdef extern from "foo.h":
enum:
FOO
BAR
so you need to hook in before the preprocessor and after the
preprocessor and dig out different stuff.
Then what happens if you have
#ifdef FOO
#define BAR 3
#else
#define BAR 4
#endif
?? I'm not saying it is hard, but perhaps no longer completely trivial :-)
And like Robert hinted at, supporting all the aspects of C++ might take
a little more work, there's just so much different syntax in C++, and
there's several C++ features Cython just does not support and must be
either ignored or hacked around (e.g., "const").
Supporting stuff like
#define MACRO(x) ((x)->field*2)
probably belongs in the category of "must be done manually".
So far in a very crude way it understands structs and functions but
nothing else i can see how it could all work but i see now why you see
it could be a very small project David's plugin system has mostly
everything done for you but i would like to add to his plugin for some
tree access code etc...
Adding a config file for telling the plugin to not care about certain
things would be a nice addition. It seems interesting the clang idea,
it could be interesting porting this to other front-ends of gcc like
gccgo.
Does gccgo use the C ABI so that Cython could call it? If so, go for it!
(Fortran is actually very much in use in the Cython userbase and would
get a lot more "customers" than Go, but if you have more of a CS
background or similar I can see why you wouldn't be so interested in
Fortran. I didn't believe people were still using Fortran either until I
started doing astrophysics, and suddenly it seems to be the default tool
everybody uses for everything.)
Dag
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Gsoc project
Dag Sverre Seljebotn, 29.03.2012 05:28:
> On 03/28/2012 07:58 PM, Philip Herron wrote:
>> I am implemented a very crude and simplistic and very badly programmed
>> version of a pxd generator i think i understand what were after now
>> but i would appreciate if you look over what i did to make sure i have
>> grasped the basic idea for now:
>>
>> So if i have:
>>
>> #include "test.h"
>>
>> int add (int x, int y)
>> {
>>return x + y;
>> }
>>
>> #ifndef __TEST_H__
>> #define __TEST_H__
>>
>> extern int add (int, int);
>>
>> struct foobar {
>>int a;
>>unsigned char * b;
>> };
>>
>> #endif //__TEST_H_
>>
>> We run gcc -fplugin=./python.so -fplugin-arg-python-script=walk.py test.c
>>
>> And i output out.pxd:
>>
>> cdef extern from "test.h":
>> int add (int, int)
>>
>> ctypedef struct foobar:
>> int a
>> unsigned char * b
>
> Nice.
Yep.
> But please use 4 spaces (see PEP 8) :-)
Yep.
> More ideas for project proposal:
>
> Another slight complication is that you should ideally turn
>
> #define FOO 3
> #define BAR 4
>
> into
>
> cdef extern from "foo.h":
> enum:
> FOO
> BAR
>
> so you need to hook in before the preprocessor and after the preprocessor
> and dig out different stuff.
>
> Then what happens if you have
>
> #ifdef FOO
> #define BAR 3
> #else
> #define BAR 4
> #endif
>
> ?? I'm not saying it is hard, but perhaps no longer completely trivial :-)
These things bring us close to what SWIG wants to achieve, though. I think
we should find a point where we should stop and leave it to users writing
their own supplemental .pxd file.
When generating the initial manual .pxd file, we could still move all stuff
in there that we think users should take care of. It's easy enough to
delete or comment out declarations once they're there - much easier than
wading through a header file trying to figure out what might work and what
might not.
> And like Robert hinted at, supporting all the aspects of C++ might take a
> little more work, there's just so much different syntax in C++, and there's
> several C++ features Cython just does not support and must be either
> ignored or hacked around (e.g., "const").
It would still be good to have them sort-of supported in the generator,
even if there's no declaration code coming out of it. In the specific case
of "const", we still want to support it in Cython at some point, and I'm
sure there are other things of that kind.
> Supporting stuff like
>
> #define MACRO(x) ((x)->field*2)
>
> probably belongs in the category of "must be done manually".
I think that's a good case for generating "something" into the editable
file (maybe even commented out) and letting the user make sense of it. We
could have standard comments for different constructs that we write into
the user file, e.g.
## please add types for macro MACRO, defined as
## MACRO(x) ((x)->field*2)
# cdef ? MACRO(? x)
Probably also with some help text at the top of the file that shows how to
define different signatures for the same function declaration (using
explicit cnames) plus a link to the relevant docs.
>> it could be interesting porting this to other front-ends of gcc like
>> gccgo.
>
> Does gccgo use the C ABI so that Cython could call it? If so, go for it!
Absolutely. We already have fwrap for Fortran, but if there's a way to use
GCC in order to generate glue code (i.e. not just .pxd files but also
header files and some adaptation code) that talks to even more languages,
that would be more than you could ever dream of as result of a GSoC. As
long as a language supports the C-ABI, it may even be easier to do than for
C/C++, because most languages simply don't have preprocessor macros and
similarly hard to analyse features. Even if it's just a "you have to write
your functions in this specific way in language X and only then the tool
can adapt them for you", that would still be extremely useful.
It certainly sounds like Philip has the right background for something like
that.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
