Re: [Cython] funding (Re: sage.math problems?)
mark florisson, 26.03.2012 19:42: > Apparently something is wrong with the py31 and py32 release jobs: > /levi/scratch/hudson/tmp/hudson738187841750656805.sh: line 17: > /levi/scratch/robertwb/hudson/hudson/jobs/cython-release-tests/workspace/BACKEND/c/PYVERSION/py31-ext/python/bin/python: > No such file or directory (see > https://sage.math.washington.edu:8091/hudson/view/release/job/cython-release-tests/BACKEND=c,PYVERSION=py31-ext/5/console). > The originating problem is: ln: accessing > `/levi/scratch/robertwb/hudson/hudson/jobs/py31-ext-hg/lastStable/archive/python31-bin.tar.gz': > No such file or directory Ah, sorry. A typo in the final archiving step. Should work now. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 0.16 release
2012/2/27 Sébastien Sablé Sablé : > Great, thanks! > > > 2012/2/25 mark florisson >> >> 2012/2/24 Sébastien Sablé Sablé : >> > Hi, >> > >> > could you please also look at incorporating the following patch before >> > releasing 0.16? (if it has not already been merged) >> > >> > https://github.com/cython/cython/pull/67 >> > >> > It has been more or less validated, but a test case is needed. >> > >> > This patch makes using C++ templates much more convenient with Cython. >> > >> > Currently I have to use hacks like the following which looks ugly and >> > make >> > the code less readable: >> > ctypedef TCacheVarData[float] TCacheVarData_float "TCacheVarData" >> > >> > Also thank you for all the work done on Cython, I have been using it >> > (and >> > Pyrex before) intensively for more than 6 years now, and it makes >> > integrating Python and C/C++ really convenient. >> > >> > Thanks in advance >> > >> > Sébastien >> > >> > ___ >> > cython-devel mailing list >> > cython-devel@python.org >> > http://mail.python.org/mailman/listinfo/cython-devel >> > >> >> Ok I merged it and added a test. I also fixed a lot of tests to run >> under MSVC on windows. I'm thinking to merge >> https://github.com/cython/cython/pull/77, see if everything still >> passes on Jenkins, and then pushing out a beta release for 0.16. I >> created some release notes, please feel free to add to the page >> (especially to the feature and improvements lists), they might be >> incomplete: http://wiki.cython.org/ReleaseNotes-0.16 >> >> Are there any other last-minute bug fixes pending? Recently I've found this bug on the tracker: http://trac.cython.org/cython_trac/ticket/766 I think I can fix it before release or wait unitl 0.16.1 since itsn't a regression. -- vitja. ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 0.16 release
On 27 March 2012 11:57, Vitja Makarov wrote: > 2012/2/27 Sébastien Sablé Sablé : >> Great, thanks! >> >> >> 2012/2/25 mark florisson >>> >>> 2012/2/24 Sébastien Sablé Sablé : >>> > Hi, >>> > >>> > could you please also look at incorporating the following patch before >>> > releasing 0.16? (if it has not already been merged) >>> > >>> > https://github.com/cython/cython/pull/67 >>> > >>> > It has been more or less validated, but a test case is needed. >>> > >>> > This patch makes using C++ templates much more convenient with Cython. >>> > >>> > Currently I have to use hacks like the following which looks ugly and >>> > make >>> > the code less readable: >>> > ctypedef TCacheVarData[float] TCacheVarData_float "TCacheVarData" >>> > >>> > Also thank you for all the work done on Cython, I have been using it >>> > (and >>> > Pyrex before) intensively for more than 6 years now, and it makes >>> > integrating Python and C/C++ really convenient. >>> > >>> > Thanks in advance >>> > >>> > Sébastien >>> > >>> > ___ >>> > cython-devel mailing list >>> > cython-devel@python.org >>> > http://mail.python.org/mailman/listinfo/cython-devel >>> > >>> >>> Ok I merged it and added a test. I also fixed a lot of tests to run >>> under MSVC on windows. I'm thinking to merge >>> https://github.com/cython/cython/pull/77, see if everything still >>> passes on Jenkins, and then pushing out a beta release for 0.16. I >>> created some release notes, please feel free to add to the page >>> (especially to the feature and improvements lists), they might be >>> incomplete: http://wiki.cython.org/ReleaseNotes-0.16 >>> >>> Are there any other last-minute bug fixes pending? > > > Recently I've found this bug on the tracker: > > http://trac.cython.org/cython_trac/ticket/766 > > I think I can fix it before release or wait unitl 0.16.1 since itsn't > a regression. > Sure, that's fine, I think there will be some time before the next beta. I tested the release in my own branch and jenkins was blue, but the release build seems to disagree. The py32 C++ build shows some refcount error: numpy_memoryview.cpp:16922: warning: ‘void __pyx_pw_5numpy_7ndarray_3__releasebuffer__(PyObject*, Py_buffer*)’ defined but not used python: Modules/gcmodule.c:327: visit_decref: Assertion `gc->gc.gc_refs != 0' failed. and some of the other python 3 tests are also failing. > -- > vitja. > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] 0.16 release
mark florisson, 27.03.2012 13:20: > I tested the release in my own branch and jenkins was blue, but > the release build seems to disagree. The release (and master) branch is tested against the "-ext" builds of CPython, which have some external packages installed, including NumPy. You have to change the config of your "tests" job to use those instead of the plain CPython builds. The build job can stay as it is. As for the inner workings, there's a normal "pyXY-hg" job to build CPython and a corresponding "pyXY-ext-hg" job that takes the build and installs a list of packages into it, then creates a new install archive. You can then reference either of the two archives in your build/test jobs by asking for a "pyXY" or "pyXY-ext" Python. > The py32 C++ build shows some refcount error: > > numpy_memoryview.cpp:16922: warning: ‘void > __pyx_pw_5numpy_7ndarray_3__releasebuffer__(PyObject*, Py_buffer*)’ > defined but not used These are legitimate warnings that are worth fixing (at some point), I think. They seem to originate from the buffer implementation in numpy.pxd. Those sort-of-external special methods shouldn't lead to the generation of a Python wrapper function. > python: Modules/gcmodule.c:327: visit_decref: Assertion > `gc->gc.gc_refs != 0' failed. It's surprising that that only occurs in one out of four Py3 test configurations. Maybe there's something indeterministic in those tests? > and some of the other python 3 tests are also failing. Yep, some of them look really funny (complaining about getting exactly the expected output), others are the typical Py3 problems (e.g. printing bytes). Note that the reason the py3k tests are not impacted is that it does not have NumPy. So the tests would equally fail in all Py3 versions. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Question on "numpy_common.pxi" in NumPy tests
Hi, the NumPy related tests use a file "numpy_common.pxi" that contains this useless code: """ cdef extern from *: bint FALSE "0" void import_array() void import_umath() if FALSE: import_array() import_umath() """ Does this serve any purpose? It currently leads to build failures of the tests in C++ mode because the "import_umath()" call seems to inline code with a bare "return" statement, which is not allowed in the module init function (which returns a reference to the module). Is there any reason why this can't just die? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Question on "numpy_common.pxi" in NumPy tests
On 27 March 2012 17:20, Stefan Behnel wrote: > Hi, > > the NumPy related tests use a file "numpy_common.pxi" that contains this > useless code: > > """ > cdef extern from *: > bint FALSE "0" > void import_array() > void import_umath() > > if FALSE: > import_array() > import_umath() > """ > > Does this serve any purpose? It currently leads to build failures of the > tests in C++ mode because the "import_umath()" call seems to inline code > with a bare "return" statement, which is not allowed in the module init > function (which returns a reference to the module). > > Is there any reason why this can't just die? > Oh! long long ago Cython testsuite was able to run without a single C compiler warning... That numpy common file was a hack to silent "defined but not used" warnings about the import array/umath functions defined in NumPy headers. I think you can safely remove that file, it serves no useful purpose IMHO... -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Question on "numpy_common.pxi" in NumPy tests
On Tue, Mar 27, 2012 at 7:20 AM, Stefan Behnel wrote: > Hi, > > the NumPy related tests use a file "numpy_common.pxi" that contains this > useless code: > > """ > cdef extern from *: > bint FALSE "0" > void import_array() > void import_umath() > > if FALSE: > import_array() > import_umath() > """ > > Does this serve any purpose? It currently leads to build failures of the > tests in C++ mode because the "import_umath()" call seems to inline code > with a bare "return" statement, which is not allowed in the module init > function (which returns a reference to the module). > > Is there any reason why this can't just die? IIRC, this was to avoid warnings about unused functions when importing the numpy headers without manually invoking these functions. Perhaps taking their address rather than "calling" them would be better (though this may involve creating yet more unused variables...). - Robert ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
[Cython] Gsoc project
Hey
I got linked to your idea
http://groups.google.com/group/cython-users/browse_thread/thread/cb8aa58083173b97/cac3cf12d438b122?show_docid=cac3cf12d438b122&pli=1
by David Malcolm on his plugin mailing list.
I am looking to apply to Gsoc once again this year i have done gsoc
2010 and 2011 on GCC implementing my own GCC front-end for python
which is still in very early stages since its a huge task. But i am
tempted to apply to this project to implement a more self contained
project to give back to the community more promptly while that hacking
on my own front-end on my own timer. And i think it would benefit me
to get to understand in more detail different aspects of python which
is what i need and would gain very much experience from.
I was wondering if you could give me some more details on how this
could all work i am not 100% familiar with cython but i think i
understand it to a good extend from playing with it for most of my
evening. I just want to make sure i understand the basic use case of
this fully, When a user could have something like:
-header foo.h
extern int add (int, int);
-source foo.c
#include "foo.h"
int add (int x, int y)
{
return x+y;
}
We use the plugin to go over the decls created and create a pxd file like:
cdef int add (int a, int b):
return a + b
Although this is a really basic example i just want to make sure i
understand whats going on. Maybe some more of you have input? I guess
this would be best suited as a proposal for Python rather than GCC?
--Phil
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Gsoc project
On 03/27/2012 02:17 PM, Philip Herron wrote:
Hey
I got linked to your idea
http://groups.google.com/group/cython-users/browse_thread/thread/cb8aa58083173b97/cac3cf12d438b122?show_docid=cac3cf12d438b122&pli=1
by David Malcolm on his plugin mailing list.
I am looking to apply to Gsoc once again this year i have done gsoc
2010 and 2011 on GCC implementing my own GCC front-end for python
which is still in very early stages since its a huge task. But i am
tempted to apply to this project to implement a more self contained
project to give back to the community more promptly while that hacking
on my own front-end on my own timer. And i think it would benefit me
to get to understand in more detail different aspects of python which
is what i need and would gain very much experience from.
Excellent! After talking to lots of people at PyCon about Cython, it is
obvious that auto-generation of pxd files is *the* most missed feature
in Cython today. If you do this, lots of Cython users will be very grateful.
I was wondering if you could give me some more details on how this
could all work i am not 100% familiar with cython but i think i
understand it to a good extend from playing with it for most of my
evening. I just want to make sure i understand the basic use case of
this fully, When a user could have something like:
-header foo.h
extern int add (int, int);
-source foo.c
#include "foo.h"
int add (int x, int y)
{
return x+y;
}
We use the plugin to go over the decls created and create a pxd file like:
cdef int add (int a, int b):
return a + b
Although this is a really basic example i just want to make sure i
understand whats going on. Maybe some more of you have input? I guess
this would be best suited as a proposal for Python rather than GCC?
This isn't quite what should be done. Cython generates C code that
includes C header files; what the pxd files are needed for is to provide
declarations for Cython about what is available on the C side (during
the Cython->C translation/compilation).
So: "foo.c" is irrelevant to Cython. And, foo.h should turn into foo.pxd
like this:
cdef extern from "foo.h":
int add(int, int)
Let us know if you have any question; you may want to look at examples
for using Cython to wrap C code, such as
https://github.com/zeromq/pyzmq/blob/master/zmq/core/libzmq.pxd
and the rest of the pyzmq code.
Moving over to the idea of making this a GSoC:
First, we have a policy of requiring patches from prospective students
in addition to their application. Often, this has been to fix a bug or
two in Cython. However, given that pxd generation can be done without
much digging into Cython itself, I think that something like a crude
prototype of the pxd generator (supporting only a subset of C) would be
a better fit (other devs, what do you think?)
The project should contain at least:
- The wrapper generator itself
- Tests for it (including the task of figuring out how to test this,
possibly both unit tests and integration tests)
- A strategy for testing it for all relevant versions of gcc; one
should probably set up Jenkins jobs for it
Even then, I feel that this is rather small for a full GSoC, even when
supporting the subset of C++ supported by Cython, I would estimate a
month or so (and GSoC is two months). So it should be extended in one
direction or another. Some ideas:
- Very often one is not interested in the full header file. One really
wants "the API", not a translation of the C header. This probably
requires a) some heuristics, and b) the possibility for, as easily as
possible, write some selectors/configuration for what should be included
and not. Making that end-user-friendly is perhaps a challenge, I'm not sure.
One idea here is to make possible an interplay where you look at the pyx
file what needs to be wrapped. I.e. you first try to use a function in
the pyx file as if it had already been declared, then run the pxd
generator feeding in the pyx files (and .h files), and out comes the
required pxd file bridging the two (containing only the used subset).
- Support using clang to parse C code in addition
- There's a problem in that an often-used Cython approach is:
1) Generate C file from pyx and pxd files
2) Ship to other computers
3) Compile C file
However, this is fragile when combined with auto-generated pxd files,
because the resulting pxd may be different depending on whether -DFOO is
given to gcc or not.
The above 3 steps are possible because Cython often does not care about
the exact type of something, just basic type and signedness. So if you do
cdef extern from "foo.h":
ctypedef int sometype_t
then sometype_t can actually be a short or a char, and Cython doesn't
care. (Similarly, not all fields of a struct needs to be exposed, only
the ones that form part of the API.)
However, I'm not sure if the quality of an auto-generated pxd file is
good enough for this approach.
So either a) th
Re: [Cython] Gsoc project
On 03/27/2012 08:05 PM, Dag Sverre Seljebotn wrote:
On 03/27/2012 02:17 PM, Philip Herron wrote:
Hey
I got linked to your idea
http://groups.google.com/group/cython-users/browse_thread/thread/cb8aa58083173b97/cac3cf12d438b122?show_docid=cac3cf12d438b122&pli=1
by David Malcolm on his plugin mailing list.
I am looking to apply to Gsoc once again this year i have done gsoc
2010 and 2011 on GCC implementing my own GCC front-end for python
which is still in very early stages since its a huge task. But i am
tempted to apply to this project to implement a more self contained
project to give back to the community more promptly while that hacking
on my own front-end on my own timer. And i think it would benefit me
to get to understand in more detail different aspects of python which
is what i need and would gain very much experience from.
Excellent! After talking to lots of people at PyCon about Cython, it is
obvious that auto-generation of pxd files is *the* most missed feature
in Cython today. If you do this, lots of Cython users will be very
grateful.
I was wondering if you could give me some more details on how this
could all work i am not 100% familiar with cython but i think i
understand it to a good extend from playing with it for most of my
evening. I just want to make sure i understand the basic use case of
this fully, When a user could have something like:
-header foo.h
extern int add (int, int);
-source foo.c
#include "foo.h"
int add (int x, int y)
{
return x+y;
}
We use the plugin to go over the decls created and create a pxd file
like:
cdef int add (int a, int b):
return a + b
Although this is a really basic example i just want to make sure i
understand whats going on. Maybe some more of you have input? I guess
this would be best suited as a proposal for Python rather than GCC?
This isn't quite what should be done. Cython generates C code that
includes C header files; what the pxd files are needed for is to provide
declarations for Cython about what is available on the C side (during
the Cython->C translation/compilation).
So: "foo.c" is irrelevant to Cython. And, foo.h should turn into foo.pxd
like this:
cdef extern from "foo.h":
int add(int, int)
Let us know if you have any question; you may want to look at examples
for using Cython to wrap C code, such as
https://github.com/zeromq/pyzmq/blob/master/zmq/core/libzmq.pxd
and the rest of the pyzmq code.
Moving over to the idea of making this a GSoC:
First, we have a policy of requiring patches from prospective students
in addition to their application. Often, this has been to fix a bug or
two in Cython. However, given that pxd generation can be done without
much digging into Cython itself, I think that something like a crude
prototype of the pxd generator (supporting only a subset of C) would be
a better fit (other devs, what do you think?)
The project should contain at least:
- The wrapper generator itself
- Tests for it (including the task of figuring out how to test this,
possibly both unit tests and integration tests)
- A strategy for testing it for all relevant versions of gcc; one should
probably set up Jenkins jobs for it
Even then, I feel that this is rather small for a full GSoC, even when
supporting the subset of C++ supported by Cython, I would estimate a
month or so (and GSoC is two months). So it should be extended in one
direction or another. Some ideas:
I should stress that even if you only include the above in the proposal,
it would definitely still get consideration. It may well be better to go
slowly but creating something rock solid, than having lots of bells and
whistles.
It is also possible to label the above the core features, and whatever
you decide on in addition as "optional bonus goals" in your proposal.
Dag
- Very often one is not interested in the full header file. One really
wants "the API", not a translation of the C header. This probably
requires a) some heuristics, and b) the possibility for, as easily as
possible, write some selectors/configuration for what should be included
and not. Making that end-user-friendly is perhaps a challenge, I'm not
sure.
One idea here is to make possible an interplay where you look at the pyx
file what needs to be wrapped. I.e. you first try to use a function in
the pyx file as if it had already been declared, then run the pxd
generator feeding in the pyx files (and .h files), and out comes the
required pxd file bridging the two (containing only the used subset).
- Support using clang to parse C code in addition
- There's a problem in that an often-used Cython approach is:
1) Generate C file from pyx and pxd files
2) Ship to other computers
3) Compile C file
However, this is fragile when combined with auto-generated pxd files,
because the resulting pxd may be different depending on whether -DFOO is
given to gcc or not.
The above 3 steps are possible because Cython often does not care about
the exact type of something, just basic type and sig
Re: [Cython] Question on "numpy_common.pxi" in NumPy tests
Lisandro Dalcin, 27.03.2012 21:34: > On 27 March 2012 17:20, Stefan Behnel wrote: >> the NumPy related tests use a file "numpy_common.pxi" that contains this >> useless code: >> >> """ >> cdef extern from *: >> bint FALSE "0" >> void import_array() >> void import_umath() >> >> if FALSE: >>import_array() >>import_umath() >> """ >> >> Does this serve any purpose? It currently leads to build failures of the >> tests in C++ mode because the "import_umath()" call seems to inline code >> with a bare "return" statement, which is not allowed in the module init >> function (which returns a reference to the module). >> >> Is there any reason why this can't just die? > > Oh! long long ago Cython testsuite was able to run without a single C > compiler warning... That numpy common file was a hack to silent > "defined but not used" warnings about the import array/umath functions > defined in NumPy headers. Ok - in that case, it should be enough to move the import calls into a cdef function body and "avoiding" to call that. And a comment in the file would be helpful ... Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
