[Cython] Cython magic annotate option is broken under IPython 3.0
Hi all, To reproduce this issue, be on cython 0.22 and IPython 3.0 and run the following notebook: http://nbviewer.ipython.org/gist/ngoldbaum/855a629d997aa7959254 Googling the error returns some several year old discussions in IPython related to the autoreload magic and I'm not sure if that is a red herring here: http://mail.scipy.org/pipermail/ipython-dev/2012-May/009288.html For how I'm working around this by calling cython at the command line in my notebook and then loading the annotated HTML with an IFrame. Thanks for your help! ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] cygdb: no debug files were found
Hi all, I'm following the directions in the link below to debug some cython code using cygdb: http://docs.cython.org/src/userguide/debugging.html Unfortunately, when I try to actually debug my program using cygdb, I get a "no debug files were found" error. It looks like the error is getting raised here: https://github.com/cython/cython/blob/master/Cython/Debugger/Cygdb.py#L33 So for some reason the "cython_debug" folder isn't being generated at build time. I tried to follow the directions closely, you can see the notes I put together during the installation process here: https://bpaste.net/show/6b11f815d449 Unfortunately the library I'm trying to debug (yt: bitbucket.org/yt_analysis/yt) uses a somewhat nonstandard build setup based on numpy distutils, and I suspect that is breaking some of the assumptions that went into the cygdb instructions in the cython docs. Does anyone have any idea what went wrong here? For now I'm going to fall back to regular gdb, and debug the autogenerated cython sources. Thanks for your help, Nathan Goldbaum ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] 'Class' redeclared error if C++ class & file name match those of the wrapper class
Hi all,
I'm forwarding this at the request of David Nemesky ("savior" on Freenode),
who I've been chatting with on the cython IRC channel. He was having
trouble signing up to the cython-devel mailing list, so I've gone ahead and
sent his bug report in for him. His report follows below.
Apparently this error happens under very specific circumstances, but I
managed to run into them. When wrapping a C++ class, Cython reports a
"'ClassName' redeclared" error, if:
- the name of the .pyx is the same as the name of the .pxd
- the name of the python class is the same as the name of the C++ class
Mismatch in either the file- or the class name results in a successful
compilation.
I have modified my fork of the cython-cpp-test repo to make the issue
easily reproducible. In https://github.com/DavidNemeskey/cython-cpp-test/,
try `python setup.py build_ext --inplace` (do not forget to delete all .cpp
and .so files between calls) in the following branches:
- cython_bug (diff file, diff class) -> OK
- diff_file_same_class -> OK
- same_file_diff_class -> OK
- same_file_same_class -> ERROR (https://bpaste.net/show/2c1af5fa5ee2)
Since Cython compiles xxx.pyx to xxx.cpp in this case, in order to avoid
problems with testclass.cpp being overwritten, I implemented the class in
testclass.hpp (in the original repo, there was a .h and a .cpp file).
However, implementing everything in the header is not a pre-requisite of
the bug, as is evident from branch same_file_same_class_diff_c.
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Static checker for cython extension dependencies?
Hi all, I'm working on a pretty large python/cython codebase (yt) that has many interdependent C extensions written in cython. I've found it to be pretty hit or miss to depend on contributors to manually update cython dependency information in our setup.py file. The dependency information seems to only be used by setuptools to trigger recompilation, but critically setuptools will happily compile a C extension for the first time if the depends information is incomplete. This means that during development, if I update a cython routine, there's no way to ensure that other cython routines that cimport the one I modified will be recompiled unless I manually ensure the depends information is updated whenever cython code gains or loses a cimport. To make that more concrete, here's a pull request I just made to yt that adds missing dependencies for a cython header. Without this pull request, setuptools fails to recompile these routines when selection_routines.pxd changes, causing a build failure. https://bitbucket.org/yt_analysis/yt/pull-requests/2220 I think it should be possible to write a test for this by defining the dependency information outside of setup.py and parsing grep and looking for all cython files that cimport other cython files defined inside yt. However, before I do that, I'm curious whether anyone has done something similar, or if there is some other way of forcing the dependency information to be complete on the first compilation, rather than just for subsequent incremental recompilations during development. Thanks for your help! -Nathan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Static checker for cython extension dependencies?
The reason we haven't done that is we would like our setup.py script to be runnable without cython being installed. I think cythonize is being invoked (or something similar to it) by setuptools, using a feature added in setuptools 18.0: https://setuptools.readthedocs.io/en/latest/history.html#id60 Is there a way to use cythonize for this build workflow without importing it at the top-level in our setup.py file? FWIW, our setup.py file is here: https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw wrote: > You should be using cythonize rather than listing and maintaining the > Extension definitions themselves. > > > http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils > https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing > > On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum > wrote: > > Hi all, > > > > I'm working on a pretty large python/cython codebase (yt) that has many > > interdependent C extensions written in cython. > > > > I've found it to be pretty hit or miss to depend on contributors to > manually > > update cython dependency information in our setup.py file. The dependency > > information seems to only be used by setuptools to trigger recompilation, > > but critically setuptools will happily compile a C extension for the > first > > time if the depends information is incomplete. This means that during > > development, if I update a cython routine, there's no way to ensure that > > other cython routines that cimport the one I modified will be recompiled > > unless I manually ensure the depends information is updated whenever > cython > > code gains or loses a cimport. > > > > To make that more concrete, here's a pull request I just made to yt that > > adds missing dependencies for a cython header. Without this pull request, > > setuptools fails to recompile these routines when selection_routines.pxd > > changes, causing a build failure. > > > > https://bitbucket.org/yt_analysis/yt/pull-requests/2220 > > > > I think it should be possible to write a test for this by defining the > > dependency information outside of setup.py and parsing grep and looking > for > > all cython files that cimport other cython files defined inside yt. > However, > > before I do that, I'm curious whether anyone has done something similar, > or > > if there is some other way of forcing the dependency information to be > > complete on the first compilation, rather than just for subsequent > > incremental recompilations during development. > > > > Thanks for your help! > > > > -Nathan > > > > ___ > > cython-devel mailing list > > cython-devel@python.org > > https://mail.python.org/mailman/listinfo/cython-devel > > > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Static checker for cython extension dependencies?
Hmm, so I've looked into this a bit further, and it looks like the metadata isn't going to be useful for us. Many of our extensions can't be autogenerated by cython because they have non-trivial dependencies or depend directly on C code. For these extensions, cythonize must be passed an instantiated Extension object, and the metadata embedded in the compiled C file just uses whatever metadata is attached to the Extension object. I think my original idea - to parse pyx files for cimports and add a test - should allow us to manage this more safely in the future while still maintaining fine-grained control over dependencies of C extensions in our setup.py file. That said, I'd love to hear alternate ideas. On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw wrote: > We write metadata in the generated C files for just this reason. You > can fall back to something like > https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if > cython is not available. > > > On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum > wrote: > > The reason we haven't done that is we would like our setup.py script to > be > > runnable without cython being installed. I think cythonize is being > invoked > > (or something similar to it) by setuptools, using a feature added in > > setuptools 18.0: > > https://setuptools.readthedocs.io/en/latest/history.html#id60 > > > > Is there a way to use cythonize for this build workflow without > importing it > > at the top-level in our setup.py file? > > > > FWIW, our setup.py file is here: > > > https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default > > > > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw > > wrote: > >> > >> You should be using cythonize rather than listing and maintaining the > >> Extension definitions themselves. > >> > >> > >> > http://docs.cython.org/src/quickstart/build.html#building-a-cython-module-using-distutils > >> > https://github.com/cython/cython/wiki/enhancements-distutils_preprocessing > >> > >> On Fri, Jun 10, 2016 at 9:18 AM, Nathan Goldbaum > > >> wrote: > >> > Hi all, > >> > > >> > I'm working on a pretty large python/cython codebase (yt) that has > many > >> > interdependent C extensions written in cython. > >> > > >> > I've found it to be pretty hit or miss to depend on contributors to > >> > manually > >> > update cython dependency information in our setup.py file. The > >> > dependency > >> > information seems to only be used by setuptools to trigger > >> > recompilation, > >> > but critically setuptools will happily compile a C extension for the > >> > first > >> > time if the depends information is incomplete. This means that during > >> > development, if I update a cython routine, there's no way to ensure > that > >> > other cython routines that cimport the one I modified will be > recompiled > >> > unless I manually ensure the depends information is updated whenever > >> > cython > >> > code gains or loses a cimport. > >> > > >> > To make that more concrete, here's a pull request I just made to yt > that > >> > adds missing dependencies for a cython header. Without this pull > >> > request, > >> > setuptools fails to recompile these routines when > selection_routines.pxd > >> > changes, causing a build failure. > >> > > >> > https://bitbucket.org/yt_analysis/yt/pull-requests/2220 > >> > > >> > I think it should be possible to write a test for this by defining the > >> > dependency information outside of setup.py and parsing grep and > looking > >> > for > >> > all cython files that cimport other cython files defined inside yt. > >> > However, > >> > before I do that, I'm curious whether anyone has done something > similar, > >> > or > >> > if there is some other way of forcing the dependency information to be > >> > complete on the first compilation, rather than just for subsequent > >> > incremental recompilations during development. > >> > > >> > Thanks for your help! > >> > > >> > -Nathan > >> > > >> > ___ > >> > cython-devel mailing list > >> > cython-devel@python.org > >> > https://mail.python.org/mailman/listinfo/cython-devel > >> > > >> ___ > >> cython-devel mailing list > >> cython-devel@python.org > >> https://mail.python.org/mailman/listinfo/cython-devel > > > > > > > > ___ > > cython-devel mailing list > > cython-devel@python.org > > https://mail.python.org/mailman/listinfo/cython-devel > > > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Static checker for cython extension dependencies?
On Fri, Jun 10, 2016 at 5:04 PM, Robert Bradshaw wrote: > On Fri, Jun 10, 2016 at 1:18 PM, Nathan Goldbaum > wrote: > > Hmm, so I've looked into this a bit further, and it looks like the > metadata > > isn't going to be useful for us. Many of our extensions can't be > > autogenerated by cython because they have non-trivial dependencies > > meaning? In my cursory glance, these all look fine, but perhaps you > could point me at a case that isn't very nice. Also, cythonize isn't > necessarily all-or-nothing--you can always manually add Extensions to > your list. > > or depend directly on C code. > > cythonize handles dependencies on C code just fine > > > For these extensions, cythonize must be passed an > > instantiated Extension object, and the metadata embedded in the compiled > C > > file just uses whatever metadata is attached to the Extension object. > > The embedded metadata is the union of what was in the Extension object > and the extra dependencies it found. > Hmm, maybe I made a mistake, but when I passed an Extension object to cythonize earlier, I found that the metadata that got written out to the autogenerated C only included the dependencies that were listed in the Extension object's declaration, not additional ones inferred by cython. Perhaps this is a bug? > > > I think my original idea - to parse pyx files for cimports and add a > test - > > should allow us to manage this more safely in the future while still > > maintaining fine-grained control over dependencies of C extensions in our > > setup.py file. > > > > That said, I'd love to hear alternate ideas. > > "Parse pyx files for cimports" (recursively, handing includes and cdef > extern from ...) is essentially re-implementing cythonize. What is the > advantage of maintaining them by hand? Cythonize was invented because > people are really bad at maintaining these lists manually...and they > can be entirely deduced. The advantage here is that in my couple hours of playing with using cythonize for our setup.py script, I couldn't get it to correctly generate the metadata I needed to validate the dependencies, so it wasn't providing any added value. Again, it's entirely possible I was making a mistake somewhere. Also, separately, the fake_cythonize script you suggested will error out if cython isn't installed and the cython-generated C files do not exist yet. That certainly makes sense if you expect developers to have cython available, but it's a downgrade with respect to our current setup. Right now we have it set up so that setuptools will install cython into the build environment while it is processing C extensions if it is not already installed. This means if new developers or users come along and want to build yt from source, they won't need to explicitly install cython first - setuptools will take care of it for us. This makes our installation instructions simpler and causes fewer head-scratching crashes when people try to compile yt from source. > > > On Fri, Jun 10, 2016 at 1:49 PM, Robert Bradshaw > wrote: > >> > >> We write metadata in the generated C files for just this reason. You > >> can fall back to something like > >> https://gist.github.com/robertwb/25ab9838cc2b9b21eed646834cf4a108 if > >> cython is not available. > >> > >> > >> On Fri, Jun 10, 2016 at 10:55 AM, Nathan Goldbaum < > nathan12...@gmail.com> > >> wrote: > >> > The reason we haven't done that is we would like our setup.py script > to > >> > be > >> > runnable without cython being installed. I think cythonize is being > >> > invoked > >> > (or something similar to it) by setuptools, using a feature added in > >> > setuptools 18.0: > >> > https://setuptools.readthedocs.io/en/latest/history.html#id60 > >> > > >> > Is there a way to use cythonize for this build workflow without > >> > importing it > >> > at the top-level in our setup.py file? > >> > > >> > FWIW, our setup.py file is here: > >> > > >> > > https://bitbucket.org/yt_analysis/yt/src/yt/setup.py?at=yt&fileviewer=file-view-default > >> > > >> > On Fri, Jun 10, 2016 at 12:49 PM, Robert Bradshaw > > >> > wrote: > >> >> > >> >> You should be using cythonize rather than listing and maintaining the > >> >> Extension definitions themselves. > >> >> > >> >> > >> >> > >> >> > http://docs.cython.org/src/quickstart/build.html#
Re: [Cython] Next Cython Release
Cython uses github for bug tracking these days: https://github.com/cython/cython/issues On Sun, Jun 11, 2017 at 10:54 AM Julian Rüth wrote: > Hello Cython developers, > > some people over at conda-forge would like to use features that are > already in master but have not been released yet > (https://github.com/conda-forge/cython-feedstock/pull/20). We would > prefer not to patch and older release for Cython but just wait for the > next release. However, since there has not been a release in 6 months, > we're wondering if there are any plans for a release? > > julian > > PS: Not sure if this is the right list for this but the link to the > bug-tracker on https://mail.python.org/mailman/listinfo/cython-devel > goes to http://cython.org/trac/cython_trac/ which produces a 404. > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Next Cython Release
Hearty +1 to getting a cython release out. I'd like to get this fix which will substantially speed up some code I've written on multicore systems: https://github.com/cython/cython/commit/8dbc7c39d1f1a3b675f69dceaac281 7b7675ca09 -Nathan On Mon, Jun 12, 2017 at 1:00 PM, Stefan Behnel wrote: > Robert Bradshaw schrieb am 12.06.2017 um 19:35: > > On Sat, Jun 10, 2017 at 10:34 AM, Julian Rüth wrote: > >> some people over at conda-forge would like to use features that are > >> already in master but have not been released yet > >> (https://github.com/conda-forge/cython-feedstock/pull/20). We would > >> prefer not to patch and older release for Cython but just wait for the > >> next release. However, since there has not been a release in 6 months, > >> we're wondering if there are any plans for a release? > > > > I've been wanting to do a release for a while, just haven't gotten > > around to it. Yes, let's get one out soon. > > I've restarted working on the integration of PEP 525 and 530 (async > generators/comprehensions), which gets closer but isn't quite there yet. > Since we didn't have a release for so long now, and we certainly have > enough on our current list of changes, I'd be ok with having it miss this > release. So, +1 for a release soon. > > > >> PS: Not sure if this is the right list for this but the link to the > >> bug-tracker on https://mail.python.org/mailman/listinfo/cython-devel > >> goes to http://cython.org/trac/cython_trac/ which produces a 404. > > > > As mentioned, we're using github these days. I added redirects > > (including specific tickets like http://cython.org/trac/ticket/101 ) > > but seemed to have missed this link, thanks for pointing that out. > > I already updated that link. Sorry, should have mentioned it. > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Should we start requiring "language_level=2" for .pyx files?
It would help if cython could generate compiler warnings for a release or two about py2-specific constructs before officially switching the default language level. On Mon, May 28, 2018 at 2:14 PM Stefan Behnel wrote: > Hi, > > Python 3 is clearly taking over the world these days, so it starts feeling > arcane to require Py2 syntax in .pyx files. Increasingly, it means that > people cannot just rename .py files anymore to start optimising them, > because the .py file has a high chance of being written in Py3 syntax. > > Eventually, we will have to switch to Py3 syntax by default in order to > follow what most people are (or will be) used to. > > As a transition, I think we could start warning about cases where the > language level is not set explicitly. If people start marking their code as > being "Cython 2.x code", either with an in-file directive or from their > setup.py, we will have less of a problem in the future to change the > default. > > What do you think? Any other ideas, comments, objections? > > Stefan > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] #cython on freenode
Hi all, The Freenode IRC network is currently undergoing a spam attack that is affecting the #cython channel there. People definitely do use #cython to ask questions. I and some others try and help them out. It would be a shame to have that community, such as it is, get destroyed by the spam attack. Freenode has a way for a project to take control of the IRC channel associated with their name by mailing proje...@freenode.net and working with an admin to prove who you say you are. For matplotlib they asked Tom Caswell to push a branch with a randomly chosen name to github. If someone associated with the project is willing to do something like that, I would be very happy to volunteer to admin the channel and set the channel settings to mitigate the spam attack. I'm currently doing a similar process to get admin powers for #ipython and #matplotlib and already have admin powers in #scipy and #numpy. Thanks for your help, -Nathan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] #cython on freenode
Hi Robert, That’s great. Let me know if you need a hand with anything. Feel free to ping me in #cython. Also in the meantime I managed to get the attention of a freenode admin who changed the channel settings so the spam problem is mostly resolved for now. It would still be good to have the channel registered with the project and to set up admins. Nathan On Mon, Aug 13, 2018 at 12:06 PM Abdur-Rahmaan Janhangeer < arj.pyt...@gmail.com> wrote: > the admin of #python-fr sent me this resource, will help : > > https://nedbatchelder.com/blog/201808/fighting_spam_on_freenode.html > > > Abdur-Rahmaan Janhangeer > https://github.com/Abdur-rahmaanJ > Mauritius > > On Sat, 11 Aug 2018, 01:43 Robert Bradshaw, wrote: > >> Thanks for the heads up. I'd be happy to help with this. >> >> On Thu, Aug 9, 2018, 12:55 AM Nathan Goldbaum >> wrote: >> >>> Hi all, >>> >>> The Freenode IRC network is currently undergoing a spam attack that is >>> affecting the #cython channel there. >>> >>> People definitely do use #cython to ask questions. I and some others try >>> and help them out. It would be a shame to have that community, such as it >>> is, get destroyed by the spam attack. >>> >>> Freenode has a way for a project to take control of the IRC channel >>> associated with their name by mailing proje...@freenode.net and working >>> with an admin to prove who you say you are. For matplotlib they asked Tom >>> Caswell to push a branch with a randomly chosen name to github. If someone >>> associated with the project is willing to do something like that, I would >>> be very happy to volunteer to admin the channel and set the channel >>> settings to mitigate the spam attack. I'm currently doing a similar process >>> to get admin powers for #ipython and #matplotlib and already have admin >>> powers in #scipy and #numpy. >>> >>> Thanks for your help, >>> >>> -Nathan >>> ___ >>> cython-devel mailing list >>> cython-devel@python.org >>> https://mail.python.org/mailman/listinfo/cython-devel >>> >> ___ >> cython-devel mailing list >> cython-devel@python.org >> https://mail.python.org/mailman/listinfo/cython-devel >> > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] OpenMP 4.5 array reductions
Hi all, I'm curious if there would be any interest in adding support for OpenMP 4.5 array reduction in the cython compiler or alternatively detecting these cases and raising a cython compiler error. Currently cython is generating code that will compile but might lead to race conditions. See: https://github.com/cython/cython/issues/2316 https://github.com/cython/cython/issues/1504 The trouble with fixing this in the cython compiler is that adding the appropriate OpenMP pragmas might generate code that will no longer compile on compilers that don't support OpenMP 4.5. However perhaps that's a better alternative than the status quo, which is generating code that might produce random results. I'd very much appreciate any feedback or advice here as this is currently blocking our ability to easily add OpenMP to our cython code in places where we'd like threads to do parallel reductions on large arrays. I would also not be surprised if there is code in the wild that is racy and silently producing incorrect results. -Nathan ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
