Hi Stefan, On Thu, 2013-10-10 at 08:49 +0200, Stefan Behnel wrote: > > Yury V. Zaytsev, 09.10.2013 15:01: > > I've been playing with my Cython-generated extension on PyPy, trying to > > load it through CPyExt, and, surprisingly, after a few fixes to PyPy, it > > generally seems to work. > > Yes, we invested quite a bit of work into that already. Generally > speaking, my drive for supporting PyPy has substantially slowed down > during the last year or so. The interest on PyPy side in doing even > the most obvious optimisations was close to non-existant, which made > it essentially uninteresting to keep working on it on our side.
I obviously don't have the full picture, but when I was hanging around #pypy while trying to get my Cython-generated extension to work, everybody seemed to be extremely helpful, so I've got the opposite impression. I can only speculate that maybe these fixes from their side required them to expose implementation details that are bound to change, or maybe they were afraid that performance fixes will encourage people to blindly offload stuff to Cython, instead fixing their Python code or PyPy JIT, or using cffi for extremely high performance... Nevertheless, I think that there is a very valid use for Cython in conjunction with PyPy, and good compatibility and acceptable performance will certainly benefit both projects. For instance, the project that I'm working on right now involves automatic Python bindings generation with Cython, and 99% of the work actually happens inside the C++ code that I'm wrapping, so I don't really care about the performance of the bindings all that much. What I do care about however is that I don't have to write crazy cffi code (wrapping templated C++!) by hand, and with Cython my extension is now only a hundred lines of code and works perfectly across the whole zoo of CPythons... Having it to work with PyPy would be a serious benefit for us, because then we could write most of the complex processing code in pure Python and still use the same bindings as if we were using CPython. On top of that, it would be truly awesome if we could write callbacks in Python and have them run at speed with PyPy, but I guess that's going to be difficult to make it fast, if these callbacks will be invoked a lot. > The C-internals of the array.array module are private even in CPython, so > I'm not surprised that PyPy doesn't expose them. The normal buffer > interface would generally be enough, though. Hmmm... then, maybe I don't understand how array.array module is handled in Cython at the moment. From what I could figure, Python 3 supports new-style buffer interface to array.array (which is unavailable in PyPy, see below), but Cython also has some code to handle older versions of Python. That's what I was referring to with "internals". It could be that I'm completely confused. Never mind. What matters is that array.array Cython-generated code just doesn't work with PyPy CPyExt at the moment... > > Second, they say that the new-style buffer access is broken and it's not > > clear if they will support it at all & when this is going to happen. > > What do you mean by "broken"? Currently broken in PyPy? I guess it primarily means that it's broken in PyPy right now, but they also claim that this interface is problematic for some reasons and therefore also 'broken' in terms of design. > Would you have a more complete quote? I couldn't find a discussion on > the pypy mailing list about this on a quick look. Sorry, I don't have a complete quote. This information was conveyed to me by fijal on the IRC. When I asked for clarification, he said that it's not that they are going to prevent anyone from implementing it, but right now it doesn't work, and they don't have motivation to fix it because of its design-wise 'brokenness'. I guess he can easily explain you what's so 'broken' about it, but it doesn't make sense for me to serve as an intermediary between you and them, because I poorly understand the subject and will only confuse everything even further... > > Third, apparently, there is already some limited support for NumPy C-API > > in NumPyPy, and it seems that there is interest in improving this > > support to make it usable: > > > > http://docs.scipy.org/doc/numpy/reference/c-api.array.html > > Interesting. Cython has legacy support for older NumPy versions through > their C-API instead of the buffer interface. We could enable that for PyPy. > Might need some minor adaptations, but might still be the easiest way to > get it working. I think so too! > > Finally, GetItem on NumPyPy arrays, which will be very slow, but at > > least should work, rather than not, is also broken. I think this will be > > easiest of all to fix. > > How is it broken? Where should it be fixed? It's broken in a way that it segfaults PyPy. I've created a reproducer, it turned out to be a PyPy problem, and they are working on a fix: https://bugs.pypy.org/issue1621 I guess it will hit PyPy main branch sometime soon. There is nothing to be done about it in Cython. I just didn't get that far at the time of writing... > It means that we don't currently have a working installation of a recent > PyPy on our build server. The Linux builds they provide depend on fairly > recent system libraries and I consider building PyPy ourselves way too much > overhead. Also, running the tests took ages because CPyExt is so slow, so > the whole setup became useless after a while. I see the problem. Yes, I also can't use their nightlies for the same reason. I figured out how to translate it myself though, so that's what I've been using so far. > If you have a server lying around somewhere that would allow you to run > nightly tests of Cython against nightly builds of PyPy, I certainly > wouldn't mind if you set it up for it. Yes, I have a build server which has hopefully recent enough libraries so that I could use stock PyPy nightlies on it without retranslating them myself. I can set up a build that tests Cython against latest PyPy nightly, but this will take some effort. Of course, it would be very nice to know, that this effort will lead to the improvement of Cython, and not just go to the trash bin :-) In any case, would it be possible for me to at least get your build scripts? I don't seem to be able to access them on Jenkins without an account. To start off with something, I did a small test run on my laptop of Cython 0.19-438-ge36fc99 on manually translated PyPy with GetItem fix: http://vps.zaytsev.net/~zaytsev/pypy/cython-pypy-2013-10-10.log I hope that this can give you an overview of what's going on with CPyExt at the moment... Is that useful? If not, how I can make it more useful? -- Sincerely yours, Yury V. Zaytsev _______________________________________________ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
