Re: [Cython] Fwd: Question about how best require compiler options for C sources
On Mon, Apr 11, 2016 at 7:49 PM, Ian Henriksen wrote: > On Mon, Apr 11, 2016 at 7:46 AM Erik Bray wrote: >> >> On Mon, Apr 11, 2016 at 2:51 PM, Nathaniel Smith wrote: >> > On Apr 11, 2016 04:18, "Erik Bray" wrote: >> >> >> >> On Fri, Apr 8, 2016 at 5:49 PM, Nathaniel Smith wrote: >> >> > Can you give a tiny concrete example? My questions are basic enough >> >> > that >> >> > I >> >> > feel like I'm missing something fundamental :-) >> >> >> >> Yes, I think you might be missing something, but I'm not sure exactly >> >> where. In the issue I provided a tiny example which you can see here: >> >> >> >> https://gist.github.com/embray/12a67edb82b213217e31f408007898e6 >> >> >> >> The C code generated by this example currently does not compile on >> >> Windows, because of how Cython uses DL_IMPORT incorrectly. Regardless >> >> of what it *does* do, Cython should not be using the DL_IMPORT macro >> >> at all actually since that no longer even exists in Python. >> > >> > Sure. >> > >> > For that example, the correct thing to do is to *not* export the >> > function. >> >> So as I wrote in my previous message, my example was incomplete. But >> indeed if the intent was *only* to share the declaration between TUs >> of the same library then I agree the *most* correct thing would be to >> not use dllexport. Unfortunately there isn't currently a way in Cython >> to make this distinction. >> >> > Backing up, to make sure we're on the same page: >> >> Yep, I agree with your analysis that follows, and it's helpful to have >> all the cases laid out in one place, so thanks for that! >> >> There are three levels of >> > symbol visibility in C: file-internal, shared library internal >> > (different .c >> > files that make up the library can see it, but users of the shared >> > library >> > can't see it), and shared library exported (everyone can see it; can >> > also >> > carry other consequences, e.g. on Linux then internal calls will become >> > noticeably slower, and it becomes possible for weird symbol >> > interposition >> > issues to occur). So the rule of thumb is to make everything as private >> > as >> > you can get away with. >> > >> > Making this more interesting: >> > - vanilla C only includes 2 ways to mark symbol visibility, which is not >> > enough to make a 3-way distinction. Hence the need for an extra >> > attribute >> > thingummy. >> > - everyone agrees that symbols marked 'static' should be file-internal, >> > but >> > different platforms disagree about what should happen if the extra >> > attribute >> > thingummy is missing. >> > >> > So on Windows, the convention is: >> > 'static' -> file internal >> > no marking -> shared library internal >> > 'dllexport' -> public >> > >> > And on Linux it's: >> > 'static' -> file internal >> > 'visibility (hidden)' -> shared library internal >> > no marking -> public >> > >> > It's generally agreed that Linux got this wrong and that you should >> > always >> > use '-fvisibility=hidden' to switch it to the windows style, but cython >> > doesn't control compiler options and thus should probably generate code >> > that >> > works correctly regardless of such compiler settings. Fortunately, Linux >> > does provide some markings to explicitly make things public: you can >> > mark a >> > symbol 'visibility (default)' (which means public), or you can use the >> > dllexport syntax, just like Windows, because gcc is helpful like that. >> > >> > OTOH, windows is annoying because of this dllimport thing that started >> > this >> > whole thread: on other systems just marking symbols as extern is enough >> > to >> > handle both shared-object-internal and shared-library-exported symbols, >> > and >> > the linker will sort it out. On Windows, you have to explicitly >> > distinguish >> > between these. (And annoyingly, if you accidentally leave out the >> > dllimport >> > making on functions then it will use some fallback hack that works but >> > silently degrades performance; on other symbols it just doesn't work, >> > and >> > ditto for if you use it when it isn't needed.) >> >> Yep. Agreed with all of the above. >> >> > So final conclusion: for non-static symbols, cython should first decide >> > whether they are supposed to be shared-library-internal or actually >> > exported >> > from the shared library. >> > >> > For shared-library-internal symbols: their definition should be marked >> > 'visibility(hidden)' on Linux, and unmarked on Windows. This is easy >> > using >> > some preprocessor gunk. (Or maybe simplest is: marked that everywhere >> > except >> > if using msvc, because I think everyone else will understand 'visibility >> > (hidden)' even if it's a no op.) Their declaration in the header file >> > should >> > just be 'extern' everywhere. >> > >> > For shared-library-exported symbols: I am dubious about whether cython >> > should even support these at all. But if it does, then the definitions >> > should be marked 'dllexport' (no macro trickery needed, because every
[Cython] Cygwin: Handling missing C99 long double functions
Hi again, Sorry if I'm spamming the list too much, but I've encountered another pretty serious and unfortunate issue with Cython on Cygwin. The problem now is that many of libm long double functions like sqrtl, tanl, etc. are missing on Cygwin (missing from newlib to be specific). I think this is a previously known issue, but nothing's ever really been done about it. Well, to be clear, sometimes they're present, but only when sizeof(double) == sizeof(long double). However on 64-bit Cygwin sizeof(long double) == 16, so the functions are simply not defined. This seems to be due to lack of interest / effort: https://www.cygwin.com/ml/cygwin/2011-04/msg00231.html That post is 5 years old, but I can't find any evidence that this has changed. There are quite a few tests in Cygwin's test suite that test long double support. I guess what I'm asking is what would be the best thing to do about it. I could just skip those tests on Cygwin, though I'm not sure the best way to go about skipping an entire test for a given platform. More generally though, outside the context of testing, this means Cygwin will sometimes generate code that cannot be compiled on this platform, and a question arises as to what to do about that. I have some thoughts, but am not sure if it's worth discussing any further or not. Thanks, Erik ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cygwin: Handling missing C99 long double functions
Hi, certainly we did something with Sage on cygwin to work around these... Just in case, Dima On Tue, Apr 26, 2016 at 3:58 PM, Erik Bray wrote: > Hi again, > > Sorry if I'm spamming the list too much, but I've encountered another > pretty serious and unfortunate issue with Cython on Cygwin. > > The problem now is that many of libm long double functions like sqrtl, > tanl, etc. are missing on Cygwin (missing from newlib to be specific). > I think this is a previously known issue, but nothing's ever really > been done about it. Well, to be clear, sometimes they're present, but > only when sizeof(double) == sizeof(long double). However on 64-bit > Cygwin sizeof(long double) == 16, so the functions are simply not > defined. > > This seems to be due to lack of interest / effort: > https://www.cygwin.com/ml/cygwin/2011-04/msg00231.html That post is 5 > years old, but I can't find any evidence that this has changed. > > There are quite a few tests in Cygwin's test suite that test long > double support. I guess what I'm asking is what would be the best > thing to do about it. > > I could just skip those tests on Cygwin, though I'm not sure the best > way to go about skipping an entire test for a given platform. > > More generally though, outside the context of testing, this means > Cygwin will sometimes generate code that cannot be compiled on this > platform, and a question arises as to what to do about that. I have > some thoughts, but am not sure if it's worth discussing any further or > not. > > Thanks, > Erik > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cygwin: Handling missing C99 long double functions
On Tue, Apr 26, 2016 at 5:16 PM, Dima Pasechnik wrote: > Hi, > certainly we did something with Sage on cygwin to work around these... > Just in case, >From what I can tell there are several places where Sage has hacked around this issue in different packages, but it's not doing anything specifically with it for Cython. Sage uses the Cephes math lib to support these functions on FreeBSD--and apparently used to use it on Cygwin too, but disabled that for some reason. Regardless, Cython should ultimately do something sensible in these cases. My general thinking is that in cases where Cython generates code containing C math functions, it ought to support a fallback. This will require some feature checks so that Cython can generate wrappers, when necessary, around the double versions of those functions (as Numpy currently does). Erik > On Tue, Apr 26, 2016 at 3:58 PM, Erik Bray wrote: >> Hi again, >> >> Sorry if I'm spamming the list too much, but I've encountered another >> pretty serious and unfortunate issue with Cython on Cygwin. >> >> The problem now is that many of libm long double functions like sqrtl, >> tanl, etc. are missing on Cygwin (missing from newlib to be specific). >> I think this is a previously known issue, but nothing's ever really >> been done about it. Well, to be clear, sometimes they're present, but >> only when sizeof(double) == sizeof(long double). However on 64-bit >> Cygwin sizeof(long double) == 16, so the functions are simply not >> defined. >> >> This seems to be due to lack of interest / effort: >> https://www.cygwin.com/ml/cygwin/2011-04/msg00231.html That post is 5 >> years old, but I can't find any evidence that this has changed. >> >> There are quite a few tests in Cygwin's test suite that test long >> double support. I guess what I'm asking is what would be the best >> thing to do about it. >> >> I could just skip those tests on Cygwin, though I'm not sure the best >> way to go about skipping an entire test for a given platform. >> >> More generally though, outside the context of testing, this means >> Cygwin will sometimes generate code that cannot be compiled on this >> platform, and a question arises as to what to do about that. I have >> some thoughts, but am not sure if it's worth discussing any further or >> not. >> >> Thanks, >> Erik >> ___ >> cython-devel mailing list >> cython-devel@python.org >> https://mail.python.org/mailman/listinfo/cython-devel > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cygwin: Handling missing C99 long double functions
On 2016-04-26 16:58, Erik Bray wrote: The problem now is that many of libm long double functions like sqrtl, tanl, etc. are missing on Cygwin Just to understand the issue better, can you elaborate how this impacts Cython? ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cygwin: Handling missing C99 long double functions
On Tue, Apr 26, 2016 at 8:36 AM, Erik Bray wrote: > On Tue, Apr 26, 2016 at 5:16 PM, Dima Pasechnik > wrote: > > Hi, > > certainly we did something with Sage on cygwin to work around these... > > Just in case, > > From what I can tell there are several places where Sage has hacked > around this issue in different packages, but it's not doing anything > specifically with it for Cython. Sage uses the Cephes math lib to > support these functions on FreeBSD--and apparently used to use it on > Cygwin too, but disabled that for some reason. > > Regardless, Cython should ultimately do something sensible in these cases. > > My general thinking is that in cases where Cython generates code > containing C math functions, it ought to support a fallback. This > will require some feature checks so that Cython can generate wrappers, > when necessary, around the double versions of those functions (as > Numpy currently does). > Let's make things concrete. You're complaining that something like cdef extern from "math.h": long double sqrtl(long double) def foo(long double x): return sqrtl(x) Doesn't work on Cygwin? The same is true for *any* C function that you use that's not totally portable (this is the bane of trying to use C). I don't think Cython should be detecting this and substituting a (less accurate) sqrt for sqrtl in this case. If you want do do this, write your own headers that (conditionally) define these things however you want. Or, are you complaining that Cython's test suite doesn't pass on some Cygwin because there are tests of features not available on Cygwin? (Your original email isn't clear.) If so, the test framework can be set up to exclude these tests on that platform. > > Erik > > > On Tue, Apr 26, 2016 at 3:58 PM, Erik Bray > wrote: > >> Hi again, > >> > >> Sorry if I'm spamming the list too much, but I've encountered another > >> pretty serious and unfortunate issue with Cython on Cygwin. > >> > >> The problem now is that many of libm long double functions like sqrtl, > >> tanl, etc. are missing on Cygwin (missing from newlib to be specific). > >> I think this is a previously known issue, but nothing's ever really > >> been done about it. Well, to be clear, sometimes they're present, but > >> only when sizeof(double) == sizeof(long double). However on 64-bit > >> Cygwin sizeof(long double) == 16, so the functions are simply not > >> defined. > >> > >> This seems to be due to lack of interest / effort: > >> https://www.cygwin.com/ml/cygwin/2011-04/msg00231.html That post is 5 > >> years old, but I can't find any evidence that this has changed. > >> > >> There are quite a few tests in Cygwin's test suite that test long > >> double support. I guess what I'm asking is what would be the best > >> thing to do about it. > >> > >> I could just skip those tests on Cygwin, though I'm not sure the best > >> way to go about skipping an entire test for a given platform. > >> > >> More generally though, outside the context of testing, this means > >> Cygwin will sometimes generate code that cannot be compiled on this > >> platform, and a question arises as to what to do about that. I have > >> some thoughts, but am not sure if it's worth discussing any further or > >> not. > >> > >> Thanks, > >> Erik > >> ___ > >> cython-devel mailing list > >> cython-devel@python.org > >> https://mail.python.org/mailman/listinfo/cython-devel > > ___ > > cython-devel mailing list > > cython-devel@python.org > > https://mail.python.org/mailman/listinfo/cython-devel > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Cygwin: Handling missing C99 long double functions
On Tue, Apr 26, 2016 at 2:56 PM Robert Bradshaw wrote: > On Tue, Apr 26, 2016 at 8:36 AM, Erik Bray wrote: > >> On Tue, Apr 26, 2016 at 5:16 PM, Dima Pasechnik >> wrote: >> > Hi, >> > certainly we did something with Sage on cygwin to work around these... >> > Just in case, >> >> From what I can tell there are several places where Sage has hacked >> around this issue in different packages, but it's not doing anything >> specifically with it for Cython. Sage uses the Cephes math lib to >> support these functions on FreeBSD--and apparently used to use it on >> Cygwin too, but disabled that for some reason. >> >> Regardless, Cython should ultimately do something sensible in these cases. >> >> My general thinking is that in cases where Cython generates code >> containing C math functions, it ought to support a fallback. This >> will require some feature checks so that Cython can generate wrappers, >> when necessary, around the double versions of those functions (as >> Numpy currently does). >> > > Let's make things concrete. You're complaining that something like > > cdef extern from "math.h": > long double sqrtl(long double) > > def foo(long double x): > return sqrtl(x) > > Doesn't work on Cygwin? > > The same is true for *any* C function that you use that's not totally > portable (this is the bane of trying to use C). I don't think Cython should > be detecting this and substituting a (less accurate) sqrt for sqrtl in this > case. If you want do do this, write your own headers that (conditionally) > define these things however you want. > > Or, are you complaining that Cython's test suite doesn't pass on some > Cygwin because there are tests of features not available on Cygwin? (Your > original email isn't clear.) If so, the test framework can be set up to > exclude these tests on that platform. > > Right, this sounds like a good place to exclude some tests. long double's behavior is pretty platform dependent. I wouldn't expect to be able to write platform independent code that uses it. There's not much Cython can do to change the situation either. Best, -Ian ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
