Re: [Cython] Fwd: Question about how best require compiler options for C sources
On Mon, Apr 11, 2016 at 7:38 PM, Jeroen Demeyer wrote: > On 2016-04-11 15:23, Erik Bray wrote: >> >> In this case I really do want the symbol "hello" to be >> exported by the DLL, as well as be understood between translation >> units making up the same library. > > > Are you really sure that you want this? I doubt that this is supported on OS > X: on OS X, there are two kinds of files: shared libraries (with .dylib > extension) and loadable modules or bundles (usually with .so extension but > officially with .bundle extension). > > C extensions for Python are compiled as a loadable module and I don't think > you can link against these. See > http://stackoverflow.com/questions/2339679/what-are-the-differences-between-so-and-dylib-on-osx > > Some details in the above might be wrong, but I remember running into this > issue in some early version of cysignals. OSX issues aside, I was under the impression that this is needed for cysignals in particular. If I'm wrong on that then this simplifies matters a good deal, and per njs Cython could just drop the use of DL_IMPORT and we'd be fine. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Question about how best require compiler options for C sources
On Mon, Apr 11, 2016 at 8:36 PM, Ian Henriksen
wrote:
> On Mon, Apr 11, 2016 at 11:50 AM Ian Henriksen
> wrote:
>>
>> To answer the original question about define macros, it appears that the
>> canonical
>> way to pass preprocessor defines through distutils is to use the
>> define_macros
>> keyword when constructing your Extension class. You should also be able to
>> do
>> this within a Cython source file by including a directive like:
>>
>> # distutils: define_macros = MY_DEFINE, MY_DEFINE_2=2
>>
>> Unfortunately, it looks like there's a bug in that that's making it so
>> that these
>> macros are undef'ed rather than being defined, so, for now, just pass the
>> appropriate flags to your Extension object.
>>
>
> Small update on this, it looks like the issue with undef/define only applies
> when a
> define is specified this way without a value, so I either haven't gotten the
> syntax quite
> right, or that's not supported yet. Specifying an actual value for the macro
> works fine.
There is an issue specifically about this that I wanted to bring up,
though it's a bit tangential to my original issue (but maybe not
entirely?)
I know about putting "# distutils: " comments in a Cython source. But
I felt that requiring this in order to get dllexport/dllimport working
correctly on Windows is too much of a burden for a developer (who
normally might not be thinking about this).
The content of these comments *do* get written out to the generated C
sources under a "distutils" property in the JSON metadata at the top
of the file. So I thought one way I might be able to influence the
compiler flags would be to modify the code that generates the JSON
metadata to include a `"distutils: {"define_macros": [ ... ]}` entry
in that metadata. However, this was wrong because Cython itself never
actually does anything with that metadata, and it instead reads the
distutils directives (a second time) directly out of the Cython
sources during cythonize() in order to update the Extension
attributes. I'm thinking it would make more sense if the distutils
options were read from the C sources, since those are what will
*actually* be handled by distutils, even if in the majority of cases
the JSON metadata is being generated directly from the Cython sources.
Changing this would have also provided a solution to my original
problem, even if we've agreed that it's mostly moot. That said, I
think it makes more sense for cythonize() to read the distutils
options from the C source instead of the Cython source, though in
practice I don't know if it's a worthwhile change or not.
Thanks,
Erik
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Question about how best require compiler options for C sources
On 2016-04-12 10:16, Erik Bray wrote: That said, I think it makes more sense for cythonize() to read the distutils options from the C source instead of the Cython source, though in practice I don't know if it's a worthwhile change or not. I don't quite get what you mean. The C file is only read by distutils (which compiles the .c files to .so), not by Cython (which compiles the .pyx files to .c). ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Question about how best require compiler options for C sources
On 2016-04-12 10:08, Erik Bray wrote: OSX issues aside, I was under the impression that this is needed for cysignals in particular. No. cysignals is complicated, but it doesn't need dynamic linking to Python modules (.so files). It does need "internal" linking: it needs to link a non-Cython-generated .c file together with a Cython-generated .c file into one Python module. It uses "cdef public" for that, so it would run in exactly the problem that this thread is about. So I personally would not mind if Cython would completely drop support for exporting symbols from C extensions (which, AFAIK, doesn't work on OS X anyway). ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Question about how best require compiler options for C sources
On Mon, Apr 11, 2016 at 7:49 PM, Ian Henriksen wrote: > That aside, I agree with Nathaniel that exporting public declarations as a > part of the > shared object interface was a design mistake. That aside, however, using an > api > declaration lets you get equivalent results without exposing anything as a > part of > the shared object API. Here's how this all works: I don't know if I would outright call it a "mistake", but in retrospect I think we're all in agreement now that maybe the behavior of "public" should be curtailed? Or should we still try to keep its current behavior and "fix" it as I've been trying to do? > public declarations: export things to C/C++ through the shared object > interface. > Provide a header that exports this interface. > api declarations: export things to C/C++ through capsule objects. Provide a > header > for the Python module that exports that interface. > cimports: Use capsule objects and parsing of pxd files to share things like > external > declarations, header includes, inline Cython functions, and Cython functions > exported by modules between Cython modules. > > The public and api use cases are essentially the same most of the time, but > api > declarations use capsules rather than the OS's linker. > > There are still some trade-offs between public and api functions. > Technically, the > api functions require that the initialization routine exported in the api > header be > called for each translation unit that uses them. The public api just > requires that the > module already be initialized. In cases where no Python functionality is > used in a > public function, you may be able to get away with using the function without > initializing the module, though I really wouldn't recommend that. Yes, this is the main problem I have with the "api" declaration. Don't get me wrong, it's a very clever approach and I think for a lot of cases it's the right thing to do. But it absolutely requires and assumes initialization of the module. In a general case that would be the right thing to do. But in theory one could have some plain C code that does not use any of the Python API, and that one can be sure doesn't require anything initialized by the module init function. > There are some more subtle issues here though. The reason api functions need > to > be initialized on a per-translation unit basis is that things exported as > api > declarations are exported as translation-unit-local (static) function > pointers. They > aren't shared by the different translation units within a module built from > multiple > source files. I think that's a mistake. It'd be ideal if we could have api > interfaces (or > something like them) provide things with shared object local visibility > rather than > translation unit local visibility. This would require that the API headers > have more > carefully structured ifdef directives so that a macro could be set in a > given > translation unit to designate when to emit the actual declarations for the > needed > pointers rather than just forward declaring them. It would also require that > the main > generated c/cpp file define the pointers it uses as shared-object-local > rather rather > than static. Yep. This is a problem cysignals is having if I recall correctly. It would be good to be able to have it both ways. > In dynd-python we currently solve this problem by defining shared object > local > wrappers for the api exported function pointers and then using those > instead, but > I'm not a huge fan of that approach. It works well, but results in another > unnecessary layer of indirection through the source files to connect the C++ > code > back to its Python bindings. Yes, I think some kind of context-dependent declarations using carefully crafted preprocessor directives, or just using separate header files entirely for the two contexts, would be best. > With regards to the dllexporting/dllimporting of things: given that public > declarations > are already designed to export things through the shared object interface, > we may > as well fix the current setup to export the right things. It's a bad design > that > probably ought to be deprecated or at least documented better so people know > not > to use it unless their case actually requires sidestepping best practices. > On the > other hand, it's also a supported interface, so there's value in "fixing" > it. > > I think the best way to do that is the following: > - mark public symbols as dllimport unless a given (module specific) > preprocessor > define is set. > - people using the public header outside of the module exporting the symbols > should not have to set the define at all. > - people using the public header to compile other source files that are > linked in to > the same Python module should set the preprocessor flag for that module. That's the approach I was trying to take originally, yes. But I think it's a burden to require developers to set that preprocessor flag--instead it
[Cython] cdef public declarations
(this thread is related to the thread "Question about how best require compiler options for C sources") I have a question for Cython users and developers: are you sure that "cdef public" actually works as documented at http://docs.cython.org/src/userguide/external_C_code.html#public-declarations I couldn't find it tested in the Cython testsuite, although it's not easy to know what to grep for. I hightly doubt that it works on OS X because of the difference between shared libraries (with .dylib extension) and loadable modules/bundles (with .so extension). ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Question about how best require compiler options for C sources
On Tue, Apr 12, 2016 at 10:27 AM, Jeroen Demeyer wrote: > On 2016-04-12 10:16, Erik Bray wrote: >> >> That said, I >> think it makes more sense for cythonize() to read the distutils >> options from the C source instead of the Cython source, though in >> practice I don't know if it's a worthwhile change or not. > > > I don't quite get what you mean. The C file is only read by distutils (which > compiles the .c files to .so), not by Cython (which compiles the .pyx files > to .c). Right. What I'm saying is that when you call cythonize() the output is a list of Extension objects, which include instructions for how to distutils should compile the C sources (in this case, the define_macros attribute in particular). cythonize(), at some step along the way, reads "# distutils:" directives directly out of the .pyx sources, but then applies them to the Extension object representing the C sources to be compiled. What I'm suggesting is that there should be a clearer separation--the Cython compiler should generate the C sources first--including outputting their JSON metadata which may include some distutils directives, but otherwise remain completely agnostic as to how the C sources will ultimately be compiled (currently this *is* true insofar as how Cython's compile() works--the confusion is in cythonize()). After the Cython sources have been compiled to C sources, the cythonize() function should then inspect the resulting C sources (which are given explicitly in CompilerResult objects, which are currently ignored by cythonize()) and use the metadata in the C sources to tell distutils what to do with them. The Cython sources shouldn't be consulted any further at this point. It's a subtle difference which currently wouldn't affect the end result. The reason I thought of this is that I wanted to be able to set "distutils" directives directly in the C sources without them needing to be manually declared in the Cython sources. But even without that use case it's a clearer separation of concerns. Erik ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] cdef public declarations
On Tue, Apr 12, 2016 at 10:36 AM, Jeroen Demeyer wrote: > (this thread is related to the thread "Question about how best require > compiler options for C sources") > > I have a question for Cython users and developers: are you sure that "cdef > public" actually works as documented at > http://docs.cython.org/src/userguide/external_C_code.html#public-declarations > > I couldn't find it tested in the Cython testsuite, although it's not easy to > know what to grep for. I hightly doubt that it works on OS X because of the > difference between shared libraries (with .dylib extension) and loadable > modules/bundles (with .so extension). Do you have a reference about this distinction on hand? I've seen .dylib files on OSX before but never understood the distinction. I tend to irrationally avoid learning anything about OSX for as long as possible, the way most OSS people do with Windows :) Seems like a good question though. ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] cdef public declarations
On 2016-04-12 10:49, Erik Bray wrote: Do you have a reference about this distinction on hand? I think these two links are useful: [1] http://docstore.mik.ua/orelly/unix3/mac/ch05_03.htm [2] http://stackoverflow.com/questions/2339679/what-are-the-differences-between-so-and-dylib-on-osx ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Question about how best require compiler options for C sources
On Tue, Apr 12, 2016 at 2:35 AM Erik Bray wrote: > On Mon, Apr 11, 2016 at 7:49 PM, Ian Henriksen > wrote: > > That aside, I agree with Nathaniel that exporting public declarations as > a > > part of the > > shared object interface was a design mistake. That aside, however, using > an > > api > > declaration lets you get equivalent results without exposing anything as > a > > part of > > the shared object API. Here's how this all works: > > I don't know if I would outright call it a "mistake", but in > retrospect I think we're all in agreement now that maybe the behavior > of "public" should be curtailed? Or should we still try to keep its > current behavior and "fix" it as I've been trying to do? > Great question, I don't know for sure. Input from others here would be nice. My thoughts are basically just that we should do one or the other. If it's supported and we want to keep supporting it, let's make it work right on platforms other than Linux. If we want to stop supporting it, let's deprecate it. Best, -Ian ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Question about how best require compiler options for C sources
On Tue, Apr 12, 2016 at 2:35 AM Erik Bray wrote: > On Mon, Apr 11, 2016 at 7:49 PM, Ian Henriksen wrote: > > With regards to the dllexporting/dllimporting of things: given that > public > > declarations > > are already designed to export things through the shared object > interface, > > we may > > as well fix the current setup to export the right things. It's a bad > design > > that > > probably ought to be deprecated or at least documented better so people > know > > not > > to use it unless their case actually requires sidestepping best > practices. > > On the > > other hand, it's also a supported interface, so there's value in "fixing" > > it. > > > > I think the best way to do that is the following: > > - mark public symbols as dllimport unless a given (module specific) > > preprocessor > > define is set. > > - people using the public header outside of the module exporting the > symbols > > should not have to set the define at all. > > - people using the public header to compile other source files that are > > linked in to > > the same Python module should set the preprocessor flag for that module. > > That's the approach I was trying to take originally, yes. But I think > it's a burden to require developers to set that preprocessor > flag--instead it should happen automatically, (or not at all, with > preference instead for using a completely separate header for > intra-module use). > > > On top of that, at some point we still need to fix our api and public > > headers so that > > they still work when included into the translation unit for the main > > Cython-generated > > c/cpp file. This use-case should just forward-declare everything since > the > > needed > > symbols are all defined later on in the Cython module. Since static > > variables > > cannot be forward declared in C, this will require that api declarations > use > > shared > > object local symbols or that the main generated c/cpp file use some ifdef > > guards > > when it initializes the various pointers in question. > > Sounds good--I see no problem with this. > > > As far as making an additional header goes, I personally prefer the extra > > preprocessor define. On the other hand, if people think an additional > header > > is > > easier to use, then why not make it do something like > > > > #define USE_DLLEXPORT_NOT_DLLIMPORT_FOR_PARTICULAR_MODULE > > #include > > > > I think, that'd cover all the use cases better. > > That would be fine too and reduce duplication. I think such a header > should still be generated though and it should be documented when to > use which header. > > After thinking this over a bit more, it seems like we're discussing things that really have more to do with the API chosen for a particular module than anything else. Maybe these options should all be made available as Compiler flags to Cython instead. As far as I can tell there are two separate options here: - Whether to provide distinct headers for defining the needed importing functionality, or declaring it as extern - Whether to make the preprocessor define flag for a given module's header work as an opt-in or an opt-out for defining the symbols needed to import and use the module. (numpy's headers use an opt-out system. DyND doesn't currently have this set up all of the way, but we'd like to use an opt-in model for working with our headers). Thoughts? Best, -Ian ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Fwd: Question about how best require compiler options for C sources
On Mon, Apr 11, 2016 at 12:36 PM Ian Henriksen < insertinterestingnameh...@gmail.com> wrote: > On Mon, Apr 11, 2016 at 11:50 AM Ian Henriksen < > insertinterestingnameh...@gmail.com> wrote: > >> To answer the original question about define macros, it appears that the >> canonical >> way to pass preprocessor defines through distutils is to use the >> define_macros >> keyword when constructing your Extension class. You should also be able >> to do >> this within a Cython source file by including a directive like: >> >> # distutils: define_macros = MY_DEFINE, MY_DEFINE_2=2 >> >> Unfortunately, it looks like there's a bug in that that's making it so >> that these >> macros are undef'ed rather than being defined, so, for now, just pass the >> appropriate flags to your Extension object. >> >> > Small update on this, it looks like the issue with undef/define only > applies when a > define is specified this way without a value, so I either haven't gotten > the syntax quite > right, or that's not supported yet. Specifying an actual value for the > macro works fine. > > Best, > Ian Henriksen > Should be fixed in https://github.com/cython/cython/pull/509. Best, Ian Henriksen ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
