Re: [Cython] Utility Codes and templates

2011-07-25 Thread Vitja Makarov
2011/7/25 Stefan Behnel :
> Vitja Makarov, 25.07.2011 08:41:
>>
>> 2011/7/23 Robert Bradshaw:
>>>
>>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson
>>>   wrote:

 For my work on the _memview branch (and also on fused types) I noticed
 that UtilityCodes started weighing heavily on me in their current
 form, so I wrote a little loader in the _memview branch:


 https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110

 The idea is simple: you put your utility codes in Cython/Utility in
 .pyx, .c, .h files etc, and then load them. It works for both
 prototypes and implementations, for UtilityCode and CythonUtilityCode:
>>>
>>> This sounds like it could be a nice way to organize our UtilityCode
>>> snippets. So far we haven't really needed any more templating than
>>> simple substitution, but for what you're doing I can see this being
>>> quite handy. This may also provide a more flexible way forward for
>>> supporting multiple backends.
>>>
 myutility.c

 // UtilityProto: MyUtility
 header code here

 // UtilityCode: MyUtility
 implementation code here

 You can add as many other utilities as you like to the same file. You
 can then load it using

    UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
>>>
>>> I agree with you that having multiple related, named snippets in same
>>> file is worthwhile. What about
>>>
>>> // MyUtility.proto ///
>>>
>>> and
>>>
>>>  MyCyUtility ##
>>>
>>> so the chunks are easy to see.
>>>
>>
>> C++ comments looks ugly. May be it's better to have something like this:
>>
>> /* UtilityCode:  MyUtility.proto */
>> and
>>
>> # UtilityCode: MyCyUtility
>>
>> That's also pretty easy to parse
>
> For the parser, it makes no difference. For a human, it does. A big fat
> marker like "/" is hard to miss, whereas a tiny one like
> "/* ... */" is easily overlooked within a longer piece of code.
>

Ok, but how many slashes should be there?

I'd prefer something like:

## UtilityCode: MyUtility 

For both C and Cython

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Utility Codes and templates

2011-07-25 Thread Stefan Behnel

Vitja Makarov, 25.07.2011 10:25:

2011/7/25 Stefan Behnel:

Vitja Makarov, 25.07.2011 08:41:


2011/7/23 Robert Bradshaw:


On Fri, Jul 22, 2011 at 3:12 AM, mark florisson
wrote:


For my work on the _memview branch (and also on fused types) I noticed
that UtilityCodes started weighing heavily on me in their current
form, so I wrote a little loader in the _memview branch:


https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110

The idea is simple: you put your utility codes in Cython/Utility in
.pyx, .c, .h files etc, and then load them. It works for both
prototypes and implementations, for UtilityCode and CythonUtilityCode:


This sounds like it could be a nice way to organize our UtilityCode
snippets. So far we haven't really needed any more templating than
simple substitution, but for what you're doing I can see this being
quite handy. This may also provide a more flexible way forward for
supporting multiple backends.


myutility.c

// UtilityProto: MyUtility
header code here

// UtilityCode: MyUtility
implementation code here

You can add as many other utilities as you like to the same file. You
can then load it using

UtilityCode.load_utility_from_file("myutility.c", "MyUtility")


I agree with you that having multiple related, named snippets in same
file is worthwhile. What about

// MyUtility.proto ///

and

 MyCyUtility ##

so the chunks are easy to see.



C++ comments looks ugly. May be it's better to have something like this:

/* UtilityCode:  MyUtility.proto */
and

# UtilityCode: MyCyUtility

That's also pretty easy to parse


For the parser, it makes no difference. For a human, it does. A big fat
marker like "/" is hard to miss, whereas a tiny one like
"/* ... */" is easily overlooked within a longer piece of code.



Ok, but how many slashes should be there?

I'd prefer something like:

## UtilityCode: MyUtility 

For both C and Cython


I don't think the parser should care, and I don't think we should put much 
thought into this kind of detail, either. I'd suggest making the separator 
lines at most 78 characters long, and putting as many slashes or hashes in 
there as are needed to fill up the lines after writing the text content. 
What's important to the parser is that there's more than one hash/slash at 
start and end, and something parsable in between. The rest is up to those 
who write the file.


Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Utility Codes and templates

2011-07-25 Thread mark florisson
On 25 July 2011 12:00, Stefan Behnel  wrote:
> Vitja Makarov, 25.07.2011 10:25:
>>
>> 2011/7/25 Stefan Behnel:
>>>
>>> Vitja Makarov, 25.07.2011 08:41:

 2011/7/23 Robert Bradshaw:
>
> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson
>     wrote:
>>
>> For my work on the _memview branch (and also on fused types) I noticed
>> that UtilityCodes started weighing heavily on me in their current
>> form, so I wrote a little loader in the _memview branch:
>>
>>
>>
>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
>>
>> The idea is simple: you put your utility codes in Cython/Utility in
>> .pyx, .c, .h files etc, and then load them. It works for both
>> prototypes and implementations, for UtilityCode and CythonUtilityCode:
>
> This sounds like it could be a nice way to organize our UtilityCode
> snippets. So far we haven't really needed any more templating than
> simple substitution, but for what you're doing I can see this being
> quite handy. This may also provide a more flexible way forward for
> supporting multiple backends.
>
>> myutility.c
>>
>> // UtilityProto: MyUtility
>> header code here
>>
>> // UtilityCode: MyUtility
>> implementation code here
>>
>> You can add as many other utilities as you like to the same file. You
>> can then load it using
>>
>>    UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
>
> I agree with you that having multiple related, named snippets in same
> file is worthwhile. What about
>
> // MyUtility.proto ///
>
> and
>
>  MyCyUtility ##
>
> so the chunks are easy to see.
>

 C++ comments looks ugly. May be it's better to have something like this:

 /* UtilityCode:  MyUtility.proto */
 and

 # UtilityCode: MyCyUtility

 That's also pretty easy to parse
>>>
>>> For the parser, it makes no difference. For a human, it does. A big fat
>>> marker like "/" is hard to miss, whereas a tiny one
>>> like
>>> "/* ... */" is easily overlooked within a longer piece of code.
>>>
>>
>> Ok, but how many slashes should be there?
>>
>> I'd prefer something like:
>>
>> ## UtilityCode: MyUtility 
>>
>> For both C and Cython
>
> I don't think the parser should care, and I don't think we should put much
> thought into this kind of detail, either. I'd suggest making the separator
> lines at most 78 characters long, and putting as many slashes or hashes in
> there as are needed to fill up the lines after writing the text content.
> What's important to the parser is that there's more than one hash/slash at
> start and end, and something parsable in between. The rest is up to those
> who write the file.

Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either
side (symmetry is not enforced). It's now 'MyUtility' and
'MyUtility.proto'. If there's no objection to the ini-style header
(with requirements and other metadata possibly), then I'll implement
that one of these days.

> Stefan
> ___
> cython-devel mailing list
> cython-devel@python.org
> http://mail.python.org/mailman/listinfo/cython-devel
>
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Transformation of pxds

2011-07-25 Thread Romain Guillebert
Hi

I can now compile pxd files, but I have encountered something rather
strange : AnalyseExpressionsTransform turns :

cimport foo

def test():
foo.printf()

into :

cimport foo

def test():
printf()

It is due to the method analyse_as_cimported_attribute of AttributeNode
in the file Cython/Compiler/ExprNodes.py

Is this useful (since I don't think it has anything to do with
expression analysis) and do you have an idea on how I can solve this (my
first idea is : extend the class to disable the effect of this method
but it's more a hack than anything else) ?

Cheers
Romain
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Utility Codes and templates

2011-07-25 Thread Robert Bradshaw
On Mon, Jul 25, 2011 at 3:07 AM, mark florisson
 wrote:
> On 25 July 2011 12:00, Stefan Behnel  wrote:
>> Vitja Makarov, 25.07.2011 10:25:
>>>
>>> 2011/7/25 Stefan Behnel:

 Vitja Makarov, 25.07.2011 08:41:
>
> 2011/7/23 Robert Bradshaw:
>>
>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson
>>     wrote:
>>>
>>> For my work on the _memview branch (and also on fused types) I noticed
>>> that UtilityCodes started weighing heavily on me in their current
>>> form, so I wrote a little loader in the _memview branch:
>>>
>>>
>>>
>>> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
>>>
>>> The idea is simple: you put your utility codes in Cython/Utility in
>>> .pyx, .c, .h files etc, and then load them. It works for both
>>> prototypes and implementations, for UtilityCode and CythonUtilityCode:
>>
>> This sounds like it could be a nice way to organize our UtilityCode
>> snippets. So far we haven't really needed any more templating than
>> simple substitution, but for what you're doing I can see this being
>> quite handy. This may also provide a more flexible way forward for
>> supporting multiple backends.
>>
>>> myutility.c
>>>
>>> // UtilityProto: MyUtility
>>> header code here
>>>
>>> // UtilityCode: MyUtility
>>> implementation code here
>>>
>>> You can add as many other utilities as you like to the same file. You
>>> can then load it using
>>>
>>>    UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
>>
>> I agree with you that having multiple related, named snippets in same
>> file is worthwhile. What about
>>
>> // MyUtility.proto ///
>>
>> and
>>
>>  MyCyUtility ##
>>
>> so the chunks are easy to see.
>>
>
> C++ comments looks ugly. May be it's better to have something like this:
>
> /* UtilityCode:  MyUtility.proto */
> and
>
> # UtilityCode: MyCyUtility
>
> That's also pretty easy to parse

 For the parser, it makes no difference. For a human, it does. A big fat
 marker like "/" is hard to miss, whereas a tiny one
 like
 "/* ... */" is easily overlooked within a longer piece of code.

>>>
>>> Ok, but how many slashes should be there?
>>>
>>> I'd prefer something like:
>>>
>>> ## UtilityCode: MyUtility 
>>>
>>> For both C and Cython
>>
>> I don't think the parser should care, and I don't think we should put much
>> thought into this kind of detail, either. I'd suggest making the separator
>> lines at most 78 characters long, and putting as many slashes or hashes in
>> there as are needed to fill up the lines after writing the text content.
>> What's important to the parser is that there's more than one hash/slash at
>> start and end, and something parsable in between. The rest is up to those
>> who write the file.
>
> Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either
> side (symmetry is not enforced).

I don't even think we need a max.

> It's now 'MyUtility' and
> 'MyUtility.proto'. If there's no objection to the ini-style header
> (with requirements and other metadata possibly), then I'll implement
> that one of these days.

One drawback with the ini style is that it detaches the metadata from
the code block itself (and require duplicating their names).

- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Utility Codes and templates

2011-07-25 Thread Vitja Makarov
2011/7/26 Robert Bradshaw :
> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson
>  wrote:
>> On 25 July 2011 12:00, Stefan Behnel  wrote:
>>> Vitja Makarov, 25.07.2011 10:25:

 2011/7/25 Stefan Behnel:
>
> Vitja Makarov, 25.07.2011 08:41:
>>
>> 2011/7/23 Robert Bradshaw:
>>>
>>> On Fri, Jul 22, 2011 at 3:12 AM, mark florisson
>>>     wrote:

 For my work on the _memview branch (and also on fused types) I noticed
 that UtilityCodes started weighing heavily on me in their current
 form, so I wrote a little loader in the _memview branch:



 https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110

 The idea is simple: you put your utility codes in Cython/Utility in
 .pyx, .c, .h files etc, and then load them. It works for both
 prototypes and implementations, for UtilityCode and CythonUtilityCode:
>>>
>>> This sounds like it could be a nice way to organize our UtilityCode
>>> snippets. So far we haven't really needed any more templating than
>>> simple substitution, but for what you're doing I can see this being
>>> quite handy. This may also provide a more flexible way forward for
>>> supporting multiple backends.
>>>
 myutility.c

 // UtilityProto: MyUtility
 header code here

 // UtilityCode: MyUtility
 implementation code here

 You can add as many other utilities as you like to the same file. You
 can then load it using

    UtilityCode.load_utility_from_file("myutility.c", "MyUtility")
>>>
>>> I agree with you that having multiple related, named snippets in same
>>> file is worthwhile. What about
>>>
>>> // MyUtility.proto ///
>>>
>>> and
>>>
>>>  MyCyUtility ##
>>>
>>> so the chunks are easy to see.
>>>
>>
>> C++ comments looks ugly. May be it's better to have something like this:
>>
>> /* UtilityCode:  MyUtility.proto */
>> and
>>
>> # UtilityCode: MyCyUtility
>>
>> That's also pretty easy to parse
>
> For the parser, it makes no difference. For a human, it does. A big fat
> marker like "/" is hard to miss, whereas a tiny one
> like
> "/* ... */" is easily overlooked within a longer piece of code.
>

 Ok, but how many slashes should be there?

 I'd prefer something like:

 ## UtilityCode: MyUtility 

 For both C and Cython
>>>
>>> I don't think the parser should care, and I don't think we should put much
>>> thought into this kind of detail, either. I'd suggest making the separator
>>> lines at most 78 characters long, and putting as many slashes or hashes in
>>> there as are needed to fill up the lines after writing the text content.
>>> What's important to the parser is that there's more than one hash/slash at
>>> start and end, and something parsable in between. The rest is up to those
>>> who write the file.
>>
>> Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either
>> side (symmetry is not enforced).
>
> I don't even think we need a max.
>
>> It's now 'MyUtility' and
>> 'MyUtility.proto'. If there's no objection to the ini-style header
>> (with requirements and other metadata possibly), then I'll implement
>> that one of these days.
>
> One drawback with the ini style is that it detaches the metadata from
> the code block itself (and require duplicating their names).
>

Are we going to move all the existing utility code away from python code?
I think that would seriously help with  moving shared C-code into
cython library (.h and .so).

Some things like generator class implementation and cyfunction could
be move into .so file.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Utility Codes and templates

2011-07-25 Thread Robert Bradshaw
On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov  wrote:
> 2011/7/26 Robert Bradshaw :
>> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson
>>  wrote:
>>> On 25 July 2011 12:00, Stefan Behnel  wrote:
 Vitja Makarov, 25.07.2011 10:25:
>
> 2011/7/25 Stefan Behnel:
>>
>> Vitja Makarov, 25.07.2011 08:41:
>>>
>>> 2011/7/23 Robert Bradshaw:

 On Fri, Jul 22, 2011 at 3:12 AM, mark florisson
     wrote:
>
> For my work on the _memview branch (and also on fused types) I noticed
> that UtilityCodes started weighing heavily on me in their current
> form, so I wrote a little loader in the _memview branch:
>
>
>
> https://github.com/markflorisson88/cython/commit/e13debed2db78680ec0bd8c343433a2b73bd5e64#L2R110
>
> The idea is simple: you put your utility codes in Cython/Utility in
> .pyx, .c, .h files etc, and then load them. It works for both
> prototypes and implementations, for UtilityCode and CythonUtilityCode:

 This sounds like it could be a nice way to organize our UtilityCode
 snippets. So far we haven't really needed any more templating than
 simple substitution, but for what you're doing I can see this being
 quite handy. This may also provide a more flexible way forward for
 supporting multiple backends.

> myutility.c
>
> // UtilityProto: MyUtility
> header code here
>
> // UtilityCode: MyUtility
> implementation code here
>
> You can add as many other utilities as you like to the same file. You
> can then load it using
>
>    UtilityCode.load_utility_from_file("myutility.c", "MyUtility")

 I agree with you that having multiple related, named snippets in same
 file is worthwhile. What about

 // MyUtility.proto ///

 and

  MyCyUtility ##

 so the chunks are easy to see.

>>>
>>> C++ comments looks ugly. May be it's better to have something like this:
>>>
>>> /* UtilityCode:  MyUtility.proto */
>>> and
>>>
>>> # UtilityCode: MyCyUtility
>>>
>>> That's also pretty easy to parse
>>
>> For the parser, it makes no difference. For a human, it does. A big fat
>> marker like "/" is hard to miss, whereas a tiny one
>> like
>> "/* ... */" is easily overlooked within a longer piece of code.
>>
>
> Ok, but how many slashes should be there?
>
> I'd prefer something like:
>
> ## UtilityCode: MyUtility 
>
> For both C and Cython

 I don't think the parser should care, and I don't think we should put much
 thought into this kind of detail, either. I'd suggest making the separator
 lines at most 78 characters long, and putting as many slashes or hashes in
 there as are needed to fill up the lines after writing the text content.
 What's important to the parser is that there's more than one hash/slash at
 start and end, and something parsable in between. The rest is up to those
 who write the file.
>>>
>>> Yeah, I simply mandated a minimum of 5 and a maximum of 30 on either
>>> side (symmetry is not enforced).
>>
>> I don't even think we need a max.
>>
>>> It's now 'MyUtility' and
>>> 'MyUtility.proto'. If there's no objection to the ini-style header
>>> (with requirements and other metadata possibly), then I'll implement
>>> that one of these days.
>>
>> One drawback with the ini style is that it detaches the metadata from
>> the code block itself (and require duplicating their names).
>>
>
> Are we going to move all the existing utility code away from python code?

Yes, though I don't see a huge hurry to get rid of the old style.

> I think that would seriously help with  moving shared C-code into
> cython library (.h and .so).
>
> Some things like generator class implementation and cyfunction could
> be move into .so file.

Yes, that would be good. Of course we don't want a single .so file for
every tiny chunk of utility code, but one needs to be careful lumping
them together as well (e.g. things may compile differently depending
on the set of headers included in the .pyx file). To do this right I
we need to compile things at a higher level than module-by-module with
Cython.Build.cythonize or similar.

Certainly CyFunction and the generator class are easy and high-benefit
targets (compared to, e.g. tuple indexing utility code).

- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Utility Codes and templates

2011-07-25 Thread Stefan Behnel

Robert Bradshaw, 26.07.2011 06:29:

On Mon, Jul 25, 2011 at 3:07 AM, mark florisson

It's now 'MyUtility' and
'MyUtility.proto'. If there's no objection to the ini-style header
(with requirements and other metadata possibly), then I'll implement
that one of these days.


One drawback with the ini style is that it detaches the metadata from
the code block itself (and require duplicating their names).


The names are duplicated in the file already, since impl and proto are 
separated and can be anywhere in the file (e.g., you could put all protos 
at the top and all implementation parts further down). The advantages of 
putting in a header is that


a) it's immediately visible from the top of the file what it contains

b) the dependencies on other files are explicit and clearly visible, again, 
from the top of the file


c) the metadata is easy and quick to parse, as the header is trivially 
separated from the rest


d) only the header needs to be parsed in order to know what can be found in 
it or what other files are required to compile it


e) we don't need to write our own parser, and the overall file format is 
trivial to parse


Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Utility Codes and templates

2011-07-25 Thread Robert Bradshaw
On Mon, Jul 25, 2011 at 11:39 PM, Stefan Behnel  wrote:
> Robert Bradshaw, 26.07.2011 06:29:
>>
>> On Mon, Jul 25, 2011 at 3:07 AM, mark florisson
>>>
>>> It's now 'MyUtility' and
>>> 'MyUtility.proto'. If there's no objection to the ini-style header
>>> (with requirements and other metadata possibly), then I'll implement
>>> that one of these days.
>>
>> One drawback with the ini style is that it detaches the metadata from
>> the code block itself (and require duplicating their names).
>
> The names are duplicated in the file already, since impl and proto are
> separated and can be anywhere in the file (e.g., you could put all protos at
> the top and all implementation parts further down). The advantages of
> putting in a header is that
>
> a) it's immediately visible from the top of the file what it contains
>
> b) the dependencies on other files are explicit and clearly visible, again,
> from the top of the file
>
> c) the metadata is easy and quick to parse, as the header is trivially
> separated from the rest
>
> d) only the header needs to be parsed in order to know what can be found in
> it or what other files are required to compile it
>
> e) we don't need to write our own parser, and the overall file format is
> trivial to parse

I prefer keeping the metadata close, but perhaps that's just a matter
of opinion. Slurping in the entire file and parsing the block headers
is pretty trivial as well, and with pre-fetching (on so many levels) I
don't think there's a technical advantage for files so small. For
those (likely common) cases where no metadata is required, I'm -1 on
having to manually keep a "table of contents."

- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Utility Codes and templates

2011-07-25 Thread Stefan Behnel

Robert Bradshaw, 26.07.2011 07:00:

On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote:

I think that would seriously help with  moving shared C-code into
cython library (.h and .so).

Some things like generator class implementation and cyfunction could
be move into .so file.


Yes, that would be good.


Well, *if* we find a way to make sure the additional modules get properly 
distributed. It's not obvious to me how to do that, certainly not in a 
cross-package way, and not even in a per-package way, as that would still 
require some kind of interaction with distutils and maybe even changes to 
the users' setup.py scripts (hopefully in a way that doesn't require them 
to know what exact additional modules will be generated).




Of course we don't want a single .so file for
every tiny chunk of utility code, but one needs to be careful lumping
them together as well (e.g. things may compile differently depending
on the set of headers included in the .pyx file). To do this right I
we need to compile things at a higher level than module-by-module with
Cython.Build.cythonize or similar.

Certainly CyFunction and the generator class are easy and high-benefit
targets (compared to, e.g. tuple indexing utility code).


Now that we have a way to group utility code in a single file, including 
automatically resolved dependencies to other files, it should be trivial to 
take such a file and to compile it into a separate module. Cython modules 
would then trivially export their public names anyway, and utility code in 
C files could have an additional Cython wrapper file that would simply 
depend on the C file and that properly exports the C functions through a 
Python API and inter-module C-API.


We could even write empty utility files that only contain dependencies, 
thus grouping together multiple utility files to one larger module.


Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel


Re: [Cython] Utility Codes and templates

2011-07-25 Thread Robert Bradshaw
On Mon, Jul 25, 2011 at 11:50 PM, Stefan Behnel  wrote:
> Robert Bradshaw, 26.07.2011 07:00:
>>
>> On Mon, Jul 25, 2011 at 9:48 PM, Vitja Makarov wrote:
>>>
>>> I think that would seriously help with  moving shared C-code into
>>> cython library (.h and .so).
>>>
>>> Some things like generator class implementation and cyfunction could
>>> be move into .so file.
>>
>> Yes, that would be good.
>
> Well, *if* we find a way to make sure the additional modules get properly
> distributed. It's not obvious to me how to do that, certainly not in a
> cross-package way, and not even in a per-package way, as that would still
> require some kind of interaction with distutils and maybe even changes to
> the users' setup.py scripts (hopefully in a way that doesn't require them to
> know what exact additional modules will be generated).

This is where the ext_modules = Cython.Build.cythonize(...) pattern
could come in handy. We'd just add an extra module to the list. (The
generated .c file could also be shipped, but might have to be listed
explicitly.)

>> Of course we don't want a single .so file for
>> every tiny chunk of utility code, but one needs to be careful lumping
>> them together as well (e.g. things may compile differently depending
>> on the set of headers included in the .pyx file). To do this right I
>> we need to compile things at a higher level than module-by-module with
>> Cython.Build.cythonize or similar.
>>
>> Certainly CyFunction and the generator class are easy and high-benefit
>> targets (compared to, e.g. tuple indexing utility code).
>
> Now that we have a way to group utility code in a single file, including
> automatically resolved dependencies to other files, it should be trivial to
> take such a file and to compile it into a separate module. Cython modules
> would then trivially export their public names anyway, and utility code in C
> files could have an additional Cython wrapper file that would simply depend
> on the C file and that properly exports the C functions through a Python API
> and inter-module C-API.
>
> We could even write empty utility files that only contain dependencies, thus
> grouping together multiple utility files to one larger module.

I was thinking about stuff like how we handle complex numbers where
the pre-processor has a large role to play. Low-level stuff like this
probably belongs in a .h file rather than a .so file though. I'm also
wondering if there are any efficiency gains to be had from actually
linking the resulting .so file at compile time rather than using the
runtime inter-module C-API stuff.

- Robert
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel