Re: [Cython] CLI for using Shared utility module
Another question popped in my mind. How we will deal with:
> Introduce new `cython` command parameter `--shared` which will take fully
qualified module name:
On Fri, 14 Feb 2025 at 16:06, matus valo wrote:
>
>> I think I tend to agree with David. Here is my point of view:
>>
>> I am not sure how to implement it. cython command *does not know* about
>> the build directory. This is more part of the build system
>> (setuptools/meson etc). My point is that the cython command should be
>> stupid and just generate a C file with a specified name/location nothing
>> more. It is also consistent with current usage of it (it just generates C
>> files). I understand that it is the build system responsibility to do all
>> the magic around it. As an example, here is working integration of current
>> "stupid" utility present in the PR implemented by scipy dev (Ralf Gommers)
>> [1]:
>> https://github.com/scipy/scipy/compare/main...rgommers:scipy:cy-shared-memoryview?expand=1
>>
>> The only question is how it can be integrated into setuptools too. The
>> most obvious and stupid solution is to generate C file (which was also
>> recommended way of working with Cython extension until recently), commit it
>> to repository and compile it as regular C extension example is already
>> present in test of the PR:
>>
>> from Cython.Build import cythonize
>> from setuptools import setup, Extension
>> from Cython.Compiler import Options
>> import numpy
>>
>> extensions = [
>> Extension("*", ["**/*.pyx"], include_dirs=[numpy.get_include()]),
>> Extension("pkg2.MemoryView", sources=["pkg2/MemoryView.c"])
>> ]
>>
>> setup(
>> ext_modules = cythonize(extensions,
>> compiler_directives={'use_shared_utility': 'pkg2'})
>> )
>>
>> > Create all necessary directories, then write the output into the target
>> directory. No need for a failure.
>>
>> Here again `cython` command I would keep consistent. If user wants to
>> generate file, directory must already exist. This is the same logic as with
>> compilation of regular .pyx file:
>>
>> (cython3.11) root@a3be2cc9ff83:~/dev/cython# cython /foo/bar.pyx
>> /root/dev/cython3.11/bin/cython: No such file or directory:
>> '/foo/bar.pyx'
>>
>> Moreover, if we decide to create all directories in cython command,
>> should we create also `__init__.py` files too? I think this is an overkill
>> for cython command. Of course, in the build system it is different story,
>> build backend could do it.
>>
>> That said, I propose following:
>>
>> 1. Introduce the cython command parameter `--generate-shared` which has
>> the argument name & path of generated C file. E.g.
>>
>> cython --generate-shared scipy/_lib/cyshared.c
>>
>> will generate `scipy._lib/cyshared.c` file. This approach should be
>> easily integrated in meson build system based on example in [1]. Also
>> `cython` command will fail if directory `scipy/_lib` does not exist. Also,
>> cython will not care whether it is importable. It will just do stupid and
>> simple C file generation. The rest should be done by build system
>> (meson/setuptools/whatever). **This will be part of this PR**.
>>
>> 2. Introduce new `cython` command parameter `--shared` which will take
>> fully qualified module name:
>>
>> cython --shared=scipy._lib.cyshared.memoryview cython_extention.pyx
>>
>> I think here is already agreement between us. We can do some good old
>> bikeshedding about parameter name. This will be **part of this PR **too.
>>
>> 3. Extend existing `cython.build.cythonize()` function. This should be
>> used in the setuptools:
>>
>> from Cython.Build import cythonize
>> from setuptools import setup, Extension
>> from Cython.Compiler import Options
>> import numpy
>>
>> extensions = [
>> Extension("*", ["**/*.pyx"], include_dirs=[numpy.get_include()]),
>> ]
>>
>> setup(
>> ext_modules = cythonize(extensions, shared_module=pkg2.MemoryView)
>> )
>>
>> This function should also ensures that all directories and `__init__.py`
>> files are generated, hence the final `.so` module is importable. Currently,
>> I cannot see all the details and all devils hiding in the details.
>> Hence, I **would postpone it in the separate issue/PR**. Users of meson
>> are fine (see [1]) and users setuptools still can use generated and
>> commited C file in repo.
>>
>>
>> Please let me know what do you think. This is blocking my further
>> progress in PR and I would like to have clear vision to avoid PR flooding
>> with changes.
>> Moreover, I would like to keep the current PR simple and focus on actual
>> code generation of shared module and its integration into the extentions.
>> Build related logic can be implemented later on.
>>
>>
>> [1] https://github.com/cython/cython/pull/6531#issuecomment-2551351591
>
>
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CLI for using Shared utility module
Snap! Clicked send by mistake.
Another question popped in my mind. How we will deal with:
> Introduce new `cython` command parameter `--shared` which will take fully
qualified module name:
When we will introduce new `--shared` option we will have 3 places where we
can specify shared module ffully qualified module name:
1. New Cython parameter `--shared` foo.bar
2. Cython Directive parameter: `-X use_shared_utility=foo.bar
3. Cython module comment: `# cython: use_shared_utility=foo.bar
What is the priority between 1., 2. and 3.? I suppose 1. and 2. should
override 3. But what we should do if 1. and 2. are set but have different
values?
Matus
On Sat, 15 Feb 2025 at 00:00, matus valo wrote:
> Another question popped in my mind. How we will deal with:
> > Introduce new `cython` command parameter `--shared` which will take
> fully qualified module name:
>
> On Fri, 14 Feb 2025 at 16:06, matus valo wrote:
>
>>
>>> I think I tend to agree with David. Here is my point of view:
>>>
>>> I am not sure how to implement it. cython command *does not know* about
>>> the build directory. This is more part of the build system
>>> (setuptools/meson etc). My point is that the cython command should be
>>> stupid and just generate a C file with a specified name/location nothing
>>> more. It is also consistent with current usage of it (it just generates C
>>> files). I understand that it is the build system responsibility to do all
>>> the magic around it. As an example, here is working integration of current
>>> "stupid" utility present in the PR implemented by scipy dev (Ralf Gommers)
>>> [1]:
>>> https://github.com/scipy/scipy/compare/main...rgommers:scipy:cy-shared-memoryview?expand=1
>>>
>>> The only question is how it can be integrated into setuptools too. The
>>> most obvious and stupid solution is to generate C file (which was also
>>> recommended way of working with Cython extension until recently), commit it
>>> to repository and compile it as regular C extension example is already
>>> present in test of the PR:
>>>
>>> from Cython.Build import cythonize
>>> from setuptools import setup, Extension
>>> from Cython.Compiler import Options
>>> import numpy
>>>
>>> extensions = [
>>> Extension("*", ["**/*.pyx"], include_dirs=[numpy.get_include()]),
>>> Extension("pkg2.MemoryView", sources=["pkg2/MemoryView.c"])
>>> ]
>>>
>>> setup(
>>> ext_modules = cythonize(extensions,
>>> compiler_directives={'use_shared_utility': 'pkg2'})
>>> )
>>>
>>> > Create all necessary directories, then write the output into the
>>> target directory. No need for a failure.
>>>
>>> Here again `cython` command I would keep consistent. If user wants to
>>> generate file, directory must already exist. This is the same logic as with
>>> compilation of regular .pyx file:
>>>
>>> (cython3.11) root@a3be2cc9ff83:~/dev/cython# cython /foo/bar.pyx
>>> /root/dev/cython3.11/bin/cython: No such file or directory:
>>> '/foo/bar.pyx'
>>>
>>> Moreover, if we decide to create all directories in cython command,
>>> should we create also `__init__.py` files too? I think this is an overkill
>>> for cython command. Of course, in the build system it is different story,
>>> build backend could do it.
>>>
>>> That said, I propose following:
>>>
>>> 1. Introduce the cython command parameter `--generate-shared` which has
>>> the argument name & path of generated C file. E.g.
>>>
>>> cython --generate-shared scipy/_lib/cyshared.c
>>>
>>> will generate `scipy._lib/cyshared.c` file. This approach should be
>>> easily integrated in meson build system based on example in [1]. Also
>>> `cython` command will fail if directory `scipy/_lib` does not exist. Also,
>>> cython will not care whether it is importable. It will just do stupid
>>> and simple C file generation. The rest should be done by build system
>>> (meson/setuptools/whatever). **This will be part of this PR**.
>>>
>>> 2. Introduce new `cython` command parameter `--shared` which will take
>>> fully qualified module name:
>>>
>>> cython --shared=scipy._lib.cyshared.memoryview cython_extention.pyx
>>>
>>> I think here is already agreement between us. We can do some good old
>>> bikeshedding about parameter name. This will be **part of this PR **too.
>>>
>>> 3. Extend existing `cython.build.cythonize()` function. This should be
>>> used in the setuptools:
>>>
>>> from Cython.Build import cythonize
>>> from setuptools import setup, Extension
>>> from Cython.Compiler import Options
>>> import numpy
>>>
>>> extensions = [
>>> Extension("*", ["**/*.pyx"], include_dirs=[numpy.get_include()]),
>>> ]
>>>
>>> setup(
>>> ext_modules = cythonize(extensions, shared_module=pkg2.MemoryView)
>>> )
>>>
>>> This function should also ensures that all directories and `__init__.py`
>>> files are generated, hence the final `.so` module is importable. Currently,
>>> I cannot see all the details and all devils hiding i
[Cython] CLI for using Shared utility module
Hi All, I am writing to you to not get lost in https://github.com/cython/cython/pull/6531. I would like to get an agreement on how we would like to proceed with a CLI. The PR has two parts: 1. Utility for generating C file of shared utility. Here, I propose to pass the full path to the file e.g.: cython_shared_module scipy/_lib/cyshared.c The reason why not to use the import path is because the target package can be hidden inside a directory (e.g. src). This does not work well with the import path. Additional questions: * what we will call the utility for creation of the module? * should it be executable or we will use python -m '...' approach? * should utility create directories or to fail if directory does not exist? 2. cython parameter to instruct cython to compile module with shared utility Here, I propose to use import path and compilation directive: cython -X use_shared_utility=scipy._lib.cyshared my_module.pyx The reason to use the import path is because the file must be internally imported in any case. The import path is a full path including the shared module. Could you help me answer the mentioned questions? I think this is last piece of puzzle and after the PR should be finally complete (there are still PR review needed and rough edges to be polished). Thanks, Matus ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CLI for using Shared utility module
Hi All, > That said, I propose following: It seems there is no opposition to my proposal so I took liberty and implemented it in the PR. Let me know what you think. > In general, changing the code is not for everyone, passing a CLI option is > easy. We should therefore prefer the CLI option over the code if both are > available. Passing a different CLI option to test something out seems quick > and easy, so contradictions should be fine and the CLI option should just win. > Passing two options on the command line is usually resolved by letting the > last one win. I think that's fine in this case as well. The "--shared" > option would just set the directive, and setting the directive with "-X" > would then change the directive value again (and vice versa). This logic was also implemented in the PR. Matus On Sat, 15 Feb 2025 at 17:37, Stefan Behnel via cython-devel < cython-devel@python.org> wrote: > matus valo via cython-devel schrieb am 15.02.25 um 00:07: > > Another question popped in my mind. How we will deal with: > > > >> Introduce new `cython` command parameter `--shared` which will take > fully > > qualified module name: > > > > When we will introduce new `--shared` option we will have 3 places where > we > > can specify shared module ffully qualified module name: > > > > 1. New Cython parameter `--shared` foo.bar > > 2. Cython Directive parameter: `-X use_shared_utility=foo.bar > > 3. Cython module comment: `# cython: use_shared_utility=foo.bar > > > > What is the priority between 1., 2. and 3.? I suppose 1. and 2. should > > override 3. But what we should do if 1. and 2. are set but have different > > values? > > In general, changing the code is not for everyone, passing a CLI option is > easy. We should therefore prefer the CLI option over the code if both are > available. Passing a different CLI option to test something out seems > quick > and easy, so contradictions should be fine and the CLI option should just > win. > > Passing two options on the command line is usually resolved by letting the > last one win. I think that's fine in this case as well. The "--shared" > option would just set the directive, and setting the directive with "-X" > would then change the directive value again (and vice versa). > > Stefan > > ___ > cython-devel mailing list > cython-devel@python.org > https://mail.python.org/mailman/listinfo/cython-devel > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Shared utility Option vs. Compilation Directive
> I don't think 3 is worthwhile. I tend to agree. > I weakly prefer 1 to 2 on the basis that we were trying to get away from Options where possible. I think I have the same opinion. `Option` variant is much simpler and I cannot see any benefit using a shared module with only a subset of .pyx files. Matus On Mon, 3 Mar 2025 at 21:07, da-woods wrote: > Personally: > > I don't think 3 is worthwhile. > > I weakly prefer 1 to 2 on the basis that we were trying to get away from > Options where possible. However, this is probably a case where Options > will usually be right (it forces a setting on every module that you're > processing in a batch, which is what most people will want here. But maybe > not everyone). > > David > > > On 03/03/2025 19:47, matus valo wrote: > > Hi All, > > Based on the comment > https://github.com/cython/cython/pull/6531#discussion_r1975502026 I would > like to start a discussion about using Option vs Compilation Directive. > Basically, we have 3 options: > > 1. Use PR https://github.com/cython/cython/pull/6531 as is. I will > proceed with the rest of the comments. > 2. Use `Option` directly with no further changes in the Cython codebase. > This is the initial implementation. I created a branch with a version of > the PR with rollback of all changes related to the Compilation Directive > adjustments: > https://github.com/cython/cython/compare/master...matusvalo:shared_library_option?expand=1 > 3. Use code changes/cleanup in https://github.com/cython/cython/pull/6646 > but Shared Utility will be configured using `Option` instead of Compilation > directive (Option 2.). I am not sure it this point makes sense, only in > case that we want to keep changes by @da-woods improves > the code internals. > > I would like to have an agreement on this topic before moving forward with > other comments in the PR. > > > Thanks, > > Matus > > ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
[Cython] Shared utility Option vs. Compilation Directive
Hi All, Based on the comment https://github.com/cython/cython/pull/6531#discussion_r1975502026 I would like to start a discussion about using Option vs Compilation Directive. Basically, we have 3 options: 1. Use PR https://github.com/cython/cython/pull/6531 as is. I will proceed with the rest of the comments. 2. Use `Option` directly with no further changes in the Cython codebase. This is the initial implementation. I created a branch with a version of the PR with rollback of all changes related to the Compilation Directive adjustments: https://github.com/cython/cython/compare/master...matusvalo:shared_library_option?expand=1 3. Use code changes/cleanup in https://github.com/cython/cython/pull/6646 but Shared Utility will be configured using `Option` instead of Compilation directive (Option 2.). I am not sure it this point makes sense, only in case that we want to keep changes by @da-woods improves the code internals. I would like to have an agreement on this topic before moving forward with other comments in the PR. Thanks, Matus ___ cython-devel mailing list cython-devel@python.org https://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] CLI for using Shared utility module
>
>
> I think I tend to agree with David. Here is my point of view:
>
> I am not sure how to implement it. cython command *does not know* about
> the build directory. This is more part of the build system
> (setuptools/meson etc). My point is that the cython command should be
> stupid and just generate a C file with a specified name/location nothing
> more. It is also consistent with current usage of it (it just generates C
> files). I understand that it is the build system responsibility to do all
> the magic around it. As an example, here is working integration of current
> "stupid" utility present in the PR implemented by scipy dev (Ralf Gommers)
> [1]:
> https://github.com/scipy/scipy/compare/main...rgommers:scipy:cy-shared-memoryview?expand=1
>
> The only question is how it can be integrated into setuptools too. The
> most obvious and stupid solution is to generate C file (which was also
> recommended way of working with Cython extension until recently), commit it
> to repository and compile it as regular C extension example is already
> present in test of the PR:
>
> from Cython.Build import cythonize
> from setuptools import setup, Extension
> from Cython.Compiler import Options
> import numpy
>
> extensions = [
> Extension("*", ["**/*.pyx"], include_dirs=[numpy.get_include()]),
> Extension("pkg2.MemoryView", sources=["pkg2/MemoryView.c"])
> ]
>
> setup(
> ext_modules = cythonize(extensions,
> compiler_directives={'use_shared_utility': 'pkg2'})
> )
>
> > Create all necessary directories, then write the output into the target
> directory. No need for a failure.
>
> Here again `cython` command I would keep consistent. If user wants to
> generate file, directory must already exist. This is the same logic as with
> compilation of regular .pyx file:
>
> (cython3.11) root@a3be2cc9ff83:~/dev/cython# cython /foo/bar.pyx
> /root/dev/cython3.11/bin/cython: No such file or directory:
> '/foo/bar.pyx'
>
> Moreover, if we decide to create all directories in cython command, should
> we create also `__init__.py` files too? I think this is an overkill for
> cython command. Of course, in the build system it is different story, build
> backend could do it.
>
> That said, I propose following:
>
> 1. Introduce the cython command parameter `--generate-shared` which has
> the argument name & path of generated C file. E.g.
>
> cython --generate-shared scipy/_lib/cyshared.c
>
> will generate `scipy._lib/cyshared.c` file. This approach should be easily
> integrated in meson build system based on example in [1]. Also `cython`
> command will fail if directory `scipy/_lib` does not exist. Also,
> cython will not care whether it is importable. It will just do stupid and
> simple C file generation. The rest should be done by build system
> (meson/setuptools/whatever). **This will be part of this PR**.
>
> 2. Introduce new `cython` command parameter `--shared` which will take
> fully qualified module name:
>
> cython --shared=scipy._lib.cyshared.memoryview cython_extention.pyx
>
> I think here is already agreement between us. We can do some good old
> bikeshedding about parameter name. This will be **part of this PR **too.
>
> 3. Extend existing `cython.build.cythonize()` function. This should be
> used in the setuptools:
>
> from Cython.Build import cythonize
> from setuptools import setup, Extension
> from Cython.Compiler import Options
> import numpy
>
> extensions = [
> Extension("*", ["**/*.pyx"], include_dirs=[numpy.get_include()]),
> ]
>
> setup(
> ext_modules = cythonize(extensions, shared_module=pkg2.MemoryView)
> )
>
> This function should also ensures that all directories and `__init__.py`
> files are generated, hence the final `.so` module is importable. Currently,
> I cannot see all the details and all devils hiding in the details.
> Hence, I **would postpone it in the separate issue/PR**. Users of meson
> are fine (see [1]) and users setuptools still can use generated and
> commited C file in repo.
>
>
> Please let me know what do you think. This is blocking my further progress
> in PR and I would like to have clear vision to avoid PR flooding with
> changes.
> Moreover, I would like to keep the current PR simple and focus on actual
> code generation of shared module and its integration into the extentions.
> Build related logic can be implemented later on.
>
>
> [1] https://github.com/cython/cython/pull/6531#issuecomment-2551351591
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel
