Re: [Cython] array expressions
On 10/14/2012 08:18 AM, Stefan Behnel wrote: mark florisson, 13.10.2012 20:30: On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: On 10/12/2012 05:50 PM, Robert Bradshaw wrote: On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: On 12 October 2012 08:36, Stefan Behnel wrote: mark florisson, 24.08.2012 20:40: Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation. Did you manage to split off a separate minivect package? I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation). Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython. I don't see a reason for a new pull request. Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories. This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history). What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users. I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it. This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level. Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is). Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time. I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless. I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.) Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] array expressions
On 10/14/2012 10:18 AM, Dag Sverre Seljebotn wrote: On 10/14/2012 08:18 AM, Stefan Behnel wrote: mark florisson, 13.10.2012 20:30: On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: On 10/12/2012 05:50 PM, Robert Bradshaw wrote: On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: On 12 October 2012 08:36, Stefan Behnel wrote: mark florisson, 24.08.2012 20:40: Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation. Did you manage to split off a separate minivect package? I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation). Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython. I don't see a reason for a new pull request. Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories. This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history). What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users. I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it. This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level. Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is). Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time. I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless. I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.) Also, I don't even know when/if numba will use it... Mark, are you able to fill us in on that? Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] array expressions
Dag Sverre Seljebotn, 14.10.2012 10:18: > On 10/14/2012 08:18 AM, Stefan Behnel wrote: >> mark florisson, 13.10.2012 20:30: >>> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: On 10/12/2012 05:50 PM, Robert Bradshaw wrote: > On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: >> On 12 October 2012 08:36, Stefan Behnel wrote: >>> mark florisson, 24.08.2012 20:40: Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 >>> >>> Mark, any news on this? I'd like to see a version merged before >>> the master branch starts diverging all too far - it already >>> requires a bit of adaptation. >>> Did you manage to split off a separate minivect package? > > I'm assuming this has already been looked at, at least to some level, > by Dag, but I'll try to take a brief pass at it too (probably more the > interface than the implementation). Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython. > I don't see a reason for a new pull request. >>> >>> Great. As for the packaging, I'm creating a distribution branch, and a >>> subtree branch. Newer versions of git have a 'subtree' command >>> (previously https://github.com/apenwarr/git-subtree), which allows one >>> to split of, merge, push, and pull subdirectories. >>> >>> This means when users pull the master project, they get the >>> sub-projects as well (without themselves needing newer git versions). >>> Any changes to a subproject can be merged into the subproject, ands. >>> changes can be pulled back in (with a squash option to avoid mixing in >>> the subproject's history). >>> >>> What about using this approach? That way Cython remains stable and >>> pinned on the right minivect version now and in the future, with no >>> burden on users. >> >> I still prefer having separate packages. I mean, we don't ship NumPy >> either, even though a lot of people use Cython together with it. > > This is a very bad comparison. NumPy is not used by Cython at all, but by > Cython-generated modules! Whereas minivect is a tool used in the compiler > itself and working on the AST level. > > Plex would be a better comparison (though bad as well, since Plex is not > optional while minivect is). > >> Keeping the two packages separate helps in keeping the interface between >> both clean. I wouldn't want to end up with Cython shipping some patched up >> version of minivect just because it's so easy, and I would like to allow >> users to install a new version of either Cython or minivect at any time. > > I think this goal (allowing separate upgrades of Cython and/or minivect) is > unrealistic and pointless. > > I think you should look at minivect as "some AST transform algorithms which > numba and Cython are able to share". It doesn't really have a life on its > own, it's just a means for Cython and numba to cooperate. (Really long-term > then hopefully NumPy, numexpr, Theano etc. would jump on too, but that > won't happen just yet. If it does, we can revisit this.) Ok, so I gave bad examples, fair enough. But I still don't see why we should integrate minivect with Cython more deeply than necessary. Could you explain what the advantage would be over having two separate packages? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] array expressions
On 10/14/2012 10:32 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 14.10.2012 10:18: On 10/14/2012 08:18 AM, Stefan Behnel wrote: mark florisson, 13.10.2012 20:30: On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: On 10/12/2012 05:50 PM, Robert Bradshaw wrote: On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: On 12 October 2012 08:36, Stefan Behnel wrote: mark florisson, 24.08.2012 20:40: Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation. Did you manage to split off a separate minivect package? I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation). Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython. I don't see a reason for a new pull request. Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories. This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history). What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users. I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it. This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level. Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is). Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time. I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless. I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.) Ok, so I gave bad examples, fair enough. But I still don't see why we should integrate minivect with Cython more deeply than necessary. Could you explain what the advantage would be over having two separate packages? Mark already raised the issue and we discussed in through in another thread where you participated, and me, Mark and Robert at least agreed on something very similar to what Mark is proposing to do now. If you want to reopen the discussion I wish you had rather dug up the old thread again and respond to arguments in it, rather than us having to repeat the arguments here. (Though please also consider what it does to discussion climate to reopen discussions about trivial details that can also trivially be changed later if better arguments crop up. Mark had a weekend of spare time to get this merged (yay!), and now who-knows-what happens because you decided to bikeshed something that had, at least to the eyes of rest of us, achieved consensus on this list.) Dag ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] array expressions
Dag Sverre Seljebotn, 14.10.2012 11:03: > On 10/14/2012 10:32 AM, Stefan Behnel wrote: >> Dag Sverre Seljebotn, 14.10.2012 10:18: >>> On 10/14/2012 08:18 AM, Stefan Behnel wrote: mark florisson, 13.10.2012 20:30: > On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: >> On 10/12/2012 05:50 PM, Robert Bradshaw wrote: >>> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: On 12 October 2012 08:36, Stefan Behnel wrote: > mark florisson, 24.08.2012 20:40: >> Here a pull request for element-wise array expressions for Cython: >> https://github.com/cython/cython/pull/144 > > Mark, any news on this? I'd like to see a version merged before > the master branch starts diverging all too far - it already > requires a bit of adaptation. > Did you manage to split off a separate minivect package? >>> >>> I'm assuming this has already been looked at, at least to some level, >>> by Dag, but I'll try to take a brief pass at it too (probably more the >>> interface than the implementation). >> >> Thanks for doing that, it'd be great to get this in (but myself I've got >> nothing to spare). I'll admit I was mostly focused on the generated >> code and >> the algorithms in minivect rather than the integration with Cython. >> >>> I don't see a reason for a new pull request. > > Great. As for the packaging, I'm creating a distribution branch, and a > subtree branch. Newer versions of git have a 'subtree' command > (previously https://github.com/apenwarr/git-subtree), which allows one > to split of, merge, push, and pull subdirectories. > > This means when users pull the master project, they get the > sub-projects as well (without themselves needing newer git versions). > Any changes to a subproject can be merged into the subproject, ands. > changes can be pulled back in (with a squash option to avoid mixing in > the subproject's history). > > What about using this approach? That way Cython remains stable and > pinned on the right minivect version now and in the future, with no > burden on users. I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it. >>> >>> This is a very bad comparison. NumPy is not used by Cython at all, but by >>> Cython-generated modules! Whereas minivect is a tool used in the compiler >>> itself and working on the AST level. >>> >>> Plex would be a better comparison (though bad as well, since Plex is not >>> optional while minivect is). >>> Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time. >>> >>> I think this goal (allowing separate upgrades of Cython and/or minivect) is >>> unrealistic and pointless. >>> >>> I think you should look at minivect as "some AST transform algorithms which >>> numba and Cython are able to share". It doesn't really have a life on its >>> own, it's just a means for Cython and numba to cooperate. (Really long-term >>> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that >>> won't happen just yet. If it does, we can revisit this.) >> >> Ok, so I gave bad examples, fair enough. But I still don't see why we >> should integrate minivect with Cython more deeply than necessary. Could you >> explain what the advantage would be over having two separate packages? > > Mark already raised the issue and we discussed in through in another thread This is still the same thread, at least for me. I replied to it when asking for progress. > where you participated, and me, Mark and Robert at least agreed on > something very similar to what Mark is proposing to do now. > > If you want to reopen the discussion I wish you had rather dug up the old > thread again and respond to arguments in it, rather than us having to > repeat the arguments here. The single argument for git based integration was that it simplifies the workflow during development. Meaning, it's a development-only thing and deployments would use separate package installations. I think that's ok as long as it's actually helpful for development and doesn't start getting in the way. I can't comment on that yet, but I wouldn't want to have to deal with it. > (Though please also consider what it does to discussion climate to reopen > discussions about trivial details that can also trivially be changed later > if better arguments crop up. It may be a trivial detail as long as it only has a reasonable impact on the development. If it has a user impact, it's no longer immediately obvious that it's trivial. > Mark had a weekend of spare time to get this > merged (yay!), and
Re: [Cython] array expressions
On 10/14/2012 12:23 PM, Stefan Behnel wrote: Dag Sverre Seljebotn, 14.10.2012 11:03: On 10/14/2012 10:32 AM, Stefan Behnel wrote: Dag Sverre Seljebotn, 14.10.2012 10:18: On 10/14/2012 08:18 AM, Stefan Behnel wrote: mark florisson, 13.10.2012 20:30: On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: On 10/12/2012 05:50 PM, Robert Bradshaw wrote: On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: On 12 October 2012 08:36, Stefan Behnel wrote: mark florisson, 24.08.2012 20:40: Here a pull request for element-wise array expressions for Cython: https://github.com/cython/cython/pull/144 Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation. Did you manage to split off a separate minivect package? I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation). Thanks for doing that, it'd be great to get this in (but myself I've got nothing to spare). I'll admit I was mostly focused on the generated code and the algorithms in minivect rather than the integration with Cython. I don't see a reason for a new pull request. Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories. This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history). What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users. I still prefer having separate packages. I mean, we don't ship NumPy either, even though a lot of people use Cython together with it. This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level. Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is). Keeping the two packages separate helps in keeping the interface between both clean. I wouldn't want to end up with Cython shipping some patched up version of minivect just because it's so easy, and I would like to allow users to install a new version of either Cython or minivect at any time. I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless. I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.) Ok, so I gave bad examples, fair enough. But I still don't see why we should integrate minivect with Cython more deeply than necessary. Could you explain what the advantage would be over having two separate packages? Mark already raised the issue and we discussed in through in another thread This is still the same thread, at least for me. I replied to it when asking for progress. where you participated, and me, Mark and Robert at least agreed on something very similar to what Mark is proposing to do now. If you want to reopen the discussion I wish you had rather dug up the old thread again and respond to arguments in it, rather than us having to repeat the arguments here. The single argument for git based integration was that it simplifies the workflow during development. Meaning, it's a development-only thing and deployments would use separate package installations. I think that's ok as long as it's actually helpful for development and doesn't start getting in the way. I can't comment on that yet, but I wouldn't want to have to deal with it. You know what -- I didn't think things through and was totally wrong here. 'git subtree' has a very different feel to it than 'git submodule'; it means that minivect will be in the tree by default and would have to be stripped out when making a release (or included by default). So revisiting this discussion is indeed in order, as git subtree wasn't mentioned last time. Dag Sverre (Though please also consider what it does to discussion climate to reopen discussions about trivial details that can also trivially be changed later if better arguments crop up. It may be a trivial detail as long as it only has a reasonable impact on the development. If it has a user impact, it's no longer immediately obvious that it's trivial. Mark had a weekend o
Re: [Cython] array expressions
On 14 October 2012 09:23, Dag Sverre Seljebotn wrote: > On 10/14/2012 10:18 AM, Dag Sverre Seljebotn wrote: >> >> On 10/14/2012 08:18 AM, Stefan Behnel wrote: >>> >>> mark florisson, 13.10.2012 20:30: On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: > > On 10/12/2012 05:50 PM, Robert Bradshaw wrote: >> >> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: >>> >>> On 12 October 2012 08:36, Stefan Behnel wrote: mark florisson, 24.08.2012 20:40: > > Here a pull request for element-wise array expressions for Cython: > https://github.com/cython/cython/pull/144 Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation. Did you manage to split off a separate minivect package? >> >> >> I'm assuming this has already been looked at, at least to some level, >> by Dag, but I'll try to take a brief pass at it too (probably more the >> interface than the implementation). > > > Thanks for doing that, it'd be great to get this in (but myself I've > got > nothing to spare). I'll admit I was mostly focused on the generated > code and > the algorithms in minivect rather than the integration with Cython. > >> I don't see a reason for a new pull request. Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories. This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history). What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users. >>> >>> >>> I still prefer having separate packages. I mean, we don't ship NumPy >>> either, even though a lot of people use Cython together with it. >> >> >> This is a very bad comparison. NumPy is not used by Cython at all, but >> by Cython-generated modules! Whereas minivect is a tool used in the >> compiler itself and working on the AST level. >> >> Plex would be a better comparison (though bad as well, since Plex is not >> optional while minivect is). >> >>> Keeping the two packages separate helps in keeping the interface between >>> both clean. I wouldn't want to end up with Cython shipping some >>> patched up >>> version of minivect just because it's so easy, and I would like to allow >>> users to install a new version of either Cython or minivect at any time. >> >> >> I think this goal (allowing separate upgrades of Cython and/or minivect) >> is unrealistic and pointless. >> >> I think you should look at minivect as "some AST transform algorithms >> which numba and Cython are able to share". It doesn't really have a life >> on its own, it's just a means for Cython and numba to cooperate. (Really >> long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, >> but that won't happen just yet. If it does, we can revisit this.) > > > Also, I don't even know when/if numba will use it... Mark, are you able to > fill us in on that? > Yes, numba uses minivect, currently as a subtree (did that yesterday). Submodules are very bad, especially if you develop the submodule itself, and switch branches in the superproject (it will just forget about unpinned changes, which means you just lose your work without so much as a warning). Numba mostly uses and extends the minivect type system and related type conversions (to/from numpy dtypes, to/from ctypes, etc). NumbaPro uses it for array expressions and ufuncs, which will at some point make it into numba. You're entirely right in saying that it breathes only through other projects. I'll add the rest of the response in answer to Stefan's question. > Dag Sverre > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] array expressions
On 14 October 2012 09:32, Stefan Behnel wrote: > Dag Sverre Seljebotn, 14.10.2012 10:18: >> On 10/14/2012 08:18 AM, Stefan Behnel wrote: >>> mark florisson, 13.10.2012 20:30: On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: > On 10/12/2012 05:50 PM, Robert Bradshaw wrote: >> On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: >>> On 12 October 2012 08:36, Stefan Behnel wrote: mark florisson, 24.08.2012 20:40: > Here a pull request for element-wise array expressions for Cython: > https://github.com/cython/cython/pull/144 Mark, any news on this? I'd like to see a version merged before the master branch starts diverging all too far - it already requires a bit of adaptation. Did you manage to split off a separate minivect package? >> >> I'm assuming this has already been looked at, at least to some level, >> by Dag, but I'll try to take a brief pass at it too (probably more the >> interface than the implementation). > > Thanks for doing that, it'd be great to get this in (but myself I've got > nothing to spare). I'll admit I was mostly focused on the generated > code and > the algorithms in minivect rather than the integration with Cython. > >> I don't see a reason for a new pull request. Great. As for the packaging, I'm creating a distribution branch, and a subtree branch. Newer versions of git have a 'subtree' command (previously https://github.com/apenwarr/git-subtree), which allows one to split of, merge, push, and pull subdirectories. This means when users pull the master project, they get the sub-projects as well (without themselves needing newer git versions). Any changes to a subproject can be merged into the subproject, ands. changes can be pulled back in (with a squash option to avoid mixing in the subproject's history). What about using this approach? That way Cython remains stable and pinned on the right minivect version now and in the future, with no burden on users. >>> >>> I still prefer having separate packages. I mean, we don't ship NumPy >>> either, even though a lot of people use Cython together with it. >> >> This is a very bad comparison. NumPy is not used by Cython at all, but by >> Cython-generated modules! Whereas minivect is a tool used in the compiler >> itself and working on the AST level. >> >> Plex would be a better comparison (though bad as well, since Plex is not >> optional while minivect is). >> >>> Keeping the two packages separate helps in keeping the interface between >>> both clean. I wouldn't want to end up with Cython shipping some patched up >>> version of minivect just because it's so easy, and I would like to allow >>> users to install a new version of either Cython or minivect at any time. >> >> I think this goal (allowing separate upgrades of Cython and/or minivect) is >> unrealistic and pointless. >> >> I think you should look at minivect as "some AST transform algorithms which >> numba and Cython are able to share". It doesn't really have a life on its >> own, it's just a means for Cython and numba to cooperate. (Really long-term >> then hopefully NumPy, numexpr, Theano etc. would jump on too, but that >> won't happen just yet. If it does, we can revisit this.) > > Ok, so I gave bad examples, fair enough. But I still don't see why we > should integrate minivect with Cython more deeply than necessary. Could you > explain what the advantage would be over having two separate packages? The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point. For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4. Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba. We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that. > Stefan > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] array expressions
On 14 October 2012 11:23, Stefan Behnel wrote: > Dag Sverre Seljebotn, 14.10.2012 11:03: >> On 10/14/2012 10:32 AM, Stefan Behnel wrote: >>> Dag Sverre Seljebotn, 14.10.2012 10:18: On 10/14/2012 08:18 AM, Stefan Behnel wrote: > mark florisson, 13.10.2012 20:30: >> On 12 October 2012 20:01, Dag Sverre Seljebotn wrote: >>> On 10/12/2012 05:50 PM, Robert Bradshaw wrote: On Fri, Oct 12, 2012 at 3:14 AM, mark florisson wrote: > On 12 October 2012 08:36, Stefan Behnel wrote: >> mark florisson, 24.08.2012 20:40: >>> Here a pull request for element-wise array expressions for Cython: >>> https://github.com/cython/cython/pull/144 >> >> Mark, any news on this? I'd like to see a version merged before >> the master branch starts diverging all too far - it already >> requires a bit of adaptation. >> Did you manage to split off a separate minivect package? I'm assuming this has already been looked at, at least to some level, by Dag, but I'll try to take a brief pass at it too (probably more the interface than the implementation). >>> >>> Thanks for doing that, it'd be great to get this in (but myself I've got >>> nothing to spare). I'll admit I was mostly focused on the generated >>> code and >>> the algorithms in minivect rather than the integration with Cython. >>> I don't see a reason for a new pull request. >> >> Great. As for the packaging, I'm creating a distribution branch, and a >> subtree branch. Newer versions of git have a 'subtree' command >> (previously https://github.com/apenwarr/git-subtree), which allows one >> to split of, merge, push, and pull subdirectories. >> >> This means when users pull the master project, they get the >> sub-projects as well (without themselves needing newer git versions). >> Any changes to a subproject can be merged into the subproject, ands. >> changes can be pulled back in (with a squash option to avoid mixing in >> the subproject's history). >> >> What about using this approach? That way Cython remains stable and >> pinned on the right minivect version now and in the future, with no >> burden on users. > > I still prefer having separate packages. I mean, we don't ship NumPy > either, even though a lot of people use Cython together with it. This is a very bad comparison. NumPy is not used by Cython at all, but by Cython-generated modules! Whereas minivect is a tool used in the compiler itself and working on the AST level. Plex would be a better comparison (though bad as well, since Plex is not optional while minivect is). > Keeping the two packages separate helps in keeping the interface between > both clean. I wouldn't want to end up with Cython shipping some patched up > version of minivect just because it's so easy, and I would like to allow > users to install a new version of either Cython or minivect at any time. I think this goal (allowing separate upgrades of Cython and/or minivect) is unrealistic and pointless. I think you should look at minivect as "some AST transform algorithms which numba and Cython are able to share". It doesn't really have a life on its own, it's just a means for Cython and numba to cooperate. (Really long-term then hopefully NumPy, numexpr, Theano etc. would jump on too, but that won't happen just yet. If it does, we can revisit this.) >>> >>> Ok, so I gave bad examples, fair enough. But I still don't see why we >>> should integrate minivect with Cython more deeply than necessary. Could you >>> explain what the advantage would be over having two separate packages? >> >> Mark already raised the issue and we discussed in through in another thread > > This is still the same thread, at least for me. I replied to it when asking > for progress. > > >> where you participated, and me, Mark and Robert at least agreed on >> something very similar to what Mark is proposing to do now. >> >> If you want to reopen the discussion I wish you had rather dug up the old >> thread again and respond to arguments in it, rather than us having to >> repeat the arguments here. > > The single argument for git based integration was that it simplifies the > workflow during development. Meaning, it's a development-only thing and > deployments would use separate package installations. > > I think that's ok as long as it's actually helpful for development and > doesn't start getting in the way. I can't comment on that yet, but I > wouldn't want to have to deal with it. > > >> (Though please also consider what it does to discussion climate to reopen >> discussions about trivial details that can also trivially be changed later >> if better arguments crop up. > > It may be a trivial detail as long as it only has a reasonable impact on > the devel
Re: [Cython] array expressions
mark florisson, 14.10.2012 13:59: > The problem with minivect as a package is that it caters to different > projects, which have different requirements. Cython and minivect are > quite closely coupled, and any future change, or in the future any > older version may not have the functionality Cython needs, it's not > exactly a stable API at this point. Ok, understood. > For instance Numba needs python > 2.7, whereas Cython needs to be compatible with python 2.4. > > Before releasing minivect I'll verify every time that it doesn't break > Cython, but I currently have no real promises for backwards or forward > compatibility. And that is really because not all use cases have yet > been anticipated, and some really require a change, as I've already > seen with Numba. > > We could list minivect as a dependency, which works for > easy_install/pip users, but I just foresee numerous people running > into problems that didn't install with pip, and I don't think an > exclusion of a 300kb addition is worth any of that. Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately. As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway. Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases. BTW, I would guess that no-one has tested hg with git subtrees yet? Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] array expressions
On 14 October 2012 14:05, Stefan Behnel wrote: > mark florisson, 14.10.2012 13:59: >> The problem with minivect as a package is that it caters to different >> projects, which have different requirements. Cython and minivect are >> quite closely coupled, and any future change, or in the future any >> older version may not have the functionality Cython needs, it's not >> exactly a stable API at this point. > > Ok, understood. > > >> For instance Numba needs python >> 2.7, whereas Cython needs to be compatible with python 2.4. >> >> Before releasing minivect I'll verify every time that it doesn't break >> Cython, but I currently have no real promises for backwards or forward >> compatibility. And that is really because not all use cases have yet >> been anticipated, and some really require a change, as I've already >> seen with Numba. >> >> We could list minivect as a dependency, which works for >> easy_install/pip users, but I just foresee numerous people running >> into problems that didn't install with pip, and I don't think an >> exclusion of a 300kb addition is worth any of that. > > Fine. In that case, I'm for not making minivect a separate package at all > but including it directly and considering it a part of Cython (and Numba > etc.) until there is enough of an interface to make it a reusable separate > package, or at least to support a separate installation and independent > update. Basically, if you can't update it separately, there's no use in > installing it separately. > > As long as we handle this so, we should take care to keep the generic parts > in their separate package directory and the Cython specific parts in > Cython, and try to keep the interface between the two as cleanly separate > as possible, so that we can actually reach a point where both have an > interface. I would guess that the need to support Numba from the same > source base will encourage this kind of separation anyway. Yes, definitely. > Note that this means that minivect will fall under the release schedules of > Cython and Numba (independently), instead of really having its own releases. It can have its own releases as well, but currently there isn't much point :) Minivect can be developed independent of the releases, since Cython and Numba need to explicitly pull in the changes. Let's make a habit of squashing the minivect pulls to avoid its history. I'll also wait for Dag and Robert to see if they have a (final) opinion before merging the subtree. > BTW, I would guess that no-one has tested hg with git subtrees yet? Heh, no. The hg support didn't work great for me (it was really slow), can I ask why you're using it? > Stefan > > ___ > cython-devel mailing list > cython-devel@python.org > http://mail.python.org/mailman/listinfo/cython-devel ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] array expressions
mark florisson, 14.10.2012 15:13: > The hg support didn't work great for me (it was really slow), It's fast enough for almost everything. What's really slow is when it has to figure out how to match branches after head changes - I consider that a pretty serious bug. But that happens quite rarely. And I usually push only the master branch anyway, which is quick. Obviously, hg is way faster against a mercurial server than against a git server. > can I ask why you're using it? I prefer the interface of hg over that of git, that's the main reason. In fact, whenever I have to touch git, it quickly lives up to its name: http://dict.leo.org/ende?lang=en&search=igitt Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] array expressions
On 10/14/2012 03:05 PM, Stefan Behnel wrote: mark florisson, 14.10.2012 13:59: The problem with minivect as a package is that it caters to different projects, which have different requirements. Cython and minivect are quite closely coupled, and any future change, or in the future any older version may not have the functionality Cython needs, it's not exactly a stable API at this point. Ok, understood. For instance Numba needs python 2.7, whereas Cython needs to be compatible with python 2.4. Before releasing minivect I'll verify every time that it doesn't break Cython, but I currently have no real promises for backwards or forward compatibility. And that is really because not all use cases have yet been anticipated, and some really require a change, as I've already seen with Numba. We could list minivect as a dependency, which works for easy_install/pip users, but I just foresee numerous people running into problems that didn't install with pip, and I don't think an exclusion of a 300kb addition is worth any of that. Fine. In that case, I'm for not making minivect a separate package at all but including it directly and considering it a part of Cython (and Numba etc.) until there is enough of an interface to make it a reusable separate package, or at least to support a separate installation and independent update. Basically, if you can't update it separately, there's no use in installing it separately. As long as we handle this so, we should take care to keep the generic parts in their separate package directory and the Cython specific parts in Cython, and try to keep the interface between the two as cleanly separate as possible, so that we can actually reach a point where both have an interface. I would guess that the need to support Numba from the same source base will encourage this kind of separation anyway. Note that this means that minivect will fall under the release schedules of Cython and Numba (independently), instead of really having its own releases. +1 to Mark's proposal, this seems nice. BTW, I would guess that no-one has tested hg with git subtrees yet? git subtree is just a utility for pushing/pulling the history of a subdirectory to/from another repository. As long as you don't need to update to a new version of minivect, you shouldn't be affected at all; when you don't actually use the "git subtree" commands, it's just a normal directory under git control. But if one needs to fix a bug in minivect when doing Cython development, I'm guessing hg-git can't help you move the patch upstream (but you can always just submit a normal Unix diff upstream instead if you don't want to use git). Dag Sverre ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
