Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
On Mo, 2016-09-05 at 18:31 -0400, Marten van Kerkwijk wrote: > Actually, on those names: an alternative to your proposal would be to > introduce only one new method which can do all types of indexing, > depending on a keyword argument, i.e., something like > ``` > def getitem(self, item, mode='outer'): > ... > ``` Have I been overthinking this, eh? Just making it `__getitem__(self, index, mode=...)` and then from `vindex` calling the subclasses `__getitem__(self, index, mode="vector")` or so would already solve the issue almost fully? Only thing I am not quite sure about: 1. Is `__getitem__` in some way special to make this difficult (also considering some new ideas like allowing object[a=4]? 2. Do we currently have serious deficiencies we want to fix, and could maybe not fix like that. > -- Marten > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
On Di, 2016-09-06 at 09:37 +0200, Sebastian Berg wrote: > On Mo, 2016-09-05 at 18:31 -0400, Marten van Kerkwijk wrote: > > > > Actually, on those names: an alternative to your proposal would be > > to > > introduce only one new method which can do all types of indexing, > > depending on a keyword argument, i.e., something like > > ``` > > def getitem(self, item, mode='outer'): > > ... > > ``` > Have I been overthinking this, eh? Just making it `__getitem__(self, > index, mode=...)` and then from `vindex` calling the subclasses > `__getitem__(self, index, mode="vector")` or so would already solve > the > issue almost fully? Only thing I am not quite sure about: > > 1. Is `__getitem__` in some way special to make this difficult (also > considering some new ideas like allowing object[a=4]? OK; I think the C-side slot cannot get the kwarg likely, but probably you can find a solution for that > 2. Do we currently have serious deficiencies we want to fix, and > could > maybe not fix like that. > > > > > > -- Marten > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org > > https://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
On Tue, Sep 6, 2016 at 8:46 AM, Sebastian Berg wrote: > > On Di, 2016-09-06 at 09:37 +0200, Sebastian Berg wrote: > > On Mo, 2016-09-05 at 18:31 -0400, Marten van Kerkwijk wrote: > > > > > > Actually, on those names: an alternative to your proposal would be > > > to > > > introduce only one new method which can do all types of indexing, > > > depending on a keyword argument, i.e., something like > > > ``` > > > def getitem(self, item, mode='outer'): > > > ... > > > ``` > > Have I been overthinking this, eh? Just making it `__getitem__(self, > > index, mode=...)` and then from `vindex` calling the subclasses > > `__getitem__(self, index, mode="vector")` or so would already solve > > the > > issue almost fully? Only thing I am not quite sure about: > > > > 1. Is `__getitem__` in some way special to make this difficult (also > > considering some new ideas like allowing object[a=4]? > > OK; I think the C-side slot cannot get the kwarg likely, but probably > you can find a solution for that Well, the solution is to use a different name, I think. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Which NumPy/Numpy/numpy spelling?
Hi, The general consensus seems to be in favour of using "NumPy" when referring to the project and "numpy" as a module name. Please note that there are currently PRs in 3 different repositories implementing this practice: - numpy docs: https://github.com/numpy/numpy/pull/8021 - numpy.org website: https://github.com/numpy/numpy.org/pull/5 - Scipy Lecture Notes: https://github.com/scipy-lectures/scipy-lecture-notes/pull/265 The name of the mailing lists still conflicts with the practice, but perhaps it would be more hassle to rename it than it's worth it. :). There is also some instances of "Numpy" spelling in numpy sources, but changing them would probably need more care and time. If everyone agrees the PRs could be merged together. Please review and comment! Thanks in advance, Bartosz ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
On Di, 2016-09-06 at 10:57 +0100, Robert Kern wrote: > On Tue, Sep 6, 2016 at 8:46 AM, Sebastian Berg s.net> wrote: > > > > On Di, 2016-09-06 at 09:37 +0200, Sebastian Berg wrote: > > > On Mo, 2016-09-05 at 18:31 -0400, Marten van Kerkwijk wrote: > > > > > > > > Actually, on those names: an alternative to your proposal would > be > > > > to > > > > introduce only one new method which can do all types of > indexing, > > > > depending on a keyword argument, i.e., something like > > > > ``` > > > > def getitem(self, item, mode='outer'): > > > > ... > > > > ``` > > > Have I been overthinking this, eh? Just making it > `__getitem__(self, > > > index, mode=...)` and then from `vindex` calling the subclasses > > > `__getitem__(self, index, mode="vector")` or so would already > solve > > > the > > > issue almost fully? Only thing I am not quite sure about: > > > > > > 1. Is `__getitem__` in some way special to make this difficult > (also > > > considering some new ideas like allowing object[a=4]? > > > > OK; I think the C-side slot cannot get the kwarg likely, but > probably > > you can find a solution for that > > Well, the solution is to use a different name, I think. > Yeah :). Which goes back to `__numpy_getitem__` or so, just with a slightly different (simpler) API. Something more along: 1. If subclass has `__numpy_getitem__` call it with the method keyword. Or just add the argument to `__getitem__` which should likely work as well. 2. Implement `ndarray.__numpy_getitem__` which takes the method keyword and subclasses would call it instead of `ndarray.__getitem__` their base class call. The option I first mentioned would be similar but allows to give a bit of extra information to the subclass which may be useful. But if no one actually needs that information (this information would be things available after inspection of the indexing object) it just adds quite a bit of code and thus a maintenance burden). Such a new method could of course do things slightly different (such as the scalar cases, I really have to understand that wrapping thing, I am always worried about the array of array case as well. Or that annoying setitem calls getitem. Or maybe not wrap the array at all.). - Sebastian > -- > Robert Kern > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
On Mon, Sep 5, 2016 at 6:02 PM, Marten van Kerkwijk < m.h.vankerkw...@gmail.com> wrote: > p.s. Just to be clear: personally, I think we should have neither > `__numpy_getitem__` nor a mixin; we should just get the quite > wonderful new indexing methods! +1 I don't maintain ndarray subclasses (I prefer composition), but I don't think it's too difficult to require implementing vindex and oindex properties from scratch. Side note: I would prefer the more verbose "legacy_index" to "lindex". We really want to discourage this one, and two new abbreviations are bad enough. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
On Tuesday, September 6, 2016, Stephan Hoyer wrote: > On Mon, Sep 5, 2016 at 6:02 PM, Marten van Kerkwijk < > m.h.vankerkw...@gmail.com > > wrote: > >> p.s. Just to be clear: personally, I think we should have neither >> `__numpy_getitem__` nor a mixin; we should just get the quite >> wonderful new indexing methods! > > > +1 > > I don't maintain ndarray subclasses (I prefer composition), but I don't > think it's too difficult to require implementing vindex and oindex > properties from scratch. > > Side note: I would prefer the more verbose "legacy_index" to "lindex". We > really want to discourage this one, and two new abbreviations are bad > enough. > Very much agreed. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
I'm also in the non-subclass array-like camp, and I'd love to just write vindex and oindex methods, then have: def __getitem__(self, idx): return np.dispatch_getitem(self, idx) Where "dispatch_getitem" does some basic argument checking and calls either vindex or oindex as appropriate. Maybe that fits better as a mixin; I don't really mind either way. On Tue, Sep 6, 2016 at 12:11 PM, Nathan Goldbaum wrote: > > > On Tuesday, September 6, 2016, Stephan Hoyer wrote: > >> On Mon, Sep 5, 2016 at 6:02 PM, Marten van Kerkwijk < >> m.h.vankerkw...@gmail.com> wrote: >> >>> p.s. Just to be clear: personally, I think we should have neither >>> `__numpy_getitem__` nor a mixin; we should just get the quite >>> wonderful new indexing methods! >> >> >> +1 >> >> I don't maintain ndarray subclasses (I prefer composition), but I don't >> think it's too difficult to require implementing vindex and oindex >> properties from scratch. >> >> Side note: I would prefer the more verbose "legacy_index" to "lindex". We >> really want to discourage this one, and two new abbreviations are bad >> enough. >> > > Very much agreed. > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
I'd love to solve it with `__getitem__`... Since most subclasses will
have that defined that with just a single argument, calling it from
`oindex` with an extra mode argument set will properly fail, so good
in that sense (although one better ensure a useful error message...).
Another option would be to store the mode as an additional part in the
index tuple (perhaps as a dict), i.e., in python form do something
like:
```
def __getitem__(self, index):
if isinstance(index, tuple) and isinstance(index[-1], dict):
*index, kwargs = index
else:
kwargs = {}
return self._internal_indexer(index, **kwargs)
```
This way, if all a subclass does is to call `super(SubClass,
self).__getitem__(index)` (as does `Quantity`; it does not look at the
index at all), it would work automagically. Indeed, one could them
decide the type of index even by regular slicing in the following way
```
array[:, 10, {'mode': 'vector'}]
```
which one could of course have a special token for (like `np.newaxis`
for `None`), so that it would be something like
```
array[:, 10, np.vector_index]
```
However, looking at the above, I fear this is too baroque even by my standards!
-- Marten
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
In a separate message, since perhaps a little less looney: yet another option would be work by analogy with np.ix_ and define pre-dispatch index preparation routines np.ox_ and np.vx_ (say), which would be used as in: ``` array[np.ox_[:, 10]] -- or -- array[np.vx_[:, 10]] ``` This could work if those functions each return something appropriate for the legacy indexer, or, if that is not possible, a specific subclass of tuple as a marker that gets interpreted further up. In the end, though, probably also too complicated. It may remain best to simply implement the new methods instead and keep it at that! -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
On Di, 2016-09-06 at 10:10 -0700, Stephan Hoyer wrote: > On Mon, Sep 5, 2016 at 6:02 PM, Marten van Kerkwijk gmail.com> wrote: > > p.s. Just to be clear: personally, I think we should have neither > > `__numpy_getitem__` nor a mixin; we should just get the quite > > wonderful new indexing methods! > +1 > > I don't maintain ndarray subclasses (I prefer composition), but I > don't think it's too difficult to require implementing vindex and > oindex properties from scratch. > Well, in some sense why I brought it up is masked arrays. They have quite a bit of code in `__getitem__` including doing an identical indexing operation to the mask. I suppose you can always solve it with such code: def __special_getitem__(self, index, method=None): # do magic if method is None: # (not sure if this gets passed the base class res = super()[index] elif method == "outer": res = super().oindex[index] # ... # more magic. def __getitem__(self, index): self.__special_getitem__(index) @property def oindex(self): # define class elsewhere I guess class _indexer(object): def __init__(self, arr): self.arr = arr def __getitem__(self, index) arr.__special_getitem__(index, method='oter') return _indexer(self) Though I am not 100% sure without testing, a superclass method that understands the `method` kwarg might work better. We can teach numpy to pass in that `method` to getitem so that you don't have to implement that `_indexer` class for the special attribute. I first started to do that for MaskedArrays, and while it is not hard, it seemed a bit tedious. If we move this to a method with a new name, a slight advantage would be that other oddities could be removed maybe. By now it seems to me that nobody really needs the extra information (i.e. preprocessing information of the indexing tuple)...? I thought it could be useful to know things about the result, but I suppose you can check most things (view vs. no view; field access; scalar access) also afterwards? > Side note: I would prefer the more verbose "legacy_index" to > "lindex". We really want to discourage this one, and two new > abbreviations are bad enough. Sounds good to me. > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)
On Di, 2016-09-06 at 13:59 -0400, Marten van Kerkwijk wrote: > In a separate message, since perhaps a little less looney: yet > another > option would be work by analogy with np.ix_ and define pre-dispatch > index preparation routines np.ox_ and np.vx_ (say), which would be > used as in: > ``` > array[np.ox_[:, 10]] -- or -- array[np.vx_[:, 10]] > ``` > This could work if those functions each return something appropriate > for the legacy indexer, or, if that is not possible, a specific > subclass of tuple as a marker that gets interpreted further up. Sure, it would be a solution, but not sure it is any better implementation wise then just passing an extra argument. As for the syntax for plain arrays, I am not convinced to be honest. - Sebastian > In the end, though, probably also too complicated. It may remain best > to simply implement the new methods instead and keep it at that! > > -- Marten > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] State-of-the-art to use a C/C++ library from Python
On Fri, Sep 2, 2016 at 1:16 AM, Peter Creasey wrote: > > I'm not quite sure which approach is state-of-the-art as of 2016. How > would > > you do it if you had to make a C/C++ library available in Python right > now? > > > > In my case, I have a C library with some scientific functions on matrices > > and vectors. You will typically call a few functions to configure the > > computation, then hand over some pointers to existing buffers containing > > vector data, then start the computation, and finally read back the data. > > The library also can use MPI to parallelize. > Cython works really well for this. ctypes is a better option if you have a "black box" shared lib you want a call a couple functions in. Cython works better if you want to write a little "thicker" wrapper around youe C code -- i.e. it may do a scalar computation, and you want to apply it to an entire numpy array, at C speed. Either would work in this case, But I like Cyton better, as long as I don't have compilation issues. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
