[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.
Ilhan Polat wrote: > I think all these point to the missing convenient functionality that > extends arrays. In matlab "[0 arr 10]" nicely extends the array to a new > one but in NumPy you need to punch quite some code and some courage to > remember whether it is hstack or vstack or concat or block as the correct > naming which decreases the "code morale". Not having a convenient workaround is not the only problem. The workaround is wastefull with memory and involves unnecessary copying of an array. Having a keyword implemented with these concerns in mind might avoid this. ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.
I was trying to get a feel for how often the work around occurs. I found three clear examples in Scipy and one unclear case. One case in holoviews. Two in numpy. One from soundappraisal's code base. Next to prepending to the output, I also see prepending to the input as a workaround. Some examples of workarounds: scipy: (prepending to the output) scipy/scipy/sparse/construct.py: '''Python row_offsets = np.append(0, np.cumsum(brow_lengths)) col_offsets = np.append(0, np.cumsum(bcol_lengths)) ''' scipy/scipy/sparse/dia.py: '''Python indptr = np.zeros(num_cols + 1, dtype=idx_dtype) indptr[1:offset_len+1] = np.cumsum(mask.sum(axis=0)) ''' scipy/scipy/sparse/csgraph/_tools.pyx: '''Python indptr = np.zeros(N + 1, dtype=ITYPE) indptr[1:] = mask.sum(1).cumsum() ''' Not sure whether this is also an example: scipy/scipy/stats/_hypotests_pythran.py '''Python # Now fill in the values. We cannot use cumsum, unfortunately. val = 0.0 if minj == 0 else 1.0 for jj in range(maxj - minj): j = jj + minj val = (A[jj + minj - lastminj] * i + val * j) / (i + j) A[jj] = val ''' holoviews: (prepending to the input) '''Python # We add a zero in the begging for the cumulative sum points = np.zeros((areas_in_radians.shape[0] + 1)) points[1:] = areas_in_radians points = points.cumsum() ''' numpy (prepending to the input): numpy/numpy/lib/_iotools.py : '''Python idx = np.cumsum([0] + list(delimiter)) ''' numpy/numpy/lib/histograms.py '''Python cw = np.concatenate((zero, sw.cumsum())) ''' soundappraisal own code: (prepending to the output) '''Python def get_cumulativepixelareas(whiteboard): whiteboard['cumulativepixelareas'] = \ np.concatenate((np.array([0, ]), np.cumsum(whiteboard['pixelareas']))) return True ''' ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.
On Fri, Aug 18, 2023 at 10:59 AM Ronald van Elburg < r.a.j.van.elb...@hetnet.nl> wrote: > I was trying to get a feel for how often the work around occurs. I found > three clear examples in Scipy and one unclear case. One case in holoviews. > Two in numpy. One from soundappraisal's code base. > Thank you Ronald. I think we indeed have more than enough evidence that allowing prepending an initial zero is useful. I think the API currently proposed in https://github.com/data-apis/array-api/pull/653 should work for that: def cumulative_sum( x: array, /, *, axis: Optional[int] = None, dtype: Optional[dtype] = None, include_initial: bool = False, ) -> array: Whether it's necessary to have other keywords to prepend anything other than zero, or append rather than prepend, is a lot less clear. Did you find a clear need for those things? Cheers, Ralf ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.
> Whether it's necessary to have other keywords to prepend anything other > than zero, or append rather than prepend, is a lot less clear. Did you find > a clear need for those things? No, I haven't found them. For streaming data there might be usecases for starting with an initial offset, but I expect there might be no need for a returned offset there. What is notable is that all examples above are 1D. To get the behavior of the API right, the simplest solution is to make the workaround part of the implementation. What I was pondering on is whether it is desirable to allocate the memory once and avoid copying the data. What is the price to pay in terms of code complexity and developer time? Also if the accumulation would run in place on a copy of the input data then prepending the input might be a good option introducing very little new overhead. ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
[Numpy-discussion] Re: Add to NumPy a function to compute cumulative sums from 0.
On Fri, Aug 18, 2023 at 4:59 AM Ronald van Elburg < r.a.j.van.elb...@hetnet.nl> wrote: > I was trying to get a feel for how often the work around occurs. I found > three clear examples in Scipy and one unclear case. One case in holoviews. > Two in numpy. One from soundappraisal's code base. > See also my comment from back in 2020: https://github.com/numpy/numpy/pull/14542#issuecomment-586494608 Anyone interested in this enhancement is encouraged to review the discussion in that pull request (https://github.com/numpy/numpy/pull/14542), and an earlier issue from 2015: https://github.com/numpy/numpy/issues/6044 Warren > Next to prepending to the output, I also see prepending to the input as a > workaround. > > Some examples of workarounds: > > scipy: (prepending to the output) > > scipy/scipy/sparse/construct.py: > > '''Python > row_offsets = np.append(0, np.cumsum(brow_lengths)) > col_offsets = np.append(0, np.cumsum(bcol_lengths)) > ''' > > scipy/scipy/sparse/dia.py: > > '''Python > indptr = np.zeros(num_cols + 1, dtype=idx_dtype) > indptr[1:offset_len+1] = np.cumsum(mask.sum(axis=0)) > ''' > > scipy/scipy/sparse/csgraph/_tools.pyx: > > '''Python > indptr = np.zeros(N + 1, dtype=ITYPE) > indptr[1:] = mask.sum(1).cumsum() > ''' > > Not sure whether this is also an example: > > scipy/scipy/stats/_hypotests_pythran.py > '''Python > # Now fill in the values. We cannot use cumsum, unfortunately. > val = 0.0 if minj == 0 else 1.0 > for jj in range(maxj - minj): > j = jj + minj > val = (A[jj + minj - lastminj] * i + val * j) / (i + j) > A[jj] = val > ''' > > holoviews: (prepending to the input) > > '''Python > # We add a zero in the begging for the cumulative sum > points = np.zeros((areas_in_radians.shape[0] + 1)) > points[1:] = areas_in_radians > points = points.cumsum() > ''' > > > numpy (prepending to the input): > > numpy/numpy/lib/_iotools.py : > > '''Python > idx = np.cumsum([0] + list(delimiter)) > ''' > > numpy/numpy/lib/histograms.py > > '''Python > cw = np.concatenate((zero, sw.cumsum())) > ''' > > > > soundappraisal own code: (prepending to the output) > > '''Python > def get_cumulativepixelareas(whiteboard): > whiteboard['cumulativepixelareas'] = \ > np.concatenate((np.array([0, ]), > np.cumsum(whiteboard['pixelareas']))) > return True > ''' > ___ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: warren.weckes...@gmail.com > ___ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
