[Numpy-discussion] A regression in numpy 1.10: VERY slow memory mapped file generation

2015-10-13 Thread Nadav Horesh
I have binary files of size range between few MB to 1GB, which I read process as memory mapped files (via np.memmap). Until numpy 1.9 the creation  of recarray on an existing file (without reading its content) was instantaneous, and now it takes ~6 seconds (system: archlinux on sandy bridge). A

Re: [Numpy-discussion] Making datetime64 timezone naive

2015-10-13 Thread Marten van Kerkwijk
> > > However, numpy datetime is optimized for compact storage and fast > computation of absolute deltas (actual hours, minutes, seconds... not > calendar units like "the next day" ). > > Except that ironically it actually can't compute absolute deltas > accurately with one second resolution, becau

Re: [Numpy-discussion] Make all comparisons with NaT false?

2015-10-13 Thread Jeff Reback
Here another oddity to add to the list In [28]: issubclass(np.datetime64,np.integer) Out[28]: False In [29]: issubclass(np.timedelta64,np.integer) Out[29]: True On Tue, Oct 13, 2015 at 5:44 PM, Chris Barker wrote: > On Sun, Oct 11, 2015 at 8:38 PM, Stephan Hoyer wrote: > >> Currently, NaT (n

Re: [Numpy-discussion] Making datetime64 timezone naive

2015-10-13 Thread Nathaniel Smith
On Oct 13, 2015 3:49 PM, "Chris Barker" wrote: > [...] > However, numpy datetime is optimized for compact storage and fast computation of absolute deltas (actual hours, minutes, seconds... not calendar units like "the next day" ). Except that ironically it actually can't compute absolute deltas a

Re: [Numpy-discussion] Making datetime64 timezone naive

2015-10-13 Thread Chris Barker
On Tue, Oct 13, 2015 at 3:21 PM, Nathaniel Smith wrote: > > If you are going to make datetime64 more like datetime.datetime, please > consider adding the "fold" bit. See PEP 495. [1] > The challenge here is that we literally do not have a bit too use :-) > hmm -- I was first thinking that this

Re: [Numpy-discussion] Deprecating unitless timedelta64 and "safe" casting of integers to timedelta64

2015-10-13 Thread Chris Barker
On Tue, Oct 13, 2015 at 10:48 AM, Stephan Hoyer wrote: > This led to my discovery that NumPy currently supports unit-less > timedeltas (e.g., "np.timedelta64(5)"), which indicate some sort of generic > time difference. The current behavior is to take the time units from the > other argument when

Re: [Numpy-discussion] Making datetime64 timezone naive

2015-10-13 Thread Nathaniel Smith
On Oct 12, 2015 11:48 AM, "Alexander Belopolsky" wrote: > > > On Mon, Oct 12, 2015 at 3:10 AM, Stephan Hoyer wrote: >> >> The tentative consensus from last year's discussion was that we should make datetime64 timezone naive, like the standard library's datetime.datetime > > > > If you are going t

Re: [Numpy-discussion] Making datetime64 timezone naive

2015-10-13 Thread Chris Barker
On Mon, Oct 12, 2015 at 12:38 AM, Nathaniel Smith wrote: > > As a temporary measure, we still will parse datetimes that include a > > timezone specification by converting them to UTC, but will issue a > > DeprecationWarning. This is important for a smooth transition, because at > > the very least

Re: [Numpy-discussion] Making datetime64 timezone naive

2015-10-13 Thread Chris Barker
On Mon, Oct 12, 2015 at 11:48 AM, Alexander Belopolsky wrote: > If you are going to make datetime64 more like datetime.datetime, please > consider adding the "fold" bit. See PEP 495. [1] > > [1]: https://www.python.org/dev/peps/pep-0495/ > well, adding any timezone support is not (yet) in the t

Re: [Numpy-discussion] Make all comparisons with NaT false?

2015-10-13 Thread Chris Barker
On Sun, Oct 11, 2015 at 8:38 PM, Stephan Hoyer wrote: > Currently, NaT (not a time) does not have any special treatment when used > in comparison with datetime64/timedelta64 objects. > > To me, this seems a little crazy for a value meant to denote a > missing/invalid time -- NaT should really hav

Re: [Numpy-discussion] Deprecating unitless timedelta64 and "safe" casting of integers to timedelta64

2015-10-13 Thread Nathaniel Smith
On Oct 13, 2015 10:48 AM, "Stephan Hoyer" wrote: > > As part of the datetime64 cleanup I've been working on over the past few days, I noticed that NumPy's casting rules for np.datetime64('NaT') were not working properly: > https://github.com/numpy/numpy/pull/6465 > > This led to my discovery that

[Numpy-discussion] Deprecating unitless timedelta64 and "safe" casting of integers to timedelta64

2015-10-13 Thread Stephan Hoyer
As part of the datetime64 cleanup I've been working on over the past few days, I noticed that NumPy's casting rules for np.datetime64('NaT') were not working properly: https://github.com/numpy/numpy/pull/6465 This led to my discovery that NumPy currently supports unit-less timedeltas (e.g., "np.ti

Re: [Numpy-discussion] Making datetime64 timezone naive

2015-10-13 Thread Stephan Hoyer
On Mon, Oct 12, 2015 at 12:38 AM, Nathaniel Smith wrote: > > One possible strategy here would be to do some corpus analysis to find > out whether anyone is actually using it, like I did for the ufunc ABI > stuff: > https://github.com/njsmith/codetrawl > https://github.com/njsmith/ufunc-abi-an

Re: [Numpy-discussion] Making datetime64 timezone naive

2015-10-13 Thread Benjamin Root
I'd be totally in support of switching to timezone naive form. While it would be ideal that everyone stores their dates in UTC, the real world is messy and most of the time, people are just loading dates as-is and don't even care about timezones. I work on machines with different TZs, and I hate it