Re: [Numpy-discussion] float128 casting rounding as if it were float64

2011-10-15 Thread Charles R Harris
On Sat, Oct 15, 2011 at 12:54 PM, Matthew Brett wrote: > Hi, > > On Wed, Oct 12, 2011 at 11:24 AM, Charles R Harris > wrote: > > > > > > On Tue, Oct 11, 2011 at 12:17 PM, Matthew Brett > > > wrote: > >> > >> Hi, > >> > >> While struggling with floating point precision, I ran into this: > >> > >>

Re: [Numpy-discussion] float128 casting rounding as if it were float64

2011-10-15 Thread Matthew Brett
Hi, On Wed, Oct 12, 2011 at 11:24 AM, Charles R Harris wrote: > > > On Tue, Oct 11, 2011 at 12:17 PM, Matthew Brett > wrote: >> >> Hi, >> >> While struggling with floating point precision, I ran into this: >> >> In [52]: a = 2**54+3 >> >> In [53]: a >> Out[53]: 18014398509481987L >> >> In [54]:

Re: [Numpy-discussion] float128 casting rounding as if it were float64

2011-10-12 Thread Charles R Harris
On Tue, Oct 11, 2011 at 12:17 PM, Matthew Brett wrote: > Hi, > > While struggling with floating point precision, I ran into this: > > In [52]: a = 2**54+3 > > In [53]: a > Out[53]: 18014398509481987L > > In [54]: np.float128(a) > Out[54]: 18014398509481988.0 > > In [55]: np.float128(a)-1 > Out[55]

[Numpy-discussion] float128 casting rounding as if it were float64

2011-10-11 Thread Matthew Brett
Hi, While struggling with floating point precision, I ran into this: In [52]: a = 2**54+3 In [53]: a Out[53]: 18014398509481987L In [54]: np.float128(a) Out[54]: 18014398509481988.0 In [55]: np.float128(a)-1 Out[55]: 18014398509481987.0 The line above tells us that float128 can exactly repres