[ python-Bugs-1770416 ] Decimal.__int__ overflows for large values
Bugs item #1770416, was opened at 2007-08-08 17:43
Message generated for change (Comment added) made by ajaksu2
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770416&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Jason G (aryx)
Assigned to: Nobody/Anonymous (nobody)
Summary: Decimal.__int__ overflows for large values
Initial Comment:
This also affects Decimal.__hash__, since it [indirectly] calls Decimal.__int__.
>>> from decimal import Decimal as D
>>> e = D("1e1234567890987654321")
>>> int(e)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/decimal.py", line 1501, in __int__
s = ''.join(map(str, self._int)) + '0'*self._exp
OverflowError: cannot fit 'long' into an index-sized integer
>>> e = D("1e1234567890")
>>> int(e)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/decimal.py", line 1501, in __int__
s = ''.join(map(str, self._int)) + '0'*self._exp
MemoryError
Also, for values that do work this is incredibly slow if they are still fairly
large.
--
Comment By: ajaksu (ajaksu2)
Date: 2007-08-09 05:09
Message:
Logged In: YES
user_id=1200609
Originator: NO
Hi Jason,
The OverflowError is related to "index-sized ints" as in "ints that are
valid indexes for sequences", try:
>>> e = "0" * 1234567890
So it seems that this error is avoiding the creation of a string of length
1234567890, which is a good thing (sorta) :)
Once I tried to implement a dec2long function that was based on numbers
instead of strings, see if it helps (it's VERY slow and naive, but IIRC it
was a bit faster than the original version and correct):
http://groups.google.com/group/comp.lang.python/msg/aba7264ab38eb25e
Now, do you really need all that precision for such huge numbers? I know I
didn't ;)
Daniel
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770416&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1770416 ] Decimal.__int__ overflows for large values
Bugs item #1770416, was opened at 2007-08-08 18:13
Message generated for change (Comment added) made by aryx
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770416&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Jason G (aryx)
Assigned to: Nobody/Anonymous (nobody)
Summary: Decimal.__int__ overflows for large values
Initial Comment:
This also affects Decimal.__hash__, since it [indirectly] calls Decimal.__int__.
>>> from decimal import Decimal as D
>>> e = D("1e1234567890987654321")
>>> int(e)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/decimal.py", line 1501, in __int__
s = ''.join(map(str, self._int)) + '0'*self._exp
OverflowError: cannot fit 'long' into an index-sized integer
>>> e = D("1e1234567890")
>>> int(e)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/decimal.py", line 1501, in __int__
s = ''.join(map(str, self._int)) + '0'*self._exp
MemoryError
Also, for values that do work this is incredibly slow if they are still fairly
large.
--
>Comment By: Jason G (aryx)
Date: 2007-08-09 12:39
Message:
Logged In: YES
user_id=1289703
Originator: YES
Hey Daniel,
The bigger issue for us is mostly the fact that Decimal.__hash__ us
calling Decimal.__int__ and not because we want an integer/long version of
a very large Decimal. We do not actually cast the decimal into an int/long
explicitly. I wouldn't have any issues if Decimal.__int__ remained as it
is, but I think it would be a good idea for Decimal.__hash__ to do
something differently. Probably something that is fast and simple, such as
hash( self.as_tuple() ), self being a Decimal of course.
Our project is a CAS and we use Decimal for our real number class. I
happened to run into this issue when I was implementing approximation of
log(x) for extremely large/small values of x. I just started keyboard
bashing numbers and behold, Decimal crashed on me :)
--
Comment By: ajaksu (ajaksu2)
Date: 2007-08-09 05:39
Message:
Logged In: YES
user_id=1200609
Originator: NO
Hi Jason,
The OverflowError is related to "index-sized ints" as in "ints that are
valid indexes for sequences", try:
>>> e = "0" * 1234567890
So it seems that this error is avoiding the creation of a string of length
1234567890, which is a good thing (sorta) :)
Once I tried to implement a dec2long function that was based on numbers
instead of strings, see if it helps (it's VERY slow and naive, but IIRC it
was a bit faster than the original version and correct):
http://groups.google.com/group/comp.lang.python/msg/aba7264ab38eb25e
Now, do you really need all that precision for such huge numbers? I know I
didn't ;)
Daniel
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770416&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1770416 ] Decimal.__int__ overflows for large values
Bugs item #1770416, was opened at 2007-08-08 17:43
Message generated for change (Comment added) made by ajaksu2
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770416&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Jason G (aryx)
Assigned to: Nobody/Anonymous (nobody)
Summary: Decimal.__int__ overflows for large values
Initial Comment:
This also affects Decimal.__hash__, since it [indirectly] calls Decimal.__int__.
>>> from decimal import Decimal as D
>>> e = D("1e1234567890987654321")
>>> int(e)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/decimal.py", line 1501, in __int__
s = ''.join(map(str, self._int)) + '0'*self._exp
OverflowError: cannot fit 'long' into an index-sized integer
>>> e = D("1e1234567890")
>>> int(e)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/decimal.py", line 1501, in __int__
s = ''.join(map(str, self._int)) + '0'*self._exp
MemoryError
Also, for values that do work this is incredibly slow if they are still fairly
large.
--
Comment By: ajaksu (ajaksu2)
Date: 2007-08-09 14:21
Message:
Logged In: YES
user_id=1200609
Originator: NO
I see. Inheriting from Decimal and overloading __hash__ is a way to solve
your problem, but it's IMHO a shallow bug and worth reporting.
I just tried hash(D.as_tuple()) and it is blazing fast. I think that
unless the official line is "don't touch decimal.py until X", this change
to hashing would be useful and (AFAICT) harmless enough to fit in e.g.
2.5.2. To avoid incompatibilities, __hash__ could check for Overflow and
only use .as_tuple for values higher than the previous maximum (keeping,
unfortunately, __hash__ slow for values below).
Could the current status of Decimal be made a bit more clear? Are bug
reports/patches welcome? Is bugging Facundo or RHettinger welcome? :)
If getting __int__ a bit faster and able to convert sans huge strings is
desired, I've updated that old function (see below) and AFAIK it could be
used to replace Lib/decimal.py/Decimal.[__int__,__long__]. It gets about
ten times faster on best cases and is about as slow on worst cases (could
be much worse if "long(rint_part + rdec_part)/exponent" is a STUPID thing
to do, but seems easy to avoid). As the original __int__ optimizes
str(Decimal._int) and doesn't split/check for substrings, using the same
path should speed this up more. I can run the tests and benchmark it (next
month...) if there's interest.
def dec2long(number):
""" Convert decimal.Decimal to long (abridged, non-checking
version)"""
decimal_string = str(number)
if "e" in decimal_string:
radix, exponent = decimal_string.split("e")
elif "E" in decimal_string:
radix, exponent = decimal_string.split("E")
else:
radix, exponent = (decimal_string, 0)
if exponent:
exponent = int(exponent)
if "." in radix:
rint, rdec = radix.split(".")
radix_decimal_part_len = long(len(rdec))
if radix_decimal_part_len <= exponent:
radix_as_long = long(rint + rdec)
corrected_exponent = exponent - radix_decimal_part_len
result = radix_as_long * 10L** corrected_exponent
else:
result = long(rint + rdec) / exponent
else:
radix_as_long = long(radix)
result = radix_as_long * 10L**exponent
else:
if "." in radix:
radix_integer_part = long(radix.split(".")[0])
else:
radix_integer_part = long(radix)
result = radix_integer_part
return result
As a comparison, here's __int__ (abridged) from decimal.py:
def __int__(number):
"""Converts self to an int, truncating if necessary."""
if number._exp >= 0:
s = ''.join(map(str, number._int)) + '0'*number._exp
else:
s = ''.join(map(str, number._int))[:number._exp]
if s == '':
s = '0'
sign = '-'*self._sign
return int(sign + s)
--
Comment By: Jason G (aryx)
Date: 2007-08-09 12:09
Message:
Logged In: YES
user_id=1289703
Originator: YES
Hey Daniel,
The bigger issue for us is mostly the fact that Decimal.__hash__ us
calling Decimal.__int__ and not because we want an integer/long version of
a very large Decimal. We do not actually cast the decimal into an int/long
explicitly. I wouldn't have any issues if Decimal.__int__ remained as it
is, but I think it would be a good idea for Decimal.__hash__ to do
something differently. Probably something that is fast and simple,
[ python-Bugs-1770416 ] Decimal.__int__ overflows for large values
Bugs item #1770416, was opened at 2007-08-08 20:43
Message generated for change (Comment added) made by gbrandl
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770416&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Jason G (aryx)
>Assigned to: Facundo Batista (facundobatista)
Summary: Decimal.__int__ overflows for large values
Initial Comment:
This also affects Decimal.__hash__, since it [indirectly] calls Decimal.__int__.
>>> from decimal import Decimal as D
>>> e = D("1e1234567890987654321")
>>> int(e)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/decimal.py", line 1501, in __int__
s = ''.join(map(str, self._int)) + '0'*self._exp
OverflowError: cannot fit 'long' into an index-sized integer
>>> e = D("1e1234567890")
>>> int(e)
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/decimal.py", line 1501, in __int__
s = ''.join(map(str, self._int)) + '0'*self._exp
MemoryError
Also, for values that do work this is incredibly slow if they are still fairly
large.
--
>Comment By: Georg Brandl (gbrandl)
Date: 2007-08-09 17:37
Message:
Logged In: YES
user_id=849994
Originator: NO
Assigning to Facundo, he's actively working on decimal ATM.
--
Comment By: ajaksu (ajaksu2)
Date: 2007-08-09 17:21
Message:
Logged In: YES
user_id=1200609
Originator: NO
I see. Inheriting from Decimal and overloading __hash__ is a way to solve
your problem, but it's IMHO a shallow bug and worth reporting.
I just tried hash(D.as_tuple()) and it is blazing fast. I think that
unless the official line is "don't touch decimal.py until X", this change
to hashing would be useful and (AFAICT) harmless enough to fit in e.g.
2.5.2. To avoid incompatibilities, __hash__ could check for Overflow and
only use .as_tuple for values higher than the previous maximum (keeping,
unfortunately, __hash__ slow for values below).
Could the current status of Decimal be made a bit more clear? Are bug
reports/patches welcome? Is bugging Facundo or RHettinger welcome? :)
If getting __int__ a bit faster and able to convert sans huge strings is
desired, I've updated that old function (see below) and AFAIK it could be
used to replace Lib/decimal.py/Decimal.[__int__,__long__]. It gets about
ten times faster on best cases and is about as slow on worst cases (could
be much worse if "long(rint_part + rdec_part)/exponent" is a STUPID thing
to do, but seems easy to avoid). As the original __int__ optimizes
str(Decimal._int) and doesn't split/check for substrings, using the same
path should speed this up more. I can run the tests and benchmark it (next
month...) if there's interest.
def dec2long(number):
""" Convert decimal.Decimal to long (abridged, non-checking
version)"""
decimal_string = str(number)
if "e" in decimal_string:
radix, exponent = decimal_string.split("e")
elif "E" in decimal_string:
radix, exponent = decimal_string.split("E")
else:
radix, exponent = (decimal_string, 0)
if exponent:
exponent = int(exponent)
if "." in radix:
rint, rdec = radix.split(".")
radix_decimal_part_len = long(len(rdec))
if radix_decimal_part_len <= exponent:
radix_as_long = long(rint + rdec)
corrected_exponent = exponent - radix_decimal_part_len
result = radix_as_long * 10L** corrected_exponent
else:
result = long(rint + rdec) / exponent
else:
radix_as_long = long(radix)
result = radix_as_long * 10L**exponent
else:
if "." in radix:
radix_integer_part = long(radix.split(".")[0])
else:
radix_integer_part = long(radix)
result = radix_integer_part
return result
As a comparison, here's __int__ (abridged) from decimal.py:
def __int__(number):
"""Converts self to an int, truncating if necessary."""
if number._exp >= 0:
s = ''.join(map(str, number._int)) + '0'*number._exp
else:
s = ''.join(map(str, number._int))[:number._exp]
if s == '':
s = '0'
sign = '-'*self._sign
return int(sign + s)
--
Comment By: Jason G (aryx)
Date: 2007-08-09 15:09
Message:
Logged In: YES
user_id=1289703
Originator: YES
Hey Daniel,
The bigger issue for us is mostly the fact that Decimal.__hash__ us
calling Decimal.__int__ and not because we want an integer/long version of
a very large Decimal. We
[ python-Bugs-1771260 ] Errors in site.py not reported properly
Bugs item #1771260, was opened at 2007-08-09 15:37 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1771260&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Adam Olsen (rhamphoryncus) Assigned to: Nobody/Anonymous (nobody) Summary: Errors in site.py not reported properly Initial Comment: (Ignore the p3yk dir name. This has been updated to the py3k branch.) [EMAIL PROTECTED]:~/src/python-p3yk/build$ make object : type: TypeError refcount: 4 address : 0x8239f0c lost sys.stderr make: *** [sharedmods] Error 1 The root cause is that (due to some local modifications) site.py failed to load and gave an error. This can be easily duplicated by opening up Lib/site.py:main and putting 1/0 on the first line. However, the ZeroDivisionError that should cause never gets printed. Python/pythonrun.c:initsite attempts to retrieve sys.stderr, which fails because site.py never got a chance to install it (!), then passes the NULL file object pointer to PyFile_WriteString, which turns that into a new exception (replacing the old one). initsite ignores the return value indicating the exception, instead clearing it, and the interpreter continues to load, no one the wiser. Several other exceptions may happen and get squashed, I'm not sure. Eventually, Python/sysmodule.c:sys_excepthook calls Python/pythonrun.c:PyErr_Display, which attempts to retrieve sys.stderr, and failing that calls _PyObject_Dump() on the exception (explaining the weird message). Oddly, there's a *second* if statement, which then prints the "lost sys.stderr" line. Possible remedies: 1. PyErr_Display's dump message is not very clear. 2. initsite should go directly to stderr if it can't retrieve sys.stderr. Alternatively, since site.py is now more important, it could be changed into a fatal error. Yet another option is to explicitly check for sys.stderr even on success and make that alone into a fatal error. 3. The error printing APIs could be modified to internalize the stderr retrieval. Upon failure they could print a brief "stderr unavailable; original exception was ..." message. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1771260&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1771381 ] bsddb can't use unicode keys
Bugs item #1771381, was opened at 2007-08-10 04:32
Message generated for change (Tracker Item Submitted) made by Item Submitter
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1771381&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Unicode
Group: Feature Request
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Erol Aktay (moghe)
Assigned to: M.-A. Lemburg (lemburg)
Summary: bsddb can't use unicode keys
Initial Comment:
bsddb throws a TypeError when I try to use an unicode string as key name;
i.e. bsddb.btopen("foobar", "c")[u'foo'] = "bar" fails
I discovered it while experimenting with the shelve module. You may find more
information in the attached file.
Python version: 2.5.1
OS: Windows XP SP2 (5.1.2600)
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1771381&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1770551 ] words able to decode but unable to encode in GB18030
Bugs item #1770551, was opened at 2007-08-08 18:34
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770551&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Unicode
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Z-flagship (zaex)
>Assigned to: Hye-Shik Chang (perky)
Summary: words able to decode but unable to encode in GB18030
Initial Comment:
Here is a list of chinese characters that can be read from a file [in GB18030
encoding], but unable to encode to GB18030 encoding
detailed:
used codecs.open(r'file name', encoding='GB18030') to read the characters from
a file, and try to encode them word by word into GB18030 with
word.encode('GB18030'). The action caused an exception with 'illegal multibyte
sequence'
the attachment is also the list.
list:
䎬䎱䅟䌷䦟䦷䲠㧏㭎㘚㘎㱮䴔䴖䴗䦆㧟䙡䙌䴕䁖䎬䴙䥽䝼䞍䓖䲡䥇䦂䦅䴓㩳㧐㳠䲢䴘㖞䜣䥺䶮䜩䥺䲟䲣䦛䦶㑳㑇㥮㤘䏝䦃
--
>Comment By: Neal Norwitz (nnorwitz)
Date: 2007-08-09 20:35
Message:
Logged In: YES
user_id=33168
Originator: NO
This seems like a cjk problem. Hye-Shik, could you take a look?
--
Comment By: Z-flagship (zaex)
Date: 2007-08-08 18:37
Message:
Logged In: YES
user_id=1863611
Originator: YES
The Python is Python2.5 , my OS is windows XP professional sp2 version
2002
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770551&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1770009 ] decimal.Decimal("trash") produces informationless exception
Bugs item #1770009, was opened at 2007-08-08 05:44
Message generated for change (Comment added) made by nnorwitz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770009&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: John Machin (sjmachin)
>Assigned to: Facundo Batista (facundobatista)
Summary: decimal.Decimal("trash") produces informationless exception
Initial Comment:
Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import decimal
>>> decimal.Decimal("-$123,456.78")
Traceback (most recent call last):
File "", line 1, in
File "C:\python25\lib\decimal.py", line 614, in __new__
self._sign, self._int, self._exp = context._raise_error(ConversionSyntax)
File "C:\python25\lib\decimal.py", line 2325, in _raise_error
raise error, explanation
decimal.InvalidOperation
It should do something like float does ... better message, and show the
offending arg:
>>> float("-$123,456.78")
Traceback (most recent call last):
File "", line 1, in
ValueError: invalid literal for float(): -$123,456.78
>>>
--
>Comment By: Neal Norwitz (nnorwitz)
Date: 2007-08-09 20:36
Message:
Logged In: YES
user_id=33168
Originator: NO
Facundo, could you take a look?
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1770009&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
