[issue39232] asyncio crashes when tearing down the proactor event loop
Joe added the comment: We are running into this all the time, ever since the Proactor became the default on Windows in 3.8. Usually it comes up when the program terminates due to an unhandled exception during a highly concurrent operation. The resulting cascade of RuntimeErrors often obscures the real reason for failure, which makes debugging more painful than it should be. But sometimes this cascade of RuntimeErrors will occur even when the program otherwise executes successfully. So it can be difficult to know if the program actually blew up or if it's just benign event loop vomit. Converting this particular error to a warning would be great, but eliminating the call to the event loop in __del__ would be even better. I understand that's easier said than done, or else ProactorBasePipeTransport wouldn't be leaning on __del__ in the first place. -- nosy: +lawsonjl.ornl ___ Python tracker <https://bugs.python.org/issue39232> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6767] Python as zip package
New submission from Joe : It would be nice, if you could offer the Windows version also as a zi package, besides the msi installer. -- components: Windows messages: 91890 nosy: Joe severity: normal status: open title: Python as zip package type: feature request ___ Python tracker <http://bugs.python.org/issue6767> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6767] Python as zip package
Joe added the comment: I meant as a zip archive package -- ___ Python tracker <http://bugs.python.org/issue6767> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6767] Python as zip package
Joe added the comment: Because, I don't need/want an installation. I only need the files whith its directory structure. -- ___ Python tracker <http://bugs.python.org/issue6767> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27467] distutils.config API different between <=3.5.1 and 3.5.2
New submission from Joe: In Python 3k releases leading up to 3.5.2, distutils.config imported "ConfigParser", but now imports "RawConfigParser" in the latest release. The documentation indicates "RawConfigParser" is considered legacy and "ConfigParser" should be used in its place. Was this change intentional, and if not, will it be reverted to the previous behavior for the next patch release? Thanks, Joe -- components: Distutils messages: 269970 nosy: dstufft, eric.araujo, jhunkeler priority: normal severity: normal status: open title: distutils.config API different between <=3.5.1 and 3.5.2 type: behavior versions: Python 3.5 ___ Python tracker <http://bugs.python.org/issue27467> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46931] zipfile library will raise uncaught oserror when reading length incorrect zip file
New submission from Vertu Joe : I intentionally made some corrupted zip archive files for testing. If some contents were removed from the archive instead of changing the bits. when trying to read such files, the zipfile will raise an uncaught OSError, instead of a badzipfile error as expected. os is windows 10 x64 not sure if this also happens on the UNIX system or it's intended to be happen. code: import zipfile with zipfile.ZipFile(r'damaged.zip') as dmg: dmg.testzip() result: OSError [Errno 22] Invalid argument File "test.py", line 20, in file = dmg.testzip() -- components: Library (Lib) files: damaged.zip messages: 414590 nosy: ultimalium priority: normal severity: normal status: open title: zipfile library will raise uncaught oserror when reading length incorrect zip file type: behavior versions: Python 3.10 Added file: https://bugs.python.org/file50658/damaged.zip ___ Python tracker <https://bugs.python.org/issue46931> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46947] unicodedata.name gives ValueError for control characters
New submission from Joe Cool :
unicodedata.name gives ValueError for control characters, for example:
>>> unicodedata.name('\x00')
Traceback (most recent call last):
File "", line 1, in
ValueError: no such name
>>> unicodedata.name('\t')
Traceback (most recent call last):
File "", line 1, in
ValueError: no such name
Where unicodedata.lookup clearly knows the names for these characters:
>>> unicodedata.lookup('NULL')
'\x00'
>>> unicodedata.lookup('TAB')
'\t'
--
components: Library (Lib)
messages: 414672
nosy: snoopyjc
priority: normal
severity: normal
status: open
title: unicodedata.name gives ValueError for control characters
type: behavior
versions: Python 3.10
___
Python tracker
<https://bugs.python.org/issue46947>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12378] smtplib.SMTP_SSL leaks socket connections on SSL error
New submission from Joe Shaw :
Start a non-SSL server on port 2525:
$ python -m smtpd -n -c DebuggingServer localhost:2525
In another terminal, fire up a python interpreter and run the following code:
>>> import smtplib
>>> s = smtplib.SMTP_SSL("localhost", 2525)
[...]
ssl.SSLError: [Errno 1] _ssl.c:480: error:140770FC:SSL
routines:SSL23_GET_SERVER_HELLO:unknown protocol
The underlying socket connection is still open, but you can't access it or
close it:
$ lsof -P -p 76318 | grep 2525
Python 76318 joeshaw3u IPv4 0x09a9fb18 0t0 TCP
localhost:64328->localhost:2525 (ESTABLISHED)
This wreaks havoc if you're trying to write a unit test using the smtpd module
and asyncore in a thread and try to clean up after yourself.
The code inside SMTP_SSL looks something like this (on 2.6.5 anyway):
def _get_socket(self, host, port, timeout):
if self.debuglevel > 0: print>>stderr, 'connect:', (host, port)
new_socket = socket.create_connection((host, port), timeout)
new_socket = ssl.wrap_socket(new_socket, self.keyfile,
self.certfile)
self.file = SSLFakeFile(new_socket)
return new_socket
Something like:
new_socket = socket.create_connection((host, port), timeout)
try:
new_socket = ssl.wrap_socket(new_socket, self.keyfile,
self.certfile)
except:
new_socket.close()
raise
self.file = SSLFakeFile(new_socket)
return new_socket
I think will do the trick.
--
components: Library (Lib)
messages: 138753
nosy: joeshaw
priority: normal
severity: normal
status: open
title: smtplib.SMTP_SSL leaks socket connections on SSL error
type: resource usage
versions: Python 2.6
___
Python tracker
<http://bugs.python.org/issue12378>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12378] smtplib.SMTP_SSL leaks socket connections on SSL error
Joe Shaw added the comment: >From some experimentation, closing the underlying socket isn't enough. You >also need to close the SSL socket, so you'd need to do something like: new_socket = socket.create_connection((host, port), timeout) ssl_socket = ssl.wrap_socket(new_socket, self.keyfile, self.certfile, do_handshake_on_connect=False) try: ssl_socket.do_handshake() except: ssl_socket.close() new_socket.close() raise self.file = SSLFakeFile(ssl_socket) return ssl_socket -- ___ Python tracker <http://bugs.python.org/issue12378> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12739] read stuck with multithreading and simultaneous subprocess.Popen
New submission from Joe Hu : When multiple threads create child processes simultaneously and redirect their stdout using subprocess.Popen, at least one thread will stuck on reading the stdout after its child process exited, until all other processes are also exited. The test case reproduces the problem. It's always reproducible on my system (Python 3.1 on Windows 7 x64 / Python 3.2 on Windows 7 x86). Here is my suspicion: When Popen is called by two threads simultaneously, the latter child processes may be started before pipe handles for the former process are closed, causing the handles be incorrectly inherited by the latter process. So these handles can only be closed after all the two processes exit, and only after that, p.stdout.read* can detect EOF and return. -- components: Library (Lib), Windows files: python-subprocess-bug-test-case.py messages: 141939 nosy: SAPikachu priority: normal severity: normal status: open title: read stuck with multithreading and simultaneous subprocess.Popen type: behavior versions: Python 3.1, Python 3.2 Added file: http://bugs.python.org/file22885/python-subprocess-bug-test-case.py ___ Python tracker <http://bugs.python.org/issue12739> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10531] write tilted text in turtle
Joe Metcalfe added the comment: Turtle is built on top of Tk, which is currently at version 8.5 - this has no ability to rotate text. When Tk version 8.6 arrives it should be able to write rotated text (see http://mail.python.org/pipermail/tkinter-discuss/2010-November/002490.html) and turtle.py could be updated to take advantage of it. -- nosy: +Joe.Metcalfe ___ Python tracker <http://bugs.python.org/issue10531> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10531] write tilted text in turtle
Changes by Joe Metcalfe : -- versions: -Python 3.2 ___ Python tracker <http://bugs.python.org/issue10531> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple crashes, does not handle negative TZ offsets, does not handle DST correctly, and outputs localtime (not UTC)
New submission from Joe Peterson : Internaldate2tuple() is broken in several ways. The last two issues have existed in the code for some time. New issues in Python 3: 1. It crashes with "KeyError". This is because the Mon2num dictionary's keys are strings, not bytes objects (note that other strings in imaplib have been updated, but not Mon2num). 2. The sign of the TZ offset (e.g. -0700) is compared to the string '-', not b'-', so the compare is never true, causing a large error when TZ offset is negative. Left over from Python 2.x: 3. DST is not handled correctly. Specifically, when the date is such that its local time form and its UTC form (if both interpreted as local time) are on different sides of a DST changeover, the result will be off by one hour. This is because the check for DST is done after treating the time as local time, even if it is not local time, causing it be tested while sometimes on the wrong side of a DST change. This can be corrected by using calls that keep time in UTC. 4. Related to #3, the result is returned in local time, whereas the documentation states that the result is in UT. The fix for #3 takes care of this one, as well. Here is an example: Run the following two dates, that represent exactly the same time, through Internaldate2tuple: '25 (INTERNALDATE "01-Apr-2000 19:02:23 -0700")' '101 (INTERNALDATE "02-Apr-2000 02:02:23 +")' Once the Mon2num issue is fixed, Python 3 will perform the conversions, but with a 15 hour difference. Python 2 will produce results with a one hour difference. Note that the last two issues (but describing only #4 above) were also addressed in a similar way in an old post I found by Colin Brown in 2004 (http://www.velocityreviews.com/forums/t336162-imaplib-function-bug.html). -- components: Library (Lib) files: imaplib_Internaldate2tuple.patch keywords: patch messages: 126386 nosy: lavajoe priority: normal severity: normal status: open title: imaplib: Internaldate2tuple crashes, does not handle negative TZ offsets, does not handle DST correctly, and outputs localtime (not UTC) versions: Python 3.2 Added file: http://bugs.python.org/file20419/imaplib_Internaldate2tuple.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple crashes, does not handle negative TZ offsets, does not handle DST correctly, and outputs localtime (not UTC)
Changes by Joe Peterson : -- type: -> crash ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly, and outputs localtime (not UTC)
Changes by Joe Peterson : -- title: imaplib: Internaldate2tuple crashes, does not handle negative TZ offsets, does not handle DST correctly, and outputs localtime (not UTC) -> imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly, and outputs localtime (not UTC) ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly
Joe Peterson added the comment: Regarding problem #4, it actually appears that returning local time is the right thing to do, since it matches the behavior of Time2Internaldate(). Returning UTC, as the doc states, would potentially break IMAP append() that can include an internaldate possibly returned from Internaldate2tuple() in typical usage (like when copying messages). In this case, the doc is wrong on Internaldate2tuple(). I have attached a new patch to both the code and doc that replaces the old patch. I now return localtime rather than gmtime, but other than that, the DST fix remains the same and now handles the cases near DST changes correctly. Ultimately, it might be desirable to be able always stay in UTC, so perhaps adding UTC variants of both Internaldate2tuple() and Time2Internaldate() (and a UTC option to append()) would be a good enhancement for later. -- assignee: -> docs@python components: +Documentation nosy: +docs@python title: imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly, and outputs localtime (not UTC) -> imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly Added file: http://bugs.python.org/file20420/imaplib_Internaldate2tuple.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20419/imaplib_Internaldate2tuple.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly
Changes by Joe Peterson : -- components: -Documentation ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly
Changes by Joe Peterson : -- components: +Documentation ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly
Joe Peterson added the comment: > I assume this means it raises a KeyError when given a bytes object as an > argument. Yes, a KeyError is raised when arg is bytes, but passing a string also fails (raising TypeError). The latter might also be a separate bug, in that strings cannot be passed as they could be in Python 2. > This looks like a 2 to 3 port oversight and we can probably fix it in RC. Probably, since many strings have been changed to bytes elsewhere in the file. BTW, I just attached a patch for Python 2.7 that fixes the subset of non-py3k-related issues. -- versions: +Python 2.7 Added file: http://bugs.python.org/file20426/imaplib_Internaldate2tuple_python27.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10929] telnetlib does not send FIN when self.close() issued
New submission from Joe Bennett : Hi, am running Python 2.6.5 on Unbuntu 10.04 and am seeing no FIN when a self.close() is issued... I do see a reset issued, but it looks like some of the M$ servers do not appreciate on a reset and would like a graceful teardown... I understand that an RST can be issued in the event the buffer may have data to send, I do not believe that is the case. If there is a way to verify that, please let me know... -- components: IO messages: 126436 nosy: jammer1 priority: normal severity: normal status: open title: telnetlib does not send FIN when self.close() issued type: behavior versions: Python 2.6 ___ Python tracker <http://bugs.python.org/issue10929> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Joe Peterson added the comment: Added fix for ParseFlags (another string/bytes issue) and now accept strings as args to ParseFlags and Internaldate2tuple. Also added unit tests for changes. -- assignee: -> docs@python components: +Tests title: imaplib: Internaldate2tuple() crashes, does not handle negative TZ offsets, does not handle DST correctly -> imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly Added file: http://bugs.python.org/file20430/imaplib_Internaldate2tuple_python32.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20420/imaplib_Internaldate2tuple.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20430/imaplib_Internaldate2tuple_python32.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Changes by Joe Peterson : Added file: http://bugs.python.org/file20431/imaplib_Internaldate2tuple_python32.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20426/imaplib_Internaldate2tuple_python27.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Changes by Joe Peterson : Added file: http://bugs.python.org/file20432/imaplib_Internaldate2tuple_python27.patch ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Joe Peterson added the comment: >There are at least 3 issues here: a documentation issue, a py3k bug and at >least one feature request. Which is a feature request? In these patches, I am attempting to fix the DST problems and regain the previous behavior in Python 2. Are you talking about the ability to accept a string vs. a bytes object? > 1. Internaldate2tuple is documented to return UTC timetuple, but implemented > to return local time (ignoring the DST issues.) The proposal is to change > the documentation. I prefer UTC, so this is a bit of a shame, I agree, but the use of the pervious interfaces assumed localtime, so changing to UTC would definitely break existing code. I do think it would be nice to extend this to deal with UTC instead, but in this patch, I am only trying to retain current functionality. 2. There are a couple of str vs bytes bugs that obviously need to be fixed. > 3. The proposed patch also make both str and bytes acceptable as > Internaldate2tuple argument. True, but given the new role of str and bytes, it is unclear what existing code would try to pass. > As discussed in issue 9864, it would be best if datetime formatting and > parsing routines would operate on datetime objects. I can see that redoing some of this would be a good idea. But I am only trying to keep the existing stuff from being broken in this patch. I agree that the interfaces could be a lot better, and I would indeed like to see it improved (and I am willing to help with doing that). > There is still a problem, though. The code above would only work as expected > in the C locale, but Time2Internaldate already has this limitation. As long as we assume strings passed are ASCII, it should work. And email headers should be ASCII (although I have seen some really weird deviations from this on old emails). It's not perfect, certainly, and going forward, the IMAP lib could be tightened up. Maybe this first patch could be thought of as a step, at least fixing what is broken until improved. -- ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10934] imaplib: Internaldate2tuple() is documented to return UTC, but it returns local time
New submission from Joe Peterson : Patched documentation for Internaldate2tuple() to correctly state it returns local time. Also, Time2Internaldate() (its inverse) did not state that it needs local time as input, so patched this as well. Patches for 3.2 and 2.7 are attached. -- assignee: docs@python components: Documentation files: imaplib_Internaldate2tuple_doc_python32.patch keywords: patch messages: 126463 nosy: docs@python, lavajoe priority: normal severity: normal status: open title: imaplib: Internaldate2tuple() is documented to return UTC, but it returns local time versions: Python 2.7, Python 3.2 Added file: http://bugs.python.org/file20438/imaplib_Internaldate2tuple_doc_python32.patch ___ Python tracker <http://bugs.python.org/issue10934> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10934] imaplib: Internaldate2tuple() is documented to return UTC, but it returns local time
Changes by Joe Peterson : Added file: http://bugs.python.org/file20439/imaplib_Internaldate2tuple_doc_python27.patch ___ Python tracker <http://bugs.python.org/issue10934> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Joe Peterson added the comment: I have started splitting these up as recommended. First one (documentation) is: issue 10934. I will split out more later today... -- ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10939] imaplib: Internaldate2tuple fails to parse month and does not work with negative TZ offset due to bytes/str issues
New submission from Joe Peterson : There are two issues with conversion to Python 3: 1. It raise "KeyError". This is because the Mon2num dictionary keys are strings (str), not bytes objects (note that many other strings in imaplib have been updated, but not Mon2num). 2. The sign character of the TZ offset (e.g. -0700) is compared to the string (str) '-', not bytes array b'-', so the compare is never true, causing a large error when the TZ offset is negative. Patch attached that also adds a unit test. -- components: Library (Lib) files: imaplib_Internaldate2tuple_bytes_fixes_python32.patch keywords: patch messages: 126499 nosy: lavajoe priority: normal severity: normal status: open title: imaplib: Internaldate2tuple fails to parse month and does not work with negative TZ offset due to bytes/str issues type: behavior versions: Python 3.2 Added file: http://bugs.python.org/file20444/imaplib_Internaldate2tuple_bytes_fixes_python32.patch ___ Python tracker <http://bugs.python.org/issue10939> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10939] imaplib: Internaldate2tuple raises KeyError parsing month and does not work with negative TZ offset due to bytes/str issues
Changes by Joe Peterson : -- title: imaplib: Internaldate2tuple fails to parse month and does not work with negative TZ offset due to bytes/str issues -> imaplib: Internaldate2tuple raises KeyError parsing month and does not work with negative TZ offset due to bytes/str issues ___ Python tracker <http://bugs.python.org/issue10939> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10941] imaplib: Internaldate2tuple produces wrong result if date is near a DST change
New submission from Joe Peterson : DST is not handled correctly. Specifically, when the input date/time, ignoring the TZ offset (and treated as if it is local time) is on the other side of the DST changeover from the actual local time represented, the result will be off by one hour. This can happen, e.g., when the input date/time is actually UTC (offset +). This is because the check for DST is done after converting the time into a local time tuple, thereby treating as real local time. This can be corrected by keeping the time in real UTC (by using calendar.timegm() instead of checking the DST flag) until the final conversion to local time. Here is an example: Run the following two dates, that represent exactly the same time, through Internaldate2tuple: '25 (INTERNALDATE "01-Apr-2000 19:02:23 -0700")' '101 (INTERNALDATE "02-Apr-2000 02:02:23 +")' Note that a variant of this issue (but identifying it as a different problem) was addressed in a similar way in an old post I found by Colin Brown in 2004 (http://www.velocityreviews.com/forums/t336162-imaplib-function-bug.html). Patch also adds unit test to check the above example dates. Python 3 version of patch assumes patch from issue 10939 has been applied. Patch also corrects code python doc that currently states time is UT, not local (see issue 10934). -- components: Library (Lib) messages: 126503 nosy: lavajoe priority: normal severity: normal status: open title: imaplib: Internaldate2tuple produces wrong result if date is near a DST change type: behavior versions: Python 2.7, Python 3.2 ___ Python tracker <http://bugs.python.org/issue10941> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10941] imaplib: Internaldate2tuple produces wrong result if date is near a DST change
Changes by Joe Peterson : -- keywords: +patch Added file: http://bugs.python.org/file20446/imaplib_Internaldate2tuple_dst_fix_python32.patch ___ Python tracker <http://bugs.python.org/issue10941> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10941] imaplib: Internaldate2tuple produces wrong result if date is near a DST change
Changes by Joe Peterson : Added file: http://bugs.python.org/file20447/imaplib_Internaldate2tuple_dst_fix_python27.patch ___ Python tracker <http://bugs.python.org/issue10941> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Joe Peterson added the comment: Two more issues split out into their own issues: issue 10939 (bytes/str issues) issue 10941 (DST handled incorrectly) -- ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10947] imaplib: Internaldate2tuple and ParseFlags require (and latter returns) bytes arrays; should allow/return str
New submission from Joe Peterson : In imaplib, there is currently a mix of bytes array and str use. Time2Internaldate(), e.g., returns (and accepts) str. Internaldate2tuple() and ParseFlags() only accept a bytes object, and the latter returns a tuple of bytes objects. Of course, these were all strings in Python 2. To regain compatibility with Python 2 behavior and make things more consistent, this patch changes the return type of ParseFlags() to a str tuple, and allow it and Internaldate2tuple() to accept either a bytes object or a str. The unit test has been expanded to test these. Note that applying this patch assumes that the the patches from issue 10939 and issue 10941 have been applied. Also, I am not sure it is proper to accept *both* bytes array and str. Perhaps only str should be allowed. Comments? -- components: Library (Lib) files: imaplib_string_compat.patch keywords: patch messages: 126532 nosy: lavajoe priority: normal severity: normal status: open title: imaplib: Internaldate2tuple and ParseFlags require (and latter returns) bytes arrays; should allow/return str type: behavior versions: Python 3.2 Added file: http://bugs.python.org/file20453/imaplib_string_compat.patch ___ Python tracker <http://bugs.python.org/issue10947> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10921] imaplib: Internaldate2tuple() string/bytes issues, does not handle negative TZ offsets, does not handle DST correctly
Joe Peterson added the comment: This issue has been split, as suggested by Alexander Belopolsky, into separate issues: issue 10934 - doc change to correctly reflect return of local time vs. UTC issue 10939 - bytes/str issues issue 10941 - DST handled incorrectly issue 10947 - compatibility using str/bytes Closing this issue (see separate ones above). -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue10921> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10934] imaplib: Internaldate2tuple() is documented to return UTC, but it returns local time
Joe Peterson added the comment: Hey Alexander, Looks great. Just a few small things: * In hunk 2 of the imaplib.rst file patch section, insert word "to" (i.e. "Convert *date_time* to an IMAP4 ``INTERNALDATE`` representation.") * Two lines, down there's only one space at end of sentence: "double-quotes). The" (all other places have 2 spaces). * In first hunk of imaplib.py patch section, there is a period that does not belong after "tuple": "time.struct_time tuple. or None" That's all I saw! Thanks for expanding on the doc! -- ___ Python tracker <http://bugs.python.org/issue10934> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10947] imaplib: Internaldate2tuple and ParseFlags require (and latter returns) bytes arrays; should allow/return str
Joe Peterson added the comment: These are all good comments. And I agree that the naming of the functions is not good (and the CamelCase). Overall, yes, it should be made consistent and the types returned and passed in should be appropriate. I did a little experimenting, using more imaplib functions in Python3, and I found the following... In the case of Internaldate2tuple(), I use it as follows (and I suspect most would do the same): typ, data = src_box.fetch(src_msg, '(INTERNALDATE)') ...check typ here for error... src_internal_date = data[0] src_date_tuple = myInternaldate2tuple(src_internal_date) So data[0] is a bytes array, returned by the fetch() method, and it is indeed therefore compatible with Internaldate2tuple(). In fact, it seems that the data, in general, is returned as bytes arrays in imaplib. Given that we are dealing with raw email headers and bodies, this is perhaps exactly what should be done. There is some grey area here. For example, for headers, RFC 2047 discusses using special 'encoded-words' that are regular ASCII in, e.g., subjects (but I've encountered a few cases of UTF-8 in headers, even though this is a dubious practice). IMAP flags are ASCII as well. On the other hand, what you get from fetching pieces of emails are thing that you usually want to deal with as strings (e.g., you will want to do find() and startswith(), etc. on them). But I guess it depends on what is really proper to use for email data. -- ___ Python tracker <http://bugs.python.org/issue10947> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10947] imaplib: Internaldate2tuple and ParseFlags require (and latter returns) bytes arrays; should allow/return str
Joe Peterson added the comment: Yep, I agree, and in light of this, we should probably just close this issue and work toward reviewing/improving imaplib in the ways you are suggesting. As I migrate my imap stuff more to Python3, I'll see if I run into any problems with using bytes throughout (beyond what is addressed in issue 10939). -- ___ Python tracker <http://bugs.python.org/issue10947> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Joe Peterson added the comment: Sebastian, Yes, in fact Alexander Belopolsky (belopolsky) brought up the the locale issue for this very function in one of the other issue comments. The invert function, Internaldate2tuple(), actually does its own parsing using a regex match (and so does not have the problem), but you are right, Time2Internaldate() has this issue. -- versions: +3rd party -Python 2.6 ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Joe Peterson added the comment: Yes, that's serious, certainly. A patch should be fairly straightforward, given that part of the formatting logic is already there (for the TZ offset at the end). You just need to format the 6 values, and do a lookup for the month name. If you want to try to work up one, I can take a look, or maybe, if I have some time today, I'll try to do one quickly... -- ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Joe Peterson added the comment: OK, I attached a patch that should work. Note that this patch works for Python 2 and Python 3. As an aside, the str type is still returned as before (even in Python 3), and the _month_names list uses str. As has been discussed, it may be more proper to return a bytes array and be consistent throughout imaplib, but this is not addressed here. Also, I return a leading zero on the day instead of a leading space, since this appears to be what is returned by two IMAP servers I have just tested (gmail's and dovecot). -- keywords: +patch Added file: http://bugs.python.org/file20556/imaplib_Time2Internaldate_locale_fix.patch ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20556/imaplib_Time2Internaldate_locale_fix.patch ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Changes by Joe Peterson : Added file: http://bugs.python.org/file20557/imaplib_Time2Internaldate_locale_fix.patch ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10947] imaplib: Internaldate2tuple and ParseFlags require (and latter returns) bytes arrays; should allow/return str
Changes by Joe Peterson : -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue10947> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Joe Peterson added the comment: > Also, isn't day supposed to be space- rather than 0- padded? This is not clear to me. RFC2822 (referenced from RFC3501 for internal date) discusses date formats, but as used in the header. In this case, day is specified as "([FWS] 1*2DIGIT)", which implies optional space and 1 or 2 digit day. I am not sure this disallows leading-zero format. But this date spec also says dates should be space-separated (like "12 Jan 2011"), and clearly INTERNALDATE needs "-" (like "12-Jan-2011"). Therefore, I cannot see this date format as being authoritative fro INTERNALDATE. Also, RFC3501, in chage #71, is extra confusing in that it puts the 3-letter month in all-caps. Python's Internaldate2tuple(), e.g., cannot handle this currently (nor can it handle a single-digit day with no space or 0, but its regex does handle a leading zero, which led me to think 0 is OK). Also, it seems that gmail's imap server and Dovecot imap server return leading zero, not leading space, when you fetch INTERNALDATE. So I concluded from all this that 0 might actually be preferred. If this is true, leading zero is better also in that it is less error-prone (e.g., strip can remove the leading space, which will cause problems). I'll keep looking for definitive info, but if you know of some I missed, please let me know. -- ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Joe Peterson added the comment: Our messages crossed... :) Hm, I see that in RFC 3501, as well (which obsoletes 2060). But... I wonder: does "(SP DIGIT) / 2DIGIT" mean that " 1" and "01" are both OK? It seems ambiguous to me. I still don't see why major IMAP servers are returning leading zeros if now allowed... -- ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20557/imaplib_Time2Internaldate_locale_fix.patch ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Joe Peterson added the comment: Here's a new patch. I would still like to discuss the leading space vs. leading zero issue, but I have reverted to using a leading space in this patch - fewer changes that way. The long line is also fixed (sorry about that - yes, long lines are ugly). And I have used your suggested Mon2num dict creation. Note that I do an encode() in there for compatibility with Python 3, which throws an exception if the keys are not bytes arrays (consistent with the fix in issue 10939). -- Added file: http://bugs.python.org/file20587/imaplib_Time2Internaldate_locale_fix.patch ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10939] imaplib: Internaldate2tuple raises KeyError parsing month and does not work with negative TZ offset due to bytes/str issues
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20444/imaplib_Internaldate2tuple_bytes_fixes_python32.patch ___ Python tracker <http://bugs.python.org/issue10939> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20587/imaplib_Time2Internaldate_locale_fix.patch ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10939] imaplib: Internaldate2tuple raises KeyError parsing month and does not work with negative TZ offset due to bytes/str issues
Changes by Joe Peterson : Added file: http://bugs.python.org/file20588/imaplib_Internaldate2tuple_bytes_fixes_python32.patch ___ Python tracker <http://bugs.python.org/issue10939> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Changes by Joe Peterson : Added file: http://bugs.python.org/file20589/imaplib_Time2Internaldate_locale_fix.patch ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20589/imaplib_Time2Internaldate_locale_fix.patch ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11024] imaplib: Time2Internaldate() returns localized strings
Joe Peterson added the comment: Not cryptic at all - looks great! New patch attached with associated tweaks. -- Added file: http://bugs.python.org/file20591/imaplib_Time2Internaldate_locale_fix.patch ___ Python tracker <http://bugs.python.org/issue11024> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10939] imaplib: Internaldate2tuple raises KeyError parsing month and does not work with negative TZ offset due to bytes/str issues
Joe Peterson added the comment: Good catch on the test. Note that for issue 10941, we'll want a new non-timezone-dependent test case that can catch the DST-related problem. I'll post a new patch to issue 10941 now and describe this some more there. -- ___ Python tracker <http://bugs.python.org/issue10939> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10939] imaplib: Internaldate2tuple raises KeyError parsing month and does not work with negative TZ offset due to bytes/str issues
Joe Peterson added the comment: Alexander, looks like you hit a weirdness in the Europe/London timezone for dates before 31-Oct-1971, as if DST was in effect even in winter. And the fail of the test is caused by the same bug that causes issue 10941: the use of mktime to interpret the values from an internal time string (which should not be dealing with timezones, but rather explicit offsets from UTC, before converting to localtime). -- ___ Python tracker <http://bugs.python.org/issue10939> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10941] imaplib: Internaldate2tuple produces wrong result if date is near a DST change
Joe Peterson added the comment: Here is a new patch that adds a test to the tests committed for issue 10939. This new test case is needed to trigger the DST issue. This test is not timezone-dependent (in the sense that one does not have to have a specific timezone set for it to work), but it does depend on the DST change happening at 02:00 on the first Sunday in April, 2000. This is the case for most of the United States, but to make it work everywhere, I have added code that sets the TZ to one that works, and then reverts the temporary TZ change after the test. The bug is triggered when the time described in the internal date string, ignoring the time offset, is between 02:00 and 03:00. This is because the raw time in the string is interpreted as local time, and local times in this range are basically invalid, since time advances to 03:00 when 02:00 is reached because of the DST change. This causes the final result to be off by one hour. [This patch is only for Python 3] -- Added file: http://bugs.python.org/file20613/issue10941.diff ___ Python tracker <http://bugs.python.org/issue10941> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10939] imaplib: Internaldate2tuple raises KeyError parsing month and does not work with negative TZ offset due to bytes/str issues
Joe Peterson added the comment: More info on the Europe/London "near the epoch" thing (there is no weirdness in the tz stuff - it's just issue 10941 causing the test to fail)... I looked at the sources for the tz data, and here is the definition for Europe/London: # Zone NAMEGMTOFF RULES FORMAT [UNTIL] ZoneEurope/London -0:01:15 - LMT 1847 Dec 1 0:00s 0:00 GB-Eire %s 1968 Oct 27 1:00 - BST 1971 Oct 31 2:00u 0:00 GB-Eire %s 1996 0:00 EU GMT/BST I appears that London was always 1 hour ahead (BST time) from 1968 Oct 27 until 1971 Oct 31 2:00u, thus triggering issue 10941: Internaldate2tuple() actually returns the wrong local time (00:00 rather than [the correct] 01:00) in its tuple for the epoch, so mktime(), doing the right thing, returns -3600. The patch in issue 10941 fixes this, so the right local time in London (01:00) is returned for the epoch (internal date string "01-Jan-1970 00:00:00 +"). Note that this also exposes another problem with Time2Internaldate(), since it uses time.timezone/time.altzone, which are only valid for the current rules, not old rules as in the London case near the epoch. -- ___ Python tracker <http://bugs.python.org/issue10939> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10941] imaplib: Internaldate2tuple produces wrong result if date is near a DST change
Changes by Joe Peterson : Removed file: http://bugs.python.org/file20613/issue10941.diff ___ Python tracker <http://bugs.python.org/issue10941> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10941] imaplib: Internaldate2tuple produces wrong result if date is near a DST change
Joe Peterson added the comment: I like the idea of adding the decorator. New patch added. -- Added file: http://bugs.python.org/file20615/issue10941.diff ___ Python tracker <http://bugs.python.org/issue10941> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue10941] imaplib: Internaldate2tuple produces wrong result if date is near a DST change
Joe Peterson added the comment: [just carrying over some info from issue 10939 that is related to this issue] Here is another manifestation of this issue, related to the local time assumption, but not to DST, per se: Here is the definition for Europe/London in the unix tz data: # Zone NAMEGMTOFF RULES FORMAT [UNTIL] ZoneEurope/London -0:01:15 - LMT 1847 Dec 1 0:00s 0:00 GB-Eire %s 1968 Oct 27 1:00 - BST 1971 Oct 31 2:00u 0:00 GB-Eire %s 1996 0:00 EU GMT/BST So London's local time was always 1 hour ahead of UTC (BST time) from 1968 Oct 27 until 1971 Oct 31 2:00u. Because of this, Internaldate2tuple() returns the wrong local time (00:00 rather than [the correct] 01:00) in its tuple when input is "01-Jan-1970 00:00:00 +" (the epoch). -- ___ Python tracker <http://bugs.python.org/issue10941> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue7162] 2to3 does not convert __builtins__.file
Joe Amenta added the comment: Yes, the idea was that it doesn't seem outlandish for someone to do: def file(something): do_stuff() You can use lib2to3.fixer_util.is_probably_builtin for this... modified the patch and attached. -- Added file: http://bugs.python.org/file16490/issue7162-3.patch ___ Python tracker <http://bugs.python.org/issue7162> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46947] unicodedata.name gives ValueError for control characters
Joe Cool added the comment: Note: This is an issue for all chars in the ordinal range 0 thru 31. -- ___ Python tracker <https://bugs.python.org/issue46947> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue46947] unicodedata.name gives ValueError for control characters
Joe Cool added the comment: My recommendation would be to add a keyword parameter, defaulting to False, to name(), something like give_full_alias, or maybe errors=“give_full_alias” like the IO functions. In the meantime, as the author of perllib, I had to make my own dict to return to the user the same thing perl does, which is the full alias for these. -- ___ Python tracker <https://bugs.python.org/issue46947> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47198] os.stat on windows doesn't take an open file even though os.stat in os.supports_fd
New submission from Joe Cool :
os.stat on windows doesn't take an open file even though os.stat in
os.supports_fd
>>> fd = open('tmp.tmp', 'w')
>>> fd
<_io.TextIOWrapper name='tmp.tmp' mode='w' encoding='cp1252'>
>>> os.stat(fd)
Traceback (most recent call last):
File "", line 1, in
TypeError: stat: path should be string, bytes, os.PathLike or integer, not
TextIOWrapper
>>> os.stat in os.supports_fd
True
--
messages: 416535
nosy: snoopyjc
priority: normal
severity: normal
status: open
title: os.stat on windows doesn't take an open file even though os.stat in
os.supports_fd
type: behavior
versions: Python 3.10
___
Python tracker
<https://bugs.python.org/issue47198>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44114] Incorrect function signatures in dictobject.c
New submission from Joe Marshall : There's a couple of wrong function signatures in dictobject.c, dictkeys_reversed and dictitems_reversed are defined as single arg functions like so: PyObject *(PyObject *), and are then passed around and called as PyCFunctions, which should be PyObject *(PyObject *self,PyObject *args). dictvalues_reversed is correct. This works fine on most C compilers as the extra arg is just ignored, but on things that use strict function pointer type checking (e.g. webassembly), it crashes (and hence any of the tests that happen to use these functions fails, which is a surprising number) I've got a fix in my pyodide repo, which I'll chuck in as a pull request on github. -- components: Interpreter Core messages: 393506 nosy: joemarshall priority: normal severity: normal status: open title: Incorrect function signatures in dictobject.c type: behavior ___ Python tracker <https://bugs.python.org/issue44114> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue44114] Incorrect function signatures in dictobject.c
Change by Joe Marshall : -- keywords: +patch pull_requests: +24703 stage: -> patch review pull_request: https://github.com/python/cpython/pull/26062 ___ Python tracker <https://bugs.python.org/issue44114> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43656] TracebackException or StackSummary.extract with capture_locals=True fail to catch exceptions raised by repr() on value of frame local variable in FrameSummary.__init__.
Joe Wells added the comment:
I would like to request that this bug be repopened and fixed.
I've changed (or at least tried to change, I'm not sure if it will let me) the
title of the bug to point out that the failure happens in
FrameSummary.__init__. It does not happen in StackSummary.format.
This problem makes the capture_locals=True feature of TracebackException and
StackSummary and the "locals" parameter of FrameSummary unreliable. If any one
of the local variables in any frame on the stack is in an inconsistent state
such that repr will raise an exception, the processing of the traceback fails.
This kind of inconsistent state is of course likely to happen during debugging,
which is precisely when you would want the capture_locals feature to actually
work so you can see what is going wrong.
Just one example of an exception traceback being created with an unsafe local
variable value in one of the stack frames is in the following line:
from _pydecimal import Decimal as D; D(None)
The _pydecimal.Decimal.__new__ method raises an exception when it sees a value
it doesn't know how to convert to Decimal. When it does this, the new object
it was creating is left in an inconsistent state missing the _sign attribute.
When you try to inspect the resulting exception traceback with
traceback.TracebackException(..., capture_locals=True), this raises an
exception.
While I was tracking this bug down, I discovered that the
traceback.TracebackException.__init__ method has the same problem: it
initializes the _str attribute used by the __str__ method quite late and when
it raised an exception before this point, the incompletely initialized
TracebackException object caused repr to fail. There's at least one more class
in traceback.py that also has this problem, but I don't remember which one it
is.
The cascade of exceptions causing exceptions causing exceptions makes the
capture_locals feature difficult to use and debug.
Here is a short snippet that reproduces the bug on Python 3.9.7:
import traceback
class TriggerTracebackBug:
def __init__(self):
raise RuntimeError("can't build a TriggerTracebackBug object for some
reason")
self._repr = 'if we reached this line, this object would have a repr
result'
def __repr__(self):
return self._repr
try:
TriggerTracebackBug()
except Exception as e:
# this method call fails because there is a stack frame with a local
# variable (self) such that repr fails on that variable's value:
traceback.TracebackException.from_exception(e, capture_locals=True)
It's clear that it is too much to expect every class to implement a safe
__repr__ method. So it also seems clear to me that
traceback.FrameSummary.__init__ is the place to fix it.
My suggested fix is to replace this line in the latest Lib/traceback.py:
self.locals = {k: repr(v) for k, v in locals.items()} if locals else
None
with something like this code (written in a somewhat awkward way because that
helped while I was debugging it):
if locals:
d = {}
self.locals = d
for k, v in locals.items():
try:
d[k] = repr(v)
except Exception:
d[k] = ''''''
else:
self.locals = None
I've tested this code in an older version of Python and it fixed the problem
for me.
--
nosy: +jbw
title: StackSummary.format fails if repr(value) fails -> TracebackException or
StackSummary.extract with capture_locals=True fail to catch exceptions raised
by repr() on value of frame local variable in FrameSummary.__init__.
___
Python tracker
<https://bugs.python.org/issue43656>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43656] TracebackException or StackSummary.extract with capture_locals=True fail to catch exceptions raised by repr() on value of frame local variable in FrameSummary.__init__.
Joe Wells added the comment:
Here are my thoughts inspired by Andrei's insightful comments.
1. In response to the major issue Andrei raises, I agree that it is desirable
that repr would never raise an exception. The reasons Andrei mentions seem
quite correct to me. However, I think the only practical way to make that
change would be the change the code of the repr builtin. (Expecting the
authors of every class in all Python code ever written to make sure that their
__repr__ method will never raise an exception is unrealistic.)
The bug that is the topic of the issue in this bug report can be fixed by
merely handling exceptions raised by one particular call to repr in the code of
FrameSummary.__init__. That change can only affect code that uses the
traceback module to get nicer tracebacks, and only if the capture_locals=True
feature is requested
Changing the implementation of the repr builtin could conceivably cause
unexpected problems for lots of deployed 3rd party code during normal use,
because repr is widely used in deployed code, and hence more care and thought
is needed. In contrast, changing FrameSummary.__init__ as I propose in this
bug report will only affect code using the capture_locals=True feature of the
traceback module, and is likely to make such code more reliable because right
now that feature is failure-prone due to this bug.
So I propose that making repr never raise exceptions should be a different bug.
This bug does not need to wait for that bug to be fixed.
2. In response to a minor point of Andrei, I would like to mention that I
encountered this because I had given students a coursework template containing
this code:
import traceback, sys
def format_exception_chunks(exception):
return (traceback.TracebackException.from_exception(exception,
capture_locals=True).format())
def print_exception(_ignored_type, exception, _ignored_traceback):
for chunk in format_exception_chunks(exception):
print(chunk, file=sys.stderr, end="")
# This change to sys.excepthook adds local variables to stack traces.
sys.excepthook = print_exception
This had the unfortunate effect that when a class constructor decided that it
did not like its arguments, the students were overwhelmed by a baffling cascade
of exception tracebacks. So while this was indeed “for convenience of
interactive debugging”, it had the actual effect of making it nearly impossible
for these beginner Python programmers to do any debugging at all. The overall
effect of this bug is that it makes it potentially unwise to routinely rely on
the capture_locals=True feature. A feature that could be useful for beginners
actually makes it worse for them.
3. In response to Andrei's suggested change to my minimal example to reproduce
the bug, I have a few comments. First, his minimal example is just as good as
mine for reproducing the bug. Second, my example is inspired by the two
classes I mention whose constructors trigger the bug: both of them do it by
leaving the incompletely initialized object missing an attribute that repr
depends on, so they both raise a missing attribute exception when an attempt is
made to print the incompletely initialized object. I suspect this is a quite
common pattern in deployed Python code. I suspect that people have not made
efforts to avoid this pattern because it only triggers a bug when exception
tracebacks are investigated, because the only reference to the incompletely
initialized object is in the stack frame of the constructor method (either
__new__ or __init__) which in turn is only referenced from the traceback.
Third, my example deliberately has as little code as possible in the __repr__
method to convey the key issue. Fourth, one of these two examples (mine or
Andrei's) should go into the unit tests when the bug is fixed.
4. I notice that the code of unittest.util.safe_repr gives an example of how to
use object.__repr__(obj) to get something formatted for an object whose normal
__repr__ method has raised an exception. So I alter my recommended fix to the
following:
if locals:
d = {}
self.locals = d
for k, v in locals.items():
try:
d[k] = repr(v)
except Exception:
d[k] = ''
else:
self.locals = None
5. Can someone please change the Stage of this issue to something other than
“resolved” and the Resolution of this issue to something other than “not a bug”?
I hope these comments are helpful and this bug gets somehow fixed.
--
___
Python tracker
<https://bugs.python.org/issue43656>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43656] TracebackException or StackSummary.extract with capture_locals=True fail to catch exceptions raised by repr() on value of frame local variable in FrameSummary.__init__.
Joe Wells added the comment: I'm sorry Andrei: I misread your alteration of my example and misunderstood its purpose. For anyone else reading these messages: In my most recent comment above, please ignore the part of my comment about Andrei's example. So yes, Andrei, that is how people should write their code! Your code does not trigger the bug because it is written in a better way. Agreed completely. However, this bug still needs fixed because it is currently not possible to use the capture_locals=True feature of the traceback module when debugging code written by people who do not follow Andrei's example of best practice. -- ___ Python tracker <https://bugs.python.org/issue43656> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43656] TracebackException or StackSummary.extract with capture_locals=True fail to catch exceptions raised by repr() on value of frame local variable in FrameSummary.__init__.
Joe Wells added the comment:
Andrei, thanks very much for the pointer to bug/issue
https://bugs.python.org/issue39228. I had not noticed the earlier comment by
Irit pointing to that bug. (Is there some way to merge bugs so that someone
visiting one of the bugs will see the merged stream of comments?)
The remarks in that bug are precisely why my recommended fix has this line:
d[k] = ''
instead of this:
d[k] = object.__repr__(v)
Does this not meet the concern expressed there? Something that would more
thoroughly meet that would be the following:
if locals:
d = {}
self.locals = d
for k, v in locals.items():
try:
d[k] = repr(v)
except Exception as e:
d[k] = ''
else:
self.locals = None
I use object.__repr__ instead of repr because when repr(v) has already failed
it is likely that repr(e) on the exception e would also be likely to fail.
Although we could also try repr(e) first to see if it raises an exception.
Your suggested reaction to this bug (documenting Python best practice) is
something that should be done regardless. I completely agree. Unfortunately,
better documentation of how to write good Python does not address the point of
this bug which is that debugging tools should not fail when used to debug buggy
code. The purpose of a debugging tool is to help with badly written code, not
to work only with well written code. Right now the capture_locals=True feature
is not safe to use without wrapping it with an exception handler. As a result
of this bug, I have been forced to rewrite this:
def format_exception_chunks(exception):
return (traceback.TracebackException.from_exception(exception,
capture_locals=True).format())
to instead be this:
def format_exception_chunks(exception):
try:
tbe = (traceback.TracebackException
. from_exception(exception, capture_locals=True))
return tbe.format()
except Exception:
pass
# the traceback module fails to catch exceptions that occur when
# formatting local variables' values, so we retry without
# requesting the local variables.
try:
tbe = traceback.TracebackException.from_exception(exception)
return ('WARNING: Exceptions were raised while trying to'
' format exception traceback with local variable'
' values,\n'
'so the exception traceback was formatted without'
' local variable values.\n',
*tbe.format())
except Exception:
return ('WARNING: Exceptions were raised while trying to format'
' exception traceback,\n'
'so no formatted traceback information is being'
' provided.\n',)
My argument is that it is better to fix the bug once in traceback.py instead of
every user of the capture_locals=True feature having to discover the hard way
(with hours of painful bug investigation) that they need to write lots of
exception handling and bug workarounds around their use of the
capture_locals=True feature.
--
___
Python tracker
<https://bugs.python.org/issue43656>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43656] TracebackException or StackSummary.extract with capture_locals=True fail to catch exceptions raised by repr() on value of frame local variable in FrameSummary.__init__.
Joe Wells added the comment: In the hopes of convincing someone to install a fix to this bug, I will mention a few additional points. When I mention “the capture_locals feature”, I mean calls of the form traceback.TracebackException(..., capture_locals=True) and traceback.StackSummary.extract(..., capture_locals=True). 1. Right now, the capture_locals feature is unreliable. If any frame on the entire stack has a single local variable for which repr raises an exception, the user gets no information whatsoever back for the entire stack except for that exception, and that exception does not identify the offending stack frame or local variable. Also, every use of the capture_locals feature must be inside a try-except statement. 2. The exceptions raised by the capture_locals feature will often be junk exceptions carrying no useful information. The meaning of these exceptions will often be “there is an incompletely initialized object in a variable somewhere on the stack”. Because this is a fairly normal state of affairs, this will often not be useful information. 3. Although it is a excellent idea to encourage Python coders to ensure that __repr__ method never raise exceptions, it seems unlikely this will cause many __repr__ methods to be fixed in the near future. My impression is that __repr__ methods that raise exceptions on incompletely initialized objects are common. One reason for this might be that authors of __repr__ methods rarely suffer from this problem personally, because they will generally supply correct arguments to their own class constructors, and even when they don't (e.g., while building unit tests), they are unlikely to try to format to strings all the local variable values on the stack in the exception traceback. 4. Right now, the caller of traceback.FrameSummary(..., locals=x) needs to go through the entire dict x and for every value v in x test whether repr(v) raises an exception and if so either remove the key/value pair or change the value to something which can be safely repr-ed. Then FrameSummary.__init__ will repeat this work and run repr on every value in x again. So using FrameSummary safely requires duplicating the work, i.e., running repr on every value in the dict both before and during the call to FrameSummary. 5. Anyone who is using the capture_locals feature on an exception traceback has already caught an exception, and therefore decided not to let that exception “bubble up”. Their attempts to log or otherwise cope with the exception will be spoiled by the capture_locals feature raising an unrelated exception. This is even worse when the exception raised by the capture_locals feature is a junk exception that merely indicates there is an incompletely initialized object on the stack. This is likely to happen because exceptions will often happen during the initialization of an object. 6. The most recent suggested fix does in fact report the extra repr exception discovered by the capture_locals feature in the string that represents the local variable's value. The main effect of the current code propagating this repr exception is to make it harder to deal with the original exception. I hope these points are useful. -- ___ Python tracker <https://bugs.python.org/issue43656> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43656] TracebackException or StackSummary.extract with capture_locals=True fail to catch exceptions raised by repr() on value of frame local variable in FrameSummary.__init__.
Joe Wells added the comment: 1. As background, it is worth remembering that a major motivation for why FrameSummary.__init__ stringifies the local variable values in its parameter locals is to prevent the resulting data structure from keeping those values live. This enables them to be garbage collected. 2. It has been suggested that TracebackException.__init__ or StackSummary.extract could be given a function parameter. This could either be a new parameter or could involve using the existing capture_locals parameter with a non-bool. The purpose of this suggestion is to allow customizing the use of repr to stringify local variable values. If this suggestion is followed, it would be quite useful if this function parameter was given not just the local variable values, but also the name of the local variable and maybe also the stack frame it is in. This would enable filtering out undesired variables. For example, I would usually prefer to filter most of the variables from the __main__ module frame, because they are just noise that makes it hard to see the important details. Some ability to filter would also help with the security issue that is discussed in issue 23597 that Irit pointed to. 3. I doubt that new students will be setting up calls to TracebackException or modifying sys.excepthook on their own. Although a simple interface is clearly good, it might be more important to have an interface that maximizes the usefulness of the feature. 4. I no longer have the tracebacks my students were getting. You can look at the traceback from the example in msg 404319 for a traceback. While I was debugging this, I got much more complicated tracebacks because two of the classes in traceback.py also throw exceptions during their __init__ method that leave the not-yet-initialized object in a repr-will-raise-an-exception state. Despite having decades of programming experience, it actually took me a long time before I realized that the exceptions I was debugging were junk exceptions due to attempts to call repr on not-yet-initialized objects. I think figuring this out would be extremely challenging for my second-year undergraduate students. 5. If one of the calls to repr in FrameSummary.__init__ raises an exception, this prevents all the other local variable values from being inspected. If it is desired to report local variable values that cause repr to raise an exception, then it would be good to collect all such exceptions, which requires handling each exception and then continuing to traverse the traceback stack. Because of point 1 above, it might often be best to turn each such exception into a string. To avoid infinite loops in the debugging/logging tools, it would often be best not to attempt to traverse the traceback stack of each of these extra exceptions. If this argument is a good argument, this seems to mean that my most recent proposed fix is a good balance. 6. It is not clear if there is a reliable way to detect whether repr raises an exception due to an object's initialization having been interrupted, but all of the failures that I observed of this sort were for the variable name “self”. In one case, the repr failure was for a normal local variable of an __new__ method which was not a parameter of this method; the variable was named “self” by convention, but this convention would be difficult to automatically verify. In the two other cases, the repr failure was for a parameter named “self” which was the first parameter of an __init__ method. So it would help to give special treatment to the first parameter of __init__ methods, but this would not solve the problem for __new__ methods. 7. In some cases, it might be a bit complicated to ensure the object is always in a safe state for repr-ing during initialization. This is because there may be many attributes that need to be modified and this is not generally done atomically. One attribute could be designated to indicate that initialization is done, so that repr will be extra careful if that attribute does not have an expected value. Given that this is only (so far) an issue for debuggers and traceback inspection tools, it is not clear how to motivate everyone who writes a __new__, __init__, or __repr__ method to do this safely. Documentation can of course help. -- ___ Python tracker <https://bugs.python.org/issue43656> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43656] TracebackException or StackSummary.extract with capture_locals=True fail to catch exceptions raised by repr() on value of frame local variable in FrameSummary.__init__.
Joe Wells added the comment: Some quick comments on Martin's pull request. 1. The changes are sufficient to let the user make things work the way it is requested that they work and anyone who does not start using the new format_locals parameter will get the old behavior. 2. A side comment: I just noticed that the docstring for FrameSummary.__init__ does not document the filename, lineno, and name parameters. Perhaps this could be fixed at the same time? 3. The new parameter format_locals is slightly confusingly named, because it does not format a single string from all the locals but instead gives a dictionary of strings with one string for each local. 4. I personally think it would be better to include something like the example value for format_locals as an extra attribute of the traceback module so everybody doesn't have to write the same function. That said, the documented example is sufficient. 5. It is not clear to me why this stringify-ing of the locals can't be done in StackSummary._extract_from_extended_frame_gen. It passes the dictionary of locals and also the function to transform it to FrameSummary. It would make more sense to transform it first. I suppose there might be some user somewhere who is directly calling FrameSummary so the interface needs to stay. 6. I fantasize that the new format_locals parameter would have access to the frame object. In order for this to happen, the frame object would have to be passed to FrameSummary instead of the 3 values (locals, name, filename) that are extracted from it. I think the current interface of FrameSummary was designed to interoperate with very old versions of Python. Oh well. 7. If anyone wanted to ever refactor FrameSummary (e.g., to enable my fantasy in point #6 just above), it becomes harder after this. This change has the effect of exposing details of the interface of FrameSummary to users of StackSummary.extract and TracebackException. The four parameters that the new parameter format_locals takes are most of the parameters of FrameSummary. -- ___ Python tracker <https://bugs.python.org/issue43656> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40467] subprocess: replacement shell on windows with executable="..." arg
Joe Cool added the comment:
Proposed solution:
if comspec.endswith('sh.exe') or comspec.endswith('sh'):#
issue 40467
args = '{} -c "{}"'.format (comspec, args) #
issue 40467
else: #
issue 40467
args = '{} /c "{}"'.format (comspec, args)
--
nosy: +snoopyjc
___
Python tracker
<https://bugs.python.org/issue40467>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39196] json fails to encode dictionary view types
New submission from Joe Gordon :
Python 3 fails to encode dictionary view objects. Assuming this is an expected
behavior, what is the thinking behind it? I was unable to find any
documentation around this.
> import json; json.dumps({}.values())
"TypeError: Object of type dict_values is not JSON serializable"
--
components: Library (Lib)
messages: 359212
nosy: jogo
priority: normal
severity: normal
status: open
title: json fails to encode dictionary view types
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue39196>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39661] TimedRotatingFileHandler doesn’t handle DST switch with daily rollover
New submission from Joe Cool : TimedRotatingFileHandler doesn’t handle the switch to/from DST when using daily/midnight rotation. It does not adjust the rollover time so the rollover will be off by an hour. Parameters: when=‘midnight’, utc=False -- components: Library (Lib) messages: 362140 nosy: snoopyjc priority: normal severity: normal status: open title: TimedRotatingFileHandler doesn’t handle DST switch with daily rollover type: behavior versions: Python 3.7 ___ Python tracker <https://bugs.python.org/issue39661> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39661] TimedRotatingFileHandler doesn’t handle DST switch with daily rollover
Joe Cool added the comment: Never mind. I was looking for the DST code in computeRollover, and I found it in doRollover. -- stage: -> resolved status: open -> closed ___ Python tracker <https://bugs.python.org/issue39661> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14354] Crash in _ctypes_alloc_callback
New submission from Joe Rumsey : I have reproduced this crash in Apple's default 2.7.1 python, and in 2.7.3 built from source myself. But only in release mode. If I rebuild 2.7.3 in debug, the crash goes away. The attached file reproduces the issue, which has to do with a union containing multiple structs being used as a value for a callback argument. I've stripped it down as much as I can. Removing any fields from either the union or the Dice struct will cause this to no longer crash. The original source of this is libtcod's python wrapper library, which seems to work fine on platforms other than Mac, though it had a lot of other problems on 64-bit systems before I got to this one. This issue may also be more 64-bit specific than Mac specific. This is the callstack from Apple's python: #0 0x0001010c7712 in _ctypes_alloc_callback () #1 0x0001010c4a1c in PyCData_AtAddress () #2 0x000100050afa in icu::DigitList::getDouble () #3 0x0001bd32 in PyObject_Call () #4 0x00010008bf63 in ubrk_swap () #5 0x00010008ecd8 in triedict_swap () #6 0x00010008ed4d in triedict_swap () #7 0x0001000a608f in ucnv_openStandardNames () #8 0x0001000a614f in ucnv_openStandardNames () #9 0x0001000a72a2 in ucnv_getUnicodeSet () #10 0x0001000b72af in ucnv_getDisplayName () #11 0x00010e88 in ?? () -- assignee: ronaldoussoren components: Macintosh, ctypes files: testctypes.py messages: 156205 nosy: ogre, ronaldoussoren priority: normal severity: normal status: open title: Crash in _ctypes_alloc_callback type: crash versions: Python 2.7 Added file: http://bugs.python.org/file24919/testctypes.py ___ Python tracker <http://bugs.python.org/issue14354> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14354] Crash in _ctypes_alloc_callback
Joe Rumsey added the comment:
I just built python 3.2.2 from source, and reproduced the issue there as well.
Same location. Here's the slightly more informative stack trace from my
release-with-symbols 3.2.2 build:
#0 _ctypes_alloc_callback (callable=0x7fff5fbfef20, converters=0x10003,
restype=0x7fff5fbfef20, flags=1606414112) at callbacks.c:432
#1 0x0001010c5395 in PyCFuncPtr_new (type=0x7fff5fbfefc0,
args=0x101010580, kwds=0x7fff5fbfefc0) at _ctypes.c:3372
#2 0x00010004986c in type_call (type=0x1006ca940, args=0x101041b90,
kwds=0x0) at typeobject.c:676
#3 0x00018605 in PyObject_Call (func=0x1006ca940, arg=0x101041b90,
kw=0x0) at abstract.c:2149
#4 0x00010008afcc in do_call [inlined] () at
/Users/ogre/src/Python-3.2.2/Python/ceval.c:4141
#5 0x00010008afcc in PyEval_EvalFrameEx (f=0x7fff5fbff1c0,
throwflag=1606414784) at ceval.c:3944
#6 0x00010009131b in PyEval_EvalCodeEx (_co=0x7fff5fbff260, globals=0x0,
locals=0x1066c880101, args=0x7fff5fbff260, argcount=1606414944,
kws=0x7fff5fbff260, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0)
at ceval.c:3350
#7 0x00010009139f in PyEval_EvalCode (co=0x101066828, globals=0x101066c88,
locals=0x0) at ceval.c:767
#8 0x0001000b0a21 in run_mod [inlined] () at
/Users/ogre/src/Python-3.2.2/Python/pythonrun.c:1783
#9 0x0001000b0a21 in PyRun_FileExFlags (fp=0x7fff7c677ee0,
filename=0x101066828 "\002", start=0, globals=0x101066c88, locals=0x0,
closeit=1, flags=0x7fff5fbff420) at pythonrun.c:1740
#10 0x0001000b2992 in PyRun_SimpleFileExFlags (fp=0x7fff7c677ee0,
filename=0x10104ac20 "testctypes.py", closeit=1606415200, flags=0x7fff5fbff360)
at pythonrun.c:1265
#11 0x0001000c45af in run_file [inlined] () at
/Users/ogre/src/Python-3.2.2/Modules/main.c:297
#12 0x0001000c45af in Py_Main (argc=1606415440, argv=0x7fff5fbff450) at
main.c:692
#13 0x00011522 in main (argc=17197096, argv=0x100609fe0) at python.c:59
converters doesn't seem to be pointing at valid data.
(gdb) p *converters
$2 = {
ob_refcnt = 3302829852670,
ob_type = 0xe000200
}
But, being an optimized build, it's hard to say (especially for me, having
never debugged python itself before) if that's the real data or some
optimizer-mangled version.
--
versions: +Python 3.2
___
Python tracker
<http://bugs.python.org/issue14354>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14354] Crash in _ctypes_alloc_callback
Joe Rumsey added the comment: Thanks for that. This does seem to be the case. I rebuilt with CC=gcc-4.2 and my short sample and the full library I took it from both work fine. -- ___ Python tracker <http://bugs.python.org/issue14354> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14354] Crash in _ctypes_alloc_callback
Joe Rumsey added the comment: It's maybe not directly relevant to fixing this, but I worked around it on the project where this came up by redefining dice as (c_int * 4) and col as (c_uint8 * 3) in the union, then using ctypes.cast to get those as pointers to the actual struct. That seems to work just fine for now. -- ___ Python tracker <http://bugs.python.org/issue14354> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37270] Manage memory lifetime for all type-related objects.
New submission from Joe Jevnik : When using PyType_FromSpec, the memory for PyType_Spec.name, Py_tp_methods, and Py_tp_members needs to somehow outlive the resulting type. This makes it hard to use this interface to generate types without just leaking the memory for these arrays, which is bad if you are programatically creating short-lived types. I posted about this on capi-sig, and the response seemed to be that it would be to replace the things that currently hold pointers into the array with copies of the data. Remove internal usages of PyMethodDef and PyGetSetDef. For PyMethodDef, change PyCFunctionObject to replace the PyMethodDef* member with a PyCFunctionBase member. The PyCFunctionBase is a new struct to hold the managed values of a PyMethodDef. This type is shared between PyCFunction and the various callable descriptor objects. A PyCFunctionBase is like a PyMethodDef but replaces the char* members with PyObject* members. For PyGetSetDef, inline the members on the resulting PyGetSetDescrObject, replacing all char* members with PyObject* members. The memory for the closure is *not* managed, adding support for that would likely require an API change and can be done in a future change. For the tp_name field, instead of setting it directly to the value of PyType_Spec.name, set it to the result of PyUnicode_AsUTF8(ht_name), where ht_name is the PyUnicode object created from the original spec name. This is the same trick used to properly manage this pointer for heap types when the __name__ is reassigned. -- components: Extension Modules messages: 345539 nosy: ll priority: normal severity: normal status: open title: Manage memory lifetime for all type-related objects. versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue37270> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37270] Manage memory lifetime for all type-related objects.
Change by Joe Jevnik : -- keywords: +patch pull_requests: +13925 stage: -> patch review pull_request: https://github.com/python/cpython/pull/14066 ___ Python tracker <https://bugs.python.org/issue37270> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30859] Can't install Python for Windows 3.6.1 on multiple profiles
New submission from Joe Jacobs: Windows 7 Ultimate: SP1. Fully pathed as of July 4th, 2017 My Windows 7 install has multiple user accounts. The main install account is set up with "Administrator" privledges. Some point in the past I had installed Python 3.6.1 on the main account. Did not install for all users. That put the python files inside the %appdata% folder. I was then trying to go through the Python Minecraft book with my daughter. We had logged in as my daughters login and tried to install Python 3.6.1 for her. Got an error stating Python was already installed, even though she couldn't access it for her Windows login. Ultimately, had to uninstall Python 3.6, and reinstall for "All users". Would think that multiple users could install Python in their own folder regardless of versions that other people have installed. -- components: Windows messages: 297776 nosy: Joe Jacobs, paul.moore, steve.dower, tim.golden, zach.ware priority: normal severity: normal status: open title: Can't install Python for Windows 3.6.1 on multiple profiles type: behavior versions: Python 3.6 ___ Python tracker <http://bugs.python.org/issue30859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30859] Can't install Python for Windows 3.6.1 on multiple profiles
Joe Jacobs added the comment: I checked, there are no install logs. Just realized, maybe an issue with the web installer rather than the base installer. python-3.6.1-amd64-webinstall Had downloaded this file and tried to run it. I wouldn't even get to the screen that asks you to do a standard or custom install. The only screen that popped up said python was already installed. I run it now after i uninstalled and re-installed for "All Users" and i don't have that issue. Are the logs for the web installer stored somewhere else? If not, maybe i'll try again and see if i can replicate again. -- ___ Python tracker <http://bugs.python.org/issue30859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30859] Can't install Python for Windows 3.6.1 on multiple profiles
Joe Jacobs added the comment: So, with the feedback, i uninstalled everything and tried installing again, but in the %appdata% folder for both users and everything worked fine. Both users have python installed. If i run the web launcher, the first time it installs on the second account and subsequent running of the launcher ask me to uninstall or repair as expected. I'm unable to find logs for the failed installs in the %temp% folder. If i'm able to duplicate the issue, i'll open a new bug report. Please go ahead and close this report. -- ___ Python tracker <http://bugs.python.org/issue30859> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28638] Optimize namedtuple creation
Joe Jevnik added the comment: I added a benchmark suite (using Victor's perf utility) to cnamedtuple. The results are here: https://github.com/ll/cnamedtuple#benchmarks To summarize: type creation is much faster; instance creation and named attribute access are a bit faster. -- nosy: +ll ___ Python tracker <http://bugs.python.org/issue28638> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue31557] tarfile: incorrectly treats regular file as directory
New submission from Joe Tsai:
The original V7 header only allocates 100B to store the file path. If a path
exceeds this length, then either the PAX format or GNU formats must be used,
which can represent arbitrarily long file paths. When doing so, most tar
writers just store the first 100B of the file path in the V7 header.
When reading, a proper reader should disregard the contents of the V7 field if
a previous and corresponding PAX or GNU header overrode it.
This currently not the case with the tarfile module, which has the following
check
(https://github.com/python/cpython/blob/c7cc14a825ec156c76329f65bed0d0bd6e03d035/Lib/tarfile.py#L1054-L1057):
# Old V7 tar format represents a directory as a regular
# file with a trailing slash.
if obj.type == AREGTYPE and obj.name.endswith("/"):
obj.type = DIRTYPE
This check should be further constrained to only activate when there were no
prior PAX or GNU records that override that value of obj.name. This check was
the source of a bug that caused tarfile to report a regular as a directory
because the file path was extra long, and when the tar write truncated the path
to the first 100B, it so happened to end on a slash.
--
messages: 302778
nosy: Joe Tsai
priority: normal
severity: normal
status: open
title: tarfile: incorrectly treats regular file as directory
___
Python tracker
<https://bugs.python.org/issue31557>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36068] Make _tuplegetter objects serializable
New submission from Joe Jevnik : The new _tuplegetter objects for accessing fields of a namedtuple are no longer serializable with pickle. Cloudpickle, a library which provides extensions to pickle to facilitate distributed computing in Python, depended on being able to pickle the members of namedtuple classes. While property isn't serializable, cloudpickle has support for properties allowing us to serialize the old property(itemgetter) members. The attached PR adds a __reduce__ method to _tuplegetter objects which will allow serialization without special support. Another option would be to expose `index` as a read-only attribute, allowing cloudpickle or other libraries to provide the pickle implementation as a third-party library. -- components: Library (Lib) messages: 336251 nosy: ll priority: normal severity: normal status: open title: Make _tuplegetter objects serializable type: enhancement versions: Python 3.8 ___ Python tracker <https://bugs.python.org/issue36068> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36068] Make _tuplegetter objects serializable
Change by Joe Jevnik : -- keywords: +patch pull_requests: +12006 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36068> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36068] Make _tuplegetter objects serializable
Joe Jevnik added the comment: Thank you for reviewing this so quickly! -- ___ Python tracker <https://bugs.python.org/issue36068> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30180] PyArg_ParseTupleAndKeywords supports required keyword only arguments
New submission from Joe Jevnik: I opened a pr to remove a line in the docs about $ needing to follow | in PyArg_ParseTupleAndKeywords. In practice, you can just use a $ to create required keyword arguments which intuitively makes sense. I was told this should raise a SystemError; however, you can create required keyword only arguments in Python so I am not sure why we would want to fail when this is done with PyArg_ParseTupleAndKeywords. -- messages: 292385 nosy: ll, serhiy.storchaka priority: normal pull_requests: 1417 severity: normal status: open title: PyArg_ParseTupleAndKeywords supports required keyword only arguments ___ Python tracker <http://bugs.python.org/issue30180> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30180] PyArg_ParseTupleAndKeywords supports required keyword only arguments
Changes by Joe Jevnik : -- assignee: -> docs@python components: +Documentation nosy: +docs@python ___ Python tracker <http://bugs.python.org/issue30180> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30477] tuple.index error message improvement
New submission from Joe Jevnik: The old error of tuple.index(x): x not in tuple seemed very confusing to me. It was also harder to quickly understand what the program was doing wrong. This saves people a second pass through the program under the debugger because they can just see what the invalid value was. The reason I am explicitly calling repr instead of using %R is that I didn't want to call repr twice if I didn't need to. If people think that is fine the format can just be tuple.index(%R): %R not in tuple instead. -- messages: 294516 nosy: brett.cannon, ll priority: normal pull_requests: 1906 severity: normal status: open title: tuple.index error message improvement versions: Python 3.7 ___ Python tracker <http://bugs.python.org/issue30477> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30477] tuple.index error message improvement
Joe Jevnik added the comment: I agree, "%R in tuple" is enough information for me. This would also remove the need to manually repr the object. -- type: enhancement -> ___ Python tracker <http://bugs.python.org/issue30477> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
