[issue43911] Queue.get() memory leak
New submission from Jens :
I use the following code to produce what looks like a memory leak after
emptying a queue with the get() method.
import queue
import os
import psutil
def run(del_after_puts, del_after_gets, n_puts, process):
mem = queue.Queue()
for msg in range(n_puts):
msg_put =
f'{msg}_03311'
if msg % 100 == 0:
print(f'puting {msg} qsize {len(mem.queue)}')
mem.put(msg_put)
print(f'-- put done - qsize {len(mem.queue)}')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
if del_after_puts:
print(f'deleting queue after puts {mem}')
del mem
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
return
for _ in range(n_puts):
msg_get = mem.get()
msg = int(msg_get.split('_')[0])
if msg % 100 == 0:
print(f'getting_q {msg} qsize {len(mem.queue)} ')
mem.task_done()
print(f'-- gets done - qsize {len(mem.queue)}')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
if del_after_gets:
print(f'deleting queue after gets {mem}')
del mem
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
return
if __name__ == '__main__':
del_after_puts = False
del_after_gets = False
n_puts = 20_000_000
print()
print('#')
print(f'del_after_puts {del_after_puts} del_after_gets
{del_after_gets} n_puts {n_puts}')
process = psutil.Process(os.getpid())
print('before run')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
run(del_after_puts, del_after_gets, n_puts, process)
print(f'after run return')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
This script can be run in 3 ways:
1. Add n_puts elements into the queue and then empty it.
2. Add n_puts elements into the queue and then delete the queue object
3. Add n_puts elements into the queue and then empty it and then delete the
queue object.
For the 1st and 3rd case, the script seems to produce a memory leak as in the
following:
1st case, before putting elements into the queue mem used is 0.15%, after
emptying it 2.22%:
> #
>
> del_after_puts False del_after_gets False n_puts 2000
>
> before run
>
> mem_pct 0.15%
>
> -- put done - qsize 2000
>
> mem_pct 37.61%
>
> -- gets done - qsize 0
>
> mem_pct 2.22%
3rd case, before putting elements into the queue mem used is 0.15%, after
emptying it 2.22%, after deleting the object, 2.22%:
> #
>
> del_after_puts False del_after_gets True n_puts 2000
>
> before run
>
> mem_pct 0.15%
>
> -- put done - qsize 2000
>
> mem_pct 37.61%
>
> -- gets done - qsize 0
>
> mem_pct 2.22%
>
> deleting queue after gets
>
> mem_pct 2.22%
For the 2nd case, mem_pct at the start is 0.15%, after putting all elements
into the queue and just deleting it, 0.16%, which is almost the same.
> #
>
> del_after_puts True del_after_gets False n_puts 2000
>
> before run
>
> mem_pct 0.15%
>
> -- put done - qsize 2000
>
> mem_pct 37.61%
>
> deleting queue after puts
>
> mem_pct 0.16%
As it can be seen, memory returns to the start level only in the first case
when only `queue.put()` is invoked, hence it seems that `queue.get()` produces
a memory leak.
This is persistent across python 3.7, 3.8, as well as 3.9.
--
messages: 391586
nosy: multiks2200
priority: normal
severity: normal
status: open
title: Queue.get() memory leak
versions: Python 3.9
___
Python tracker
<https://bugs.python.org/issue43911>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment: I've tried to profile the memory with tracemalloc as well as pympler, but they dont show any leaks at python level, so I suspect that might be a C level leak. -- ___ Python tracker <https://bugs.python.org/issue43911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment: Hi, Thanks for your reply, so I've run same script with queue.PriorityQueue, queue.LifoQueue, queue.SimpleQueue, Asyncio.Queue as well as collections.dequeue 1. PriorityQueue ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize 2000 >mem_pct 37.49% >-- gets done - qsize 0 >mem_pct 0.16% >deleting queue after gets >mem_pct 0.16% 2. LifoQueue ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize 2000 >mem_pct 37.49% >-- gets done - qsize 0 >mem_pct 0.16% >deleting queue after gets >mem_pct 0.16% 3. SimpleQueue ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize 2000 >mem_pct 37.49% >-- gets done - qsize 0 >mem_pct 0.28% >deleting queue after gets <_queue.SimpleQueue object at 0x7f1da93d03b0> >mem_pct 0.28% 4. asyncio.Queue ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.23% >-- put done - qsize 2000 >mem_pct 37.69% >-- gets done - qsize 0 >mem_pct 2.3% >deleting queue after gets >mem_pct 0.25% 5. collections.deque ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize 2000 >mem_pct 37.61% >-- gets done - qsize 0 >mem_pct 2.22% >deleting queue after gets deque([]) >mem_pct 0.16% So from the result it can be seen, that PriorityQueue and LifoQueue dont leak at all, and return to original memory after queues are emptied, so deleting the object reference has no effect. SimpleQueue seems to leak in the same way as Queue, but to a way lesser extent and deleting object does not return the memory. asyncio.Queue and collections.deque seem to leak to the same extent as Queue, but memory can be returned by deleting the object. As result, this got be to just trying to use Queue._put() and Queue._get() to populate the deque inside the Queue object and bypass all conditions/locks logic, which produces the following result: ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize 2000 >mem_pct 37.61% >-- gets done - qsize 0 >mem_pct 2.22% >deleting queue after gets >mem_pct 0.16% It can be seen that for a Queue where only _put() and _get() methods are used, the memory still leaks, but can be released by deleting the object reference. Perhaps that indicates a possible leak in the Condition class? I run a Queue with threads on my production applications that run for weeks straight, and the Queue leaks, and memory profiler always shows a leak at waiters_to_notify = _deque(_islice(all_waiters, n)) in Condition.notify() I dont want to make it more confusing by bringing it up here since it might be not the same kind of leak, because I dont simply populate up to 20million objects and then pop them out of the Queue in my application, but from the memory profiling results I see here this seems to be related. Thanks -- ___ Python tracker <https://bugs.python.org/issue43911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment: Just inspected the PriorityQueue and LifoQueue classes, they dont employ a deque at all but simply a list, but all other Queues tested do (except the native SimpleQueue). Since they don't leak, the leak itself seems to be coming from deque, and the fact that it does not get released after deletion only in a single case when Conditions are involved, makes me think the combination of two produces this unreleasable leak. -- ___ Python tracker <https://bugs.python.org/issue43911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment: Results for queue._PySimpleQueue: ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize 2000 >mem_pct 37.61% >-- gets done - qsize 0 >mem_pct 2.22% >deleting queue after gets >mem_pct 2.22% The leak occurs and not getting released after deletion. -- ___ Python tracker <https://bugs.python.org/issue43911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment: Thanks for your input. So i've run the tests with the List of Lists Queue class, there seems to be a resulting difference depending on what qsize() method I define, that is called my script. For an instance where qsize just return None, class QueueLists(list): def put(self, x): if not self or len(self[-1]) >= 66: self.append([]) self[-1].append(x) def get(self): if not self: raise IndexError block = self[0] x = block.pop(0) if not block: self.pop(0) return x # def qsize(self): # tot = 0 # for elem in self: # tot += len(elem) # return tot def qsize(self): return None The results are: ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize None >mem_pct 38.06% >-- gets done - qsize None >mem_pct 2.35% >deleting queue after gets [] >mem_pct 2.35% >time elapsed 0:01:04.703969 For a Queue instance, where qsize() returns the actual size class QueueLists(list): def put(self, x): if not self or len(self[-1]) >= 66: self.append([]) self[-1].append(x) def get(self): if not self: raise IndexError block = self[0] x = block.pop(0) if not block: self.pop(0) return x def qsize(self): tot = 0 for elem in self: tot += len(elem) return tot the results are: ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize 2000 >mem_pct 38.05% >-- gets done - qsize 0 >mem_pct 2.35% >deleting queue after gets [] >mem_pct 0.18% >time elapsed 0:00:53.347746 So both instances leak as you've indicated, but the one that returns None as queue size does not get it's leak released after the instance is deleted which is a weird difference. -- ___ Python tracker <https://bugs.python.org/issue43911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment: Regarding deque, the leak indeed does not seem to be releasable after it is inited to up the size of the number of elements that are going to put into the queue, as: qmem = collections.deque(range(n_puts)) qmem.clear() The results are: ># >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.14% >-- put done - qsize 2000 >mem_pct 37.61% >-- gets done - qsize 0 >mem_pct 2.22% >deleting queue after gets deque([]) >mem_pct 2.22% >time elapsed 0:00:16.800156 -- ___ Python tracker <https://bugs.python.org/issue43911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment: So this got me thinking of trying to use some other linked list implementations. I've used a llist library - https://github.com/ajakubek/python-llist Using their doubly linked list implementation: class DllistQueue(queue.Queue): def _init(self, maxsize): self.queue = dllist() Results are: >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >-- put done - qsize 2000 >mem_pct 55.34% >-- gets done - qsize 0 >mem_pct 0.15% >deleting queue after gets <__main__.DllistQueue object at 0x7f494ba91450> >mem_pct 0.15% >time elapsed 0:02:17.642700 Using their singly listed list implementation: class SllistQueue(queue.Queue): def _init(self, maxsize): self.queue = sllist() results are: >del_after_puts False del_after_gets True n_puts 2000 >before run >mem_pct 0.15% >puting 0 qsize 0 >-- put done - qsize 2000 >mem_pct 55.34% >-- gets done - qsize 0 >mem_pct 0.15% >deleting queue after gets <__main__.SllistQueue object at 0x7ff07bf484d0> >mem_pct 0.15% >time elapsed 0:02:03.495047 I have not dived in how their C implementations differ from deque, but it seems to use more memory and it's slower, but it does not seem to leak at all. Thanks -- ___ Python tracker <https://bugs.python.org/issue43911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment: Also compared this library to deque, and Queues based on this: https://github.com/kata198/python-cllist It seems to be as fast as deque, uses a bit more memory at the top usage, but does not leak at all. -- ___ Python tracker <https://bugs.python.org/issue43911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment: Ok, I see, thanks Raymond. Queue based logging leaks seem to hang my deployed applications atm, so this seemed like a possible reason for it. I use locally 8GB Ubuntu, and it gets to 2.2% after return with 20million puts, and on a remote 1GB Ubuntu instance, it gets to 3.79% up from 1.24% with 3 million puts after return. So I don't think this is necessarily specific to my local build and seems like a genuine issue. I understand where are you coming from stating that it's not a bug, but the alternative deque implementations don't seem to suffer from the same issue, at least to such a big extent, as I have shown. I will continue and try to adapt alternate deque implementations and see if it solves Queues leaking and hanging my instances. Thanks for your time and involvement in this. -- ___ Python tracker <https://bugs.python.org/issue43911> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue43911] Queue.get() memory leak
Jens added the comment:
Raymond, thanks for your suggestions.
My deployed applications don't hold up 20m items at a time, that was a way to
show the leak.
I was able to resolve the threading, queue-based leaks on my instances by
modifying the Queue, Event and Conditions classes to use external doubly linked
list library in the following manner (python3.7), not the clearest rewrite, but
just to get the idea:
class DllistCondition(threading.Condition):
def __init__(self, lock=None):
if lock is None:
lock = threading.RLock()
self._lock = lock
self.acquire = lock.acquire
self.release = lock.release
try:
self._release_save = lock._release_save
except AttributeError:
pass
try:
self._acquire_restore = lock._acquire_restore
except AttributeError:
pass
try:
self._is_owned = lock._is_owned
except AttributeError:
pass
self._waiters = dllist()
def notify(self, n=1):
if not self._is_owned():
raise RuntimeError("cannot notify on un-acquired lock")
all_waiters = self._waiters
waiters_to_notify = all_waiters
#_islize would be empty only if there are no waiters avail, for any
n
if len(waiters_to_notify) < 1:
return
node = waiters_to_notify.first
i = 1
while True:
#simulate _islice
if i > n:
return
waiter = node.value
#get next node before release
node_next = node.next
waiter.release()
try:
all_waiters.remove(node)
except ValueError:
pass
i += 1
node = node_next
#if it's the last node, return
if node is None:
return
def wait(self, timeout=None):
if not self._is_owned():
raise RuntimeError("cannot wait on un-acquired lock")
waiter = threading._allocate_lock()
waiter.acquire()
node = self._waiters.append(waiter)
saved_state = self._release_save()
gotit = False
try:# restore state no matter what (e.g., KeyboardInterrupt)
if timeout is None:
waiter.acquire()
gotit = True
else:
if timeout > 0:
gotit = waiter.acquire(True, timeout)
else:
gotit = waiter.acquire(False)
return gotit
finally:
self._acquire_restore(saved_state)
if not gotit:
try:
self._waiters.remove(node)
except ValueError:
pass
class DllistEvent(threading.Event):
def __init__(self):
self._cond = DllistCondition(threading.Lock())
self._flag = False
class DllistQueue(queue.Queue):
def __init__(self, maxsize=0):
self.maxsize = maxsize
self._init(maxsize)
self.mutex = threading.Lock()
self.not_empty = DllistCondition(self.mutex)
self.not_full = DllistCondition(self.mutex)
self.all_tasks_done = DllistCondition(self.mutex)
self.unfinished_tasks = 0
def _init(self, maxsize):
self.queue = dllist()
Now, I'm not exactly sure that the Queue itself required the `self.queue =
deque()`modification, but I'm sure that conditions required getting rid of the
stock `self._waiters = deque()` and the consequent use of it.
Memory profiling constantly showed a leak at waiters_to_notify =
_deque(_islice(all_waiters, n) in the threading.Condition class, which is both
employed by threading.Queue and treading.Event classes.
After the modifiction this leak is gone and I suspect it has to do something
with reiniting the deque at `waiters_to_notify = _deque(_islice(all_waiters, n)`
In any cases, I'm leaving this out here in case anyone get to deal with it as
well and it might be helpful.
Thanks
--
___
Python tracker
<https://bugs.python.org/issue43911>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42936] Decimal module performs wrong floor division with negative numbers
New submission from Jens :
from decimal import Decimal
print(-0.9//0.123)
# prints -8.0
print(Decimal('-0.9')//Decimal('0.123'))
# prints -7
print(-10//4.2)
# prints -3.0
print(Decimal('-10')//Decimal('4.2'))
# prints -2
--
messages: 385113
nosy: multiks2200
priority: normal
severity: normal
status: open
title: Decimal module performs wrong floor division with negative numbers
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue42936>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11638] pysetup un sdist crashes with weird trace if version is unicode by accident
Changes by Jens Diemer : -- nosy: +jens ___ Python tracker <http://bugs.python.org/issue11638> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11638] pysetup un sdist crashes with weird trace if version is unicode by accident
Jens Diemer added the comment: I have the same problem, using distutils (and not distutils2): Traceback (most recent call last): File "./setup.py", line 60, in test_suite="creole.tests.run_all_tests", File "/usr/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/home/jens/python2creole_env/local/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/sdist.py", line 147, in run File "/usr/lib/python2.7/distutils/command/sdist.py", line 448, in make_distribution owner=self.owner, group=self.group) File "/usr/lib/python2.7/distutils/cmd.py", line 392, in make_archive owner=owner, group=group) File "/usr/lib/python2.7/distutils/archive_util.py", line 237, in make_archive filename = func(base_name, base_dir, **kwargs) File "/usr/lib/python2.7/distutils/archive_util.py", line 101, in make_tarball tar = tarfile.open(archive_name, 'w|%s' % tar_compression[compress]) File "/usr/lib/python2.7/tarfile.py", line 1687, in open _Stream(name, filemode, comptype, fileobj, bufsize), File "/usr/lib/python2.7/tarfile.py", line 431, in __init__ self._init_write_gz() File "/usr/lib/python2.7/tarfile.py", line 459, in _init_write_gz self.__write(self.name + NUL) File "/usr/lib/python2.7/tarfile.py", line 475, in __write self.buf += s UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 1: ordinal not in range(128) The Problem seems that tarfile._Stream() can't handle 'name' as unicode. With this changes, it works: class _Stream: ... def __init__(self, name, mode, comptype, fileobj, bufsize): ... self.name = str(name) or "" + Don't know it this is related to the usage of: from __future__ import unicode_literals ? -- components: +Distutils ___ Python tracker <http://bugs.python.org/issue11638> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue5752] xml.dom.minidom does not escape CR, LF and TAB characters within attribute values
Changes by Jens Grivolla : -- nosy: +Jens.Grivolla ___ Python tracker <http://bugs.python.org/issue5752> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3332] DocTest and dict sort.
New submission from Jens Diemer <[EMAIL PROTECTED]>:
The doctest doesn't work good, if a function returns a dict.
Here a simple example:
def test(d):
"""
This works:
>>> test({"A":1, "B":2, "C":3})
{'A': 1, 'C': 3, 'B': 2}
This failed, because of different dict sort:
>>> test({"A":1, "B":2, "C":3})
{'A': 1, 'B': 2, 'C': 3}
The Error messages:
Failed example:
test({"A":1, "B":2, "C":3})
Expected:
{'A': 1, 'B': 2, 'C': 3}
Got:
{'A': 1, 'C': 3, 'B': 2}
"""
return d
The problem is IMHO that doctest.py [1] OutputChecker.check_output()
does compare the repr() of the dict and not the real dict as data.
One solution: Use eval() to convert the string repr. of the dict into
the real dict:
...
#--
try:
if eval(got) == eval(want):
return True
except:
pass #*pfeif* kein schoener stil, aber pragmatisch
#-
# We didn't find any match; return false.
return False
German discuss can be found here:
http://www.python-forum.de/topic-15321.html
[1] http://svn.python.org/view/python/trunk/Lib/doctest.py?view=markup
--
components: Extension Modules
messages: 69501
nosy: jedie
severity: normal
status: open
title: DocTest and dict sort.
type: behavior
versions: Python 2.5
___
Python tracker <[EMAIL PROTECTED]>
<http://bugs.python.org/issue3332>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue3974] collections.namedtuple uses exec to create new classes
New submission from Jens Kadenbach <[EMAIL PROTECTED]>: Rewrite of the namedtuple implementation to avoid the use of exec for class generation. The new code uses a custom class dictionary and the builtin type to create new classes dynamically. -- components: Library (Lib) files: new_namedtuples.diff keywords: patch messages: 73844 nosy: audax severity: normal status: open title: collections.namedtuple uses exec to create new classes versions: Python 2.6 Added file: http://bugs.python.org/file11608/new_namedtuples.diff ___ Python tracker <[EMAIL PROTECTED]> <http://bugs.python.org/issue3974> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37943] mimetypes.guess_extension() doesn’t get JPG right
New submission from Jens Troeger :
I think this one’s quite easy to reproduce:
Python 3.7.4 (default, Jul 11 2019, 01:08:00)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mimetypes
>>> mimetypes.guess_extension("image/jpg") # Expected ".jpg"
>>> mimetypes.guess_extension("image/jpeg") # Expected ".jpg"
'.jpe'
According to MDN
https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Complete_list_of_MIME_types
only "image/jpeg" is a valid MIME type; however, I’ve seen quite a bit of
"image/jpg" out in the wild and I think that ought to be accounted for too.
Before I look into submitting a PR I wanted to confirm that this is an issue
that ought to be fixed. I think it is.
--
components: Library (Lib)
messages: 350408
nosy: _savage
priority: normal
severity: normal
status: open
title: mimetypes.guess_extension() doesn’t get JPG right
type: behavior
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue37943>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37943] mimetypes.guess_extension() doesn’t get JPG right
Jens Troeger added the comment:
Oops, forgot…
>>> mimetypes.guess_extension("image/jpeg") # Expected ".jpg" or ".jpeg"
as per referenced MDN. I personally would go with ".jpg" because that's the
more common file name extension.
--
___
Python tracker
<https://bugs.python.org/issue37943>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue45722] documentation missing information on objects in submodules
New submission from Jens Rapp : Documentation 5.4.2. Submodules tells what happens to modules which are imported inside __init__.py of a package> from .foo import Foo from .bar import Bar then executing the following puts a name binding to foo and bar in the spam module: >>> >>> import spam >>> spam.foo >>> spam.bar I miss information on what happes to Foo and Bar. is it directly usable under spam.Bar() or does one have to use spam.bar.Bar()? To my mind, that example should tell this. -- assignee: docs@python components: Documentation messages: 405769 nosy: docs@python, rapp.jens priority: normal severity: normal status: open title: documentation missing information on objects in submodules type: enhancement ___ Python tracker <https://bugs.python.org/issue45722> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39957] bpo39775 not fixed - inspect.Signature.parameters still dict/mappingproxy
New submission from Jens Reidel : Hi guys, compiling CPython from the master branch will result in a git history with the commit https://github.com/python/cpython/commit/211055176157545ce98e6c02b09d624719e6dd30 included and in Lib/inspect.py, however the return type is still like before and behaviour has not changed. Python 3.9.0a4+ (heads/master:be79373a78, Mar 11 2020, 16:36:27) [GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import inspect >>> inspect.signature(lambda x, y: None).parameters == inspect.signature(lambda y, x: None).parameters True >>> I have been able to confirm this on all builds I've done. To get it to do expected behaviour and return False on above code, I need to patch back the commit that changed OrderedDict to dict (https://raw.githubusercontent.com/Gelbpunkt/python-image/master/inspect.patch is the file I am using to patch). I have compiled against the codebase of https://github.com/python/cpython/commit/be79373a78c0d75fc715ab64253c9b757987a848 and believe this is some issue with the Lib/inspect.py code internally if the patch file can fix it. -- components: Library (Lib) messages: 364118 nosy: gelbpunkt priority: normal severity: normal status: open title: bpo39775 not fixed - inspect.Signature.parameters still dict/mappingproxy type: behavior versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue39957> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39957] bpo39775 not fixed - inspect.Signature.parameters still dict/mappingproxy around dict
Change by Jens Reidel : -- title: bpo39775 not fixed - inspect.Signature.parameters still dict/mappingproxy -> bpo39775 not fixed - inspect.Signature.parameters still dict/mappingproxy around dict ___ Python tracker <https://bugs.python.org/issue39957> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39957] bpo39775 not fixed - inspect.Signature.parameters still dict/mappingproxy around dict
Jens Reidel added the comment:
Just to show the types are inequal:
Without patch file:
>>> inspect.signature(lambda x: None).parameters
mappingproxy({'x': })
With patch file:
>>> inspect.signature(lambda x: None).parameters
mappingproxy(OrderedDict([('x', )]))
--
___
Python tracker
<https://bugs.python.org/issue39957>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue39957] bpo39775 not fixed - inspect.Signature.parameters still dict/mappingproxy around dict
Change by Jens Reidel : -- keywords: +patch pull_requests: +18335 stage: -> patch review pull_request: https://github.com/python/cpython/pull/18988 ___ Python tracker <https://bugs.python.org/issue39957> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40376] documentation for os.getgrouplist potentially misleading
New submission from Jens Holzkämper :
https://docs.python.org/3/library/os.html#os.getgrouplist states „Return list
of group ids that user belongs to. If group is not in the list, it is included;
typically, group is specified as the group ID field from the password record
for user.“, but the function is at least on Linux a wrapper for getgrouplist
from the C standard library, which lists only the membership in groups in the
group-database. Users can be members of groups without it being declared in the
group database, this is often the case with the default group of the user which
is only declared in the passwd database.
e.g.
/etc/passwd: woodfighter:x:1000:1000:,,,:/home/woodfighter:/bin/bash
/etc/group: woodfighter:x:1000:
os.getgrouplist("woodfighter",65534) then doesn't contain group id 1000.
The documentation tries to steer a developer in the right direction with the
second sentence but fails to state, that the list will be possibly incomplete
otherwise. I would add something like „, because that group ID will otherwise
be potentially omitted.“ before the last period.
--
assignee: docs@python
components: Documentation
messages: 367187
nosy: docs@python, woodfighter
priority: normal
severity: normal
status: open
title: documentation for os.getgrouplist potentially misleading
type: enhancement
___
Python tracker
<https://bugs.python.org/issue40376>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue40376] documentation for os.getgrouplist potentially misleading
Change by Jens Holzkämper : -- keywords: +patch pull_requests: +19020 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19702 ___ Python tracker <https://bugs.python.org/issue40376> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue41471] After installing python 3.x on Mac pip doesn't work at all
New submission from Jens Petersen :
Using pip fails with the same error messages independent of version an user.
Also su or sudo -H doesn't change anything:
How to Reproduce
install python 3.x on your Mac and try pip
After a quite long search I found that it is a problem with the proxy config.
You need to remove all settings in network settings/proxy even those defining
which domains shouldn't use a proxy. Proxy settings work fine with everything
except python.
xcode-select install
ruby -e "$(curl -fsSL
https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install pyenv
pyenv install 3.7.3
pip list --outdated or pip install something
Exception
xxx-MBP:~ xxx$ pip list --outdated
Exception:
Traceback (most recent call last):
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/cli/base_command.py",
line 179, in main
status = self.run(options, args)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/commands/list.py",
line 144, in run
packages = self.get_outdated(packages, options)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/commands/list.py",
line 152, in get_outdated
dist for dist in self.iter_packages_latest_infos(packages, options)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/commands/list.py",
line 152, in
dist for dist in self.iter_packages_latest_infos(packages, options)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/commands/list.py",
line 179, in iter_packages_latest_infos
all_candidates = finder.find_all_candidates(dist.key)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/index.py",
line 610, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/index.py",
line 743, in _get_pages
page = _get_html_page(location, session=self.session)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/index.py",
line 229, in _get_html_page
resp = _get_html_response(url, session=session)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/index.py",
line 177, in _get_html_response
"Cache-Control": "max-age=0",
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py",
line 546, in get
return self.request('GET', url, **kwargs)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_internal/download.py",
line 403, in request
return super(PipSession, self).request(method, url, *args, **kwargs)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py",
line 524, in request
prep.url, proxies, stream, verify, cert
File
"/Users/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_vendor/requests/sessions.py",
line 700, in merge_environment_settings
env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_vendor/requests/utils.py",
line 761, in get_environ_proxies
if should_bypass_proxies(url, no_proxy=no_proxy):
File
"/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/site-packages/pip/_vendor/requests/utils.py",
line 745, in should_bypass_proxies
bypass = proxy_bypass(parsed.hostname)
File "/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/urllib/request.py", line
2610, in proxy_bypass
return proxy_bypass_macosx_sysconf(host)
File "/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/urllib/request.py", line
2587, in proxy_bypass_macosx_sysconf
return _proxy_bypass_macosx_sysconf(host, proxy_settings)
File "/Users/xxx/.pyenv/versions/3.7.3/lib/python3.7/urllib/request.py", line
2573, in _proxy_bypass_macosx_sysconf
if (hostIP >> mask) == (base >> mask):
ValueError: negative shift count
--
components: macOS
messages: 374800
nosy: ned.deily, peteje66, ronaldoussoren
priority: normal
pull_requests: 20872
severity: normal
status: open
title: After installing python 3.x on Mac pip doesn't work at all
type: crash
versions: Python 3.8
___
Python tracker
<https://bugs.python.org/issue41471>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37943] mimetypes.guess_extension() doesn’t get JPG right
Jens Troeger added the comment:
This is still not working: tried it on Python 3.8.5 and Python 3.7.8.
>>> import mimetypes
>>> mimetypes.guess_extension('image/jpg')
>>> mimetypes.guess_extension('image/jpeg')
'.jpg'
Both should return the same value; I expected the mimetype 'image/jpg' to
return extension '.jpg' because that mimetype is used a lot.
--
resolution: out of date -> remind
status: closed -> open
___
Python tracker
<https://bugs.python.org/issue37943>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37943] mimetypes.guess_extension() doesn’t get JPG right
Jens Troeger added the comment: @fbidu, oh I missed that, thank you! Shall I close the issue again, or what’s the common procedure in this case? -- ___ Python tracker <https://bugs.python.org/issue37943> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue42253] xml.dom.minidom.rst missed informations
New submission from Jens Diemer : The standalone arguments was added in Python 3.9. This information is missed in the docu. -- messages: 380277 nosy: jedie2 priority: normal pull_requests: 22042 severity: normal status: open title: xml.dom.minidom.rst missed informations versions: Python 3.9 ___ Python tracker <https://bugs.python.org/issue42253> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34424] Unicode names break email header
Jens Troeger added the comment: Can somebody please review and merge https://github.com/python/cpython/pull/8803 ? I am still waiting for this fix the become mainstream. -- ___ Python tracker <https://bugs.python.org/issue34424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30717] Add unicode grapheme cluster break algorithm
Change by Jens Troeger : -- nosy: +_savage ___ Python tracker <https://bugs.python.org/issue30717> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28686] py.exe ignored PATH when using python3 shebang
Jens Lindgren added the comment: I just got hit by this bug and would like to add my thoughts on this. If you are in an activated venv, no matter if you launch with command python or python3, it will launch the version in venv (version 3.6.1 in this case). I expect the py-launcher and shebang to work the same way. In fact it works as expected on Linux and '#! /usr/bin/env pyton3' are in fact using the venv version. This is a pretty major bug that needs to be fixed asap in my opinion. -- nosy: +Jens Lindgren ___ Python tracker <http://bugs.python.org/issue28686> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28686] py.exe ignored PATH when using python3 shebang
Jens Lindgren added the comment: Sorry I need to clarify. On Linux both python and python3 works as there is a symlink created from python to python3 in the venv folder. On Windows only python.exe is created. I copied it to python3.exe. Now I can use python3 script.py to start but py-launcher and shebang still didn't work with '/usr/bin/env python3'. I expect this to work the same on Windows as it does on Linux. -- ___ Python tracker <http://bugs.python.org/issue28686> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36633] py_compile.compile: AttributeError on importlib.utils
New submission from Jens Vagelpohl : The following code in py_compile.compile fails (tested on 3.6.6 and 3.7.3) with tracebacks that end like the one shown at the bottom. There's an AttributeError about importlib.utils. """ if cfile is None: if optimize >= 0: optimization = optimize if optimize >= 1 else '' cfile = importlib.util.cache_from_source(file, optimization=optimization) else: cfile = importlib.util.cache_from_source(file) """ Sample tail end of traceback: """ File "/Users/jens/src/.eggs/Chameleon-3.6-py3.7.egg/chameleon/template.py", line 243, in _cook cooked = self.loader.build(source, filename) File "/Users/jens/src/.eggs/Chameleon-3.6-py3.7.egg/chameleon/loader.py", line 177, in build py_compile.compile(name) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/py_compile.py", line 130, in compile cfile = importlib.util.cache_from_source(file) AttributeError: module 'importlib' has no attribute 'util' """ -- components: Library (Lib) messages: 340271 nosy: dataflake priority: normal severity: normal status: open title: py_compile.compile: AttributeError on importlib.utils versions: Python 3.6, Python 3.7 ___ Python tracker <https://bugs.python.org/issue36633> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36633] py_compile.compile: AttributeError on importlib.utils
Jens Vagelpohl added the comment: Thank you for the prompt reply. It turns out this is not a bug in py_compile. Other code we use imports importlib.util briefly for a quick check at module level and then deletes it, also at module scope. Removing the deletion fixes the issue. Thanks again and apologies! -- resolution: -> third party ___ Python tracker <https://bugs.python.org/issue36633> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34424] Unicode names break email header
Jens Troeger added the comment: Cheryl, if you can find somebody to approve and merge this fix, that would be greatly appreciated! Anything I can do, please let me know. -- ___ Python tracker <https://bugs.python.org/issue34424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33398] From, To, Cc lines break when calling send_message()
New submission from Jens Troeger :
It looks like non-ascii characters in an Address()’s display_name parameter
cause their lines in the header to get mangled when the message is being sent.
For example, a case to reproduce:
>>> msg = EmailMessage()
>>> msg["To"] = Address(display_name="Jens Tröger",
addr_spec="jens.troe...@gmail.com")
>>> msg["From"] = Address(display_name="Jens Troeger",
addr_spec="jens.troe...@gmail.com")
>>> msg.set_content("Some content.")
>>> msg.as_string()
'To: Jens =?utf-8?q?Tr=C3=B6ger?= \nContent-Type:
text/plain; charset="utf-8"\nContent-Transfer-Encoding: 7bit\nMIME-Version:
1.0\nFrom: Jens Troeger \n\nSome content.\n'
Sending this email creates the following SMTP debug output:
>>> smtpsrv = smtplib.SMTP("smtp.gmail.com:587")
>>> …
>>> smtpsrv.send_message(msg)
send: 'mail FROM: size=220\r\n'
reply: b'250 2.1.0 OK z23sm16924622pfe.110 - gsmtp\r\n'
reply: retcode (250); Msg: b'2.1.0 OK z23sm16924622pfe.110 - gsmtp'
send: 'rcpt TO:\r\n'
reply: b'250 2.1.5 OK z23sm16924622pfe.110 - gsmtp\r\n'
reply: retcode (250); Msg: b'2.1.5 OK z23sm16924622pfe.110 - gsmtp'
send: 'data\r\n'
reply: b'354 Go ahead z23sm16924622pfe.110 - gsmtp\r\n'
reply: retcode (354); Msg: b'Go ahead z23sm16924622pfe.110 - gsmtp'
data: (354, b'Go ahead z23sm16924622pfe.110 - gsmtp')
send: b'To: Jens =?utf-8?q?Tr=C3=B6ger?=
\r\r\r\r\r\nContent-Type: text/plain;
charset="utf-8"\r\nContent-Transfer-
Encoding: 7bit\r\nMIME-Version: 1.0\r\nFrom: Jens Troeger
\r\n\r\nSome content.\r\n.\r\n'
reply: b'250 2.0.0 OK 1525174591 z23sm16924622pfe.110 - gsmtp\r\n'
reply: retcode (250); Msg: b'2.0.0 OK 1525174591 z23sm16924622pfe.110 -
gsmtp'
data: (250, b'2.0.0 OK 1525174591 z23sm16924622pfe.110 - gsmtp')
{}
Notice the string of "\r\r\…" for the "To" field which consequently breaks off
the remainder of the email’s header into a premature body:
[…]
Message-ID: <5ae8513e.17b9620a.eebf7.d...@mx.google.com>
Date: Tue, 01 May 2018 04:36:30 -0700 (PDT)
From: jens.troe...@gmail.com
To: Jens Tröger
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
From: Jens Troeger
Some content.
Also notice the two From fields. The first one, I suspect, is supplied from the
SMTP server’s login, the second one from them EmailMessage. Without a From in
the EmailMessage, I get the following error:
>>> smtpsrv.send_message(msg)
Traceback (most recent call last):
File "", line 1, in
File "/…/lib/python3.6/smtplib.py", line 936, in send_message
from_addr = email.utils.getaddresses([from_addr])[0][1]
File "/…/lib/python3.6/email/utils.py", line 112, in getaddresses
all = COMMASPACE.join(fieldvalues)
TypeError: sequence item 0: expected str instance, NoneType found
Similar breakage of the header into premature body can be achieved with the Cc
header field.
--
components: email
messages: 315994
nosy: _savage, barry, r.david.murray
priority: normal
severity: normal
status: open
title: From, To, Cc lines break when calling send_message()
type: behavior
versions: Python 3.6
___
Python tracker
<https://bugs.python.org/issue33398>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24218] Also support SMTPUTF8 in smtplib's send_message method.
Jens Troeger added the comment: I was about to open an issue when I found this one. Consider an email message with the following: message = EmailMessage() message["From"] = Address(addr_spec="b...@foo.com", display_name="Jens Troeger") message["To"] = Address(addr_spec="f...@bar.com", display_name="Martín Córdoba") It’s important here that the email itself is `ascii` encodable, but the names are not. With that combination, send_message() falsely assumes plain text addresses (see https://github.com/python/cpython/blob/master/Lib/smtplib.py#L949 where it checks only email addresses, not display names!) and therefore the `international` flag stays False. As a result of that, flattening the email object (https://github.com/python/cpython/blob/master/Lib/smtplib.py#L964) incorrectly inserts multiple linefeeds, thus breaking the email header, thus mangling the entire email: flatmsg: b'From: Jens Troeger \r\nTo: Fernando =?utf-8?q?Mart=C3=ADn_C=C3=B3rdoba?= \r\r\r\r\r\nSubject:\r\n Confirmation: …\r\n…' I think a proper fix would be in line 949, where email addresses and display names should be checked for encoding. The comment to that function should also be adjusted to mention display names? Note also that the attached patch does not test the above scenario, and should probably be extended as well. -- nosy: +_savage ___ Python tracker <https://bugs.python.org/issue24218> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24218] Also support SMTPUTF8 in smtplib's send_message method.
Jens Troeger added the comment: (continuing the previous message msg322761) …unless the addresses should be checked separately from the display names, in which case the BytesGenerator’s flatten() function should be fixed. Without reading the RFC, please let me know how to continue from here. -- ___ Python tracker <https://bugs.python.org/issue24218> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue33398] From, To, Cc lines break when calling send_message()
Jens Troeger added the comment: See also this issue comment: https://bugs.python.org/issue24218#msg322761 -- ___ Python tracker <https://bugs.python.org/issue33398> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24218] Also support SMTPUTF8 in smtplib's send_message method.
Jens Troeger added the comment: > Well, posting on a closed issue is generally not the best way :) Fair enough ;) > The multiple carriage returns is a bug, and there is an open issue for it, > though I'm not finding it at the moment. Oh good, yes that should be fixed! My current workaround is setting `international = True` _always_, which prevents the multiple CRs. Not pretty but it works… -- ___ Python tracker <https://bugs.python.org/issue24218> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24218] Also support SMTPUTF8 in smtplib's send_message method.
Jens Troeger added the comment: So that’s interesting. I thought that setting `international = True` (see line https://github.com/python/cpython/blob/master/Lib/smtplib.py#L947) would be a neat workaround, but the opposite. When delivering those emails to Gmail I started seeing Failed to send email: (555, b'5.5.2 Syntax error, goodbye. s53-v6sm1864855qts.5 - gsmtp', 'f...@bar.com') and it turns out (according to the IETF message linter, https://tools.ietf.org/tools/msglint/) that: --- UNKNOWN: unknown header 'User-Agent' at line 4 ERROR: missing mandatory header 'date' lines 1-7 ERROR: missing mandatory header 'return-path' lines 1-7 OK: found part text/plain line 9 WARNING: line 13 too long (109 chars); text/plain shouldn't need folding (RFC 2046-4.1.1) WARNING: line 15 too long (124 chars); text/plain shouldn't need folding (RFC 2046-4.1.1) WARNING: Character set mislabelled as 'utf-8' when 'us-ascii' suffices, body part ending line 22 --- It seems that now “Date” and “Return-Path” header entries are missing when the email is generated. I reverted the initial change. Any updates on the multiple CR problem when flattening? -- ___ Python tracker <https://bugs.python.org/issue24218> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24218] Also support SMTPUTF8 in smtplib's send_message method.
Jens Troeger added the comment:
David, I tried to find the mentioned '\r\r…\n' issue but I could not find it
here. However, from an initial investigation into the BytesGenerator, here is
what’s happening.
Flattening the body and attachments of the EmailMessage object works, and
eventually _write_headers() is called to flatten the headers which happens
entry by entry
(https://github.com/python/cpython/blob/master/Lib/email/generator.py#L417-L418).
Flattening a header entry is a recursive process over the parse tree of the
entry, which builds the flattened and encoded final string by descending into
the parse tree and encoding & concatenating the individual “parts” (tokens of
the header entry).
Given the parse tree for a header entry like "Martín Córdoba "
eventually results in the correct flattened string:
'=?utf-8?q?Mart=C3=ADn_C=C3=B3rdoba?= \r\n'
at the bottom of the recursion for this “Mailbox” part. The recursive callstack
is then:
_refold_parse_tree _header_value_parser.py:2687
fold [Mailbox] _header_value_parser.py:144
_refold_parse_tree _header_value_parser.py:2630
fold [Address] _header_value_parser.py:144
_refold_parse_tree _header_value_parser.py:2630
fold [AddressList] _header_value_parser.py:144
_refold_parse_tree _header_value_parser.py:2630
fold [Header] _header_value_parser.py:144
fold [_UniqueAddressHeader] headerregistry.py:258
_fold [EmailPolicy] policy.py:205
fold_binary [EmailPolicy] policy.py:199
_write_headers [BytesGenerator] generator.py:418
_write [BytesGenerator] generator.py:195
The problem now arises from the interplay of
#
https://github.com/python/cpython/blob/master/Lib/email/_header_value_parser.py#L2629
encoded_part = part.fold(policy=policy)[:-1] # strip nl
which strips the '\n' from the returned string, and
#
https://github.com/python/cpython/blob/master/Lib/email/_header_value_parser.py#L2686
return policy.linesep.join(lines) + policy.linesep
which adds the policy’s line separation string linesep="\r\n" to the end of the
flattened string upon unrolling the recursion.
I am not sure about a proper fix here, but considering that the linesep policy
can be any string length (in this case len("\r\n") == 2) a fixed truncation of
one character [:-1] seems wrong. Instead, using:
encoded_part = part.fold(policy=policy)[:-len(policy.linesep)] # strip nl
seems to work for entries with and without Unicode characters in their display
names.
David, please advise on how to proceed from here.
--
___
Python tracker
<https://bugs.python.org/issue24218>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24218] Also support SMTPUTF8 in smtplib's send_message method.
Jens Troeger added the comment: @David, any thoughts on this? -- ___ Python tracker <https://bugs.python.org/issue24218> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24218] Also support SMTPUTF8 in smtplib's send_message method.
Jens Troeger added the comment: Thanks David: PR on Github (which is R/O) or where should I submit to? -- ___ Python tracker <https://bugs.python.org/issue24218> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24218] Also support SMTPUTF8 in smtplib's send_message method.
Jens Troeger added the comment: New issue: https://bugs.python.org/issue34424 -- ___ Python tracker <https://bugs.python.org/issue24218> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34424] Unicode names break email header
New submission from Jens Troeger :
See also this comment and ensuing conversation:
https://bugs.python.org/issue24218?#msg322761
Consider an email message with the following:
message = EmailMessage()
message["From"] = Address(addr_spec="b...@foo.com", display_name="Jens Troeger")
message["To"] = Address(addr_spec="f...@bar.com", display_name="Martín Córdoba")
It’s important here that the email itself is `ascii` encodable, but the names
are not. Flattening the object
(https://github.com/python/cpython/blob/master/Lib/smtplib.py#L964) incorrectly
inserts multiple linefeeds, thus breaking the email header, thus mangling the
entire email:
flatmsg: b'From: Jens Troeger \r\nTo: Fernando
=?utf-8?q?Mart=C3=ADn_C=C3=B3rdoba?= \r\r\r\r\r\nSubject:\r\n
Confirmation: …\r\n…'
After an initial investigation into the BytesGenerator (used to flatten an
EmailMessage object), here is what’s happening.
Flattening the body and attachments of the EmailMessage object works, and
eventually _write_headers() is called to flatten the headers which happens
entry by entry
(https://github.com/python/cpython/blob/master/Lib/email/generator.py#L417-L418).
Flattening a header entry is a recursive process over the parse tree of the
entry, which builds the flattened and encoded final string by descending into
the parse tree and encoding & concatenating the individual “parts” (tokens of
the header entry).
Given the parse tree for a header entry like "Martín Córdoba "
eventually results in the correct flattened string:
'=?utf-8?q?Mart=C3=ADn_C=C3=B3rdoba?= \r\n'
at the bottom of the recursion for this “Mailbox” part. The recursive callstack
is then:
_refold_parse_tree _header_value_parser.py:2687
fold [Mailbox] _header_value_parser.py:144
_refold_parse_tree _header_value_parser.py:2630
fold [Address] _header_value_parser.py:144
_refold_parse_tree _header_value_parser.py:2630
fold [AddressList] _header_value_parser.py:144
_refold_parse_tree _header_value_parser.py:2630
fold [Header] _header_value_parser.py:144
fold [_UniqueAddressHeader] headerregistry.py:258
_fold [EmailPolicy] policy.py:205
fold_binary [EmailPolicy] policy.py:199
_write_headers [BytesGenerator] generator.py:418
_write [BytesGenerator] generator.py:195
The problem now arises from the interplay of
#
https://github.com/python/cpython/blob/master/Lib/email/_header_value_parser.py#L2629
encoded_part = part.fold(policy=policy)[:-1] # strip nl
which strips the '\n' from the returned string, and
#
https://github.com/python/cpython/blob/master/Lib/email/_header_value_parser.py#L2686
return policy.linesep.join(lines) + policy.linesep
which adds the policy’s line separation string linesep="\r\n" to the end of the
flattened string upon unrolling the recursion.
I am not sure about a proper fix here, but considering that the linesep policy
can be any string length (in this case len("\r\n") == 2) a fixed truncation of
one character [:-1] seems wrong. Instead, using:
encoded_part = part.fold(policy=policy)[:-len(policy.linesep)] # strip nl
seems to work for entries with and without Unicode characters in their display
names.
--
components: email
messages: 323686
nosy: _savage, barry, r.david.murray
priority: normal
severity: normal
status: open
title: Unicode names break email header
type: behavior
versions: Python 3.7
___
Python tracker
<https://bugs.python.org/issue34424>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue34424] Unicode names break email header
Jens Troeger added the comment: Pull request https://github.com/python/cpython/pull/8803/ -- ___ Python tracker <https://bugs.python.org/issue34424> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue28879] smtplib send_message should add Date header if it is missing, per RFC5322
Jens Troeger added the comment: Any updates on this? Looks like the proposed change has not been merged into mainstream yet? I’m having problems with Google rejecting emails: (555, b'5.5.2 Syntax error, goodbye. r10-v6sm7321838qtj.41 - gsmtp', '…') and using IETF’s message linter (https://tools.ietf.org/tools/msglint/) I get the following: ERROR: missing mandatory header 'date' lines 1-7 ERROR: missing mandatory header 'return-path' lines 1-7 amongst a few others. -- nosy: +_savage ___ Python tracker <https://bugs.python.org/issue28879> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4130] Intel icc 9.1 does not support __int128_t used by ctypes
Jens Timmerman added the comment: I believe this pull request fixed this upstream (it did for me) https://github.com/atgreen/libffi/pull/43 I fixed this by including "xmmintrin.h" instead of switching to reg_args->sse[ssecount].m (as per http://software.intel.com/en-us/forums/topic/303826 ) -- nosy: +Jens.Timmerman ___ Python tracker <http://bugs.python.org/issue4130> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14910] argparse disable abbreviation
Changes by Jens Jährig : -- components: None nosy: jens.jaehrig priority: normal severity: normal status: open title: argparse disable abbreviation type: enhancement versions: Python 2.6, Python 2.7 ___ Python tracker <http://bugs.python.org/issue14910> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14910] argparse disable abbreviation
New submission from Jens Jährig :
argparse uses per default abbreviation in unambiguous cases.
I don't want abbreviation and i'd like to disable it.
But theres no option to do this.
http://docs.python.org/library/argparse.html#argument-abbreviations
Only to override the Handler like suggested here:
http://stackoverflow.com/questions/10750802/python-argparse-disable-abbreviation/10751356#10751356
# Example #
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--send', action='store_true')
parser.parse_args(['--se']) # returns Namespace(send=True)
But i want it only to be true when the full parameter is supplied. To prevent
user errors.
--
___
Python tracker
<http://bugs.python.org/issue14910>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue14910] argparse: disable abbreviation
Changes by Jens Jährig : -- title: argparse disable abbreviation -> argparse: disable abbreviation ___ Python tracker <http://bugs.python.org/issue14910> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16936] Documentation for stat.S_IFMT inconsistent
New submission from Jens Lechtenboerger: The documentation for the stat module is inconsistent (Doc/library/stat.rst, at least for Python 2.7.2 and 3.3.0): It talks about a function stat.S_IFMT() and a bit mask stat.S_IFMT. Only the former does exist. Besides, it states: "For complete details about the stat(), fstat() and lstat() calls, consult the documentation for your system." I suggest to add some pointers on what systems one might consult what documentation: "(e.g., on GNU/Linux invoke 'man 2 stat')" I don't know about other systems, though. So, doing "man 2 stat", which refers to the POSIX standard and which seems to have served as blueprint for stat's interface, I expected to find the bit mask S_IFMT. However, that does not exist. In contrast, in stat.py that bit mask is hard-coded as 0o17 in the definition of S_IFMT(). As long-term, backwards-incompatible fix, I suggest to export S_IFMT = 0o17 and to rename the function S_IFMT(). That way, stat would in fact be aligned with the documentation for my system. As short-term fix, I suggest to correct stat.rst. Replace > Use of the functions above is more portable than use of the first > set of flags: > > stat.S_IFMT >Bit mask for the file type bit fields. with > Use of the functions above may be more portable than use of the > first set of flags. > > Warning: Note that the stat module does not export a bit mask > S_IFMT. (As stated incorrectly in previous versions of the > documentation.) Here, I replaced "is more portable" with "may be more portable" as the constant 0o17 is hard-coded into S_IFMT() in stat.py. Maybe somebody could add a hint in what sense portability is improved? Alternatively that sentence could be deleted. A diff for stat.rst (Python 3.3.0) is attached. -- assignee: docs@python components: Documentation files: stat.rst.diff keywords: patch messages: 179687 nosy: docs@python, lechten priority: normal severity: normal status: open title: Documentation for stat.S_IFMT inconsistent type: behavior versions: Python 3.3 Added file: http://bugs.python.org/file28697/stat.rst.diff ___ Python tracker <http://bugs.python.org/issue16936> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16998] Lost updates with multiprocessing.Value
New submission from Jens Lechtenboerger:
Maybe I'm misreading the documentation of multiprocessing.Value and
multiprocessing.sharedctypes.Value.
I thought that access to the value field of Value instances was protected by
locks to avoid lost updates.
Specifically, for multiprocessing.Value(typecode_or_type, *args[, lock]) and
multiprocessing.sharedctypes.Value(typecode_or_type, *args[, lock]) the
documentation states:
> By default the return value is actually a synchronized wrapper for the
> object. [...]
> If lock is True (the default) then a new lock object is created to
> synchronize access to the value. If lock is a Lock or RLock object then that
> will be used to synchronize access to the value. If lock is False then
> access to the returned object will not be automatically protected by a lock,
> so it will not necessarily be “process-safe”.
(By the way, I'm not sure why both, multiprocessing.Value and
multiprocessing.sharedctypes.Value are documented. They appear to be the same
thing.)
The following tests (also attached as file) show that lost updates may occur
if several instances of multiprocessing.Process increment the same Value that
is passed as args parameter.
def do_inc(integer):
"""Increment integer.value for multiprocessing.Value integer."""
integer.value += 1
def do_test(notasks):
"""Create notasks processes, each incrementing the same Value.
As the Value is initialized to 0, its final value is expected to be
notasks.
"""
tasks = list()
integer = multiprocessing.sharedctypes.Value("i", 0)
for run in range(notasks):
proc = multiprocessing.Process(target=do_inc, args=(integer,))
proc.start()
tasks.append(proc)
for proc in tasks:
proc.join()
if integer.value != notasks:
logging.error(
"Unexpected value: %d (expected: %d)", integer.value, notasks)
if __name__ == "__main__":
do_test(100)
Sample invocations and results:
Note that on a single CPU machine the error is not reported for every
execution but only for about every third run.
$ python --version
Python 2.6.5
$ uname -a
Linux ubuntu-desktop 2.6.32.11+drm33.2 #2 Fri Jun 18 20:30:49 CEST 2010 i686
GNU/Linux
$ python test_multiprocessing.py
ERROR:root:Unexpected value: 99 (expected: 100)
On a quadcore, the error occurs almost every time.
$ uname -a
Linux PC 2.6.35.13 #4 SMP Tue Dec 20 15:22:02 CET 2011 x86_64 GNU/Linux
$ ~/local/Python-2.7.3/python test_multiprocessing.py
ERROR:root:Unexpected value: 95 (expected: 100)
$ ~/local/Python-3.3.0/python test_multiprocessing.py
ERROR:root:Unexpected value: 86 (expected: 100)
--
components: Library (Lib)
files: test_multiprocessing.py
messages: 180251
nosy: lechten
priority: normal
severity: normal
status: open
title: Lost updates with multiprocessing.Value
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file28782/test_multiprocessing.py
___
Python tracker
<http://bugs.python.org/issue16998>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16998] Lost updates with multiprocessing.Value
Jens Lechtenboerger added the comment: > Loads and stores are both atomic. But "+=" is made up of two operations, a > load followed by a store, and the lock is dropped between the two. I see. Then this is a documentation bug. The examples in the documentation use such non-thread-safe assignments (combined with the statement "These shared objects will be process and thread safe."). -- ___ Python tracker <http://bugs.python.org/issue16998> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue16998] Lost updates with multiprocessing.Value
Jens Lechtenboerger added the comment: > It only says that accesses are synchronized. The problem is that you were > assuming that "+=" involves a single access -- but that is not how python > works. Yes, I understand that by now (actually since your first comment). > Where in the examples is there "non-process-safe" access? (Note that waiting > for the only process which modifies a value to terminate using join() will > prevent races.) In section "The multiprocessing.sharedctypes module" the assignments in the first example (function modify()) are unsafe. None of them is protected by a lock as suggested in your first comment. Strictly speaking, no lock is necessary in the example as there are no race conditions (the processes work in an alternating fashion without concurrency). I certainly did not see that the example (for a *shared* memory data structure) is based on the implicit assumption of a single writer. In contrast, I assumed that some "magic" should offer process-safe usage of "+=", which made me file this bug. That assumption has turned out to be wrong. To prevent others from being mislead in the same way I suggest to either protect those operations with locks (and comment on their effect) or to state the implicit assumption explicitly. Maybe add the following after "Below is an example where a number of ctypes objects are modified by a child process:" Note that assignments such n.value **= 2 are not executed atomically but involve two operations, a load followed by a store. Each of these operations is protected by the Value's lock, which is dropped in between. Thus, in scenarios with concurrent modifying processes you may want to explicitly acquire the Value's lock to ensure atomic execution (avoiding race conditions and lost updates), e.g.: with n.get_lock(): n.value **= 2 -- ___ Python tracker <http://bugs.python.org/issue16998> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4130] Intel icc 9.1 does not support __int128_t used by ctypes
Jens Timmerman added the comment: Since this is fixed in upstream libffi, can this be synced with the libffi included in python? -- ___ Python tracker <http://bugs.python.org/issue4130> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13492] ./configure --with-system-ffi=LIBFFI-PATH
Jens Timmerman added the comment: As a workaround, you can make the libffi build work by applying this patch. https://github.com/atgreen/libffi/pull/43 (indeed, see also http://bugs.python.org/issue4130 ) -- nosy: +Jens.Timmerman ___ Python tracker <http://bugs.python.org/issue13492> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22561] PyUnicode_InternInPlace crashes
New submission from Jens Troeger: This might be an issue with Python, or an issue with Libre/OpenOffice not setting up the UNO environment correctly. The crash happens during "import uno" of Python 3.3 in the PyUnicode_InternInPlace function. I've done some digging and posted more information about this crash in this forum: http://en.libreofficeforum.org/node/9195 -- components: Unicode messages: 228635 nosy: _savage, ezio.melotti, haypo priority: normal severity: normal status: open title: PyUnicode_InternInPlace crashes type: crash versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue22561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22561] PyUnicode_InternInPlace crashes
Jens Troeger added the comment: Thanks Victor. I had the suspicion that UNO might set up somewhat incorrectly, and consequently cause this problem. To answer your questions: - Debug symbols: agreed. I haven't built a vanilla Python with symbols yet. I'm using MacPorts default Python 3.3. - Yes, this is Python on Mac using MacPorts. - Yes, UNO is compatible with Python 3.3. When you install LibreOffice (on Mac) then it ships with a Python 3.3 interpreter. Interestingly, using THAT interpreter the crash does not happen when I import the module, but it does happen -- on occasion -- upon exiting. However, I do not know what options Python is compiled with for LibreOffice. Shall I bounce this issue back to the LibreOffice folks and see if I can find whoever owns that piece of code? (If anybody does...) -- ___ Python tracker <http://bugs.python.org/issue22561> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4130] Intel icc 9.1 does not support __int128_t used by ctypes
Jens Timmerman added the comment: sorry for my confusion, libffi's website stated libffi-3.0.14 was released on TBD. I must have missed the TBD part. -- ___ Python tracker <http://bugs.python.org/issue4130> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22874] gzip bug in python 2.7.3?
New submission from Jens Bonerz: I am getting the "Not a gzipped file" exception while retrieving a gzipped sitemap xml (tested on amazon.de) using scrapy- I am using Python 2.7.3 and Scrapy 0.24.4 Can anyone confirm gzip being broken in 2.7.3 or am I overseeing something? -- messages: 231167 nosy: jbonerz priority: normal severity: normal status: open title: gzip bug in python 2.7.3? ___ Python tracker <http://bugs.python.org/issue22874> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue22874] gzip bug in python 2.7.3?
Jens Bonerz added the comment: closed. Problem caused by 3rd party app -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue22874> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24086] Configparser interpolation is unexpected
Jens Diemer added the comment:
IMHO i ran into the same bug, try to "./setup.py upload" with my new password
in ~/.pypirc:
configparser.InterpolationSyntaxError: '%' must be followed by '%' or '(',
found: "%foobar"
This is a limitation of the configparser... But what's about to validate this
on the "set PyPi password view" ?
So that's impossible to set a PyPi password which can't be parsed by
configparser ?
--
nosy: +jens
___
Python tracker
<http://bugs.python.org/issue24086>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21417] Compression level for zipfile
Changes by Jens Diemer : -- nosy: +jens ___ Python tracker <http://bugs.python.org/issue21417> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue19894] zipfile ignores deflate level settings in zipinfo object
Jens Diemer added the comment: IMHO it should be possible to set compression level not only for DEFLATE. And it should be similar with the tarfile API. Seems that http://bugs.python.org/issue21417 will cover this. -- nosy: +jens ___ Python tracker <http://bugs.python.org/issue19894> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21417] Compression level for zipfile
Jens Diemer added the comment: btw. hacked work-a-round is: zlib.Z_DEFAULT_COMPRESSION = 9 -- ___ Python tracker <http://bugs.python.org/issue21417> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue24931] _asdict breaks when inheriting from a namedtuple
Jens Troeger added the comment:
With my update from Python 3.4.3 to Python 3.4.4 (default, Dec 25 2015,
06:14:41) I started experiencing crashes of my applications and I suspect this
change is the culprit.
I have a class that inherits from namedtuple, and code calls vars() (i.e.
retrieve __dict__) to iterate over an instance's attributes. Much like Raymond
points out in http://bugs.python.org/msg249100
For example with 3.4.3:
>>> from collections import namedtuple
>>> Point = namedtuple('Point', ['x', 'y'])
>>> p = Point(1,2)
>>> p
Point(x=1, y=2)
>>> p.__dict__
OrderedDict([('x', 1), ('y', 2)])
>>> vars(p)
OrderedDict([('x', 1), ('y', 2)])
After the most recent update this breaks with 3.4.4:
>>> from collections import namedtuple
>>> Point = namedtuple('Point', ['x', 'y'])
>>> p = Point(1,2)
>>> p
Point(x=1, y=2)
>>> p.__dict__
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'Point' object has no attribute '__dict__'
>>> vars(p)
Traceback (most recent call last):
File "", line 1, in
TypeError: vars() argument must have __dict__ attribute
I am not sure about the fix on my side. Should I use _asdict() instead of
vars() although I would argue that vars() should remain functional across this
change. Calling _asdict() seems messy to me, but it works:
>>> p._asdict()
OrderedDict([('x', 1), ('y', 2)])
Why not keep the __dict__ property in tact?
@property
def __dict__(self):
return self._asdict()
Thanks!
--
nosy: +_savage
___
Python tracker
<http://bugs.python.org/issue24931>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18199] Windows: support path longer than 260 bytes using "\\?\" prefix
Jens Diemer added the comment: I also with this problems. I have made a test script. There is a problem with os.chdir(): It doesn't work with \\?\ notation. And there is also a problem, if you use ``` import os import pathlib import tempfile with tempfile.TemporaryDirectory(prefix="path_test_") as path: new_path = pathlib.Path(path, "A"*255, "B"*255) extended_path = "?\\%s" % new_path os.makedirs(extended_path) print(len(extended_path), extended_path) ``` os.makedirs() will work, but the cleanup will failed. Output is: ``` 567 \\?\C:\Users\jens\AppData\Local\Temp\path_test_8fe6utdz\AAA\BBB Traceback (most recent call last): File "D:\test2.py", line 18, in print(len(extended_path), extended_path) File "C:\Program Files (x86)\Python35-32\lib\tempfile.py", line 807, in __exit__ self.cleanup() File "C:\Program Files (x86)\Python35-32\lib\tempfile.py", line 811, in cleanup _shutil.rmtree(self.name) File "C:\Program Files (x86)\Python35-32\lib\shutil.py", line 488, in rmtree return _rmtree_unsafe(path, onerror) File "C:\Program Files (x86)\Python35-32\lib\shutil.py", line 383, in _rmtree_unsafe onerror(os.unlink, fullname, sys.exc_info()) File "C:\Program Files (x86)\Python35-32\lib\shutil.py", line 381, in _rmtree_unsafe os.unlink(fullname) FileNotFoundError: [WinError 3] Das System kann den angegebenen Pfad nicht finden: 'C:\\Users\\jens\\AppData\\Local\\Temp\\path_test_8fe6utdz\\AAA' ``` -- nosy: +jens Added file: http://bugs.python.org/file41820/test.py ___ Python tracker <http://bugs.python.org/issue18199> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue18199] Windows: support path longer than 260 bytes using "\\?\" prefix
Jens Diemer added the comment: I have made https://github.com/jedie/pathlib_revised to address this, see: https://github.com/jedie/pathlib_revised#windows-max_path The idea is to add a property (I call it 'extended_path') and this will add the \\?\ prefix on all absolute path under windows. Under non-Windows the property will return ".path" And i patch methods like 'chmod', 'unlink', 'rename' etc. to use the 'extended_path' property and i add more methods like 'link', 'listdir', 'copyfile' etc. The source code is here: https://github.com/jedie/pathlib_revised/blob/master/pathlib_revised/pathlib.py This is another thing: Why are not all filesystem access methods implemented in pathlib?!? e.g.: There is Path.unlink() but no Path.link() There is Path.rmdir() but no Path.chdir() and many more. And the last thing is: Why is pathlib so bad designed? It's ugly to extend it. But this address: https://bugs.python.org/issue24132 -- ___ Python tracker <http://bugs.python.org/issue18199> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue13492] ./configure --with-system-ffi=LIBFFI-PATH
Jens Timmerman added the comment: Wel, I can confirm that this is fixed in new libffi shipped with python now, and the problem no longer occurs on 3.4.3 (only version I checked) -- ___ Python tracker <http://bugs.python.org/issue13492> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue4130] Intel icc 9.1 does not support __int128_t used by ctypes
Jens Timmerman added the comment: yep, newer versions of python with newer libffi do not longer have this issue, confirmed with python 3.4.3 -- ___ Python tracker <http://bugs.python.org/issue4130> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23102] distutils: isinstance checks fail with setuptools-monkeypatched Extension/Distribution
Jens Timmerman added the comment: I'm also regularly running into this, it is really annoying, Can I do anything to help getting this merged in? -- nosy: +Jens.Timmerman ___ Python tracker <http://bugs.python.org/issue23102> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue23102] distutils: isinstance checks fail with setuptools-monkeypatched Extension/Distribution
Jens Timmerman added the comment: small setup.py file to reproduce this problem if people still had trouble reproducing (this works with the attached d002-distutils-type-checks-can-fail-issue-23102.patch) ``` #!/usr/bin/env python ''' Installation script that breaks ''' from distutils.command import install from distutils.core import setup class installer(install.install): def run(self): import setuptools install.install.run(self) setup(name='test', cmdclass = dict(install=installer) ) ``` run with: python setup.py install --user result: ``` running install running build running install_egg_info Traceback (most recent call last): File "mysetup.py", line 14, in cmdclass = dict(install=installer) File "/usr/lib64/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib64/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib64/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "mysetup.py", line 10, in run install.install.run(self) File "/usr/lib64/python2.7/distutils/command/install.py", line 575, in run self.run_command(cmd_name) File "/usr/lib64/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/usr/lib64/python2.7/distutils/dist.py", line 970, in run_command cmd_obj = self.get_command_obj(command) File "/usr/lib64/python2.7/distutils/dist.py", line 846, in get_command_obj cmd_obj = self.command_obj[command] = klass(self) File "/usr/lib64/python2.7/distutils/cmd.py", line 59, in __init__ raise TypeError, "dist must be a Distribution instance" TypeError: dist must be a Distribution instance ``` expected result: ``` running install running build running install_egg_info Writing /home/jens/.local/lib/python2.7/site-packages/test-0.0.0-py2.7.egg-info ``` -- ___ Python tracker <http://bugs.python.org/issue23102> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue47082] No protection: `import numpy` in two different threads can lead to race-condition
New submission from Jens Henrik Goebbert :
While using [Xpra](https://github.com/Xpra-org/xpra) we came across a bug which
might be a Python or a NumPy issue.
Perhaps some of you can help us understanding some internals.
Calling `import numpy` at the same time in two different threads of the Python
program can lead to a race-condition.
This happens for example with Xpra when loading the encoder nvjpeg:
```
2022-03-20 12:54:59,298 cannot load enc_nvjpeg (nvjpeg encoder)
Traceback (most recent call last):
File "/lib/python3.9/site-packages/xpra/codecs/loader.py", line
52, in codec_import_check
ic = __import__(class_module, {}, {}, classnames)
File "xpra/codecs/nvjpeg/encoder.pyx", line 8, in init
xpra.codecs.nvjpeg.encoder
File "/lib/python3.9/site-packages/numpy/__init__.py", line 150,
in
from . import core
File "/lib/python3.9/site-packages/numpy/core/__init__.py", line
51, in
del os.environ[envkey]
File "/lib/python3.9/os.py", line 695, in __delitem__
raise KeyError(key) from None
KeyError: 'OPENBLAS_MAIN_FREE'
```
Here the environment variable OPENBLAS_MAIN_FREE is set in the `numpy` code:
https://github.com/numpy/numpy/blob/maintenance/1.21.x/numpy/core/__init__.py#L18
and short after that it is deleted
https://github.com/numpy/numpy/blob/maintenance/1.21.x/numpy/core/__init__.py#L51
But this deletion fails ... perhaps because the initialization runs at the same
time in two threads :thinking:
Shouldn't Python protect us by design?
@seberg comments
[HERE](https://github.com/numpy/numpy/issues/21223#issuecomment-1074008386):
```
So, my current hypothesis (and I have briefly checked the Python code) is that
Python does not do manual locking. But it effectively locks due to this going
into C and thus holding the GIL. But somewhere during the import of NumPy,
NumPy probably releases the GIL briefly and that could allow the next thread to
go into the import machinery.
[..]
NumPy may be doing some worse than typical stuff here, but right now it seems
to me that Python should be protecting us.
```
Can anyone comment on this?
--
components: Interpreter Core
messages: 415687
nosy: jhgoebbert
priority: normal
severity: normal
status: open
title: No protection: `import numpy` in two different threads can lead to
race-condition
type: behavior
versions: Python 3.11
___
Python tracker
<https://bugs.python.org/issue47082>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27846] Base64 expansion factor is 4 to 3, not 6 to 4
New submission from Jens Jørgen Mortensen: The documentation has a note saying: "Base64 has an expansion factor of 6 to 4". I believe it should be 4 to 3. https://docs.python.org/dev/library/base64.html -- assignee: docs@python components: Documentation messages: 273543 nosy: Jens.Jørgen.Mortensen, docs@python priority: normal severity: normal status: open title: Base64 expansion factor is 4 to 3, not 6 to 4 type: enhancement versions: Python 3.6 ___ Python tracker <http://bugs.python.org/issue27846> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27846] Base64 expansion factor is 4 to 3, not 6 to 4
Jens Jørgen Mortensen added the comment: That would also be a solution. Also, the sentence is not quite grammatically correct: "when space [is] expensive". -- versions: -Python 3.5 ___ Python tracker <https://bugs.python.org/issue27846> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27846] Base64 expansion factor is 4 to 3, not 6 to 4
Changes by Jens Jørgen Mortensen : -- versions: +Python 3.5 ___ Python tracker <https://bugs.python.org/issue27846> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue26943] Datetime.strptime crash
New submission from Jens de Bruijn:
Datetime crashes while running script from the command line. When running the
same date string from the interpreter (Ubuntu 16.04) it does not crash.
date = datetime.datetime.strptime('May 01 23:59:59 + 2016', '%b %d %H:%M:%S
+ %Y')
--
components: Extension Modules
files: Screenshot from 2016-05-04 08-52-30.png
messages: 264784
nosy: Jens de Bruijn
priority: normal
severity: normal
status: open
title: Datetime.strptime crash
type: crash
versions: Python 3.5
Added file: http://bugs.python.org/file42706/Screenshot from 2016-05-04
08-52-30.png
___
Python tracker
<http://bugs.python.org/issue26943>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue15764] Sqlite3 performance
New submission from Jens Henrik Leonhard Jensen: The _sqlite modules uses sqlite3_prepare and not sqlite3_prepare_v2. With sqlite3_prepare_v2 enables more optiomation. Workaround: Do not use parameters as the right hand argument to GLOB or LIKE. For more documentation see http://www.sqlite.org/optoverview.html. -- components: Extension Modules files: db-test.py messages: 168881 nosy: jhlj priority: normal severity: normal status: open title: Sqlite3 performance type: performance versions: Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3, Python 3.4 Added file: http://bugs.python.org/file26964/db-test.py ___ Python tracker <http://bugs.python.org/issue15764> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
