[issue30441] os.environ raises RuntimeError: dictionary changed size during iteration
Osvaldo Santana Neto added the comment: Pull Request #2409 (https://github.com/python/cpython/pull/2409) opened. -- pull_requests: +2457 ___ Python tracker <http://bugs.python.org/issue30441> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30441] os.environ raises RuntimeError: dictionary changed size during iteration
Osvaldo Santana Neto added the comment: Hi Serhiy Storchaka, I need to thank you for your valuable guidance. -- ___ Python tracker <http://bugs.python.org/issue30441> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30441] os.environ raises RuntimeError: dictionary changed size during iteration
New submission from Osvaldo Santana Neto:
We're facing ocasional RuntimeError exceptions in a multithreaded application
when one of the threads creates new entries to the environment (os.environ).
I'm not so sure if the attached patch fixes this issue the right way, so, feel
free to propose another approach.
Traceback Sample of the issue in our production environment:
RuntimeError: dictionary changed size during iteration
File "python3.5/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
[ internal code ]
File "newrelic/api/background_task.py", line 103, in wrapper
return wrapped(*args, **kwargs)
File "python3.5/contextlib.py", line 30, in inner
return func(*args, **kwds)
File "python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
File "raven/base.py", line 851, in make_decorator
yield
File "python3.5/contextlib.py", line 30, in inner
return func(*args, **kwds)
[ internal code ]
File "retry/api.py", line 74, in retry_decorator
logger)
File "retry/api.py", line 33, in __retry_internal
return f()
[ internal code ]
File "requests/sessions.py", line 531, in get
return self.request('GET', url, **kwargs)
File "requests/sessions.py", line 509, in request
prep.url, proxies, stream, verify, cert
File "requests/sessions.py", line 686, in merge_environment_settings
env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
File "requests/utils.py", line 696, in get_environ_proxies
return getproxies()
File "urllib/request.py", line 2393, in getproxies_environment
for name, value in os.environ.items():
File "_collections_abc.py", line 676, in __iter__
for key in self._mapping:
File "python3.5/os.py", line 702, in __iter__
for key in self._data:
--
components: Library (Lib)
files: fix_os_environ_iteration_issue.diff
keywords: patch
messages: 294250
nosy: osantana
priority: normal
severity: normal
status: open
title: os.environ raises RuntimeError: dictionary changed size during iteration
type: behavior
versions: Python 3.5, Python 3.6
Added file: http://bugs.python.org/file46889/fix_os_environ_iteration_issue.diff
___
Python tracker
<http://bugs.python.org/issue30441>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30441] os.environ raises RuntimeError: dictionary changed size during iteration
Osvaldo Santana Neto added the comment: I agree with Jelle about the fix's implementation. But I disagree of suggestion to implement lock at the application side. I disagree because some external libraries may want to access os.environ in a concurrent fashion without lock acquiring. That's the case reported in my previous comment: urllib/requests.py (https://github.com/python/cpython/blob/master/Lib/urllib/request.py#L2468) iterates over os.environ with no lock control. How could I handle this issue in this case? Would it be a good idea implement the lock mechanism in current os._Environ (https://github.com/python/cpython/blob/master/Lib/os.py#L666) mapping class? -- ___ Python tracker <http://bugs.python.org/issue30441> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30441] os.environ raises RuntimeError: dictionary changed size during iteration
Osvaldo Santana Neto added the comment: There are exceptions being raised in many applications (as reported here and in http://bugs.python.org/issue25641) and there are three paths to follow: 1. We handle this exception somewhere; 2. We avoid raising it; 3. Just leave it. I don't care about this exception. If we chose to handle this exception we need to decide where we'll do it. At os.py module? At urllib/request.py? At all http client libraries (eg. python-requests)? At our applications? If we chose to avoid this exception we, probably, need to implement some kind of lock/mutex in os.environ. I can implement any of these options. You just need to decide which one is best. If we chose the third option, we can just close this issue. -- ___ Python tracker <http://bugs.python.org/issue30441> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30441] os.environ raises RuntimeError: dictionary changed size during iteration
Osvaldo Santana Neto added the comment: This patch implements a lock protection (as suggested by Antoine) using `_thread.allocate_lock()` module (instead of `threading.Lock()`) due to the fact that CPython interpreter already imports `_thread` module during its bootstrap process. -- Added file: http://bugs.python.org/file46896/fix_os_environ_iter_issue_low_level_thread_lock.diff ___ Python tracker <http://bugs.python.org/issue30441> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30441] os.environ raises RuntimeError: dictionary changed size during iteration
Osvaldo Santana Neto added the comment: This is an alternative implementation using `threading.Lock()`. The main issue with this implementation relies on the fact that we've to import the `threading` module during CPython interpreter loading (currently it's not imported by default). This is consequence of the fact that CPython interpreter uses `os` module when bootstrapping. -- Added file: http://bugs.python.org/file46897/fix_os_environ_iter_issue_threading_lock.diff ___ Python tracker <http://bugs.python.org/issue30441> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue30441] os.environ raises RuntimeError: dictionary changed size during iteration
Osvaldo Santana Neto added the comment: New version of patch: 1. Use RLock instead of Lock (Mark & Antoine recomendation) 2. Add lock acquire at `__setitem__` and `__delitem__` as following (Mark & Antoine recomendation) 3. Add lock protection in `__repr__` implementation. I've some questions pending: 1. Is it a good idea to wrap self._data access with lock in __getitem__? (I suspect it's not) 2. Is it a good idea to lock protect self.copy() and self.setdefault() (I suspect it's not because both uses __iter__ that is already protected) -- Added file: http://bugs.python.org/file46902/fix_os_environ_iter_issue_low_level_thread_lock_2.diff ___ Python tracker <http://bugs.python.org/issue30441> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21183] Doctest capture only AssertionError but not printed text
New submission from Osvaldo Santana Neto:
The following doctest should pass, but it fails:
>>> def spam(): print("eggs")
...
>>> assert spam()
eggs
Traceback (most recent call last):
AssertionError
But if we remove the print output from printed results the test pass:
>>> def spam(): print("eggs")
...
>>> assert spam()
Traceback (most recent call last):
AssertionError
I'm writing the 2nd edition of a book about Python (covering python3) and
Django and I'm using doctest to run the interactive sessions that I use as
examples in book.
--
components: Library (Lib)
files: doctest_bug.py
messages: 215796
nosy: osantana
priority: normal
severity: normal
status: open
title: Doctest capture only AssertionError but not printed text
type: behavior
versions: Python 2.7, Python 3.4
Added file: http://bugs.python.org/file34771/doctest_bug.py
___
Python tracker
<http://bugs.python.org/issue21183>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue21183] Doctest capture only AssertionError but not printed text
Osvaldo Santana Neto added the comment: Hi Tim, I tried to find more information in documentation before opening this bug but I need to confess that I didn't read the footnote. Thanks and sorry for the invalid report. -- resolution: -> invalid status: open -> closed ___ Python tracker <http://bugs.python.org/issue21183> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
