[issue42154] Bad proxy returned immediately after BaseManager server restarted
New submission from john ryan :
I am building an application that is made up of several separate processes,
where each process is a python program. They are all started by the supervisord
utility and execute within a venv running Python 3.8.5 (default, Aug 13 2020,
15:42:06) [GCC 7.5.0] on linux, under Ubuntu 18.04.
I am using a multiprocessing BaseManager to implement a repository of queues.
Each process asks for a queue by name then uses put/get on that queue.
The application needs to be resilient so it must be possible to restart the
respository process and have the various client processes re-connect to the
queues hosted by it.
The problem I am getting is that the first call to `get_queue()` after
restarting the BaseManager server process does not return a queue.
The sequence below shows some testing by hand. (My test environment runs Ubuntu
in a virtualbox hosted on Windows 8.1)
Here I started the server in a different terminal then started python as below
(both pythons in the same venv).
This works as expected with the first call to get_queue returning a queue.
```
(hydra_env) john@u1804-VirtualBox:~/sw/code/hydra$ python
Python 3.8.5 (default, Aug 13 2020, 15:42:06)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing.managers import BaseManager
>>> class QueueManager(BaseManager): pass
...
>>> QueueManager.register('get_queue')
>>> mgr = QueueManager(address=('localhost', 5), authkey=b'abracadabra' )
>>> mgr.connect()
>>> q = mgr.get_queue('name', 'src'); print(str(q))
>>> q = mgr.get_queue('name', 'src'); print(str(q))
```
Stop and restart the server to see the problem. The first call to get_queue
seems to succeed but in fact it has failed as shown by the print(str...). The
second call to get_queue succeeds.
```
>>> mgr.connect()
>>> q = mgr.get_queue('name', 'src'); print(str(q))
>>> q = mgr.get_queue('name', 'src'); print(str(q))
```
The server logs show it sent queues on all 4 calls
```
^C(hydra_env) john@u1804-VirtualBox:~/sw/code/hydra$ python
../../trials/test_mgr.py
starting
serving
serving
^C(hydra_env) john@u1804-VirtualBox:~/sw/code/hydra$ python
../../trials/test_mgr.py
starting
serving
serving
```
I get the same behaviour if I re-instantiate the local manager object
```
>>> mgr = QueueManager(address=('localhost', 5), authkey=b'abracadabra' )
>>> mgr.connect()
>>> q = mgr.get_queue('name', 'src'); print(str(q))
>>> q = mgr.get_queue('name', 'src'); print(str(q))
>>>
```
I even get the same behaviour if I just call `get_queue()` after restarting the
server (ie without explicitly reconnecting).
I would have expected the first call to `get_queue()` to return a valid queue
since neither it nor the call to `connect()` raised any kind of error.
It seems to me that there is some kind of state held that is the underlying
cause of the issue. I did some investigating in but I was not able to work out
what was happening.
I found that it was possible to get into a state where a valid queue was never
returned by `get_queue()` if an error had been raised by `get_nowait()` first.
Stop the server
```
>>> q.get_nowait()
Traceback (most recent call last):
File "", line 1, in
File "", line 2, in get_nowait
File
"/home/john/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/managers.py",
line 835, in _callmethod
kind, result = conn.recv()
File
"/home/john/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/connection.py",
line 250, in recv
buf = self._recv_bytes()
File
"/home/john/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/connection.py",
line 414, in _recv_bytes
buf = self._recv(4)
File
"/home/john/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/connection.py",
line 383, in _recv
raise EOFError
EOFError
```
Restart the server but do not call `get_queue()`
```
>>> q.get_nowait()
Traceback (most recent call last):
File "", line 1, in
File "", line 2, in get_nowait
File
"/home/john/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/managers.py",
line 834, in _callmethod
conn.send((self._id, methodname, args, kwds))
File
"/home/john/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/connection.py",
line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File
"/home/john/.pyenv/versions/3.8.5/lib/python3.8/multiprocessing/connection.py",
line 411, in _send_bytes
self._send(header + buf)
File
"/home/john/.pyenv/versions/3
[issue42203] Unexpected behaviour NameError: name 'open' is not defined
New submission from john ryan : My test environment runs Ubuntu 18.04 in a virtualbox hosted on Windows 8.1 and Python executes within a venv running Python 3.9.0 (Python 3.9.0 (default, Oct 26 2020, 09:02:51) [GCC 7.5.0] on linux Running a test with unittest.IsolatedAsyncioTestCase my code hung so I hit ctrl-C twice to stop it and got the below traceback. The really odd behaviour is that open is reported as not defined. The behaviour is repeatable in my test, but I do not know how to produce a simple test case. I do not know what the issue is with my code. ^C^CTraceback (most recent call last): File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/async_case.py", line 158, in run return super().run(result) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/case.py", line 593, in run self._callTestMethod(testMethod) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/async_case.py", line 65, in _callTestMethod self._callMaybeAsync(method) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/async_case.py", line 88, in _callMaybeAsync return self._asyncioTestLoop.run_until_complete(fut) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete self.run_forever() File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 596, in run_forever self._run_once() File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 1854, in _run_once event_list = self._selector.select(timeout) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/selectors.py", line 469, in select fd_event_list = self._selector.poll(timeout, max_ev) KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/__main__.py", line 18, in main(module=None) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/main.py", line 101, in __init__ self.runTests() File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/main.py", line 271, in runTests self.result = testRunner.run(self.test) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/runner.py", line 176, in run test(result) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/suite.py", line 122, in run test(result) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/suite.py", line 122, in run test(result) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/suite.py", line 122, in run test(result) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/case.py", line 653, in __call__ return self.run(*args, **kwds) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/async_case.py", line 160, in run self._tearDownAsyncioLoop() File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/unittest/async_case.py", line 126, in _tearDownAsyncioLoop loop.run_until_complete(self._asyncioCallsQueue.join()) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete self.run_forever() File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 596, in run_forever self._run_once() File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 1854, in _run_once event_list = self._selector.select(timeout) File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/selectors.py", line 469, in select fd_event_list = self._selector.poll(timeout, max_ev) KeyboardInterrupt Exception ignored in: > Traceback (most recent call last): File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/asyncio/base_events.py", line 1771, in call_exception_handler File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/logging/__init__.py", line 1463, in error File "/home/john/.pyenv/versions/3.9.0/lib/python3.9/lo
[issue42203] Unexpected behaviour NameError: name 'open' is not defined
john ryan added the comment: Thanks. That is understandable. I reported it in case it was helpful. -- ___ Python tracker <https://bugs.python.org/issue42203> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
