[issue34563] invalid assert on big output of multiprocessing.Process

2018-09-01 Thread Oleksandr Buchkovskyi 


New submission from Oleksandr Buchkovskyi :

the bug is reproduced on big multiprocessing.Process output
when the size of the output gets bigger than signed int the value becomes 
negative, thus
```
assert left > 0
```
in multiprocessing/connection.py:337 raises an exception like the following

```
Traceback (most recent call last):
  File "D:\GitHub\cpython\lib\threading.py", line 917, in _bootstrap_inner
self.run()
  File "D:\GitHub\cpython\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
  File "D:\GitHub\cpython\lib\multiprocessing\pool.py", line 470, in 
_handle_results
task = get()
  File "D:\GitHub\cpython\lib\multiprocessing\connection.py", line 250, in recv
buf = self._recv_bytes()
  File "D:\GitHub\cpython\lib\multiprocessing\connection.py", line 318, in 
_recv_bytes
return self._get_more_data(ov, maxsize)
  File "D:\GitHub\cpython\lib\multiprocessing\connection.py", line 337, in 
_get_more_data
assert left > 0
AssertionError
```


this assert looks invalid in this case because in C code the left value is 
DWORD (unsigned long), which cannot be negative by definition.

--
components: Library (Lib)
files: invalid_assert_bug_reproduce.py
messages: 324465
nosy: Oleksandr Buchkovskyi
priority: normal
severity: normal
status: open
title: invalid assert on big output of multiprocessing.Process
type: crash
versions: Python 3.6
Added file: https://bugs.python.org/file47778/invalid_assert_bug_reproduce.py

___
Python tracker 
<https://bugs.python.org/issue34563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34563] invalid assert on big output of multiprocessing.Process

2018-09-01 Thread Oleksandr Buchkovskyi 


Change by Oleksandr Buchkovskyi :


--
pull_requests: +8493

___
Python tracker 
<https://bugs.python.org/issue34563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34563] invalid assert on big output of multiprocessing.Process

2018-09-01 Thread Oleksandr Buchkovskyi 


Change by Oleksandr Buchkovskyi :


--
pull_requests: +8494

___
Python tracker 
<https://bugs.python.org/issue34563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34563] invalid assert on big output of multiprocessing.Process

2018-09-02 Thread Oleksandr Buchkovskyi 


Change by Oleksandr Buchkovskyi :


--
versions: +Python 3.7

___
Python tracker 
<https://bugs.python.org/issue34563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34563] invalid assert on big output of multiprocessing.Process

2018-09-02 Thread Oleksandr Buchkovskyi 


Oleksandr Buchkovskyi  added the comment:

By bigger than signed int i meant bigger than positive signed int

--

___
Python tracker 
<https://bugs.python.org/issue34563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34563] invalid assert on big output of multiprocessing.Process

2018-09-04 Thread Oleksandr Buchkovskyi 


Change by Oleksandr Buchkovskyi :


--
pull_requests: +8528

___
Python tracker 
<https://bugs.python.org/issue34563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue34563] invalid assert on big output of multiprocessing.Process

2018-09-07 Thread Oleksandr Buchkovskyi 


Oleksandr Buchkovskyi  added the comment:

thank you for all the help in merging this fix!

--

___
Python tracker 
<https://bugs.python.org/issue34563>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35152] too small type for struct.pack/unpack in mutliprocessing.Connection

2018-11-03 Thread Oleksandr Buchkovskyi 

New submission from Oleksandr Buchkovskyi :

the problem is reproduced on big multiprocessing.Process output
when the size of the output gets bigger than signed int a struct error is raised
```
python3 test.py
Process ForkPoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib64/python3.5/multiprocessing/pool.py", line 125, in worker
put((job, i, result))
  File "/usr/lib64/python3.5/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 200, in 
send_bytes
self._send_bytes(m[offset:offset + size])
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 393, in 
_send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.5/multiprocessing/process.py", line 252, in 
_bootstrap
self.run()
  File "/usr/lib64/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.5/multiprocessing/pool.py", line 130, in worker
put((job, i, (False, wrapped)))
  File "/usr/lib64/python3.5/multiprocessing/queues.py", line 347, in put
self._writer.send_bytes(obj)
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 200, in 
send_bytes
self._send_bytes(m[offset:offset + size])
  File "/usr/lib64/python3.5/multiprocessing/connection.py", line 393, in 
_send_bytes
header = struct.pack("!i", n)
struct.error: 'i' format requires -2147483648 <= number <= 2147483647
```

this error can be fixed by changing types that are used herŅƒ for 
struct.pack/unpack to !Q

--
components: Library (Lib)
files: error_on_bug_multiprocessing_output.py
messages: 329189
nosy: ahcub
priority: normal
severity: normal
status: open
title: too small type for struct.pack/unpack in mutliprocessing.Connection
type: crash
versions: Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: 
https://bugs.python.org/file47903/error_on_bug_multiprocessing_output.py

___
Python tracker 
<https://bugs.python.org/issue35152>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue35152] too small type for struct.pack/unpack in mutliprocessing.Connection

2018-11-03 Thread Oleksandr Buchkovskyi 


Change by Oleksandr Buchkovskyi :


--
keywords: +patch
pull_requests: +9613
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue35152>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17560] problem using multiprocessing with really big objects?

2018-11-05 Thread Oleksandr Buchkovskyi 


Change by Oleksandr Buchkovskyi :


--
keywords: +patch
pull_requests: +9646
stage:  -> patch review

___
Python tracker 
<https://bugs.python.org/issue17560>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com