[issue40440] allow array.array construction from memoryview w/o copy

[Benjamin Keen]

New submission from Benjamin Keen :

Currently the array.array object can export a memoryview, but there is no way 
to construct one from a memoryview without making a copy of the underlying 
data.  So in that sense array.array only supports one half of the buffer 
protocol and this is to allow for the other half.

This proposal is to allow the array object to be constructed from an existing 
exported buffer without copying and reallocating the memory, permitting 
operations that can modify the underlying buffer's contents but not the 
allocation.

This is useful when working with many small pieces of one very large underlying 
buffer that you do not want to copy, when desiring to work with different parts 
of it with different types, and as part of a way to work with shared memory in 
multiple processes.

I will shortly have a PR for this, including updates for the documentation and 
unit tests.

 - Modules/arraymodule.c already must check if the array object has exported a 
buffer for methods that might resize. If the array was constructed from an 
imported buffer, the same restrictions apply.  So the object just needs to know 
whether it is constructed from a Py_Buffer or not and check in the same places 
it checks for the export count being nonzero. So the code doesn't need to be 
perturbed that much.

- Only MemoryView objects with contiguous layout, size, and alignment 
compatible with the data type of the array element are allowed.

- I'm proposing this is only for when it's an actual memoryview object, not 
just if the object can export buffers. This preserves more of the existing 
behavior.

- Currently you /can/ initialize an array with a type-compatible memoryview - 
but it makes a copy, iterating the elements and the types have to match, not 
just in size. We could maintain exact backward compatibility by adding an extra 
argument to array.array() or another letter to the format specifier; my current 
patch doesn't do this though.

---
Example of current behavior:

>>> import array
>>> x = array.array('b', [1,2,3,4])
>>> y = memoryview(x)
>>> z = array.array('b', y)
>>> z
array('b', [1, 2, 3, 4])
>>> z[0] = 42
>>> x
array('b', [1, 2, 3, 4])
>>> z
array('b', [42, 2, 3, 4])
 # x and z are backed by different memory
>>> x.append(17)
Traceback (most recent call last):
  File "", line 1, in 
BufferError: cannot resize an array that is exporting buffers
 # this is because y is still a live object
>>> z.append(17)
 # it is really a copy, x and y are irrelevant to z
>>> z
array('b', [42, 2, 3, 4, 17])


Example of new behavior:

>>> import array
>>> x = array.array('b', [1,2,3,4])
>>> x
array('b', [1, 2, 3, 4])
>>> y = memoryview(x)
>>> z = array.array('b', y)
>>> z
array('b', [1, 2, 3, 4])
>>> z[0] = 42
>>> x
array('b', [42, 2, 3, 4])
>>> x.append(4)
Traceback (most recent call last):
  File "", line 1, in 
BufferError: cannot resize an array that is exporting buffers
>>> z.append(4)
Traceback (most recent call last):
  File "", line 1, in 
BufferError: cannot resize an array constructed from an imported buffer

--
components: Library (Lib)
messages: 367688
nosy: bjkeen
priority: normal
severity: normal
status: open
title: allow array.array construction from memoryview w/o copy
type: enhancement

___
Python tracker 
<https://bugs.python.org/issue40440>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40440] allow array.array construction from memoryview w/o copy

[Benjamin Keen]

Change by Benjamin Keen :


--
keywords: +patch
pull_requests: +19121
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/19800

___
Python tracker 
<https://bugs.python.org/issue40440>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40440] allow array.array construction from memoryview w/o copy

[Benjamin Keen]

Benjamin Keen  added the comment:

memoryview has a lot of overlap with array, but there are still useful methods 
(index and count for instance) that memoryview does not have. I don't see a 
workaround that will run with equivalent speed without writing some extension 
or adding them to memoryview.

Constructing an array from the memoryview when one wants these isn't a 
workaround because there may not be enough memory to make a copy. For instance 
- a memoryview of a mapped disk file that is much larger than the physical 
memory in the machine.

When writing functions that use an array you don't always know ahead of time 
whether you are going to be in this situation.  The functions may come from 
someone else with a bigger machine. This lets the client of the function decide 
what the right thing to do is as needed without changing the function itself.

So for that reason this brings something that just using the memoryview 
directly won't provide.  There's value in being able to write things that use 
the array interface consistently.

You could also think of this as a way of providing compiled-speed index() and 
count() on certain memoryviews without needing to add new code for that to 
memoryview itself.

--

___
Python tracker 
<https://bugs.python.org/issue40440>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com