convert triply-nested array to void pointer using ctypes
I'd like to send an array containing arrays of 2-element float arrays to a
foreign function, as the following struct:
class _FFIArray(Structure):
_fields_ = [("data", c_void_p),
("len", c_size_t)]
@classmethod
def from_param(cls, seq):
""" Allow implicit conversions """
return seq if isinstance(seq, cls) else cls(seq)
def __init__(self, seq):
self.data = cast(
np.array(seq, dtype=np.float64).ctypes.data_as(POINTER(c_double)),
c_void_p
)
self.len = len(seq)
This works for data such as [[1.0, 2.0], [3.0, 4.0], [5.0, 6.0], … ], (.shape
is n, 2 where n is an arbitrary length) but I'd like it to work for data such as
[
[[1.0, 2.0], [3.0, 4.0], [5.0, 6.0], … ],
[[7.0, 8.0], [9.0, 10.0], [11.0, 12.0], … ],
…
]
which has .shape p, q, 2 where p and q are of arbitrary length.
Is this possible?
--
https://mail.python.org/mailman/listinfo/python-list
Re: NumPy frombuffer giving nonsense values when reading C float array on Windows
On Tuesday, 26 July 2016 15:21:14 UTC+1, Peter Otten wrote: > > > I'm using ctypes to interface with a binary which returns a void pointer > > (ctypes c_void_p) to a nested 64-bit float array: > > [[1.0, 2.0], [3.0, 4.0], … ] > > then return the pointer so it can be freed > > > > I'm using the following code to de-reference it: > > > > # a 10-element array > > shape = (10, 2) > > array_size = np.prod(shape) > > mem_size = 8 * array_size > > array_str = ctypes.string_at(ptr, mem_size) > > # convert to NumPy array,and copy to a list > > ls = np.frombuffer(array_str, dtype="float64", > > count=array_size).reshape(shape).tolist() > > # return pointer so it can be freed > > drop_array(ptr) > > return ls > > > > This works correctly and consistently on Linux and OSX using NumPy 1.11.0, > > but fails on Windows 32 bit and 64-bit about 50% of the time, returning > > nonsense values. Am I doing something wrong? Is there a better way to do > > this? > > I'd verify that the underlying memory has not been freed by the "binary" > when you are doing the ctypes/numpy processing. You might get the correct > values only when you are "lucky" and the memory has not yet been reused for > something else, and you are "lucky" on Linux/OSX more often than on > Windows... I'm pretty sure the binary isn't freeing the memory prematurely; I wrote it and I'm testing it, and the Python tests run 10^6 loops of: array retrieval -> numpy array allocation + copying to list -> passing original array back to be freed. I'm not completely ruling it out (it's difficult to test a .dylib / .so using valgrind), but getting the ctypes / numpy side right would at least allow me to eliminate one potential source of problems. -- https://mail.python.org/mailman/listinfo/python-list
Re: NumPy frombuffer giving nonsense values when reading C float array on Windows
On Tuesday, 26 July 2016 16:36:33 UTC+1, Christian Gollwitzer wrote: > Am 26.07.16 um 17:09 schrieb sth: > > it's difficult to test a .dylib / .so using valgrind > > Why is it difficult? If you have a python script such that > > python mytests.py > > loads the .so and runs the tests, then > > valgrind --tool=memcheck python mytests.py > > should work. This should immediately spit out an error in case that > numpy accesses deleted memory. > > Of course debug information should be enabled when compiling the .so for > more useful output, and it could be helpful to have it for numpy/python, > too, but that's not a requirement. > > Christian Valgrind isn't giving me any errors; it's reporting possibly-lost memory, but this is constant -- if I call the foreign functions once or 10^6 times, the number of bytes it reports possibly lost (and the number of bytes it reports still reachable) remain the same. -- https://mail.python.org/mailman/listinfo/python-list
Re: NumPy frombuffer giving nonsense values when reading C float array on Windows
On Tuesday, 26 July 2016 19:10:46 UTC+1, eryk sun wrote: > On Tue, Jul 26, 2016 at 12:06 PM, sth wrote: > > I'm using ctypes to interface with a binary which returns a void pointer > > (ctypes c_void_p) to a nested 64-bit float array: > > If this comes from a function result, are you certain that its restype > is ctypes.c_void_p? I commonly see typos here such as setting > "restypes" instead of "restype". > > > [[1.0, 2.0], [3.0, 4.0], … ] > > then return the pointer so it can be freed > > > > I'm using the following code to de-reference it: > > > > # a 10-element array > > shape = (10, 2) > > array_size = np.prod(shape) > > mem_size = 8 * array_size > > array_str = ctypes.string_at(ptr, mem_size) > > # convert to NumPy array,and copy to a list > > ls = np.frombuffer(array_str, dtype="float64", > > count=array_size).reshape(shape).tolist() > > # return pointer so it can be freed > > drop_array(ptr) > > return ls > > > > This works correctly and consistently on Linux and OSX using NumPy 1.11.0, > > but fails on > > Windows 32 bit and 64-bit about 50% of the time, returning nonsense values. > > Am I doing > > something wrong? Is there a better way to do this? > > numpy.ctypeslib facilitates working with ctypes functions, pointers > and arrays via the factory functions as_array, as_ctypes, and > ndpointer. > > ndpointer creates a c_void_p subclass that overrides the default > from_param method to allow passing arrays as arguments to ctypes > functions and also implements the _check_retval_ hook to automatically > convert a pointer result to a numpy array. > > The from_param method validates an array argument to ensure it has the > proper data type, shape, and memory layout. For example: > > g = ctypes.CDLL(None) # Unix only > Base = np.ctypeslib.ndpointer(dtype='B', shape=(4,)) > > # strchr example > g.strchr.argtypes = (Base, ctypes.c_char) > g.strchr.restype = ctypes.c_char_p > > d = np.array(list(b'012\0'), dtype='B') > e = np.array(list(b'0123\0'), dtype='B') # wrong shape > > >>> g.strchr(d, b'0'[0]) > b'012' > >>> g.strchr(e, b'0'[0]) > Traceback (most recent call last): > File "", line 1, in > ctypes.ArgumentError: argument 1: : > array must have shape (4,) > > The _check_retval_ hook of an ndpointer calls numpy.array on the > result of a function. Its __array_interface__ property is used to > create a copy with the defined data type and shape. For example: > > g.strchr.restype = Base > > >>> d.ctypes._as_parameter_ # source address > c_void_p(24657952) > >>> a = g.strchr(d, b'0'[0]) > >>> a > array([48, 49, 50, 0], dtype=uint8) > >>> a.ctypes._as_parameter_ # it's a copy > c_void_p(19303504) > > As a copy, the array owns its data: > > >>> a.flags > C_CONTIGUOUS : True > F_CONTIGUOUS : True > OWNDATA : True > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False > > You can subclass the ndpointer type to have _check_retval_ instead > return a view of the result (i.e. copy=False), which may be desirable > for a large result array but probably isn't worth it for small arrays. > For example: > > class Result(Base): > @classmethod > def _check_retval_(cls, result): > return np.array(result, copy=False) > > g.strchr.restype = Result > > >>> a = g.strchr(d, b'0'[0]) > >>> a.ctypes._as_parameter_ # it's NOT a copy > c_void_p(24657952) > > Because it's not a copy, the array view doesn't own the data, but note > that it's not a read-only view: > > >>> a.flags > C_CONTIGUOUS : True > F_CONTIGUOUS : True > OWNDATA : False > WRITEABLE : True > ALIGNED : True > UPDATEIFCOPY : False The restype is a ctypes Structure instance with a single __fields__ entry (coords), which is a Structure with two fields (len and data) which are the FFI array's length and the void pointer to its memory: https://github.com/urschrei/pypolyline/blob/master/pypolyline/util.py#L109-L117 I'm only half-following your explanation of how ctypeslib works, but it seems clear that I'm doing something wrong. -- https://mail.python.org/mailman/listinfo/python-list
need help removing 'dev-r0' part from a python package
My package 'webpreview'[https://github.com/ludbek/webpreview] has new version '1.0.3'. I used 'sdist' to bundle it. Unfortunately it names it 'webpreview-1.0.3dev-r0.tar.gz' instead of 'webpreview-1.0.3.tar.gz' making it unsuitable to upload to pypi. What is causing sdist to append 'dev' stuff in the package. The source can be available in the link above. Thanx for you help. -- https://mail.python.org/mailman/listinfo/python-list
