Re: [Numpy-discussion] State-of-the-art to use a C/C++ library from Python

[Neal Becker]

Jason Newton wrote:

> I just wanted to follow up on the C++ side of OP email - Cython has quite
> a
> few difficulties working with C++ code at the moment.  It's really more of
> a C solution most of the time and you must split things up into a mostly C
> call interface (that is the C code Cython can call) and limit
> exposure/complications with templates and  complex C++11+ constructs. 
> This may change in the longer term but in the near, that is the state.
> 
> I used to use Boost.Python but I'm getting my feet wet with Pybind (which
> is basically the same api but works more as you expect it to with it's
> signature/type plumbing  (including std::shared_ptr islanding), with some
> other C++11 based improvements, and is header only + submodule friendly!).
> I also remembered ndarray thanks to Neal's post but I haven't figured out
> how to leverage it better than pybind, at the moment.  I'd be interested
> to see ndarray gain support for pybind interoperability...
> 
> -Jason
> 
> On Wed, Aug 31, 2016 at 1:08 PM, David Morris  wrote:
> 
>> On Wed, Aug 31, 2016 at 2:28 PM, Michael Bieri  wrote:
>>
>>> Hi all
>>>
>>> There are several ways on how to use C/C++ code from Python with NumPy,
>>> as given in http://docs.scipy.org/doc/numpy/user/c-info.html .
>>> Furthermore, there's at least pybind11.
>>>
>>> I'm not quite sure which approach is state-of-the-art as of 2016. How
>>> would you do it if you had to make a C/C++ library available in Python
>>> right now?
>>>
>>> In my case, I have a C library with some scientific functions on
>>> matrices and vectors. You will typically call a few functions to
>>> configure the computation, then hand over some pointers to existing
>>> buffers containing vector data, then start the computation, and finally
>>> read back the data. The library also can use MPI to parallelize.
>>>
>>
>> I have been delighted with Cython for this purpose.  Great integration
>> with NumPy (you can access numpy arrays directly as C arrays), very
>> python like syntax and amazing performance.
>>
>> Good luck,
>>
>> David
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>

pybind11 looks very nice.  My problem is that the numpy API exposed by 
pybind11 is fairly weak at this point, as far as I can see from the docs.  
ndarray exposes a lot of functionality through the Array object, including 
convenient indexing and slicing.  AFAICT, the interface in pybind11 is 
pretty low level - just pointers.

There is also some functionality exposed by pybind11 using eigen.  
Personally, I find eigen rather baroque, and only use it when I see no 
alternative.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reading in a mesh file

[Florian Lindner]

Hello,

thanks for your reply which was really helpful!

My problem is that I discovered that the data I got is rather unordered.

The documentation for reshape says: Read the elements of a using this index 
order, and place the elements into the
reshaped array using this index order. ‘C’ means to read / write the elements 
using C-like index order, with the last
axis index changing fastest, back to the first axis index changing slowest. ‘F’ 
means to read / write the elements using
Fortran-like index order, with the first index changing fastest, and the last 
index changing slowest.

With my data both dimensions change, so there is no specific ordering of the 
points, just a bunch of arbitrarily mixed
"x y z value" data.

My idea is:

out = np.loadtxt(...)
x = np.unique(out[:,0])
y = np.unique[out]:,1])
xx, yy = np.meshgrid(x, y)

values = lookup(xx, yy, out)

lookup is ufunc (I hope that term is correct here) that looks up the value of 
every x and y in out, like
x_filtered = out[ out[:,0] == x, :]
y_filtered  = out[ out[:,1] == y, :]
return y_filtered[2]

(untested, just a sketch)

Would this work? Any better way?

Thanks,
Florian


Am 31.08.2016 um 17:06 schrieb Robert Kern:
> On Wed, Aug 31, 2016 at 4:00 PM, Florian Lindner  > wrote:
>>
>> Hello,
>>
>> I have mesh (more exactly: just a bunch of nodes) description with values 
>> associated to the nodes in a file, e.g. for a
>> 3x3 mesh:
>>
>> 0   0   10
>> 0   0.3 11
>> 0   0.6 12
>> 0.3 0   20
>> 0.3 0.3 21
>> 0.3 0.6 22
>> 0.6 0   30
>> 0.6 0.3 31
>> 0.6 0.6 32
>>
>> What is best way to read it in and get data structures like the ones I get 
>> from np.meshgrid?
>>
>> Of course, I know about np.loadtxt, but I'm having trouble getting the 
>> resulting arrays (x, y, values) in the right form
>> and to retain association to the values.
> 
> For this particular case (known shape and ordering), this is what I would do. 
> Maybe throw in a .T or three depending on
> exactly how you want them to be laid out.
> 
> [~/scratch]
> |1> !cat mesh.txt
> 
> 0   0   10
> 0   0.3 11
> 0   0.6 12
> 0.3 0   20
> 0.3 0.3 21
> 0.3 0.6 22
> 0.6 0   30
> 0.6 0.3 31
> 0.6 0.6 32
> 
> [~/scratch]
> |2> nodes = np.loadtxt('mesh.txt')
> 
> [~/scratch]
> |3> nodes
> array([[  0. ,   0. ,  10. ],
>[  0. ,   0.3,  11. ],
>[  0. ,   0.6,  12. ],
>[  0.3,   0. ,  20. ],
>[  0.3,   0.3,  21. ],
>[  0.3,   0.6,  22. ],
>[  0.6,   0. ,  30. ],
>[  0.6,   0.3,  31. ],
>[  0.6,   0.6,  32. ]])
> 
> [~/scratch]
> |4> reshaped = nodes.reshape((3, 3, -1))
> 
> [~/scratch]
> |5> reshaped
> array([[[  0. ,   0. ,  10. ],
> [  0. ,   0.3,  11. ],
> [  0. ,   0.6,  12. ]],
> 
>[[  0.3,   0. ,  20. ],
> [  0.3,   0.3,  21. ],
> [  0.3,   0.6,  22. ]],
> 
>[[  0.6,   0. ,  30. ],
> [  0.6,   0.3,  31. ],
> [  0.6,   0.6,  32. ]]])
> 
> [~/scratch]
> |7> x = reshaped[..., 0]
> 
> [~/scratch]
> |8> y = reshaped[..., 1]
> 
> [~/scratch]
> |9> values = reshaped[..., 2]
> 
> [~/scratch]
> |10> x
> array([[ 0. ,  0. ,  0. ],
>[ 0.3,  0.3,  0.3],
>[ 0.6,  0.6,  0.6]])
> 
> [~/scratch]
> |11> y
> array([[ 0. ,  0.3,  0.6],
>[ 0. ,  0.3,  0.6],
>[ 0. ,  0.3,  0.6]])
> 
> [~/scratch]
> |12> values
> array([[ 10.,  11.,  12.],
>[ 20.,  21.,  22.],
>[ 30.,  31.,  32.]])
> 
> --
> Robert Kern
> 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Reading in a mesh file

[Robert Kern]

On Thu, Sep 1, 2016 at 3:49 PM, Florian Lindner  wrote:
>
> Hello,
>
> thanks for your reply which was really helpful!
>
> My problem is that I discovered that the data I got is rather unordered.
>
> The documentation for reshape says: Read the elements of a using this
index order, and place the elements into the
> reshaped array using this index order. ‘C’ means to read / write the
elements using C-like index order, with the last
> axis index changing fastest, back to the first axis index changing
slowest. ‘F’ means to read / write the elements using
> Fortran-like index order, with the first index changing fastest, and the
last index changing slowest.
>
> With my data both dimensions change, so there is no specific ordering of
the points, just a bunch of arbitrarily mixed
> "x y z value" data.
>
> My idea is:
>
> out = np.loadtxt(...)
> x = np.unique(out[:,0])
> y = np.unique[out]:,1])
> xx, yy = np.meshgrid(x, y)
>
> values = lookup(xx, yy, out)
>
> lookup is ufunc (I hope that term is correct here) that looks up the
value of every x and y in out, like
> x_filtered = out[ out[:,0] == x, :]
> y_filtered  = out[ out[:,1] == y, :]
> return y_filtered[2]
>
> (untested, just a sketch)
>
> Would this work? Any better way?

If the (x, y) values are actually drawn from a rectilinear grid, then you
can use np.lexsort() to sort the rows before reshaping.

[~/scratch]
|4> !cat random-mesh.txt
0.3 0.3 21
0   0   10
0   0.3 11
0.3 0.6 22
0   0.6 12
0.6 0.3 31
0.3 0   20
0.6 0.6 32
0.6 0   30


[~/scratch]
|5> scrambled_nodes = np.loadtxt('random-mesh.txt')

# Note! Put the "faster" column before the "slower" column!
[~/scratch]
|6> i = np.lexsort([scrambled_nodes[:, 1], scrambled_nodes[:, 0]])

[~/scratch]
|7> sorted_nodes = scrambled_nodes[i]

[~/scratch]
|8> sorted_nodes
array([[  0. ,   0. ,  10. ],
   [  0. ,   0.3,  11. ],
   [  0. ,   0.6,  12. ],
   [  0.3,   0. ,  20. ],
   [  0.3,   0.3,  21. ],
   [  0.3,   0.6,  22. ],
   [  0.6,   0. ,  30. ],
   [  0.6,   0.3,  31. ],
   [  0.6,   0.6,  32. ]])


Then carry on with the reshape()ing as before. If the grid points that
"ought to be the same" are not actually identical, then you may end up with
some problems, e.g. if you had "0.3001 0.0 20.0" as a row, but all
of the other "x=0.3" rows had "0.3", then that row would get sorted out of
order. You would have to clean up the grid coordinates a bit first.

--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion