[Numpy-discussion] ANN: SfePy 2016.3

[Robert Cimrman]

I am pleased to announce release 2016.3 of SfePy.

Description
---

SfePy (simple finite elements in Python) is a software for solving systems of
coupled partial differential equations by the finite element method or by the
isogeometric analysis (limited support). It is distributed under the new BSD
license.

Home page: http://sfepy.org
Mailing list: http://groups.google.com/group/sfepy-devel
Git (source) repository, issue tracker: http://github.com/sfepy/sfepy

Highlights of this release
--

- Python 3 support
- testing with Travis CI
- new classes for homogenized coefficients
- using argparse instead of optparse

For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1
(rather long and technical).

Cheers,
Robert Cimrman

---

Contributors to this release in alphabetical order:

Robert Cimrman
Jan Heczko
Thomas Kluyver
Vladimir Lukes
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Vendorize tempita

[Charles R Harris]

Hi All,

There is a PR  to vendorize
tempita. This removes tempita as a dependency and simplifies some things.
Feedback on this step is welcome. One question is whether the package
should be renamed to something like `npy_tempita`, as otherwise installed
tempita, if any has priority.

Thoughts?

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Stephan Hoyer]

One way to do this is to move to vendorized dependencies into an submodule
of numpy itself (e.g., sklearn.externals.joblib, though maybe even a little
more indirection than that would be valuable to make it clear that it isn't
part of NumPy public API). This would avoid further enlarging the set of
namespaces we use.

In any case, I'm perfectly OK with using something like npy_tempita
internally, too, as long as we can be sure that we're using NumPy's
vendorized version, not whatever version is installed locally. We're not
planning to actually install "npy_tempita" when installing numpy (even for
dev installs), right?


On Fri, Sep 30, 2016 at 7:30 AM, Charles R Harris  wrote:

> Hi All,
>
> There is a PR  to vendorize
> tempita. This removes tempita as a dependency and simplifies some things.
> Feedback on this step is welcome. One question is whether the package
> should be renamed to something like `npy_tempita`, as otherwise installed
> tempita, if any has priority.
>
> Thoughts?
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Benjamin Root]

This is the first I am hearing of tempita (looks to be a templating
language). How is it a dependency of numpy? Do I now need tempita in order
to use numpy, or is it a build-time-only dependency?

Ben

On Fri, Sep 30, 2016 at 11:13 AM, Stephan Hoyer  wrote:

> One way to do this is to move to vendorized dependencies into an submodule
> of numpy itself (e.g., sklearn.externals.joblib, though maybe even a little
> more indirection than that would be valuable to make it clear that it isn't
> part of NumPy public API). This would avoid further enlarging the set of
> namespaces we use.
>
> In any case, I'm perfectly OK with using something like npy_tempita
> internally, too, as long as we can be sure that we're using NumPy's
> vendorized version, not whatever version is installed locally. We're not
> planning to actually install "npy_tempita" when installing numpy (even for
> dev installs), right?
>
>
> On Fri, Sep 30, 2016 at 7:30 AM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>> Hi All,
>>
>> There is a PR  to vendorize
>> tempita. This removes tempita as a dependency and simplifies some things.
>> Feedback on this step is welcome. One question is whether the package
>> should be renamed to something like `npy_tempita`, as otherwise installed
>> tempita, if any has priority.
>>
>> Thoughts?
>>
>> Chuck
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Charles R Harris]

On Fri, Sep 30, 2016 at 9:13 AM, Stephan Hoyer  wrote:

> One way to do this is to move to vendorized dependencies into an submodule
> of numpy itself (e.g., sklearn.externals.joblib, though maybe even a little
> more indirection than that would be valuable to make it clear that it isn't
> part of NumPy public API). This would avoid further enlarging the set of
> namespaces we use.
>
> In any case, I'm perfectly OK with using something like npy_tempita
> internally, too, as long as we can be sure that we're using NumPy's
> vendorized version, not whatever version is installed locally. We're not
> planning to actually install "npy_tempita" when installing numpy (even for
> dev installs), right?
>
>
>
The only thing in the tools directory included in a source distribution is
the swig directory. Tempita is only currently used by the cythonize script
also in the tools directory. The search path for the cythonize script is 1)
installed modules, 2) modules in same directory, which is why it might be
good to rename the module npy_tempita` so that is always the one used.



Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Charles R Harris]

On Fri, Sep 30, 2016 at 9:21 AM, Benjamin Root  wrote:

> This is the first I am hearing of tempita (looks to be a templating
> language). How is it a dependency of numpy? Do I now need tempita in order
> to use numpy, or is it a build-time-only dependency?
>

Build time only. The virtue of tempita is that it can be used to generate
cython sources. We could adapt one of our current templating scripts to do
that also, but that would seem to be more work. Note that tempita is
currently included in cython, but the cython folks consider that an
implemention detail that should not be depended upon.



Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Evgeni Burovski]

On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris
 wrote:
>
>
> On Fri, Sep 30, 2016 at 9:21 AM, Benjamin Root  wrote:
>>
>> This is the first I am hearing of tempita (looks to be a templating
>> language). How is it a dependency of numpy? Do I now need tempita in order
>> to use numpy, or is it a build-time-only dependency?
>
>
> Build time only. The virtue of tempita is that it can be used to generate
> cython sources. We could adapt one of our current templating scripts to do
> that also, but that would seem to be more work. Note that tempita is
> currently included in cython, but the cython folks consider that an
> implemention detail that should not be depended upon.
>
> 
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>


Ideally, it's packaged in such a way that it's usable for scipy too --
at the moment it's used in scipy.sparse via Cython.Tempita + a
fallback to system installed tempita if Cython.Tempita is not
available (however I'm not sure that fallback is ever exercised).
Since scipy needs to support numpy down to 1.8.2, a vendorized copy
will not be usable for scipy for quite a while.

So, it'd be great to handle it like numpydoc: to have npy_tempita as a
small self-contained package with the repo under the numpy
organization and include it via a git submodule. Chuck, do you think
tempita would need much in terms of maintenance?

To put some money where my mouth is, I can offer to do some legwork
for packaging it up.

Cheers,

Evgeni
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Charles R Harris]

On Fri, Sep 30, 2016 at 9:48 AM, Evgeni Burovski  wrote:

> On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris
>  wrote:
> >
> >
> > On Fri, Sep 30, 2016 at 9:21 AM, Benjamin Root 
> wrote:
> >>
> >> This is the first I am hearing of tempita (looks to be a templating
> >> language). How is it a dependency of numpy? Do I now need tempita in
> order
> >> to use numpy, or is it a build-time-only dependency?
> >
> >
> > Build time only. The virtue of tempita is that it can be used to generate
> > cython sources. We could adapt one of our current templating scripts to
> do
> > that also, but that would seem to be more work. Note that tempita is
> > currently included in cython, but the cython folks consider that an
> > implemention detail that should not be depended upon.
> >
> > 
> >
> > Chuck
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
>
> Ideally, it's packaged in such a way that it's usable for scipy too --
> at the moment it's used in scipy.sparse via Cython.Tempita + a
> fallback to system installed tempita if Cython.Tempita is not
> available (however I'm not sure that fallback is ever exercised).
> Since scipy needs to support numpy down to 1.8.2, a vendorized copy
> will not be usable for scipy for quite a while.
>
> So, it'd be great to handle it like numpydoc: to have npy_tempita as a
> small self-contained package with the repo under the numpy
> organization and include it via a git submodule. Chuck, do you think
> tempita would need much in terms of maintenance?
>
> To put some money where my mouth is, I can offer to do some legwork
> for packaging it up.
>
>
It might be better to keep tempita and cythonize together so that the
search path works out right. It is also possible that other scripts might
be wanted as cythonize is currently restricted to cython files (*.pyx.in, *.
pxi.in). There are two other templating scripts in numpy/distutils, and I
think f2py has a dependency on one of those.

If there is a set of tools that would be common to both scipy and numpy,
having them included as a submodule would be a good idea.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Charles R Harris]

On Fri, Sep 30, 2016 at 10:10 AM, Charles R Harris <
charlesr.har...@gmail.com> wrote:

>
>
> On Fri, Sep 30, 2016 at 9:48 AM, Evgeni Burovski <
> evgeny.burovs...@gmail.com> wrote:
>
>> On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Fri, Sep 30, 2016 at 9:21 AM, Benjamin Root 
>> wrote:
>> >>
>> >> This is the first I am hearing of tempita (looks to be a templating
>> >> language). How is it a dependency of numpy? Do I now need tempita in
>> order
>> >> to use numpy, or is it a build-time-only dependency?
>> >
>> >
>> > Build time only. The virtue of tempita is that it can be used to
>> generate
>> > cython sources. We could adapt one of our current templating scripts to
>> do
>> > that also, but that would seem to be more work. Note that tempita is
>> > currently included in cython, but the cython folks consider that an
>> > implemention detail that should not be depended upon.
>> >
>> > 
>> >
>> > Chuck
>> >
>> > ___
>> > NumPy-Discussion mailing list
>> > NumPy-Discussion@scipy.org
>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>> >
>>
>>
>> Ideally, it's packaged in such a way that it's usable for scipy too --
>> at the moment it's used in scipy.sparse via Cython.Tempita + a
>> fallback to system installed tempita if Cython.Tempita is not
>> available (however I'm not sure that fallback is ever exercised).
>> Since scipy needs to support numpy down to 1.8.2, a vendorized copy
>> will not be usable for scipy for quite a while.
>>
>> So, it'd be great to handle it like numpydoc: to have npy_tempita as a
>> small self-contained package with the repo under the numpy
>> organization and include it via a git submodule. Chuck, do you think
>> tempita would need much in terms of maintenance?
>>
>> To put some money where my mouth is, I can offer to do some legwork
>> for packaging it up.
>>
>>
> It might be better to keep tempita and cythonize together so that the
> search path works out right. It is also possible that other scripts might
> be wanted as cythonize is currently restricted to cython files (*.pyx.in,
> *.pxi.in). There are two other templating scripts in numpy/distutils, and
> I think f2py has a dependency on one of those.
>
> If there is a set of tools that would be common to both scipy and numpy,
> having them included as a submodule would be a good idea.
>
>
Hmm, I suppose it just depends on where submodule is, so a npy_tempita
alone would work fine.  There isn't much maintenance needed if you resist
the urge to refactor the code. I removed a six dependency, but that is now
upstream as well.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] automatically avoiding temporary arrays

[Julian Taylor]

On 30.09.2016 23:09, josef.p...@gmail.com wrote:
> On Fri, Sep 30, 2016 at 9:38 AM, Julian Taylor
>  wrote:
>> hi,
>> Temporary arrays generated in expressions are expensive as the imply
>> extra memory bandwidth which is the bottleneck in most numpy operations.
>> For example:
>>
>> r = a + b + c
>>
>> creates the b + c temporary and then adds a to it.
>> This can be rewritten to be more efficient using inplace operations:
>>
>> r = b + c
>> r += a
> 
> general question (I wouldn't understand the details even if I looked.)
> 
> how is this affected by broadcasting and type promotion?
> 
> Some of the main reasons that I don't like to use inplace operation in
> general is that I'm often not sure when type promotion occurs and when
> arrays expand during broadcasting.
> 
> for example b + c is 1-D, a is 2-D, and r has the broadcasted shape.
> another case when I switch away from broadcasting is when b + c is int
> or bool and a is float. Thankfully, we get error messages for casting
> now.

the temporary is only avoided when the casting follows the safe rule, so
it should be the same as what you get without inplace operations. E.g.
float32-temporary + float64 will not be converted to the unsafe float32
+= float64 which a normal inplace operations would allow. But
float64-temp + float32 is transformed.

Currently the only broadcasting that will be transformed is temporary +
scalar value, otherwise it will only work on matching array sizes.
Though there is not really anything that prevents full broadcasting but
its not implemented yet in the PR.

> 
>>
>> This saves some memory bandwidth and can speedup the operation by 50%
>> for very large arrays or even more if the inplace operation allows it to
>> be completed completely in the cpu cache.
> 
> I didn't realize the difference can be so large. That would make
> streamlining some code worth the effort.
> 
> Josef
> 
> 
>>
>> The problem is that inplace operations are a lot less readable so they
>> are often only used in well optimized code. But due to pythons
>> refcounting semantics we can actually do some inplace conversions
>> transparently.
>> If an operand in python has a reference count of one it must be a
>> temporary so we can use it as the destination array. CPython itself does
>> this optimization for string concatenations.
>>
>> In numpy we have the issue that we can be called from the C-API directly
>> where the reference count may be one for other reasons.
>> To solve this we can check the backtrace until the python frame
>> evaluation function. If there are only numpy and python functions in
>> between that and our entry point we should be able to elide the temporary.
>>
>> This PR implements this:
>> https://github.com/numpy/numpy/pull/7997
>>
>> It currently only supports Linux with glibc (which has reliable
>> backtraces via unwinding) and maybe MacOS depending on how good their
>> backtrace is. On windows the backtrace APIs are different and I don't
>> know them but in theory it could also be done there.
>>
>> A problem is that checking the backtrace is quite expensive, so should
>> only be enabled when the involved arrays are large enough for it to be
>> worthwhile. In my testing this seems to be around 180-300KiB sized
>> arrays, basically where they start spilling out of the CPU L2 cache.
>>
>> I made a little crappy benchmark script to test this cutoff in this branch:
>> https://github.com/juliantaylor/numpy/tree/elide-bench
>>
>> If you are interested you can run it with:
>> python setup.py build_ext -j 4 --inplace
>> ipython --profile=null check.ipy
>>
>> At the end it will plot the ratio between elided and non-elided runtime.
>> It should get larger than one around 180KiB on most cpus.
>>
>> If no one points out some flaw in the approach, I'm hoping to get this
>> into the next numpy version.
>>
>> cheers,
>> Julian
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> 




signature.asc
Description: OpenPGP digital signature
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Charles R Harris]

On Fri, Sep 30, 2016 at 10:36 AM, Charles R Harris <
charlesr.har...@gmail.com> wrote:

>
>
> On Fri, Sep 30, 2016 at 10:10 AM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>>
>>
>> On Fri, Sep 30, 2016 at 9:48 AM, Evgeni Burovski <
>> evgeny.burovs...@gmail.com> wrote:
>>
>>> On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris
>>>  wrote:
>>> >
>>> >
>>> > On Fri, Sep 30, 2016 at 9:21 AM, Benjamin Root 
>>> wrote:
>>> >>
>>> >> This is the first I am hearing of tempita (looks to be a templating
>>> >> language). How is it a dependency of numpy? Do I now need tempita in
>>> order
>>> >> to use numpy, or is it a build-time-only dependency?
>>> >
>>> >
>>> > Build time only. The virtue of tempita is that it can be used to
>>> generate
>>> > cython sources. We could adapt one of our current templating scripts
>>> to do
>>> > that also, but that would seem to be more work. Note that tempita is
>>> > currently included in cython, but the cython folks consider that an
>>> > implemention detail that should not be depended upon.
>>> >
>>> > 
>>> >
>>> > Chuck
>>> >
>>> > ___
>>> > NumPy-Discussion mailing list
>>> > NumPy-Discussion@scipy.org
>>> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>> >
>>>
>>>
>>> Ideally, it's packaged in such a way that it's usable for scipy too --
>>> at the moment it's used in scipy.sparse via Cython.Tempita + a
>>> fallback to system installed tempita if Cython.Tempita is not
>>> available (however I'm not sure that fallback is ever exercised).
>>> Since scipy needs to support numpy down to 1.8.2, a vendorized copy
>>> will not be usable for scipy for quite a while.
>>>
>>> So, it'd be great to handle it like numpydoc: to have npy_tempita as a
>>> small self-contained package with the repo under the numpy
>>> organization and include it via a git submodule. Chuck, do you think
>>> tempita would need much in terms of maintenance?
>>>
>>> To put some money where my mouth is, I can offer to do some legwork
>>> for packaging it up.
>>>
>>>
>> It might be better to keep tempita and cythonize together so that the
>> search path works out right. It is also possible that other scripts might
>> be wanted as cythonize is currently restricted to cython files (*.pyx.in,
>> *.pxi.in). There are two other templating scripts in numpy/distutils,
>> and I think f2py has a dependency on one of those.
>>
>> If there is a set of tools that would be common to both scipy and numpy,
>> having them included as a submodule would be a good idea.
>>
>>
> Hmm, I suppose it just depends on where submodule is, so a npy_tempita
> alone would work fine.  There isn't much maintenance needed if you resist
> the urge to refactor the code. I removed a six dependency, but that is now
> upstream as well.
>

There don't seem to be any objections, so I will put the current
vendorization in. Evgeni, if you think it a good idea to make a repo for
this and use submodules, go ahead with that. I have left out the testing
infrastructure at https://github.com/gjhiggins/tempita which runs a sparse
set of doctests.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Vendorize tempita

[Ralf Gommers]

On Sat, Oct 1, 2016 at 1:42 PM, Charles R Harris 
wrote:

>
>
> On Fri, Sep 30, 2016 at 10:36 AM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
>
>>
>>
>> On Fri, Sep 30, 2016 at 10:10 AM, Charles R Harris <
>> charlesr.har...@gmail.com> wrote:
>>
>>>
>>>
>>> On Fri, Sep 30, 2016 at 9:48 AM, Evgeni Burovski <
>>> evgeny.burovs...@gmail.com> wrote:
>>>
 On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris
  wrote:
 >
 >
 > On Fri, Sep 30, 2016 at 9:21 AM, Benjamin Root 
 wrote:
 >>
 >> This is the first I am hearing of tempita (looks to be a templating
 >> language). How is it a dependency of numpy? Do I now need tempita in
 order
 >> to use numpy, or is it a build-time-only dependency?
 >
 >
 > Build time only. The virtue of tempita is that it can be used to
 generate
 > cython sources. We could adapt one of our current templating scripts
 to do
 > that also, but that would seem to be more work. Note that tempita is
 > currently included in cython, but the cython folks consider that an
 > implemention detail that should not be depended upon.
 >
 > 
 >
 > Chuck
 >
 > ___
 > NumPy-Discussion mailing list
 > NumPy-Discussion@scipy.org
 > https://mail.scipy.org/mailman/listinfo/numpy-discussion
 >


 Ideally, it's packaged in such a way that it's usable for scipy too --
 at the moment it's used in scipy.sparse via Cython.Tempita + a
 fallback to system installed tempita if Cython.Tempita is not
 available (however I'm not sure that fallback is ever exercised).
 Since scipy needs to support numpy down to 1.8.2, a vendorized copy
 will not be usable for scipy for quite a while.

 So, it'd be great to handle it like numpydoc: to have npy_tempita as a
 small self-contained package with the repo under the numpy
 organization and include it via a git submodule. Chuck, do you think
 tempita would need much in terms of maintenance?

 To put some money where my mouth is, I can offer to do some legwork
 for packaging it up.


>>> It might be better to keep tempita and cythonize together so that the
>>> search path works out right. It is also possible that other scripts might
>>> be wanted as cythonize is currently restricted to cython files (*.pyx.in,
>>> *.pxi.in). There are two other templating scripts in numpy/distutils,
>>> and I think f2py has a dependency on one of those.
>>>
>>> If there is a set of tools that would be common to both scipy and numpy,
>>> having them included as a submodule would be a good idea.
>>>
>>>
>> Hmm, I suppose it just depends on where submodule is, so a npy_tempita
>> alone would work fine.  There isn't much maintenance needed if you resist
>> the urge to refactor the code. I removed a six dependency, but that is now
>> upstream as well.
>>
>
> There don't seem to be any objections, so I will put the current
> vendorization in.
>

LGTM as is. tools/ seems to be the right place, its outside the numpy
package so no one can import it as numpy.something, which is better than a
numpy.externals or numpy.vendor submodule.


> Evgeni, if you think it a good idea to make a repo for this and use
> submodules, go ahead with that. I have left out the testing infrastructure
> at https://github.com/gjhiggins/tempita which runs a sparse set of
> doctests.
>

There's a problem with that: it will then not be possible to do:
git clone ...
python setup.py install  # or equivalent
We shouldn't force everyone to mess with "git submodule" for this. I
suspect a submodule would also break a pip install directly from github.
There'll be very few (if any) changes from upstream Tempita, so making a
reusable npy_tempita seems premature.

Cheers,
Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion