[Cython] Bug - PyPy tuple leak in __Pyx_PyObject_CallOneArg?

[Jason Madden]

Hello,

I'm one of the maintainers of the gevent concurrency library, which has 
recently been ported to run on PyPy. Under PyPy, a small portion of the code is 
compiled with Cython in order to get the desired atomic semantics 
(specifically, a semaphore class). We recently had a user report an easily 
reproducible leak of tuples of one element. Tracking it down, it appears that 
__Pyx_PyObject_CallOneArg creates a new tuple under PyPy, but neglects to free 
it. This was tested with Cython 0.23.3 and PyPy 2.6.1.

Our Cython code contained a loop like this, and every iteration of the loop 
leaked a tuple:

for link in links:
link(self)

The C output for that last line looked like this:

__Pyx_PyObject_CallOneArg(__pyx_t_10, ((PyObject *)__pyx_v_self));...

And, under PyPy, the implementation of __Pyx_PyObject_CallOneArg is different 
than it is under CPython. Specifically, with Cython 0.23.3 this is the code:

#if CYTHON_COMPILING_IN_CPYTHON
...
#else
static CYTHON_INLINE PyObject* __Pyx_PyObject_CallOneArg(PyObject *func, 
PyObject *arg) {
PyObject* args = PyTuple_Pack(1, arg);
return (likely(args)) ? __Pyx_PyObject_Call(func, args, NULL) : NULL;
}
#endif

PyTuple_Pack is documented as returning a new reference 
(https://docs.python.org/2/c-api/tuple.html#c.PyTuple_Pack), so it seems to me 
like this code should be decrementing the reference to the tuple before 
returning (much like the __Pyx__PyObject_CallOneArg function does under 
CPython).

Changing the loop to avoid the use of CallOneArg seemed to resolve the tuple 
leak, presumably at some execution time cost:

   args = (self,)
   for link in links:
   link(*self)

There's some additional background at 
https://bitbucket.org/pypy/pypy/issues/2149/memory-leak-for-python-subclass-of-cpyext#comment-22347393

Am I interpreting this correctly to be a bug, or could there be something wrong 
in the way we're handling callbacks? Please let me know if there's any further 
information I can provide.

Thanks,
Jason
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bug - PyPy tuple leak in __Pyx_PyObject_CallOneArg?

[Stefan Behnel]

Jason Madden schrieb am 09.10.2015 um 18:04:
> We recently had a user report an easily reproducible leak of tuples of
> one element. Tracking it down, it appears that __Pyx_PyObject_CallOneArg
> creates a new tuple under PyPy, but neglects to free it. This was tested
> with Cython 0.23.3 and PyPy 2.6.1.

Thanks for the report and the fix.

https://github.com/cython/cython/pull/450

Stefan
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

[Cython] Cython 0.23.4 released

[Stefan Behnel]

Hi everyone,

here is a minor bug-fix release for the 0.23 release series that repairs
some problems and regressions.

https://pypi.python.org/pypi/Cython/0.23.4

Complete changelog follows below.


You can get the signed release from here:

http://cython.org/

http://cython.org/release/Cython-0.23.4.tar.gz

http://cython.org/release/Cython-0.23.4.zip


SHA1 sums:
fc574c5050cd5a8e34435432e2a4a693353ed807  Cython-0.23.4.tar.gz
ad477040a22190b5506a3d8ddde646b7586d5703  Cython-0.23.4.zip

Have fun,

Stefan


0.23.4 (2015-10-10)
===

Bugs fixed
--

* Memory leak when calling Python functions in PyPy.

* Compilation problem with MSVC in C99-ish mode.

* Warning about unused values in a helper macro.
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython coverage and multiple projects

[Stefan Behnel]

Masood Malekghassemi via cython-devel schrieb am 08.10.2015 um 01:06:
> I'm having trouble with getting Cython to play nicely with the coverage.py
> tool via the Cython.Coverage plug-in.
> 
> We have tests that are running from a different setup.py project than the
> one which contains the Cython files (the projects respectively being for
> the duration of this discussion the 'tests' project and the 'source'
> project). The generated coverage file contains the line data for individual
> .pyx files, but the paths of the .pyx files themselves *within* the
> .coverage file appear to be rooted in the tests project rather than the
> site-packages folder in which the source project's files were installed
> (i.e. paths that never exist throughout the edit-build-test cycle).

This seems to be a rather unusual setup. The "expected" setting is that
tests live somewhere within the source checkout and are run on the
non-installed sources, usually as part of the CI build process, sometimes
even just as part of a separate reporting process. Why do you want to
create coverage reports after the installation?


> A secondary issue: the .pyx files are never copied over to the installation
> directory, so even if the paths were 'correct' (with respect to being
> copied into the site-packages folder or being accessible via developer mode
> pip-installs), they wouldn't be found.

Sure, that's a difference from .py/.pyc files which are getting executed
directly. For Cython code, there is an additional compile/build step that
generates the executable code from the sources, which are then no longer
needed. That's another reason why support for coverage reporting after the
installation seems foreign here and wouldn't be of much use.

So, my suggestion would be to run the coverage reports within the source
tree after building it, when all source files are readily available. But
I'm obviously missing a lot of information about your actual project setup
and the reasons behind what you are doing here.

Stefan

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

[Cython] git master fails to compile scipy

[Julian Taylor via cython-devel]

hi,
since ab78f93b3ffa88183a0d2aae6b692e394c51f860 scipy does not build anymore:
the failing file is:
https://github.com/scipy/scipy/blob/master/scipy/sparse/csgraph/_reordering.pyx

error:
Traceback (most recent call last):
  File "/usr/bin/cython", line 8, in 
main(command_line = 1)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 704, in main
result = compile(sources, options)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 679, in compile
return compile_multiple(source, options)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 657, in compile_multiple
result = run_pipeline(source, options, context=context)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
line 487, in run_pipeline
err, enddata = Pipeline.run_pipeline(pipeline, source)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py",
line 365, in run_pipeline
data = phase(data)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py",
line 53, in generate_pyx_code_stage
module_node.process_implementation(options, result)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
line 118, in process_implementation
self.generate_c_code(env, options, result)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
line 349, in generate_c_code
self.body.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 436, in generate_function_definitions
stat.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 436, in generate_function_definitions
stat.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/FusedNode.py",
line 785, in generate_function_definitions
stat.generate_function_definitions(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 3068, in generate_function_definitions
FuncDefNode.generate_function_definitions(self, env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 1935, in generate_function_definitions
self.generate_function_body(env, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 1691, in generate_function_body
self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 442, in generate_execution_code
stat.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/UtilNodes.py",
line 321, in generate_execution_code
self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 6419, in generate_execution_code
self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 442, in generate_execution_code
stat.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 5865, in generate_execution_code
if_clause.generate_execution_code(code, end_label, is_last=i == last)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 5908, in generate_execution_code
self.body.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 442, in generate_execution_code
stat.generate_execution_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 4800, in generate_execution_code
self.generate_assignment_code(code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
line 5091, in generate_assignment_code
self.lhs.generate_assignment_code(self.rhs, code)
  File
"/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ExprNodes.py",
line 3897, in generate_assignment_code
self.generate_

Re: [Cython] git master fails to compile scipy

[Robert Bradshaw]

Thanks for the report. This should be fixed with
https://github.com/cython/cython/commit/eef2bc650945d69158e888653a2f8bd6bbad3db3

On Sat, Oct 10, 2015 at 2:11 PM, Julian Taylor via cython-devel
 wrote:
> hi,
> since ab78f93b3ffa88183a0d2aae6b692e394c51f860 scipy does not build anymore:
> the failing file is:
> https://github.com/scipy/scipy/blob/master/scipy/sparse/csgraph/_reordering.pyx
>
> error:
> Traceback (most recent call last):
>   File "/usr/bin/cython", line 8, in 
> main(command_line = 1)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
> line 704, in main
> result = compile(sources, options)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
> line 679, in compile
> return compile_multiple(source, options)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
> line 657, in compile_multiple
> result = run_pipeline(source, options, context=context)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Main.py",
> line 487, in run_pipeline
> err, enddata = Pipeline.run_pipeline(pipeline, source)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py",
> line 365, in run_pipeline
> data = phase(data)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Pipeline.py",
> line 53, in generate_pyx_code_stage
> module_node.process_implementation(options, result)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
> line 118, in process_implementation
> self.generate_c_code(env, options, result)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/ModuleNode.py",
> line 349, in generate_c_code
> self.body.generate_function_definitions(env, code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 436, in generate_function_definitions
> stat.generate_function_definitions(env, code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 436, in generate_function_definitions
> stat.generate_function_definitions(env, code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/FusedNode.py",
> line 785, in generate_function_definitions
> stat.generate_function_definitions(env, code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 3068, in generate_function_definitions
> FuncDefNode.generate_function_definitions(self, env, code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 1935, in generate_function_definitions
> self.generate_function_body(env, code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 1691, in generate_function_body
> self.body.generate_execution_code(code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 442, in generate_execution_code
> stat.generate_execution_code(code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/UtilNodes.py",
> line 321, in generate_execution_code
> self.body.generate_execution_code(code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 6419, in generate_execution_code
> self.body.generate_execution_code(code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 442, in generate_execution_code
> stat.generate_execution_code(code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 5865, in generate_execution_code
> if_clause.generate_execution_code(code, end_label, is_last=i == last)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 5908, in generate_execution_code
> self.body.generate_execution_code(code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 442, in generate_execution_code
> stat.generate_execution_code(code)
>   File
> "/tmp/local/lib/python2.7/site-packages/Cython-0.24.0a0-py2.7-linux-x86_64.egg/Cython/Compiler/Nodes.py",
> line 4800, in generate_execution_code
> self.generate_assign