[Cython] Control flow graph

[Vitja Makarov]

Hi!

In order to implement "reaching definitions" algorithm.
I'm now working on control-flow (or data-flow) graph.

Here is funny picture made with graphviz ;)

http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control flow graph

[Vitja Makarov]

2011/2/15 Vitja Makarov :
> 2011/2/15 Robert Bradshaw :
>> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov  
>> wrote:
>>> Hi!
>>>
>>> In order to implement "reaching definitions" algorithm.
>>> I'm now working on control-flow (or data-flow) graph.
>>>
>>> Here is funny picture made with graphviz ;)
>>>
>>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/
>>
>> Cool. Any plans on handling exceptions?
>>
>

Ooops forget to CC cython-devel...

Sure, but I don't have much time for this :(

Linear block inside try...except body should be split by assignments
and each subblock should point to exception handling entry point.

As result I want to have set of possible assignments for each NameNode position.

So handling of uninitialized variables, unused, unused results should be easy,
later it may help to implement local variable deletion. And guess that
could help type inference.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control flow graph

[Vitja Makarov]

2011/2/15 Stefan Behnel :
> Robert Bradshaw, 15.02.2011 08:21:
>>
>> On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote:
>>>
>>> 2011/2/15 Robert Bradshaw:
>>>>
>>>> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote:
>>>>>
>>>>> Hi!
>>>>>
>>>>> In order to implement "reaching definitions" algorithm.
>>>>> I'm now working on control-flow (or data-flow) graph.
>>>>>
>>>>> Here is funny picture made with graphviz ;)
>>>>>
>>>>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/
>>>>
>>>> Cool. Any plans on handling exceptions?
>>>
>>> Sure, but I don't have much time for this :(
>>>
>>> Linear block inside try...except body should be split by assignments
>>> and each subblock should point to exception handling entry point.
>>
>> Would every possible failing sub-expression have to point to the
>> exception handling point(s)?
>

Not sure here as now graph node is list of NameNode ref and AssignmentNode.
I'm not sure but it seems that every sub-expression could raise
exception now, is there any flag?

> Well, in most cases (especially the interesting ones), this will be the
> function exit point, so it'll be easy. And in some cases, we may be able to
> infer that a specific exception that an expression (e.g. arithmetics or a
> 'raise' statement) can raise will not get caught by a given except clause
> (although that's certainly a tricky optimisation).
>
> But in general, I think any subexpression that potentially raises an
> exception must point to the next exception handling point.
>

Right, and each exception handling point should have reference to the
next one or function exit point.

>
>> I suppose it depends on whether you'll be handling more than assignment
>> tracking.
>
> We *may* get away with a statement-level graph in that case, but I somehow
> doubt it already. For example, list comprehensions leak their variable in
> Py2 code, so it's important to know if they are executed or not, and they
> may appear in any kind of expression.
>

This should be special case for scoped expression node.

Now I build graph "from scratch" it include only name node references
and assignments, and positions marks to draw gv.

May be node tree should be translated into graph and then local
variables graph could be created from it.

On the other hand CreateAbstractGraph transformation could be used to
create specialized graph.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control flow graph

[Vitja Makarov]

2011/2/15 Stefan Behnel :
> Robert Bradshaw, 15.02.2011 08:21:
>>
>> On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote:
>>>
>>> 2011/2/15 Robert Bradshaw:
>>>>
>>>> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote:
>>>>>
>>>>> Hi!
>>>>>
>>>>> In order to implement "reaching definitions" algorithm.
>>>>> I'm now working on control-flow (or data-flow) graph.
>>>>>
>>>>> Here is funny picture made with graphviz ;)
>>>>>
>>>>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/
>>>>
>>>> Cool. Any plans on handling exceptions?
>>>
>>> Sure, but I don't have much time for this :(
>>>
>>> Linear block inside try...except body should be split by assignments
>>> and each subblock should point to exception handling entry point.
>>
>> Would every possible failing sub-expression have to point to the
>> exception handling point(s)?
>
> Well, in most cases (especially the interesting ones), this will be the
> function exit point, so it'll be easy. And in some cases, we may be able to
> infer that a specific exception that an expression (e.g. arithmetics or a
> 'raise' statement) can raise will not get caught by a given except clause
> (although that's certainly a tricky optimisation).
>
> But in general, I think any subexpression that potentially raises an
> exception must point to the next exception handling point.
>
>
>> I suppose it depends on whether you'll be handling more than assignment
>> tracking.
>
> We *may* get away with a statement-level graph in that case, but I somehow
> doubt it already. For example, list comprehensions leak their variable in
> Py2 code, so it's important to know if they are executed or not, and they
> may appear in any kind of expression.
>

Hmm... both python and codespeaks in the thread

Here is my commit it's mostly broken now but anyway
https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control flow graph

[Vitja Makarov]

2011/2/16 Vitja Makarov :
> 2011/2/15 Stefan Behnel :
>> Robert Bradshaw, 15.02.2011 08:21:
>>>
>>> On Mon, Feb 14, 2011 at 9:49 PM, Vitja Makarov wrote:
>>>>
>>>> 2011/2/15 Robert Bradshaw:
>>>>>
>>>>> On Sun, Feb 13, 2011 at 11:40 PM, Vitja Makarov wrote:
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> In order to implement "reaching definitions" algorithm.
>>>>>> I'm now working on control-flow (or data-flow) graph.
>>>>>>
>>>>>> Here is funny picture made with graphviz ;)
>>>>>>
>>>>>> http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/
>>>>>
>>>>> Cool. Any plans on handling exceptions?
>>>>
>>>> Sure, but I don't have much time for this :(
>>>>
>>>> Linear block inside try...except body should be split by assignments
>>>> and each subblock should point to exception handling entry point.
>>>
>>> Would every possible failing sub-expression have to point to the
>>> exception handling point(s)?
>>
>> Well, in most cases (especially the interesting ones), this will be the
>> function exit point, so it'll be easy. And in some cases, we may be able to
>> infer that a specific exception that an expression (e.g. arithmetics or a
>> 'raise' statement) can raise will not get caught by a given except clause
>> (although that's certainly a tricky optimisation).
>>
>> But in general, I think any subexpression that potentially raises an
>> exception must point to the next exception handling point.
>>
>>
>>> I suppose it depends on whether you'll be handling more than assignment
>>> tracking.
>>
>> We *may* get away with a statement-level graph in that case, but I somehow
>> doubt it already. For example, list comprehensions leak their variable in
>> Py2 code, so it's important to know if they are executed or not, and they
>> may appear in any kind of expression.
>>
>
> Hmm... both python and codespeaks in the thread
>
> Here is my commit it's mostly broken now but anyway
> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a
>
>

I've update stuff:
 - algo for finding definitions
 - warnings for uninitialized and may be uninitialised use
 - few test cases

Trying to compile ParseTreeTransforms.py I've found this for example:

warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable
'template' may be used uninitialized

def create_Property(self, entry):
if entry.visibility == 'public':
if entry.type.is_pyobject:
template = self.basic_pyobject_property
else:
template = self.basic_property
elif entry.visibility == 'readonly':
template = self.basic_property_ro
property = template.substitute({
u"ATTR": ExprNodes.AttributeNode(pos=entry.pos,

obj=ExprNodes.NameNode(pos=entry.pos, name="self"),
 attribute=entry.name),
}, pos=entry.pos).stats[0]



-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control flow graph

[Vitja Makarov]

2011/2/22 Stefan Behnel :
> Vitja Makarov, 20.02.2011 18:23:
>>
>> 2011/2/16 Vitja Makarov:
>>>
>>> Hmm... both python and codespeaks in the thread
>
> Yes, we should keep it to cython-devel only. Sorry for mixing it up.
>
>
>>> Here is my commit it's mostly broken now but anyway
>>>
>>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a
>
> Flow control support is large enough to merit its own module. Not sure how
> 'smart' git is here, but you can always keep the history by explicitly
> copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated
> sections from both files.
>

Ok.

> You are duplicating some code from the type inferencer. We might want to
> clean that up at some point. However, given that flow control analysis will
> allow us to improve the type inferencer, I think it's best to keep this code
> in the FCA part.
>

Yes, I think it could replace MarkAssignments transform later.
Unreachable code could be delete there too.

>
>> I've update stuff:
>>  - algo for finding definitions
>>  - warnings for uninitialized and may be uninitialised use
>>  - few test cases
>
> That looks very nice so far. Any idea how well it scales?
>

"Usually iterative algorithm takes no more then 5 iterations"

For ExprNodes.py max number is 15 while avg is about 3

About execution time:

ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325,
~10% slow down.
-O flag could be introduced but I don't think that's a good idea.

Should later try to execute cython compiled code.

>
>> Trying to compile ParseTreeTransforms.py I've found this for example:
>>
>> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable
>> 'template' may be used uninitialized
>>
>>     def create_Property(self, entry):
>>         if entry.visibility == 'public':
>>             if entry.type.is_pyobject:
>>                 template = self.basic_pyobject_property
>>             else:
>>                 template = self.basic_property
>>         elif entry.visibility == 'readonly':
>>             template = self.basic_property_ro
>>         property = template.substitute({
>>                 u"ATTR": ExprNodes.AttributeNode(pos=entry.pos,
>>
>> obj=ExprNodes.NameNode(pos=entry.pos, name="self"),
>>                                                  attribute=entry.name),
>>             }, pos=entry.pos).stats[0]
>
> Ok, I guess that code generally works, but it's better to get rid of the
> code smell.
>

Might be used warning should be disabled by default, because algorithm
isn't smart enough:

a = 1
if (a): b = 1
if (a): print b

See also:
http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized-325

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control flow graph

[Vitja Makarov]

2011/2/23 Vitja Makarov :
> 2011/2/23 Vitja Makarov :
>> 2011/2/22 Stefan Behnel :
>>> Vitja Makarov, 20.02.2011 18:23:
>>>>
>>>> 2011/2/16 Vitja Makarov:
>>>>>
>>>>> Hmm... both python and codespeaks in the thread
>>>
>>> Yes, we should keep it to cython-devel only. Sorry for mixing it up.
>>>
>>>
>>>>> Here is my commit it's mostly broken now but anyway
>>>>>
>>>>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a
>>>
>>> Flow control support is large enough to merit its own module. Not sure how
>>> 'smart' git is here, but you can always keep the history by explicitly
>>> copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated
>>> sections from both files.
>>>
>>
>> Ok.
>>
>
> Oops there is no copy command in git.
>
>
>>> You are duplicating some code from the type inferencer. We might want to
>>> clean that up at some point. However, given that flow control analysis will
>>> allow us to improve the type inferencer, I think it's best to keep this code
>>> in the FCA part.
>>>
>>
>> Yes, I think it could replace MarkAssignments transform later.
>> Unreachable code could be delete there too.
>>
>>>
>>>> I've update stuff:
>>>>  - algo for finding definitions
>>>>  - warnings for uninitialized and may be uninitialised use
>>>>  - few test cases
>>>
>>> That looks very nice so far. Any idea how well it scales?
>>>
>>
>> "Usually iterative algorithm takes no more then 5 iterations"
>>
>> For ExprNodes.py max number is 15 while avg is about 3
>>
>> About execution time:
>>
>> ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325,
>> ~10% slow down.
>> -O flag could be introduced but I don't think that's a good idea.
>>
>> Should later try to execute cython compiled code.
>>
>>>
>>>> Trying to compile ParseTreeTransforms.py I've found this for example:
>>>>
>>>> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable
>>>> 'template' may be used uninitialized
>>>>
>>>>     def create_Property(self, entry):
>>>>         if entry.visibility == 'public':
>>>>             if entry.type.is_pyobject:
>>>>                 template = self.basic_pyobject_property
>>>>             else:
>>>>                 template = self.basic_property
>>>>         elif entry.visibility == 'readonly':
>>>>             template = self.basic_property_ro
>>>>         property = template.substitute({
>>>>                 u"ATTR": ExprNodes.AttributeNode(pos=entry.pos,
>>>>
>>>> obj=ExprNodes.NameNode(pos=entry.pos, name="self"),
>>>>                                                  attribute=entry.name),
>>>>             }, pos=entry.pos).stats[0]
>>>
>>> Ok, I guess that code generally works, but it's better to get rid of the
>>> code smell.
>>>
>>
>> Might be used warning should be disabled by default, because algorithm
>> isn't smart enough:
>>
>> a = 1
>> if (a): b = 1
>> if (a): print b
>>
>> See also:
>> http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized-325
>>


I've updated things:

 - Now it passes all the tests (some tests are run with
remove_unreachable=False)
 - Control flow graph support all the control statements (hope so)
 - Analyse PyClassDefNode (now only inner, need some more work)
 - Now we have the following optional warnings:
  1. uninitialized variable use [default on]
  2. possibly uninitialized use [-Wextra]
  3. unreachable code [default on] (this needs rework, unreachable
code detection could be better handled inside CreateControlFlowGraph)
  4. unused variable [-Wextra]
  5. unused argument [-X warn.unused_arg=True]
  6. unused result [-X warn_unused_result=True]

 - Optional dot graph output:

 $ cython foo.pyx -X control_flow.dot_output=foo.dot
 $ dot -Tpng -o foo.png foo.dot

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] "__ipow__ has wrong number of arguments" is back here

[Vitja Makarov]

Hi!

I noticed that this error came back again.

https://sage.math.washington.edu:8091/hudson/view/cython-devel/job/cython-devel-tests-py26-c/lastCompletedBuild/testReport/(root)/CythonRunTestCase/compiling__c__and_running_special_methods_T561/

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] strange bug in starred target

[Vitja Makarov]

2011/3/3 Vitja Makarov :
> This doesn't work:
> def assign():
>    a, *b = 1,2,3,4,5
>    return a, b
>
>>>> import x
>>>> x.assign()
> Traceback (most recent call last):
>  File "", line 1, in 
>  File "x.pyx", line 6, in x.assign (x.c:445)
>    a, *b = 1,2,3,4,5
> TypeError: Expected list, got int
>
> And this is ok:
>
> def assign():
>    a, *b = 1,2,3,4,5
>    return a, b
>    *a, b = 1,2,3,4,5
>    return a, b
>
>>>> import x
>>>> x.assign()
> (1, [2, 3, 4, 5])
>
>

I hope problem is somewhere in map_starred_assignment()

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control flow graph

[Vitja Makarov]

2011/2/28 Vitja Makarov :
> 2011/2/23 Vitja Makarov :
>> 2011/2/23 Vitja Makarov :
>>> 2011/2/22 Stefan Behnel :
>>>> Vitja Makarov, 20.02.2011 18:23:
>>>>>
>>>>> 2011/2/16 Vitja Makarov:
>>>>>>
>>>>>> Hmm... both python and codespeaks in the thread
>>>>
>>>> Yes, we should keep it to cython-devel only. Sorry for mixing it up.
>>>>
>>>>
>>>>>> Here is my commit it's mostly broken now but anyway
>>>>>>
>>>>>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a
>>>>
>>>> Flow control support is large enough to merit its own module. Not sure how
>>>> 'smart' git is here, but you can always keep the history by explicitly
>>>> copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated
>>>> sections from both files.
>>>>
>>>
>>> Ok.
>>>
>>
>> Oops there is no copy command in git.
>>
>>
>>>> You are duplicating some code from the type inferencer. We might want to
>>>> clean that up at some point. However, given that flow control analysis will
>>>> allow us to improve the type inferencer, I think it's best to keep this 
>>>> code
>>>> in the FCA part.
>>>>
>>>
>>> Yes, I think it could replace MarkAssignments transform later.
>>> Unreachable code could be delete there too.
>>>
>>>>
>>>>> I've update stuff:
>>>>>  - algo for finding definitions
>>>>>  - warnings for uninitialized and may be uninitialised use
>>>>>  - few test cases
>>>>
>>>> That looks very nice so far. Any idea how well it scales?
>>>>
>>>
>>> "Usually iterative algorithm takes no more then 5 iterations"
>>>
>>> For ExprNodes.py max number is 15 while avg is about 3
>>>
>>> About execution time:
>>>
>>> ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325,
>>> ~10% slow down.
>>> -O flag could be introduced but I don't think that's a good idea.
>>>
>>> Should later try to execute cython compiled code.
>>>
>>>>
>>>>> Trying to compile ParseTreeTransforms.py I've found this for example:
>>>>>
>>>>> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable
>>>>> 'template' may be used uninitialized
>>>>>
>>>>>     def create_Property(self, entry):
>>>>>         if entry.visibility == 'public':
>>>>>             if entry.type.is_pyobject:
>>>>>                 template = self.basic_pyobject_property
>>>>>             else:
>>>>>                 template = self.basic_property
>>>>>         elif entry.visibility == 'readonly':
>>>>>             template = self.basic_property_ro
>>>>>         property = template.substitute({
>>>>>                 u"ATTR": ExprNodes.AttributeNode(pos=entry.pos,
>>>>>
>>>>> obj=ExprNodes.NameNode(pos=entry.pos, name="self"),
>>>>>                                                  attribute=entry.name),
>>>>>             }, pos=entry.pos).stats[0]
>>>>
>>>> Ok, I guess that code generally works, but it's better to get rid of the
>>>> code smell.
>>>>
>>>
>>> Might be used warning should be disabled by default, because algorithm
>>> isn't smart enough:
>>>
>>> a = 1
>>> if (a): b = 1
>>> if (a): print b
>>>
>>> See also:
>>> http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized-325
>>>
>
>
> I've updated things:
>
>  - Now it passes all the tests (some tests are run with
> remove_unreachable=False)
>  - Control flow graph support all the control statements (hope so)
>  - Analyse PyClassDefNode (now only inner, need some more work)
>  - Now we have the following optional warnings:
>  1. uninitialized variable use [default on]
>  2. possibly uninitialized use [-Wextra]
>  3. unreachable code [default on] (this needs rework, unreachable
> code detection could be better handled inside CreateControlFlowGraph)
>  4. unused variable [-Wextra]
>  5. unused argument [-X warn.unused_arg=True]
>  6. unused result [-X warn_unused_result=True]
>
>  - Optional dot graph output:
>
>  $ cython foo.pyx -X control_flow.dot_output=foo.dot
>  $ dot -Tpng -o foo.png foo.dot
>

I've pushed unused varaiable/result removal.
Unused argument removal require some more work, especially generic
args and kwargs.

Now I'm going to handle local variable deletion and then remove
Py_None initialization.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Multiple modules in one compilation unit

[Vitja Makarov]

2011/3/3 mark florisson :
> On 3 March 2011 07:43, Stefan Behnel  wrote:
>> Lisandro Dalcin, 03.03.2011 05:38:
>>>
>>> On 2 March 2011 21:01, Greg Ewing  wrote:

 Stefan Behnel wrote:
>
> you'd call "cython" on a package and it would output a directory with a
> single __init__.so that contains the modules compiled from all .pyx/.py
> files in that package. Importing the package would then trigger an
> import of
> that __init__.so, which in turn will execute code in its init__init__()
> function to register the other modules.

 I don't think it even has to be a directory with an __init__,
 it could just be an ordinary .so file with the name of the
 package.

 I just tried an experiment in Python:

 # onefilepackage.py
 import new, sys
 blarg = new.module("blarg")
 blarg.thing = "This is the thing"
 sys.modules["onefilepackage.blarg"] = blarg

 and two different ways of importing it:

 >>> from onefilepackage import blarg
 >>> blarg
 
 >>> blarg.thing
 'This is the thing'

 >>> import onefilepackage.blarg
 >>> onefilepackage.blarg.thing
 'This is the thing'

>>>
>>> I'm hacking around these lines. However, I'm working to maintain
>>> different modules in different C compilation units, in order to
>>> workaround the obvious issue with duplicated global C symbols.
>>
>> That should be ok as a first attempt to get it working quickly. I'd still
>> like to see the modules merged in the long term in order to increase the
>> benefit of the more compact format. They'd all share the same code generator
>> and Cython's C internals, C helper functions, constants, builtins, etc., but
>> each of them would use a separate (name mangling) scope to keep the visible
>> C names separate.
>
> I was thinking that perhaps we could share declarations common to all
> cython modules (compiled with that version of Cython) in a cython.h
> header file (which is also imho nicer to maintain than a code.put() in
> e.g. ModuleNode.generate_module_preamble), and put it in e.g.
> Cython/Includes and set the -I C compiler flag to point to the
> Includes directory.
> Module-specific functions would still be declared static, of course.
> And if users want to ship generated C files to avoid Cython as a
> dependency, they could simply ship the header and adjust their
> setup.py.
>
> If you want to merge modules and the "package-approach" is chosen, it
> should have well-defined semantics for in-place builds, and
> package/__init__.py is preferred over package.so. So how would you
> solve that problem without either placing package.so in the package
> itself, or giving it another name (and perhaps star-importing it from
> __init__.py)? Basically, if people want to combine several modules
> into one they could use the 'include' statement.
>
> e.g. in spam.pyx you 'include "ham.pyx"' and in spam.pxd you 'include
> "ham.pxd"'.
>
> (although you'd probably rename ham.pyx to ham.pxi, and you'd probably
> merge spam.pxd with ham.pxd)
>
> In any case, I'm just wondering, would this functionality be more
> useful than our current include statement and a cython.h header that
> is shared by default?
>
 So assuming the same thing works with a .so instead of a .py,
 all you need to do is emit a .so whose init function stuffs
 appropriate entries into sys.modules to make it look like
 a package.

>>>
>>> However, the import machinery does not treat .so the same as *.pyx.
>>> For example, I have a problem with Python 3. For .py modules, before
>>> the module code starts to execute, the matching entries in sys.modules
>>> is already there.
>>
>> And it has to be, in order to prevent accidental reimports.
>>
>>
>>> The same happens in Python 2 for .so modules, as
>>> Py_InitModule() add the entry in sys.modules early. However, in Python
>>> 3 that is not te case (and only for the .so, for .py is the same as in
>>> Py2), the import machinery adds the entry later, after the
>>> finalization of the module init function. I'm tempted to workaround
>>> this by setting the entry in sys.modules right after the call to
>>> PyModule_Create() ... What do you think about this? Any potential
>>> issue?
>>
>> No idea. I'd say, read the source and give it a try.
>>
>> Stefan


To share common sources is a good idea, we can also share "code" in
libcython-.so
But then we should handle ABI compatibility problems.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] strange bug in starred target

[Vitja Makarov]

2011/3/3 Vitja Makarov :
> 2011/3/3 Vitja Makarov :
>> This doesn't work:
>> def assign():
>>    a, *b = 1,2,3,4,5
>>    return a, b
>>
>>>>> import x
>>>>> x.assign()
>> Traceback (most recent call last):
>>  File "", line 1, in 
>>  File "x.pyx", line 6, in x.assign (x.c:445)
>>    a, *b = 1,2,3,4,5
>> TypeError: Expected list, got int
>>
>> And this is ok:
>>
>> def assign():
>>    a, *b = 1,2,3,4,5
>>    return a, b
>>    *a, b = 1,2,3,4,5
>>    return a, b
>>
>>>>> import x
>>>>> x.assign()
>> (1, [2, 3, 4, 5])
>>
>>
>
> I hope problem is somewhere in map_starred_assignment()
>

That wasn't as simple as patch is...

Ticket on trac http://trac.cython.org/cython_trac/ticket/664
Pull request https://github.com/cython/cython/pull/7

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] InlinedGeneratorExpression question

[Vitja Makarov]

Hi!

Here is example code:

def foo(seq):
   cdef int x
   return any(x for x in seq)

Here inner x have type int, how does cdef affects nested scope?
Is that correct?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] unbound local variables

[Vitja Makarov]

What is the right way to handle cdefed unbounds?

cdef object foo
print foo

cdef int foo
print foo

And how buffers and arrays should be handled? Now I'm skipping
buffers, arrays and structs.


There are some examples in test suite:


...
cdef void spam():
cdef long long L
cdef unsigned long long U
cdef object x
L = x
^


ass2longlong.pyx:5:9: local variable 'x' referenced before assignment

cdef void spam():
cdef object x
del x[17:42]
^


delslice.pyx:3:9: local variable 'x' referenced before assignment

cdef void tomato():
cdef Spam spam
cdef SuperSpam superspam
spam = superspam
   ^


extcmethcall.pyx:16:20: local variable 'superspam' referenced before assignment

def f(Grail g):
cdef int i = 0
cdef Swallow s
cdef object x
g = x
^


extcoerce.pyx:13:9: local variable 'x' referenced before assignment


Should this raise error message or not?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)

[Vitja Makarov]

2011/3/11 Stefan Behnel :
> Hi,
>
> ticket 654 describes a code generation problem where the arguments to a cdef
> function are not being evaluated in the order they are written down in the
> code.
>
> http://trac.cython.org/cython_trac/ticket/654
>
> This introduces problems when the arguments have side effects or are not
> simple, e.g. they are function calls themselves or are taken from object
> attributes or live in a closure. For example,
>
>    f(g(a), a.x, h(a))
>
> will produce different results depending on the evaluation order if g or h
> change the value of a.x.
>
> However, apparently, the order of evaluation is only guaranteed by Python,
> not by C. Now, the question is: what are the right semantics for Cython
> here: follow Python or C?
>

+1 for Python, Cython is mostly Python than C, btw I don't think that
it makes much sense.

> Personally, I think it would be nice to keep up Python's semantics, but when
> I implemented this I broke quite some code in Sage (you may have noticed
> that the sage-build project in Hudson has been red for a while). There are
> things in C and especially in C++ that cannot be easily copied into a
> temporary variable in order to make sure they are evaluated before the
> following arguments. This is not a problem for Python function calls where
> all arguments end up being copied (and often converted) anyway. It is a
> problem for C function calls, though.
>
> What do you think about this?
>


>    f(g(a), a.x, h(a))

Why could not this be translated into:

tmp1 = g(a)
tmp2 = a.x
tmp3 = h(a)

f(tmp1, tmp2, tmp3)


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)

[Vitja Makarov]

2011/3/17 Robert Bradshaw :
> On Thu, Mar 17, 2011 at 6:15 AM, Jason Grout
>  wrote:
>> On 3/16/11 11:05 PM, Robert Bradshaw wrote:
>>>
>>> On Wed, Mar 16, 2011 at 7:53 AM, Stefan Behnel
>>>  wrote:

 I'm actually leaning towards not guaranteeing the order of execution if C
 doesn't do it either. If this is really required, it's easy to work
 around
 for users, but it's severely hard to fix for Cython in all cases, and the
 gain is truly small. After all, we'd only make it easier for users to
 write
 bad code.
>>>
>>> Yep. Lets keep the code in for the above case.
>>
>>
>> Is there a huge big warning in the docs?  Maybe on this page would be a good
>> place: http://docs.cython.org/src/userguide/limitations.html
>
> This doesn't affect Python functions at all, so I'm not sure it
> belongs on that page. I agree that it should be mentioned that c(p)def
> functions have C calling semantics, *including* an unspecified order
> or argument evaluation.
>

As you noticed above, how should be handled def function inlining?

I guess there are restrictions on def function argument type, so
side-effect isn't issue here.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Message system refactoring

[Vitja Makarov]

Now error/warning messages are stored in global variables at
Cython.Compiler.Errors

I think it's much better to move error handling into some object,
Main.Context for example.

Some benefits:
 - reduce use of global variables
 - allow more then one cython compiler instance at the time
 - make it much easy to implement -Werror
 - cython directives can affect message system (fast_fail, werror)
 - messages could be easily sorted

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Message system refactoring

[Vitja Makarov]

2011/3/21 Dag Sverre Seljebotn :
> On 03/21/2011 11:45 AM, Vitja Makarov wrote:
>>
>> Now error/warning messages are stored in global variables at
>> Cython.Compiler.Errors
>>
>> I think it's much better to move error handling into some object,
>> Main.Context for example.
>>
>> Some benefits:
>>  - reduce use of global variables
>>  - allow more then one cython compiler instance at the time
>>  - make it much easy to implement -Werror
>>  - cython directives can affect message system (fast_fail, werror)
>>  - messages could be easily sorted
>
> +1. I assume the reason this is not done is simply because it would be a lot
> of work and the payback is less than spending time on other stuff.
>
> By attaching the error context to "env" and "code" one can avoid a lot of
> signature changes. I think transforms should take the context in their
> constructor.
>

It could be something like proxy object that takes compiler directives
into account and proxies messages to context.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bug in cython-mode.el under GNU Emacs 23.2.1

[Vitja Makarov]

2011/3/22 Rafe Kettler :
> I've just tried out cython-mode.el and I was a bit disappointed when it
> didn't work right off the bat.
>
> I added the following to my .emacs:
>
> (setq load-path (cons "~/lib/emacs-plugins" load-path))
> (require 'cython-mode)
>
> this initially gave me the following error when I restarted emacs:
>
>     Warning (initialization): An error occurred while loading
> `/home/rafe/.emacs':
>
>     File error: Cannot open load file, python-mode
>
>     To ensure normal operation, you should investigate and remove the
>     cause of the error in your initialization file.  Start Emacs with
>     the `--debug-init' option to view a complete error backtrace.
>
> So, seeing that python-mode wasn't around, I changed line 3 of
> cython-mode.el from (require 'python-mode) to (require 'python) and
> everything works now.
>
> I got this mode from Github this morning and it is the latest version (you
> can see it at
> https://github.com/cython/cython/blob/master/Tools/cython-mode.el).
>
> I'm not an emacs lisp expert, so I'm not sure if there's something I'm
> missing here or if I somehow loaded the package correctly. Anyway, I thought
> you guys should know that I had a problem with the file but managed to get a
> fix. If my fix needs to be applied, I'll patch it on Github.
>
> This is GNU Emacs 23.2.1 x64 on Fedora 14, btw.
>
> Rafe
>

Emacs23 have native support for python, you can install python-mode package:

$ sudo apt-get install python-mode

Instead of requiring python it's better to try python-mode first then python:

(when (not (require 'python-mode nil t))
  (require 'python))

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython

[Vitja Makarov]

2011/3/22 Stefan Behnel :
> Stefan Behnel, 22.03.2011 07:10:
>>
>> there seems to be quite some interest in a project to get parts of CPython
>> and specifically its stdlib rewritten in Cython.
>> [...] I gave it a try with difflib and it turned out to be quite easy.
>>
>> http://blog.behnel.de/index.php?p=155
>
> BTW, given how short that patch is, wouldn't that make a nice little
> presentation for the Open Cython Day? Show off how to quickly optimise a
> stdlib module with Cython?
>

That should be nice ;) Greate step forward for cython and python.

Btw I think that stdlib module cython optimization should be as simple
as writting .pxd file with minimal .py or none modifications
As in difflib

Also it would be good to have:
 - performance comparison/monitoring infrastructure
 - more tests on stdlib

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython

[Vitja Makarov]

2011/3/23 Stefan Behnel :
> Craig Citro, 23.03.2011 08:11:
>>>
>>> We have a clear 1.0 goal, being able to compile the full Python
>>> language. We're not there yet, but very close. It may make sense at
>>> that point to also clean up any cruft we don't want to continue
>>> supporting forever. I agree, until that point, there's no way we would
>>> be a Python development dependency.
>>
>> Are we aiming for 100% compatibility, or 99.9%? For instance, I seem to
>> recall a few dark corners we aren't planning on covering
>
> I think it's more like 98%. CPython, PyPy and friends consider the more
> obscure features like frame access a core Python language feature, but I
> don't see an interest in making that work in Cython. If someone wants to
> write the code, fine, as long as it's switched off outside of a future
> "strict compatibility mode". I've never seen this used in any code that I
> would have wanted to compile with Cython, and even in Python code it's only
> really used for more or less "clever hacks".
>
>
>> maybe some details of traceback construction?
>
> What I would like to see here is simply a fallback of the frames that we
> create for tracebacks to look for a .py file (or even .pyx/.pxi file) next
> to the compiled .so file, so that they can print the source line for the
> executed line in the .so.
>
> Apart from that, it boils down to the issue of frame construction. See
> above. Keep in mind that CPython's own C code currently gives you no frames
> at all, so Cython modules already have a serious advantage over what's there
> today.
>
>
>> (I want to say multiple inheritance, too,
>> but I think that's only an issue for cdef classes, right?)
>
> Yes. That's how things work in CPython at the C level, and I'm fine with
> that.
>
>
>> I think it would
>> be good to have this written down -- in particular, it seems like there's
>> some momentum right now for clearly delineating "Python language
>> semantics"
>> vs. "CPython implementation detail" in the python-devel community, so it
>> might be a particularly good time to raise these questions.
>
> +1, a "where do we stand?" topic is planned for the workshop anyway.
>


One more interesting thing is introspection support via inspect module

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bug report: lambda and numpy.vectorize segfaults

[Vitja Makarov]

2011/3/23 René Rex :
> Hello
>
> The attached code creates a segfault with Cython 0.14.1. A workaround
> is to use a real function instead of lambda. Using the normal python
> interpreter the same code works flawlessly.
>
> Here are my build commands:
>
> cython --embed vectorizeBug.pyx
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O0 -Wall -fPIC
> -lpython2.6 -I/usr/include/python2.6 -o vectorizeBug vectorizeBug.c
>
> --
> René Rex
> TU Braunschweig
> Bioinformatics & Biochemistry
> Langer Kamp 19 b
> 38106 Braunschweig
> Phone: +49-531-391-8315
>

This is simplified code that segfaults too:

def f():
return lambda x=0: x
f()()


Also I found that this code doesn't even compiles:

# Module level lambda with default argument value
lambda x=0: x


  File "../cython.py", line 8, in 
main(command_line = 1)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line
818, in main
result = compile(sources, options)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line
793, in compile
return compile_multiple(source, options)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line
765, in compile_multiple
result = run_pipeline(source, options)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line
629, in run_pipeline
err, enddata = context.run_pipeline(pipeline, source)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line
248, in run_pipeline
data = phase(data)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line
571, in parse
tree = context.parse(source_desc, scope, pxd = 0, full_module_name
= full_module_name)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Main.py", line
510, in parse
tree = Parsing.p_module(s, pxd, full_module_name)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 2816, in p_module
body = p_statement_list(s, Ctx(level = level), first_statement = 1)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 1778, in p_statement_list
stats.append(p_statement(s, ctx, first_statement = first_statement))
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 1771, in p_statement
s, ctx, first_statement = first_statement)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 1632, in p_simple_statement_list
stat = p_simple_statement(s, first_statement = first_statement)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 1626, in p_simple_statement
node = p_expression_or_assignment(s)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 1049, in p_expression_or_assignment
expr_list = [p_testlist_star_expr(s)]
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 1005, in p_testlist_star_expr
expr = p_test_or_starred_expr(s)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 191, in p_test_or_starred_expr
return p_test(s)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 121, in p_test
return p_lambdef(s)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 101, in p_lambdef
s, terminator=':', annotated=False)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 2612, in p_varargslist
annotated = annotated)
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 2261, in p_c_arg_list
annotated = annotated))
  File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
line 2307, in p_c_arg_decl
if 'pxd' in s.level:
AttributeError: 'PyrexScanner' object has no attribute 'level'






-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bug report: lambda and numpy.vectorize segfaults

[Vitja Makarov]

2011/3/23 Stefan Behnel :
> Vitja Makarov, 23.03.2011 17:25:
>>
>>   File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
>> line 2307, in p_c_arg_decl
>>     if 'pxd' in s.level:
>> AttributeError: 'PyrexScanner' object has no attribute 'level'
>
> Yes it does:
>
> """
> # Cython/Plex/Scanners.pxd
> cdef class Scanner:
>    [...]
>    cdef public level
> """
>
> Try a clean build.
>

Don't actually understand what's going wrong here?
I'm running plain python cython, probably I'm wrong.

lambda x=0:0 # give same error, but lambda x:0 work fine


About bug: it seems that analyse_expressions() is not called for LambdaNode.
Hope that will help.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bug report: lambda and numpy.vectorize segfaults

[Vitja Makarov]

2011/3/24 Stefan Behnel :
> Vitja Makarov, 23.03.2011 20:11:
>>
>> 2011/3/23 Stefan Behnel:
>>>
>>> Vitja Makarov, 23.03.2011 17:25:
>>>>
>>>>   File "/home/vitja/tmp/cython-my-git/Cython/Compiler/Parsing.py",
>>>> line 2307, in p_c_arg_decl
>>>>     if 'pxd' in s.level:
>>>> AttributeError: 'PyrexScanner' object has no attribute 'level'
>>>
>>> Yes it does:
>>>
>>> """
>>> # Cython/Plex/Scanners.pxd
>>> cdef class Scanner:
>>>    [...]
>>>    cdef public level
>>> """
>>>
>>> Try a clean build.
>>
>> Don't actually understand what's going wrong here?
>> I'm running plain python cython, probably I'm wrong.
>
> These things usually happen to me when I use e.g. a compiled Scanners.so but
> a plain Parsing.py.
>

This is actally cython bug PyrexScanner.level attribute is not
initialized at module scope.

Simple fix is to set level to 'other' or 'unknown'


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] cygdb gdb script problem

[Vitja Makarov]

Running cygdb with Cython installed in the system leads to the
following problem:

vitja@vitja-laptop:~/work/cython-vitek-git/zzz$ python ../cygdb.py
GNU gdb (GDB) 7.2-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
.
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named Debugger
/tmp/tmp1ZvOf9:3: Error in sourced command file:
Error while executing Python code.
(gdb)

So may be it's better to explicitly specify correct path to Cython
package in gdb script?

Small example patch is attached.

-- 
vitja.
diff --git a/Cython/Debugger/Cygdb.py b/Cython/Debugger/Cygdb.py
index faa98d7..bbb0de7 100644
--- a/Cython/Debugger/Cygdb.py
+++ b/Cython/Debugger/Cygdb.py
@@ -31,11 +31,15 @@ def make_command_file(path_to_debug_info, prefix_code='', no_import=False):
 sys.exit('%s.\nNo debug files were found in %s. Aborting.' % (
usage, os.path.abspath(path_to_debug_info)))
 
+cython_root = os.path.abspath(
+os.path.join(os.path.dirname(__file__),'../..'))
+
 fd, tempfilename = tempfile.mkstemp()
 f = os.fdopen(fd, 'w')
 f.write(prefix_code)
 f.write('set breakpoint pending on\n')
 f.write("set print pretty on\n")
+f.write("python import sys; sys.path.insert(0, %r)\n" % cython_root)
 f.write('python from Cython.Debugger import libcython, libpython\n')
 
 if no_import:
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] cygdb gdb script problem

[Vitja Makarov]

2011/3/29 mark florisson :
> On 29 March 2011 21:11, Vitja Makarov  wrote:
>> Running cygdb with Cython installed in the system leads to the
>> following problem:
>>
>> vitja@vitja-laptop:~/work/cython-vitek-git/zzz$ python ../cygdb.py
>> GNU gdb (GDB) 7.2-ubuntu
>> Copyright (C) 2010 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>.
>> Traceback (most recent call last):
>>  File "", line 1, in 
>> ImportError: No module named Debugger
>> /tmp/tmp1ZvOf9:3: Error in sourced command file:
>> Error while executing Python code.
>> (gdb)
>>
>> So may be it's better to explicitly specify correct path to Cython
>> package in gdb script?
>>
>> Small example patch is attached.
>>
>> --
>> vitja.
>>
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>>
>
> Hmm, when using cygdb it expects Cython to be in the path. I believe
> your traceback means that it can find Cython, but not Cython.Debugger,
> right? Are you sure you're using Python 2.6+? It doesn't work with any
> version below 2.6 as they don't use the TP_FLAGS_*_SUBCLASS tp_flags.
>

That's correct when cygdb is executed it's executed with right cython version,
and gdb import system version of cython

> In your patch you probably want to join() the dirname() with
> os.pardir. However, I don't think it would really solve anything,
> because if you install Cython, cygdb will be installed in e.g.
> /usr/bin, so you'd be adding / to the path.

No it will add /usr/lib to path, path is taken from Cython.Debugger.Cygdb

> If you want to run cygdb
> from the cython/bin directory, then cython should be listed in
> PYTHONPATH.

Ok. But you may get in trouble importing system version instead of
development one.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] problem building master with python3

[Vitja Makarov]

2011/4/4 Darren Dale :
> On Mon, Apr 4, 2011 at 3:32 PM, Darren Dale  wrote:
>> I'm attempting to install cython from the git repository to benefit
>> from this fix: http://trac.cython.org/cython_trac/ticket/597 . When I
>> run "python3 setup.py install --user", I get an error:
>>
>> cythoning /Users/darren/Projects/cython/Cython/Compiler/Code.py to
>> /Users/darren/Projects/cython/Cython/Compiler/Code.c
>>
>> Error compiling Cython file:
>> 
>> ...
>>        self.cname = cname
>>        self.text = text
>>        self.escaped_value = StringEncoding.escape_byte_string(byte_string)
>>        self.py_strings = None
>>
>>    def get_py_string_const(self, encoding, identifier=None, is_str=False):
>>   ^
>> 
>>
>> Cython/Compiler/Code.py:320:4: Signature not compatible with previous
>> declaration
>>
>> Error compiling Cython file:
>> 
>> ...
>>    cdef public object text
>>    cdef public object escaped_value
>>    cdef public dict py_strings
>>
>>    @cython.locals(intern=bint, is_str=bint, is_unicode=bint)
>>    cpdef get_py_string_const(self, encoding, identifier=*, is_str=*)
>>                             ^
>> 
>>
>> Cython/Compiler/Code.pxd:64:30: Previous declaration is here
>> building 'Cython.Compiler.Code' extension
>> /usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g
>> -fwrapv -O3 -Wall -Wstrict-prototypes -O2
>> -I/opt/local/Library/Frameworks/Python.framework/Versions/3.2/include/python3.2m
>> -c /Users/darren/Projects/cython/Cython/Compiler/Code.c -o
>> build/temp.macosx-10.6-x86_64-3.2/Users/darren/Projects/cython/Cython/Compiler/Code.o
>> /Users/darren/Projects/cython/Cython/Compiler/Code.c:1:2: error:
>> #error Do not use this file, it is the result of a failed Cython
>> compilation.
>> error: command '/usr/bin/gcc-4.2' failed with exit status 1
>>
>
> Actually, I get this same error when I try to build with python-2.7 as well.
>
> Darren

This one fails too :(

Generators branch is okay. But upstream after merge isn't :(

vitja@vitja-laptop:~/work/cython.git$ cat ttt.py
def foo(is_str=False):
pass
vitja@vitja-laptop:~/work/cython.git$ cat ttt.pxd
cimport cython

@cython.locals(is_str=cython.bint)
cdef foo(is_str=*)





-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] speed.pypy.org

[Vitja Makarov]

2011/4/11 Stefan Behnel :
> Hi,
>
> I'm currently discussing with Maciej Fijalkowski (PyPy) how to get Cython
> running on speed.pypy.org (that's what I wrote "cythonrun" for). If it works
> out well, we may have it up in a couple of days.
>
> I would expect that Cython won't be a big winner in this game, given that it
> will only compile plain untyped Python code. It's also going to fail
> entirely in some of the benchmarks. But I think it's worth having it up
> there, simply as a way for us to see where we are performance-wise and to
> get quick (nightly) feed-back about optimisations we try. The benchmark
> suite is also a nice set of real-world Python code that will allow us to
> find compliance issues.
>
> Stefan


Cool, that would be nice!

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control flow graph

[Vitja Makarov]

Can I use cython-generators project on hudson for control-flow tests?
So I'll move cf branch development to my master branch.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Recent bugs in generators

[Vitja Makarov]

Hi!

1. Lambda-generator:

Previous implementation was inspired by Python2.6:
>>> list((lambda:((yield 1), (yield 2)))())
[1, 2, (None, None)]

things changed in python2.7:
>>> list((lambda:((yield 1), (yield 2)))())
[1, 2]

2. GeneratorExit is initialized to StopIteration when running
generators_py doctests

This is strange behaviour of doctest module, try this:
x = __builtins__
def foo():
"""
>>> type(x)

"""

3. check_yield_in_exception()

Cython calls __Pyx_ExceptionReset when except block is done, so when
yield is there no exception reset is called.

I'm not sure how to fix this.

import sys

def foo():
"""
>>> list(foo())
[, None]
"""
try:
raise ValueError
except ValueError:
yield sys.exc_info()[0]
yield sys.exc_info()[0] # exc_info is lost here


Here is quick fix for 1 and 2 https://github.com/cython/cython/pull/25

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Recent bugs in generators

[Vitja Makarov]

2011/4/18 Stefan Behnel :
> Vitja Makarov, 17.04.2011 17:57:
>>
>> 3. check_yield_in_exception()
>
> I added this because I found a failing pyregr test that uses it (testing the
> @contextmanager decorator).
>
>
>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>> yield is there no exception reset is called.
>>
>> I'm not sure how to fix this.
>
> I'm not completely sure either.
>
>
>> import sys
>>
>> def foo():
>>     """
>>     >>>  list(foo())
>>     [, None]
>>     """
>>     try:
>>         raise ValueError
>>     except ValueError:
>>         yield sys.exc_info()[0]
>>         yield sys.exc_info()[0] # exc_info is lost here
>
> I think (!), the difference here is that CPython actually keeps the
> exception in the generator frame. We don't have a frame, so we have to
> emulate it using the closure class. I guess we'll have to store away the
> exception into the closure when we yield while an exception is being
> handled, and restore it afterwards. Note: this is not the exception that is
> freshly *being* raised (the "_cur*" fields in the thread state), it's the
> exception that *was* raised and is now being handled, i.e. the thread state
> fields without the "_cur", that are reflected by sys.exc_info().
>

Interesting difference between py2 and py3:

def foo():
try:
raise ValueError
except ValueError:
yield
raise
list(foo())

  File "xxx.py", line 7, in 
list(foo())
  File "xxx.py", line 6, in foo
raise
TypeError: exceptions must be old-style classes or derived from
BaseException, not NoneType

It seems that exception info is completely lost (tried 2.6, 2.7) and
seems to be fixed in python3.

Btw exception info temps are already saved and restored between yields.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Recent bugs in generators

[Vitja Makarov]

2011/4/18 Stefan Behnel :
> Vitja Makarov, 18.04.2011 06:38:
>>
>> 2011/4/18 Stefan Behnel:
>>>
>>> Vitja Makarov, 17.04.2011 17:57:
>>>>
>>>> 3. check_yield_in_exception()
>>>
>>> I added this because I found a failing pyregr test that uses it (testing
>>> the
>>> @contextmanager decorator).
>>>
>>>
>>>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>>>> yield is there no exception reset is called.
>>>>
>>>> I'm not sure how to fix this.
>>>
>>> I'm not completely sure either.
>>>
>>>
>>>> import sys
>>>>
>>>> def foo():
>>>>     """
>>>>     >>>    list(foo())
>>>>     [, None]
>>>>     """
>>>>     try:
>>>>         raise ValueError
>>>>     except ValueError:
>>>>         yield sys.exc_info()[0]
>>>>         yield sys.exc_info()[0] # exc_info is lost here
>>>
>>> I think (!), the difference here is that CPython actually keeps the
>>> exception in the generator frame. We don't have a frame, so we have to
>>> emulate it using the closure class. I guess we'll have to store away the
>>> exception into the closure when we yield while an exception is being
>>> handled, and restore it afterwards. Note: this is not the exception that
>>> is
>>> freshly *being* raised (the "_cur*" fields in the thread state), it's the
>>> exception that *was* raised and is now being handled, i.e. the thread
>>> state
>>> fields without the "_cur", that are reflected by sys.exc_info().
>>
>> Interesting difference between py2 and py3:
>>
>> def foo():
>>     try:
>>         raise ValueError
>>     except ValueError:
>>         yield
>>         raise
>> list(foo())
>>
>>   File "xxx.py", line 7, in
>>     list(foo())
>>   File "xxx.py", line 6, in foo
>>     raise
>> TypeError: exceptions must be old-style classes or derived from
>> BaseException, not NoneType
>>
>> It seems that exception info is completely lost (tried 2.6, 2.7) and
>> seems to be fixed in python3.
>
> Not surprising. The implementation is completely different in Py2 and Py3,
> both in CPython and in Cython. It's actually much simpler in Cython under
> Py3, due to better semantics and C-API support. That also implies that
> there's much less Cython can do wrong in that environment. ;-)
>
>
>> Btw exception info temps are already saved and restored between yields.
>
> Right, but the exc_info itself is not reset and recovered around the yield.
> As I said above, generators have their own lifetime frame in CPython, and
> exceptions don't leak from that. So, whenever it's the generator (or code
> called by it) that raises an exception, that must be kept local to the
> generator.
>

There is one more interesting thing:

def test_yield_inside_genexp():
"""
>>> o = test_yield_inside_genexp()
>>> list(o)
[0, None, 1, None, 2, None]
"""
return ((yield i) for i in range(3))


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Recent bugs in generators

[Vitja Makarov]

2011/4/19 Stefan Behnel :
> Vitja Makarov, 18.04.2011 15:19:
>>
>> There is one more interesting thing:
>>
>> def test_yield_inside_genexp():
>>     """
>>     >>>  o = test_yield_inside_genexp()
>>     >>>  list(o)
>>     [0, None, 1, None, 2, None]
>>     """
>>     return ((yield i) for i in range(3))
>
> Impressively ugly.
>
> I guess we should start testing these things with other Python
> implementations than CPython, in order to see if these corner cases are
> really being accepted and knowingly being implemented, or if it's just
> something that no-one has really cared about so far and that's actually
> worth discussing.
>

The case above is yield inside genexp and it works now.

Btw we've found one more funny thing this time with parser: 0lor(1)


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Recent bugs in generators

[Vitja Makarov]

2011/4/18 Stefan Behnel :
> Vitja Makarov, 18.04.2011 06:38:
>>
>> 2011/4/18 Stefan Behnel:
>>>
>>> Vitja Makarov, 17.04.2011 17:57:
>>>>
>>>> 3. check_yield_in_exception()
>>>
>>> I added this because I found a failing pyregr test that uses it (testing
>>> the
>>> @contextmanager decorator).
>>>
>>>
>>>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>>>> yield is there no exception reset is called.
>>>>
>>>> I'm not sure how to fix this.
>>>
>>> I'm not completely sure either.
>>>
>>>
>>>> import sys
>>>>
>>>> def foo():
>>>>     """
>>>>     >>>    list(foo())
>>>>     [, None]
>>>>     """
>>>>     try:
>>>>         raise ValueError
>>>>     except ValueError:
>>>>         yield sys.exc_info()[0]
>>>>         yield sys.exc_info()[0] # exc_info is lost here
>>>
>>> I think (!), the difference here is that CPython actually keeps the
>>> exception in the generator frame. We don't have a frame, so we have to
>>> emulate it using the closure class. I guess we'll have to store away the
>>> exception into the closure when we yield while an exception is being
>>> handled, and restore it afterwards. Note: this is not the exception that
>>> is
>>> freshly *being* raised (the "_cur*" fields in the thread state), it's the
>>> exception that *was* raised and is now being handled, i.e. the thread
>>> state
>>> fields without the "_cur", that are reflected by sys.exc_info().
>>
>> Interesting difference between py2 and py3:
>>
>> def foo():
>>     try:
>>         raise ValueError
>>     except ValueError:
>>         yield
>>         raise
>> list(foo())
>>
>>   File "xxx.py", line 7, in
>>     list(foo())
>>   File "xxx.py", line 6, in foo
>>     raise
>> TypeError: exceptions must be old-style classes or derived from
>> BaseException, not NoneType
>>
>> It seems that exception info is completely lost (tried 2.6, 2.7) and
>> seems to be fixed in python3.
>
> Not surprising. The implementation is completely different in Py2 and Py3,
> both in CPython and in Cython. It's actually much simpler in Cython under
> Py3, due to better semantics and C-API support. That also implies that
> there's much less Cython can do wrong in that environment. ;-)
>
>
>> Btw exception info temps are already saved and restored between yields.
>
> Right, but the exc_info itself is not reset and recovered around the yield.
> As I said above, generators have their own lifetime frame in CPython, and
> exceptions don't leak from that. So, whenever it's the generator (or code
> called by it) that raises an exception, that must be kept local to the
> generator.
>

Please review:
https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858

I've added __Pyx_Generator_SwapExceptions() method and call it right
before resume switch and before return from yield. It swaps generator
exception state with thread local.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Recent bugs in generators

[Vitja Makarov]

2011/4/20 Stefan Behnel :
> Vitja Makarov, 20.04.2011 10:26:
>>
>> 2011/4/18 Stefan Behnel:
>>>
>>> Vitja Makarov, 18.04.2011 06:38:
>>>>
>>>> 2011/4/18 Stefan Behnel:
>>>>>
>>>>> Vitja Makarov, 17.04.2011 17:57:
>>>>>>
>>>>>> 3. check_yield_in_exception()
>>>>>
>>>>> I added this because I found a failing pyregr test that uses it
>>>>> (testing
>>>>> the
>>>>> @contextmanager decorator).
>>>>>
>>>>>
>>>>>> Cython calls __Pyx_ExceptionReset when except block is done, so when
>>>>>> yield is there no exception reset is called.
>>>>>>
>>>>>> I'm not sure how to fix this.
>>>>>
>>>>> I'm not completely sure either.
>>>>>
>>>>>
>>>>>> import sys
>>>>>>
>>>>>> def foo():
>>>>>>     """
>>>>>>     >>>      list(foo())
>>>>>>     [, None]
>>>>>>     """
>>>>>>     try:
>>>>>>         raise ValueError
>>>>>>     except ValueError:
>>>>>>         yield sys.exc_info()[0]
>>>>>>         yield sys.exc_info()[0] # exc_info is lost here
>>>>>
>>>>> I think (!), the difference here is that CPython actually keeps the
>>>>> exception in the generator frame. We don't have a frame, so we have to
>>>>> emulate it using the closure class. I guess we'll have to store away
>>>>> the
>>>>> exception into the closure when we yield while an exception is being
>>>>> handled, and restore it afterwards. Note: this is not the exception
>>>>> that
>>>>> is
>>>>> freshly *being* raised (the "_cur*" fields in the thread state), it's
>>>>> the
>>>>> exception that *was* raised and is now being handled, i.e. the thread
>>>>> state
>>>>> fields without the "_cur", that are reflected by sys.exc_info().
>>>>
>>>> Interesting difference between py2 and py3:
>>>>
>>>> def foo():
>>>>     try:
>>>>         raise ValueError
>>>>     except ValueError:
>>>>         yield
>>>>         raise
>>>> list(foo())
>>>>
>>>>   File "xxx.py", line 7, in
>>>>     list(foo())
>>>>   File "xxx.py", line 6, in foo
>>>>     raise
>>>> TypeError: exceptions must be old-style classes or derived from
>>>> BaseException, not NoneType
>>>>
>>>> It seems that exception info is completely lost (tried 2.6, 2.7) and
>>>> seems to be fixed in python3.
>>>
>>> Not surprising. The implementation is completely different in Py2 and
>>> Py3,
>>> both in CPython and in Cython. It's actually much simpler in Cython under
>>> Py3, due to better semantics and C-API support. That also implies that
>>> there's much less Cython can do wrong in that environment. ;-)
>>>
>>>
>>>> Btw exception info temps are already saved and restored between yields.
>>>
>>> Right, but the exc_info itself is not reset and recovered around the
>>> yield.
>>> As I said above, generators have their own lifetime frame in CPython, and
>>> exceptions don't leak from that. So, whenever it's the generator (or code
>>> called by it) that raises an exception, that must be kept local to the
>>> generator.
>>
>> Please review:
>>
>> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>>
>> I've added __Pyx_Generator_SwapExceptions() method and call it right
>> before resume switch and before return from yield. It swaps generator
>> exception state with thread local.
>
> Looks good to me. I assume this fixes the problem? Then please push it into
> mainline.
>

old pull request is still there ;)

https://github.com/cython/cython/pull/25

Does __Pyx_ExceptionReset() steal references to args, so they should
not be decrefed later?


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Recent bugs in generators

[Vitja Makarov]

2011/4/20 Stefan Behnel :
> Vitja Makarov, 20.04.2011 11:50:
>>
>> 2011/4/20 Stefan Behnel:
>>>
>>> Vitja Makarov, 20.04.2011 10:26:
>>>>
>>>> 2011/4/18 Stefan Behnel:
>>>>>
>>>>> generators have their own lifetime frame in CPython, and
>>>>> exceptions don't leak from that. So, whenever it's the generator (or
>>>>> code
>>>>> called by it) that raises an exception, that must be kept local to the
>>>>> generator.
>>>>
>>>> Please review:
>>>>
>>>>
>>>> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>>>>
>>>> I've added __Pyx_Generator_SwapExceptions() method and call it right
>>>> before resume switch and before return from yield. It swaps generator
>>>> exception state with thread local.
>>>
>>> Looks good to me. I assume this fixes the problem? Then please push it
>>> into
>>> mainline.
>>
>> old pull request is still there ;)
>>
>> https://github.com/cython/cython/pull/25
>>
>> Does __Pyx_ExceptionReset() steal references to args, so they should
>> not be decrefed later?
>
> Hmm, good call. The refcounting looks correct over the two function calls,
> but it would be nicer if it could be avoided instead. We are really just
> swapping around pointers here, no refcounting is needed.
>
> I think it's worth using a dedicated utility function for this.
>

Do you mean new utility call __Pyx_ExceptionSwap()?

> Oh, and one more thing: what should be done when exiting the generator
> normally? Would that just raise GeneratorExit anyway, so that we can swallow
> any original outer exception? I think there might be a problem with Python
> 3, where the GeneratorExit should be raised in the context of the original
> exception. Worth testing how CPython behaves here...
>


Right! It's better to wrap call to generator:

ExceptionSwap();
generator_body();
ExceptionSwap();

Ok, I'll take a look.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Recent bugs in generators

[Vitja Makarov]

2011/4/20 Stefan Behnel :
> Vitja Makarov, 20.04.2011 12:51:
>>
>> 2011/4/20 Stefan Behnel:
>>>
>>> Vitja Makarov, 20.04.2011 11:50:
>>>>
>>>> 2011/4/20 Stefan Behnel:
>>>>>
>>>>> Vitja Makarov, 20.04.2011 10:26:
>>>>>>
>>>>>> 2011/4/18 Stefan Behnel:
>>>>>>>
>>>>>>> generators have their own lifetime frame in CPython, and
>>>>>>> exceptions don't leak from that. So, whenever it's the generator (or
>>>>>>> code
>>>>>>> called by it) that raises an exception, that must be kept local to
>>>>>>> the
>>>>>>> generator.
>>>>>>
>>>>>> Please review:
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://github.com/vitek/cython/commit/73014aaed10b82a3f632d7f86212f86280c55858
>>>>>>
>>>>>> I've added __Pyx_Generator_SwapExceptions() method and call it right
>>>>>> before resume switch and before return from yield. It swaps generator
>>>>>> exception state with thread local.
>>>>>
>>>>> Looks good to me. I assume this fixes the problem? Then please push it
>>>>> into
>>>>> mainline.
>>>>
>>>> old pull request is still there ;)
>>>>
>>>> https://github.com/cython/cython/pull/25
>>>>
>>>> Does __Pyx_ExceptionReset() steal references to args, so they should
>>>> not be decrefed later?
>>>
>>> Hmm, good call. The refcounting looks correct over the two function
>>> calls,
>>> but it would be nicer if it could be avoided instead. We are really just
>>> swapping around pointers here, no refcounting is needed.
>>>
>>> I think it's worth using a dedicated utility function for this.
>>>
>>
>> Do you mean new utility call __Pyx_ExceptionSwap()?
>
> Yes. I think it doesn't even have to be specific to the generator code. Just
> pass in "&gen->exc_type" etc.
>
>
>>> Oh, and one more thing: what should be done when exiting the generator
>>> normally? Would that just raise GeneratorExit anyway, so that we can
>>> swallow
>>> any original outer exception? I think there might be a problem with
>>> Python
>>> 3, where the GeneratorExit should be raised in the context of the
>>> original
>>> exception. Worth testing how CPython behaves here...
>>>
>>
>>
>> Right! It's better to wrap call to generator:
>>
>> ExceptionSwap();
>> generator_body();
>> ExceptionSwap();
>
> Good idea.
>
>
>> Ok, I'll take a look.
>
> Cool. Thanks!
>

https://github.com/vitek/cython/compare/73014aaed1...01286645d0

Another one try ;)


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Build module script

[Vitja Makarov]

Now we have cythonrun build script, may be it's time to create script
for easy module building?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Build module script

[Vitja Makarov]

2011/4/21 mark florisson :
> On 21 April 2011 08:57, Vitja Makarov  wrote:
>> Now we have cythonrun build script, may be it's time to create script
>> for easy module building?
>>
>> --
>> vitja.
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> We have cythonize():
> http://wiki.cython.org/enhancements/distutils_preprocessing . Is
> something like that what you mean? Unfortunately I believe it doesn't
> parse sys.argv (yet). Perhaps a recursive option would also be useful.

Not exactly cythonize is for distutils and friends.

I mean simple script that run cython compiler and then build shared
python module.
It's much better then manually run cython then gcc or write makefile for this.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson pyregr testing takes too long

[Vitja Makarov]

2011/4/25 Stefan Behnel :
> Stefan Behnel, 07.04.2011 13:52:
>>
>> Stefan Behnel, 07.04.2011 13:46:
>>>
>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>> minutes for a run to ~4 hours when we added generator support.
>>>
>>>
>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>
>>> I currently have no idea why that is (well, it's likely because we
>>> compile
>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>> would
>>> be great if someone could find the time to analyse this problem. The
>>> current run time makes it basically impossible to keep these tests
>>> enabled.
>>
>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The Py2.7
>> tests take 30-45 minutes, which is very long, but not completely out of
>> bounds. I've disabled the Py2.6 pyregr tests for now.
>
> There seems to be a huge memory leak which almost certainly accounts for
> this. The Python process that runs the pyregr suite ends up with about 50GB
> of memory at the end, also in the latest Py3k builds.
>
> I have no idea where it may be, but it started to show when we merged the
> generator support. That's where I noticed the instant jump in the runtime.
>

That's very strange for my branch it takes about 30 minutes that is ok.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson pyregr testing takes too long

[Vitja Makarov]

2011/4/25 Stefan Behnel :
> Vitja Makarov, 25.04.2011 08:19:
>>
>> 2011/4/25 Stefan Behnel:
>>>
>>> Stefan Behnel, 07.04.2011 13:52:
>>>>
>>>> Stefan Behnel, 07.04.2011 13:46:
>>>>>
>>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>>> minutes for a run to ~4 hours when we added generator support.
>>>>>
>>>>>
>>>>>
>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>>
>>>>> I currently have no idea why that is (well, it's likely because we
>>>>> compile
>>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>>> would
>>>>> be great if someone could find the time to analyse this problem. The
>>>>> current run time makes it basically impossible to keep these tests
>>>>> enabled.
>>>>
>>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The
>>>> Py2.7
>>>> tests take 30-45 minutes, which is very long, but not completely out of
>>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>>
>>> There seems to be a huge memory leak which almost certainly accounts for
>>> this. The Python process that runs the pyregr suite ends up with about
>>> 50GB
>>> of memory at the end, also in the latest Py3k builds.
>>>
>>> I have no idea where it may be, but it started to show when we merged the
>>> generator support. That's where I noticed the instant jump in the
>>> runtime.
>>
>> That's very strange for my branch it takes about 30 minutes that is ok.
>
> There's also a second path that's worth investigating. As part of the merge,
> there was another change that came in: the CythonPyregrTestCase
> implementation. This means that the regression tests are now being run
> differently than before. The massive memory consumption may simply be due to
> the mass of unit tests being loaded into memory.
>


   def run_test():
..
try:
module = __import__(self.module)
if hasattr(module, 'test_main'):
module.test_main()
except (unittest.SkipTest, support.ResourceDenied):
result.addSkip(self, 'ok')


It seems that all the modules stay loaded so may be they should be
unloaded with del sys.modules[module_name]?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson pyregr testing takes too long

[Vitja Makarov]

2011/4/25 Stefan Behnel :
> Vitja Makarov, 25.04.2011 11:04:
>>
>> 2011/4/25 Stefan Behnel:
>>>
>>> Vitja Makarov, 25.04.2011 08:19:
>>>>
>>>> 2011/4/25 Stefan Behnel:
>>>>>
>>>>> Stefan Behnel, 07.04.2011 13:52:
>>>>>>
>>>>>> Stefan Behnel, 07.04.2011 13:46:
>>>>>>>
>>>>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>>>>> minutes for a run to ~4 hours when we added generator support.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>>>>
>>>>>>> I currently have no idea why that is (well, it's likely because we
>>>>>>> compile
>>>>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>>>>> would
>>>>>>> be great if someone could find the time to analyse this problem. The
>>>>>>> current run time makes it basically impossible to keep these tests
>>>>>>> enabled.
>>>>>>
>>>>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The
>>>>>> Py2.7
>>>>>> tests take 30-45 minutes, which is very long, but not completely out
>>>>>> of
>>>>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>>>>
>>>>> There seems to be a huge memory leak which almost certainly accounts
>>>>> for
>>>>> this. The Python process that runs the pyregr suite ends up with about
>>>>> 50GB
>>>>> of memory at the end, also in the latest Py3k builds.
>>>>>
>>>>> I have no idea where it may be, but it started to show when we merged
>>>>> the
>>>>> generator support. That's where I noticed the instant jump in the
>>>>> runtime.
>>>>
>>>> That's very strange for my branch it takes about 30 minutes that is ok.
>>>
>>> There's also a second path that's worth investigating. As part of the
>>> merge,
>>> there was another change that came in: the CythonPyregrTestCase
>>> implementation. This means that the regression tests are now being run
>>> differently than before. The massive memory consumption may simply be due
>>> to
>>> the mass of unit tests being loaded into memory.
>>
>>    def run_test():
>> ..
>>         try:
>>             module = __import__(self.module)
>>             if hasattr(module, 'test_main'):
>>                 module.test_main()
>>         except (unittest.SkipTest, support.ResourceDenied):
>>             result.addSkip(self, 'ok')
>>
>>
>> It seems that all the modules stay loaded so may be they should be
>> unloaded with del sys.modules[module_name]?
>
> (Binary) module unloading isn't really supported in CPython. There's PEP
> 3121 that has the potential to change it, but it's not completely
> implemented, neither in CPython nor in Cython. A major problem is that
> unloading a module deletes its globals but not necessarily the code that
> uses them. For example, instances of types defined in the module can still
> be alive at that point.
>
> The way runtests.py deals with this is forking before loading a module.
> However, this does not currently work with the "xmlrunner" which we use on
> Hudson, so we let all tests run in a single process there.
>


Btw when running plain python code with generators total ref counter
doesn't get back to initial value.
I tried to trace scope and generator destructors and they are run as
expected. So I'm not sure about leaks in generators.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] speed.pypy.org

[Vitja Makarov]

2011/4/27 Stefan Behnel :
> Robert Bradshaw, 27.04.2011 19:08:
>>
>> On Wed, Apr 27, 2011 at 12:45 AM, Stefan Behnel wrote:
>>>
>>> Actually, if we want a proper history, I'd suggest a separate codespeed
>>> installation somewhere.
>>
>> Makes sense. How many CPU-hours does it take?
>
> Including the Cython build, it's more like 25 minutes currently, given that
> we only support a smaller part of the benchmark suite. It will obviously
> take longer when we start supporting the larger benchmarks, such as Django
> templates or Twisted.
>
>
>> If it's not to
>> intensive, we could probably run it, say, daily as a normal-priority
>> job.
>
> We could certainly do that for now, and check again when we see that it
> starts running substantially longer.
>

Nice thing! Can I run this tests at home?
-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Dead code removal

[Vitja Makarov]

Hi!

I can move unreachable code removal into its own branch and then make
a pull request.

But it seems to me that dead code removal should be done in control
flow analysis so there are few options:

1. remove unreachable code in control flow transformation
2. set is_terminator flag in control flow and then run removal
transformation later

For instance, RemoveUnreachableCode doesn't handle this case:

try:
raise Error
finally:
pass
print 'unreachable'

This case could be easily handled if I set is_terminator flag on the
whole TryFinallyStatNode,
but I'm not sure that this is the only unhandled case.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] jenkins problems

[Vitja Makarov]

Hi!

Jenkins doesn't work for me. It seems that it can't do pull and is
running tests again obsolete sources.
May be because of forced push.

There are only 6 errors here:
https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py27-c/

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] jenkins problems

[Vitja Makarov]

2011/5/4 Vitja Makarov :
> Hi!
>
> Jenkins doesn't work for me. It seems that it can't do pull and is
> running tests again obsolete sources.
> May be because of forced push.
>
> There are only 6 errors here:
> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py27-c/
>

Can you please provide me jenkins account and I'll try to fix the issues myself?


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] jenkins problems

[Vitja Makarov]

2011/5/4 Vitja Makarov :
> 2011/5/4 Vitja Makarov :
>> Hi!
>>
>> Jenkins doesn't work for me. It seems that it can't do pull and is
>> running tests again obsolete sources.
>> May be because of forced push.
>>
>> There are only 6 errors here:
>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py27-c/
>>
>
> Can you please provide me jenkins account and I'll try to fix the issues 
> myself?
>

It's better to use:

$ git fetch origin
$ git checkout -f origin/master

Instead of git pull

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] jenkins problems

[Vitja Makarov]

2011/5/4 Yury V. Zaytsev :
> On Wed, 2011-05-04 at 10:35 +0400, Vitja Makarov wrote:
>> > Can you please provide me jenkins account and I'll try to fix the issues 
>> > myself?
>> >
>>
>> It's better to use:
>>
>> $ git fetch origin
>> $ git checkout -f origin/master
>>
>> Instead of git pull
>
> Or
>
> $ git fetch origin
> $ git reset --hard origin/master
>
> which is what we used for our buildbot.
>
> --
> Sincerely yours,
> Yury V. Zaytsev
>

Thanks! Am I right: when you do reset '--hard origin/master' you are
on the master branch and when you do checkout you are in a 'detached
state'?

But it seems to me that the problem is somewhere in the jenkins configuration.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] jenkins problems

[Vitja Makarov]

2011/5/4 Stefan Behnel :
> Vitja Makarov, 04.05.2011 07:09:
>>
>> Jenkins doesn't work for me. It seems that it can't do pull and is
>> running tests again obsolete sources.
>> May be because of forced push.
>>
>> There are only 6 errors here:
>>
>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py27-c/
>
> According to the build logs of the sdist job, it was previously checking out
> the "master" branch and it seems you reconfigured it to use the
> "unreachable_code" branch now. At least the recent checkouts have used the
> latest snapshot of the branches, so ISTM that everything is working
> correctly. Could you point me to a build where something was going wrong for
> you?
>
> Stefan


I've added the following line to sdist target

+rm -fr $WORKSPACE/dist
$WORKSPACE/python/bin/python setup.py clean sdist --formats=gztar
--cython-profile --no-cython-compile

Hope that should help, that's the only difference between
cython-devel-sdist and cython-vitek-sdist.


See here:
https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py27-c/166/

You can't find w_unreachable in the logs, it seems that cython code
there is outdated.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] jenkins problems

[Vitja Makarov]

2011/5/4 Stefan Behnel :
> Vitja Makarov, 04.05.2011 21:14:
>>
>> 2011/5/4 Stefan Behnel:
>>>
>>> Vitja Makarov, 04.05.2011 07:09:
>>>>
>>>> Jenkins doesn't work for me. It seems that it can't do pull and is
>>>> running tests again obsolete sources.
>>>> May be because of forced push.
>>>>
>>>> There are only 6 errors here:
>>>>
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py27-c/
>>>
>>> According to the build logs of the sdist job, it was previously checking
>>> out
>>> the "master" branch and it seems you reconfigured it to use the
>>> "unreachable_code" branch now. At least the recent checkouts have used
>>> the
>>> latest snapshot of the branches, so ISTM that everything is working
>>> correctly. Could you point me to a build where something was going wrong
>>> for
>>> you?
>>
>> I've added the following line to sdist target
>>
>> +rm -fr $WORKSPACE/dist
>> $WORKSPACE/python/bin/python setup.py clean sdist --formats=gztar
>> --cython-profile --no-cython-compile
>>
>> Hope that should help, that's the only difference between
>> cython-devel-sdist and cython-vitek-sdist.
>>
>>
>> See here:
>>
>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py27-c/166/
>>
>> You can't find w_unreachable in the logs, it seems that cython code
>> there is outdated.
>
> Ah, right, that's it. You can see the problem here:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-vitek-sdist/97/artifact/dist/Cython-0.14+.tar.gz/*fingerprint*/
>
> It's been using the 0.14+ sdist for ages instead of the 0.14.1+ one.
>
> That could also explain why your CPython regression tests are running much
> faster than in cython-devel.
>

Ok, so I should take a look at pyregr tests closer.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson pyregr testing takes too long

[Vitja Makarov]

2011/4/25 Vitja Makarov :
> 2011/4/25 Stefan Behnel :
>> Vitja Makarov, 25.04.2011 11:04:
>>>
>>> 2011/4/25 Stefan Behnel:
>>>>
>>>> Vitja Makarov, 25.04.2011 08:19:
>>>>>
>>>>> 2011/4/25 Stefan Behnel:
>>>>>>
>>>>>> Stefan Behnel, 07.04.2011 13:52:
>>>>>>>
>>>>>>> Stefan Behnel, 07.04.2011 13:46:
>>>>>>>>
>>>>>>>> I just noticed that the CPython pyregr tests have jumped up from ~14
>>>>>>>> minutes for a run to ~4 hours when we added generator support.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-tests-pyregr-py26-c/buildTimeTrend
>>>>>>>>
>>>>>>>> I currently have no idea why that is (well, it's likely because we
>>>>>>>> compile
>>>>>>>> more tests now, but Vitja's branch ran the tests in ~30 minutes). It
>>>>>>>> would
>>>>>>>> be great if someone could find the time to analyse this problem. The
>>>>>>>> current run time makes it basically impossible to keep these tests
>>>>>>>> enabled.
>>>>>>>
>>>>>>> Ok, it looks like this is mostly an issue with the Py2.6 tests. The
>>>>>>> Py2.7
>>>>>>> tests take 30-45 minutes, which is very long, but not completely out
>>>>>>> of
>>>>>>> bounds. I've disabled the Py2.6 pyregr tests for now.
>>>>>>
>>>>>> There seems to be a huge memory leak which almost certainly accounts
>>>>>> for
>>>>>> this. The Python process that runs the pyregr suite ends up with about
>>>>>> 50GB
>>>>>> of memory at the end, also in the latest Py3k builds.
>>>>>>
>>>>>> I have no idea where it may be, but it started to show when we merged
>>>>>> the
>>>>>> generator support. That's where I noticed the instant jump in the
>>>>>> runtime.
>>>>>
>>>>> That's very strange for my branch it takes about 30 minutes that is ok.
>>>>
>>>> There's also a second path that's worth investigating. As part of the
>>>> merge,
>>>> there was another change that came in: the CythonPyregrTestCase
>>>> implementation. This means that the regression tests are now being run
>>>> differently than before. The massive memory consumption may simply be due
>>>> to
>>>> the mass of unit tests being loaded into memory.
>>>
>>>    def run_test():
>>> ..
>>>         try:
>>>             module = __import__(self.module)
>>>             if hasattr(module, 'test_main'):
>>>                 module.test_main()
>>>         except (unittest.SkipTest, support.ResourceDenied):
>>>             result.addSkip(self, 'ok')
>>>
>>>
>>> It seems that all the modules stay loaded so may be they should be
>>> unloaded with del sys.modules[module_name]?
>>
>> (Binary) module unloading isn't really supported in CPython. There's PEP
>> 3121 that has the potential to change it, but it's not completely
>> implemented, neither in CPython nor in Cython. A major problem is that
>> unloading a module deletes its globals but not necessarily the code that
>> uses them. For example, instances of types defined in the module can still
>> be alive at that point.
>>
>> The way runtests.py deals with this is forking before loading a module.
>> However, this does not currently work with the "xmlrunner" which we use on
>> Hudson, so we let all tests run in a single process there.
>>
>
>
> Btw when running plain python code with generators total ref counter
> doesn't get back to initial value.
> I tried to trace scope and generator destructors and they are run as
> expected. So I'm not sure about leaks in generators.
>

Recently I've found that pyregr.test_dict (test_mutatingiteration)
test makes it slow:

def test_mutatingiteration():
d = {}
d[1] = 1
for i in d:
print i
d[i+1] = 1

test_mutatingiteration()


In CPython this code raises: RuntimeError: dictionary changed size
during iteration
And in Cython you have infinite loop. So we can disable this test for now.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hudson pyregr testing takes too long

[Vitja Makarov]

2011/5/5 Stefan Behnel :
> Vitja Makarov, 05.05.2011 08:41:
>>
>> Recently I've found that pyregr.test_dict (test_mutatingiteration)
>> test makes it slow:
>>
>> def test_mutatingiteration():
>>     d = {}
>>     d[1] = 1
>>     for i in d:
>>         print i
>>         d[i+1] = 1
>>
>> test_mutatingiteration()
>>
>>
>> In CPython this code raises: RuntimeError: dictionary changed size
>> during iteration
>> And in Cython you have infinite loop. So we can disable this test for now.
>
> Interesting. Yes, if it's a bug in Cython then it's worth disabling the test
> until the bug is fixed.
>
> Anyway, I'll try to come up with a fix quickly.
>

Cool! It seems that you've fixed the issue and now pyregr takes about
40 minutes )

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Git workflow, branches, pull requests

[Vitja Makarov]

2011/5/6 Robert Bradshaw :
> I don't like the default to be "don't pull from me"--I'd rather there
> be some convention to indicate a branch is being used as a queue.
> Maybe even foo-queue, or a leading underscore if people like that.
>
> On Thu, May 5, 2011 at 2:03 PM, Dag Sverre Seljebotn
>  wrote:
>> Yes, that is the only time it happens.
>>
>> Do we agree on a) ask before you pull anything that is not in cython/* (ie
>> in private repos), b) document it in hackerguide?
>>
>> DS
>>
>>
>> --
>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>
>> Robert Bradshaw  wrote:
>>>
>>> On Thu, May 5, 2011 at 1:22 PM, Stefan Behnel  wrote:
>>> > Dag Sverre Seljebotn, 05.05.2011 21:52: >> >> There was just a messup in
>>> git history: Mark's OpenMP pull request got >> merged twice; all commits
>>> show up two times. > > What (I think) happened, was that Vitja pulled in
>>> Mark's changes into his > unreachable code removal branch, and they ended up
>>> in his pull request. I > guess I was assuming that git wouldn't care too
>>> much about branch > duplication, so I just accepted the pull request via the
>>> web interface. > Apparently, it did care. > > I tend to rebase my local
>>> change sets before pushing them, and I think it > makes sense to continue
>>> doing that. +1, I think for as-yet-unpublished changes, it makes the most
>>> sense to rebase, but for a longer-term branch, merging isn't as disruptive
>>> to the history (in fact is probably more reflective of what's going on) and
>>> is much better than duplication. To clarify, is this only a problem when we
>>> have A cloned from master B cloned from A (or from master and then pulls in
>>> A) A rebases A+B merged into master ? If this is the case, then we could
>>> simply make the rule that you should ask before hacking a clone atop
>>> anything but master. (Multiple people can share a repeatedly-rebased branch,
>>> right.) We could also us the underscore (or another) convention to mean
>>> "this branch is being used as a queue, puller beware." Surely other projects
>>> have dealt with this. - Robert


About my branch:

I've rebased it from upstream/master at home and made "forced push"
At work I pulled it back and rebased from origin, then I tried to
rebase if again from upstream/master

Guess I was wrong somewhere. So I've lost two latest commits
(generators related fix)
Sometimes it's much easy to branch from upstream and then make
cherry-pick (manual rebase).

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Git workflow, branches, pull requests

[Vitja Makarov]

2011/5/6 Dag Sverre Seljebotn :
> On 05/06/2011 08:20 AM, Vitja Makarov wrote:
>>
>> 2011/5/6 Robert Bradshaw:
>>>
>>> I don't like the default to be "don't pull from me"--I'd rather there
>>> be some convention to indicate a branch is being used as a queue.
>>> Maybe even foo-queue, or a leading underscore if people like that.
>>>
>>> On Thu, May 5, 2011 at 2:03 PM, Dag Sverre Seljebotn
>>>   wrote:
>>>>
>>>> Yes, that is the only time it happens.
>>>>
>>>> Do we agree on a) ask before you pull anything that is not in cython/*
>>>> (ie
>>>> in private repos), b) document it in hackerguide?
>>>>
>>>> DS
>>>>
>>>>
>>>> --
>>>> Sent from my Android phone with K-9 Mail. Please excuse my brevity.
>>>>
>>>> Robert Bradshaw  wrote:
>>>>>
>>>>> On Thu, May 5, 2011 at 1:22 PM, Stefan Behnel
>>>>>  wrote:
>>>>>>
>>>>>> Dag Sverre Seljebotn, 05.05.2011 21:52:>>  >>  There was just a messup
>>>>>> in
>>>>>
>>>>> git history: Mark's OpenMP pull request got>>  merged twice; all
>>>>> commits
>>>>> show up two times.>  >  What (I think) happened, was that Vitja pulled
>>>>> in
>>>>> Mark's changes into his>  unreachable code removal branch, and they
>>>>> ended up
>>>>> in his pull request. I>  guess I was assuming that git wouldn't care
>>>>> too
>>>>> much about branch>  duplication, so I just accepted the pull request
>>>>> via the
>>>>> web interface.>  Apparently, it did care.>  >  I tend to rebase my
>>>>> local
>>>>> change sets before pushing them, and I think it>  makes sense to
>>>>> continue
>>>>> doing that. +1, I think for as-yet-unpublished changes, it makes the
>>>>> most
>>>>> sense to rebase, but for a longer-term branch, merging isn't as
>>>>> disruptive
>>>>> to the history (in fact is probably more reflective of what's going on)
>>>>> and
>>>>> is much better than duplication. To clarify, is this only a problem
>>>>> when we
>>>>> have A cloned from master B cloned from A (or from master and then
>>>>> pulls in
>>>>> A) A rebases A+B merged into master ? If this is the case, then we
>>>>> could
>>>>> simply make the rule that you should ask before hacking a clone atop
>>>>> anything but master. (Multiple people can share a repeatedly-rebased
>>>>> branch,
>>>>> right.) We could also us the underscore (or another) convention to mean
>>>>> "this branch is being used as a queue, puller beware." Surely other
>>>>> projects
>>>>> have dealt with this. - Robert
>>
>>
>> About my branch:
>>
>> I've rebased it from upstream/master at home and made "forced push"
>> At work I pulled it back and rebased from origin, then I tried to
>> rebase if again from upstream/master
>
> Do I understand correctly that you:
>
>  a) You make local changes at home
>  b) Rebase them on cython/master
>  c) Force-push to vitja/somebranch
>  d) Go to work, where you have other local changes
>  e) Rebase your work changes at work on top of vitja/somebranch
>


Right.

> If this is correct; then this can't work. The reason is that after the
> force-push in c), there are no shared commits (apart from what's shared from
> cython/master) between your work computer and vitja/somebranch.
>
> So the rule is: If you rebase a branch, then if you have other copies of
> that branch (like on a work computer), destroy them (e.g., git branch -D)!
>  And then fetch new copies of the branches.
>
> (And as you say, if you do have different changes in many places then you
> can recover from an unfortunate rebase by cherry-picking. And you can always
> undo a rebase by looking at "git reflog" and manually check out the old
> HEAD.)
>

Thank you for explanation.

So btw, when I do rebase and my changes were already pushed I have to
use forced push.
Is forced push ok?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] buffers and flow-control

[Vitja Makarov]

I've never been using buffers so my question is:

Should uninitialized buffer access raise UnboundLocalVariable error?

Like this:

def foo():
 cdef object bar
 print bar

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Scoped expressions and generators problem

[Vitja Makarov]

I don't know how to handle scopedexprs in generators, here are some examples:

[(yield i) for i in l] or [i for i in yield]

In Python3 language mode scoped expression is created for list
comprehension and loop variable is moved there.
So now it isn't stored inside generator closure and is lost between yields.

Btw there is one more problem I hope that's easy to solve:
yield expression inside scoped expression is counted twice.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] nonecheck directive

[Vitja Makarov]

2011/5/22 Stefan Behnel :
> Dag Sverre Seljebotn, 21.05.2011 09:07:
>>
>> On 05/21/2011 07:57 AM, Stefan Behnel wrote:
>>>
>>> In the future, I think we should be more careful with potentially
>>> harmful options, and always prefer safety over speed - *especially* when
>>> we know that the safe way will improve at some point.
>>
>> There wasn't a point where anybody wasn't careful about this;
>
> Sorry if I sounded offensive. It just felt too wrong from today's POV
> (which, I hope, is properly reflected by my paragraph above).
>
>
>> it is simply
>> something that was inherited from Pyrex. The nonecheck directive came much
>> later.
>
> Well, it wouldn't have been the first time we change the default behaviour
> from the way Pyrex originally worked to something we deem more correct.
> Looks like we missed that opportunity back then.
>
> I think Vitja's "uninitialised" branch is our way out of this.
>

Now it's called "_control_flow" btw.
Now None initialization is removed and everything seems to work.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] local variable handling in generators

[Vitja Makarov]

2011/5/22 Stefan Behnel :
> Hi,
>
> I've been looking at the nqueens benchmark for a while, and I think it's
> actually not that a bad benchmark for generators.
>
> http://hg.python.org/benchmarks/file/tip/performance/bm_nqueens.py
>
> A better implementation only for Py2.7/Py3 is here:
>
> https://github.com/cython/cython/blob/master/Demos/benchmarks/nqueens.py
>
> Cython currently runs the first implementation about as fast as Py3.3:
>
> https://sage.math.washington.edu:8091/hudson/job/cython-devel-pybenchmarks-py3k/lastSuccessfulBuild/artifact/chart.html
>
> and the second one more than 3x as fast:
>
> https://sage.math.washington.edu:8091/hudson/view/bench/job/cython-devel-cybenchmarks-py3k/lastSuccessfulBuild/artifact/chart.html
>
> However, I think there's still some space for improvements, and local
> variables are part of that. For generator functions that do non-trivial
> things between yields, I think that local variables will quickly become a
> bottleneck. Currently, they are always closure fields, so any access to them
> will use a pointer indirection to a foreign struct, originally passed in as
> an argument to the function. Given that generators often do Python object
> manipulation through C-API calls, any such call will basically require the C
> compiler to assume that all values in the closure may have changed, thus
> disabling any optimisations for them. The same applies to many other object
> related operations or pointer operations (even DECREF!), as the C compiler
> cannot know that the generator function owns the closure during its lifetime
> exclusively.
>
> I think it would be worth changing the current implementation to use local C
> variables for local Cython variables in the generator, and to copy the
> values back into/from the closure around yields. I'd even let local Python
> references start off as NULL when the generator is created, given that
> Vitek's branch can eliminate None initialisations now.
>
> I started looking into this a bit, but it's not a quick change. The main
> problem I see is the current code duplication between the generator body
> node and the DefNode. It would be better if both could share more code.
> Basically, if local variables become truly local, the only differences will
> be that they come from call arguments in one case and from the closure in
> the other, and that generators have the additional jump-to-yield entry step.
> Everything else could hopefully be identical.
>
> Once again, this goes hand in hand with the still pending DefNode
> refactoring...
>

With live variable analysis that should be easy to save/restore only
active variables at the yield point.
Btw now only reaching definitions analysis is implemented. I'm going
to optimize by replacing sets with bitsets. And then try to implement
live varaiables.

I'm going to delete variable reference using active variable info, but
that could introduce small incompatiblity with CPython:
a = X
print a # <- a will be decrefed here
print 'the end'

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] local variable handling in generators

[Vitja Makarov]

2011/5/23 Stefan Behnel :
> Vitja Makarov, 23.05.2011 10:13:
>>
>> With live variable analysis that should be easy to save/restore only
>> active variables at the yield point.
>
> "Active" in the sense of "modified", I suppose? That's what I was expecting.
>

Active means that variable value will be used. In my example after
'print a' a isn't used anymore.

>
>> Btw now only reaching definitions analysis is implemented. I'm going
>> to optimize by replacing sets with bitsets. And then try to implement
>> live varaiables.
>>
>> I'm going to delete variable reference using active variable info, but
>> that could introduce small incompatiblity with CPython:
>> a = X
>> print a #<- a will be decrefed here
>> print 'the end'
>
> That incompatibility is not small at all. It breaks this code:
>
>    x = b'abc'
>    cdef char* c = x
>
> Even if 'x' is no longer used after this point, it *must not* get freed
> before 'c' is going away as well. That's basically impossible to decide, as
> users may pass 'c' into a function that stores it away for alter use.
>

Yeah. That's hard to detect. But x could be marked as "don't decref
when not-active"

> I'm fine with deallocating variables that are no longer used after the user
> explicitly assigned None to them (i.e. replace the None assignment by a
> simple "DECREF + set to NULL" in that case). I don't think we should be
> doing more than that.
>

Hmm. Why should that be NULL if user sets it to None?


For instance:

for i in args:
print i

this code will be translated into:

PyObject *i = NULL;

for (;;)
{
   tmp = next();
   if (!tmp) break;

  Pyx_XDECREF(i);
  i = tmp;
  print(i);
}

using active variables information this could be translated into:

PyObject *i = NULL;

for (;;)
{
   tmp = next();
   if (!tmp) break;

  i = tmp;
  print(i);
  Pyx_DECREF(i);
}


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] local variable handling in generators

[Vitja Makarov]

2011/5/23 Stefan Behnel :
> Vitja Makarov, 23.05.2011 10:50:
>>
>> 2011/5/23 Stefan Behnel:
>>>
>>> Vitja Makarov, 23.05.2011 10:13:
>>>>
>>>> With live variable analysis that should be easy to save/restore only
>>>> active variables at the yield point.
>>>
>>> "Active" in the sense of "modified", I suppose? That's what I was
>>> expecting.
>>
>> Active means that variable value will be used. In my example after
>> 'print a' a isn't used anymore.
>
> That's not correct then. In a generator, a modified value must be kept alive
> over a yield, even if it is no longer used afterwards.
>
> We can safely reduce the write-back code to modified values, but we cannot
> reduce it to values to that will be used later on.
>

I'm not sure how to get modified variables list at yield point. Now I
only know which assignments reach yield point.


>
>>>> Btw now only reaching definitions analysis is implemented. I'm going
>>>> to optimize by replacing sets with bitsets. And then try to implement
>>>> live varaiables.
>>>>
>>>> I'm going to delete variable reference using active variable info, but
>>>> that could introduce small incompatiblity with CPython:
>>>> a = X
>>>> print a #<- a will be decrefed here
>>>> print 'the end'
>>>
>>> That incompatibility is not small at all. It breaks this code:
>>>
>>>    x = b'abc'
>>>    cdef char* c = x
>>>
>>> Even if 'x' is no longer used after this point, it *must not* get freed
>>> before 'c' is going away as well. That's basically impossible to decide,
>>> as
>>> users may pass 'c' into a function that stores it away for alter use.
>>
>> Yeah. That's hard to detect. But x could be marked as "don't decref
>> when not-active"
>
> How would you know that it needs to be marked like that? You won't
> necessarily see it in the code that a pointer was taken from the value, that
> might have happened within a called function.
>
>
>>> I'm fine with deallocating variables that are no longer used after the
>>> user
>>> explicitly assigned None to them (i.e. replace the None assignment by a
>>> simple "DECREF + set to NULL" in that case). I don't think we should be
>>> doing more than that.
>>
>> Hmm. Why should that be NULL if user sets it to None?
>
> Because there is no user visible difference. None will always be available,
> even if the Cython code no longer holds a reference to it. So changing "x =
> None" into "Py_DECREF(x); x=NULL" is just fine, as long as we can make sure
> 'x' is never accessed after this point.
>
>
>> For instance:
>>
>> for i in args:
>>     print i
>>
>> this code will be translated into:
>>
>> PyObject *i = NULL;
>>
>> for (;;)
>> {
>>    tmp = next();
>>    if (!tmp) break;
>>
>>   Pyx_XDECREF(i);
>>   i = tmp;
>>   print(i);
>> }
>>
>> using active variables information this could be translated into:
>>
>> PyObject *i = NULL;
>>
>> for (;;)
>> {
>>    tmp = next();
>>    if (!tmp) break;
>>
>>   i = tmp;
>>   print(i);
>>   Pyx_DECREF(i);
>> }
>
> That's not correct, though. Python semantics dictate that 'i' must keep its
> value until the end of the function or until it's being reassigned to,
> whatever comes first. Remember that objects can have deallocators in Python.
> That must not be called at an undefined point.
>
> The only thing that can safely be special cased is None. It's common in
> Python code to set a variable to None when the value is worth being
> deallocated (e.g. a large data structure). Cython can optimise this as I
> indicated above.
>

Ohh, I see that variable references couldn't be simply removed.

Unused result reference removal seems safe to me:

a = foo() # a will be assigned to NULL here
print

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] local variable handling in generators

[Vitja Makarov]

2011/5/23 Robert Bradshaw :
>>
>> Then again, "del x" would be a more obvious way to spell this ...
>
> And then setting it to NULL would actually be correct.
>

That is already done ;) But usually you wouldn't use del statement in your code.

> In any case, I'm -1 to deleting references once their no longer used,
> we must wait 'till the end of the function, and I'm not sure the
> savings would be that great either.
>

Ok. Example with char * makes me think that this is not a good idea.

> In terms of packing/unpacking the variables onto the local C stack,
> until we have live variable analysis (and even then) it may still be
> more expensive in some cases than just leaving it there, right?
>

We can try, that isn't hard to implement and compare.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] local variable handling in generators

[Vitja Makarov]

2011/5/23 Stefan Behnel :
>
> However, once we really know which values change between yield calls, i.e.
> which ones need to be stored away, it will actually be less expensive in
> most cases. We currently pay the indirection penalty for each access, even
> read access, whereas the C compiler can then keep important variables in
> registers and only write them back once per yield.
>

I think that all not NULL variables should be saved/restored inside yield.
I can not really track changes only assignments.

for i in a:
yield i  # j is NULL here
for j in b:
yield j # a, b, i ,j should be saved/restored

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] CF and finally clause

[Vitja Makarov]

Hi!

When I create control flow graph I have to visit finally clause twice.
This is required because of different outputs should be generated for
success and exception case:

try:
a = might_raise()
finally:
pass # 'a' might be uninitialized here
print(a)  # and definitely defined here

So after tracking both cases I have to merge states back because same
code is generated for each.

Maybe it's a good idea to split success/exception case by copying it
at PostParse transform?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] subclassing list

[Vitja Makarov]

cdef class Foo(list):
pass

def foo():
f = Foo()
f.append(1)

gcc -g3 -fPIC -I/usr/include/python2.6 -g3 -W -Wall-c -o lll.o lll.c
lll.c: In function ‘__pyx_pf_3lll_foo’:
lll.c:468: error: ‘struct __pyx_obj_3lll_Foo’ has no member named ‘None’

Is this a known bug?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Small issue with buffers

[Vitja Makarov]

Hi!

Trying to fix some tests broken by CF I've found this issue:

def fuu(object[int] b1):
cdef object[int] b2
print b1

buff.c: In function ‘__pyx_pf_4buff_fuu’:
buff.c:504: error: ‘__pyx_bstruct_b2’ undeclared (first use in this function)
buff.c:504: error: (Each undeclared identifier is reported only once
buff.c:504: error: for each function it appears in.)
buff.c: In function ‘__Pyx_BufFmt_TypeCharToAlignment’:
buff.c:809: warning: unused parameter ‘is_complex’

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Broken tests

[Vitja Makarov]

2011/5/27 Stefan Behnel :
> Hi,
>
> could we please make it a requirement to run the complete test suite after a
> branch merge? Or at least taking a close look at Jenkins after pushing a
> merge? And then fixing the bugs it finds?
>
> Stefan
>

There are failing tests in _control_flow branch.
I guess it's better to fix them after merge.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Broken tests

[Vitja Makarov]

2011/5/27 Dag Sverre Seljebotn :
> On 05/27/2011 02:41 PM, Vitja Makarov wrote:
>>
>> 2011/5/27 Stefan Behnel:
>>>
>>> Hi,
>>>
>>> could we please make it a requirement to run the complete test suite
>>> after a
>>> branch merge? Or at least taking a close look at Jenkins after pushing a
>>> merge? And then fixing the bugs it finds?
>>>
>>> Stefan
>>>
>>
>> There are failing tests in _control_flow branch.
>> I guess it's better to fix them after merge.
>
> Wouldn't it be better to rebase _control_flow on master, then fix tests,
> then merge it into master?
>
> (Replace "rebase" with "merge in master" if you wish, I'm going by the _
> prefix here.)
>

Sure. But I think that I need some help to make decision which tests
should be fixed and which removed.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Control Flow

[Vitja Makarov]

I've recently fixed some issues:

 - closure variables were not tracked
 - scoped expression variables were initialized to None

So, I should fix broken tests, here are some of them:

nogil (should be fixed)
purecdef (upstream)
cfunc_directive_in_pyclass (upstream)
bufaccess_noassignT444 (should be removed?)
bufaccess (should be fixed)
ass2longlong (should be fixed)
builtinfuncs (should be fixed)
extcmethcall (should be fixed)
extcoerce (fixed or removed?)
e_extweakref (should be fixed)

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Redundant Cython exception message strings

[Vitja Makarov]

2011/5/28 Robert Bradshaw :
> On Fri, May 27, 2011 at 3:32 PM, Stefan Behnel  wrote:
>> Hi,
>>
>> I recently stumbled over a tradeoff question with AttributeError, and now
>> found the same situation for UnboundLocalError in Vitja's control flow
>> branch. So here it is.
>>
>> When we raise an exception several times in different parts of the code with
>> a message that only differs slightly each time (usually something like
>> "'NoneType' has no attribute X", or "local variable X referenced before
>> assignment"), we have three choices to handle this:
>>
>> 1) Optimise for speed: create a Python string object at module
>> initialisation time and call PyErr_SetObject(exc_type, msg_str_obj).
>>
>> 2) Current way: let CPython create the string object when raising the
>> exception and just call PyErr_SetString(exc_type, "complete message").
>>
>> 3) Trade speed for size and allow the C compiler to reduce the storage
>> redundancy: write only the message template and the names as C char*
>> constants by calling PyErr_Format(exc_type, "message template %s", "X").
>>
>> Assuming that exceptions should be exceptional, I'm leaning towards 3). This
>> would allow the C compiler to collapse multiple usages of the same C string
>> into one data constant, thus reducing a bit of redundancy in the shared
>> library size and the memory footprint. However, it would (slightly?) slow
>> down the exception raising due to the additional string formatting, even
>> when compared to the need to build a Python string object that it shares
>> with 2). While 1) would obviously be the fastest way to raise an exception
>> (no memory allocation, only refcounting), I think it's not worth it for
>> exceptions as it increases both the runtime memory overhead and the module
>> startup time.
>>
>> Thoughts?
>
> Any back-of-the-envelope calculations on how much the savings would
> be? I think I'm leaning towards 3 as well, certainly not option 1.
>

For UnboundLocalError and NameError I used 2) way:

https://github.com/vitek/cython/commit/1fe86b85d965753244cd09db38b1089b40f09a58

So maybe I should add functions like __Pyx_RaiseUnboundLocalError and
__Pyx_RaiseClosureNameError that will  use 3) way.
How do you like put_error_if_unbound CCodeWriter method is that right
place for it?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control Flow

[Vitja Makarov]

2011/5/28 Robert Bradshaw :
> On Fri, May 27, 2011 at 7:27 AM, Vitja Makarov  
> wrote:
>> I've recently fixed some issues:
>>
>>  - closure variables were not tracked
>>  - scoped expression variables were initialized to None
>>
>> So, I should fix broken tests, here are some of them:
>>
>> nogil (should be fixed)
>> purecdef (upstream)
>> cfunc_directive_in_pyclass (upstream)
>> bufaccess_noassignT444 (should be removed?)
>> bufaccess (should be fixed)
>> ass2longlong (should be fixed)
>> builtinfuncs (should be fixed)
>> extcmethcall (should be fixed)
>> extcoerce (fixed or removed?)
>
> Unless you see this code being exercised elsewhere (a run test would
> be better her anyways) it's an easy fix that should be done.
>
>> e_extweakref (should be fixed)
>>

I've fixed all the tests. Now I see this strange error in py2.4:

https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py24-c/lastCompletedBuild/testReport/(root)/CythonCompileTestCase/compiling__c__w_uninitialized_py3/

*
compiling (c) w_uninitialized_py3
=== Expected errors: ===
7:11: local variable 'i' referenced before assignment
13:11: local variable 'i' referenced before assignment


=== Got errors: ===
7:9: undeclared name not builtin: print
7:11: local variable 'i' referenced before assignment
13:11: local variable 'i' referenced before assignment  


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control Flow

[Vitja Makarov]

2011/5/28 Stefan Behnel :
> Vitja Makarov, 28.05.2011 11:48:
>>
>> 2011/5/28 Robert Bradshaw:
>>>
>>> On Fri, May 27, 2011 at 7:27 AM, Vitja Makarov wrote:
>>>>
>>>> I've recently fixed some issues:
>>>>
>>>>  - closure variables were not tracked
>>>>  - scoped expression variables were initialized to None
>>>>
>>>> So, I should fix broken tests, here are some of them:
>>>>
>>>> nogil (should be fixed)
>>>> purecdef (upstream)
>>>> cfunc_directive_in_pyclass (upstream)
>>>> bufaccess_noassignT444 (should be removed?)
>>>> bufaccess (should be fixed)
>>>> ass2longlong (should be fixed)
>>>> builtinfuncs (should be fixed)
>>>> extcmethcall (should be fixed)
>>>> extcoerce (fixed or removed?)
>>>
>>> Unless you see this code being exercised elsewhere (a run test would
>>> be better her anyways) it's an easy fix that should be done.
>>>
>>>> e_extweakref (should be fixed)
>>>>
>>
>> I've fixed all the tests.
>
> Cool.
>
>
>> Now I see this strange error in py2.4:
>>
>>
>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py24-c/lastCompletedBuild/testReport/(root)/CythonCompileTestCase/compiling__c__w_uninitialized_py3/
>>
>> *
>> compiling (c) w_uninitialized_py3
>> === Expected errors: ===
>> 7:11: local variable 'i' referenced before assignment
>> 13:11: local variable 'i' referenced before assignment
>>
>>
>> === Got errors: ===
>> 7:9: undeclared name not builtin: print
>> 7:11: local variable 'i' referenced before assignment
>> 13:11: local variable 'i' referenced before assignment
>
> I guess that's
>
> http://trac.cython.org/cython_trac/ticket/69
>
> In short, print() still isn't supported before Py2.6. I'd just disable that
> test in older Python versions for now.
>

I've replaced print() with dummy ref(obj) function


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control Flow

[Vitja Makarov]

2011/5/28 Stefan Behnel :
> Vitja Makarov, 28.05.2011 13:48:
>>
>> 2011/5/28 Stefan Behnel:
>>>
>>> Vitja Makarov, 28.05.2011 11:48:
>>>>
>>>> 2011/5/28 Robert Bradshaw:
>>>>>
>>>>> On Fri, May 27, 2011 at 7:27 AM, Vitja Makarov wrote:
>>>>>>
>>>>>> I've recently fixed some issues:
>>>>>>
>>>>>>  - closure variables were not tracked
>>>>>>  - scoped expression variables were initialized to None
>>>>>>
>>>>>> So, I should fix broken tests, here are some of them:
>>>>>>
>>>>>> nogil (should be fixed)
>>>>>> purecdef (upstream)
>>>>>> cfunc_directive_in_pyclass (upstream)
>>>>>> bufaccess_noassignT444 (should be removed?)
>>>>>> bufaccess (should be fixed)
>>>>>> ass2longlong (should be fixed)
>>>>>> builtinfuncs (should be fixed)
>>>>>> extcmethcall (should be fixed)
>>>>>> extcoerce (fixed or removed?)
>>>>>
>>>>> Unless you see this code being exercised elsewhere (a run test would
>>>>> be better her anyways) it's an easy fix that should be done.
>>>>>
>>>>>> e_extweakref (should be fixed)
>>>>>>
>>>>
>>>> I've fixed all the tests.
>>>
>>> Cool.
>>>
>>>
>>>> Now I see this strange error in py2.4:
>>>>
>>>>
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-py24-c/lastCompletedBuild/testReport/(root)/CythonCompileTestCase/compiling__c__w_uninitialized_py3/
>>>>
>>>> *
>>>> compiling (c) w_uninitialized_py3
>>>> === Expected errors: ===
>>>> 7:11: local variable 'i' referenced before assignment
>>>> 13:11: local variable 'i' referenced before assignment
>>>>
>>>>
>>>> === Got errors: ===
>>>> 7:9: undeclared name not builtin: print
>>>> 7:11: local variable 'i' referenced before assignment
>>>> 13:11: local variable 'i' referenced before assignment
>>>
>>> I guess that's
>>>
>>> http://trac.cython.org/cython_trac/ticket/69
>>>
>>> In short, print() still isn't supported before Py2.6. I'd just disable
>>> that
>>> test in older Python versions for now.
>>>
>>
>> I've replaced print() with dummy ref(obj) function
>
> Ok, I think we're done then, right? Anything else to do before merging it
> back?
>

I hope so ;)


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control Flow

[Vitja Makarov]

2011/5/28 Stefan Behnel :
> Vitja Makarov, 28.05.2011 14:12:
>>
>> 2011/5/28 Stefan Behnel:
>>>
>>> Ok, I think we're done then, right? Anything else to do before merging it
>>> back?
>>
>> I hope so ;)
>
> I gave your branch a test with lxml and it spit out a couple of missing
> initialisation warnings. Most of them were due to the usage of C out
> arguments, e.g.
>
>    cdef Py_ssize_t step
>    ...
>    _PyEval_SliceIndex(sliceobject.step, &step)
>
> Here, it doesn't matter that "step" isn't initialised. I guess it's still ok
> to get a warning, since it's not obvious that "step" will always be set by
> the function call.
>
> Similarly:
>
>    cdef void _receiveXSLTError(void* c_log_handler, char* msg, ...) nogil:
>        cdef cstd.va_list args
>        ...
>        cstd.va_start(args, msg)
>
> Cython now tells me that "args" isn't initialised. That's true again,
> although this is an even more special case here. I guess I'll have to live
> with that as well for now, maybe until there's dedicated va_list support in
> Cython.
>
> Next, I got a crash in a part of code where I was manually releasing the
> GIL, but only conditionally, and Cython figured out that a variable wasn't
> initialised in one case and injected an exception. I would have liked to get
> a compile time warning here, but didn't. I assume that's because it was only
> a "may be used uninitialised", not a "is used uninitialised"?
>
> I'm currently looking through the other issues. A major problem is that I
> often relied on Cython setting uninitialised variables to None, especially
> in performance critical code where I wanted to avoid double initialisations.
> Cython is smarter now, so this needs fixing in the code. I'm sure I'm not
> the only one who'll run into that, so more warnings would be very helpful.
>

Do you run cython as "cython -Wextra"?
 -Wextra enables "maybe uninitialized" warning and unused entry.
By default error is generated for uninitialized pyobject variables and
warning for ctypes.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control Flow

[Vitja Makarov]

2011/5/28 Stefan Behnel :
> Stefan Behnel, 28.05.2011 18:06:
>>
>> Let's see how much of Sage we break.
>
> ... quite a bit. From a superficial glance, there seem to be some real bugs,
> but most of the failures and warnings are most likely just uninitialised
> variables that are expected to be None.
>
> https://sage.math.washington.edu:8091/hudson/view/All/job/sage-build/740/consoleFull
>


I've started new branch called control_flow_fixes.
I've fixed the issue with RaiseStatNode. And unused variables that
were not assigned or referenced now you get correct message.

And I started to look inside sage problems.

Error compiling Cython file:

...
# allow lambda functions

if self._nrows==0 or self._ncols==0:
return self.__copy__()
v = [z.derivative(var) for z in self.list()]
if R is None:
^


sage/matrix/matrix_dense.pyx:451:13: local variable 'R' referenced
before assignment


I've looked into the sources and I think as cython is right here. 'R'
is allways NULL at that point.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control Flow

[Vitja Makarov]

2011/5/29 Stefan Behnel :
> Vitja Makarov, 29.05.2011 10:11:
>>
>> 2011/5/28 Stefan Behnel:
>>>
>>> Stefan Behnel, 28.05.2011 18:06:
>>>>
>>>> Let's see how much of Sage we break.
>>>
>>> ... quite a bit. From a superficial glance, there seem to be some real
>>> bugs,
>
> Bugs in Sage, to be clear.
>

Ok.

>
>>> but most of the failures and warnings are most likely just uninitialised
>>> variables that are expected to be None.
>>>
>>>
>>> https://sage.math.washington.edu:8091/hudson/view/All/job/sage-build/740/consoleFull
>>
>>
>> I've started new branch called control_flow_fixes.
>
> I think you can safely apply the subsequent fixes to the master branch, now
> that the feature is in. That will also give you broader feedback from
> Jenkins.
>
>

I don't have write access to upstream repo.
Btw there is one more commit in control_flow_fixes branch with few new tests.

>> I've fixed the issue with RaiseStatNode. And unused variables that
>> were not assigned or referenced now you get correct message.
>
> Cool.
>

What about "Error running command, failed with status 256." lines in
sage build log?
Is that cython crash?


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control Flow

[Vitja Makarov]

2011/5/29 Stefan Behnel :
> Vitja Makarov, 29.05.2011 11:17:
>>
>> 2011/5/29 Stefan Behnel:
>>>
>>> Vitja Makarov, 29.05.2011 10:11:
>>>>
>>>> I've started new branch called control_flow_fixes.
>>>
>>> I think you can safely apply the subsequent fixes to the master branch,
>>> now
>>> that the feature is in. That will also give you broader feedback from
>>> Jenkins.
>>
>> I don't have write access to upstream repo.
>
> Ah, sorry. I added you to the project members.
>
>
>> Btw there is one more commit in control_flow_fixes branch with few new
>> tests.
>
> That makes a good test for your write access then. ;)
>

Thanks. I've successfully pushed my changes.
I removed old ControlFlow stuff completely.


>
>> What about "Error running command, failed with status 256." lines in
>> sage build log?
>> Is that cython crash?
>
> Don't think so. Not sure what those failures are, but given that some
> modules fail to compile, maybe it's just the late result of such a build
> failure.
>


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control Flow

[Vitja Makarov]

2011/5/29 Vitja Makarov :
> 2011/5/29 Stefan Behnel :
>> Vitja Makarov, 29.05.2011 11:17:
>>>
>>> 2011/5/29 Stefan Behnel:
>>>>
>>>> Vitja Makarov, 29.05.2011 10:11:
>>>>>
>>>>> I've started new branch called control_flow_fixes.
>>>>
>>>> I think you can safely apply the subsequent fixes to the master branch,
>>>> now
>>>> that the feature is in. That will also give you broader feedback from
>>>> Jenkins.
>>>
>>> I don't have write access to upstream repo.
>>
>> Ah, sorry. I added you to the project members.
>>
>>
>>> Btw there is one more commit in control_flow_fixes branch with few new
>>> tests.
>>
>> That makes a good test for your write access then. ;)
>>
>
> Thanks. I've successfully pushed my changes.
> I removed old ControlFlow stuff completely.
>
>
>>
>>> What about "Error running command, failed with status 256." lines in
>>> sage build log?
>>> Is that cython crash?
>>
>> Don't think so. Not sure what those failures are, but given that some
>> modules fail to compile, maybe it's just the late result of such a build
>> failure.
>>


I see sigsegv in lxml tests:

1222/1364 ( 89.6%): test_xmlschema_import_file
(lxml.tests.test_xmlschema.ETreeXMLSchemaTestCase) ... Segmentation
fault
https://sage.math.washington.edu:8091/hudson/job/cython-devel-lxml-trunk-py27/890/console

Is that control flow related issue?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control Flow

[Vitja Makarov]

2011/5/30 Stefan Behnel :
> Vitja Makarov, 30.05.2011 20:53:
>>
>> I see sigsegv in lxml tests:
>>
>> 1222/1364 ( 89.6%): test_xmlschema_import_file
>> (lxml.tests.test_xmlschema.ETreeXMLSchemaTestCase) ... Segmentation
>> fault
>>
>> https://sage.math.washington.edu:8091/hudson/job/cython-devel-lxml-trunk-py27/890/console
>>
>> Is that control flow related issue?
>
> No, don't bother with that one.
>
> It's been crashing at that point for ages and I can't figure out why. It
> only crashes on sage.math, and only when running two of the tests in
> combination (a threading test and the one above). So far, I couldn't
> reproduce this on any other machine, nor is it any clear what may trigger
> this. I'm pretty sure it's not a Cython related bug, though.
>

Ok, according to last 4 builds it's fails 75% so maybe try to run it
under gdb with bt hook?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] wiki.cython.org is down

[Vitja Makarov]

With error:

Service Temporarily Unavailable

The server is temporarily unable to service your request due to
maintenance downtime or capacity problems. Please try again later.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] local variable handling in generators

[Vitja Makarov]

2011/5/24 Stefan Behnel :
> Vitja Makarov, 23.05.2011 21:33:
>>
>> 2011/5/23 Stefan Behnel:
>>>
>>> However, once we really know which values change between yield calls,
>>> i.e.
>>> which ones need to be stored away, it will actually be less expensive in
>>> most cases. We currently pay the indirection penalty for each access,
>>> even
>>> read access, whereas the C compiler can then keep important variables in
>>> registers and only write them back once per yield.
>>>
>>
>> I think that all not NULL variables should be saved/restored inside yield.
>> I can not really track changes only assignments.
>
> I mean "assignments" when I write "changes". When a variable is being
> assigned to, we should assume that its value changed. Users can be expected
> to be smart enough to avoid unnecessary assignments in most cases, there's
> no need to put work into optimising them away automatically.
>
>
>> for i in a:
>>     yield i  # j is NULL here
>> for j in b:
>>     yield j # a, b, i ,j should be saved/restored
>
> Right. However, in general, we can expect that most variables will have been
> initialised on a yield.
>
> We can still avoid storing away C typed variables that are not being used
> later on, because they are not reference counted.
>
> And if the user sets 'a' and 'i' to None between the loops, they won't need
> to be saved on the second yield either. But that's an optimisation, not a
> requirement.
>

What about copying C++ structs and objects with constructor (and/or
overloaded = method)?
That wouldn't work well.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Bindings performance issue

[Vitja Makarov]

Hi!

Is bindings performance issue valuable?

$ cat bindbench.pyx
def wo_bindings():
pass

def outer():
def inner():
pass
return inner
with_bindings = outer()

$ python
>>> import timeit
>>> timeit.repeat('with_bindings()', setup='from bindbench import wo_bindings, 
>>> with_bindings', repeat=1, number=1)
[6.169871807098389]
>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, 
>>> with_bindings', repeat=1, number=1)
[4.609416961669922]

PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
slower for CPython interpreter execution.
As CPython has some optimizations for CFunctions and PyCFunctions.

Does it make sense for us? Or we can easily switch to bindings?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bindings performance issue

[Vitja Makarov]

2011/6/1 mark florisson :
> On 31 May 2011 20:25, Vitja Makarov  wrote:
>> Hi!
>>
>> Is bindings performance issue valuable?
>>
>> $ cat bindbench.pyx
>> def wo_bindings():
>>    pass
>>
>> def outer():
>>    def inner():
>>        pass
>>    return inner
>> with_bindings = outer()
>>
>> $ python
>>>>> import timeit
>>>>> timeit.repeat('with_bindings()', setup='from bindbench import 
>>>>> wo_bindings, with_bindings', repeat=1, number=1)
>> [6.169871807098389]
>>>>> timeit.repeat('wo_bindings()', setup='from bindbench import wo_bindings, 
>>>>> with_bindings', repeat=1, number=1)
>> [4.609416961669922]
>>
>> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
>> slower for CPython interpreter execution.
>> As CPython has some optimizations for CFunctions and PyCFunctions.
>>
>> Does it make sense for us? Or we can easily switch to bindings?
>>
>> --
>> vitja.
>> ___
>> cython-devel mailing list
>> cython-devel@python.org
>> http://mail.python.org/mailman/listinfo/cython-devel
>>
>
> I think switching should be fine, if you'd desperately need the speed
> you'd be calling c(p)def functions from Cython. In fact, when the
> fused cfunction will be ready it will be even slightly slower, as it
> overrides the tp_call. But perhaps that should really be made into a
> subclass...
>
> Anyway, would you use these for Python classes and module-level def
> functions (and closures) only, or would you also use them in extension
> classes? Because I found at least a bit of problem there, as extension
> class methods get 'self' passed as the first argument to the C
> function (as PyCFunctionObject.m_self), whereas methods in Python
> classes get 'self' passed in the args tuple (and the m_self is
> unused).
>

Recently I've found a problem with static methods (__new__ for
example) of inner classes (classes created inside a function, for
instance)
I think binding version should be used for all regular def functions
(defs, staticmethods, classmethods and so on).
I also think that it's better to rename binding_PyCFunction_Type to
something like CyFunctionType and make it more like PyFunction.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bindings performance issue

[Vitja Makarov]

2011/6/2 Robert Bradshaw :
> On Wed, Jun 1, 2011 at 7:26 AM, Vitja Makarov  wrote:
>> 2011/6/1 mark florisson :
>>> On 31 May 2011 20:25, Vitja Makarov  wrote:
>>>> Hi!
>>>>
>>>> Is bindings performance issue valuable?
>>>>
>>>> $ cat bindbench.pyx
>>>> def wo_bindings():
>>>>    pass
>>>>
>>>> def outer():
>>>>    def inner():
>>>>        pass
>>>>    return inner
>>>> with_bindings = outer()
>>>>
>>>> $ python
>>>>>>> import timeit
>>>>>>> timeit.repeat('with_bindings()', setup='from bindbench import 
>>>>>>> wo_bindings, with_bindings', repeat=1, number=1)
>>>> [6.169871807098389]
>>>>>>> timeit.repeat('wo_bindings()', setup='from bindbench import 
>>>>>>> wo_bindings, with_bindings', repeat=1, number=1)
>>>> [4.609416961669922]
>>>>
>>>> PyCBindings makes it 1.3 (difference is about 15ns on my laptop) times
>>>> slower for CPython interpreter execution.
>>>> As CPython has some optimizations for CFunctions and PyCFunctions.
>>>>
>>>> Does it make sense for us? Or we can easily switch to bindings?
>>>>
>>>> --
>>>> vitja.
>>>> ___
>>>> cython-devel mailing list
>>>> cython-devel@python.org
>>>> http://mail.python.org/mailman/listinfo/cython-devel
>>>>
>>>
>>> I think switching should be fine, if you'd desperately need the speed
>>> you'd be calling c(p)def functions from Cython. In fact, when the
>>> fused cfunction will be ready it will be even slightly slower, as it
>>> overrides the tp_call. But perhaps that should really be made into a
>>> subclass...
>
> Yes, this should be a subclass, slowing all calls down is a bad idea.
>
>>> Anyway, would you use these for Python classes and module-level def
>>> functions (and closures) only, or would you also use them in extension
>>> classes? Because I found at least a bit of problem there, as extension
>>> class methods get 'self' passed as the first argument to the C
>>> function (as PyCFunctionObject.m_self), whereas methods in Python
>>> classes get 'self' passed in the args tuple (and the m_self is
>>> unused).
>>>
>>
>> Recently I've found a problem with static methods (__new__ for
>> example) of inner classes (classes created inside a function, for
>> instance)
>> I think binding version should be used for all regular def functions
>> (defs, staticmethods, classmethods and so on).
>
> +1 to moving that way. (Well, static and class methods bind
> differently, right?)
>
>> I also think that it's better to rename binding_PyCFunction_Type to
>> something like CyFunctionType and make it more like PyFunction.
>
> +1
>
> Is there any advantage to the method descriptors (other than that
> they're easier for a human to write?)

Initially bindings was written to support bound class methods (am I right?)
So when we use it for regular functions 'binding' in the name doesn't
reflect its purpose.

One the other hand it's much more easy to write manually.

About staticmethods: I think that CyFunction type should handle it as well
because staticmethods can have custom attributes and act just like
normal def one.

The difference is insde descr_get it should return bound method for
normal and self for staticmethods.

Here I've tried to support staticmethods inside bindings type:
https://github.com/vitek/cython/commit/c0725ab340a8173d8e6724c62be3a135df58980e

>
> I think that small speed regression for better compatibility is OK if
> we add a directive to not create binding functions. (It'd be nice if
> we could work around it, but that's hard as CPython has special
> hooks...) The bigger issue is that the binding behavior is backwards
> incompatible, and perhaps in a subtle way. Perhaps we need to have a
> phase where all currently non-binding functions that will become
> binding functions will raise an error (or at least a warning) to wean
> people off of the old behavior before making a switch.
>

Is the difference so significant?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bindings performance issue

[Vitja Makarov]

2011/6/2 Robert Bradshaw :
>> Initially bindings was written to support bound class methods (am I right?)
>> So when we use it for regular functions 'binding' in the name doesn't
>> reflect its purpose.
>
> There's three kinds of functions we create: PyCFunctions, binding
> PyCFunctions, and extension class methods (via descriptors). I was
> asking about the latter.
>
> As for the first two, it reflects the difference in behavior. If I
> take the function and assign it to a class, it will bind as a method
> when I go to look it up. This is arguably the biggest difference
> between built-in functions and Python functions.
>
>> One the other hand it's much more easy to write manually.
>>
>> About staticmethods: I think that CyFunction type should handle it as well
>> because staticmethods can have custom attributes and act just like
>> normal def one.
>>
>> The difference is insde descr_get it should return bound method for
>> normal and self for staticmethods.
>
> Yep. Does that merit a new class, or a flag that's checked on every
> function invocation (false 99.9% of the time)?
>

tp_descr_get() is called each time function gets bound, isn't so?
If it makes sens we can create both CyFunction and CyStaticFunction types.

>> Here I've tried to support staticmethods inside bindings type:
>> https://github.com/vitek/cython/commit/c0725ab340a8173d8e6724c62be3a135df58980e
>>
>>>
>>> I think that small speed regression for better compatibility is OK if
>>> we add a directive to not create binding functions. (It'd be nice if
>>> we could work around it, but that's hard as CPython has special
>>> hooks...) The bigger issue is that the binding behavior is backwards
>>> incompatible, and perhaps in a subtle way. Perhaps we need to have a
>>> phase where all currently non-binding functions that will become
>>> binding functions will raise an error (or at least a warning) to wean
>>> people off of the old behavior before making a switch.
>>>
>>
>> Is the difference so significant?
>
> If anyone is assigning a Cython function to an object and then using
> it they're counting on the current non-binding behavior, and it will
> break. The speed is probably a lesser issue, which is what benchmarks
> are for.
>




-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bindings performance issue

[Vitja Makarov]

I've tried that: https://github.com/vitek/cython/compare/master..._bindings

Results are not bad: 168 failing tests for pyregr2.7 and 463 for py3

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bindings performance issue

[Vitja Makarov]

2011/6/4 mark florisson :
> On 4 June 2011 12:24, Vitja Makarov  wrote:
>> I've tried that: https://github.com/vitek/cython/compare/master..._bindings
>>
>> Results are not bad: 168 failing tests for pyregr2.7 and 463 for py3
>
> Nice, it partly duplicates work in my fusedtypes branch but I suppose
> it will have to be reworked anyway into a subclass.

Sure. I've taken some code from your branch ;)

> So this works only
> for normal classes right, i.e. not for extension classes? It's also
> pickle-able, cool :)
>

Yes. It works for python classes and python def functions.
__reduce__ only returns function name, so it's possible to
pickle/unpickle it in some cases
but that was enough to fix some broken tests from pyregr.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] local variable handling in generators

[Vitja Makarov]

2011/5/31 Dag Sverre Seljebotn :
> On 05/31/2011 01:07 PM, Vitja Makarov wrote:
>>
>> 2011/5/24 Stefan Behnel:
>>>
>>> Vitja Makarov, 23.05.2011 21:33:
>>>>
>>>> 2011/5/23 Stefan Behnel:
>>>>>
>>>>> However, once we really know which values change between yield calls,
>>>>> i.e.
>>>>> which ones need to be stored away, it will actually be less expensive
>>>>> in
>>>>> most cases. We currently pay the indirection penalty for each access,
>>>>> even
>>>>> read access, whereas the C compiler can then keep important variables
>>>>> in
>>>>> registers and only write them back once per yield.
>>>>>
>>>>
>>>> I think that all not NULL variables should be saved/restored inside
>>>> yield.
>>>> I can not really track changes only assignments.
>>>
>>> I mean "assignments" when I write "changes". When a variable is being
>>> assigned to, we should assume that its value changed. Users can be
>>> expected
>>> to be smart enough to avoid unnecessary assignments in most cases,
>>> there's
>>> no need to put work into optimising them away automatically.
>>>
>>>
>>>> for i in a:
>>>>     yield i  # j is NULL here
>>>> for j in b:
>>>>     yield j # a, b, i ,j should be saved/restored
>>>
>>> Right. However, in general, we can expect that most variables will have
>>> been
>>> initialised on a yield.
>>>
>>> We can still avoid storing away C typed variables that are not being used
>>> later on, because they are not reference counted.
>>>
>>> And if the user sets 'a' and 'i' to None between the loops, they won't
>>> need
>>> to be saved on the second yield either. But that's an optimisation, not a
>>> requirement.
>>>
>>
>> What about copying C++ structs and objects with constructor (and/or
>> overloaded = method)?
>> That wouldn't work well.
>
> Currently I believe C++ objects are only supported through pointers, which
> of course are copyable. If or when we support C++ objects on the stack, then
> we should definitely avoid copying them.
>
> If you declare a C++ class as a struct Cython-side then I think we should
> assume that we can freely invoke the copy constructor and assignment
> operator in this context.
>

May be it's better to declare struct with all the locals and then copy
it with memcpy()?

Btw cpp still wouldn't love it :(
So probably it's better to look forward for stack switching library...

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Constant folding

[Vitja Makarov]

Hi!

When CF is there it is possible to improve constant folding.
Some NameNode references could be easily replaced by appropriate ConstNode.

Could that help GCC to produce faster code?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Some bugs found while testing cython on django

[Vitja Makarov]

Hi!

Trying to compile django I've found some problems:

1. It seems that cython currently doesn't support tuples inside args definition:

def foo((a, b), c):

Currently this gives missing argument name and crash inside
CreateControlFlow graph

cython -X binding=True django/contrib/gis/forms/fields.py -o
django/contrib/gis/forms/fields.c

2. Concatenating unicode and str isn't supported:

I guess that this is mostly django problem

Error compiling Cython file:

...
default_error_messages = {
'no_geom' : _(u'No geometry value provided.'),
'invalid_geom' : _(u'Invalid geometry value.'),
'invalid_geom_type' : _(u'Invalid geometry type.'),
'transform_error' : _(u'An error occurred when transforming
the geometry '
  'to the SRID of the geometry form field.'),
 ^


django/contrib/gis/forms/fields.py:21:30: Cannot mix string literals
of different types, expected u'', got ''

3.Reraise not inside except clause, here is simple example

def foo():
print 'catched'
raise
try:
raise IndexError
except IndexError:
foo()


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Some bugs found while testing cython on django

[Vitja Makarov]

2011/6/20 Stefan Behnel :
> Vitja Makarov, 19.06.2011 21:37:
>>
>> Trying to compile django I've found some problems:
>>
>> 1. It seems that cython currently doesn't support tuples inside args
>> definition:
>>
>> def foo((a, b), c):
>>
>> Currently this gives missing argument name and crash inside
>> CreateControlFlow graph
>>
>> cython -X binding=True django/contrib/gis/forms/fields.py -o
>> django/contrib/gis/forms/fields.c
>
> It may be valid Py2-only syntax, but I consider its usage a bug in Django.
>

Ok. Any way I should fix crash in FlowControl.

>
>> 2. Concatenating unicode and str isn't supported:
>>
>> I guess that this is mostly django problem
>>
>> Error compiling Cython file:
>> 
>> ...
>>     default_error_messages = {
>>         'no_geom' : _(u'No geometry value provided.'),
>>         'invalid_geom' : _(u'Invalid geometry value.'),
>>         'invalid_geom_type' : _(u'Invalid geometry type.'),
>>         'transform_error' : _(u'An error occurred when transforming
>> the geometry '
>>                               'to the SRID of the geometry form field.'),
>>                              ^
>> 
>>
>> django/contrib/gis/forms/fields.py:21:30: Cannot mix string literals
>> of different types, expected u'', got ''
>
> That's a bug in Django. Worth submitting a patch to their bug tracker. It
> works in this case, but it's not what they want in their code:
>
>    >>> u'abc\u1234'
>    u'abc\u1234'
>    >>> u'abc' '\u1234'
>    u'abc\\u1234'
>

Ok.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bindings performance issue

[Vitja Makarov]

Wow now we have about 11K tests with 171 errors!

https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bindings performance issue

[Vitja Makarov]

2011/6/21 Stefan Behnel :
> Vitja Makarov, 20.06.2011 22:23:
>>
>> Wow now we have about 11K tests with 171 errors!
>>
>>
>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/
>
> Yes, that was a low hanging, high value bug.
>
> """
> Make cyfunction __name__ attribute writable
> """
>
> Could you provide a pull request with that change?
>

Do you mean single change or whole branch?

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bindings performance issue

[Vitja Makarov]

2011/6/21 Vitja Makarov :
> 2011/6/21 Stefan Behnel :
>> Vitja Makarov, 20.06.2011 22:23:
>>>
>>> Wow now we have about 11K tests with 171 errors!
>>>
>>>
>>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/
>>
>> Yes, that was a low hanging, high value bug.
>>
>> """
>> Make cyfunction __name__ attribute writable
>> """
>>
>> Could you provide a pull request with that change?
>>
>
> Do you mean single change or whole branch?
>

This commit
https://github.com/vitek/cython/commit/7b4471bcc5eeb09b2e4851d3a1c2c3d2a5bd085c

disables bindings for inner functions and breaks closure_decorators_T478 test.

I can remove that or fix testcase.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bindings performance issue

[Vitja Makarov]

2011/6/21 Stefan Behnel :
> Vitja Makarov, 21.06.2011 09:07:
>>
>> 2011/6/21 Stefan Behnel:
>>>
>>> Vitja Makarov, 20.06.2011 22:23:
>>>>
>>>> Wow now we have about 11K tests with 171 errors!
>>>>
>>>>
>>>> https://sage.math.washington.edu:8091/hudson/view/cython-vitek/job/cython-vitek-tests-pyregr-py27-c/
>>>
>>> Yes, that was a low hanging, high value bug.
>>>
>>> """
>>> Make cyfunction __name__ attribute writable
>>> """
>>>
>>> Could you provide a pull request with that change?
>>
>> Do you mean single change or whole branch?
>
> Hmm, I didn't look at the other changes in the branch yet. Actually, I
> followed the link in Jenkins and the changeset page didn't even show me
> which branch it was. This one, I guess:
>
> https://github.com/vitek/cython/commits/_bindings
>
> I'll give it a review when I get to it.
>
> Do you think this branch is ready for a merge, or is there more to be done?
>

I think it's almost done. But it's better to review it because lots of
new C code there.
So I can create pull request to make code review easy.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] strange bug in parallel assignment

[Vitja Makarov]

Hi!

Looking at hudson pyregr output I've found this issue:

(a, b), (c,) = (1, 2), {1: 2}

Traceback (most recent call last):
  File "cython.py", line 17, in 
main(command_line = 1)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
821, in main
result = compile(sources, options)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
796, in compile
return compile_multiple(source, options)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
768, in compile_multiple
result = run_pipeline(source, options)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
632, in run_pipeline
err, enddata = context.run_pipeline(pipeline, source)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
251, in run_pipeline
data = phase(data)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
173, in generate_pyx_code
module_node.process_implementation(options, result)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
line 73, in process_implementation
self.generate_c_code(env, options, result)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
line 310, in generate_c_code
self.generate_module_init_func(modules[:-1], env,
globalstate['init_module'])
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
line 1924, in generate_module_init_func
self.body.generate_execution_code(code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
347, in generate_execution_code
stat.generate_execution_code(code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
3811, in generate_execution_code
stat.generate_assignment_code(code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
3709, in generate_assignment_code
self.lhs.generate_assignment_code(self.rhs, code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 4036, in generate_assignment_code
self.generate_parallel_assignment_code(rhs, code)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 4088, in generate_parallel_assignment_code
self.generate_generic_parallel_unpacking_code(code, rhs)
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 4137, in generate_generic_parallel_unpacking_code
item.result(),
  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 2047, in result
print self.pos, self.type, self.temp_cname
AttributeError: 'PyTempNode' object has no attribute 'temp_cname'



-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] strange bug in parallel assignment

[Vitja Makarov]

2011/6/22 Vitja Makarov :
> Hi!
>
> Looking at hudson pyregr output I've found this issue:
>
> (a, b), (c,) = (1, 2), {1: 2}
>
> Traceback (most recent call last):
>  File "cython.py", line 17, in 
>    main(command_line = 1)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 821, in main
>    result = compile(sources, options)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 796, in compile
>    return compile_multiple(source, options)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 768, in compile_multiple
>    result = run_pipeline(source, options)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 632, in run_pipeline
>    err, enddata = context.run_pipeline(pipeline, source)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 251, in run_pipeline
>    data = phase(data)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/Main.py", line
> 173, in generate_pyx_code
>    module_node.process_implementation(options, result)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
> line 73, in process_implementation
>    self.generate_c_code(env, options, result)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
> line 310, in generate_c_code
>    self.generate_module_init_func(modules[:-1], env,
> globalstate['init_module'])
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/ModuleNode.py",
> line 1924, in generate_module_init_func
>    self.body.generate_execution_code(code)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
> 347, in generate_execution_code
>    stat.generate_execution_code(code)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
> 3811, in generate_execution_code
>    stat.generate_assignment_code(code)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/Nodes.py", line
> 3709, in generate_assignment_code
>    self.lhs.generate_assignment_code(self.rhs, code)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
> line 4036, in generate_assignment_code
>    self.generate_parallel_assignment_code(rhs, code)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
> line 4088, in generate_parallel_assignment_code
>    self.generate_generic_parallel_unpacking_code(code, rhs)
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
> line 4137, in generate_generic_parallel_unpacking_code
>    item.result(),
>  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
> line 2047, in result
>    print self.pos, self.type, self.temp_cname
> AttributeError: 'PyTempNode' object has no attribute 'temp_cname'
>
>

Oops. last line is incorrect it should be something like this:

  File "/home/vitja/work/cython-vitek/Cython/Compiler/ExprNodes.py",
line 2050, in result
assert False, "Remember to call allocate/release on TempNode"
AssertionError: Remember to call allocate/release on TempNode

Btw this code leads to the same error:
(a, b), (c,) = (1, 2), set([1])



-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] PEP 380 approved: syntax for generator delegation

[Vitja Makarov]

2011/6/27 Stefan Behnel :
> Hi,
>
> PEP 380 ("yield from") recently went through the approval gate.
>
> http://www.python.org/dev/peps/pep-0380/
>
> The implementation in CPython isn't rounded up yet and lots of tests are
> missing from the test suite. But it's agreed that it'll make it into 3.3,
> and we should try to support it as well. It's a really nice feature.
>

Wow! Interesting thing.
I think that could be implemented but I think we should try to
optimize cython generators first.


-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] Hudson is down?

[Vitja Makarov]

It seems that hudson server is down.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] PEP 3135 -- New Super

[Vitja Makarov]

I was thinking about implementing new super() with no arguments.

The problem is where to store __class__, I see two options here:

1. Add func_class member to CyFunction, this way __class__ will be
private and not visible for inner functions:
2. Put it into closure

Actually, I like the first one.
And I don't think that __class__ should be use somewhere outside super()

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] PEP 3135 -- New Super

[Vitja Makarov]

2011/7/5 Stefan Behnel :
> Vitja Makarov, 05.07.2011 08:21:
>>
>> I was thinking about implementing new super() with no arguments.
>
> Please do :)
>
> If you start working on it, please assign the ticket to you:
>
> http://trac.cython.org/cython_trac/ticket/696
>

Ok, I'll do this if I start.

>
>> The problem is where to store __class__, I see two options here:
>>
>> 1. Add func_class member to CyFunction, this way __class__ will be
>> private and not visible for inner functions:
>> 2. Put it into closure
>
> The second option has the advantage of requiring the field only when super()
> is used, whereas the first impacts all functions.
>
> I would expect that programs commonly have a lot more functions than
> specifically methods that use a no-argument call to super(), so this may
> make a difference.
>

So, now classes are created the following way:

class_dict = {}
class_dict.foo = foo_func
class = CreateClass(class_dict)

So after class is created I should check its dict for CyFunction
members (maybe only ones that actually require __class__)
and set __class__:

for value in class.__dict__.itervalues():
   if isinstance(value, CyFunction) and value.func_class is WantClass:
   value.func_class = class


> OTOH, not all methods have a closure, so creating one just to store the
> "__class__" field is very wasteful, in terms of both runtime and memory
> overhead. A lot more wasteful than paying 8 bytes of memory for each
> function, with no additional time overhead.
>

Going this way it only requires to initialize closure:

closure.func_class = class


Btw, first way requires cyfunction signature change, it would accept
cyfunction object as first argument.
This also could help to solve default args problem.


>
>> And I don't think that __class__ should be use somewhere outside super()
>
> Agreed. CPython simply uses a compile time heuristic ("is there a function
> call to something global named 'super'?") when creating this field, so it's
> strictly reserved for this use case.
>
> BTW, I like the irony in the fact that CPython essentially gives Cython
> semantics to the "super" builtin here, by (partially) evaluating it at
> compile time.
>

Yeah, I think Cython super with no args should be a little bit faster
then classic one.

-- 
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel