Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Sat, Feb 19, 2011 at 06:31:26PM -0500, W. Trevor King wrote: > On Sat, Feb 19, 2011 at 02:04:16PM -0800, Robert Bradshaw wrote: > > On Sat, Feb 19, 2011 at 1:45 PM, W. Trevor King wrote: > > > Ah. Sorry for all the c(p)def/qualifier confusion, but I'm trying to > > > consolidate the way these are handled in Parsing/Nodes/Symtab and I > > > want to make sure I don't implement the wrong interpretation. Can you > > > clarify how one knows if "public" means "expose a read/write Python > > > interface to this object" or "expose this symbol to external C code"? > > > > Public has had several different meanings. I wish there were a spec > > and full grammer for Cython, but there's not (yet?). The meaning is > > implicit in the code, and there's enough users out there that we > > should stay backwards compatible. It may be worth doing some > > backwards-incompatible normalization before we hit 1.0, but compiling > > the entire Python grammar is higher priority than that. > > Since I'm going to have lots of similar stuff (classes, enums, > structs, unions) all with the same (hopefully) cdef/cpdef/visibility > stuff for members, I'd like to consolidate now. I will of course, add > special-case code as necessary to support the current syntax, which > can then be removed whenever you think it is appropriate, but writing > separate, near-identical handlers for each type seems like a recipe > for disaster ;). > > I'll look to the code for guidance on public, and try to work out the > appropriate meaning during the parse phase. I've been working on a more explicit parser that removes the ambiguity behind the various visibilities. This will help me ensure proper impolementation of my cdef-ed enums/structs/..., and make it easier to update visibility syntax in the future. Take a look and let me know if you think this is a useful direction to take: git: http://www.physics.drexel.edu/~wking/code/git/cython.git branch: cdef-enums-stucts-and-unions gitweb: http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;h=refs/heads/cdef-enums-stucts-and-unions -- This email may be signed or encrypted with GPG (http://www.gnupg.org). The GPG signature (if present) will be attached as 'signature.asc'. For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt pgpExMhfTUJMY.pgp Description: PGP signature ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
W. Trevor King, 22.02.2011 18:55:
I've been working on a more explicit parser that removes the ambiguity
behind the various visibilities. This will help me ensure proper
impolementation of my cdef-ed enums/structs/..., and make it easier to
update visibility syntax in the future. Take a look and let me know
if you think this is a useful direction to take:
git: http://www.physics.drexel.edu/~wking/code/git/cython.git
branch: cdef-enums-stucts-and-unions
gitweb:
http://www.physics.drexel.edu/~wking/code/git/gitweb.cgi?p=cython.git;a=log;h=refs/heads/cdef-enums-stucts-and-unions
It doesn't seem like I can leave comments in the gitweb version, so I'll
comment here.
First thing that caught my eyes: I hope you do not intend to leave the
logging usage in the parser. This is seriously performance critical code
that Cython compiles down to pretty fast C code. Note that you can use a
debugger and Python's profiling/tracing interface to find out what's
happening, without code impact.
Some of the log statements span more than one line, which makes it trickier
to strip them out with sed&friends (but backing out the initial changeset
would likely make it simple enough to remove the rest manually).
Also note that it's best to write runnable tests ("tests/run/"). The tests
in "tests/compile/" are only compiled and imported. See the hacking guide
in the wiki. I know you're not there yet with your implementation, I'm just
mentioning it.
Most important point, however: I think it's a good idea to clean up the
parsing context the way you did it. The semantic distinction of the three
new classes you added makes sense to me.
CtxAttribute is the wrong name, though. And the class copy implementation
gets even more generic than it already was in the Ctx. I'm not a big fan of
that, especially not in the parser. For one, it prevents changing the
classes into cdef classes, which had long been on my list for Ctx.
CSource: doesn't sound like quite the right name - it does not describe a C
source file but information that Cython has about non-Cython things.
I also doubt that Cython allows you to call an attribute "cdef", you'll
need to change that.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants
On Tue, Feb 22, 2011 at 08:18:21PM +0100, Stefan Behnel wrote:
> W. Trevor King, 22.02.2011 18:55:
> > I've been working on a more explicit parser that removes the ambiguity
> > behind the various visibilities. This will help me ensure proper
> > impolementation of my cdef-ed enums/structs/..., and make it easier to
> > update visibility syntax in the future. Take a look and let me know
> > if you think this is a useful direction to take:
>
> First thing that caught my eyes: I hope you do not intend to leave the
> logging usage in the parser. This is seriously performance critical code
> that Cython compiles down to pretty fast C code. Note that you can use a
> debugger and Python's profiling/tracing interface to find out what's
> happening, without code impact.
I can strip them out afterwards, but it helps me figure out what I've
broken if I shift too much around at the same time.
I don't know enough about Python's trace module to know if I can turn
on tracing only for functions defined in a single module or not, since
otherwise its hard for me to separate signal from noise.
> Some of the log statements span more than one line, which makes it trickier
> to strip them out with sed&friends (but backing out the initial changeset
> would likely make it simple enough to remove the rest manually).
Hmm, perhaps I'll condense the logging statements down onto one (long)
line a piece, that will make it easy to comment/uncomment them with
sed/emacs/etc. I suppose once Cython can compile the logging module
we could leave them in with reduced overhead ;).
> Also note that it's best to write runnable tests ("tests/run/"). The
> tests in "tests/compile/" are only compiled and imported. See the
> hacking guide in the wiki. I know you're not there yet with your
> implementation, I'm just mentioning it.
Thanks for the tip.
> CtxAttribute is the wrong name, though. And the class copy
> implementation gets even more generic than it already was in the
> Ctx. I'm not a big fan of that, especially not in the parser. For
> one, it prevents changing the classes into cdef classes, which had
> long been on my list for Ctx.
An easy, if uglier, workaround would be to prepend attributes with the
class name, e.g. CBinding.visibility -> CBinding.c_binding_visiblity.
Then the Ctx class could subclass the current CtxAttribute classes
instead of binding instances of each of them. That way Ctx would keep
its traditional flat attribute namespace and easy deepcopy, but
eveyone in Nodes, etc. that will use the attributes would become
class-name dependent.
The CtxAttribute class is, as its docstring says, just a hook for its
deepcopy method. With an alternative deepcopy implementation,
CtxAttribute could be replaced with the standard `object`, so don't
worry too much about its name at this point ;).
> CSource: doesn't sound like quite the right name - it does not describe a C
> source file but information that Cython has about non-Cython things.
It's a container for attributes that describe the presence and
location of backing C definitions.
* cdef: "Will there be a backing C defintion?
* extern: "Has someone else already written it?"
* name/namespace: "What did they call it?"
If you'd rather I called the class something else, I'm certainly
willing to change it.
> I also doubt that Cython allows you to call an attribute "cdef", you'll
> need to change that.
It seems to work for me:
>>> import Cython.Compiler.Parsing as P
>>> P.__file__
'Cython/Compiler/Parsing.so'
>>> c = P.CSource()
>>> dir(c)
[..., 'cdef', 'deepcopy', 'extern', 'name', 'namespace']
>>> c.cdef
0
>>> c.cdef = 1
>>> c.cdef
1
However, I agree that it's generally a bad idea to play around with
keywords. I'll revert it to the wordier-but-less-confusing
`cdef_flag`.
--
This email may be signed or encrypted with GPG (http://www.gnupg.org).
The GPG signature (if present) will be attached as 'signature.asc'.
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
My public key is at http://www.physics.drexel.edu/~wking/pubkey.txt
pgptJivrf8KRn.pgp
Description: PGP signature
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Control flow graph
Vitja Makarov, 20.02.2011 18:23:
2011/2/16 Vitja Makarov:
Hmm... both python and codespeaks in the thread
Yes, we should keep it to cython-devel only. Sorry for mixing it up.
Here is my commit it's mostly broken now but anyway
https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a
Flow control support is large enough to merit its own module. Not sure how
'smart' git is here, but you can always keep the history by explicitly
copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated
sections from both files.
You are duplicating some code from the type inferencer. We might want to
clean that up at some point. However, given that flow control analysis will
allow us to improve the type inferencer, I think it's best to keep this
code in the FCA part.
I've update stuff:
- algo for finding definitions
- warnings for uninitialized and may be uninitialised use
- few test cases
That looks very nice so far. Any idea how well it scales?
Trying to compile ParseTreeTransforms.py I've found this for example:
warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable
'template' may be used uninitialized
def create_Property(self, entry):
if entry.visibility == 'public':
if entry.type.is_pyobject:
template = self.basic_pyobject_property
else:
template = self.basic_property
elif entry.visibility == 'readonly':
template = self.basic_property_ro
property = template.substitute({
u"ATTR": ExprNodes.AttributeNode(pos=entry.pos,
obj=ExprNodes.NameNode(pos=entry.pos, name="self"),
attribute=entry.name),
}, pos=entry.pos).stats[0]
Ok, I guess that code generally works, but it's better to get rid of the
code smell.
Stefan
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
[Cython] hasattr() swallows any exception (Py<3.2)
Take a look here: http://bugs.python.org/issue9666 'hasattr' default behaviour should be changed to suppress only AttributeError exceptions. Other should pass through. Should we do something about this in Cython? We currently use PyObject_HasAttr(), but even in Python 3.2 this is not the same as builtins.hasattr(), it swallows any exception (do you think this is a bug in core Python?). I'm inclined to fix the behavior for ALL Python versions to suppress only AttributeError. -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] hasattr() swallows any exception (Py<3.2)
Lisandro Dalcin, 22.02.2011 21:41: Take a look here: http://bugs.python.org/issue9666 'hasattr' default behaviour should be changed to suppress only AttributeError exceptions. Other should pass through. +1, I think I even faintly recall that discussion. What a lengthy thread... http://mail.python.org/pipermail/python-dev/2010-August/103178.html I don't even recall when I last used hasattr(). Given how dumb it is to ask for an attribute, let someone else throw it away for you and then ask again to get it, I don't see much point in using it at all. I can write exception catching code myself, thanks. Should we do something about this in Cython? We currently use PyObject_HasAttr(), but even in Python 3.2 this is not the same as builtins.hasattr(), it swallows any exception (do you think this is a bug in core Python?). It was at least considered: http://mail.python.org/pipermail/python-dev/2010-August/103203.html I'm inclined to fix the behavior for ALL Python versions to suppress only AttributeError. How? Would you implement a hasattr() helper that uses PyObject_GetAttr() and Does The Right Thing? In any case, I personally don't care so much about hasattr(), but I'm +1 for Python compatibility, and +1 for getting the fixed Py3.2 behaviour in all Python versions. There's a reason this was fixed in CPython, it's a pretty clear bug. Stefan ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] hasattr() swallows any exception (Py<3.2)
On 22 February 2011 18:21, Stefan Behnel wrote: > Lisandro Dalcin, 22.02.2011 21:41: >> >> I'm inclined to fix the behavior for ALL Python >> versions to suppress only AttributeError. > > How? Would you implement a hasattr() helper that uses PyObject_GetAttr() and > Does The Right Thing? > Yes, more or less the implementation of builtin_hasattr from bltinmodule.c from Py3.2 > In any case, I personally don't care so much about hasattr(), but I'm +1 for > Python compatibility, and +1 for getting the fixed Py3.2 behaviour in all > Python versions. There's a reason this was fixed in CPython, it's a pretty > clear bug. > OK, I'll go for it. Thanks. -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] hasattr() swallows any exception (Py<3.2)
On 22 February 2011 19:09, Lisandro Dalcin wrote: > On 22 February 2011 18:21, Stefan Behnel wrote: >> Lisandro Dalcin, 22.02.2011 21:41: >>> >>> I'm inclined to fix the behavior for ALL Python >>> versions to suppress only AttributeError. >> >> How? Would you implement a hasattr() helper that uses PyObject_GetAttr() and >> Does The Right Thing? >> > > Yes, more or less the implementation of builtin_hasattr from > bltinmodule.c from Py3.2 > >> In any case, I personally don't care so much about hasattr(), but I'm +1 for >> Python compatibility, and +1 for getting the fixed Py3.2 behaviour in all >> Python versions. There's a reason this was fixed in CPython, it's a pretty >> clear bug. >> > > OK, I'll go for it. Thanks. > https://github.com/cython/cython/commit/90120314bcc8bd5a4f71f2629e1065f5c943071c -- Lisandro Dalcin --- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169 ___ cython-devel mailing list cython-devel@python.org http://mail.python.org/mailman/listinfo/cython-devel
Re: [Cython] Control flow graph
2011/2/22 Stefan Behnel :
> Vitja Makarov, 20.02.2011 18:23:
>>
>> 2011/2/16 Vitja Makarov:
>>>
>>> Hmm... both python and codespeaks in the thread
>
> Yes, we should keep it to cython-devel only. Sorry for mixing it up.
>
>
>>> Here is my commit it's mostly broken now but anyway
>>>
>>> https://github.com/vitek/cython/commit/5579b23c3c1c06981331b6427a73e5cb19980b8a
>
> Flow control support is large enough to merit its own module. Not sure how
> 'smart' git is here, but you can always keep the history by explicitly
> copying ParseTreeTransforms.py to FlowControl.py and removing the unrelated
> sections from both files.
>
Ok.
> You are duplicating some code from the type inferencer. We might want to
> clean that up at some point. However, given that flow control analysis will
> allow us to improve the type inferencer, I think it's best to keep this code
> in the FCA part.
>
Yes, I think it could replace MarkAssignments transform later.
Unreachable code could be delete there too.
>
>> I've update stuff:
>> - algo for finding definitions
>> - warnings for uninitialized and may be uninitialised use
>> - few test cases
>
> That looks very nice so far. Any idea how well it scales?
>
"Usually iterative algorithm takes no more then 5 iterations"
For ExprNodes.py max number is 15 while avg is about 3
About execution time:
ExprNodes.py compilation with c/f enabled takes 10.120 ms, w/o 9.325,
~10% slow down.
-O flag could be introduced but I don't think that's a good idea.
Should later try to execute cython compiled code.
>
>> Trying to compile ParseTreeTransforms.py I've found this for example:
>>
>> warning: Cython/Compiler/ParseTreeTransforms.py:1182:27: Variable
>> 'template' may be used uninitialized
>>
>> def create_Property(self, entry):
>> if entry.visibility == 'public':
>> if entry.type.is_pyobject:
>> template = self.basic_pyobject_property
>> else:
>> template = self.basic_property
>> elif entry.visibility == 'readonly':
>> template = self.basic_property_ro
>> property = template.substitute({
>> u"ATTR": ExprNodes.AttributeNode(pos=entry.pos,
>>
>> obj=ExprNodes.NameNode(pos=entry.pos, name="self"),
>> attribute=entry.name),
>> }, pos=entry.pos).stats[0]
>
> Ok, I guess that code generally works, but it's better to get rid of the
> code smell.
>
Might be used warning should be disabled by default, because algorithm
isn't smart enough:
a = 1
if (a): b = 1
if (a): print b
See also:
http://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wuninitialized-325
--
vitja.
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
