Re: [Cython] Bug in NULL handling introduced 0.14.1-1

[Greg Ewing]

Chris Colbert wrote:
I have cython file which is using PyObject_GenericSetAttr 

Now in my script I am using that function to generically delete an 
attribute by passing a NULL as the last value (this is proper way to 
trigger a generic delattr in the Python c-api)


I would have thought the proper way to do that was to use
PyObject_DelAttr, which Pyrex exposes as delattr().

I don't think PyObject_GenericSetAttr is even meant to be
called directly -- it's intended for filling the tp_setattr
slot of type objects.

This causes a segfault because the NULL is getting increfed via 
Py_INCREF instead of Py_XINCREF.


I would recommend against trying to "fix" this. You got
away with it before because you happened to be passing the
NULL value directly to a function which is expecting it.
But casting NULL to an object reference is not something
that should be encouraged, because in any other context it
would quickly lead to disaster.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bug in NULL handling introduced 0.14.1-1

[Greg Ewing]

Chris Colbert wrote:
The problem with delattr (and thus PyObject_DelAttr) arises when you 
define a __delattr__ method on your class. There is not easy way to then 
call back into the "normal" python delattr semantics, except by doing 
object.__delattr__ (which is not optimized by Cython). 


Hmmm, perhaps it should be? And similarly for all the other type
slots.

I would argue that there should be at least some way to pass a NULL 
pointer in Cython where a PyObject* is expected.


The trouble is that supporting this in general both safely and
efficiently would require keeping track of which object references
are allowed to be NULL. Could be done, but might require quite a
lot of work.

Maybe something more limited could be done such as allowing
a literal NULL to be passed to a function argument that is
specially marked as permitting this.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bug in NULL handling introduced 0.14.1-1

[Greg Ewing]

Chris Colbert wrote:

changing the include definition to:

cdef extern from "Python.h":
 int PyObject_GenericSetAttr(PyObject*, PyObject*, PyObject*) except -1


This suggests another possible workaround:

  cdef extern from "Python.h":
int PyObject_GenericDelAttr "PyObject_GenericSetAttr" (object, object, 
PyObject*)


This creates an alias for PyObject_GenericSetAttr with a different
signature, which you then call as

  PyObject_GenericDelAttr(obj, name, NULL)

If you're willing to write a tiny bit of C, you could make this tidier
by using a .h file containing

#define PyObject_GenericDelAttr(obj, name) PyObject_GenericSetAttr(obj, name, 
NULL)

and then declare PyObject_GenericDelAttr to Cython with just two
arguments.

Maybe Cython should include something like this in its standard preamble,
along with similar macros for other NULL-taking API functions.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Control flow graph

[Greg Ewing]

Vitja Makarov wrote:


Here is funny picture made with graphviz ;)

http://piccy.info/view3/1099337/ca29d7054d09bd0503cefa25f5f49420/1200/


Gives the term "spaghetti code" a whole new meaning!

--
Greg

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Greg Ewing]

Robert Bradshaw wrote:


BTW, the "public" keyword is the wrong thing to use here, as that
actually controls name mangling and (c-level) symbol exporting. The
fact that means a different thing for members than for top-level
symbols isn't ideal, but at least it's unambiguous as members need not
be mangled.


The overloading of 'public' is really a bit of a mess. I've been
thinking for a while that there really ought to be a different
keyword such as "exposed" for declaring that things are to be
exposed to Python. It would be useful in lots of ways:

cdef class Foo:
  cdef exposed int i   # formerly 'public'

cdef exposed enum E:
  a, b, c  # creates Python bindings for these names

cdef exposed struct S: # Exposed but mangled as usual
  ...

cdef public exposed struct S:  # Exposed and unmangled
  ...

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Greg Ewing]

W. Trevor King wrote:

On Sat, Feb 19, 2011 at 04:41:27PM -0800, Robert Bradshaw wrote:


On Sat, Feb 19, 2011 at 3:31 PM, W. Trevor King  wrote:


  cython $ grep class Cython/Includes/numpy.pxd
  ctypedef class numpy.dtype [object PyArray_Descr]:
  ctypedef extern class numpy.flatiter [object PyArrayIterObject]:
  ctypedef extern class numpy.broadcast [object PyArrayMultiIterObject]:
  ctypedef class numpy.ndarray [object PyArrayObject]:
  ctypedef extern class numpy.ufunc [object PyUFuncObject]:


"numpy.dtype" is the fully qualified name of the class it's
declaring--the module is "numpy."


Aren't the module names in the class declarations redundant here?
Since they're in a file called numpy.pxd, declarations will go into
the numpy module namespace by default. You should be able to write
these as just

  ctypedef class dtype [object PyArray_Descr]:
  ctypedef class flatiter [object PyArrayIterObject]:
  ctypedef class broadcast [object PyArrayMultiIterObject]:
  ctypedef class ndarray [object PyArrayObject]:
  ctypedef class ufunc [object PyUFuncObject]:

The 'extern' syntax for extension classes, including a module name,
is really an obsolete feature. Before Pyrex had .pxd files, it was
the only way to declare an externally implemented extension type
to Pyrex. But now that .pxd files exist, there should be no need
for it.


Hmm, that means that

cimport numpy
a = numpy.numpy.ndarray

also compiles.


Compiles and runs, or just compiles? If this works as a way of
getting hold of the ndarray type, then it's a bug -- it's not
supposed to work that way.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Greg Ewing]

Stefan Behnel wrote:


Given that Cython has "cpdef" already, why not just use that?


That seems like a reasonable idea.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Greg Ewing]

Stefan Behnel wrote:


With "preferred way", I was suggesting that we could *deprecate*

cdef public int x
cdef readonly object y

for cdef class properties in favour of

cpdef int x
cpdef readonly object y


I think I've just realised one of the reasons for my gut
dislike of the "cpdef" keyword -- it looks too similar to
"def". At first glance, it's hard to spot the difference
between the cdef and cpdef versions of two otherwise
identical declarations.

I think this is too subtle for something that makes such
a big difference to semantics. It's often important that
Python code is not allowed to mess with an object's
internal state, so making it possible should require
something more obvious.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Cython .pxd introspection: listing defined constants

[Greg Ewing]

Stefan Behnel wrote:
The same argument could be brought up against 
"cdef" vs. "def" (between which the semantic difference is *huge*)


There are a couple of differences:

- 'cdef' and 'def' look very different (at least to me) because
  they *start* with a different letter. Whereas 'cdef' and 'cpdef'
  both start and end with the same letter, maknig tehm mcuh eazier
  to conufse.

- There is much less semantic overlap betwen 'def' and 'cdef'. If
  you use the wrong one, you usually find out about it fairly
  quickly one way or another -- you get a syntax error (e.g.
  'def struct'), or something doesn't work the way you expected
  (e.g. a function you thought you had exposed doesn't show up
  in Python).

  It's possible to accidentally expose a function, but only if
  you declare it with an implicit return type, which I never do
  when writing what I intend to be a C function (this could
  perhaps be made illegal to further reduce the chances of such
  an error).

  Mistaken use of 'cpdef' on a C attribute, on the other hand,
  would very often pass silently.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Multiple modules in one compilation unit

[Greg Ewing]

Stefan Behnel wrote:
you'd call "cython" on a package and it would output a 
directory with a single __init__.so that contains the modules compiled 
from all .pyx/.py files in that package. Importing the package would 
then trigger an import of that __init__.so, which in turn will execute 
code in its init__init__() function to register the other modules.


I don't think it even has to be a directory with an __init__,
it could just be an ordinary .so file with the name of the
package.

I just tried an experiment in Python:

# onefilepackage.py
import new, sys
blarg = new.module("blarg")
blarg.thing = "This is the thing"
sys.modules["onefilepackage.blarg"] = blarg

and two different ways of importing it:

>>> from onefilepackage import blarg
>>> blarg

>>> blarg.thing
'This is the thing'

>>> import onefilepackage.blarg
>>> onefilepackage.blarg.thing
'This is the thing'

So assuming the same thing works with a .so instead of a .py,
all you need to do is emit a .so whose init function stuffs
appropriate entries into sys.modules to make it look like
a package.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Multiple modules in one compilation unit

[Greg Ewing]

Lisandro Dalcin wrote:

However, in Python
3 that is not te case (and only for the .so, for .py is the same as in
Py2), the import machinery adds the entry later, after the
finalization of the module init function.


Might be an idea to raise this issue on python-dev, to see
if there is a reason for this and whether it could be changed.


I'm tempted to workaround
this by setting the entry in sys.modules right after the call to
PyModule_Create() ... What do you think about this?


It's worth a try. I can't think of anything it might break
offhand -- but then I'm not really up with the play on what's
been happening to the importing machinery over the years.

--
Greg

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] OpenMP support

[Greg Ewing]

Stefan Behnel wrote:

You *can* use Python references inside of nogil blocks, even if there's 
a lot of stuff that you can't do with them, ... But you can access cdef 

> attributes on them,

Strictly speaking it's not even safe to do that, unless
you know there's some locking mechanism in effect that
prevents the underlying Python object going away from
under you.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)

[Greg Ewing]

Stefan Behnel wrote:

This introduces problems when the arguments have side effects or are not 
simple, e.g.


f(g(a), a.x, h(a))

>

What do you think about this?


I think it's a bad idea to write code that relies on the order
of evaluation like this. If the order matters, it's better to
be explicit about it:

   arg1 = g(a)
   arg2 = h(a)
   f(arg1, a.x, arg2)

So I would say it's not something worth worrying about overly
much.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)

[Greg Ewing]

Stefan Behnel wrote:

To be a little clearer here, it's a problem in C for example with struct 
values. Copying them by value into a temp variable can be expensive, 
potentially twice as expensive as simply passing them into the function 
normally.


What are you actually proposing to do here? Anything that
Cython does in the generated code to guarantee evaluation
order is going to involve using temp variables, and thus
incur the same overhead.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] OpenMP support

[Greg Ewing]

mark florisson wrote:


Have we ever thought about supporting 'with gil' as an actual
statement instead of just as part of a function declaration or
definition?


That's the way I was originally going to do it in Pyrex, but
it turned out to be problematic, because there is some setup
that gets done on entering a function -- increfing parameters,
initialising locals to None, etc. -- that is needed in order
to support Python operations being done anywhere in the function.

Allowing for a 'with gil' block would require deferring those
things until entering the block, and making sure nothing is done
outside the block that relies on them. Probably not impossible,
but it would require some extensive rearrangements.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[Greg Ewing]

mark florisson wrote:

I just think it is important for users
to realize that in general, they cannot unblock threads recursively.


I don't think that nested 'with nogil' blocks in any way
suggest that they can. Saying 'with nogil' means "I want
this code executed with the gil released". If it's
already released, the logical meaning is to do nothing
further.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[Greg Ewing]

On 03/16/2011 01:55 PM, mark florisson wrote:


but it doesn't catch this:

cdef void func() nogil:
 with nogil:
 pass

with nogil:
 func()


I would say that in that case, func() should acquire the
gil on entry so that it can be released in the 'with nogil'
block.

Either that or find some way to generate code that tests
whether the gil is currently held and releases it if it
is.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Out of order side effects of argument evaluation in function calls (ticket #654)

[Greg Ewing]

Stefan Behnel wrote:
That means that the average 
innocent future maintainer could break the code by manually rearranging 
the expression in a way that's perfectly valid mathematically,


Another thing is that when the subexpressions concerned
are function arguments, it only works if the order of the
arguments happens to be the same as the required order of
evaluation. Changes in the function signature can therefore
lead to subtle breakage.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[Greg Ewing]

Stefan Behnel wrote:

I'm not sure if this is a good idea. "nogil" blocks don't have a way to 
handle exceptions, so simply jumping out of them because an inner 'with 
gil' block raised an exception can have unexpected side effects.


Seems to me that the __Pyx_WriteUnraisable should be done at
the end of the 'with gil' block, and execution should then
continue from there.

In other words, the effect on exception handling should be
the same as if the 'with gil' block had been factored out into
a separate function having no exception return value.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[Greg Ewing]

Dag Sverre Seljebotn wrote:


def f():
with nogil:
for ...:
A
if something_exceptional:
with gil:
raise Exception(...)
B
C


If that's to be supported, the following really ought to be
supported as well:

  def f():
 with nogil:
try:
   ...
   with gil:
  raise Exception()
finally:
   ...do some cleanup...

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] 'with gil:' statement

[Greg Ewing]

mark florisson wrote:

I think we could support it without having to acquire
the GIL in the finally clause.


That was the intention -- the code in the finally clause would
be subject to the same nogil restrictions as the rest of
the nogil block.

My point is that as long as you're allowing exceptions to be
tunnelled through nogil blocks, they should respect any finally
clauses that they pass through on the way.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Rewriting/compiling parts of CPython's stdlib in Cython

[Greg Ewing]

Sturla Molden wrote:

Why stop with the standard library? Why not implement the whole CPython 
interpreter in Cython?


That would be tricky, because the code that Cython generates
depends on large chunks of CPython as an infrastructure.

--
Greg

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP: prange for parallel loops

[Greg Ewing]

Nathaniel Smith wrote:

Surely it should be 'pfor i in range(...)'.


Or 'pfhor', just to let you know it's really something out of
this world.

http://marathongame.wikia.com/wiki/Pfhor_%28Race%29

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] GSoC Proposal - Reimplement C modules in CPython's standard library in Cython.

[Greg Ewing]

Arthur de Souza Ribeiro wrote:


def _make_iterencode(dict markers, _default, _encoder, _indent, _floatstr,
_key_separator, _item_separator, bint _sort_keys, bint 
_skipkeys, bint _one_shot,

## HACK: hand-optimized bytecode; turn globals into locals
ValueError=ValueError,
dict=dict,
float=float,
^


encoder.pyx:273:13: Empty declarator


You may need to choose something other than 'float' for the local name to avoid
confusing the parser (it thinks you're about to declare a parameter of type
'float').

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

[Cython] What *is* a fused type?

[Greg Ewing]

I seem to have missed the beginning of the discussion about this
fused type business. Is there a document somewhere describing
what a "fused type" is and what it's meant to be used for?

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[Greg Ewing]

Stefan Behnel wrote:
Meaning, we'd find a new type during type analysis 
or inference, split off a new version of the function and then analyse 
it.


I'm not sure that this degree of smartness would really
be a good idea. I'd worry that if I made a type error
somewhere, the compiler would go off and spend a few
hours generating umpteen zillion versions of the code
instead of stopping and telling me something was wrong.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[Greg Ewing]

I'm a bit confused about how fused types combine to
create further fused types. If you have something
like

  ctypedef struct Vector:
floating x
floating y
floating z

then is it going to generate code for all possible
combinations of types for x, y and z?

That's probably not what the user intended -- it's
more likely that he wants *two* versions of type
Vector, one with all floats and one with all doubles.
But without explicit type parameters, there's no way
to express that.

I worry that you're introducing a parameterised type
system in a half-baked way here without properly
thinking through all the implications.

--
Greg

___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[Greg Ewing]

mark florisson wrote:


cdef func(floating x, floating y):
...

you get a "float, float" version, and a "double, double" version, but
not "float, double" or "double, float".


It's hard to draw conclusions from this example because
it's degenerate. You don't really need multiple versions of a
function like that, because of float <-> double coercions.

A more telling example might be

  cdef double dot_product(floating *u, floating *v, int length)

By your current rules, this would give you one version that
takes two float vectors, and another that takes two double
vectors.

But if you want to find the dot product of a float vector and
a double vector, you're out of luck.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[Greg Ewing]

Dag Sverre Seljebotn wrote:


ctypedef fused_type(float, double) speed_t
ctypedef fused_type(float, double) acceleration_t

then you get 4 specializations.

ctypedef speed_t acceleration_t

I guess only 2 specializations.

Treating the typedefs in this way is slightly fishy of course.


Indeed. This whole business seems rather too implicit to
me. I think I'd rather have explicit type parameters in
some form. Maybe

  cdef func2[floating F](F x, F y):
# 2 specialisations

  cdef func4[floating F, floating G](F x, G y):
# 4 specialisations

This also makes it clear how to refer to particular
specialisations: func2[float] or func4[float, double].

Pointers are handled in a natural way:

  cdef funcfp[floating F](F x, F *y):
# 2 specialisations

It also extends naturally to fused types used in other
contexts:

  cdef struct Vector[floating F]:
F x, y, z

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fused Types

[Greg Ewing]

Dag Sverre Seljebotn wrote:

The argument to the above goes that you *should* be out of luck. For 
instance, talking about dot products, BLAS itself has float-float and 
double-double, but not float-double AFAIK.


Seems to me that's more because generating lots of versions
of a function in C is hard work, and the designers of BLAS
didn't think it was worth providing more than two versions.
If they'd had a tool that would magically generate all the
combinations for them, they might have made a different choice.

What you seem to be trying to do here is enable compile-time
duck typing, so that you can write a function that "just works"
with a variety of argument types, without having to think
about the details.

With that mindset, seeing a function declared as

  cdef func(floating x, floating y)

one would expect that x and y could be independently chosen
as any of the types classified as "floating", because that's
the way duck typing usually works. For example, if a Python
function is documented as taking two sequences, you expect
that to mean *any* two sequences, not two sequences of the
same type.


What you are saying that this does not have the full power of C++ 
templates. And the answer is that yes, this does not have the full power 
of C++ templates.


What I'm suggesting doesn't have the full power of C++ templates
either, because the range of possible values for each type
parameter would still have to be specified in advance. However,
it makes the dependencies between the type parameters explicit,
rather than being hidden in some rather unintuitive implicit rules.

Would it be feasible to have a Skype 
session sometimes this week where everybody interested in the outcome of 
this come together for an hour and actually decide on something?


I'm not sure that would help much. Reaching good decisions about
things like this requires time to think through all the issues.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Header file bug

[Greg Ewing]

On Sat, Jun 4, 2011 at 10:36 AM, Seth Shannin  wrote:


test.h:11: warning: 'struct __pyx_obj_4test_foo' declared inside parameter
list
test.h:11: warning: its scope is only this definition or declaration, which
is probably not what you want


Not sure about Cython, but the way to to this in Pyrex
is to declare the class 'public' as well, and then the
struct declaration for it will be put in the .h file.

You will need to supply C names for the struct and type
object as well, e.g.

   cdef public class foo[type FooType, object FooObject]:
  ...

The generated test.h file then contains

   struct FooObject {
 PyObject_HEAD
 int a;
 int b;
   };

   __PYX_EXTERN_C DL_IMPORT(void) bar(struct FooObject *);
   __PYX_EXTERN_C DL_IMPORT(struct FooObject) *make_foo(int,int);

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Header file bug

[Greg Ewing]

Seth Shannin wrote:

I would still make the case that currently
something is a bit off in that the header file tries to use something
which it has no information about


Yes, I agree that it could do with improvement. I'm not sure
that automatically putting the struct declaration in the
header is the best idea, though, since with mangled names it's
hard to do anything with it from external C code (that's why
you're required to manually supply type and object struct names).

Maybe it should just be an error to make a declaration public
if it uses something that's non-public?

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] How to define C-consts in python module scope

[Greg Ewing]

Chris Colbert wrote:


cdef public enum:
   EV_READ  = 1
   EV_WRITE = 2

However, I do not like it, because I would like to use "public" for
other meaning (API generation).


My suggestion is

  cdef exposed enum:
...

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Willing to contribute

[Greg Ewing]

Dag Sverre Seljebotn wrote:

 - One idea is coercion of C pointers to ctypes Python objects and back 
again.


Some way of requesting this manually might be useful, but
I don't think I'd like it to happen automatically. Slinging
raw pointers around in Python isn't something to be done
lightly -- even if all the code that dereferences it is
in Cython, there are problems with managing the lifetime
of whatever it references.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Re: callback function pointer problem

[Greg Ewing]

Robert Bradshaw wrote:


On this note, eventually I would like coerce structs (and unions,
enums) to auto-generated wrapper classes, visible in the Python module
namespace if one declares them as "cpdef struct ..."


Would these wrapper classes contain a copy of the struct,
or would they reference the struct? If they reference it,
there would be issues with the lifetime of the referenced
data.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Acquisition counted cdef classes

[Greg Ewing]

mark florisson wrote:

These will by default not lock for operations to allow
e.g. one thread to iterate over the list and another thread to index
it without lock contention and other general overhead.


I don't think that's safe. You can't say "I'm not modifying
this, so I don't need to lock it" because there may be another
thread that *is* in the midst of modifying it.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Acquisition counted cdef classes

[Greg Ewing]

Dag Sverre Seljebotn wrote:

I'd gladly take a factor two (or even four) slowdown of CPython code any 
day to get rid of the GIL :-). The thing is, sometimes one has 48 cores 
and consider a 10x speedup better than nothing...


Another thing to consider is that locking around refcount
changes may not be as expensive in typical Cython code as
it is in Python.

The trouble with Python is that you can't so much as scratch
your nose without touching a big pile of ref counts. But
if the Cython code is only dealing with a few Python objects
and doing most of its work at the C level, the relative
overhead of locking around refcount changes may not be
significant.

So it may be worth trying the strategy of just acquiring
the GIL whenever a refcount needs to be changed in a nogil
section, and damn the consequences.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Correct way of defining enums

[Greg Ewing]

Stéfan van der Walt wrote:

Hi Licandro

On Tue, Nov 29, 2011 at 10:32 AM, Lisandro Dalcin  wrote:


Try "ctypedef enum ..."


Unfortunately, that doesn't work either :/


Did you try both together? I.e.

  cdef extern from "ndarraytypes.h":
ctypedef enum NPY_SEARCHSIDE:
  NPY_SEARCHLEFT = 0
  NPY_SEARCHRIGHT = 1

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Greg Ewing]

Dag Sverre Seljebotn wrote:


1) It doesn't work well with multiple interpreter states. Ok, nothing works
with that at the moment, but it is on the roadmap for Python


Is it really? I got the impression that it's not considered feasible,
since it would require massive changes to the entire implementation
and totally break the existing C API. Has someone thought of a way
around those problems?

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Greg Ewing]

Robert Bradshaw wrote:


Has anyone done any experiments/timings to see if having constants vs.
globals even matters?


My gut feeling is that one extra memory read is going to be
insignificant compared to the time taken by the call itself
and whatever it does. But of course gut feelings are always
better when backed up (or refuted!) by measurements.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Greg Ewing]

Dag Sverre Seljebotn wrote:


if (obj has signature "id)i") {


This is an aside, but is it really necessary to define the
signature syntax in a way that involves unmatched parens?
Some editors (such as the one I like to use) get confused
by this, even when they're inside quotes.

The answer "get a better editor" would be entirely
appropriate if there were some advantage to this syntax,
over a non-unbalanced one, but I can't see any.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Greg Ewing]

Stefan Behnel wrote:


It wasn't really a proposed syntax, I guess, more of a way to write down an
example.


That's okay, although you might want to mention in the PEP
that the actual syntax is yet to be determined. Being a PEP,
anything it says tends to come across as being a specification
otherwise.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Greg Ewing]

Robert Bradshaw wrote:


Brevity, especially if the signature is inlined. (Encoding could take
care of this by, e.g. ignoring the redundant opening, or we could just
write di=d.)


Yes, I was thinking in terms of replacing the paren with
some other character, rather than inserting more parens.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] CEP1000: Native dispatch through callables

[Greg Ewing]

Dag Sverre Seljebotn wrote:
if you use 
Python bytes in a dict in sys.modules['_nativecall'], the bytes objects 
could be deallocated before callables containing the interned string -- 
unless you Py_INCREF once too many,


I don't understand that. Is there some reason that refcounting
of the interned strings can't be done correctly?

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Julialang

[Greg Ewing]

Dag Sverre Seljebotn wrote:

I'm excited about Julia because it's basically what I'd *like* to 
program in. My current mode of development for much stuff is Jinja2 or 
Tempita used for generating C code; Julia would be a real step forward.


It looks interesting, but I have a few reservations about
it as it stands:

* No modules, just one big global namespace. This makes it
unsuitable for large projects, IMO.

* Multiple dispatch... I have mixed feelings about it. When
methods belong to classes, the class serves as a namespace,
and as we all know, namespaces are a honking great idea.
Putting methods outside of classes throws away one kind of
namespace.

* One-based indexing? Yuck. I suppose it's what Fortran and
Matlab users are familiar with, but it's not the best
technical decision, IMO.

On the plus side, it does seem to have a very nice and
unobtrusive type system.

the next natural step is to implement Python in 
Julia, with CPython C-API compatability. Which would be great.


That would indeed be an interesting thing to explore.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Julialang

[Greg Ewing]

Dag Sverre Seljebotn wrote:

Well, there's still the namespace of the argument type. I think it is 
really a syntactic rewrite of


obj->foo(bar)

to

foo(obj, bar)


This is where I disagree. It's *not* just a syntactic rewrite,
it's a lot more than that.

With a Python method, I have a fairly good idea of where to
start looking for a definition, documentation, etc. Or if it's
a stand-alone function, I can follow the imports and find out
which module it's defined in.

But with generic functions, the implementation for the particular
combination of argument types concerned could be *anywhere*, even
in a file that I haven't explicitly imported myself. It's
monkeypatching on steroids.

I acknowledge that multiple dispatch is very powerful and lets
you do all sorts of wonderful things. But that power comes at
the expense of some features that I particularly value about
Python's namespace system.

And in Python there's all sorts of problems with who wins the battle 
over __add__ and __radd__


I think Python's solution to this is rather good, actually. It
seems to work quite well in practice.

And multiple dispatch systems have all the same problems when
the "best" match to the argument types is ambiguous, so it's
not as if multimethods are a magic solution to this.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Fwd: Re: [cython-users] checking for "None" in nogil function

[Greg Ewing]

Stefan Behnel wrote:


The main reason we didn't change this behaviour back then was that it would
clearly break user code and we thought we could do without that. That's
different from considering it "right" and "good".


I changed the None-checking behaviour in Pyrex because I *wanted*
to break user code. Or rather, I didn't think it would be a
bad thing to make people revisit their code and think properly
about whether they really wanted to allow None in each case.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Bug, or changed array assignment in 0.17beta1?

[Greg Ewing]

mark florisson wrote:


I'm wondering, what was the original motivation to reuse temporaries?


It goes back to Pyrex, where I didn't really give it much
thought -- it just seemed like the tidiest thing to do.
Once you have the logic to release temp references as soon
as it's safe to do so, it's not much harder to return the
variable to a pool as well.


Do we ever have C++
stack-allocated object temporaries?


Theoretically, yes --  temporaries aren't necessarily
object references, they can be of any type (or they can
in Pyrex, at least).

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] import relative cimport

[Greg Ewing]

Christoph Groth wrote:


in Cython/Compiler/Parsing.py I find code that emits "Relative cimport
is not supported yet".



Is this related to the behavior of Pyrex?


There's no such code in Pyrex's Parsing.py, and if I try to
compile this with Pyrex:

   from .bar.stuff cimport thing

I get

   /Users/greg/foo/foo.pyx:1:5: Expected an identifier

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] import relative cimport

[Greg Ewing]

Christoph Groth wrote:


kwant/_system.pyx:8:0: 'graph.defs.pxd' not found


That part *is* related to Pyrex -- it's a holdover from
the old method of dealing with .pyx files in packages.
It doesn't have anything to do with relative imports,
though.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Recommendations for efficient typed arrays in Cython?

[Greg Ewing]

On 01/02/13 09:31, Sturla Molden wrote:

cdef object a
cdef list b
cdef foobar c

etc to define Python variables. 'cdef' seems to indicate that it is a C
declaration, yet here it is not.


Yes, it is. In this context, the cdef isn't about the type of the
variable, it's about where and how it's stored and accessed. The
above declarations result in the generation of C code something like:

   PyObject *a;
   PyListObject *b;
   Foobar *c;

They are then accessed directly by the generated C code.

Without the cdef, these variables would be stored wherever Python
normally stores variables for the relevant scope, which could be
in a module or instance dict, and the usual Python/C API machinery
is used to access them.

Distinguishing between Python and C types would be problematic
anyway, since a PyObject* is both a Python type *and* a C type.


Neither does this cdef syntax allow us to declare Python int and float
statically.


I've never found the need to declare a Python int or float
statically, but a way could be provided to access these types
if need be. Maybe Cython has already done this, I don't know.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Recommendations for efficient typed arrays in Cython?

[Greg Ewing]

Sturla Molden wrote:
The way I see it, "object" is a Python type and "PyObject*" is a C type. 
That is, PyObject* is just a raw C pointer with respect to behavior.


Well... while it's possible to declare something as PyObject*
in Cython and get raw pointer behaviour, it's something you
should only do in very rare circumstances, because you're
then totally on your own when it comes to reference counting
and exception handling.

If you're suggesting that 'def object foo' should give Python
reference semantics and 'cdef object foo' raw C pointer
semantics, that would lead to a world of pain.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [cython-users] Recommendations for efficient typed arrays in Cython?

[Greg Ewing]

Sturla Molden wrote:
I was replying to a Cython 
user who thought anything declared 'cdef' was reference counted


That's a matter of user education. We can't use syntax to
address every possible misconception a user might have.

"cdef" refers to storage in the generated C, not to the semantics of 
Cython. But how and where variables are stored in the generated C is an 
implementation detail.


Not entirely -- you can't access a cdef attribute of an
extension type using getattr(), for example. And external
C code only has direct access to cdef variables and
attributes.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cython syntax to pre-allocate lists for performance

[Greg Ewing]

Nikita Nemkin wrote:

Sorry, accidental early send. Previous mail continued...

[None] * N makes an extra pass over the list to assign None to each item
(and also incref None n times).


Maybe this could be optimised by adding N to the reference
count instead of incrementing it N times?

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Surprising behaviour wrt. generated tp_clear and tp_dealloc functions

[Greg Ewing]

Stefan Behnel wrote:

Torsten Landschoff, 22.04.2013 13:07:



One could even think about building a graph of possible object
relationships ...  Slot refers only to Slots and Slots only to Context, so
these can't build a cycle.


Interesting. Yes, that might work.


Only if subclassing of Slot and Context are forbidden.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] nogil doesn't seem to work when defined on extern cpp functions

[Greg Ewing]

Ryan Pessa wrote:
It doesn't seem like Cython 
respects the `nogil` statement on extern cpp functions.


The nogil declaration doesn't work that way. Declaring a
function as nogil just says that it's safe to call it
without the GIL -- it doesn't actually cause the GIL to
be released. You need a 'with nogil' block to do that.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] How should be C-clobal deletion handled?

[Greg Ewing]

Vitja Makarov wrote:

Recently I've found that the following code causes segmentation fault:

cdef object f
del f
print f


FWIW, Pyrex disallows using del on cdef names.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Declaration syntax change

[Greg Ewing]

Robert Bradshaw wrote:

cdef int *a, b, c, *d[3]

is IMHO quite ugly but also adds a lot of complexity to the parser.
What if instead we required

cdef int* a
cdef int b, c
cdef int[3]* d


What would be the benefit of this? You're proposing to change
from something identical to C declaration syntax, which is
second nature for a great many people, to something that
looks deceptively like C syntax but isn't.

I can't see that causing anything other than a massive
amount of confusion, anguish and hair-tearing.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Wrong order of __Pyx_DECREF when calling a function with an implicit str → char* conversion.

[Greg Ewing]

Stefan Behnel wrote:

Stefan Behnel, 30.05.2014 15:01:


Emmanuel Gil Peyrot, 28.05.2014 13:25:


I was testing my cython codebase on top of pypy/cpyext, and I found a
memory corruption.  After investigating it with the pypy guys (thanks
arigato!), we identified it as a cython bug:

cdef void test(char *string):
print(string)

def run(array):
test(array[0])


I wouldn't call it a bug, more a limitation. When casting a
Python object to char *, it's the programmer's responsibility
to ensure that a reference is kept to the underlying object
for as long as necessary.

Having said that, keeping the reference for the duration of
the call would result in less surprising behaviour in cases
like this.


Cython has a mechanism to reject this kind of code, not sure why it
wouldn't strike here.


Not sure about Cython, but Pyrex only catches the most glaring
cases of this, such as adding two string and then immediately
casting the result to char *, which is almost guaranteed to
fail. Trying to catch anything more would have resulted in huge
numbers of false positives and been very annoying, so I
didn't try.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] New function (pointer) syntax.

[Greg Ewing]

Robert Bradshaw wrote:

If you want a hint, the last is something that returns numerical
integration algorithm given a string name. Yes, you could use
typedefs, but you shouldn't have to.


I don't find *any* of those particularly easy to read in the third
case, or even the second. Using typedefs to make the intention
clear, even if the syntax doesn't require it, seems entirely
appropriate to me.


I am curious, when you read "cdef int * p" do you parse this as "cdef
(int*) p" or "cdef int (*p)"


C syntax is sufficiently well embedded in my brain that I
parse it as "int (*p)", and read it as "p points to an int"
or "p is of type pointer-to-int".

But I don't find myself needing to parse "int * p" very often,
because I always write it as "int *p", and so do most other
people who write C. Whenever I see something like "int * p"
or "int* p", it makes me think it was written by someone who
doesn't understand C very well.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Possible bug (or wrong documentation) WRT "not None"

[Greg Ewing]

Samuele Kaplun wrote:
The self parameter of a method of an extension type is guaranteed never to be 
None. (Note: arithmetic methods might be an exception to this rule. 


The arithmetic methods don't *have* a "self" parameter
(or at least it's not always the first parameter), so
technically what the docs currently say is correct.
It's probably still worth clarifying this somehow, though.
Maybe something like

   The self parameter of a method of an extension type is guaranteed never to be
   None. (However, note that some operator methods do not always receive "self"
   as the first parameter; see ).

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] IPython with Cython support on Android

[Greg Ewing]

Apps Embedded wrote:

> 2015-02-01 9:25 GMT+01:00 Stefan Behnel  >:
>
> (I don't understand why it requests the right to track where I am,
> though,
> so I didn't try it out.)
>

The Free app has an advertising banner based on the Mobfox library.
When integrating this library, it is written in their integration guide 
to add this permission.


So the price of using a "free" app is getting subjected to
advertising *and* being spied upon? Seems a tad obnoxious
to me.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Compilation failes if a class member is named "INFINITY"

[Greg Ewing]

Stefan Behnel wrote:

Python extension types are just structs at the C level, and struct member
names cannot be mangled.


Well, in principle it *could* mangle the names in
structs that aren't declared "cdef extern". I didn't
do that in Pyrex because it didn't seem necessary;
I hadn't anticipated problems like collision with
macros.

It's probably too late to change it now, since it
would break existing code that interoperates with
foreign code.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] operator() bug in cython

[Greg Ewing]

Ulrich Dobramysl wrote:

Thanks! I haven't figured out how to call heap allocated objects, as the 
code

---
cdef OperatorTest *t = new OperatorTest()
t()
---
is not translatable by Cython.


Have you tried:

   t[0]()

?

A quick and dirty fix for this would be this patch 
for NameNode.calculate_result_code:

---
diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py
index f99ec6e..f894a64 100644
--- a/Cython/Compiler/ExprNodes.py
+++ b/Cython/Compiler/ExprNodes.py
@@ -1904,6 +1904,8 @@ class NameNode(AtomicExprNode):
 entry = self.entry
 if not entry:
 return "" # There was an error earlier
+if entry.cname=='operator()':
+return self.name 
 return entry.cname


I haven't been following the development of Cython's internals,
so I may be speaking naively here, but it looks wrong to me that
the cname of that entry should be 'operator()' rather than the
c-level name of the variable that the NameNode refers to.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] __dealloc__ called even if __cinit__ failed

[Greg Ewing]

Jeroen Demeyer wrote:
The problem is that __cinit__ is supposed to set up the C-level 
attributes for that object. If this somehow fails, then the object isn't 
in a consistent state yet, so it makes no sense to call __dealloc__ on it.


Do you consider this a Cython bug or should I manually protect my code 
against this?


It's probably a good thing that __dealloc__ is called,
because it provides an opportunity to clean up a partially
initialised object that would otherwise leak memory.

It does mean that you need to design your __dealloc__
to be robust against this situation. However, the
alternative would be to catch exceptions in __cinit__
and clean up there, which means you have cleanup code
in two places.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] can we deprecate for-from loops?

[Greg Ewing]

Stefan Behnel wrote:

Hi!

The syntax construct "for i from 0 <= i < 10" has been silently outdated
for years. Can we start issuing a warning that normal range() loops are
preferred?


I'd be in favour of replacing it with just 'for 0 <= i < 10',
but -1 on removing it altogether.

I introduced it in Pyrex for a reason -- to clearly express
iterations over ranges of integers with arbitrary combinations
of open/closed endpoints, for use in conjunction with C code.
I believe that reason is still valid.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] import-free setuptools integration

[Greg Ewing]

Nils Werner wrote:
 - Having `cythonize()` internally automatically compile *.pyx files 
when *.c files
   are provided is disabled by default and must be enabled using the 
`replace_extension`

   keyword argument (`cython_modules` enables that option).


I'd be happier if it were still disabled by default even then.
If I'm simply installing a package, I do *not* want it to
try to recompile any .pyx files just because I happen to have
cython installed.

I know that isn't supposed to happen unless the .pyx is newer
than the .c, but I wouldn't like to rely on that. There's a
big difference between using a package and modifying it, and
I'd rather be explicit about when I'm doing the latter.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Cygwin: Handling missing C99 long double functions

[Greg Ewing]

Erik Bray wrote:

A developer who writes some code that
happens to use long double isn't going to think themselves "Gosh, I
guess I can't accept `long double` here because it may cause Cython to
generate code that contains "truncl"


Sounds to me like Cython shouldn't be taking it upon
itself to generate calls to truncl just because long
double is being used. The programmer should have to
call it explicitly if that's what they want.

--
Greg

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Virtual cpdef methods

[Greg Ewing]

Robert Bradshaw wrote:

All methods (cdef, cpdef, and def) are virtual by default in Cython,
just like Python.

On Mon, Jun 20, 2016 at 6:08 AM, Jeroen Demeyer  wrote:


I would like to have a "virtual" cpdef method: something which allows a fast
Cython call *when implemented* but without a default implementation.


It sounds like Jeroen really means "abstract" here, not "virtual".

The usual way to do this kind of thing in Python is to write
a stub method that raises NotImplementedError or such like.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] RFC: an inline_ function that dumps c/c++ code to the code emitter

[Greg Ewing]

Robert Bradshaw wrote:

Name mangling is done for the standard reasons--to avoid possible
conflicts with all other symbols that may be defined.


The main reason for mangling names is that top-level declarations
in Cython belong to a module namespace, whereas C just has a
single namespace for all globals. So some way is needed to
distinguish things with the same name in different modules.

Also, a single Cython declaration often leads to several
different related things being declared in C, e.g. a class
has a PyType instance, a struct declaration, and possibly
a vtable, all of which need distinct names in the C code,
so prefixes are used to distinguish them.

It might be possible to get away without mangling a few
things, such as local variables, but for simplicity they're
treated the same as everything else.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hooking tp_clear()

[Greg Ewing]

Jeroen Demeyer wrote:

I have a concrete use case where I want something like __dealloc__ but 

> *before* Python attributes are cleared. So this really belongs in tp_clear().

Are you sure you can't do it in __del__?  From what I gather,
the presence of __del__ no longer prevents cyclic garbage
collection.

I never really understood the technical difference between 
tp_clear() and tp_dealloc(). It seems to me that these serve a very 
similar purpose: why can't the garbage collector just call tp_dealloc()?


tp_dealloc is the inverse of tp_alloc -- its purpose is to
free the memory occupied by the object. This must not be done
until there are no more references to the object.

tp_clear is used to break reference cycles. After calling it,
there may still be references to the object from other objects
in the cycle, so tp_dealloc can't be done at that point.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hello

[Greg Ewing]

On 27/01/20 6:56 pm, John Skaller2 wrote:

Felix binds C/C++ code with statements like:

type PyObject = “PyObject*”;
fun add: PyObject * PyObject -> PyObject = “Py_AddLong($1)”;

and can use the bindings like:

var a : PyObject = ….
var b: PyObject = ...
var sum = add (a,b);


How does it deal with reference counting and exception handling?

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hello

[Greg Ewing]

On 29/01/20 1:33 am, John Skaller2 wrote:


# This conflicts with the C++ bool type, and unfortunately
# C++ is too liberal about PyObject* <-> bool conversions,
# resulting in unintuitive runtime behavior and segfaults.
#("bool","PyBool_Type", []),

Now you have me worried!  Any more
detailed explanation of this issue?


I think it's just that the name "bool" refers to different
things in Python and C++, and trying to unify them would
lead to problems. So Cython has chosen to use "bool" to mean
the Python type, and introduced "bint" for the C++ notion
of a bool.

This is consistent with the general philosophy that
the Cython language is an extension of Python, so where
there is a conflict between Python smantics and C or C++
semantics, Python wins.

--
Greg

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hello

[Greg Ewing]

On 29/01/20 8:37 am, John Skaller2 wrote:

If I implement that with just static casts for coercions, then say,
given a dictionary and a function accepting a mapping argument,
you can pass a dictionary or mapping, but not an object.
If you want to pass an object you have to explicitly coerce it.


That sounds fine. Cython follows similar rules.


The *problem* with this idea I think is that from memory the reference counting
rules for PyObject and dictionaries are not the same as for methods
on abstractions.


The vast majority of CPython API functions follow consistent
refcounting rules -- they take borrowed references as arguments,
and return new references as return values.

However, a few of the low-level functions that operate on
specific types follow different rules. Cython knows about
these and treats them specially.

For your system, you may need a way of annotating function
declarations with refcounting rules where they differ from
the defaults.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Status

[Greg Ewing]

On 30/01/20 8:58 pm, John Skaller2 wrote:
 import oldtest
> Traceback (most recent call last):
>File "", line 1, in 
> ModuleNotFoundError: No module named 'oldtest'
What happens if you use a .so extension instead of .dylib?

--
Greg

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Status

[Greg Ewing]

On 31/01/20 10:00 am, John Skaller2 wrote:

It works. Why Python would look for a linux extension only on MacOS I have no 
idea.


It's not really a Linux extension. It goes back to when shared
libraries first appeared in BSD, from which large parts of MacOSX
are derived, and MacOSX also recognises it as a valid extension
for dynamically linked object files. I guess the Python developers
didn't think it was worth making a distinction between different
flavours of unix in this area.

The mystery to me is why MacOSX introduced .dylib instead of
sticking with .so.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Status

[Greg Ewing]

On 31/01/20 9:47 am, John Skaller2 wrote:


2. pyport is plain wrong. It contains conflicting C typedefs.


PRs welcome.


Is this your prefered method (pull request)?


I'm sure PRs are very welcome, but at the least you could
give us some idea of what these conflicting typedefs are!


ob should be PyObject*


No, the declaration looks correct to me. The input is an object.


I don’t understand. ob isn’t a type, is it? A type is required.


It's a (dummy) parameter name. Cython defaults to "object" when a
type isn't specified.

Looking at the other declarations in that file, it was probably
*meant* to say "object ob", but it's not wrong -- it still works
that way.


Else where “object” is used as an alias for PyObject*.
It also suggests standard ref counting is required whereas
plain PyObject* suggests non-standard ref counting.


And you want standard refcounting here.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Status

[Greg Ewing]

On 1/02/20 12:25 am, John Skaller2 wrote:

cdef extern from "Python.h":
 ctypedef int int32_t
 ctypedef int int64_t
 ctypedef unsigned int uint32_t
 ctypedef unsigned int uint64_t


These work because Cython doesn't need to know the exact
sizes of these types. All it needs to know is that they're
some kind of integer so that its type checks will pass.
The typedef names end up in the generated C code, and the
C compiler figures out their actual sizes.


Obviously this is an incorrect translation of the original source.


Extern declarations in Cython are not meant to be exact
translations. They only need to tell Cython enough about
the thing being declared so that it can cope.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Status

[Greg Ewing]

On 1/02/20 12:34 am, John Skaller2 wrote:


Ok, but now the syntax is made very context sensitive.
To interpret it correctly, you have to know “ob” is not a type.


Yes, Cython mostly follows C declaration syntax, and C also has
this property.


In C this would not work because there is no default type,


Yes, there is -- the default type in C is int. This is
a valid function definition in C:

  f(x) {
  }

It's equivalent to

  int f(int x) {
  }


And my translator script got fooled, because it assumes
any single identifier used as a parameter is a type,


Then it's making an incorrect assumption.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Status

[Greg Ewing]

On 1/02/20 3:03 am, John Skaller2 wrote:


 So this is some kind of hack way
of getting something a bit like Haskell type classes,
you’re basically saying int32_t and int64_t are of class “Integer”.


I suppose you could think of it that way, but it's really
not that formal.


This also explains the conflict for me, because Felix is the opposite:
it aims to make the types of things more precise (and has actual
type classes for generalisation).


To define them any more precisely, Cython would need to
know how things vary depending on the platform, which would
mean conditional compilation, etc. It's much easier to leave
all that up to the C compiler and system headers. It also
ensures that there can't be any mismatch between the two.

--
Greg

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Status

[Greg Ewing]

On 1/02/20 3:08 am, John Skaller2 wrote:


I don’t think that is true in C99 but I’m not sure.


You may be right, I haven't been keeping up with all the
twists and turns of recent C standards. The gcc I just
tried it on allowed it, but warned about it.


In any case its a bad idea in an interface specification even if it’s legal.


Perhaos in C, but I think it makes sense for types in
Cython to default to object, because it deals with objects
so much. It means that functions that take and return
objects exclusively look like Python.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Size of output

[Greg Ewing]

On 1/02/20 3:17 am, John Skaller2 wrote:

When I ran Cython on a two line Python function I got this from wc:

 4276   13798  161338 oldtest.c


That seems a bit excessive.


A lot of the emitted code appeared to be run time and compile
time support code which wasn’t actually used.


Not sure what's going on there. Pyrex made efforts to only
include support code that was actually used, but Cython
has changed a lot since then and I haven't been following
its development closely. Either it's slipped on that, or
the support code has become more bloated.

Can you remove any of it and still have it compile? If
so, filing a bug report might be useful.


Is there an option to use an #include for the standard stuff?


There are upsides and downsides to that as well. The way
things are, the generated file is self-contained, and can
be shipped without worrying about it becoming disconnected
from a compatible version of the include file. This is
important when details of the support code can change
without notice between Cython releases.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Status

[Greg Ewing]

On 1/02/20 3:29 pm, John Skaller2 wrote:

But the all hell breaks loose for pointers. Your hack only
works for rvalues.


Yes, using these it's possible for Cython to accept something
that will later be rejected by the C compiler. It's not perfect,
but it gets the job done most of the time.

--
Greg


___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Most efficient way to create an object with Python C-API

[Greg Ewing]

On 17/06/20 8:32 am, Daniele Nicolodi wrote:

PyObject* decimal = PyImport_ImportModule("decimal");
PyObject* constructor = PyObject_GetAttrString(m, "Decimal");
PyObject* obj = PyObject_CallFunction(PyDec_Type, "s#", str, len);

Is there a better way?

I was thinking about getting to the PyTypeObject for Decimal, like

PyTypeObject* type = Py_TYPE(obj);


That will get you exactly the same object you get by importing
the decimal.Decimal class. In Python3 there is no difference
between a type and a class. So I don't think there's any
substantially faster way.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [ENH] Compile-time code execution (macros / templates)

[Greg Ewing]

On 26/02/21 9:29 am, Celelibi wrote:

Maybe in the future if cython
support compiling closures to C?


Your "twice" example already needs some closure functionality.
It relies on being able to manufacture a cdef function inside
a Python function, with the expectation that it will have
access to arguments of the Python function. With all or some
of that happening at compile time rather than run time. I'm
having trouble imagining how it would be implemented.

--
Greg

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] [ENH] Compile-time code execution (macros / templates)

[Greg Ewing]

On 26/02/21 3:21 pm, Celelibi wrote:

def twice(f):
 fun = CdefFunction(... use f ...)
 return fun

foo = CdefFunction(...)
ast.append(AstCdefFunction("foo", twice(foo)))

I think this might even be doable without having to even detect the
closure in cython. We'd just have to let python perform the name lookup.


The cdef function created inside twice() is going to have
to be some kind of closure, because it needs access at run
time to the particular f that was passed to twice() at
compile time.

--
Greg
___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel