from:"John Skaller2"

[Cython] Hello

2020-01-26 Thread John Skaller2

Hi! I have just built Cython but haven’t used it yet.

I want to check I understand what it does whilst examing the sources in the 
repository.
Please let me know if I have it wrong!

Given a Python file, Cython parses it, and translates it to the equivalent C, 
which can
then be compiled to binary, thereby bypassing the overhead of interpreting 
bytecode,
but still executing the same accesses to the CPython run time API that the 
interpreter would.
So there may be a small speed improvement, or maybe not (because it misses 
optimisations
the interpreter might be able to spot). The binary will typically be a C 
extension module
that can be loaded and operate the same way as the original Python.

Now, Cython is an extension of Python which allows some extra stuff, including
type annotations, and other directives related to integration with C. These can 
be
used to facilitate integration with external C libraries directly, and mapping 
into
Python, as if the code were written in C, only we’re using a Python like 
language
representing a subset of C instead of C.

Additionally, the compiler recognises the type annotations, and can reduce or
eliminate run time type checks, improving performance, or even replacing
common constructions in Python which much faster ones that do the same job
“closer to the metal”.

To make this work, the CPython API itself is represented in a set of *.pxd files
found in the repository in Includes/cpython, splitting the logic of the compiler
roughly into two parts: the front and back end. The front end groks Python
and Cython code whilst the back end generates the actual C.

===

Just FYI, I’m the developer of a programming language, Felix, which is C++ code
generator. You can think of it as a meta-programming language for C++ with a 
proper type system. Felix binds C/C++ code with statements like:

type PyObject = “PyObject*”;
fun add: PyObject * PyObject -> PyObject = “Py_AddLong($1)”;

and can use the bindings like:

var a : PyObject = ….
var b: PyObject = ...
var sum = add (a,b);

so in some ways its doing the same kind of job as Cython, except it isn’t
specialised to bind to Python, it can bind to anything written in C or C++.
Including the Python API as illustrated.

Right now, I’ve writing a program to translate Include/python/*.pxd files to 
Felix bindings, to save typing everything by hand. Its even possible I can teach
my compiler to read *.pxd files directly, which would make Cython a sublanguage
of Felix. Felix is very good at defining DSSLs, Domain Specific Sub Languages,
the grammar of the language is defined in the standard library.

In the process of doing this I may find some issues in Cython which I’ll report.
Hope that helps. As I learn more I may be able to contribute to the project
directly.

One possible future goal is to replace NumPy with something much better.

I also have code written in Python that I might translate to C using Cython.
It would be kind of interesting to use Cython to generate C, and then create
bindings to that C in Felix, so instead of calling the Python C API, we call
the Cython generated API instead, allowing people to write libraries for
Felix in Cython instead of C or Felix. However that’s a more major integration 
task.
You’d want Cython to generate the Felix bindings, or at least output meta-data
that would allow them to be generated easily .. such as .. a *.pxd file !!


—
John Skaller
skal...@internode.on.net





___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hello

2020-01-27 Thread John Skaller2

> On 27 Jan 2020, at 20:23, Greg Ewing  wrote:
> 
> On 27/01/20 6:56 pm, John Skaller2 wrote:
>> Felix binds C/C++ code with statements like:
>>  type PyObject = “PyObject*”;
>>  fun add: PyObject * PyObject -> PyObject = “Py_AddLong($1)”;
>> and can use the bindings like:
>>  
>>  var a : PyObject = ….
>>  var b: PyObject = ...
>>  var sum = add (a,b);
> 
> How does it deal with reference counting and exception handling?

The Felix binding generates C. So the answer is, the same way as you would in C.

However the above is a low level binding, one to one with C. 
Higher level stuff is possible. Won’t know until the binding is operational.
My script is basically working translating the *.pxd files in Cython.
(Just the functions, the rest can be done by hand).

Basically I can write, say, a concurrent matrix multiplication routine and
map it to Python. Of course you can do that in C as well, but Felix is a very
high level language so you can do stuff you wouldn’t bother attempting
in low level languages like C and C++.

—
John Skaller
skal...@internode.on.net

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hello

2020-01-27 Thread John Skaller2

Ok, quick pass on Cython reference .. pretty good. 
Easy to read, good coverage, well written.

Beginning to understand the design.

Interesting thing is the limitations are the best clue. For example:

"The “?” indicates that the value -1 only indicates a possible error. 
In this case, Cython generates a call to PyErr_Occurred() if the exception 
value is returned, 
to make sure it really is an error.”

really tells me how Cython generates code to interface C and Python.

Thanks.

—
John Skaller
skal...@internode.on.net





___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hello

2020-01-28 Thread John Skaller2

What’s a “descr” when its at home?

There are some special words in pxd files like “list” which mean
PyObject that happens to be a list. Is there a list of these somewhere?

—
John Skaller
skal...@internode.on.net





___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hello

2020-01-28 Thread John Skaller2

> On 28 Jan 2020, at 21:07, Stefan Behnel  wrote:
> 
> John Skaller2 schrieb am 28.01.20 um 10:57:
>> What’s a “descr” when its at home?
> 
> A descriptor, a special protocol in Python.
> 
> https://docs.python.org/3/howto/descriptor.html

Got it, thanks!

> 
>> There are some special words in pxd files like “list” which mean
>> PyObject that happens to be a list. Is there a list of these somewhere?
> 
> See Builtin.py.

Got it, thanks! Very useful file.

 I’m curious about this comment in that file:

# This conflicts with the C++ bool type, and unfortunately
# C++ is too liberal about PyObject* <-> bool conversions,
# resulting in unintuitive runtime behavior and segfaults.
#("bool","PyBool_Type", []),

I can believe it. Bool is a mess in C, twice as bad in C++ ..
and exponentially screwed up in Felix :-)

The problem is compatibility in bindings..

The thing is I did translate bint in pxd files to Felix bool,
which is a bit, embedded in a 64 bit unsigned int .. different
to both C and C++ :-)

Now you have me worried!  Any more
detailed explanation of this issue?

—
John Skaller
skal...@internode.on.net

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hello

2020-01-28 Thread John Skaller2

So I just had a thought, I might try something to enhance the bindings.
Not sure if it will work. 

My system supports subtyping. For bindings and other nominal types
you can simple define a subtyping coercion.

Now in Python we have subtyping rules like:

dictionary -> mapping -> object

If I implement that with just static casts for coercions, then say,
given a dictionary and a function accepting a mapping argument,
you can pass a dictionary or mapping, but not an object.
If you want to pass an object you have to explicitly coerce it.

The *problem* with this idea I think is that from memory the reference counting
rules for PyObject and dictionaries are not the same as for methods
on abstractions. 

Cython must do something like this, right? And get the ref counting right.


—
John Skaller
skal...@internode.on.net





___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

Re: [Cython] Hello

2020-01-28 Thread John Skaller2


> 
> For your system, you may need a way of annotating function
> declarations with refcounting rules where they differ from
> the defaults.

Yeah, and i have to encode that to some extent in the type system.
However I have two techs to play with: Felix and C++. Both can,
for example, do subtyping. In addition, going

pxd [Cython] -> C -> Felix

allow safely and efficiently extending the “Python C API”.

—
John Skaller
skal...@internode.on.net





___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

[Cython] Status

2020-01-29 Thread John Skaller2

Ok, I have processed most of the Cython/Include/python/*.pxd files.

Seems these were mechanically derived from docs? Because some
of the bugs in the docs are also in these files. Not sure what you
would like to do so I’ll ask here before doing github issues.

1. Its not all there. I couldn’t find the API for generators for example.
If the API is extracted from the docs the script could be run again.

2. pyport is plain wrong. It contains conflicting C typedefs.

3. Some of the files describe extension modules (datetime, array)
rather than the core API.

4. ../cython/Cython/Includes/cpython/weakref.pxd:bint
PyWeakref_CheckProxy(ob)

ob should be PyObject*

there’s a couple more (maybe in ref as well).

Python keeps changing the API. Its not clear exactly which version is
represented.
I have the same issue, i have no idea at the moment how to deal with it.

I’m thinking to do some test runs of Cython on some Python code. I’ll be
running Cython as a translator directly from the command line.

Now when I built Cython before with “make” I saw lots of “python2.7” in the
output which is not what I want to process.

My Felix test code is executing plain C code bound againt a brew installed
Python run time library, the meta data required to link that is this:

Name: Pycore
Description: Macosx bindings to Python shared run time library core.
cflags:
-I/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/include/python3.6m
includes: '"Python.h"'
provides_dlib:
-L/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib
-lpython3.6m
provides_slib:
-L/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib
-lpython3.6m

And I can run the corresponding Python:

~/felix>/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/bin/python3.6m
Python 3.6.5 (default, Apr 25 2018, 14:26:36)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

And here is a test program written in Felix that uses the bindings:

println$ Py_GetVersion();
Py_Initialize();

var x : PyObject = PyLong_FromLong (42);
var y : long = PyLong_AsLong (x);
println$ PyLong_Check x;
println$ y;
println$ (f"%x" header_version);

and here’s the output:

~/felix>flx pycheck.flx
3.6.5 (default, Apr 25 2018, 14:26:36)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
true
42
30605f0

Note this code is generating a shared library BUT it is being run by Felix,
it is NOT being loaded as a module under Python. I will check that works
next.

So I’m showing all this because I would now like to run Cython on a little
test script using the same run time library as above, and perhaps the
same Python interpreter. I have too many Python’s installed all over the
place to really know what’s what.

So I would appreciate step by step instructions on how to build and use
Cython so it uses and generates code binding to the exact same
Python/run time as above. I guess I’ll test some kind of hello world
program.

===

I’m expecting that on OSX and Windows, CPython extensions will work,
but on Linux they will fail. This is because last time I looked, extensions
were built assuming a flat linker namespace with the API provided by the
Python executable. Indeed, I’m not sure if you can even get a Python
executable on Linux that isn’t statically linked to the run time.

However the shared library extensions I generate will always be linked
against a shared library. My build tech does not permit flat namespaces
for dynamic linkage. I know you CAN get Python run time as a shared
library on Debian based systems, but I don’t think you can get
a stub Python executable. Its essential that all the global data is held
by the library not the executable, otherwise it won’t be shared by
libraries that link to it.

However stuff may have changed recently. I can of course build Python
from source myself and hack it to build correctly, its not that hard,
but Felix clients will not be able to do that.

This issue will impact Cython uses exactly the same way for the same reason.

—
John Skaller
skal...@internode.on.net

___
cython-devel mailing list
cython-devel@python.org
https://mail.python.org/mailman/listinfo/cython-devel

30 matches

Mail list logo