[Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python
Hi Has anyone used Cython for developing a module to be used from postgreSQL/pl/python. Something that calls back to PostgreSQL internals. --- Hannu Krosing PostgreSQL Unlimited Scalability and Performance Consultant 2ndQuadrant Nordic PG Admin Book: http://www.2ndQuadrant.com/books/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python
On Sat, 2012-03-31 at 14:41 +0200, Stefan Behnel wrote: > Hannu Krosing, 31.03.2012 14:22: > > Has anyone used Cython for developing a module to be used from > > postgreSQL/pl/python. > > > > Something that calls back to PostgreSQL internals. > > Note that this is the CPython core developers mailing list, for which your > question is off-topic. Please ask on the cython-users mailing list (see > http://cython.org). Thanks for the pointer ! Can you recommend a similar place for asking the same queston about ctypes ? That is using ctypes for calling back to "outer" c application which embeds the python interpreter. > Stefan > > ___ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/hannu%40krosing.net ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Cython for developing a module to be used from postgreSQL/pl/python
On Sun, 2012-04-01 at 00:44 +1100, Ben Finney wrote:
> Hannu Krosing writes:
>
> > Has anyone used Cython for developing a module to be used from
> > postgreSQL/pl/python.
> >
> > Something that calls back to PostgreSQL internals.
>
> Are you aware that PostgreSQL has long provided Python as a language for
> writing stored procedures in the database?
>
> http://www.postgresql.org/docs/current/static/plpython.html>
>
> Does that meet your needs?
Sure, I have even contributed code to it ;)
But what i want is some way to call _back_ into PostgreSQL to use its
internal functions from a module imported from pl/python.
I tried ctypes module and got a segfault when i used the following
create or replace function callback_to_postgres(rn_text text)
returns text
language plpythonu
as
$$
from ctypes import *
import struct
pg = cdll.LoadLibrary('/usr/lib/postgresql/9.1/bin/postmaster')
pg.pq_flush()
return rn_text
$$;
select send_raw_notice('do you see me?');
I determined that the call to pg.pq_flush() was the one crashing the
backend, so I assume that cdll.LoadLibrary did not get the already
running backend, but loaded another copy of it as library.
Now I'm looking for a simplest way to do some C-wrapping. I have used
Cyuthon for wrapping simple library calls and I really don'r think there
would be hard problems doing this inside the postgreSQL extension build
framework.
I was just hoping that someboduy had already taken care of all the
nitty-gritty detaiuls of setting this up
---
Hannu Krosing
PostgreSQL Unlimited Scalability and Performance Consultant
2ndQuadrant Nordic
PG Admin Book: http://www.2ndQuadrant.com/books/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Iterating generator from C (PostgreSQL's pl/python RETUN SETOF/RECORD iterator support broken on RedHat buggy libs)
I try to move this to -dev as I hope there more people reading it who
are competent in internal working :). So please replay to -dev only.
-
The question is about use of generators in embedde v2.4 with asserts
enabled.
Can somebody explain, why the code in try2.c works with wrappers 2 and 3
but crashes on buggy exception for all others, that is pure generator
and wrappers 1,4,5 ?
In other words, what does [i in i for gen] do differently than other
ways of iterating over gen, which helps it to avoid the assert, and
also, how could I write something that is still a generator, but does
not trigger the assert.
While the right solution is to just fix python (which is done for v2.5)
we need a workaround for the following reason
We have added support for returning rows (as tuples and or dictionaries
or classes) and sets of boths scalars and rows to PostgreSQL's
pl/plpython embedded language, but have some objections to getting this
into distribution, because there is a bug in python 2.4 doing a wrong
assert and additional bug in how RedHat rpm build process which leaves
the buggy asserts in python.so.
So I hoped to write a simple wrapper class, but it only seems to work,
when the generator is turned into list, which is not a good solution as
it works directly against what generators are good for.
--
Hannu Krosing
Database Architect
Skype Technologies OÜ
Akadeemia tee 21 F, Tallinn, 12618, Estonia
Skype me: callto:hkrosing
Get Skype for free: http://www.skype.com
#include
#include
const char *py_source =
"def fn():\n"
"print 'one'\n"
"yield 1\n"
"print 'two'\n"
"yield 2\n"
;
const char *wrapper_source1 =
"class gwrap:\n"
"def __init__(self,gen):\n"
"self.gen = gen\n"
"def __iter__(self):\n"
"return iter(self.gen)\n"
;
const char *wrapper_source2 =
"def gwrap(gen):\n"
"return [i for i in gen]\n"
;
const char *wrapper_source3 =
"def gwrap(gen):\n"
"return (x for x in [i for i in gen])\n"
;
const char *wrapper_source4 =
"class gwrap:\n"
"def __init__(self,gen):\n"
"list = [i for i in gen]\n"
"self.gen = list.__iter__()\n"
"def __iter__(self):\n"
"return iter(self.gen)\n"
;
const char *wrapper_source5 =
"def gwrap(gen):\n"
"return (i for i in gen)\n"
;
int main (int argc, char *argv[])
{
Py_Initialize();
PyObject *main_module = PyImport_AddModule("__main__");
PyObject *globals = PyObject_GetAttrString( main_module, "__dict__");
// insert function code into interpreter
PyObject *code = PyRun_String(py_source, Py_file_input, globals, NULL);
Py_DECREF(code);
// insert wrapper class into interpreter
PyObject *code2 = PyRun_String(wrapper_source5, Py_file_input, globals, NULL);
Py_DECREF(code2);
// compile call to the function
code = Py_CompileString("gwrap(fn())", "", Py_eval_input);
// code = Py_CompileString("fn()", "", Py_eval_input);
// do call
PyObject *gen = PyEval_EvalCode((PyCodeObject *)code, globals, NULL);
gen = PyObject_GetIter((PyObject *)gen);
// iterate result
PyObject *item;
while ((item = PyIter_Next(gen))) {
printf("> %ld\n", PyInt_AsLong(item));
Py_DECREF(item);
}
Py_DECREF(gen);
Py_DECREF(code);
Py_DECREF(globals);
Py_Finalize();
return 0;
}
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Iterating generator from C (PostgreSQL's pl/python RETUN SETOF/RECORD iterator support broken on RedHat buggy libs)
Ühel kenal päeval, P, 2006-05-28 kell 14:18, kirjutas Thomas Wouters: > > On 5/20/06, Hannu Krosing <[EMAIL PROTECTED]> wrote: > I try to move this to -dev as I hope there more people reading > it who > are competent in internal working :). So please replay to -dev > only. > > I'm not sure if you have found the problem on another mailinglist > then, but I saw no answers on python-dev. > > > - > > The question is about use of generators in embedde v2.4 with > asserts > enabled. > > Can somebody explain, why the code in try2.c works with > wrappers 2 and 3 > but crashes on buggy exception for all others, that is pure > generator > and wrappers 1,4,5 ? > > Your example code does not crash for me, not for any of the > 'wrapper_source' variants, linking against Python 2.4 (debian's python > 2.4.3 on amd64, both in 64-bit and 32-bit mode) or Python 2.5 (current > trunk, same hardware.) I don't know what kind of crash you were > expecting, but I do see unchecked return values, which can cause > crashes for the simplest of reasons ;-) Fedora Core distributes its python libs with asserts on, and this code triggers an assert, both without a wrapper and with most wrappers [EMAIL PROTECTED] plpython]$ ./try2 one try2: Objects/genobject.c:53: gen_iternext: Assertion `f->f_back != ((void *)0)' failed. Aborted This is reported to be fixed for 2.5 (they changed the assert), but I'd like to have a workaround which would not trigger the old buggy assert. > -- > Thomas Wouters <[EMAIL PROTECTED]> > > Hi! I'm a .signature virus! copy me into your .signature file to help > me spread! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
