Re: on GNU EMACS's python-mode, loading entire buffer
On Sun, 04 Sep 2022 16:47:07 -0300 Meredith Montgomery wrote: > Meredith Montgomery writes: > >> Meredith Montgomery writes: >> >> [...] >> >>> I would also be interested in a command that restarts the REPL afresh >>> and reloads my buffer --- sort of like keyboard's [F5] of the IDLE. >> >> A partial solution for this is the following procedure. >> >> (defun python-revert-and-send-buffer-to-repl () >> "Revert current buffer and sends it to the Python REPL." >> (interactive) >> (revert-buffer "ignore-auto-no" "no-confirm") >> (python-shell-send-buffer)) >> >> We can map this to the F5-key and that improves things. But a restart >> of the REPL would be the ideal. (Sometimes we really want to start >> afresh. Sometimes. Most often we don't want that.) > > It's not easy to restart the REPL. You can send "quit()" to it and > invoke run-python again interactively by typing out one command after > another, but if you write a procedure such as this one below, it doesn't > work: it gives me the impression that there's a timing issue, that is, > perhaps the procedure is too fast and something happens before it > should. > > (defun python-save-send-buffer-to-repl () > (interactive) > (save-buffer) > (python-shell-send-string "quit()") > (run-python) > (python-shell-send-buffer) > (python-shell-switch-to-shell)) It does seem like a timing issue. This works for me: (defun python-save-send-buffer-to-repl () (interactive) (save-buffer) (python-shell-send-string "quit()") (sit-for 0.1) (run-python) (python-shell-send-buffer) (python-shell-switch-to-shell)) But if I decrease the wait to 0.05 it doesn't work. Steve Berman -- https://mail.python.org/mailman/listinfo/python-list
PyObject_CallFunctionObjArgs segfaults
Recently I completed a project where I used PyObject_CallFunctionObjArgs
extensively with the NLTK library from a program written in NASM, with no
problems. Now I am on a new project where I call the Python random library. I
use the same setup as before, but I am getting a segfault with random.seed.
At the start of the NASM program I call a C API program that gets PyObject
pointers to “seed” and “randrange” in the same way as I did before:
int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);
if (pMod_random == 0x0){
PyErr_Print();
return 1;}
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;
return 0;
}
Later in the same program I call a C API program to call random.seed:
int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
if (p_seed_calc == 0x0){
PyErr_Print();
return 1;}
//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);
return return_val;
}
The first program correctly imports “random” and gets pointers to “seed” and
“randrange.” I verified that the same pointer is correctly passed into
C_API_2, and the seed value (1234) is passed as Py_ssize_t value_1. But I get
this segfault:
Program received signal SIGSEGV, Segmentation fault.
0x764858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459
459 ../Include/object.h: No such file or directory.
So I tried Py_INCREF in the first program:
Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);
Then I moved Py_INCREF(pAttr_seed) to the second program. Same segfault.
Finally, I initialized “random” and “seed” in the second program, where they
are used. Same segfault.
The segfault refers to Py_INCREF, so this seems to do with reference counting,
but Py_INCREF didn’t solve it.
I’m using Python 3.8 on Ubuntu.
Thanks for any ideas on how to solve this.
Jen
--
https://mail.python.org/mailman/listinfo/python-list
Re: PyObject_CallFunctionObjArgs segfaults
On 2022-09-29 16:54, Jen Kris via Python-list wrote:
Recently I completed a project where I used PyObject_CallFunctionObjArgs
extensively with the NLTK library from a program written in NASM, with no
problems. Now I am on a new project where I call the Python random library. I
use the same setup as before, but I am getting a segfault with random.seed.
At the start of the NASM program I call a C API program that gets PyObject
pointers to “seed” and “randrange” in the same way as I did before:
int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);
Both PyUnicode_FromString and PyImport_Import return new references or
null pointers.
if (pMod_random == 0x0){
PyErr_Print();
You're leaking a reference here (pName_random).
return 1;}
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;
You're leaking 2 references here (pName_random and pMod_random).
return 0;
}
Later in the same program I call a C API program to call random.seed:
int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
It's expecting all of the arguments to be PyObject*, but value_1 is
Py_ssize_t instead of PyObject* (a pointer to a _Python_ int).
The argument list must end with a null pointer.
It returns a new reference or a null pointer.
if (p_seed_calc == 0x0){
PyErr_Print();
return 1;}
//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);
You're leaking a reference here (p_seed_calc).
return return_val;
}
The first program correctly imports “random” and gets pointers to “seed” and
“randrange.” I verified that the same pointer is correctly passed into
C_API_2, and the seed value (1234) is passed as Py_ssize_t value_1. But I get
this segfault:
Program received signal SIGSEGV, Segmentation fault.
0x764858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459
459 ../Include/object.h: No such file or directory.
So I tried Py_INCREF in the first program:
Py_INCREF(pMod_random);
Py_INCREF(pAttr_seed);
Then I moved Py_INCREF(pAttr_seed) to the second program. Same segfault.
Finally, I initialized “random” and “seed” in the second program, where they
are used. Same segfault.
The segfault refers to Py_INCREF, so this seems to do with reference counting,
but Py_INCREF didn’t solve it.
I’m using Python 3.8 on Ubuntu.
Thanks for any ideas on how to solve this.
Jen
--
https://mail.python.org/mailman/listinfo/python-list
Re: on implementing a toy oop-system
[email protected] (Stefan Ram) writes: > Meredith Montgomery writes: >>The code below works, but you can see it's kinda ugly. I wish I could >>uncurry a procedure, but I don't think this is possible. (Is it?) > > from functools import partial > from operator import add > add5 = partial( add, 5 ) > print( add5( 2 )) > # might be dependent on implementation details of "functools": > uncurried = add5.func > print( uncurried( 14, 7 )) It works on my system here. Thank you so much for your help. (I've been learning a lot with you!) > However, to evaluate a method call such as "o.m( a, a1, ... )", > currying does not necessarily have to be used. One can as well > determine the function to be used for "m" from the type of "o" > and then call that function with arguments "o", "a", "a1", ... Was that your last approach? I only glanced at it so far. But I remember seeing you describing an object and listing out which methods were defined for it and then there was one procedure that would take the role of invoking the methods (with the proper arguments). -- https://mail.python.org/mailman/listinfo/python-list
Re: Implementation of an lru_cache() decorator that ignores the first argument
Hi Chris and dh, thanks for your --as usually-- thoughtful and interesting answers. Indeed, when doing these web applications I find that there are several layers of useful, maybe less useful, and unknown caching. Many of my requests rely on a notoriously unreliable read-only database outside of my control, so I cache the required data into a local DB on my server, then I do some in-memory caching of expensive data plots because I haven't figured out how to reliably exploit the client-side caching ... then every middleware on that path may or may not implement its own version of clever or not-so-clever caching. Probably not a good idea to try and outsmart that by adding yet another thing that may break or not be up-to-date at the wrong moment. That said, the only caching that SQLAlchemy does (to my knowledge) is that it stores retrieved DB items by their primary keys in the session. Not worth much since the session gets created and dumped on each request by SQA's unit of work paradigm. But the DB backend itself may be caching repeated queries. Back to Python-theory: The "Cloak" object is the only way I could think of to sneak changing data past lru_cache's key lookup mechanism. Is there some other method? Just curious. -- https://mail.python.org/mailman/listinfo/python-list
Re: on implementing a toy oop-system
[email protected] (Stefan Ram) writes: [...] >>>However, to evaluate a method call such as "o.m( a, a1, ... )", >>>currying does not necessarily have to be used. One can as well >>>determine the function to be used for "m" from the type of "o" >>>and then call that function with arguments "o", "a", "a1", ... >>Was that your last approach? > > Yes, I think so. > > (There are also languages with "multi-methods", where upon > a function call "m( o, o1, ... )" the decision which function > to call depends on all the types of all the arguments.) I think Clojure is one such. I've read Part 1 of ``Clojure in Action'' by Amit Rathore, 2012, Manning, ISBN 9781935182597. I liked it. -- https://mail.python.org/mailman/listinfo/python-list
Re: non-standard glibc location
On Friday, September 8, 2017 at 3:16:12 AM UTC+2, Fetchinson . wrote:
> On 9/7/17, Thomas Jollans wrote:
> > On 2017-09-06 16:14, Fetchinson . via Python-list wrote:
> >> Hi folks,
> >>
> >> I'm trying to install a binary package (tensorflow) which contains
> >> some binary C extensions. Now my system glibc is 2.15 but the binaries
> >> in the C extensions were created (apparently) with glibc 2.17. So I
> >> thought no problemo I installed glibc 2.17 to a custom location, built
> >> python2.7 from source (hopefully using my custom glibc) and installed
> >> pip and everything else using this custom built python. But still when
> >> I try to import tensorflow I get:
> >>
> >> ImportError: /lib64/libc.so.6: version `GLIBC_2.17' not found
> >> (required by
> >> /home/nogradi/fetch/custom/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
> >>
> >>
> >> So apparently it's trying to use my system glibc, not the custom one.
> >>
> >> How do I tell this extension to use the custom glibc? Is it even
> >> possible?
> >
> > It's going to use the same libc as python, so first of all check which
> > libc your python interpreter is actually linked to. Maybe your custom
> > build didn't do quite what you wanted.
> >
> > ldd `which python` # or something like that
> >
> > Once you've convinced yourself that python has the correct libc, you
> > could try building tensorflow from source rather than installing the
> > binaries.
> >
> > Maybe something in here helps:
> > https://github.com/tensorflow/tensorflow/issues/53
> Thanks a lot for all the comments, my problem was indeed that the
> compiled python was still using the system glibc. The solution was to
> set the environment variables
>
> p=/path/to/custom/glibc
> export CFLAGS=-I${p}/include
> export LDFLAGS="-Wl,--rpath=${p}/lib
> -Wl,--dynamic-linker=${p}/lib/ld-linux-x86-64.so.2"
>
> And then the compiled python was using the new glibc.
>
> Side note 1 on tensorflow: the compiled tensorflow binary uses unicode
> ucs4, so for python I had to ./configure --enable-unicode=ucs4 because
> the default is ucs2
>
> Side note 2 on tensorflow: it also depends on libstdc++ and my version
> was too old for that as well. Instead of compiling gcc from source
> (which includes libstdc++) I copied a binary libstdc++ from a newer
> linux distro and it was working fine.
>
> And the reason, if anyone cares, I had to go through the above is that
> I couldn't compile tensorflow from source.
>
> Thanks again,
> Daniel
> >
> >>
> >> But maybe I have an even more basic issue: how do I link python not
> >> with the system glibc but with my custom glibc?
> >>
> >> Cheers,
> >> Daniel
> >>
> >>
> >>
> >
> >
> > --
> > Thomas Jollans
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
>
>
> --
> Psss, psss, put it down! - http://www.cafepress.com/putitdown
First of all, thank you a lot Fetchinson, it helped on my system (Centos 6) to
today recompile the newest Python (3.10.7)!
Of course I had foss2018 available on the system, so I "just" had to compile
glibc, zlib, and openssl 1.1.1q ... but hey it was worth it! I have my
environment up and running!
FYI:
1. to me it helped that since I had nonstandard openssl location to edit
Modules/Setup.dist as suggested here
https://stackoverflow.com/questions/5937337/building-python-with-ssl-support-in-non-standard-location
2. to me it helped to configure with
./configure --with-libc=/tmp/glibc/lib/libc.so --prefix=/tmp/python
--with-openssl=/tmp/openssl
Hope this helps someone else who will try to do the same.
Cheers
--
https://mail.python.org/mailman/listinfo/python-list
python developer
Hi all, I has been using python in scientific computing for many years I want to become a python developer so what is a good reference to follow ? Thanks in advance ! Walid -- https://mail.python.org/mailman/listinfo/python-list
Re: PyObject_CallFunctionObjArgs segfaults
Thanks very much to @MRAB for taking time to answer. I changed my code to
conform to your answer (as best I understand your comments on references), but
I still get the same error. My comments continue below the new code
immediately below.
int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);
Py_INCREF(pName_random);
Py_INCREF(pMod_random);
if (pMod_random == 0x0){
PyErr_Print();
return 1;}
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;
return 0;
}
int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = (PyObject * )value_1;
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr,
NULL);
if (p_seed_calc == 0x0){
PyErr_Print();
return 1;}
//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);
return return_val;
}
So I incremented the reference to all objects in Get_LibModules, but I still
get the same segfault at PyObject_CallFunctionObjArgs. Unfortunately,
reference counting is not well documented so I’m not clear what’s wrong.
Sep 29, 2022, 10:06 by [email protected]:
> On 2022-09-29 16:54, Jen Kris via Python-list wrote:
>
>> Recently I completed a project where I used PyObject_CallFunctionObjArgs
>> extensively with the NLTK library from a program written in NASM, with no
>> problems. Now I am on a new project where I call the Python random library.
>> I use the same setup as before, but I am getting a segfault with
>> random.seed.
>>
>> At the start of the NASM program I call a C API program that gets PyObject
>> pointers to “seed” and “randrange” in the same way as I did before:
>>
>> int64_t Get_LibModules(int64_t * return_array)
>> {
>> PyObject * pName_random = PyUnicode_FromString("random");
>> PyObject * pMod_random = PyImport_Import(pName_random);
>>
> Both PyUnicode_FromString and PyImport_Import return new references or null
> pointers.
>
>> if (pMod_random == 0x0){
>> PyErr_Print();
>>
>
> You're leaking a reference here (pName_random).
>
>> return 1;}
>>
>> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
>> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
>> "randrange");
>>
>> return_array[0] = (int64_t)pAttr_seed;
>> return_array[1] = (int64_t)pAttr_randrange;
>>
>
> You're leaking 2 references here (pName_random and pMod_random).
>
>> return 0;
>> }
>>
>> Later in the same program I call a C API program to call random.seed:
>>
>> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
>> {
>> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
>>
>
> It's expecting all of the arguments to be PyObject*, but value_1 is
> Py_ssize_t instead of PyObject* (a pointer to a _Python_ int).
>
> The argument list must end with a null pointer.
>
> It returns a new reference or a null pointer.
>
>>
>> if (p_seed_calc == 0x0){
>> PyErr_Print();
>> return 1;}
>>
>> //Prepare return values
>> long return_val = PyLong_AsLong(p_seed_calc);
>>
> You're leaking a reference here (p_seed_calc).
>
>> return return_val;
>> }
>>
>> The first program correctly imports “random” and gets pointers to “seed” and
>> “randrange.” I verified that the same pointer is correctly passed into
>> C_API_2, and the seed value (1234) is passed as Py_ssize_t value_1. But I
>> get this segfault:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x764858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459
>> 459 ../Include/object.h: No such file or directory.
>>
>> So I tried Py_INCREF in the first program:
>>
>> Py_INCREF(pMod_random);
>> Py_INCREF(pAttr_seed);
>>
>> Then I moved Py_INCREF(pAttr_seed) to the second program. Same segfault.
>>
>> Finally, I initialized “random” and “seed” in the second program, where they
>> are used. Same segfault.
>>
>> The segfault refers to Py_INCREF, so this seems to do with reference
>> counting, but Py_INCREF didn’t solve it.
>>
>> I’m using Python 3.8 on Ubuntu.
>>
>> Thanks for any ideas on how to solve this.
>>
>> Jen
>>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: PyObject_CallFunctionObjArgs segfaults
To update my previous email, I found the problem, but I have a new problem. Previously I cast PyObject * value_ptr = (PyObject * )value_1 but that's not correct. Instead I used PyObject * value_ptr = PyLong_FromLong(value_1) and that works. HOWEVER, while PyObject_CallFunctionObjArgs does work now, it returns -1, which is not the right answer for random.seed. I use "long return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long. So my question is why do I get -1 as return value? When I query p_seed calc : get: (gdb) p p_seed_calc $2 = (PyObject *) 0x769be120 <_Py_NoneStruct> Thanks again. Jen Sep 29, 2022, 13:02 by [email protected]: > Thanks very much to @MRAB for taking time to answer. I changed my code to > conform to your answer (as best I understand your comments on references), > but I still get the same error. My comments continue below the new code > immediately below. > > int64_t Get_LibModules(int64_t * return_array) > { > PyObject * pName_random = PyUnicode_FromString("random"); > PyObject * pMod_random = PyImport_Import(pName_random); > > Py_INCREF(pName_random); > Py_INCREF(pMod_random); > > if (pMod_random == 0x0){ > PyErr_Print(); > return 1;} > > PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed"); > PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange"); > > Py_INCREF(pAttr_seed); > Py_INCREF(pAttr_randrange); > > return_array[0] = (int64_t)pAttr_seed; > return_array[1] = (int64_t)pAttr_randrange; > > return 0; > } > > int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) > { > PyObject * value_ptr = (PyObject * )value_1; > PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, > NULL); > > if (p_seed_calc == 0x0){ > PyErr_Print(); > return 1;} > > //Prepare return values > long return_val = PyLong_AsLong(p_seed_calc); > > return return_val; > } > > So I incremented the reference to all objects in Get_LibModules, but I still > get the same segfault at PyObject_CallFunctionObjArgs. Unfortunately, > reference counting is not well documented so I’m not clear what’s wrong. > > > > > Sep 29, 2022, 10:06 by [email protected]: > >> On 2022-09-29 16:54, Jen Kris via Python-list wrote: >> >>> Recently I completed a project where I used PyObject_CallFunctionObjArgs >>> extensively with the NLTK library from a program written in NASM, with no >>> problems. Now I am on a new project where I call the Python random >>> library. I use the same setup as before, but I am getting a segfault with >>> random.seed. >>> >>> At the start of the NASM program I call a C API program that gets PyObject >>> pointers to “seed” and “randrange” in the same way as I did before: >>> >>> int64_t Get_LibModules(int64_t * return_array) >>> { >>> PyObject * pName_random = PyUnicode_FromString("random"); >>> PyObject * pMod_random = PyImport_Import(pName_random); >>> >> Both PyUnicode_FromString and PyImport_Import return new references or null >> pointers. >> >>> if (pMod_random == 0x0){ >>> PyErr_Print(); >>> >> >> You're leaking a reference here (pName_random). >> >>> return 1;} >>> >>> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed"); >>> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, >>> "randrange"); >>> >>> return_array[0] = (int64_t)pAttr_seed; >>> return_array[1] = (int64_t)pAttr_randrange; >>> >> >> You're leaking 2 references here (pName_random and pMod_random). >> >>> return 0; >>> } >>> >>> Later in the same program I call a C API program to call random.seed: >>> >>> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) >>> { >>> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1); >>> >> >> It's expecting all of the arguments to be PyObject*, but value_1 is >> Py_ssize_t instead of PyObject* (a pointer to a _Python_ int). >> >> The argument list must end with a null pointer. >> >> It returns a new reference or a null pointer. >> >>> >>> if (p_seed_calc == 0x0){ >>> PyErr_Print(); >>> return 1;} >>> >>> //Prepare return values >>> long return_val = PyLong_AsLong(p_seed_calc); >>> >> You're leaking a reference here (p_seed_calc). >> >>> return return_val; >>> } >>> >>> The first program correctly imports “random” and gets pointers to “seed” >>> and “randrange.” I verified that the same pointer is correctly passed into >>> C_API_2, and the seed value (1234) is passed as Py_ssize_t value_1. But I >>> get this segfault: >>> >>> Program received signal SIGSEGV, Segmentation fault. >>> 0x764858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459 >>> 459 ../Include/object.h: No such file or directory. >>> >>> So I tried Py_INCREF in the first program: >>> >>> Py_INCREF(pMod_random); >>> Py_INCREF(pAttr_seed); >>> >>> Then I moved Py_INCREF(pAttr_seed) to the second program. Same segfault. >>> >>> Finally, I initialized “random” and “seed” in the second program, where >>> they are use
Re: PyObject_CallFunctionObjArgs segfaults
On 2022-09-29 21:47, Jen Kris wrote: To update my previous email, I found the problem, but I have a new problem. Previously I cast PyObject * value_ptr = (PyObject * )value_1 but that's not correct. Instead I used PyObject * value_ptr = PyLong_FromLong(value_1) and that works. HOWEVER, while PyObject_CallFunctionObjArgs does work now, it returns -1, which is not the right answer for random.seed. I use "long return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long. random.seed returns None, so when you call PyObject_CallFunctionObjArgs it returns a new reference to Py_None. If you then pass to PyLong_AsLong a reference to something that's not a PyLong, it'll set an error and return -1. So my question is why do I get -1 as return value? When I query p_seed calc : get: (gdb) p p_seed_calc $2 = (PyObject *) 0x769be120 <_Py_NoneStruct> Exactly. It's Py_None, not a PyLong. Thanks again. Jen Sep 29, 2022, 13:02 by [email protected]: Thanks very much to @MRAB for taking time to answer. I changed my code to conform to your answer (as best I understand your comments on references), but I still get the same error. My comments continue below the new code immediately below. int64_t Get_LibModules(int64_t * return_array) { PyObject * pName_random = PyUnicode_FromString("random"); PyObject * pMod_random = PyImport_Import(pName_random); Py_INCREF(pName_random); Py_INCREF(pMod_random); if (pMod_random == 0x0){ PyErr_Print(); return 1;} PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed"); PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange"); Py_INCREF(pAttr_seed); Py_INCREF(pAttr_randrange); return_array[0] = (int64_t)pAttr_seed; return_array[1] = (int64_t)pAttr_randrange; return 0; } int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) { PyObject * value_ptr = (PyObject * )value_1; PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL); if (p_seed_calc == 0x0){ PyErr_Print(); return 1;} //Prepare return values long return_val = PyLong_AsLong(p_seed_calc); return return_val; } So I incremented the reference to all objects in Get_LibModules, but I still get the same segfault at PyObject_CallFunctionObjArgs. Unfortunately, reference counting is not well documented so I’m not clear what’s wrong. Sep 29, 2022, 10:06 by [email protected]: On 2022-09-29 16:54, Jen Kris via Python-list wrote: Recently I completed a project where I used PyObject_CallFunctionObjArgs extensively with the NLTK library from a program written in NASM, with no problems. Now I am on a new project where I call the Python random library. I use the same setup as before, but I am getting a segfault with random.seed. At the start of the NASM program I call a C API program that gets PyObject pointers to “seed” and “randrange” in the same way as I did before: int64_t Get_LibModules(int64_t * return_array) { PyObject * pName_random = PyUnicode_FromString("random"); PyObject * pMod_random = PyImport_Import(pName_random); Both PyUnicode_FromString and PyImport_Import return new references or null pointers. if (pMod_random == 0x0){ PyErr_Print(); You're leaking a reference here (pName_random). return 1;} PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed"); PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange"); return_array[0] = (int64_t)pAttr_seed; return_array[1] = (int64_t)pAttr_randrange; You're leaking 2 references here (pName_random and pMod_random). return 0; } Later in the same program I call a C API program to call random.seed: int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) { PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1); It's expecting all of the arguments to be PyObject*, but value_1 is Py_ssize_t instead of PyObject* (a pointer to a _Python_ int). The argument list must end with a null pointer. It returns a new reference or a null pointer. if (p_seed_calc == 0x0){ PyErr_Print(); return 1;} //Prepare return values long return_val = PyLong_AsLong(p_seed_calc); You're leaking a reference here (p_seed_calc). return return_val; } The first program correctly imports “random” and gets pointe
Re: PyObject_CallFunctionObjArgs segfaults
I just solved this C API problem, and I’m posting the answer to help anyone
else who might need it.
The errors were:
(1) we must call Py_INCREF on each object when it’s created.
(2) in C_API_2 (see below) we don’t cast value_1 as I did before with PyObject
* value_ptr = (PyObject * )value_1. Instead we use PyObject * value_ptr =
PyLong_FromLong(value_1);
(3) The command string to PyObject_CallFunctionObjArgs must be null terminated.
Here’s the revised code:
First we load the modules, and increment the reference to each object:
int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);
Py_INCREF(pName_random);
Py_INCREF(pMod_random);
if (pMod_random == 0x0){
PyErr_Print();
return 1;}
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;
return 0;
}
Next we call a program to initialize the random number generator with
random.seed(), and increment the reference to its return value p_seed_calc:
int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr,
NULL);
// _
if (p_seed_calc == 0x0){
PyErr_Print();
return 1;}
Py_INCREF(p_seed_calc);
return 0;
}
Now we call another program to get a random number:
int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
PyObject * p_randrange_calc = PyObject_CallFunctionObjArgs(pAttr_randrange,
value_ptr, NULL);
if (p_randrange_calc == 0x0){
PyErr_Print();
return 1;}
//Prepare return values
long return_val = PyLong_AsLong(p_randrange_calc);
return return_val;
}
That returns 28, which is what I get from the Python command line.
Thanks again to MRAB for helpful comments.
Jen
Sep 29, 2022, 15:31 by [email protected]:
> On 2022-09-29 21:47, Jen Kris wrote:
>
>> To update my previous email, I found the problem, but I have a new problem.
>>
>> Previously I cast PyObject * value_ptr = (PyObject * )value_1 but that's not
>> correct. Instead I used PyObject * value_ptr = PyLong_FromLong(value_1) and
>> that works. HOWEVER, while PyObject_CallFunctionObjArgs does work now, it
>> returns -1, which is not the right answer for random.seed. I use "long
>> return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long.
>>
> random.seed returns None, so when you call PyObject_CallFunctionObjArgs it
> returns a new reference to Py_None.
>
> If you then pass to PyLong_AsLong a reference to something that's not a
> PyLong, it'll set an error and return -1.
>
>> So my question is why do I get -1 as return value? When I query p_seed calc
>> : get:
>>
>> (gdb) p p_seed_calc
>> $2 = (PyObject *) 0x769be120 <_Py_NoneStruct>
>>
> Exactly. It's Py_None, not a PyLong.
>
>> Thanks again.
>>
>> Jen
>>
>>
>>
>>
>> Sep 29, 2022, 13:02 by [email protected]:
>>
>> Thanks very much to @MRAB for taking time to answer. I changed my
>> code to conform to your answer (as best I understand your comments
>> on references), but I still get the same error. My comments
>> continue below the new code immediately below.
>>
>> int64_t Get_LibModules(int64_t * return_array)
>> {
>> PyObject * pName_random = PyUnicode_FromString("random");
>> PyObject * pMod_random = PyImport_Import(pName_random);
>>
>> Py_INCREF(pName_random);
>> Py_INCREF(pMod_random);
>>
>> if (pMod_random == 0x0){
>> PyErr_Print();
>> return 1;}
>>
>> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
>> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
>> "randrange");
>>
>> Py_INCREF(pAttr_seed);
>> Py_INCREF(pAttr_randrange);
>>
>> return_array[0] = (int64_t)pAttr_seed;
>> return_array[1] = (int64_t)pAttr_randrange;
>>
>> return 0;
>> }
>>
>> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
>> {
>> PyObject * value_ptr = (PyObject * )value_1;
>> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
>> value_ptr, NULL);
>>
>> if (p_seed_calc == 0x0){
>> PyErr_Print();
>> return 1;}
>>
>> //Prepare return values
>> long return_val = PyLong_AsLong(p_seed_calc);
>>
>> return return_val;
>> }
>>
>> So I incremented the reference to all objects in Get_LibModules,
>> but I still get the same segfault at
>> PyObject_CallFunctionObjArgs. Unfortunately, reference counting
>> is not well documented so I’m not clear what’s wrong.
>>
>>
>>
>>
>> Sep 29, 2022, 10:06 by [email protected]:
>>
>> On 2022-09-29 16:54, Jen Kris via Python-list wrote:
>>
>> Recently I completed a project where I used
>> PyObject_CallFunctionObjArgs extensively wi
Re: python developer
On 30/09/2022 5:09 am, Walid AlMasri wrote: Hi all, I has been using python in scientific computing for many years I want to become a python developer so what is a good reference to follow ? A developer makes products whereas a scientist understands complexities. I would recommend reading about Scrum https://www.scrum.org/resources/what-is-scrum After many years you probably already know enough Python to do anything. Your task as a developer is to understand the relationship between a problem (business or scientific) and a maintainable, perhaps packaged, product which handles the drudgery and leaves expert judgement to the human user(s). Scrum is the easiest way down that road. You need a customer with a problem. You might be your own first customer. Cheers Mike Thanks in advance ! Walid -- Signed email is an absolute defence against phishing. This email has been signed with my private key. If you import my public key you can automatically decrypt my signature and be sure it came from me. Just ask and I'll send it to you. Your email software can handle signing. OpenPGP_signature Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
Re: PyObject_CallFunctionObjArgs segfaults
On 2022-09-29 23:41, Jen Kris wrote:
I just solved this C API problem, and I’m posting the answer to help
anyone else who might need it.
The errors were:
(1) we must call Py_INCREF on each object when it’s created.
Some functions return an object that has already been incref'ed ("new
reference"). This occurs when it has either created a new object (the
refcount will be 1) or has returned a pointer to an existing object (the
refcount will be > 1 because it has been incref'ed).
Other functions return an object that hasn't been incref'ed. This occurs
when you're looking up something, for example, looking at a member of a
list or the value of an attribute.
(2) in C_API_2 (see below) we don’t cast value_1 as I did before with
PyObject * value_ptr = (PyObject * )value_1. Instead we use PyObject
* value_ptr = PyLong_FromLong(value_1);
(3) The command string to PyObject_CallFunctionObjArgs must be null
terminated.
Always read the docs carefully!
Here’s the revised code:
First we load the modules, and increment the reference to each object:
int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);
Possible null pointers here.
Py_INCREF(pName_random);
Py_INCREF(pMod_random);
Py_INCREF will fail on a null pointer, which you haven't check for yet.
If they aren't null, then they're new references (incref'ed) that you've
incref'ed.
if (pMod_random == 0x0){
PyErr_Print();
Leaks here because of the refcount.
return 1;}
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");
Possible null pointers here too.
Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);
Same issue as above.
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;
return 0;
}
Next we call a program to initialize the random number generator with
random.seed(), and increment the reference to its return value
p_seed_calc:
int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
New reference.
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);
// _
if (p_seed_calc == 0x0){
PyErr_Print();
Leak.
return 1;}
Py_INCREF(p_seed_calc);
p_seed_calc will be a new reference to Py_None, incref'ed again, so leak.
return 0;
}
Now we call another program to get a random number:
int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
PyObject * p_randrange_calc =
PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
if (p_randrange_calc == 0x0){
Leak.
PyErr_Print();
return 1;}
//Prepare return values
long return_val = PyLong_AsLong(p_randrange_calc);
Leak.
return return_val;
}
That returns 28, which is what I get from the Python command line.
Thanks again to MRAB for helpful comments.
Jen
Getting refcounting right can be difficult!
Sep 29, 2022, 15:31 by [email protected]:
On 2022-09-29 21:47, Jen Kris wrote:
To update my previous email, I found the problem, but I have a
new problem.
Previously I cast PyObject * value_ptr = (PyObject * )value_1
but that's not correct. Instead I used PyObject * value_ptr =
PyLong_FromLong(value_1) and that works. HOWEVER, while
PyObject_CallFunctionObjArgs does work now, it returns -1,
which is not the right answer for random.seed. I use "long
return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long.
random.seed returns None, so when you call
PyObject_CallFunctionObjArgs it returns a new reference to Py_None.
If you then pass to PyLong_AsLong a reference to something that's
not a PyLong, it'll set an error and return -1.
So my question is why do I get -1 as return value? When I
query p_seed calc : get:
(gdb) p p_seed_calc
$2 = (PyObject *) 0x769be120 <_Py_NoneStruct>
Exactly. It's Py_None, not a PyLong.
Thanks again.
Jen
Sep 29, 2022, 13:02 by [email protected]:
Thanks very much to @MRAB for taking time to answer. I changed my
code to conform to your answer (as best I understand your comments
on references), but I still get the same error. My comments
continue below the new code immediately below.
int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);
Py_INCREF(pName_random);
Py_INCREF(pMod_random);
if (pMod_random == 0x0){
PyErr_Print();
return 1;}
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random,
"seed");
PyObject * pAttr_randr
Re: PyObject_CallFunctionObjArgs segfaults
On 2022-09-30 01:02, MRAB wrote:
On 2022-09-29 23:41, Jen Kris wrote:
I just solved this C API problem, and I’m posting the answer to help
anyone else who might need it.
[snip]
What I like to do is write comments that state which variables hold a
reference, followed by '+' if it's a new reference (incref'ed) and '?'
if it could be null. '+?' means that it's probably a new reference but
could be null. Once I know that it's not null, I can remove the '?', and
once I've decref'ed it (if required) and no longer need it, I remobe it
from the comment.
Clearing up references, as soon as they're not needed, helps to keep the
number of current references more manageable.
int64_t Get_LibModules(int64_t * return_array) {
PyObject * pName_random = PyUnicode_FromString("random");
//> pName_random+?
if (!pName_random) {
PyErr_Print();
return 1;
}
//> pName_random+
PyObject * pMod_random = PyImport_Import(pName_random);
//> pName_random+ pMod_random+?
Py_DECREF(pName_random);
//> pMod_random+?
if (!pMod_random) {
PyErr_Print();
return 1;
}
//> pMod_random+
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
//> pMod_random+ pAttr_seed?
if (!pAttr_seed) {
Py_DECREF(pMod_random);
PyErr_Print();
return 1;
}
//> pMod_random+ pAttr_seed
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");
//> pMod_random+ pAttr_seed pAttr_randrange?
Py_DECREF(pMod_random);
//> pAttr_seed pAttr_randrange?
if (!pAttr_randrange) {
PyErr_Print();
return 1;
}
//> pAttr_seed pAttr_randrange
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;
return 0;
}
int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);
//> value_ptr+?
if (!!value_ptr) {
PyErr_Print();
return 1;
}
//> value_ptr+
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);
//> value_ptr+ p_seed_calc+?
Py_DECREF(value_ptr);
//> p_seed_calc+?
if (!p_seed_calc) {
PyErr_Print();
return 1;
}
//> p_seed_calc+
Py_DECREF(p_seed_calc);
return 0;
}
int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);
//> value_ptr+?
if (!value_ptr) {
PyErr_Print();
return 1;
}
//> value_ptr+
PyObject * p_randrange_calc =
PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
//> value_ptr+ p_randrange_calc+?
Py_DECREF(value_ptr);
//> p_randrange_calc+?
if (!p_randrange_calc) {
PyErr_Print();
return 1;
}
//Prepare return values
//> p_randrange_calc+
return_val = PyLong_AsLong(p_randrange_calc);
Py_DECREF(p_randrange_calc);
return return_val;
}
--
https://mail.python.org/mailman/listinfo/python-list
