Re: python developer
2022-09-29, Mike Dewhirst schrieb: > This is an OpenPGP/MIME signed message (RFC 4880 and 3156) Why? [Schnipp] -- Jan v/d Broek [email protected] -- https://mail.python.org/mailman/listinfo/python-list
Re: PyObject_CallFunctionObjArgs segfaults
Thanks very much for your detailed reply. I have a few followup questions.
You said, “Some functions return an object that has already been incref'ed
("new reference"). This occurs when it has either created a new object (the
refcount will be 1) or has returned a pointer to an existing object (the
refcount will be > 1 because it has been incref'ed). Other functions return an
object that hasn't been incref'ed. This occurs when you're looking up
something, for example, looking at a member of a list or the value of an
attribute.”
In the official docs some functions show “Return value: New reference” and
others do not. Is there any reason why I should not just INCREF on every new
object, regardless of whether it’s a new reference or not, and DECREF when I am
finished with it? The answer at
https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to
says “With out-of-order execution, the INCREF/DECREF are basically free
operations, so performance is no reason to leave them out.” Doing so means I
don’t have to check each object to see if it needs to be INCREF’d or not, and
that is a big help.
Also:
What is a borrowed reference, and how does it effect reference counting?
According to https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use
Py_INCREF on a borrowed PyObject pointer you already have. This increments the
reference count on the object, and obligates you to dispose of it properly.”
So I guess it’s yes, but I’m confused by “pointer you already have.”
What does it mean to steal a reference? If a function steals a reference does
it have to decref it without incref (because it’s stolen)?
Finally, you said:
if (pMod_random == 0x0){
PyErr_Print();
Leaks here because of the refcount
Assuming pMod_random is not null, why would this leak?
Thanks again for your input on this question.
Jen
Sep 29, 2022, 17:33 by [email protected]:
> On 2022-09-30 01:02, MRAB wrote:
>
>> On 2022-09-29 23:41, Jen Kris wrote:
>>
>>>
>>> I just solved this C API problem, and I’m posting the answer to help anyone
>>> else who might need it.
>>>
> [snip]
>
> What I like to do is write comments that state which variables hold a
> reference, followed by '+' if it's a new reference (incref'ed) and '?' if it
> could be null. '+?' means that it's probably a new reference but could be
> null. Once I know that it's not null, I can remove the '?', and once I've
> decref'ed it (if required) and no longer need it, I remobe it from the
> comment.
>
> Clearing up references, as soon as they're not needed, helps to keep the
> number of current references more manageable.
>
>
> int64_t Get_LibModules(int64_t * return_array) {
> PyObject * pName_random = PyUnicode_FromString("random");
> //> pName_random+?
> if (!pName_random) {
> PyErr_Print();
> return 1;
> }
>
> //> pName_random+
> PyObject * pMod_random = PyImport_Import(pName_random);
> //> pName_random+ pMod_random+?
> Py_DECREF(pName_random);
> //> pMod_random+?
> if (!pMod_random) {
> PyErr_Print();
> return 1;
> }
>
> //> pMod_random+
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
> //> pMod_random+ pAttr_seed?
> if (!pAttr_seed) {
> Py_DECREF(pMod_random);
> PyErr_Print();
> return 1;
> }
>
> //> pMod_random+ pAttr_seed
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
> "randrange");
> //> pMod_random+ pAttr_seed pAttr_randrange?
> Py_DECREF(pMod_random);
> //> pAttr_seed pAttr_randrange?
> if (!pAttr_randrange) {
> PyErr_Print();
> return 1;
> }
>
> //> pAttr_seed pAttr_randrange
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> //> value_ptr+?
> if (!!value_ptr) {
> PyErr_Print();
> return 1;
> }
>
> //> value_ptr+
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr,
> NULL);
> //> value_ptr+ p_seed_calc+?
> Py_DECREF(value_ptr);
> //> p_seed_calc+?
> if (!p_seed_calc) {
> PyErr_Print();
> return 1;
> }
>
> //> p_seed_calc+
> Py_DECREF(p_seed_calc);
> return 0;
> }
>
> int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> //> value_ptr+?
> if (!value_ptr) {
> PyErr_Print();
> return 1;
> }
>
> //> value_ptr+
> PyObject * p_randrange_calc = PyObject_CallFunctionObjArgs(pAttr_randrange,
> value_ptr, NULL);
> //> value_ptr+ p_randrange_calc+?
> Py_DECREF(value_ptr);
> //> p_randrange_calc+?
> if (!p_randrange_calc) {
> PyErr_Print();
> return 1;
> }
>
> //Prepare return values
> //> p_randrange_calc+
> return_val = PyLong_AsLong(p_randrange_calc);
> Py_DECREF(p_randrange_calc);
>
> return return_val;
> }
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
Re: Implementation of an lru_cache() decorator that ignores the first argument
On Thursday, September 29th, 2022 at 07:18, Robert Latest via Python-list
wrote:
> Hi Chris and dh,
>
> thanks for your --as usually-- thoughtful and interesting answers. Indeed,
> when
> doing these web applications I find that there are several layers of useful,
> maybe less useful, and unknown caching. Many of my requests rely on a
> notoriously unreliable read-only database outside of my control, so I cache
> the
> required data into a local DB on my server, then I do some in-memory caching
> of
> expensive data plots because I haven't figured out how to reliably exploit the
> client-side caching ... then every middleware on that path may or may not
> implement its own version of clever or not-so-clever caching. Probably not a
> good idea to try and outsmart that by adding yet another thing that may break
> or not be up-to-date at the wrong moment.
>
> That said, the only caching that SQLAlchemy does (to my knowledge) is that it
> stores retrieved DB items by their primary keys in the session. Not worth much
> since the session gets created and dumped on each request by SQA's unit of
> work
> paradigm. But the DB backend itself may be caching repeated queries.
>
> Back to Python-theory: The "Cloak" object is the only way I could think of to
> sneak changing data past lru_cache's key lookup mechanism. Is there some other
> method? Just curious.
>
> --
> https://mail.python.org/mailman/listinfo/python-list
You could use closures. For example, something like this:
import functools
import time
def my_cache(timeout):
start = time.monotonic()
def cache_decorator(func):
wrapper = _my_cache_wrapper(func, timeout, start)
return functools.update_wrapper(wrapper, func)
return cache_decorator
def _my_cache_wrapper(func, timeout, start):
first = None
@functools.cache
def _cached(timeout_factor, *args):
print("In the _cached function")
return func(first, *args)
def wrapper(*args):
print("In the wrapper")
nonlocal first
first, *rest = args
elapsed = time.monotonic() - start
timeout_factor = elapsed // timeout
return _cached(timeout_factor, *rest)
return wrapper
@my_cache(3)
def expensive(first, second, third):
print("In the expensive function")
return (first, second, third)
if __name__ == "__main__":
print(expensive(1, 2, 3))
print()
time.sleep(2)
print(expensive(2, 2, 3))
print()
time.sleep(2)
print(expensive(3, 2, 3))
This should output the following:
In the wrapper
In the _cached function
In the expensive function
(1, 2, 3)
In the wrapper
(1, 2, 3)
In the wrapper
In the _cached function
In the expensive function
(3, 2, 3)
It's not necessarily better than your version though. :D
Kind regards,
Heinrich Kruger
--
https://mail.python.org/mailman/listinfo/python-list
Re: python developer
On 2022-09-30 05:31:15 -, Jan van den Broek wrote: > 2022-09-29, Mike Dewhirst schrieb: > > This is an OpenPGP/MIME signed message (RFC 4880 and 3156) > > Why? Why not? Signing an email protects against impersonation and tampering. While you may not be assured that my name is indeed "Peter J. Holzer", you can be sure that all the messages signed with the same key came from the same person and that they were not modified by somebody else (at least not the signed part). I've been signing all my (private) mails for 25 years or so. hp -- _ | Peter J. Holzer| Story must make more sense than reality. |_|_) || | | | [email protected] |-- Charles Stross, "Creative writing __/ | http://www.hjp.at/ | challenge!" signature.asc Description: PGP signature -- https://mail.python.org/mailman/listinfo/python-list
Re: PyObject_CallFunctionObjArgs segfaults
On 2022-09-30 17:02, Jen Kris wrote:
Thanks very much for your detailed reply. I have a few followup
questions.
You said, “Some functions return an object that has already been
incref'ed ("new reference"). This occurs when it has either created a
new object (the refcount will be 1) or has returned a pointer to an
existing object (the refcount will be > 1 because it has been
incref'ed). Other functions return an object that hasn't been
incref'ed. This occurs when you're looking up something, for example,
looking at a member of a list or the value of an attribute.”
In the official docs some functions show “Return value: New reference”
and others do not. Is there any reason why I should not just INCREF
on every new object, regardless of whether it’s a new reference or
not, and DECREF when I am finished with it? The answer at
https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to
says “With out-of-order execution, the INCREF/DECREF are basically
free operations, so performance is no reason to leave them out.”
Doing so means I don’t have to check each object to see if it needs to
be INCREF’d or not, and that is a big help.
It's OK to INCREF them, provided that you DECREF them when you no longer
need them, and remember that if it's a "new reference" you'd need to
DECREF it twice.
Also:
What is a borrowed reference, and how does it effect reference
counting? According to
https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use
Py_INCREF on a borrowed PyObject pointer you already have. This
increments the reference count on the object, and obligates you to
dispose of it properly.” So I guess it’s yes, but I’m confused by
“pointer you already have.”
A borrowed reference is when it hasn't been INCREFed.
You can think of INCREFing as a way of indicating ownership, which is
often shared ownership (refcount > 1). When you're borrowing a
reference, you're using it temporarily, but not claiming ownership. When
the last owner releases its ownership (DECREF reduces the refcount to
0), the object can be garbage collected.
When, say, you lookup an attribute, or get an object from a list with
PyList_GetItem, it won't have been INCREFed. You're using it
temporarily, just borrowing a reference.
What does it mean to steal a reference? If a function steals a
reference does it have to decref it without incref (because it’s stolen)?
When function steals a reference, it's claiming ownership but not
INCREFing it.
Finally, you said:
if (pMod_random == 0x0){
PyErr_Print();
Leaks here because of the refcount
Assuming pMod_random is not null, why would this leak?
It's pName_random that's the leak.
PyUnicode_FromString("random") will either create and return a new
object for the string "random" (refcount == 1) or return a reference to
an existing object (refcount > 1). You need to DECREF it before
returning from the function.
Suppose it created a new object. You call the function, it creates an
object, you use it, then return from the function. The object still
exists, but there's no reference to it. Now call the function again. It
creates another object, you use it, then return from the function. You
now have 2 objects with no reference to them.
Thanks again for your input on this question.
Jen
Sep 29, 2022, 17:33 by [email protected]:
On 2022-09-30 01:02, MRAB wrote:
On 2022-09-29 23:41, Jen Kris wrote:
I just solved this C API problem, and I’m posting the
answer to help anyone else who might need it.
[snip]
What I like to do is write comments that state which variables
hold a reference, followed by '+' if it's a new reference
(incref'ed) and '?' if it could be null. '+?' means that it's
probably a new reference but could be null. Once I know that it's
not null, I can remove the '?', and once I've decref'ed it (if
required) and no longer need it, I remobe it from the comment.
Clearing up references, as soon as they're not needed, helps to
keep the number of current references more manageable.
int64_t Get_LibModules(int64_t * return_array) {
PyObject * pName_random = PyUnicode_FromString("random");
//> pName_random+?
if (!pName_random) {
PyErr_Print();
return 1;
}
//> pName_random+
PyObject * pMod_random = PyImport_Import(pName_random);
//> pName_random+ pMod_random+?
Py_DECREF(pName_random);
//> pMod_random+?
if (!pMod_random) {
PyErr_Print();
return 1;
}
//> pMod_random+
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
//> pMod_random+ pAttr_seed?
if (!pAttr_seed) {
Py_DECREF(pMod_random);
PyErr_Print();
return 1;
}
//> pMod_random+ pAttr_seed
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");
//> pMod_random+ pAttr_seed pAttr_
Re: PyObject_CallFunctionObjArgs segfaults
That's great. It clarifies things a lot for me, particularly re ref count for new references. I would have had trouble if I didn't decref it twice. Thanks very much once again. Sep 30, 2022, 12:18 by [email protected]: > On 2022-09-30 17:02, Jen Kris wrote: > >> >> Thanks very much for your detailed reply. I have a few followup questions. >> >> You said, “Some functions return an object that has already been incref'ed >> ("new reference"). This occurs when it has either created a new object (the >> refcount will be 1) or has returned a pointer to an existing object (the >> refcount will be > 1 because it has been incref'ed). Other functions return >> an object that hasn't been incref'ed. This occurs when you're looking up >> something, for example, looking at a member of a list or the value of an >> attribute.” >> >> In the official docs some functions show “Return value: New reference” and >> others do not. Is there any reason why I should not just INCREF on every >> new object, regardless of whether it’s a new reference or not, and DECREF >> when I am finished with it? The answer at >> https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to >> says “With out-of-order execution, the INCREF/DECREF are basically free >> operations, so performance is no reason to leave them out.” Doing so means >> I don’t have to check each object to see if it needs to be INCREF’d or not, >> and that is a big help. >> > It's OK to INCREF them, provided that you DECREF them when you no longer need > them, and remember that if it's a "new reference" you'd need to DECREF it > twice. > >> Also: >> >> What is a borrowed reference, and how does it effect reference counting? >> According to https://jayrambhia.com/blog/pythonc-api-reference-counting, >> “Use Py_INCREF on a borrowed PyObject pointer you already have. This >> increments the reference count on the object, and obligates you to dispose >> of it properly.” So I guess it’s yes, but I’m confused by “pointer you >> already have.” >> > > A borrowed reference is when it hasn't been INCREFed. > > You can think of INCREFing as a way of indicating ownership, which is often > shared ownership (refcount > 1). When you're borrowing a reference, you're > using it temporarily, but not claiming ownership. When the last owner > releases its ownership (DECREF reduces the refcount to 0), the object can be > garbage collected. > > When, say, you lookup an attribute, or get an object from a list with > PyList_GetItem, it won't have been INCREFed. You're using it temporarily, > just borrowing a reference. > >> >> What does it mean to steal a reference? If a function steals a reference >> does it have to decref it without incref (because it’s stolen)? >> > When function steals a reference, it's claiming ownership but not INCREFing > it. > >> >> Finally, you said: >> >> if (pMod_random == 0x0){ >> PyErr_Print(); >> Leaks here because of the refcount >> >> Assuming pMod_random is not null, why would this leak? >> > It's pName_random that's the leak. > > PyUnicode_FromString("random") will either create and return a new object for > the string "random" (refcount == 1) or return a reference to an existing > object (refcount > 1). You need to DECREF it before returning from the > function. > > Suppose it created a new object. You call the function, it creates an object, > you use it, then return from the function. The object still exists, but > there's no reference to it. Now call the function again. It creates another > object, you use it, then return from the function. You now have 2 objects > with no reference to them. > >> Thanks again for your input on this question. >> >> Jen >> >> >> >> Sep 29, 2022, 17:33 by [email protected]: >> >> On 2022-09-30 01:02, MRAB wrote: >> >> On 2022-09-29 23:41, Jen Kris wrote: >> >> >> I just solved this C API problem, and I’m posting the >> answer to help anyone else who might need it. >> >> [snip] >> >> What I like to do is write comments that state which variables >> hold a reference, followed by '+' if it's a new reference >> (incref'ed) and '?' if it could be null. '+?' means that it's >> probably a new reference but could be null. Once I know that it's >> not null, I can remove the '?', and once I've decref'ed it (if >> required) and no longer need it, I remobe it from the comment. >> >> Clearing up references, as soon as they're not needed, helps to >> keep the number of current references more manageable. >> >> >> int64_t Get_LibModules(int64_t * return_array) { >> PyObject * pName_random = PyUnicode_FromString("random"); >> //> pName_random+? >> if (!pName_random) { >> PyErr_Print(); >> return 1; >> } >> >> //> pName_random+ >> PyObject * pMod_random = PyImport_Import(pName_random); >> //> pName_random+ pMod_random+? >> Py_DECREF(pName_random); >> //> pMod_random+? >> if (!pMod_r
Re: PyObject_CallFunctionObjArgs segfaults
On 1/10/22 8:18 am, MRAB wrote: It's OK to INCREF them, provided that you DECREF them when you no longer need them, and remember that if it's a "new reference" you'd need to DECREF it twice. Which means there would usually be no point in doing the extra INCREF/DECREF. You still need to know whether it's a new reference or not and treat it accordingly. -- Greg -- https://mail.python.org/mailman/listinfo/python-list
Re: python developer
On 30/09/2022 3:31 pm, Jan van den Broek wrote: 2022-09-29, Mike Dewhirst schrieb: This is an OpenPGP/MIME signed message (RFC 4880 and 3156) Why? Good question. Further to Peter's explanation, email is the primary conduit for hackers. At this point in time human education and training is the only defence. The bad guys count on human error because that's what humans are good at. If everyone signed their mail and all mail clients defaulted to expecting signed email from correspondents for whom the public key is known, some of that defence can be delegated to the computer. I'm not expecting this any time soon but it doesn't hurt to get the message out. Most email activists demand end-to-end encryption and obviously signing email is part of that. However, my view is that email privacy, while very important, is an oxymoron. If you need encrypted messages you would never use email. You would meet under a waterfall. So the answer to your question is signed email is easy and if it becomes popular it has potential to defeat hackers. Cheers Mike [Schnipp] -- Signed email is an absolute defence against phishing. This email has been signed with my private key. If you import my public key you can automatically decrypt my signature and be sure it came from me. Just ask and I'll send it to you. Your email software can handle signing. OpenPGP_signature Description: OpenPGP digital signature -- https://mail.python.org/mailman/listinfo/python-list
