[issue37347] Reference-counting problem in sqlite
New submission from Aleksandr Balezin :
There are a couple of bugs in sqlite bindings have been found related to
reference-counting.
Short info:
sqlite connection class uses a dict to keep references to callbacks. This
mechanism is not suitable for objects which equals but not the same.
con = sqlite3.connect()
con.set_trace_callback(logger.debug)
con.set_trace_callback(logger.debug) # logger.debug == logger.debug is True,
but logger.debug is logger.debug is False because logger.debug bound method is
creating in every call
leads to segmentation fault during calling of trace_callback.
My patch fixes this behavior by using a dedicated variable for keeping
references to each callback and using dict indexed by function name in case of
named callbacks(e.g. create_function()).
Also, due to keeping objects in a variable or in a dict value, it is possible
to use unhashable objects as callbacks. e.g. issue7478
Long version:
Sqlite under the hood use dict(called function_pinboard) to keep references to
callbacks like progress_handler.
It needs to decref callbacks objects after closing sqlite connection.
This mechanism works tolerably(see bug with leaks) with functions but if you
try to use bounded methods it causes a segmentation fault.
Let see how it works.
static PyObject *
Custom_set_callback(CustomObject *self, PyObject* args)
{
PyObject* display_str;
display_str = PyUnicode_FromFormat("set_callback called with cb=%R
id=%i ob_refcnt=%i\n", args, args, args->ob_refcnt);
PyObject_Print(display_str, stdout, Py_PRINT_RAW);
if (PyDict_SetItem(self->function_pinboard, args, Py_None) == -1)
return NULL;
//sqlite3_trace(self->db, _trace_callback, trace_callback);
self->callback_handler = args;
display_str = PyUnicode_FromFormat("set_callback done for cb=%R id=%i
ob_refcnt=%i\n", args, args, args->ob_refcnt);
PyObject_Print(display_str, stdout, Py_PRINT_RAW);
Py_RETURN_NONE;
}
static PyObject *
Custom_call_callback(CustomObject *self)
{
PyObject* display_str;
display_str = PyUnicode_FromFormat("call with id=%i ob_refcnt=%i\n",
self->callback_handler ,
self->callback_handler->ob_refcnt);
PyObject_Print(display_str, stdout, Py_PRINT_RAW);
Py_RETURN_NONE;
}
Python code:
>>>> class TEST:
def log(self, msg=""):
pass
>>>> t = TEST()
>>>> conn = Custom()
>>>> conn.set_trace_callback(t.log)
set_callback called with cb=> id=196094408 ob_refcnt=1
set_callback done for cb=> id=196094408 ob_refcnt=2
>>>> conn.set_trace_callback(t.log)
set_callback called with cb=> id=196095112 ob_refcnt=1
set_callback done for cb=> id=196095112 ob_refcnt=1
conn.call()
call with id=196095112 ob_refcnt=0
After second conn.set_trace_callback(t.log) call, object t.log reference-count
is not increased because 't.log in self->function_pinboard' returns True thus
self->function_pinboard[t.log] is not replaced and t.log is not increfed, but
it replaces old object in self->callback_handler.
In the end, self->callback_handler keeps a pointer to t.log with ob_refcnt = 0.
Also, there is no cleaning of self->function_pinboard. This leads to leaks
every object passed as callback(see test_leak() in bug.py).
--
components: Extension Modules
messages: 346114
nosy: gescheit, ghaering
priority: normal
severity: normal
status: open
title: Reference-counting problem in sqlite
type: crash
versions: Python 2.7, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python 3.9
___
Python tracker
<https://bugs.python.org/issue37347>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37347] Reference-counting problem in sqlite
Change by Aleksandr Balezin : Added file: https://bugs.python.org/file48430/bug.py ___ Python tracker <https://bugs.python.org/issue37347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37347] Reference-counting problem in sqlite
Change by Aleksandr Balezin : -- keywords: +patch pull_requests: +14092 stage: -> patch review pull_request: https://github.com/python/cpython/pull/14265 ___ Python tracker <https://bugs.python.org/issue37347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37347] Reference-counting problem in sqlite
Change by Aleksandr Balezin : -- pull_requests: +14094 pull_request: https://github.com/python/cpython/pull/14268 ___ Python tracker <https://bugs.python.org/issue37347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37347] Reference-counting problem in sqlite
Change by Aleksandr Balezin : -- pull_requests: -14092 ___ Python tracker <https://bugs.python.org/issue37347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37347] Reference-counting problem in sqlite
Aleksandr Balezin added the comment: Because destructor can be registered only for particular functions. For example sqlite3_progress_handler() can't register destructor. -- ___ Python tracker <https://bugs.python.org/issue37347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37347] Reference-counting problem in sqlite
Aleksandr Balezin added the comment: At my point of view, dererencing from sqlite destructor has next flaws: - relying on external library which can has bugs - using different routine for reference-counting(complicates code) - loosing support for old sqlite version Of cause, if it's necessary to use sqlite3_create_function_v2, I'm ready to make appropriate changes. -- ___ Python tracker <https://bugs.python.org/issue37347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue37347] Reference-counting problem in sqlite
Aleksandr Balezin added the comment: I've pushed changes https://github.com/python/cpython/pull/14268/commits/bce7fdc952b14c23821e184e82b3944f6e10aaa9. Could I ask for clarification on callback and Py_DECREF(something)? -- ___ Python tracker <https://bugs.python.org/issue37347> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25430] speed up ipaddress __contain__ method
Aleksandr Balezin added the comment: I think this patch can be easily applied. There are no any breaking changes. My colleagues and I assume that ipaddress module would work fast because it is easy to imagine tasks, operate with ton of networks. Also, current comparison implementation works as not so good pattern for other developers. -- ___ Python tracker <http://bugs.python.org/issue25430> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue20825] containment test for "ip_network in ip_network"
Aleksandr Balezin added the comment:
I've reviewed this patch and want to make some advices.
- hasattr is unwanted here. There is no any similar usage hasattr in this
module. Also before hasattr there is a call of _ipversion method. If other is
not instance of BaseNetwork it will raise AttributeError exception before
hasattr check.
- It is not a good manner comparing thru "other.network_address >=
self.network_address and other.broadcast_address <= self.broadcast_address"(see
issue25430). Networks must be compared thru "other._prefixlen >=
self._prefixlen and other.network._ip & self.netmask._ip == self.network._ip"
for performance reason.
- _containment_check function is excessive. There is no common logic in
supernet_of/subnet_of function except _ipversion and type checking. I think
this two functions should be simple as:
def subnet_of(self, other):
if self._version != other._version:
raise TypeError('%s and %s are not of the same version' % (self, other))
if other._prefixlen >= self._prefixlen and other.network._ip &
self.netmask._ip == self.network._ip:
return True
return False
--
nosy: +gescheit
___
Python tracker
<http://bugs.python.org/issue20825>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36613] asyncio._wait() don't remove callback in case of exception
New submission from Aleksandr Balezin : Attached script shows unexpected behavior of the wait() function. The wait_ function adds done callback on every call and removes it only if a waiter is successfully awaited. In case of CancelledError exception during "await waiter", callbacks are being accumulated infinitely in task._callbacks. -- components: asyncio files: asyncio_wait_callbacks_leak.py messages: 340034 nosy: asvetlov, gescheit, yselivanov priority: normal severity: normal status: open title: asyncio._wait() don't remove callback in case of exception type: resource usage versions: Python 3.6, Python 3.7, Python 3.8 Added file: https://bugs.python.org/file48263/asyncio_wait_callbacks_leak.py ___ Python tracker <https://bugs.python.org/issue36613> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue36613] asyncio._wait() don't remove callback in case of exception
Change by Aleksandr Balezin : -- keywords: +patch pull_requests: +12728 stage: -> patch review ___ Python tracker <https://bugs.python.org/issue36613> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25430] speed up ipaddress __contain__ method
Changes by Aleksandr Balezin : -- versions: +Python 3.4, Python 3.6 ___ Python tracker <http://bugs.python.org/issue25430> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25431] implement address in network in ipaddress module
Changes by Aleksandr Balezin : -- versions: +Python 3.4, Python 3.6 ___ Python tracker <http://bugs.python.org/issue25431> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue25431] implement address in network in ipaddress module
Aleksandr Balezin added the comment: Any news? -- ___ Python tracker <http://bugs.python.org/issue25431> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
