#36939: Connecting the same signal multiple times with weak=True allocates new
memory that is never deallocated
-------------------------------------+-------------------------------------
Reporter: phantom-jacob | Type: Bug
Status: new | Component: Core
| (Other)
Version: 6.0 | Severity: Normal
Keywords: signal connect weak | Triage Stage:
weakref | Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
We recently observed steadily increasing memory usage in our Django
application running under uwsgi with celery workers, which we tracked down
to multiple calls to connect the same signal handlers. We eventually
figured out that when `weak=True` on signal connection (the default
behavior), each call to `connect()` allocates new memory (which
`tracemalloc` indicates is largely from `weakref`) on every call. This
memory is not cleared by disconnecting the signal, and in practice we
never saw the memory free before the system eventually OOMed.
Anyway, I've developed a standalone reproduction script, which can be run
with `uv run django_mem_leak.py`. If the `--weak` argument is provided,
you can see memory grow drastically:
{{{
uv run django_mem_leak.py
Installed 3 packages in 92ms
Running memory audit with 100000 iterations, weak=False
Starting memory usage: 28.88 MiB
[100000/100000] Memory usage: 30.48 MiB
Final memory usage: 30.52 MiB
----------------------------------------------------------------------------------------------------
[ Top 10 memory allocations ]
<frozen importlib._bootstrap_external>:784: size=178 KiB (+178 KiB),
count=1678 (+1678), average=109 B
<frozen importlib._bootstrap>:488: size=119 KiB (+119 KiB), count=1379
(+1379), average=89 B
<frozen importlib._bootstrap_external>:801: size=8156 B (+8156 B),
count=68 (+68), average=120 B
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/pathlib/_local.py:59: size=4133 B (+4133 B),
count=18 (+18), average=230 B
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/pathlib/_abc.py:403: size=3595 B (+3595 B),
count=10 (+10), average=360 B
<frozen importlib._bootstrap_external>:133: size=3479 B (+3479 B),
count=21 (+21), average=166 B
~/.cache/uv/environments-v2/django-mem-leak-
5f36d9b360a80206/lib/python3.13/site-
packages/django/conf/global_settings.py:433: size=3264 B (+3264 B),
count=1 (+1), average=3264 B
<frozen importlib._bootstrap_external>:1704: size=3086 B (+3086 B),
count=33 (+33), average=94 B
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/pathlib/_abc.py:55: size=2915 B (+2915 B),
count=11 (+11), average=265 B
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/pathlib/_local.py:482: size=2802 B (+2802 B),
count=13 (+13), average=216 B
uv run django_mem_leak.py --weak
Running memory audit with 100000 iterations, weak=True
Starting memory usage: 27.36 MiB
[100000/100000] Memory usage: 102.73 MiB
Final memory usage: 102.73 MiB
----------------------------------------------------------------------------------------------------
[ Top 10 memory allocations ]
~/.cache/uv/environments-v2/django-mem-leak-
5f36d9b360a80206/lib/python3.13/site-
packages/django/dispatch/dispatcher.py:120: size=9375 KiB (+9375 KiB),
count=200001 (+200001), average=48 B
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/weakref.py:576: size=7813 KiB (+7813 KiB),
count=100001 (+100001), average=80 B
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/weakref.py:575: size=7812 KiB (+7812 KiB),
count=100000 (+100000), average=80 B
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/weakref.py:582: size=5120 KiB (+5120 KiB),
count=1 (+1), average=5120 KiB
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/weakref.py:581: size=2727 KiB (+2727 KiB),
count=99743 (+99743), average=28 B
<frozen importlib._bootstrap_external>:784: size=233 KiB (+233 KiB),
count=2363 (+2363), average=101 B
<frozen importlib._bootstrap>:488: size=40.9 KiB (+40.9 KiB), count=328
(+328), average=128 B
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/pathlib/_local.py:59: size=4133 B (+4133 B),
count=18 (+18), average=230 B
~/.local/share/uv/python/cpython-3.13.2-macos-
aarch64-none/lib/python3.13/pathlib/_abc.py:403: size=3715 B (+3715 B),
count=11 (+11), average=338 B
<frozen importlib._bootstrap_external>:133: size=3479 B (+3479 B),
count=21 (+21), average=166 B
}}}
Use this script to reproduce:
{{{
# /// script
# requires-python = ">=3.13"
# dependencies = [
# "django",
# ]
# ///
import argparse
import resource
import sys
import tracemalloc
from django.dispatch import Signal
def signal_handler(sender, **kwargs):
pass
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Run a memory audit to demonstrate signal handler
leaks"
)
parser.add_argument(
"num",
type=int,
default=100000,
nargs="?",
help="Number of iterations to run the memory audit for",
)
parser.add_argument(
"--weak",
action="store_true",
help="Whether to use weak references for audit signals",
)
return parser.parse_args()
def _get_memory_usage() -> float:
maxrss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
# Mac reports in bytes, Linux reports in KiB
if sys.platform == "darwin":
return maxrss / 1024 / 1024
return maxrss / 1024
def main(
*,
num: int,
weak: bool,
) -> None:
signal = Signal()
print(f"Running memory audit with {num} iterations, {weak=}")
print(f"Starting memory usage: {_get_memory_usage():.2f} MiB")
tracemalloc.start()
start_snapshot = tracemalloc.take_snapshot()
for i in range(1, num + 1):
print(
f"[{i}/{num}] Memory usage: {_get_memory_usage():.2f} MiB",
end="\r",
flush=True,
)
signal.connect(signal_handler, weak=weak,
dispatch_uid="test_signal")
signal.disconnect(signal_handler, dispatch_uid="test_signal")
print("")
print(f"Final memory usage: {_get_memory_usage():.2f} MiB")
end_snapshot = tracemalloc.take_snapshot()
top_stats = end_snapshot.compare_to(start_snapshot, "lineno")
print("\n\n")
print("-" * 100)
print("\n\n[ Top 10 memory allocations ]")
for stat in top_stats[:10]:
print(stat)
tracemalloc.stop()
if __name__ == "__main__":
args = parse_args()
main(**vars(args))
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36939>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
--
You received this message because you are subscribed to the Google Groups
"Django updates" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/django-updates/0107019c7f25af1c-56a120b5-e6fd-4a78-baf1-a5f15ee79343-000000%40eu-central-1.amazonses.com.
