Public bug reported: We're facing an issue where sometimes Python 3.8 and 3.9 crash while running out suite of pure-Python unit tests.
Python 3.10 and higher are not affected. The key seems to be multithreading and forking/multiprocessing. The segmentation fault seem to happen in ContextVar.set Symptom when using pytest-xdist: worker 'gw0' crashed while running 'testing/tests/test_e2e/test_...' I believe it's the same issue in py 3.8 and 3.9. I don't have a backtrace for 3.8 though. 3.9 provides a magic env var where Python dumps stack of all threads and I get this: ``` 2025-05-30T00:34:40.0891620Z Fatal Python error: Segmentation fault 2025-05-30T00:34:40.0892247Z testing/tests/test_e2e/test_state.py::test_bare_event 2025-05-30T00:34:40.0892663Z 2025-05-30T00:34:40.0893997Z [gw0]�[36m [ 85%] �[0m�[32mPASSED�[0m testing/tests/test_e2e/test_state.py::test_bare_event 2025-05-30T00:34:40.0895862Z Thread 0x00007f1a631b66c0 (most recent call first): 2025-05-30T00:34:40.0898168Z File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 316 in wait 2025-05-30T00:34:40.0899660Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 258 in worker 2025-05-30T00:34:40.0906958Z File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 917 in run 2025-05-30T00:34:40.0908249Z File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 980 in _bootstrap_inner 2025-05-30T00:34:40.0909611Z File "/opt/hostedtoolcache/Python/3.9.22/x64/lib/python3.9/threading.py", line 937 in _bootstrap 2025-05-30T00:34:40.0910341Z 2025-05-30T00:34:40.0910646Z Thread 0x00007f1a6a9ff6c0 (most recent call first): 2025-05-30T00:34:40.0911883Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 534 in read 2025-05-30T00:34:40.0913638Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 567 in from_io 2025-05-30T00:34:40.0915652Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver 2025-05-30T00:34:40.0917394Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 341 in run 2025-05-30T00:34:40.0919223Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn 2025-05-30T00:34:40.0920247Z 2025-05-30T00:34:40.0935905Z Current thread 0x00007f1a6be30b80 (most recent call first): 2025-05-30T00:34:40.0937432Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/context/contextvars_context.py", line 41 in attach 2025-05-30T00:34:40.0939416Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/context/__init__.py", line 144 in attach 2025-05-30T00:34:40.0941588Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/trace/__init__.py", line 585 in use_span 2025-05-30T00:34:40.0943532Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/util/_decorator.py", line 61 in __enter__ 2025-05-30T00:34:40.0945978Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/sdk/trace/__init__.py", line 1099 in start_as_current_span 2025-05-30T00:34:40.0948097Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/util/_decorator.py", line 61 in __enter__ 2025-05-30T00:34:40.0950167Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/trace/__init__.py", line 451 in start_as_current_span 2025-05-30T00:34:40.0952114Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/opentelemetry/util/_decorator.py", line 61 in __enter__ 2025-05-30T00:34:40.0966905Z File "/home/runner/work/operator/operator/ops/framework.py", line 1014 in _reemit 2025-05-30T00:34:40.0967960Z File "/home/runner/work/operator/operator/ops/framework.py", line 924 in _emit 2025-05-30T00:34:40.0969147Z File "/home/runner/work/operator/operator/testing/src/scenario/_runtime.py", line 397 in _wrapped_emit 2025-05-30T00:34:40.0970440Z File "/home/runner/work/operator/operator/testing/src/scenario/_runtime.py", line 397 in _wrapped_emit 2025-05-30T00:34:40.0971891Z File "/home/runner/work/operator/operator/testing/src/scenario/_runtime.py", line 397 in _wrapped_emit 2025-05-30T00:34:40.0988312Z File "/home/runner/work/operator/operator/ops/framework.py", line 351 in emit 2025-05-30T00:34:40.0989384Z File "/home/runner/work/operator/operator/ops/charm.py", line 1432 in _evaluate_status 2025-05-30T00:34:40.0990396Z File "/home/runner/work/operator/operator/ops/_main.py", line 405 in _emit 2025-05-30T00:34:40.0991303Z File "/home/runner/work/operator/operator/ops/_main.py", line 459 in run 2025-05-30T00:34:40.0992346Z File "/home/runner/work/operator/operator/testing/src/scenario/context.py", line 121 in run 2025-05-30T00:34:40.0993485Z File "/home/runner/work/operator/operator/testing/tests/helpers.py", line 58 in trigger 2025-05-30T00:34:40.0994803Z File "/home/runner/work/operator/operator/testing/tests/test_e2e/test_state.py", line 118 in test_status_setting 2025-05-30T00:34:40.0996537Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call 2025-05-30T00:34:40.0998158Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall 2025-05-30T00:34:40.0999724Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec 2025-05-30T00:34:40.1001281Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__ 2025-05-30T00:34:40.1002849Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/python.py", line 1792 in runtest 2025-05-30T00:34:40.1004433Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call 2025-05-30T00:34:40.1006300Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall 2025-05-30T00:34:40.1007899Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec 2025-05-30T00:34:40.1009392Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__ 2025-05-30T00:34:40.1010951Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 262 in <lambda> 2025-05-30T00:34:40.1012530Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 341 in from_call 2025-05-30T00:34:40.1014321Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 261 in call_runtest_hook 2025-05-30T00:34:40.1016168Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 222 in call_and_report 2025-05-30T00:34:40.1018049Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 133 in runtestprotocol 2025-05-30T00:34:40.1019805Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol 2025-05-30T00:34:40.1021411Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall 2025-05-30T00:34:40.1022944Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec 2025-05-30T00:34:40.1024443Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__ 2025-05-30T00:34:40.1026253Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/xdist/remote.py", line 195 in run_one_test 2025-05-30T00:34:40.1027834Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/xdist/remote.py", line 174 in pytest_runtestloop 2025-05-30T00:34:40.1029603Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall 2025-05-30T00:34:40.1031155Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec 2025-05-30T00:34:40.1032647Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__ 2025-05-30T00:34:40.1034098Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/main.py", line 325 in _main 2025-05-30T00:34:40.1035736Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/main.py", line 271 in wrap_session 2025-05-30T00:34:40.1037330Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main 2025-05-30T00:34:40.1038988Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_callers.py", line 121 in _multicall 2025-05-30T00:34:40.1040543Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_manager.py", line 120 in _hookexec 2025-05-30T00:34:40.1042109Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/pluggy/_hooks.py", line 512 in __call__ 2025-05-30T00:34:40.1043572Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/xdist/remote.py", line 393 in <module> 2025-05-30T00:34:40.1045389Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1291 in executetask 2025-05-30T00:34:40.1047023Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 341 in run 2025-05-30T00:34:40.1048665Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn 2025-05-30T00:34:40.1050508Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread 2025-05-30T00:34:40.1052307Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1273 in serve 2025-05-30T00:34:40.1053855Z File "/home/runner/work/operator/operator/.tox/unit/lib/python3.9/site-packages/execnet/gateway_base.py", line 1806 in serve 2025-05-30T00:34:40.1054927Z File "<string>", line 8 in <module> 2025-05-30T00:34:40.1075786Z File "<string>", line 1 in <module> 2025-05-30T00:34:40.1123245Z testing/tests/test_e2e/test_state.py::test_leader_get ``` We can hit this both in local env (python from astral) as well as github actions (ubuntu latest). We're tracking it here https://github.com/canonical/operator/issues/1771 More backtraces, etc., are attached there. Incidence: takes about one hour on a multi-core system running unit tests back to back. That's between once per 100K to once per 10K tests. An upstream ticket, https://github.com/python/cpython/issues/122529 looks very similar to my report. CPython no longer accepts bug fixes for 3.8 through 3.12 howerver, so if this is ever going to be fixed, that would be in the context of Ubuntu LTS. ** Affects: python3.8 (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2112506 Title: Segmentation fault while multithreading and multiprocessing (Python 3.8, 3.9; pytest-xdist; opentelemetry-sdk), intermittently reproducible To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python3.8/+bug/2112506/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
