Hi Matthew, its been a number of days since we heard anything regarding this
bug, from a stability perspective we're keen to have this resolved as soon as
possible, aside from the initial assessment neither Canonical nor AWS have
given us anymore progress reports or working theories or temporary mitigations
to help us overcome this issue. I've used a combination of my local knowledge
of the problem and an LLM to help me distill down the problem, below is an
initial analysis. I'm sure there will be gaps in theory/analysis but I hope
that at the very least it stimulates some dialogue/discussion with others in
Canonical to hopefully drive us to a resolution. Attached is the analysis.
# LP: #2165410 — systemd (PID 1) D-state during cpuset migration on
nohz_full CPUs
Root-cause analysis and proposed fix for the Ubuntu 26.04 (resolute)
`linux-aws` 7.0 kernel.
Prepared from the Launchpad bug, its attachments (reproducer, test logs) and the
`Ubuntu-aws-7.0.0-1013.13` source tree. All commit hashes below were verified
against
Linus' mainline tree (`git describe --contains` gives the first release tag).
Status of the proposal: **compile-tested and link-tested** with the real `aws`
amd64
config (`make vmlinux` succeeds, no new warnings); patches apply cleanly to
`Ubuntu-aws-7.0.0-1011.11`, `-1013.13`, and upstream `v7.0`/`v7.1`. **Not yet
runtime-tested** on a c8a.metal-48xl; the validation plan in section 7 covers
that.
---
## 1. Executive summary
The bug is real, and it has **two independent kernel-side causes**, both of
which
come down to the same thing: a 7.0 kernel with `isolcpus=nohz,<list>
nohz_full=<list>`
(no `domain` flag) still targets the nohz_full CPUs with deferred kernel work
that is
executed by `SCHED_NORMAL` kernel threads, and a `SCHED_FIFO` userspace busy
loop on
those CPUs never lets such a thread run. PID 1 ends up waiting for that work.
1. **The captured stacks (all three tested kernels) are not `flush_work` in
`__lru_add_drain_all()` at all** — they show the `cpuset_migrate_mm` kworker
blocked
in `synchronize_rcu_expedited()` inside `lru_cache_disable()`, with
`rcu_exp_gp_kthread_worker` in `kthread_flush_work()`. The per-`rcu_node`
expedited
grace-period workers (`rcu_exp_par_gp_kthread_worker/N`) are **pinned to the
nohz_full
CPUs** on this configuration, because 7.0's `kthread_fetch_affinity()`
filters a
kthread's preferred affinity only against `HK_TYPE_DOMAIN`, and with
`isolcpus=nohz`
the domain-housekeeping mask is *all* CPUs. Being `SCHED_NORMAL`
(`CONFIG_RCU_BOOST=n` so `kthread_prio=0`), they starve behind the FIFO
spinners,
and **every** `synchronize_rcu_expedited()` on the system stalls with them.
2. **The mechanism described in the bug (Matthew Ruffell's reproducer)** is
also real:
`__lru_add_drain_all()` queues `lru_add_drain_per_cpu` onto every CPU whose
per-CPU
folio batch is non-empty, then `flush_work()`s it; on a FIFO-hogged CPU that
per-CPU
kworker never runs. The reproducer keeps 1–14 folios in the `lru_add` batch
of every
isolated CPU at all times, so every `lru_cache_disable()` /
`lru_add_drain_all()`
caller in the system blocks.
Why this looks new in 26.04: three things changed between 6.8 and 7.0 for this
configuration. RT bandwidth throttling, which used to give non-RT tasks 50 ms
per second
regardless of FIFO hogs, is gone (v6.12; replaced by the deadline "fair
server", which
is evidently not rescuing these kworkers — see section 4 and the open
questions).
`cpu_is_isolated()` stopped considering nohz_full CPUs isolated (v7.0-rc1), so
every
isolation-aware user (buffer_head LRU, vmstat, memcg drain, blk-mq) now targets
them.
And kthread affinity management moved from "inherit kthreadd's `HK_TYPE_KTHREAD`
mask" to "filter with `HK_TYPE_DOMAIN` only" (v7.0-rc1). `CONFIG_PREEMPT_LAZY`
itself
is a red herring as far as we can tell (section 4).
Upstream explicitly made the assumption that "nohz_full implies domain
isolation"
(commit messages quoted in section 3). The kernel does not enforce that, the
customer's
command line violates it, and the fix proposed here makes the kernel honour
`nohz_full` on its own again, plus stops mm from ever queueing LRU drain work on
isolated CPUs. **Independently of any kernel patch, adding `domain` to
`isolcpus`
(`isolcpus=domain,nohz,18-95,108-178`) removes cause 1 on the stock kernel, and
is
what upstream considers the supported configuration** (section 8).
## 2. Evidence, reconciled
Source: the three test-log bundles attached to the bug (comment #2) and the
reproducer (comment #5). Copies are in `bug-attachments/`.
Captured on 7.0.0-1006, -1008 and -1011, twice per run, 6 s apart,
identical:
```
PID 1 (systemd) Ds wchan flush_workqueue
__flush_workqueue+0x14e/0x430
flush_migrate_mm_task_workfn+0x19/0x30
task_work_run+0x62/0x90
exit_to_user_mode_loop ...
kworker/u768:1+cpuset_migrate_mm D psr 3/7 (housekeeping CPU)
synchronize_rcu_expedited+0x1b2/0x220 <- wait_event(rnp->exp_wq[...]) in
tree_exp.h:976
lru_cache_disable+0x15/0x30 <- mm/swap.c:929
do_migrate_pages+0x4e/0x2b0
cpuset_migrate_mm_workfn+0x26/0x50
rcu_exp_gp_kthread_worker (PID 19) D psr 100/101 (housekeeping CPU), wchan
kthread_flush_work
```
`pgmigrate_success` advanced ~36 pages/s during the stall and in bursts of
~8200 pages/s
otherwise; PID 1 stayed in D for 60–62 s (1006/1011) and 16 s (1008). Matthew's
improved reproducer produces 2–5 minutes; production sees 5–10 minutes.
What the RCU stack means (7.0 `kernel/rcu/tree_exp.h`):
* `synchronize_rcu_expedited()` hands the grace period to
`rcu_exp_gp_kthread_worker`
and waits (`tree_exp.h:924-981`).
* That worker runs `sync_rcu_exp_select_cpus()` which, for every leaf
`rcu_node` except
the last, `kthread_queue_work(rnp->exp_kworker, ...)` and then
`kthread_flush_work(&rnp->rew.rew_work)` (`tree_exp.h:505-533`, `468-482`).
* `kthread_flush_work()` completes only after the
`rcu_exp_par_gp_kthread_worker/N`
thread has *run* `__sync_rcu_exp_select_node_cpus()` and the flush item
behind it
(`kernel/kthread.c:1346-1375`). The work itself is microseconds (an EQS
snapshot per
CPU plus an async IPI to CPUs that are in the kernel). A seconds-long wait
therefore
means **the par worker is not being scheduled**.
Where the par workers live (7.0):
* `rcu_spawn_exp_par_gp_kworker()` creates one `kthread_worker` per leaf
`rcu_node`,
leaves it `SCHED_NORMAL` unless `CONFIG_RCU_EXP_KTHREAD` (not set: needs
`RCU_BOOST`), and calls `rcu_thread_affine_rnp()`
(`kernel/rcu/tree.c:4179-4202`).
* `rcu_thread_affine_rnp()` prefers the CPUs of that `rcu_node`
(`kthread_affine_preferred()`, `tree.c:4161-4175`).
* `kthread_fetch_affinity()` intersects the preferred mask with
`housekeeping_cpumask(HK_TYPE_DOMAIN)` **only** (`kernel/kthread.c:330-348`).
With `isolcpus=nohz,18-95,108-178` no `domain` flag is given, so
`HK_FLAG_DOMAIN` is not
set (`kernel/sched/isolation.c:314-366`) and
`housekeeping_cpumask(HK_TYPE_DOMAIN)`
returns `cpu_possible_mask` (`isolation.c:64-75`). With `RCU_FANOUT_LEAF=16`
and 192
CPUs, the leaf nodes covering CPUs 32–95 and 112–175 consist **only** of
nohz_full CPUs,
so their expedited workers are pinned to nothing but FIFO-hogged CPUs. This is a
deterministic explanation of the captured `kthread_flush_work` wait, and it
affects
every `synchronize_rcu_expedited()` user on the box, not just mm.
The bug's original theory (`flush_work` in `__lru_add_drain_all()`) is the
second stage
of the same `lru_cache_disable()` call (`mm/swap.c:913-935`:
`synchronize_rcu_expedited()`
first, then `__lru_add_drain_all(true)`), so once the RCU wait is fixed the
drain wait
is next in line. Matthew's reproducer, whose worker keeps folios batched on
every
isolated CPU, exercises it directly. Both must be fixed.
A note on the 36 pages/s and 8200 pages/s pattern in the sampler logs: the
`cpuset_migrate_mm` workqueue is *ordered*, so the mms of the 32 stress-ng
workers are
migrated one after another, each paying one `lru_cache_disable()` (RCU + drain
wait).
The regular ~8200-page bursts are consistent with one ~32 MB mm being migrated
each time
a wait completes; the ~36 pages/s stretches are the long RCU waits.
## 3. Root cause in the 7.0 source
### 3.1 `cpu_is_isolated()` no longer covers nohz_full CPUs
`include/linux/sched/isolation.h` (7.0):
```c
static inline bool cpu_is_isolated(int cpu)
{
return !housekeeping_test_cpu(cpu, HK_TYPE_DOMAIN);
}
```
v6.8 had `!housekeeping_test_cpu(cpu, HK_TYPE_DOMAIN) ||
!housekeeping_test_cpu(cpu,
HK_TYPE_TICK) || cpuset_cpu_is_isolated(cpu)`. The change is
**1c3329df409b** "sched/isolation: Remove HK_TYPE_TICK test from
cpu_is_isolated()"
(Frederic Weisbecker, v7.0-rc1), whose message says:
> It doesn't make sense to use nohz_full without also isolating the related
> CPUs from
> the domain topology, either through the use of isolcpus= or cpuset isolated
> partitions. [...] Therefore if a CPU is not HK_TYPE_DOMAIN, it shouldn't be
> HK_TYPE_KERNEL_NOISE either. Testing the former is then enough.
On the customer's command line `cpu_is_isolated()` is therefore **false for
every CPU**
(`/sys/devices/system/cpu/isolated` will read empty while `nohz_full` reads
`18-95,108-178`). Users that silently lost isolation awareness versus 6.8:
| user | effect on nohz_full-only CPUs in 7.0 |
|---|---|
| `fs/buffer.c bh_lru_install()` (line 1335) | buffer_heads cached again →
`has_bh_in_lru()` true → `__lru_add_drain_all()` queues work there |
| `mm/vmstat.c vmstat_shepherd()` (line 2138) | `vmstat_update` queued there
again |
| `mm/memcontrol.c schedule_drain_work()` (line 2031) | memcg stock drain work
queued there |
| `block/blk-mq.c` (line 4279) | CPUs kept in `hctx->cpumask` |
### 3.2 Unbound kthreads may be affined to nohz_full CPUs
`kernel/kthread.c:330-348` (7.0) filters preferred/node affinity with
`HK_TYPE_DOMAIN`
only. Introduced by **041ee6f3727a** "kthread: Rely on HK_TYPE_DOMAIN for
preferred
affinity management" (Frederic Weisbecker, v7.0-rc1):
> Unbound kthreads want to run neither on nohz_full CPUs nor on domain isolated
> CPUs.
> And since nohz_full implies domain isolation, checking the latter is enough
> to verify
> both.
History: up to v6.13 `kthreadd` did `set_cpus_allowed_ptr(tsk,
housekeeping_cpumask(HK_TYPE_KTHREAD))` and children inherited it (`v6.8
kernel/kthread.c:743`, and `:370` in `kthread()`); v6.14's **d1a89197589c** /
**4d13f4304fa4** introduced `kthread_fetch_affinity()` filtering with
`HK_TYPE_KTHREAD`;
v7.0 switched the filter to `HK_TYPE_DOMAIN` (041ee6f3727a) and included
kthreadd itself
in the managed list (**92a734606e90**, **5564c12385b7**). `nohz_full=` sets
`HK_FLAG_KERNEL_NOISE` = `HK_TYPE_KTHREAD` precisely to keep kthreads away
(`isolation.c:304-311`), and 7.0 ignores it for placement.
Affected threads on this box: `rcu_exp_par_gp_kthread_worker/N` (the captured
stall),
`rcub/N` (RCU boost, only if `RCU_BOOST`), and any NUMA-node-affine kthread
(`kswapd1`, `kcompactd1`, ... on node 1 = CPUs 96–191, which includes 108–178).
### 3.3 `__lru_add_drain_all()` queues per-CPU work on isolated CPUs
`mm/swap.c:776-890` (7.0). `cpu_needs_drain(cpu)` is true if any of the six
`cpu_fbatches` batches, the `mlock_fbatch`, or the buffer_head LRU of that CPU
is
non-empty; work is then `queue_work_on(cpu, mm_percpu_wq, ...)`'d and
`flush_work()`'d.
`mm_percpu_wq` is a bound (per-CPU) workqueue; there is no housekeeping check
for bound
work and there cannot be one (the batches are protected by a `local_lock`, so
only the
owning CPU can drain them).
The batches are filled by `__folio_batch_add_and_move()` (`swap.c:182-203`) and
`mlock_folio()/mlock_new_folio()/munlock_folio()` (`mm/mlock.c:242-305`); both
already
bypass batching when `lru_cache_disabled()` is set — the same code path the fix
reuses.
`lru_cache_disable()` (`swap.c:913-935`) is what `do_migrate_pages()`
(cpuset.mems,
`mm/mempolicy.c:1343`), `do_mbind()` (`mempolicy.c:1531`), `move_pages()`
(`mm/migrate.c:2360`), `alloc_contig_range()` (`mm/page_alloc.c:6865`) and
memory
offlining (`mm/memory_hotplug.c:1967`) call. Plain `lru_add_drain_all()` is
called by
compaction, khugepaged, memfd, ksm, memory-failure, vmscan, fadvise,
drop_caches,
bdev invalidation, GUP long-term pinning and cgroup v1 memcg reclaim. All of
them are
exposed to the same hang.
### 3.4 Why PID 1 is the victim
`cpuset_write_resmask()` (`kernel/cgroup/cpuset.c:3217-3260`) →
`update_nodemask()` →
`cpuset_update_tasks_nodemask()` iterates **every thread** of the cpuset and
queues one
`cpuset_migrate_mm_workfn` per thread that has an mm onto the **global,
ordered**
`cpuset_migrate_mm_wq`; each work item runs `do_migrate_pages(mm, from, to,
MPOL_MF_MOVE_ALL)`. The writer then runs `flush_migrate_mm_task_workfn()` as
task_work
(`TWA_RESUME`) before returning to userspace and sleeps `TASK_UNINTERRUPTIBLE`
in
`flush_workqueue()` until *every* queued migration — including ones queued by
other
writers — has finished. Because the workqueue is ordered, one migration stuck in
`lru_cache_disable()` holds up all the others. On cgroup v2 memory migration is
unconditional (`CS_MEMORY_MIGRATE` forced at `cpuset_css_alloc()`); the only
short cut is
`nodes_equal()` when the nodemask does not change.
The writer is systemd itself (PID 1): `AllowedMemoryNodes=` is applied through
`cgroup_apply_cpuset()` → `cg_set_attribute()` at unit start, on `systemctl
set-property`,
and again at boot finish for `Startup*` settings. Since systemd v259 (Ubuntu
26.04 ships
259.5) `cg_set_attribute()` opens every cgroup v2 attribute file with
**`O_NONBLOCK`**,
explicitly to opt into asynchronous semantics wherever the kernel offers them —
the 7.0
cpuset code ignores that flag (optional patch 5 below honours it). While PID 1
is in D
state, logind's requests to PID 1 are not serviced, `pam_systemd` cannot create
sessions,
and SSH logins fail.
History (verified): e93ad19d0564 "cpuset: make mm migration asynchronous"
(v4.5-rc4,
introduced the workqueue plus flush "so operations still seem synchronous to
userland");
ee9707e8593d (v5.15-rc1, migration unconditional on v2); 9f72daf7edfa
(v5.15-rc1, skip
when nodemasks match, citing the `lru_cache_disable()` cost); c0fb16ef887d +
3514309e0322 (v6.18-rc1, flush moved to task_work to get it out from under
`cgroup_mutex`); **eda17a3a7084** "cgroup/cpuset: Rebind/migrate mm only for
threadgroup
leader" (v7.3-rc1) — a backport candidate on its own, since it removes the
per-*thread* multiplication of migration work items (32 stress-ng workers ×
threads in the comment-#2 test). Full details in `research/cpuset-systemd.md`.
### 3.5 Kernel configuration facts (`linux-aws` amd64, exported from the
annotations)
```
CONFIG_PREEMPT_LAZY=y CONFIG_PREEMPT_DYNAMIC=n (aws override, LP: #2051342)
CONFIG_PREEMPTION=y CONFIG_TREE_RCU=y (PREEMPT_RCU is *not* set with LAZY
alone)
CONFIG_RCU_BOOST=n CONFIG_RCU_EXP_KTHREAD=n -> RCU kthreads are
SCHED_NORMAL
CONFIG_NO_HZ_FULL=y CONFIG_RCU_NOCB_CPU=y CONFIG_HZ=1000
CONFIG_RT_GROUP_SCHED=n -> no RT bandwidth throttling at all in 7.0
CONFIG_CPU_ISOLATION=y CONFIG_CSD_LOCK_WAIT_DEBUG=n
CONFIG_RCU_EXP_CPU_STALL_TIMEOUT=0
```
## 4. Why 24.04's 6.8 kernel behaved differently
Verified in the v6.8 sources:
* **RT throttling existed** (`kernel/sched/rt.c:25 sysctl_sched_rt_runtime =
950000`):
a FIFO spinner was throttled for 50 ms every second, so a starved per-CPU
kworker ran
within ≤1 s and any single `lru_cache_disable()` cost at most a second or
two. v6.12
removed root RT bandwidth control in favour of the deadline fair server. On
this box
the fair server is evidently not giving `SCHED_NORMAL` tasks their 50 ms/s
(stress-ng
workers confined to the isolated CPUs got ~1% CPU in total; kworkers waited
minutes) —
whether that is expected behaviour of the deferred server or a separate
scheduler bug
is analysed in `research/scheduler-fair-server.md` (see open questions).
* **`cpu_is_isolated()` included `HK_TYPE_TICK`** (`v6.8
include/linux/sched/isolation.h:68-73`),
so bh_lru/vmstat/blk-mq treated nohz_full CPUs as isolated.
* **kthreads inherited `HK_TYPE_KTHREAD` housekeeping affinity from kthreadd**
(`v6.8 kernel/kthread.c:743`), so no unbound kthread lived on nohz_full CPUs.
* The 6.8 expedited GP (without `RCU_EXP_KTHREAD`) queued its per-node work on
the
first online CPU of each `rcu_node` via the per-CPU `rcu_par_gp_wq`
(`v6.8 tree_exp.h:486-497`), which *could* also be an isolated CPU — but RT
throttling bounded that wait to ~1 s as well.
`CONFIG_PREEMPT_LAZY` changes when a *fair* task may preempt another fair task;
it does
not let a `SCHED_NORMAL` kworker preempt a `SCHED_FIFO` task and does not affect
cross-class preemption by the deadline server. Nothing in the two mechanisms
above
depends on it. The aws flavour also builds with `PREEMPT_LAZY` but *without*
`PREEMPT_RCU`, so blocked-reader effects of preemptible RCU are not in play
either.
## 5. Upstream prior art (verified)
| commit | subject | first tag | relevance |
|---|---|---|---|
| ff042f4a9b05 | mm: lru_cache_disable: replace work queue synchronization with
synchronize_rcu (M. Tosatti) | v5.18-rc1 | removed the *first* "queue work on
every CPU" step from `lru_cache_disable()` for isolated-CPU reasons |
| 31733463372e | mm: lru_cache_disable: use synchronize_rcu_expedited (M.
Tosatti) | v5.19-rc4 | the expedited GP that stalls here |
| 8a237adf213d | fs/buffer.c: disable per-CPU buffer_head cache for isolated
CPUs (M. Tosatti) | v6.6-rc1 | direct template for patch 3 (`if (...
cpu_is_isolated(smp_processor_id())) return;`) |
| d1a89197589c / 4d13f4304fa4 | kthread: Default affine kthread to its
preferred NUMA node / Implement preferred affinity (F. Weisbecker) | v6.14-rc1
| introduced `kthread_fetch_affinity()` (then filtered with `HK_TYPE_KTHREAD`) |
| 6010d245ddc9 / 1174b9344bc7 | sched/isolation: Consolidate housekeeping
cpumasks / Make "isolcpus=nohz" equivalent to "nohz_full" (W. Long) | v6.14-rc1
| `HK_TYPE_KERNEL_NOISE`; explains why `isolcpus=nohz` behaves as `nohz_full=` |
| 03ff73510169 / 6440966067dc | cpuset: Update HK_TYPE_DOMAIN cpumask from
cpuset / Remove cpuset_cpu_is_isolated() (F. Weisbecker) | v7.0-rc1 | cpuset
isolated partitions folded into `HK_TYPE_DOMAIN` |
| 1c3329df409b | sched/isolation: Remove HK_TYPE_TICK test from
cpu_is_isolated() (F. Weisbecker) | v7.0-rc1 | **regression 1** (patch 1
`Fixes:`) |
| 5564c12385b7 / 92a734606e90 / 041ee6f3727a | kthread: managed affinity list /
kthreadd included / Rely on HK_TYPE_DOMAIN (F. Weisbecker) | v7.0-rc1 |
**regression 2** (patch 2 `Fixes:` 041ee6f3727a) |
Also relevant in 7.0 (same design direction as patch 3): `mm/memcontrol.c
schedule_drain_work()` refuses to queue memcg stock drains on isolated CPUs, and
`housekeeping_update()` flushes `mm_percpu_wq` and `memcg_wq` when a cpuset
isolated
partition changes (`kernel/sched/isolation.c:121-163`). The fuller prior-art
list with
lore references is in `research/prior-art.md` (if present).
Post-7.0 mainline that matters for forward-porting: `mm/swap.c` was renamed to
`mm/folio.c` for 7.3 (0ba14428abc2), `cpu_needs_drain()` got a data-race
annotation
(f078b0a0727c) and LRU drain tracepoints were added (094470f3f45b). Nothing in
7.1/7.2/7.3-rc2 changes `kthread_fetch_affinity()` or `cpu_is_isolated()`, so
the two
regressions are still present upstream.
## 6. The proposed fix (`patches/`)
Based on `Ubuntu-aws-7.0.0-1013.13`; all five patches also apply to `-1011.11`
and to
upstream `v7.0` and `v7.1` (`git apply --check`); patches 1, 2, 4 and 5 also
apply to
`v7.2`. Patch 3 needs a trivial include-context refresh for `v7.2`, and a
rename to
`mm/folio.c` for 7.3+. Patches 1–3 are the fix; 4 and 5 are optional hardening
and
mitigation that Canonical can take or drop independently.
### 0001 sched/isolation: Consider nohz_full CPUs isolated in cpu_is_isolated()
again
* `include/linux/sched/isolation.h`: test `HK_TYPE_KERNEL_NOISE` in addition to
`HK_TYPE_DOMAIN`, through the inline `housekeeping_cpu()` wrapper. Partial
revert of
1c3329df409b, restoring 6.x semantics.
* Effect: bh_lru, vmstat shepherd, memcg drain, blk-mq (and patch 3) treat
nohz_full
CPUs as isolated again. No change where nohz_full CPUs are also domain
isolated.
* Risk: low. On systems without any CPU isolation the whole test is a
patched-out
static branch (previously one out-of-line call); on isolated systems it is
two cpumask
tests instead of one.
### 0002 kthread: Keep unbound kthreads off nohz_full CPUs
* `kernel/kthread.c kthread_fetch_affinity()`: intersect with
`housekeeping_cpumask(HK_TYPE_KTHREAD)` too; fall back to the CPUs
housekeeping for
both types (never empty: `housekeeping_setup()` and cpuset's
`isolated_cpus_can_update()` guarantee one), and finally to the domain mask.
* Effect: `rcu_exp_par_gp_kthread_worker/N`, `rcub/N`, node-affine kthreads
leave the
nohz_full CPUs → the observed `synchronize_rcu_expedited()` stall disappears.
* Risk: low; restores the 6.14–6.19 behaviour. Alternative, narrower form if
preferred:
filter with `housekeeping_cpumask(HK_TYPE_RCU)` inside
`rcu_thread_affine_rnp()` only
(what 6.10–6.13 `rcutree_affinity_setting()` did) — but that leaves other
kthreads on
the nohz_full CPUs.
### 0003 mm/swap, mm/mlock: Don't batch folios in per-CPU LRU caches on
isolated CPUs
* New helper `lru_batching_disabled()` in `mm/internal.h`:
`lru_cache_disabled() || cpu_is_isolated(smp_processor_id())`, used at the
four batch
sites (`__folio_batch_add_and_move()`, `mlock_folio()`, `mlock_new_folio()`,
`munlock_folio()`). On an isolated CPU the folio goes to the LRU immediately
(exactly
the existing `lru_cache_disabled()` path), so `cpu_needs_drain()` is always
false for
isolated CPUs and `__lru_add_drain_all()` never queues work there.
* This is the "avoid batching on isolated CPUs, like the buffer_head cache"
suggestion
from comment #7, made complete: it covers all six `cpu_fbatches` batches and
the mlock
batch, and — together with patch 1 — the buffer_head term of
`cpu_needs_drain()`.
* Cost: one lruvec-lock round trip per folio instead of per 15 folios, on
isolated CPUs
only. On systems without isolation `cpu_is_isolated()` is a patched-out
static branch.
* Not chosen: draining isolated CPUs by IPI (comment #6 experiment). The
batches are
`local_lock`-protected (preempt-disable, not irq-disable, for five of the
six), so an
IPI handler racing with the owning CPU is unsafe without a locking redesign,
and it
did not help in Matthew's test anyway; skipping isolated CPUs in
`__lru_add_drain_all()`
would leave folios unmigratable and break `lru_cache_disable()`'s contract.
### 0004 sched/isolation: Drain per-CPU LRU batches when CPUs become isolated
(optional hardening)
* `housekeeping_update()`: call `lru_add_drain_all()` once after the domain
mask update,
next to the existing vmstat/memcg workqueue flushes, so CPUs that become
isolated at
runtime via a cpuset isolated partition do not keep pre-isolation folios in
their
batches. Boot-time isolation (this bug) does not need it; Canonical may drop
it.
### 0005 cgroup/cpuset: Don't wait for memory migration when cpuset.mems is
written with O_NONBLOCK (optional mitigation)
* `cpuset_write_resmask()`: skip `schedule_flush_migrate_mm()` when the writer
opened the
file with `O_NONBLOCK`; the migration still runs to completion on
`cpuset_migrate_mm_wq`, the writer just does not wait for it. Mirrors memcg's
`memory.max`/`memory.high` behaviour (c8e6002bd611, v6.16-rc1). Blocking
writers and
the `cgroup.procs` attach path are unchanged.
* Effect: systemd ≥ 259 already opens cgroup attributes with `O_NONBLOCK`, so
PID 1 stops
going into D state on `AllowedMemoryNodes=` writes at all, whatever else
stalls the
migration. It protects init (and SSH) against any *future* migration stall,
but it does
not make the migration itself faster — patches 1–3 do that.
* Risk: uAPI semantics change for non-blocking writers only ("write returned
but memory
may still be moving"); needs the cgroup maintainers' agreement (Tejun Heo,
Waiman Long,
Chuyi Zhou). Applies cleanly to v7.0, v7.1 and v7.2. Note also that upstream
v7.3-rc1's
eda17a3a7084 (migrate once per thread group instead of once per thread) is a
cheap,
independent backport that shrinks the amount of queued migration work.
Build verification performed here (x86-64, gcc 13.3, aws amd64 config exported
from
`debian.aws/config/annotations`, BTF/signing keys disabled for the local build):
`make mm/ kernel/sched/ kernel/rcu/ fs/buffer.o block/blk-mq.o
kernel/kthread.o` clean;
`make vmlinux` clean (only the three pre-existing apparmor/maple_tree warnings).
`checkpatch.pl --strict`: 0 errors; the only warnings are the Ubuntu-style
`BugLink:`
trailer, the `Co-Authored-By` casing and "unknown commit id" (the shallow tree
lacks
upstream history). The patches carry no `Signed-off-by` yet — add yours (and
DCO) before
sending.
## 7. Validation plan (c8a.metal-48xl)
Build: apply `patches/000[1-4]*.patch` on the resolute `linux-aws` tree (or
`master-next`), build the aws flavour as usual, install on a c8a.metal-48xl
with the
customer's command line (`isolcpus=nohz,18-95,108-178 nohz_full=18-95,108-178
rcu_nocbs=18-95,108-178 rcu_nocb_poll idle=poll ...`). Keep an unpatched
`7.0.0-1011`/`-1013` to compare.
Static A/B (30 s, no workload), with `validate.sh snapshot`:
| check | unpatched | patched |
|---|---|---|
| `/sys/devices/system/cpu/isolated` | empty | empty (unchanged; DOMAIN mask is
what it is) |
| `Cpus_allowed_list` of `rcu_exp_par_gp_kthread_worker/*`, `rcub/*`,
`kswapd*`, `kcompactd*` | inside 18-95 / 108-178 for several | never intersects
18-95,108-178 |
Dynamic A/B with Matthew's reproducer
(`bug-attachments/reproducer/run_repro.sh`) and
the comment-#2 baseline (`repro-baseline-v3.sh`), while `validate.sh watch 900`
and
`validate.sh trace 120` run:
| check | unpatched | patched |
|---|---|---|
| PID 1 D-state duration (`run_repro.sh` prints it) | 2–5 min (1006/1011: ~60 s
baseline) | only the migration itself: seconds for 50 GB at ~8 K–40 K pages/s,
no plateau |
| `trace`: `workqueue_queue_work function=lru_add_drain_per_cpu req_cpu=` |
many `req_cpu` in 18-95,108-178 | none in 18-95,108-178 |
| `watch`: `cpuset_migrate_mm` kworker stack | `synchronize_rcu_expedited`
(then `flush_work`) for tens of seconds | only
`migrate_pages`/`folio_lock`-style frames, progressing |
| `watch`: `rcu_exp_gp_kthread_worker` | D in `kthread_flush_work` | never
blocked for >1 tick |
| `sampler.log` pgmigrate | 36 pages/s plateaus | steady progress |
| `ssh` login during the run | fails/hangs | works |
Bisecting the two causes on the unpatched kernel (each confirms one
half):
* boot with `isolcpus=domain,nohz,18-95,108-178` (everything else unchanged) →
the RCU
wait disappears (`/sys/devices/system/cpu/isolated` now lists the CPUs, the
par workers
move to housekeeping CPUs, `cpu_is_isolated()` is true). The `flush_work`
wait remains
reproducible with `run_repro.sh` because per-CPU kworkers stay per-CPU; that
isolates
patch 3's contribution.
* boot with `rcupdate.rcu_normal=1` (0444 module param, boot only) → expedited
GPs become
normal GPs, which need no per-node kworker; the RCU half disappears without
patch 2.
* run the FIFO workers as `SCHED_OTHER` (drop `chrt -f 80`) → both halves
disappear on
any kernel; confirms the FIFO starvation dependency.
Regression checks for the patched kernel: `stress-ng --vm` / page-fault
microbenchmark
pinned to an isolated CPU vs housekeeping CPU (expect the isolated CPU to pay a
small
per-fault cost, housekeeping unchanged); `mlock`/`munlock` of a large region on
an
isolated CPU; cpuset isolated-partition create/destroy under load (patch 4);
NUMA
`migratepages`/`mbind` throughput on housekeeping CPUs (unchanged); RCU
expedited GP
latency (`rcutorture` or `perf stat -e rcu:rcu_exp_grace_period` — should
improve).
## 8. Mitigations available without a new kernel
1. **`isolcpus=domain,nohz,18-95,108-178`** (or an equivalent cpuset isolated
partition)
instead of `isolcpus=nohz,...`. This is the configuration upstream assumes;
it moves the
RCU expedited workers and all unbound kthreads/workqueues off the isolated
CPUs and
re-enables the buffer_head/vmstat/memcg isolation logic. It does **not** fix
the
`__lru_add_drain_all()` per-CPU drain wait (patch 3). Side effect: the
isolated CPUs
leave the load balancer; pinned threads are unaffected.
2. **Do not change `cpuset.mems` (AllowedMemoryNodes=) of a slice while it
contains
large-RSS processes**; set it before tasks join, or use
`numactl`/`set_mempolicy` in
the application. This avoids the synchronous migration behind PID 1 entirely.
3. **`rcupdate.rcu_normal=1`** on the command line removes the expedited-GP
half at the
cost of slower `synchronize_rcu_expedited()` system-wide (ms instead of µs).
RT users
commonly run this way (`PREEMPT_RT` defaults `rcu_normal_after_boot=1`).
4. Lower the spinners below `SCHED_FIFO`, or make them yield periodically, so
per-CPU
kworkers can run (comment #7 suggestion 1). Only if the workload allows it.
## 9. Open questions and caveats
* **Runtime confirmation is outstanding.** Everything above is derived from the
captured
stacks and the 7.0 sources plus compile/link testing; the validation plan in
section 7
is the acceptance test. In particular the stacks of the
`rcu_exp_par_gp_kthread_worker/N`
threads were never captured; `validate.sh watch` does that and should show
them
runnable-but-not-running on isolated CPUs on the unpatched kernel.
* **Fair server behaviour.** With RT throttling gone, the deadline fair server
(50 ms /
1 s, deferred) is supposed to keep `SCHED_NORMAL` tasks alive under a FIFO
hog. The
observed minutes-long starvation says it did not on this box. If that is a
scheduler
bug (rather than intended deferred-server semantics on nohz_full CPUs), it is
a third,
independent issue worth its own report; the patches here do not depend on its
outcome.
See `research/scheduler-fair-server.md`.
* **Upstream acceptance.** Patches 1 and 2 argue against an explicit upstream
design
assumption ("nohz_full implies domain isolation"). They restore pre-7.0
behaviour for a
configuration the kernel accepts and documents, so they are defensible, but
upstream
may instead answer "use `isolcpus=domain`" and/or make the kernel warn or
imply domain
isolation. Patch 3 follows established precedent (8a237adf213d, memcg
`schedule_drain_work()`) and is the piece with the clearest upstream path; it
should be
posted to linux-mm with Marcelo Tosatti, Frederic Weisbecker, Waiman Long and
Minchan Kim
on Cc. For an Ubuntu SRU all four can go in as-is with `BugLink`.
* Patch 3 changes behaviour for every CPU reported isolated, including cpuset
isolated
partitions on systems that never set nohz_full; that is intended (those CPUs
also cannot
be drained remotely) but should be called out in the SRU justification.
ANALYSIS.md this document
patches/0001-*.patch .. 0004-* the series (git format-patch, base
Ubuntu-aws-7.0.0-1013.13)
validate.sh A/B evidence collector for the c8a.metal-48xl
(snapshot | trace | watch)
bug-attachments/ reproducer sources, comment-#2 test logs,
baseline script, all LP comments
research/ supporting research reports (prior art,
scheduler, cpuset/systemd), when complete
** Attachment added: "analysis-files.tar.gz"
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/2165410/+attachment/5999653/+files/analysis-files.tar.gz
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2165410
Title:
Ubuntu LTS 26.04 linux-aws: systemd enters D state and blocks SSH
during cpuset migration on nohz_full CPUs
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/2165410/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
