Public bug reported:
[Impact]
On a default Ubuntu 26.04 desktop install (Wayland session), any process that
performs hwloc topology discovery with I/O devices enabled blocks FOREVER in
connect() to an X display socket, inside hwloc's GL plugin
(/usr/lib/x86_64-linux-gnu/hwloc/hwloc_gl.so). There is no timeout, no error
message, and in the shipped hwloc version no runtime knob dedicated to this
plugin — the process simply hangs.
The simplest demonstration is that plain "lstopo", with no arguments, never
returns.
The most prominent victim is Open MPI. Every MPI program launched directly
("./app", i.e. singleton init, which the MPI standard supports and which
worked on every previous Ubuntu release) hangs inside MPI_Init /
MPI_Init_thread before reaching user code. Programs launched via
"mpirun -n N app" are NOT affected, because the MPI processes then receive
their topology pre-computed from the PRRTE daemon via PMIx instead of running
hwloc discovery themselves. This asymmetry hides the bug from most testing
(anything exercised under mpirun passes) while breaking direct invocation,
scripted single-process runs, test suites, and mpi4py-style usage.
Aggravating factors:
* The hang occurs on a fully working NVIDIA setup with the proprietary
driver installed — this is not a missing-hardware or missing-driver
situation. The GL plugin's sole purpose is NVIDIA NV-CONTROL display
locality, and it blocks even where that hardware is present and healthy.
* libhwloc-plugins is installed on any system with Open MPI, because
libopenmpi40 and libpmix2t64 both hard-Depend on it (see the secondary
issue at the bottom). It is therefore active in every MPI process.
* This is a regression relative to Ubuntu 24.04 LTS, on a brand-new LTS with
years of support ahead, and it is present on FRESH installs — in no way an
artefact of upgrading. Confirmed on two distinct hardware configurations,
both clean 26.04 installs (see Reproducibility).
* DISPLAY is irrelevant: the plugin synthesises its own display names, so
users cannot dodge this by unsetting DISPLAY (verified below).
* The failure mode is a silent hang with no diagnostic output; identifying
the cause required attaching gdb to the stuck process.
Upstream hwloc has already acknowledged and fixed exactly this problem: as of
hwloc 2.14, the GL component is disabled by default and must be explicitly
opted into with HWLOC_GL=1, because (per the upstream documentation) the code
"may hang when trying to connect to the X server". Resolute ships 2.13.0-2 —
one release short of the fix, so it has the hang without the remedy.
Requesting an SRU that backports the upstream default-off behaviour (a small,
self-contained change) or updates to a 2.14+ point release.
[Test Plan]
All on a default Ubuntu 26.04 desktop install with an active graphical
(Wayland) session.
1. Without MPI (minimal reproducer). Note that lstopo lives in the "hwloc"
package, which is not part of a default install:
sudo apt install hwloc
timeout 15 lstopo > /dev/null; echo $?
Affected system: prints "124" (killed by the timeout; without it, lstopo
never returns — observed for many minutes, and indefinitely in the MPI
case).
Fixed package: prints "0" and returns essentially instantly.
Plain "lstopo" with no arguments is sufficient: lstopo enables I/O device
discovery by default, so no --whole-io or other flag is needed. Verified
on the affected system that "lstopo" returns 124 while
"HWLOC_COMPONENTS=-gl lstopo" returns 0 immediately, in the same shell.
Optional cross-check, to rule out lstopo's own graphical X11 support
rather than the GL plugin: repeat with the non-X build of the tools from
hwloc-nox. Note hwloc-nox and hwloc conflict, so this replaces hwloc;
confirm the binary name with "dpkg -L hwloc-nox":
sudo apt install hwloc-nox
timeout 15 lstopo-no-graphics > /dev/null; echo $?
The attached backtrace already settles the point — the blocking call sits
inside hwloc_gl.so, reached from hwloc_topology_load, with no lstopo frame
anywhere in the stack, and the MPI reproducer involves no lstopo at all.
2. With MPI (the real-world symptom). hello.c:
#include <mpi.h>
int main(void) {
int tl;
MPI_Init_thread(NULL, NULL, MPI_THREAD_MULTIPLE, &tl);
MPI_Finalize();
return 0;
}
mpicc hello.c -o hello
./hello # affected: hangs forever; fixed: exits 0
mpirun -n 1 ./hello # works before and after (control case)
3. Opt-in still functional after the fix (upstream 2.14 semantics):
HWLOC_GL=1 lstopo > /dev/null
should re-enable the GL probe, and may therefore hang again on affected
configurations — that is the expected, documented opt-in behaviour.
[Where problems could occur]
The backport changes the default component set: topologies will no longer
contain GL OS device objects (names like ":0.0") unless HWLOC_GL=1 is set.
Only software using the hwloc/gl.h interoperability API — NVIDIA
display-to-GPU locality, e.g. some visualisation schedulers — would notice,
and the environment variable restores the previous behaviour. CUDA, NVML,
OpenCL and PCI discovery are unaffected, being separate components. The
resulting default matches upstream hwloc >= 2.14, so this does not diverge
from where later Ubuntu releases will land anyway. Risk is low and confined
to a feature that is recoverable by opt-in.
[Other Info]
* HWLOC_GL=0 is silently accepted and IGNORED by the shipped version: the
variable only acquires meaning in 2.14, where it is the opt-IN. The
documented upstream mitigation therefore does not exist in resolute.
Verified: "HWLOC_GL=0 lstopo" still hangs.
* The only working runtime mitigation on the shipped version is the generic
component blacklist, HWLOC_COMPONENTS=-gl (verified; see Workarounds).
That mechanism has existed since hwloc 1.6.0, so it is dependable, but it
is undiscoverable for a user facing a silent hang.
* DISPLAY makes no difference. With DISPLAY=:0 exported by the session, and
again with DISPLAY unset entirely, lstopo hangs identically (both return
124 under a 15 s timeout). This is consistent with the GL backend
synthesising candidate display names of the form ":port.device" itself
rather than honouring DISPLAY.
* Possibly relevant to why connect() blocks rather than failing fast: this
system has two X socket files but only one X server process. Xwayland is
running on :0, yet /tmp/.X11-unix/ contains both X0 and X1 (see
Environment). A connect() to a Unix socket whose server is absent normally
fails immediately with ECONNREFUSED, so an indefinite block suggests
either a listener that never accepts or a socket left behind by a departed
server. Offered as an observation only — the hwloc-side defect stands
regardless, which is precisely the reasoning upstream applied when they
disabled the component by default.
* Related upstream issue (same backend, same XOpenDisplay path, milder
symptom): https://github.com/open-mpi/hwloc/issues/483
Upstream default-off change: first appears in include/hwloc/gl.h as of
hwloc v2.14 ("This code may hang when trying to connect to the X server.
Hence the GL component is disabled unless HWLOC_GL=1 in the environment.")
The same wording is in the current upstream "Components and plugins"
documentation.
* Ubuntu has defaulted to a Wayland session since 25.10, which is the likely
reason this surfaced now: the GL plugin has always probed X during topology
load, but under Xorg sessions the probe completed or failed quickly. A
hardware-discovery library that blocks indefinitely on X inside its init
path, with no timeout, is the defect either way.
--------------------------------------------------------------------------------
Steps to reproduce
--------------------------------------------------------------------------------
1. Install Ubuntu 26.04 and log into the default desktop session. A fresh
install is sufficient — this is NOT specific to upgraded systems.
2. Build any Open MPI application against the distro Open MPI.
3. Run it directly: ./App -> hangs at MPI_Init_thread(...)
Run it as: mpirun -n 1 ./App -> runs normally.
4. sudo apt install hwloc, then run: lstopo
-> also hangs (no MPI involved).
Expected: singleton launch initialises just as "mpirun -n 1" does, as it did
on 24.04 and every earlier release; lstopo returns promptly.
Actual: indefinite hang, no output, no error.
Reproducibility: 100% reproducible, with the distro packages alone, on both
hardware configurations tested. Every system was a FRESH/CLEAN Ubuntu 26.04
install — no upgrade path, no configuration carried over from a previous
release, no self-built or third-party hwloc/MPI involved. The two
configurations are:
A. AMD Ryzen 9 9950X + NVIDIA GeForce RTX 5060 Ti (GB206)
B. AMD Ryzen 9 7950X + NVIDIA GeForce RTX 4060 Ti
Both have an NVIDIA discrete GPU alongside AMD integrated graphics. Systems
with no NVIDIA GPU at all were not tested; based on the code path the probe
should run there too, but this report does not claim it.
--------------------------------------------------------------------------------
Backtrace of the hung process (relevant thread; full dump attached)
--------------------------------------------------------------------------------
Thread 1 (main thread), obtained with
"gdb -p <pid> -batch -ex 'thread apply all bt'":
#3 __libc_connect (...) <- blocked here
#4 xcb_connect_to_display_with_auth_info () libxcb.so.1
#5 _XConnectXCB () libX11.so.6
#6 XOpenDisplay () libX11.so.6
#7 ?? () hwloc/hwloc_gl.so
#8 ?? () libhwloc.so.15
#9 hwloc_topology_load () libhwloc.so.15
#10 opal_hwloc_base_get_topology () libopen-pal.so.80
#11-#15 mca_base_framework_open (...) libopen-pal.so.80
#16 ?? () libmpi.so.40
#17 ompi_mpi_instance_init () libmpi.so.40
#18 ompi_mpi_init () libmpi.so.40
#19 PMPI_Init_thread () libmpi.so.40
#20 <application>
All other threads are idle — the PMIx progress thread in epoll_wait, an
OpenBLAS worker pool in pthread_cond_wait — i.e. PMIx and runtime bring-up
completed normally. The hang is exclusively the GL plugin's X connect.
--------------------------------------------------------------------------------
Root cause
--------------------------------------------------------------------------------
hwloc's GL backend (topology-gl.c, shipped as hwloc_gl.so in
libhwloc-plugins) queries the NVIDIA NV-CONTROL X extension to annotate GPU
display locality. During hwloc_topology_load with I/O discovery enabled —
which Open MPI's opal_hwloc_base_get_topology performs, and which lstopo
enables by default — it calls XOpenDisplay on candidate display names it
constructs itself. On this system that connect() never completes. In hwloc
2.13 the plugin runs unconditionally: no dedicated runtime disable, no
timeout, no diagnostic.
--------------------------------------------------------------------------------
Workarounds for affected users
--------------------------------------------------------------------------------
1. Per-process / per-shell (both verified working):
HWLOC_COMPONENTS=-gl ./App
HWLOC_COMPONENTS=-gl lstopo
2. NOT workarounds on this version: HWLOC_GL=0 (silently ignored), unsetting
DISPLAY (no effect), and removing libhwloc-plugins (would remove the MPI
stack — see below).
--------------------------------------------------------------------------------
SECONDARY ISSUE: the X-linked plugin cannot be declined without removing MPI
--------------------------------------------------------------------------------
hwloc's own packaging is correct here — libhwloc15 only Recommends the plugin
package. The hard dependency comes from the MPI side:
$ dpkg -S /usr/lib/x86_64-linux-gnu/hwloc/hwloc_gl.so
libhwloc-plugins:amd64: /usr/lib/x86_64-linux-gnu/hwloc/hwloc_gl.so
$ apt-cache show libhwloc15 libpmix2t64 libopenmpi40 \
| grep -E '^(Package|Version|Depends|Recommends)'
Package: libhwloc15
Version: 2.13.0-2
Recommends: libhwloc-plugins
Depends: libc6 (>= 2.38), libudev1 (>= 183)
Package: libpmix2t64
Version: 6.0.0+really5.0.9-3build1
Depends: libc6 (>= 2.38), libevent-core-2.1-7t64 (>= 2.1.8-stable),
libevent-pthreads-2.1-7t64 (>= 2.1.8-stable), libhwloc15 (>= 2.12.2),
libmunge2 (>= 0.5.8), zlib1g (>= 1:1.2.0), libhwloc-plugins
Package: libopenmpi40
Version: 5.0.10-1
Depends: libc6 (>= 2.38), libevent-core-2.1-7t64 (>= 2.1.8-stable),
libevent-pthreads-2.1-7t64 (>= 2.1.8-stable), libfabric1 (>= 2.0.0),
libgcc-s1 (>= 3.0), libhwloc15 (>= 2.13.0), libnl-3-200 (>= 3.11.0),
libpmix2t64 (>= 6.0.0+really5.0.9), libpsm2-2 (>= 10.3-37),
libucc1 (>= 1.7.0~rc1), libucx0, libhwloc-plugins
Consequences:
* "apt remove libhwloc-plugins" would remove libopenmpi40 and libpmix2t64
with it, i.e. the entire MPI stack. There is no supported way to decline
an X11-connecting plugin that is dlopen'd into every MPI process, and no
way to do so through apt at all.
* The plugin is installed and active on headless servers and HPC compute
nodes as readily as on desktops. An hwloc-nox package exists for the
command-line utilities, but there is no equivalent for the plugin package.
Note that those Depends are presumably not gratuitous: libhwloc-plugins
bundles the GL plugin together with plugins that MPI does legitimately want
for locality (PCI/CUDA/NVML-class discovery). That is precisely why the right
remedy is to separate the X-dependent plugin rather than to weaken the MPI
dependencies:
a) Split hwloc_gl.so into its own binary package — there is precedent in
libhwloc-contrib-plugins — so that it can be omitted without touching
PCI discovery or the MPI stack; or
b) failing that, stop building the gl plugin for Ubuntu, since its only
function is NVIDIA NV-CONTROL display locality, which is of marginal
value on the systems that install it by default.
Happy to split this section into a separate bug against openmpi/pmix if the
maintainers prefer.
--------------------------------------------------------------------------------
Environment
--------------------------------------------------------------------------------
$ lsb_release -rd
Description: Ubuntu 26.04 LTS
Release: 26.04
$ lstopo --version
lstopo 2.13.0
$ apt-cache policy hwloc libhwloc15 libhwloc-plugins openmpi-bin libopenmpi40
libpmix2t64
hwloc: Installed: 2.13.0-2 (resolute/universe)
libhwloc15: Installed: 2.13.0-2 (resolute/universe)
libhwloc-plugins: Installed: 2.13.0-2 (resolute/universe)
openmpi-bin: Installed: 5.0.10-1 (resolute/universe)
libopenmpi40: Installed: 5.0.10-1 (resolute/universe)
libpmix2t64: Installed: 6.0.0+really5.0.9-3build1 (resolute/universe)
All from the release pocket; no updates, no PPAs, no local builds.
hwloc 2.13.0 is one upstream release short of 2.14, where the GL component
was disabled by default.
$ dpkg -S /usr/lib/x86_64-linux-gnu/hwloc/hwloc_gl.so
libhwloc-plugins:amd64: /usr/lib/x86_64-linux-gnu/hwloc/hwloc_gl.so
$ echo $XDG_SESSION_TYPE
wayland
$ echo $DISPLAY
:0
$ pgrep -a Xwayland
10464 /usr/bin/Xwayland :0 -rootless -noreset -accessx -core -auth
/run/user/1000/.mutter-Xwaylandauth.IXZNT3 -listenfd 4 -listenfd 5
-displayfd 6 -initfd 7 -byteswappedclients -enable-ei-portal
$ pgrep -a Xorg
(no output — no Xorg server running)
$ ls -l /tmp/.X11-unix/
total 0
srwxrwxr-x 1 roy roy 0 Aug 4 08:43 X0
srwxrwxr-x 1 roy roy 0 Aug 4 08:43 X1
(note: two socket files, but only one X server process, on :0 — see the
observation in [Other Info])
$ lspci | grep -Ei 'vga|3d|display'
01:00.0 VGA compatible controller: NVIDIA Corporation GB206 [GeForce RTX 5060
Ti] (rev a1)
74:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
Granite Ridge [Radeon Graphics] (rev c1)
$ lscpu | grep -E 'Model name|^CPU\(s\)|Socket|Core|Thread|NUMA node\(s\)'
Model name: AMD Ryzen 9 9950X 16-Core Processor
CPU(s): 32
Thread(s) per core: 2
Core(s) per socket: 16
Socket(s): 1
NUMA node(s): 1
Reproduction results on this machine, same shell, 15 s timeout each:
$ timeout 15 lstopo >/dev/null 2>&1; echo $?
124 <- hangs
$ env -u DISPLAY timeout 15 lstopo >/dev/null 2>&1; echo $?
124 <- still hangs; DISPLAY is irrelevant
$ HWLOC_COMPONENTS=-gl timeout 15 lstopo >/dev/null 2>&1; echo $?
0 <- succeeds immediately with GL blacklisted
Note on tooling: lstopo is not present on a default install — the "hwloc"
package must be added to run the non-MPI reproducer. The DEFECT is present on
a default install regardless, because libhwloc-plugins (which contains
hwloc_gl.so) arrives automatically with the Open MPI runtime libraries.
Regression baseline: the identical application and workflow ran correctly on
Ubuntu 24.04 (hwloc 2.10.0-1build1, Open MPI 4.1.6) for years. It fails on
26.04 (hwloc 2.13.0-2, Open MPI 5.0.10-1) on both upgraded and freshly
installed systems.
ProblemType: Bug
DistroRelease: Ubuntu 26.04
Package: hwloc 2.13.0-2
ProcVersionSignature: Ubuntu 7.0.0-28.28-generic 7.0.12
Uname: Linux 7.0.0-28-generic x86_64
ApportVersion: 2.34.1-0ubuntu0.1
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Thu Aug 6 08:57:58 2026
InstallationDate: Installed on 2026-07-17 (20 days ago)
InstallationMedia: Ubuntu 26.04 "Resolute Raccoon" - Release amd64 (20260423.1)
RebootRequiredPkgs: Error: path contained symlinks.
SourcePackage: hwloc
UpgradeStatus: No upgrade log present (probably fresh install)
** Affects: hwloc (Ubuntu)
Importance: Undecided
Status: New
** Affects: openmpi (Ubuntu)
Importance: Undecided
Status: New
** Affects: pmix (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug regression-release resolute wayland-session
** Attachment added: "gdb "thread apply all bt" of a hung lstopo process.
Reproduced with distro packages only — no MPI, no user application. The main
thread is blocked in connect() inside hwloc_gl.so, reached via XOpenDisplay
from hwloc_topology_load."
https://bugs.launchpad.net/bugs/2162940/+attachment/5989900/+files/lstopo-hang-backtrace.txt
** Also affects: openmpi (Ubuntu)
Importance: Undecided
Status: New
** Also affects: pmix (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2162940
Title:
hwloc GL plugin hangs in XOpenDisplay() during topology load: plain
"lstopo" and every Open MPI singleton launch hang on 26.04 (fixed
upstream in hwloc 2.14)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/hwloc/+bug/2162940/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
