Hello there,
I would like to ask for comments on a change in how libgfortran performs
asynchronous WRITE: copy the output list into a runtime buffer when the
statement executes, so that the program can continue to modify its data
while the I/O thread writes it. Patches exist and are tested; before
posting them I would like to agree on the user interface (environment
variable, option, or default) and on whether compression on the I/O
thread, which builds on it, would be welcome as a follow-up.
This is my first RFC to GCC, so I would also be grateful for advice on
how you would like the patches split and presented.
1. Use case: restarts and output of climate models
--------------------------------------------------
A climate model run of several centuries is a chain of batch jobs.
At the
end of every job, and often more frequently, each process writes its
complete state (a restart) so that the next job can continue. At high
resolution a single restart is terabytes. In between, the model writes
its science output (daily or monthly means). What matters is throughput
in simulated years per day, and every second a process spends in a WRITE
comes straight out of it.
This is exactly what Fortran asynchronous I/O was designed for, and
it has
been in the language since Fortran 2003. In practice models do not
use it.
FESOM2 (the AWI ocean model) and others carry their own threaded writers
in C or C++, or depend on libraries such as netCDF, PnetCDF, XIOS or the
HDF5 asynchronous VOL to get the write off the critical path.
The reason is the zero-copy implementation. libgfortran queues a
pointer
to the data, so the program must not change it until WAIT. A model
changes its whole state in the next time step, so a conforming program
has to WAIT before that step, and nothing overlaps.
2. Measurements
---------------
FESOM2, CORE2 mesh (medium size short ocean simulations) , raw
restart written with unformatted asynchronous
WRITE, measured on a workstation and on an HPC system.
a) Workstation (one node, local NVMe disk), time per process inside the
restart routine:
processes x threads synchronous asynchronous, copy
2 x 4 1.83 s 0.44 s
4 x 2 1.03 s 0.35 s
8 x 1 0.54 s 0.29 s
The write lands in the page cache, so even the synchronous write is
cheap and the absolute gain is fractions of a second. This is
probably why the limitation has not been an issue so far.
b) Levante (DKRZ, Lustre), 32 processes x 4 threads per node, restarts
every 16 steps (much more often than in production, so that the cost
is measurable in a short run). Whole run loop in seconds, median of
5 runs, with the range:
16 nodes 32 nodes
no restart 3.65 3.61
synchronous WRITE 7.75 (5.37 - 13.70) 12.30 (4.50 -
14.22)
asynchronous, copy 5.36 (4.90 - 6.40) 6.02 (5.55 -
8.96)
There is no page cache to absorb the write: synchronous restarts
doubled (16 nodes) or tripled (32 nodes) the run time, and varied by
up to a factor of three between runs. With the copy they cost
47-67% and stay within a narrow range.
So the copy helps on both systems, but it matters on the parallel file
systems that large simulations run on.
Without the copy, FESOM2 overwrote its arrays while the I/O thread was
still writing them: with every intermediate restart kept, 146 of 896
restart files differed from those of the synchronous run. With the copy
all of them are byte-identical.
3. Other compilers
------------------
On the same test programs (Intel ifx 2025.3, NVIDIA nvfortran 25.11):
ifx makes the same choice as gfortran, an I/O thread per unit and no
copy. nvfortran completes the transfer inside the WRITE statement,
which
is safe but not asynchronous. The standard allows all of these, and a
conforming program cannot tell the difference. So today no compiler
offers asynchronous output that a model can use without freezing its
state, and gfortran could be the first, it has a high impact in
climate simulations in saving energy.
4. Prerequisite fixes
---------------------
Using asynchronous I/O in FESOM2 also exposed four bugs, now reported
with patches:
PR128719 WAIT with ID= returns before the statement completes
PR128720 defined I/O in an asynchronous transfer crashes
PR128721 skipped items leak after an asynchronous I/O error
PR128722 pointer arrays are transferred element by element
(a FESOM2 restart took 120 s instead of 4 s)
https://inbox.sourceware.org/gcc-patches/[email protected]/
ifx crashes on the defined I/O case as well.
5. The proposal
---------------
Branch async-copy on https://github.com/suvarchal/gcc, on top of the
fixes:
- Copy on enqueue: an asynchronous WRITE copies its items into
buffers
from a pool owned by the unit and returns; the I/O thread writes
and
releases them. The amount of outstanding copies per unit can be
limited; above the limit the statement first waits for the unit.
- Buffered asynchronous READ: data is read into a buffer and
copied to
the program's variables at WAIT.
- -fno-async-volatile: ASYNCHRONOUS variables are currently
compiled as
VOLATILE, which stops vectorization of loops that use them
(about 45%
slower on a stencil kernel; ifx loses about 50% for the same
reason).
The option compiles them like other variables. It also needs the
READ transfer calls of an asynchronous statement to declare
their data
argument as escaping, which is part of the series.
- Optionally, CPU placement of the I/O thread
(GFORTRAN_ASYNC_IO_CPUS) and several threads for large copies.
These are separate patches and can be reviewed and taken one by one; the
copy for WRITE is the part that matters most.
A standard-conforming program behaves the same with and without the
copy.
The cost is memory for the outstanding output and one memory copy in the
calling thread.
The question I would most like opinions on is how the copy is enabled:
a) an environment variable (GFORTRAN_ASYNC_COPY=yes, as on the
branch);
b) a compiler option that sets a flag in the statement's parameter
block, so that the compiled code and the runtime cannot disagree;
c) by default for every asynchronous WRITE, with a memory limit
and an
environment variable to switch it off.
I would argue for (c). Asynchronous WRITE is rarely used today, and the
copy turns its most common misuse (modifying the data before WAIT)
from a
silent data race into the intended result. The cost is one memory copy,
which is small compared with writing the data.
6. Next step: compression on the I/O thread
-------------------------------------------
Once the data is copied, the I/O thread can also compress it without
cost to the program. On the same runs, zstd level 1 on the I/O thread
cost nothing extra at the median and made the restarts 37% smaller (1.07
to 0.67 GB per restart on 32 nodes); compressing synchronously made it
the slowest restart case. Writing compressed data in fixed-size chunks
would also allow aligning the writes with the stripes or objects of
parallel file systems. A prototype (zstd, zlib, lz4 through c-blosc2,
loaded with dlopen) is on the async-compress branch, but it involves a
file format, optional libraries and an interface, so I would send a
separate RFC for it after this one.
Questions
---------
1. Is copying the data of asynchronous WRITE acceptable in
libgfortran,
and should it be the default (5c) or opt-in (5a/5b)?
2. Is -fno-async-volatile acceptable, and should it be tied to the
copy?
3. Would you prefer one RFC for copy and compression together, or
separate ones as above?
4. As trunk is now in stage 3, as i see, should I aim as
improvement for 17? or for GCC 18 stage 1. I think this path being
only fortran has little risk? I am excited seeing potential
computational savings and energy saving this enables.
Thanks,
Suvarchal
--
Suvarchal Kumar Cheedela
Alfred Wegener Institute (AWI)
[email protected]