Hi Janne, Harald,

 One more thought on below part, which I think after thought I did not answer properly:

On 11/16/26 22:43, Suvarchal K Cheedela wrote:
Thank you both for the comments.  I reply to both here.

On 11/16/26 22:00, Harald Anlauf wrote:
Am 16.11.26 um 9:18 PM schrieb Janne Blomqvist:
On Mon, Nov 16, 2026 at 5:46 PM Suvarchal Kumar Cheedela
<[email protected]> wrote:

Hello there,

I would like to ask for comments on a change in how libgfortran performs asynchronous WRITE: copy the output list into a runtime buffer when the statement executes, so that the program can continue to modify its data
while the I/O thread writes it.

I believe this to be misguided. The Fortran standard is quite clear
that the programmer is not to meddle with variables that are
in-progress for asynchronous I/O. So what a feature like this would
accomplish would be to make some broken programs work correctly (for
some definition of work not defined in the Fortran standard), at the
cost of making standard-conforming programs slower and use more memory
by virtue of the extra buffer copies.

The solution, clearly, is then to fix the broken programs to manage
the extra buffering themselves, if that makes sense for the particular
application.

I do agree here.

  The cost of an asynchronous WRITE for a conforming program already
  depends on the file system, and a copy is usually made anyway, only
  not by libgfortran:

  - On a local file system, write() copies the data into the page cache
    and returns, so even the current asynchronous WRITE (with the bugs
    fixed) costs one copy and the same amount of memory, held by the
    kernel.  On a workstation (ext4), a synchronous WRITE of a 24 MB
    array took about 4-5 ms, about the same as a copy in user space.
    The opt-in copy adds a second copy of the same size: more, but of
    the same order as what the program pays already.

  - On a parallel file system such as Lustre, the client caches dirty
    pages up to a per-target limit and the servers cache as well. When
    a write() blocks, and for how long, depends on those limits and on
    the load from other jobs, none of which the program can see. The
    run-to-run spread  (4.9-17.5 s synchronous) most likely comes from that.

  So in both cases the program pays for buffering already; the question
  is only who holds the buffer and whether the program can control when
  it waits.

  I also assume that HPC is of interest to Fortran/gfortran, given the work on
  OpenMP, OpenACC and coarrays.  For HPC codes an opt-in copy makes
  buffering a performance trade-off the user can select for their file
  system and memory budget, rather than a cost imposed on every program
  or something each application has to build itself.

I agree that conforming programs must not pay for it, so I withdraw the
proposal to make the copy the default.  I would still ask you to
consider it as an option that is off by default, so that the default
path stays exactly as it is.  My reasons:

- Few programs seem to use Fortran asynchronous I/O for large data.
  WAIT with ID= has returned too early since 2018 (PR128719), and an
  asynchronous WRITE of a pointer array queued one item per element
  (PR128722, 120 s instead of 4 s for a FESOM2 restart).  As far as I
  can find, neither had been reported.

- Climate and weather models built asynchronous output outside the
  language instead: XIOS (NEMO), CDI-PIO (ICON, MPI-ESM), PIO (CESM,
  E3SM), FESOM2's own C++ writer threads.  Most of them use dedicated
  I/O processes: every field is sent over the network to processes or
  nodes that do nothing but write, and those cores are taken from the
  computation.  An I/O thread in each process writes from where the
  data already is, with no extra network traffic, and can use the cores
  or hardware threads that a node often has left over.

- In many models a restart is written from many modules, often as
  pointers into derived types.  Buffering in the application means a
  buffer per field or a packing layer, their lifetime, and WAITs in all
  of those modules.
IIUC, these types of models write restart files in a compiler-dependent
fashion, each MPI rank separately.  So this is already non-portable.
A portable approach would be an abstraction layer that is written maybe
in C and callable from Fortran that does those things the user wants.
NetCDF and HDF5 are libraries that I do use and that have a well-defined
external data representation and which are portable.  And HDF5 already supports compression (like aec) and other features which are thought to
be good for performance.  However, I've never tried to use them for
asynchronous data transfers.

Restarts and output are different here.  A restart is a dump of the
model state, read back by the same executable to continue the run, so
portability is rarely a concern, and many models write restarts as
plain binary for that reason.  A plain WRITE also takes any kind and
type the model uses, including kinds such as 16-byte reals for which
netCDF has no numeric type, and arrays in derived types, exactly as
they are in memory.  The same makes it useful for debugging: the
state at the entry or exit of a routine can be dumped and compared
with one WRITE statement, which with an external library means
declaring every variable and dimension first.  FESOM2 writes portable
netCDF restarts when portability is needed.

For output I agree that netCDF and HDF5 are the right formats, and we
tried them, including the C/C++ abstraction layer you describe. FESOM2
on Levante (Lustre), 16 nodes, all output fields every 8 steps (far
more often than in production, to make the cost visible).  Run loop in
seconds, median of 5 runs (C++ writer and PnetCDF: best of 2); 3.6 s
without output:

  netCDF, FESOM2 default (deflate level 1, one writer)    71
  netCDF, no compression                                    9.2
  FESOM2's C++ writer threads (netCDF, deflate level 1)    14.7
  PnetCDF, every rank writes its part                     250
  Fortran WRITE, file per rank, synchronous                 8.0
  Fortran WRITE, file per rank, asynchronous with copy      5.9

The netCDF paths gather every field to one writer, so the write does
not get faster with more nodes, and the C++ writer threads hide the
compression but not the gather.  PnetCDF needs a collective call per
variable and record; its output time grew from 54 s on 1 node to 247 s
on 16.  Collective netCDF-4 on HDF5 with 4 writers was as slow as the
default at 4 nodes, and the HDF5 asynchronous VOL, used through netCDF,
made the output 5-25 times slower in tests on a workstation.  So the
portable libraries do not hide the write at this scale either.
Writing from each process avoids the gather, and the asynchronous WRITE
with copy takes the rest off the critical path and makes it predictable
(5 runs: 4.9-17.5 s synchronous, 4.9-7.0 s asynchronous).  The price is
a file per process, which needs a conversion step before standard tools
can read it.

So in this case the simplest approach, a plain WRITE from each process
with no library at all, was the fastest and the most predictable. If
the compiler supports the copy, a model gets that with a few lines of
standard Fortran and no extra dependency, and I think for many codes
this outweighs the cost of the conversion step.
   - -fno-async-volatile: ASYNCHRONOUS variables are currently compiled as      VOLATILE, which stops vectorization of loops that use them (about 45%      slower on a stencil kernel; ifx loses about 50% for the same reason).
     The option compiles them like other variables.  It also needs the
     READ transfer calls of an asynchronous statement to declare their data
     argument as escaping, which is part of the series.

I would be very wary of bugs with an approach like this. The volatile
is needed as the variables can, from the perspective of the compiler,
mysteriously change behind its back.

I have no idea if this is feasible, but I wonder if the volatile
attribute could be applied to the variable only when it's part of an
in-progress async I/O, and then removed when that I/O is waited on?

I think this can be expressed through escape analysis rather than
volatile.  If the data argument of the transfer call of an asynchronous
statement is declared as escaping, the optimizer has to assume that any
later call, in particular WAIT, may read or modify the variable, so it
cannot move accesses across the WAIT.  Between the transfer and the WAIT
a conforming program does not access the variable at all.  This is also
how a buffer passed to MPI_Isend is protected until MPI_Wait.

You are right to be wary, though: the branch makes only READ items
escaping so far.  WRITE items still use the non-escaping 'W' spec, so
with volatile removed the optimizer could in principle move a store
that follows the WAIT above it.  They need the same treatment before
the option is safe, and volatile would stay the default.
When you use MPI with non-blocking communication, you also want to use
the ASYNCHRONOUS attribute for your program's buffers taking part in
the data transfer.  You'd manage the buffers within the program in
a rather obvious and transparent way.  All performance-relevant things
happen within the MPI library, which can offer this to any (Fortran)
compiler.  Why handle file I/O differently?

MPI is a good comparison, but it also offers both: next to MPI_Isend,
where the program keeps the buffer until MPI_Wait, it has had the
buffered send (MPI_Bsend) since MPI-1, where the library copies the
data into an attached buffer and the program may reuse its own buffer
at once.  The option I am asking for is the same thing for WRITE.
   - Optionally, CPU placement of the I/O thread
     (GFORTRAN_ASYNC_IO_CPUS) and several threads for large copies.

One idea I was thinking about myself many years ago would be to use
io_uring for async I/O rather than threads. But I never got as far as
implementing it. Food for thought, maybe.

Interesting, thanks.  One reason thread may be relavent here is that it can also
do CPU work off the critical path, such as compression.
Isn't this Linux specific?

6. Next step: compression on the I/O thread
-------------------------------------------

Once the data is copied, the I/O thread can also compress it without
cost to the program.  On the same runs, zstd level 1 on the I/O thread
cost nothing extra at the median and made the restarts 37% smaller (1.07
to 0.67 GB per restart on 32 nodes); compressing synchronously made it
the slowest restart case.  Writing compressed data in fixed-size chunks
would also allow aligning the writes with the stripes or objects of
parallel file systems.  A prototype (zstd, zlib, lz4 through c-blosc2,
loaded with dlopen) is on the async-compress branch, but it involves a
file format, optional libraries and an interface, so I would send a
separate RFC for it after this one.

Fortran I/O has many gnarly cases that need to be handled wrt to
jumping back and forth in a file, in-place updates of records, and
whatnot. Not sure how this would work in the face of compression where
the compressed size is unknown. This might be another of those cases
where it might be a better idea to implement in the application or as
a library rather than building into the compiler.

That's a reason to use HDF5 or something similar.
The prototype only handles unformatted sequential files, where a WRITE
always ends the file, so there are no in-place updates; the record
markers hold the compressed lengths, so BACKSPACE works.  Still, I take
the point that compression is a large step for libgfortran, and I will
not pursue it before the copy question is settled.

Best,

Suvarchal

Reply via email to