Hello again,

Looking back, I was carried away by what this could do for climate
model restarts, output and compression all at once, and I did not state
my intent clearly; the proposal may have read as a way to make
non-conforming programs work.  That is not my intention.  My intention
is to make it simple to overlap computation with I/O, to save compute
time and energy.  Since the first mail I have made the copy opt-in only
and left compression out.  So let me try again with one narrower but
important case as an example: restarts, and what it currently takes to
overlap them with computation in a conforming program.  Some
background: I have worked on climate models for many years, but
compilers are new ground for me.  Part of my motivation is to close the
gap between model developers and compiler developers, and to show that
such collaboration is possible with open-source compilers, now also
with the help of AI tools.  Please excuse any ignorance on my side.

The use case

- Climate and weather models have many components, such as land,
  ocean and atmosphere, each with sub-components, and they typically
  keep each component's state in derived types.  As an example, the
  restart of the ocean model FESOM2 consists of about 50 fields from
  tens of modules; atmosphere components often have many more modules
  and fields.  The fields include array sections such as UV(1,:,:),
  pointers and nested types, in different precisions such as REAL(4)
  and REAL(8).
- A restart dumps that state as a binary image.  Runs must restart
  because of queue limits; in km-scale runs restarts are expensive in
  storage and compute, and critical for debugging.  They are also
  essential for reproducibility: a state that is not restored
  completely, or a default applied on initialization from a restart,
  can take the model onto a different trajectory.
- Today this is usually a synchronous unformatted WRITE, and the model
  waits for it.  The goal i am exploring is to overlap that write with the computation
  of the following steps, which do not need the file; in that case the
  savings are significant.

What it costs, and why it depends on the file system

- On a local disk, write() copies into the page cache and returns: a
  synchronous WRITE of 8 MB takes 1.3 ms.  So there is little to gain
  there.
- On a parallel file system such as Lustre, write() returns once the
  client cache takes the data, or blocks on the servers when its limit
  is reached, depending on other jobs.
- Measured: FESOM2 alone at about 100 km (1.07 GB per restart) on the
  Levante supercomputer runs at 173 simulated years per day (SYPD) on
  16 nodes without restarts.  With a restart every 16 steps (frequent,
  so that the cost shows in a short run) it drops to 81 SYPD with
  synchronous writes, -53% (median of 5 runs; -32% to -73%), and to
  118 SYPD, -32% (-26% to -43%), when the data is copied at the WRITE
  and written by a background thread.  On 32 nodes: 175 SYPD without
  restarts, 51 SYPD synchronous (-71%) and 105 SYPD copied (-40%).
  The copied case still waits here only because 16 steps take less
  time than writing the restart; with a longer interval the write is
  hidden.

What the current asynchronous I/O allows (with the reported bugs fixed)

- WRITE(ASYNCHRONOUS='YES') returns at once, but the standard requires
  that the data not change until the WAIT, and that every routine that
  references it in the meantime gives it the ASYNCHRONOUS attribute.
  A model step changes its whole state, so the WAIT has to come
  before the next step and nothing overlaps.  Even overlapping reads
  would cost: gfortran compiles ASYNCHRONOUS variables as volatile,
  which made a cache-resident stencil about 45% slower in my tests
  (ifx +52%).
- The unit buffer does not change this: with asynchronous I/O it is
  filled by the I/O thread after the statement returns, and it holds
  128 KB, far less than a typical restart.
- What remains is double buffering in the application, as some of you
   proposed: copy the state,
  into buffers of its own, write those asynchronously, and WAIT at the
  next restart.  It is conforming and gives the overlap, but:
  - every model has to write it again, with a buffer and packing code
    for each component, section, pointer and nested type, and extend it
    whenever a component is added; a field that is forgotten is a
    restart that silently differs;
  - it costs the same file-system time as a copy in the runtime, and
    the buffers stay allocated between restarts unless the code
    manages that too.
  The compiler and runtime already walk all of these structures for
  every WRITE (descriptors, scalarized components, defined I/O), so
  this is a pattern every model repeats for what the runtime already
  knows.

What I propose

- Overlap needs two things: a copy made at the statement, so that the
  variable is free when the WRITE returns, and a thread that waits for
  the file system.  Today a synchronous WRITE has the first (for the
  pieces that go through the unit buffer) and asynchronous I/O has the
  second.  The proposal combines them: in effect, a synchronous WRITE
  into a large buffer whose flush runs on the I/O thread.
- An opt-in runtime setting, next to GFORTRAN_UNFORMATTED_BUFFER_SIZE:
  for an unformatted WRITE, synchronous or asynchronous, the runtime
  copies the data at the statement and returns, and a background
  thread writes it.  For an asynchronous WRITE this means the data is
  no longer pending when the statement returns, so the WAIT returns at
  once.
- Nothing is pending after the WRITE, so the program needs no
  ASYNCHRONOUS attribute, no volatile and no WAIT.  It also works for
  defined I/O, which the standard does not(?) allow to be asynchronous.

  Conforming programs only, no new syntax, the same
  results with the option off and with every compiler.
- READ, FLUSH, CLOSE, INQUIRE of the size, WAIT and
  program end drain the queue first, so the file is consistent wherever
  a program can observe it.
- Trade-off: an I/O error is reported at the next of those statements
  rather than at the WRITE, as already happens for data that sits in
  the unit buffer when a WRITE returns.

Why in libgfortran, and why it is one pass

- The runtime already copies most of this data: pieces of up to 64 KB
  (half the unit buffer) go through it, i.e. every element of a
  section, pointer array or array component, every column of a 2-D
  array (PR128796), and everything under GFORTRAN_CONVERT_UNIT --intrusive.
  Only contiguous blocks over 64 KB go to write() directly.  The option adds a copy for
  those, and makes the rest in one pass into its own buffer instead of
  many small ones.
- It has the descriptors and sees the child WRITEs of defined I/O, so
  one implementation serves every model, whatever its types.
- It also makes extensions simple.  A plugin loaded under the program,
  for example one that compresses, runs on the runtime's I/O thread
  and reads the runtime's copy as it is passed to write(): it needs no
  thread of its own and no copy of the data, only its output buffer.
  Nothing of that needs to be in libgfortran.

I understand the bug fixes come first (PR128719-128722, PR128796).  Would such an opt-in write-behind for unformatted output may be acceptable in principle later or evaluated?
If so, I would prepare a new patch after those.  I am happy to provide
measurements, test programs or more details for any of the points
above.

Regards,
Suvarchal

On 11/17/26 10:47, Janne Blomqvist wrote:
On Tue, Nov 17, 2026 at 4:33 AM Suvarchal K Cheedela
<[email protected]> wrote:
    Hi Janne, Harald,

   One more thought on below part, which I think after thought I did not
answer properly:
    The cost of an asynchronous WRITE for a conforming program already
    depends on the file system, and a copy is usually made anyway, only
    not by libgfortran:

    - On a local file system, write() copies the data into the page cache
      and returns, so even the current asynchronous WRITE (with the bugs
      fixed) costs one copy and the same amount of memory, held by the
      kernel.  On a workstation (ext4), a synchronous WRITE of a 24 MB
      array took about 4-5 ms, about the same as a copy in user space.
      The opt-in copy adds a second copy of the same size: more, but of
      the same order as what the program pays already.

    - On a parallel file system such as Lustre, the client caches dirty
      pages up to a per-target limit and the servers cache as well. When
      a write() blocks, and for how long, depends on those limits and on
      the load from other jobs, none of which the program can see. The
      run-to-run spread  (4.9-17.5 s synchronous) most likely comes from
that.

    So in both cases the program pays for buffering already; the question
    is only who holds the buffer and whether the program can control when
    it waits.
No? If you add an extra layer of buffering, it's yet another layer of
buffers that have to be copied. It does not replace the page cache,
which is still there. Unless your patch uses O_DIRECT (I haven't
looked) which is its own huge can of worms and will likely not work
well as a general purpose solution.

    I also assume that HPC is of interest to Fortran/gfortran, given the
work on
    OpenMP, OpenACC and coarrays.  For HPC codes an opt-in copy makes
    buffering a performance trade-off the user can select for their file
    system and memory budget, rather than a cost imposed on every program
    or something each application has to build itself.
The extra buffering is something a conforming program must do anyway,
if they want to keep accessing data which is in-progress for
asynchronous I/O. Either by implementing it in the application itself
or by using some library.

In general Fortran has a long and sordid history of vendors trying to
entice users by adding all kinds of extensions to their compilers,
creating a legacy of portability quagmires. Luckily that is mostly
behind us, but let's not go there again please. The very small
development community has enough troubles in implementing the
standardized features as it is. (With the caveat that I haven't been
an active GFortran developer for several years, so my comments in this
thread are more from the perspective of an interested bystander than
an official position of the development team!)


Reply via email to