https://github.com/mjklemm updated https://github.com/llvm/llvm-project/pull/216751
>From 983c11a0c78e447adf3d9080d868e20a7110e88a Mon Sep 17 00:00:00 2001 From: Michael Klemm <[email protected]> Date: Fri, 14 Aug 2026 20:45:42 +0200 Subject: [PATCH 1/2] [flang-rt] Add OpenMP target-memory allocator runtime Introduce a Fortran runtime component that registers an allocator (index 1 in the Fortran runtime's allocator registry) backed by the OpenMP host API routines omp_target_alloc/omp_target_free. Enables ALLOCATABLE variables to live in device memory when compiled with -fopenmp-default-allocate=target (compiler support added separately). Contents: * flang-rt/lib/openmp/omp_alloc.cpp -- OpenMPAlloc/OpenMPFree callbacks plus the RTDEF entry points OpenMPRegisterAllocator and OpenMPAllocatableSetAllocIdx. * flang-rt/lib/openmp/omp_util.cpp -- PointerDeviceMap, a small thread-safe pointer->device-id map used so that OpenMPFree can pass the correct device to omp_target_free even when the default device changes between allocation and deallocation. * flang/include/flang/Runtime/OpenMP/{omp_alloc,omp_util}.h -- public declarations. * flang-rt/lib/openmp/CMakeLists.txt -- builds flang_rt.openmp as a STATIC+SHARED library, installed with the toolchain. * flang-rt/lib/CMakeLists.txt -- gates the new subdirectory on "openmp" being present in LLVM_ENABLE_RUNTIMES. The OpenMP host entry points are declared locally to avoid a build-time dependency on omp.h from the OpenMP runtime headers. [flang][OpenMP] Add -fopenmp-default-allocate= driver flag and lowering Introduce a new experimental driver flag, -fopenmp-default-allocate=, that selects the default allocator for OpenMP-managed ALLOCATABLE variables. Two values are accepted: * =host -- no behavior change (allocations stay in host memory). * =target -- allocations are routed through the Fortran runtime's allocator index 1, which is provided by the flang_rt.openmp component and dispatches to omp_target_alloc / omp_target_free on the current default OpenMP device. Compiler pieces: * clang/include/clang/Options/Options.td -- flag definition. * clang/include/clang/Basic/DiagnosticDriverKinds.td -- warning that labels the option as experimental. * clang/lib/Driver/ToolChains/Flang.cpp -- validates the value, emits the experimental-feature warning, forwards the flag to -fc1, and (for =target) enables the existing -use-alloc-runtime MLIR pass so ALLOCATE always goes through the runtime path. * flang/include/flang/Support/Fortran-features.h -- adds a new LanguageFeature bit OpenMPDefaultAllocator that carries the choice into semantics/lowering. * flang/lib/Frontend/CompilerInvocation.cpp -- fc1 parses the flag and sets the language feature. * flang/lib/Optimizer/Builder/Runtime/Main.{h,cpp} -- genMain gains an enableOpenMPAllocator parameter; when set, it emits a call to _FortranAOpenMPRegisterAllocator so the allocator is registered once at program start. * flang/lib/Lower/Bridge.cpp -- passes the feature bit through to genMain. * flang/lib/Lower/Allocatable.cpp -- on the runtime-allocate path, emits _FortranAOpenMPAllocatableSetAllocIdx to tag the descriptor with allocator id 1 before ALLOCATE. A helper skips the tag when the code is already inside an omp.target region or a declare-target function, where the target runtime handles allocations directly. Depends on the flang_rt.openmp runtime component (added in the preceding commit); the compiler references its RT entry points OpenMPRegisterAllocator and OpenMPAllocatableSetAllocIdx. Tests: * flang/test/Driver/fopenmp-default-allocate.f90 -- driver forwarding and value validation. * flang/test/Lower/OpenMP/omp_alloc_init{,_host}.f90 -- verifies the program-start OpenMPRegisterAllocator call is (or is not) emitted. * flang/test/Lower/AMDGPU/allocate_*_omp_*.f90, allocate_runtime_alloc_idx{,_host}.f90 -- verifies the OpenMPAllocatableSetAllocIdx tagging and its suppression in device code. --- .../clang/Basic/DiagnosticDriverKinds.td | 2 + clang/include/clang/Options/Options.td | 4 + clang/lib/Driver/ToolChain.cpp | 4 +- clang/lib/Driver/ToolChains/Flang.cpp | 15 +++ clang/test/Misc/warning-flags.c | 3 +- flang-rt/lib/CMakeLists.txt | 3 + flang-rt/lib/openmp/CMakeLists.txt | 27 +++++ flang-rt/lib/openmp/omp_alloc.cpp | 107 ++++++++++++++++++ flang-rt/lib/openmp/omp_util.cpp | 73 ++++++++++++ .../flang/Optimizer/Builder/Runtime/Main.h | 3 +- .../include/flang/Runtime/OpenMP/omp_alloc.h | 38 +++++++ flang/include/flang/Runtime/OpenMP/omp_util.h | 53 +++++++++ .../include/flang/Support/Fortran-features.h | 1 + flang/lib/Frontend/CompilerInvocation.cpp | 13 +++ flang/lib/Lower/Allocatable.cpp | 57 ++++++++++ flang/lib/Lower/Bridge.cpp | 2 + flang/lib/Optimizer/Builder/Runtime/Main.cpp | 10 +- flang/lib/Support/Fortran-features.cpp | 1 + .../test/Driver/fopenmp-default-allocate.f90 | 28 +++++ ...allocate_deallocate_omp_declare_target.f90 | 24 ++++ ...e_deallocate_omp_declare_target_nested.f90 | 25 ++++ .../AMDGPU/allocate_deallocate_omp_target.f90 | 24 ++++ .../AMDGPU/allocate_runtime_alloc_idx.f90 | 20 ++++ .../allocate_runtime_alloc_idx_host.f90 | 17 +++ flang/test/Lower/OpenMP/omp_alloc_init.f90 | 5 + .../test/Lower/OpenMP/omp_alloc_init_host.f90 | 5 + 26 files changed, 560 insertions(+), 4 deletions(-) create mode 100644 flang-rt/lib/openmp/CMakeLists.txt create mode 100644 flang-rt/lib/openmp/omp_alloc.cpp create mode 100644 flang-rt/lib/openmp/omp_util.cpp create mode 100644 flang/include/flang/Runtime/OpenMP/omp_alloc.h create mode 100644 flang/include/flang/Runtime/OpenMP/omp_util.h create mode 100644 flang/test/Driver/fopenmp-default-allocate.f90 create mode 100644 flang/test/Lower/AMDGPU/allocate_deallocate_omp_declare_target.f90 create mode 100644 flang/test/Lower/AMDGPU/allocate_deallocate_omp_declare_target_nested.f90 create mode 100644 flang/test/Lower/AMDGPU/allocate_deallocate_omp_target.f90 create mode 100644 flang/test/Lower/AMDGPU/allocate_runtime_alloc_idx.f90 create mode 100644 flang/test/Lower/AMDGPU/allocate_runtime_alloc_idx_host.f90 create mode 100644 flang/test/Lower/OpenMP/omp_alloc_init.f90 create mode 100644 flang/test/Lower/OpenMP/omp_alloc_init_host.f90 diff --git a/clang/include/clang/Basic/DiagnosticDriverKinds.td b/clang/include/clang/Basic/DiagnosticDriverKinds.td index 91895d4957cf7..fe8a73e6c5dd3 100644 --- a/clang/include/clang/Basic/DiagnosticDriverKinds.td +++ b/clang/include/clang/Basic/DiagnosticDriverKinds.td @@ -186,6 +186,8 @@ def warn_openmp_spec_incomplete : Warning< "the specification for OpenMP version %0 is still under development; " "the syntax and semantics of new features may be subject to change">, InGroup<ExperimentalOption>; +def warn_openmp_default_allocate_experimental + : Warning<"-fopenmp-default-allocate= is an experimental feature">; def err_drv_invalid_thread_model_for_target : Error< "invalid thread model '%0' in '%1' for this target">; def err_drv_invalid_linker_name : Error< diff --git a/clang/include/clang/Options/Options.td b/clang/include/clang/Options/Options.td index 3abb04a5f36c7..6a9eceed05012 100644 --- a/clang/include/clang/Options/Options.td +++ b/clang/include/clang/Options/Options.td @@ -4145,6 +4145,10 @@ def fomit_frame_pointer : Flag<["-"], "fomit-frame-pointer">, Group<f_Group>, "Some stack unwinding cases, such as profilers and sanitizers, may prefer specifying -fno-omit-frame-pointer. " "On many targets, -O1 and higher omit the frame pointer by default. " "-m[no-]omit-leaf-frame-pointer takes precedence for leaf functions">; +def fopenmp_default_allocate_EQ : Joined<["-"], "fopenmp-default-allocate=">, + Group<f_Group>, Visibility<[FlangOption, FC1Option]>, + HelpText<"Set default allocator for OpenMP offloading (=target or =host)">, + Values<"target,host">; def fopenmp : Flag<["-"], "fopenmp">, Group<f_Group>, Flags<[NoArgumentUnused]>, Visibility<[ClangOption, CC1Option, FlangOption, FC1Option]>, diff --git a/clang/lib/Driver/ToolChain.cpp b/clang/lib/Driver/ToolChain.cpp index 37db75a1bfc14..edb9b0e560825 100644 --- a/clang/lib/Driver/ToolChain.cpp +++ b/clang/lib/Driver/ToolChain.cpp @@ -997,9 +997,11 @@ void ToolChain::addFortranRuntimeLibs(const ArgList &Args, CmdArgs.push_back("-lexecinfo"); } - // libomp needs libatomic for atomic operations if using libgcc if (Args.hasFlag(options::OPT_fopenmp, options::OPT_fopenmp_EQ, options::OPT_fno_openmp, false)) { + CmdArgs.push_back("-lflang_rt.openmp"); + + // libomp needs libatomic for atomic operations if using libgcc Driver::OpenMPRuntimeKind OMPRuntime = getDriver().getOpenMPRuntime(Args); ToolChain::RuntimeLibType RuntimeLib = GetRuntimeLibType(Args); if ((OMPRuntime == Driver::OMPRT_OMP && diff --git a/clang/lib/Driver/ToolChains/Flang.cpp b/clang/lib/Driver/ToolChains/Flang.cpp index a48e41159f367..84f1734cd19eb 100644 --- a/clang/lib/Driver/ToolChains/Flang.cpp +++ b/clang/lib/Driver/ToolChains/Flang.cpp @@ -1310,6 +1310,21 @@ void Flang::ConstructJob(Compilation &C, const JobAction &JA, addFortranDialectOptions(Args, CmdArgs); + if (const Arg *A = + Args.getLastArg(options::OPT_fopenmp_default_allocate_EQ)) { + StringRef Val(A->getValue()); + if (Val != "target" && Val != "host") { + D.Diag(diag::err_drv_invalid_value) << A->getAsString(Args) << Val; + } else { + D.Diag(diag::warn_openmp_default_allocate_experimental); + CmdArgs.push_back(Args.MakeArgString("-fopenmp-default-allocate=" + Val)); + if (Val == "target") { + CmdArgs.push_back("-mmlir"); + CmdArgs.push_back("-use-alloc-runtime"); + } + } + } + // 'flang -E' always produces output that is suitable for use as fixed form // Fortran. However it is only valid free form source if the original is also // free form. Ensure this logic does not incorrectly assume fixed-form for diff --git a/clang/test/Misc/warning-flags.c b/clang/test/Misc/warning-flags.c index 3dc4bb55aa69c..f4d38386c25c9 100644 --- a/clang/test/Misc/warning-flags.c +++ b/clang/test/Misc/warning-flags.c @@ -18,7 +18,7 @@ This test serves two purposes: The list of warnings below should NEVER grow. It should gradually shrink to 0. -CHECK: Warnings without flags (56): +CHECK: Warnings without flags (57): CHECK-NEXT: ext_expected_semi_decl_list CHECK-NEXT: ext_missing_whitespace_after_macro_name @@ -61,6 +61,7 @@ CHECK-NEXT: warn_not_compound_assign CHECK-NEXT: warn_objc_property_copy_missing_on_block CHECK-NEXT: warn_objc_protocol_qualifier_missing_id CHECK-NEXT: warn_on_superclass_use +CHECK-NEXT: warn_openmp_default_allocate_experimental CHECK-NEXT: warn_pp_convert_to_positive CHECK-NEXT: warn_pp_expr_overflow CHECK-NEXT: warn_pp_line_decimal diff --git a/flang-rt/lib/CMakeLists.txt b/flang-rt/lib/CMakeLists.txt index 58a7a24c19e0c..7b84e38ff6498 100644 --- a/flang-rt/lib/CMakeLists.txt +++ b/flang-rt/lib/CMakeLists.txt @@ -9,6 +9,9 @@ if (FLANG_RT_ENABLE_STATIC OR FLANG_RT_ENABLE_SHARED) add_subdirectory(quadmath) add_subdirectory(runtime) + if ("openmp" IN_LIST LLVM_ENABLE_RUNTIMES) + add_subdirectory(openmp) + endif() if (FLANG_RT_INCLUDE_CUF) add_subdirectory(cuda) endif() diff --git a/flang-rt/lib/openmp/CMakeLists.txt b/flang-rt/lib/openmp/CMakeLists.txt new file mode 100644 index 0000000000000..ced202478d253 --- /dev/null +++ b/flang-rt/lib/openmp/CMakeLists.txt @@ -0,0 +1,27 @@ +#===-- lib/openmp/CMakeLists.txt --------------------------------------------===# +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +#===------------------------------------------------------------------------===# + +# Check that Umpire exits in the directory given at CMake +# TODO: this was disabled to get to an easier build procedure for now +#message(STATUS "Using Umpire in directory ${FLANG_RT_UMPIRE_DIR}") +#set(umpire_DIR ${FLANG_RT_UMPIRE_DIR}) +#find_package(umpire REQUIRED PATHS ${FLANG_RT_UMPIRE_DIR}/lib/cmake/umpire) + +add_flangrt_library(flang_rt.openmp STATIC SHARED + omp_alloc.cpp + omp_util.cpp + INSTALL_WITH_TOOLCHAIN +) + +#if (TARGET flang_rt.openmp.static) +# target_include_directories(flang_rt.openmp.static PRIVATE ${FLANG_RT_UMPIRE_DIR}/include) +#endif() +# +#if (TARGET flang_rt.openmp.shared) +# target_include_directories(flang_rt.openmp.shared PRIVATE ${FLANG_RT_UMPIRE_DIR}/include) +#endif() diff --git a/flang-rt/lib/openmp/omp_alloc.cpp b/flang-rt/lib/openmp/omp_alloc.cpp new file mode 100644 index 0000000000000..4cc93c4b0ca40 --- /dev/null +++ b/flang-rt/lib/openmp/omp_alloc.cpp @@ -0,0 +1,107 @@ +//===-- lib/openmp/omp_alloc.cpp ---------------------------------*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#define ALLOC_DEBUG 1 + +#include "flang/Runtime/OpenMP/omp_alloc.h" +#include "flang-rt/runtime/allocator-registry.h" +#include "flang-rt/runtime/descriptor.h" +#include "flang-rt/runtime/terminator.h" +#include "flang/Runtime/OpenMP/omp_util.h" +#include "flang/Support/Fortran.h" +#include <cstdio> +#include <cstdlib> + +namespace Fortran::runtime::omp { + +static bool debugEnabled; + +// Declare OpenMP memory management routines to avoid importing +// definitions via "omp.h" (and thus create a dependency to the +// OpenMP runtime library code). +extern "C" int omp_get_default_device(void); +extern "C" void *omp_target_alloc(std::size_t, int); +extern "C" void omp_target_free(void *, int); + +// Track which device each pointer was allocated on so that +// OpenMPFree can pass the correct device ID to omp_target_free, +// even if omp_set_default_device() was called between ALLOCATE +// and DEALLOCATE. +static PointerDeviceMap allocDeviceMap; + +/// Allocate \p AllocationSize bytes on the current default OpenMP device. +static void *OpenMPAlloc(std::size_t AllocationSize, std::size_t, std::int64_t *) { +#if ALLOC_DEBUG + if (debugEnabled) { + std::fprintf(stderr, "[OMP_ALLOC] %s(%zu) (%s:%d)\n", __PRETTY_FUNCTION__, + AllocationSize, __FILE__, __LINE__); + } +#endif + int device{omp_get_default_device()}; + void *pointer{omp_target_alloc(AllocationSize, device)}; + if (pointer) { + allocDeviceMap.insert(pointer, device); + } +#if ALLOC_DEBUG + if (debugEnabled) { + std::fprintf(stderr, + "[OMP_ALLOC] pointer of size %zu allocated at %p" + " on device %d.\n", + AllocationSize, pointer, device); + } +#endif + return pointer; +} + +/// Free a pointer previously allocated by OpenMPAlloc on the correct device. +static void OpenMPFree(void *pointer) { + int device{allocDeviceMap.removeAndGet(pointer)}; + if (device == -1) { + Terminator{__FILE__, __LINE__}.Crash( + "OpenMPFree: pointer %p was not allocated by OpenMPAlloc", pointer); + } +#if ALLOC_DEBUG + if (debugEnabled) { + std::fprintf(stderr, "[OMP_ALLOC] %s(%p) device %d (%s:%d)\n", + __PRETTY_FUNCTION__, pointer, device, __FILE__, __LINE__); + } +#endif + omp_target_free(pointer, device); +} + +extern "C" { +void RTDEF(OpenMPRegisterAllocator)() { +#if ALLOC_DEBUG + debugEnabled = false; + if (const char *env = std::getenv("OMP_ALLOC_DEBUG")) { + debugEnabled = env[0] != '0' && env[0] != '\0'; + } + if (debugEnabled) { + std::fprintf(stderr, "[OMP_ALLOC] %s (%s:%d)\n", __PRETTY_FUNCTION__, + __FILE__, __LINE__); + std::fprintf( + stderr, "[OMP_ALLOC] registering OpenMP device memory allocator\n"); + } +#endif + allocatorRegistry.Register(1, {&OpenMPAlloc, &OpenMPFree}); +} + +void RTDEF(OpenMPAllocatableSetAllocIdx)(Descriptor &descriptor, int pos) { + if (descriptor.IsAllocatable() && !descriptor.IsAllocated()) { +#if ALLOC_DEBUG + if (debugEnabled) { + std::fprintf( + stderr, "[OMP_ALLOC] OpenMPAllocatableSetAllocIdx = %d \n", pos); + } +#endif + descriptor.SetAllocIdx(pos); + } +} +} // extern "C" + +} // namespace Fortran::runtime::omp diff --git a/flang-rt/lib/openmp/omp_util.cpp b/flang-rt/lib/openmp/omp_util.cpp new file mode 100644 index 0000000000000..cabef7574363d --- /dev/null +++ b/flang-rt/lib/openmp/omp_util.cpp @@ -0,0 +1,73 @@ +//===-- lib/openmp/omp_util.cpp ----------------------------------*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// Implementation of PointerDeviceMap -- a thread-safe pointer-to-device-ID +// map used by the OpenMP allocator runtime to track allocation origins. +// +//===----------------------------------------------------------------------===// + +#include "flang/Runtime/OpenMP/omp_util.h" +#include "flang-rt/runtime/lock.h" +#include "flang-rt/runtime/terminator.h" +#include <cstdlib> +#include <cstring> + +namespace Fortran::runtime::omp { + +static constexpr std::size_t initialCapacity{256}; + +static Lock pointerDeviceMapLock; + +/// Double the capacity of the entries array (or set it to initialCapacity if +/// empty). Crashes on allocation failure. Must be called under the lock. +void PointerDeviceMap::grow() { + std::size_t newCapacity = capacity_ ? capacity_ * 2 : initialCapacity; + Entry *newEntries = + static_cast<Entry *>(std::realloc(entries_, newCapacity * sizeof(Entry))); + if (!newEntries) { + Terminator{__FILE__, __LINE__}.Crash( + "PointerDeviceMap: realloc failed (capacity %zu)", newCapacity); + } + entries_ = newEntries; + capacity_ = newCapacity; +} + +/// Record that \p pointer was allocated on \p device. Thread-safe. +void PointerDeviceMap::insert(void *pointer, int device) { + CriticalSection guard(pointerDeviceMapLock); + if (count_ == capacity_) { + grow(); + } + entries_[count_++] = {pointer, device}; +} + +/// Remove \p pointer from the map and return its device ID, or -1 if not +/// found. Uses swap-with-last for O(1) removal. Thread-safe. +int PointerDeviceMap::removeAndGet(void *pointer) { + CriticalSection guard(pointerDeviceMapLock); + for (std::size_t i = 0; i < count_; ++i) { + if (entries_[i].pointer == pointer) { + int device = entries_[i].device; + // Swap with last entry and shrink. + entries_[i] = entries_[--count_]; + return device; + } + } + return -1; +} + +/// Print all (pointer, device) entries to stderr. Thread-safe. +/// Can be used for debugging purposes. +void PointerDeviceMap::dump() const { + CriticalSection guard(pointerDeviceMapLock); + for (std::size_t i = 0; i < count_; ++i) { + std::fprintf(stderr, "%p -> %d\n", entries_[i].pointer, entries_[i].device); + } +} + +} // namespace Fortran::runtime::omp diff --git a/flang/include/flang/Optimizer/Builder/Runtime/Main.h b/flang/include/flang/Optimizer/Builder/Runtime/Main.h index 1acc3cbed35c5..4bce249c2c3b2 100644 --- a/flang/include/flang/Optimizer/Builder/Runtime/Main.h +++ b/flang/include/flang/Optimizer/Builder/Runtime/Main.h @@ -25,7 +25,8 @@ namespace fir::runtime { void genMain(fir::FirOpBuilder &builder, mlir::Location loc, const std::vector<Fortran::lower::EnvironmentDefault> &defs, - bool initCuda = false, bool initCoarrayEnv = false, + bool initCuda = false, bool enableOpenMPAllocator = false, + bool initCoarrayEnv = false, unsigned fpExceptionTraps = 0); } diff --git a/flang/include/flang/Runtime/OpenMP/omp_alloc.h b/flang/include/flang/Runtime/OpenMP/omp_alloc.h new file mode 100644 index 0000000000000..ba5fb95c2cade --- /dev/null +++ b/flang/include/flang/Runtime/OpenMP/omp_alloc.h @@ -0,0 +1,38 @@ +//===-- include/flang/Runtime/OpenMP/omp_alloc.h ----------------*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#ifndef FORTRAN_RUNTIME_OMP_ALLOC_H_ +#define FORTRAN_RUNTIME_OMP_ALLOC_H_ + +#include "flang/Runtime/descriptor-consts.h" +#include "flang/Runtime/entry-names.h" + +namespace Fortran::runtime::omp { + +extern "C" { + +/// Register the OpenMP target device allocator with the Fortran runtime's +/// allocator registry. Called once from the generated main() when +/// -fopenmp-default-allocate=target is active. The allocator uses +/// omp_target_alloc/omp_target_free to place Fortran ALLOCATABLE storage +/// on the current default device. The environment variable OMP_ALLOC +/// (default: "openmp") selects the allocator backend; OMP_ALLOC_DEBUG +/// enables diagnostic tracing to stderr. +void RTDECL(OpenMPRegisterAllocator)(); + +/// Set the allocator index on an allocatable descriptor so that subsequent +/// AllocatableAllocate calls route through the registered OpenMP allocator. +/// \p descriptor must be an unallocated ALLOCATABLE; \p pos is the allocator +/// registry slot (typically 1). No-op if the descriptor is already allocated +/// or is not allocatable. +void RTDECL(OpenMPAllocatableSetAllocIdx)(Descriptor &descriptor, int pos); + +} + +} // namespace Fortran::runtime::omp +#endif // FORTRAN_RUNTIME_OMP_ALLOC_H_ diff --git a/flang/include/flang/Runtime/OpenMP/omp_util.h b/flang/include/flang/Runtime/OpenMP/omp_util.h new file mode 100644 index 0000000000000..447954cdc1240 --- /dev/null +++ b/flang/include/flang/Runtime/OpenMP/omp_util.h @@ -0,0 +1,53 @@ +//===-- include/flang/Runtime/OpenMP/omp_util.h -----------------*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// + +#ifndef FORTRAN_RUNTIME_OMP_UTIL_H_ +#define FORTRAN_RUNTIME_OMP_UTIL_H_ + +#include <cstddef> + +namespace Fortran::runtime::omp { + +/// A thread-safe map from allocation pointer to device ID. +/// +/// Used to remember which OpenMP device each pointer was allocated on, +/// so that deallocation can target the correct device even if +/// omp_set_default_device() was called in between. +/// +/// Implemented as a dynamically-grown flat array with linear search and +/// a global lock, to avoid pulling in C++ runtime dependencies (e.g. +/// std::unordered_map). This is adequate for the expected allocation +/// counts in typical Fortran programs. +class PointerDeviceMap { +public: + /// Record that \p pointer was allocated on \p device. + void insert(void *pointer, int device); + + /// Remove the entry for \p pointer and return the device ID it was + /// allocated on. Returns -1 if \p pointer is not in the map. + int removeAndGet(void *pointer); + + /// Print all entries to stderr (for debugging). + void dump() const; + +private: + struct Entry { + void *pointer; + int device; + }; + + void grow(); + + Entry *entries_{nullptr}; + std::size_t count_{0}; + std::size_t capacity_{0}; +}; + +} // namespace Fortran::runtime::omp + +#endif // FORTRAN_RUNTIME_OMP_UTIL_H_ diff --git a/flang/include/flang/Support/Fortran-features.h b/flang/include/flang/Support/Fortran-features.h index 4ee956a0b4a4f..6f835bc82388f 100644 --- a/flang/include/flang/Support/Fortran-features.h +++ b/flang/include/flang/Support/Fortran-features.h @@ -55,6 +55,7 @@ ENUM_CLASS(LanguageFeature, BackslashEscapes, OldDebugLines, SavedLocalInSpecExpr, PrintNamelist, AssumedRankPassedToNonAssumedRank, IgnoreIrrelevantAttributes, Unsigned, ContiguousOkForSeqAssociation, ForwardRefExplicitTypeDummy, InaccessibleDeferredOverride, + OpenMPDefaultAllocator, CudaWarpMatchFunction, DoConcurrentOffload, TransferBOZ, Coarray, PointerPassObject, MultipleIdenticalDATA, DefaultStructConstructorNullPointer, AssumedRankIoItem, diff --git a/flang/lib/Frontend/CompilerInvocation.cpp b/flang/lib/Frontend/CompilerInvocation.cpp index b57bc4583be38..f14c990bbf0da 100644 --- a/flang/lib/Frontend/CompilerInvocation.cpp +++ b/flang/lib/Frontend/CompilerInvocation.cpp @@ -947,6 +947,19 @@ static bool parseFrontendArgs(FrontendOptions &opts, llvm::opt::ArgList &args, clang::options::OPT_fno_save_main_program, false)); + // -fopenmp-default-allocate={target,host} + if (const auto *arg = + args.getLastArg(clang::options::OPT_fopenmp_default_allocate_EQ)) { + llvm::StringRef val = arg->getValue(); + if (val == "target") { + opts.features.Enable( + Fortran::common::LanguageFeature::OpenMPDefaultAllocator); + } else if (val != "host") { + diags.Report(clang::diag::err_drv_invalid_value) + << arg->getAsString(args) << val; + } + } + if (args.hasArg(clang::options::OPT_falternative_parameter_statement)) { opts.features.Enable(Fortran::common::LanguageFeature::OldStyleParameter); } diff --git a/flang/lib/Lower/Allocatable.cpp b/flang/lib/Lower/Allocatable.cpp index f51342b27a19d..865bc5fd38f9f 100644 --- a/flang/lib/Lower/Allocatable.cpp +++ b/flang/lib/Lower/Allocatable.cpp @@ -33,8 +33,11 @@ #include "flang/Parser/parse-tree.h" #include "flang/Runtime/allocatable.h" #include "flang/Runtime/pointer.h" +#include "flang/Runtime/OpenMP/omp_alloc.h" #include "flang/Semantics/tools.h" #include "flang/Semantics/type.h" +#include "mlir/Dialect/OpenMP/OpenMPDialect.h" +#include "mlir/Dialect/OpenMP/OpenMPInterfaces.h" #include "flang/Support/Fortran-features.h" #include "llvm/Support/CommandLine.h" @@ -163,6 +166,50 @@ static void genRuntimeInitCharacter(fir::FirOpBuilder &builder, fir::CallOp::create(builder, loc, callee, convertedArgs); } +/// Return true if \p region is (transitively) nested inside an omp.target +/// region or inside a function marked as declare target. +static bool isRegionNestedInOmpTarget(mlir::Region ®ion) { + mlir::Operation *parentOp = region.getParentOp(); + while (parentOp) { + if (auto declareTargetOp = + llvm::dyn_cast<mlir::omp::DeclareTargetInterface>(parentOp)) { + if (declareTargetOp.isDeclareTarget()) + return true; + } + if (llvm::isa<mlir::omp::TargetOp>(parentOp)) + return true; + mlir::Region *parentRegion = parentOp->getParentRegion(); + if (!parentRegion) + break; + parentOp = parentRegion->getParentOp(); + } + return false; +} + +/// Emit a call to the runtime that records the allocator index (\p allocatorId) +/// on \p box so subsequent runtime allocations for \p box are serviced by the +/// OpenMP target allocator. Skipped inside device code, where allocations are +/// handled directly by the target runtime. +static void genOpenMPRuntimeDescriptorSetAllocIdx( + fir::FirOpBuilder &builder, mlir::Location loc, + const fir::MutableBoxValue &box, int allocatorId) { + if (isRegionNestedInOmpTarget(builder.getRegion())) + return; + auto *context = builder.getContext(); + mlir::Type descriptorTy = box.getAddr().getType(); + mlir::IntegerType posTy = builder.getI32Type(); + mlir::func::FuncOp callee = builder.createFunction( + loc, RTNAME_STRING(OpenMPAllocatableSetAllocIdx), + mlir::FunctionType::get(context, {descriptorTy, posTy}, {})); + llvm::SmallVector<mlir::Value> args{box.getAddr()}; + args.push_back( + builder.createIntegerConstant(loc, builder.getI32Type(), allocatorId)); + llvm::SmallVector<mlir::Value> operands; + for (auto [fst, snd] : llvm::zip(args, callee.getFunctionType().getInputs())) + operands.emplace_back(builder.createConvert(loc, snd, fst)); + fir::CallOp::create(builder, loc, callee, operands); +} + /// Generate a sequence of runtime calls to allocate memory. static mlir::Value genRuntimeAllocate(fir::FirOpBuilder &builder, mlir::Location loc, @@ -534,6 +581,9 @@ class AllocateStmtHelper { !alloc.type.IsPolymorphic() && !alloc.hasCoarraySpec() && !useAllocateRuntime && !box.isPointer() && !implicitManagedBacking; + const auto &langFeatures = converter.getFoldingContext().languageFeatures(); + bool isOpenMPAllocatorEnabled = langFeatures.IsEnabled( + Fortran::common::LanguageFeature::OpenMPDefaultAllocator); if (inlineAllocation && !alloc.hasCoarraySpec() && ((isCudaAllocate && isCudaDeviceContext) || !isCudaAllocate)) { @@ -569,6 +619,8 @@ class AllocateStmtHelper { alloc.getCoarraySpec(), errorManager.errMsgAddr, errorManager.hasStatSpec()); } else if (!isCudaAllocate) { + if (isOpenMPAllocatorEnabled) + genOpenMPRuntimeDescriptorSetAllocIdx(builder, loc, box, 1); stat = genRuntimeAllocate(builder, loc, box, errorManager); setPinnedToFalse(); } else { @@ -700,6 +752,9 @@ class AllocateStmtHelper { isCudaAllocate = propagateCUDAAttrsFromParent(alloc, cudaSymForAlloc); unsigned allocatorIdx = Fortran::lower::getAllocatorIdx(*cudaSymForAlloc); fir::ExtendedValue exv = isSource ? sourceExv : moldExv; + const auto &langFeatures = converter.getFoldingContext().languageFeatures(); + bool isOpenMPAllocatorEnabled = langFeatures.IsEnabled( + Fortran::common::LanguageFeature::OpenMPDefaultAllocator); bool sourceIsDevice = false; if (const Fortran::semantics::Symbol *sym{GetLastSymbol(sourceExpr)}) @@ -730,6 +785,8 @@ class AllocateStmtHelper { } else if (isCudaAllocate || sourceIsDevice) { stat = genCudaAllocate(builder, loc, box, errorManager, *cudaSymForAlloc); } else { + if (isOpenMPAllocatorEnabled) + genOpenMPRuntimeDescriptorSetAllocIdx(builder, loc, box, 1); if (isSource) stat = genRuntimeAllocateSource(builder, loc, box, exv, errorManager); else diff --git a/flang/lib/Lower/Bridge.cpp b/flang/lib/Lower/Bridge.cpp index bff6b51e50e18..93b7c44610eeb 100644 --- a/flang/lib/Lower/Bridge.cpp +++ b/flang/lib/Lower/Bridge.cpp @@ -620,6 +620,8 @@ class FirConverter : public Fortran::lower::AbstractConverter { Fortran::common::LanguageFeature::CUDA) && getFoldingContext().languageFeatures().IsEnabled( Fortran::common::LanguageFeature::CUDAInit)), + getFoldingContext().languageFeatures().IsEnabled( + Fortran::common::LanguageFeature::OpenMPDefaultAllocator), getFoldingContext().languageFeatures().IsEnabled( Fortran::common::LanguageFeature::Coarray), bridge.getLoweringOptions().getFPExceptionTraps()); diff --git a/flang/lib/Optimizer/Builder/Runtime/Main.cpp b/flang/lib/Optimizer/Builder/Runtime/Main.cpp index 2427088ec7f83..4a14331671e8a 100644 --- a/flang/lib/Optimizer/Builder/Runtime/Main.cpp +++ b/flang/lib/Optimizer/Builder/Runtime/Main.cpp @@ -24,7 +24,8 @@ using namespace Fortran::runtime; void fir::runtime::genMain( fir::FirOpBuilder &builder, mlir::Location loc, const std::vector<Fortran::lower::EnvironmentDefault> &defs, bool initCuda, - bool initCoarrayEnv, unsigned fpExceptionTraps) { + bool enableOpenMPAllocator, bool initCoarrayEnv, + unsigned fpExceptionTraps) { auto *context = builder.getContext(); auto argcTy = builder.getDefaultIntegerType(); auto ptrTy = mlir::LLVM::LLVMPointerType::get(context); @@ -72,6 +73,13 @@ void fir::runtime::genMain( if (initCoarrayEnv) mif::InitOp::create(builder, loc); + if (enableOpenMPAllocator) { + auto registerFn = + builder.createFunction(loc, RTNAME_STRING(OpenMPRegisterAllocator), + mlir::FunctionType::get(context, {}, {})); + fir::CallOp::create(builder, loc, registerFn); + } + if (fpExceptionTraps != 0) { auto i32Ty = builder.getI32Type(); auto enableFn = diff --git a/flang/lib/Support/Fortran-features.cpp b/flang/lib/Support/Fortran-features.cpp index 0af3ff61d18e1..cf2cd3c664cf7 100644 --- a/flang/lib/Support/Fortran-features.cpp +++ b/flang/lib/Support/Fortran-features.cpp @@ -156,6 +156,7 @@ LanguageFeatureControl::LanguageFeatureControl() { disable_.set(LanguageFeature::AssumedRankPassedToNonAssumedRank); disable_.set(LanguageFeature::Coarray); disable_.set(LanguageFeature::OpenAccDefaultNoneScalarsStrict); + disable_.set(LanguageFeature::OpenMPDefaultAllocator); // These warnings are enabled by default, but only because they used // to be unconditional. TODO: prune this list warnLanguage_.set(LanguageFeature::ExponentMatchingKindParam); diff --git a/flang/test/Driver/fopenmp-default-allocate.f90 b/flang/test/Driver/fopenmp-default-allocate.f90 new file mode 100644 index 0000000000000..9e637d2add4ec --- /dev/null +++ b/flang/test/Driver/fopenmp-default-allocate.f90 @@ -0,0 +1,28 @@ +! Check that the driver passes -fopenmp-default-allocate= through to fc1 +! and only adds -mmlir -use-alloc-runtime for target mode. + +! RUN: %flang -### -S -fopenmp-default-allocate=target %s -o - 2>&1 | FileCheck %s --check-prefix=TARGET +! RUN: %flang -### -S -fopenmp-default-allocate=host %s -o - 2>&1 | FileCheck %s --check-prefix=HOST + +! TARGET: warning: -fopenmp-default-allocate= is an experimental feature +! TARGET: "-fc1" +! TARGET-SAME: "-fopenmp-default-allocate=target" +! TARGET-SAME: "-mmlir" "-use-alloc-runtime" + +! HOST: warning: -fopenmp-default-allocate= is an experimental feature +! HOST: "-fc1" +! HOST-SAME: "-fopenmp-default-allocate=host" +! HOST-NOT: "-mmlir" +! HOST-NOT: "-use-alloc-runtime" + +! Check that invalid values are rejected at the driver level. +! RUN: not %flang -fopenmp-default-allocate=invalid -S %s 2>&1 | FileCheck %s --check-prefix=DRV-INVALID +! DRV-INVALID: error: invalid value 'invalid' in '-fopenmp-default-allocate=invalid' + +! Check that invalid values are also rejected at the frontend level. +! RUN: not %flang_fc1 -fopenmp-default-allocate=invalid -S %s 2>&1 | FileCheck %s --check-prefix=FC1-INVALID +! FC1-INVALID: error: invalid value 'invalid' in '-fopenmp-default-allocate=invalid' + +program fopenmp_default_allocate + ! do nothing +end program fopenmp_default_allocate \ No newline at end of file diff --git a/flang/test/Lower/AMDGPU/allocate_deallocate_omp_declare_target.f90 b/flang/test/Lower/AMDGPU/allocate_deallocate_omp_declare_target.f90 new file mode 100644 index 0000000000000..b3adac6e331d6 --- /dev/null +++ b/flang/test/Lower/AMDGPU/allocate_deallocate_omp_declare_target.f90 @@ -0,0 +1,24 @@ +! RUN: %flang -fopenmp-default-allocate=target -S -emit-llvm -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa --offload-arch=gfx90a -o - %s | FileCheck %s --check-prefix=CHECK-OMP +! RUN: %flang -fopenmp-default-allocate=target -S -emit-llvm -target amdgcn-- -o - %s | FileCheck %s --check-prefix=CHECK +!REQUIRES: AFAR +subroutine func_t_device() + !$omp declare target enter(func_t_device) device_type(nohost) + integer, ALLOCATABLE :: poly + +! CHECK-OMP-NOT: call void @_FortranAOpenMPAllocatableSetAllocIdx({{.*}}, i32 1) +! CHECK: call void @_FortranAOpenMPAllocatableSetAllocIdx({{.*}}, i32 1) +! CHECK-OMP: call i32 @_FortranAAllocatableAllocate +! CHECK: call i32 @_FortranAAllocatableAllocate + ALLOCATE(poly) + +! CHECK-OMP: call i32 @_FortranAAllocatableDeallocate +! CHECK: call i32 @_FortranAAllocatableDeallocate + DEALLOCATE(poly) +end subroutine func_t_device + +program main + implicit none + !$omp target + call func_t_device() + !$omp end target +end program diff --git a/flang/test/Lower/AMDGPU/allocate_deallocate_omp_declare_target_nested.f90 b/flang/test/Lower/AMDGPU/allocate_deallocate_omp_declare_target_nested.f90 new file mode 100644 index 0000000000000..d794cc6cc7cc5 --- /dev/null +++ b/flang/test/Lower/AMDGPU/allocate_deallocate_omp_declare_target_nested.f90 @@ -0,0 +1,25 @@ +! RUN: %flang -fopenmp-default-allocate=target -S -emit-llvm -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa --offload-arch=gfx90a -o - %s | FileCheck %s --check-prefix=CHECK-OMP +! RUN: %flang -fopenmp-default-allocate=target -S -emit-llvm -target amdgcn-- -o - %s | FileCheck %s --check-prefix=CHECK +!REQUIRES: AFAR +subroutine func_t_device() + !$omp declare target enter(func_t_device) device_type(nohost) + integer, ALLOCATABLE :: poly + do j=1,10 +! CHECK-OMP-NOT: call void @_FortranAOpenMPAllocatableSetAllocIdx({{.*}}, i32 1) +! CHECK: call void @_FortranAOpenMPAllocatableSetAllocIdx({{.*}}, i32 1) +! CHECK-OMP: call i32 @_FortranAAllocatableAllocate +! CHECK: call i32 @_FortranAAllocatableAllocate + ALLOCATE(poly) + +! CHECK-OMP: call i32 @_FortranAAllocatableDeallocate +! CHECK: call i32 @_FortranAAllocatableDeallocate + DEALLOCATE(poly) + end do +end subroutine func_t_device + +program main + implicit none + !$omp target + call func_t_device() + !$omp end target +end program diff --git a/flang/test/Lower/AMDGPU/allocate_deallocate_omp_target.f90 b/flang/test/Lower/AMDGPU/allocate_deallocate_omp_target.f90 new file mode 100644 index 0000000000000..f425e6b9d93ca --- /dev/null +++ b/flang/test/Lower/AMDGPU/allocate_deallocate_omp_target.f90 @@ -0,0 +1,24 @@ +! RUN: %flang -fopenmp-default-allocate=target -S -emit-llvm -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa --offload-arch=gfx90a -o - %s | FileCheck %s --check-prefix=CHECK-OMP +! RUN: %flang -fopenmp-default-allocate=target -S -emit-llvm -target amdgcn-- -o - %s | FileCheck %s --check-prefix=CHECK +!REQUIRES: AFAR +program main + implicit none + !$omp requires unified_shared_memory + REAL, DIMENSION(:), ALLOCATABLE :: poly + integer,parameter :: n = 10 + integer :: i,j + !$omp target teams distribute parallel do private(poly) + do j=1,n + +! CHECK-OMP-NOT: call void @_FortranAOpenMPAllocatableSetAllocIdx({{.*}}, i32 1) +! CHECK: call void @_FortranAOpenMPAllocatableSetAllocIdx({{.*}}, i32 1) +! CHECK-OMP: call i32 @_FortranAAllocatableAllocate +! CHECK: call i32 @_FortranAAllocatableAllocate + ALLOCATE(poly(1:3)) + poly = 2.0_8 +! CHECK-OMP: call i32 @_FortranAAllocatableDeallocate +! CHECK: call i32 @_FortranAAllocatableDeallocate + DEALLOCATE(poly) + enddo + !$omp end target teams distribute parallel do +end program diff --git a/flang/test/Lower/AMDGPU/allocate_runtime_alloc_idx.f90 b/flang/test/Lower/AMDGPU/allocate_runtime_alloc_idx.f90 new file mode 100644 index 0000000000000..6c215ec6c97e7 --- /dev/null +++ b/flang/test/Lower/AMDGPU/allocate_runtime_alloc_idx.f90 @@ -0,0 +1,20 @@ +! RUN: %flang -fopenmp-default-allocate=target -S -emit-llvm --offload-targets=amdgcn-amd-amdhsa -o - %s | FileCheck %s + +subroutine allocate_deallocate() + real, allocatable :: x +! CHECK: call void @_FortranAOpenMPAllocatableSetAllocIdx({{.*}}, i32 1) +! CHECK: call i32 @_FortranAAllocatableAllocate + allocate(x) + +! CHECK: call i32 @_FortranAAllocatableDeallocate + deallocate(x) +end subroutine + +subroutine test_allocatable_scalar(a) + real, save, allocatable :: x1, x2 + real :: a + +! CHECK: call void @_FortranAOpenMPAllocatableSetAllocIdx({{.*}}, i32 1) +! CHECK: call i32 @_FortranAAllocatableAllocateSource + allocate(x1, x2, source = a) +end diff --git a/flang/test/Lower/AMDGPU/allocate_runtime_alloc_idx_host.f90 b/flang/test/Lower/AMDGPU/allocate_runtime_alloc_idx_host.f90 new file mode 100644 index 0000000000000..2ebccd537d784 --- /dev/null +++ b/flang/test/Lower/AMDGPU/allocate_runtime_alloc_idx_host.f90 @@ -0,0 +1,17 @@ +! RUN: %flang -fopenmp-default-allocate=host -S -emit-llvm --offload-targets=amdgcn-amd-amdhsa -o - %s 2>&1 | FileCheck %s + +! Verify that host mode does not insert OpenMPAllocatableSetAllocIdx calls. + +! CHECK-NOT: call void @_FortranAOpenMPAllocatableSetAllocIdx + +subroutine allocate_deallocate() + real, allocatable :: x + allocate(x) + deallocate(x) +end subroutine + +subroutine test_allocatable_scalar(a) + real, save, allocatable :: x1, x2 + real :: a + allocate(x1, x2, source = a) +end \ No newline at end of file diff --git a/flang/test/Lower/OpenMP/omp_alloc_init.f90 b/flang/test/Lower/OpenMP/omp_alloc_init.f90 new file mode 100644 index 0000000000000..2375a9d668c39 --- /dev/null +++ b/flang/test/Lower/OpenMP/omp_alloc_init.f90 @@ -0,0 +1,5 @@ +!RUN: %flang_fc1 -fopenmp -fopenmp-default-allocate=target -emit-fir %s -o - | FileCheck %s + +program omp_alloc_init + !CHECK: fir.call @_FortranAOpenMPRegisterAllocator() +end program omp_alloc_init diff --git a/flang/test/Lower/OpenMP/omp_alloc_init_host.f90 b/flang/test/Lower/OpenMP/omp_alloc_init_host.f90 new file mode 100644 index 0000000000000..8fec15d83932c --- /dev/null +++ b/flang/test/Lower/OpenMP/omp_alloc_init_host.f90 @@ -0,0 +1,5 @@ +!RUN: %flang_fc1 -fopenmp -fopenmp-default-allocate=host -emit-fir %s -o - | FileCheck %s + +program omp_alloc_init_host + !CHECK-NOT: fir.call @_FortranAOpenMPRegisterAllocator() +end program omp_alloc_init_host >From 0905dc68a7e3752a49c74e7e7e8b9745df811cbe Mon Sep 17 00:00:00 2001 From: Michael Klemm <[email protected]> Date: Tue, 18 Aug 2026 09:46:07 +0200 Subject: [PATCH 2/2] Remove obsolete comments about Umpire --- flang-rt/lib/openmp/CMakeLists.txt | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/flang-rt/lib/openmp/CMakeLists.txt b/flang-rt/lib/openmp/CMakeLists.txt index ced202478d253..8f580fafee3e7 100644 --- a/flang-rt/lib/openmp/CMakeLists.txt +++ b/flang-rt/lib/openmp/CMakeLists.txt @@ -6,22 +6,9 @@ # #===------------------------------------------------------------------------===# -# Check that Umpire exits in the directory given at CMake -# TODO: this was disabled to get to an easier build procedure for now -#message(STATUS "Using Umpire in directory ${FLANG_RT_UMPIRE_DIR}") -#set(umpire_DIR ${FLANG_RT_UMPIRE_DIR}) -#find_package(umpire REQUIRED PATHS ${FLANG_RT_UMPIRE_DIR}/lib/cmake/umpire) - add_flangrt_library(flang_rt.openmp STATIC SHARED omp_alloc.cpp omp_util.cpp INSTALL_WITH_TOOLCHAIN ) -#if (TARGET flang_rt.openmp.static) -# target_include_directories(flang_rt.openmp.static PRIVATE ${FLANG_RT_UMPIRE_DIR}/include) -#endif() -# -#if (TARGET flang_rt.openmp.shared) -# target_include_directories(flang_rt.openmp.shared PRIVATE ${FLANG_RT_UMPIRE_DIR}/include) -#endif() _______________________________________________ cfe-commits mailing list [email protected] https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
