[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 created
https://github.com/llvm/llvm-project/pull/99283
This patch is the middle end implementation for the coroutine HALO improvement
project published on discourse:
https://discourse.llvm.org/t/language-extension-for-better-more-deterministic-halo-for-c-coroutines/80044/7
Previously CoroElide depends on inlining, and its analysis does not work very
well with code generated by the C++ frontend due the existence of many
customization points. There has been issue reported to upstream how ineffective
the original CoroElide was in real world applications.
For C++ users, this set of patches aim to fix this problem by providing library
authors and users deterministic HALO behaviour for some well-behaved coroutine
`Task` types. The stack begins with a library side attribute on the `Task`
class that guarantees no unstructured concurrency when coroutines are awaited
directly with `co_await`ed as a prvalue. This attribute on Task types gives us
lifetime guarantees and makes C++ FE capable to telling the ME which coroutine
calls are elidable. We convey such information from FE through the attribute
`coro_must_elide`.
This patch modifies CoroSplit to create a variant of the coroutine ramp
function that 1) does not use heap allocated frame, instead take an additional
parameter as the pointer to the frame. Such parameter is attributed with
`dereferenceble` and `align` to convey size and align requirements for the
frame. 2) always stores cleanup instead of destroy address for `coro.destroy()`
actions.
In a later patch, we will have a new pass that runs right after CoroSplit to
find usages of the callee coroutine attributed `coro_must_elide` in presplit
coroutine callers, allocates the frame on its "stack", transform those usages
to call the `noalloc` ramp function variant.
(note I put quotes on the word "stack" here, because for presplit coroutine,
any alloca will be spilled into the frame when it's being split)
The C++ Frontend attribute implementation that works with this change can be
found at https://github.com/llvm/llvm-project/pull/99282
>From 19bd17bc91ebc3cd03d4a8ea85ec2b1b4cb3f024 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] Implement LLVM bits
---
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 123 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 2 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 7 +
9 files changed, 142 insertions(+), 33 deletions(-)
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index 5716fd0ea4ab9..d91cccd99a703 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 9e4da5f8ca961..9c0db4f29056e 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1179,6 +1180,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1194,10 +1203,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From 7c14a807d0140c638b056a59a9c8c3f9fcf90e57 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] Implement noalloc in CoroSplit
---
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 123 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 2 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 7 +
9 files changed, 142 insertions(+), 33 deletions(-)
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index 5716fd0ea4ab9..d91cccd99a703 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 9e4da5f8ca961..9c0db4f29056e 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1179,6 +1180,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1194,10 +1203,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1455,6 +1462,64 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
+SmallVectorImpl &Clones) {
+auto *OrigFnTy = F.getFunctionType();
+auto OldParams = OrigFnTy->params();
+
+SmallVector NewParams;
+NewParams.reserve(OldParams.size() + 1);
+for (Type *T : OldParams) {
+ NewParams.push_back(T);
+}
+NewParams.push_back(PointerType::getUnqual(Shape.FrameTy));
+
+auto *NewFnTy = FunctionType::get(OrigFnTy->getReturnType(), NewParams,
+ OrigFnTy->isVarArg());
+Function *NoAllocF =
+Function::Create(NewFnTy, F.getLinkage(), F.getName() + ".noalloc");
+ValueToValueMapTy VMap;
+unsigned int Idx = 0;
+for (const auto &I : F.args()) {
+ VMap[&I] = NoAllocF->getArg(Idx++);
+}
+SmallVector Returns;
+CloneFunctionInto(NoAllocF, &F, VMap,
+ CloneFunctionChangeType::LocalChangesOnly, Returns);
+
+if (Shape.CoroBegin) {
+ auto *NewCoroBegin =
+ cast_if_present(VMap[Shape.CoroBegin]);
+ auto *NewCoroId = cast(NewCoroBegin->getId());
+ coro::replaceCoroFree(NewCoroId, /*Elide=*/true);
+ coro::suppressCoroAllocs(NewCoroId);
+ NewCoroBegin->replaceAllUsesWith(NoAllocF->getArg(Idx));
+ NewCoroBegin->eraseFromParent();
+}
+
+Module *M = F.getParent();
+M->getFunctionList().insert(M->end(), NoAllocF);
+
+removeUnreachableBlocks(*NoAllocF);
+auto NewAttrs = NoAllocF->getAttributes();
+// We just
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 08760faf31683e8372322305846465a810198ce3 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] add CoroAnnotationElidePass
---
.../Coroutines/CoroAnnotationElide.h | 34 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 3 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 135 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Other/new-pm-thinlto-prelink-defaults.ll | 1 +
.../new-pm-thinlto-prelink-pgo-defaults.ll| 1 +
...w-pm-thinlto-prelink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 ++
14 files changed, 257 insertions(+), 1 deletion(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 0..dae1cc0c689a3
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,34 @@
+//===- CoroAnnotationElide.h - Optimizing a coro_must_elide call
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_must_elide" to call the `.noalloc` variant of coroutine instead.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 929690c2c74d6..6b7d5089b1727 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -134,6 +134,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 4fd5ee1946bb7..2d5a6e6861f23 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -32,6 +32,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -968,8 +969,8 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
// it's been modified since.
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
-
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+ MainCGPipeline.addPass(CoroAnnotationElidePass());
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
diff --git a/llvm/lib/Passes/PassRegistry.def b/llvm/lib/Passes/PassRegistry.def
index 3b92823cd283b..f3d77f8def4a6 100644
--- a/llvm/lib/Passes/PassRegistry.def
+++ b/llvm/lib/Passes/PassRegistry.def
@@ -234,6 +234,7 @@ CGSCC_PASS("attributor-light-cgscc",
AttributorLightCGSCCPass())
CGSCC_PASS("invalidate", InvalidateAllAnalysesPass())
CGSCC_PASS("no-op-cgscc", NoOpCG
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -1455,6 +1462,64 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
+SmallVectorImpl &Clones) {
+auto *OrigFnTy = F.getFunctionType();
+auto OldParams = OrigFnTy->params();
+
+SmallVector NewParams;
+NewParams.reserve(OldParams.size() + 1);
+for (Type *T : OldParams) {
+ NewParams.push_back(T);
+}
+NewParams.push_back(PointerType::getUnqual(Shape.FrameTy));
+
+auto *NewFnTy = FunctionType::get(OrigFnTy->getReturnType(), NewParams,
+ OrigFnTy->isVarArg());
+Function *NoAllocF =
+Function::Create(NewFnTy, F.getLinkage(), F.getName() + ".noalloc");
yuxuanchen1997 wrote:
Does ThinLTO count as Cross-TU?
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -1967,22 +2047,13 @@ splitCoroutine(Function &F, SmallVectorImpl
&Clones,
for (DbgVariableRecord *DVR : DbgVariableRecords)
coro::salvageDebugInfo(ArgToAllocaMap, *DVR, Shape.OptimizeFrame,
false /*UseEntryValue*/);
- return Shape;
-}
-/// Remove calls to llvm.coro.end in the original function.
-static void removeCoroEndsFromRampFunction(const coro::Shape &Shape) {
- if (Shape.ABI != coro::ABI::Switch) {
-for (auto *End : Shape.CoroEnds) {
- replaceCoroEnd(End, Shape, Shape.FramePtr, /*in resume*/ false, nullptr);
-}
- } else {
-for (llvm::AnyCoroEndInst *End : Shape.CoroEnds) {
- auto &Context = End->getContext();
- End->replaceAllUsesWith(ConstantInt::getFalse(Context));
- End->eraseFromParent();
-}
+ removeCoroEndsFromRampFunction(Shape);
+
+ if (!isNoSuspendCoroutine && Shape.ABI == coro::ABI::Switch) {
yuxuanchen1997 wrote:
We currently don't propagate the attribute from the CXXRecordDecl to the LLVM
type. We can potentially add another attribute at the coroutine function level
if that sounds good to you.
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From d42ce99fac00de6d35e423490f2603796a10 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] Implement noalloc in CoroSplit
---
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 123 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 2 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 7 +
9 files changed, 142 insertions(+), 33 deletions(-)
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index 5716fd0ea4ab9..d91cccd99a703 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 9e4da5f8ca961..9c0db4f29056e 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1179,6 +1180,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1194,10 +1203,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1455,6 +1462,64 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
+SmallVectorImpl &Clones) {
+auto *OrigFnTy = F.getFunctionType();
+auto OldParams = OrigFnTy->params();
+
+SmallVector NewParams;
+NewParams.reserve(OldParams.size() + 1);
+for (Type *T : OldParams) {
+ NewParams.push_back(T);
+}
+NewParams.push_back(PointerType::getUnqual(Shape.FrameTy));
+
+auto *NewFnTy = FunctionType::get(OrigFnTy->getReturnType(), NewParams,
+ OrigFnTy->isVarArg());
+Function *NoAllocF =
+Function::Create(NewFnTy, F.getLinkage(), F.getName() + ".noalloc");
+ValueToValueMapTy VMap;
+unsigned int Idx = 0;
+for (const auto &I : F.args()) {
+ VMap[&I] = NoAllocF->getArg(Idx++);
+}
+SmallVector Returns;
+CloneFunctionInto(NoAllocF, &F, VMap,
+ CloneFunctionChangeType::LocalChangesOnly, Returns);
+
+if (Shape.CoroBegin) {
+ auto *NewCoroBegin =
+ cast_if_present(VMap[Shape.CoroBegin]);
+ auto *NewCoroId = cast(NewCoroBegin->getId());
+ coro::replaceCoroFree(NewCoroId, /*Elide=*/true);
+ coro::suppressCoroAllocs(NewCoroId);
+ NewCoroBegin->replaceAllUsesWith(NoAllocF->getArg(Idx));
+ NewCoroBegin->eraseFromParent();
+}
+
+Module *M = F.getParent();
+M->getFunctionList().insert(M->end(), NoAllocF);
+
+removeUnreachableBlocks(*NoAllocF);
+auto NewAttrs = NoAllocF->getAttributes();
+// We just
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 48c436e668a6b89aef09d300f3260e35b6960dfc Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] add CoroAnnotationElidePass
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: https://phabricator.intern.facebook.com/D60250514
---
.../Coroutines/CoroAnnotationElide.h | 34 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 3 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 136 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Other/new-pm-thinlto-prelink-defaults.ll | 1 +
.../new-pm-thinlto-prelink-pgo-defaults.ll| 1 +
...w-pm-thinlto-prelink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 ++
14 files changed, 258 insertions(+), 1 deletion(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 0..dae1cc0c689a3
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,34 @@
+//===- CoroAnnotationElide.h - Optimizing a coro_must_elide call
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_must_elide" to call the `.noalloc` variant of coroutine instead.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 5dbb1e2f49871..8349ea728520f 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -135,6 +135,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 757b20dcd6693..9d56bbd69ed52 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -32,6 +32,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -978,8 +979,8 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
// it's been modified since.
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
-
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+ MainCGPipeline.addPass(CoroAnnotationElidePass());
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
diff --git a/llvm/lib/Passes/PassRegistry.def b/llvm/lib/Passes/PassRegistry.def
index 3b92823cd283b..f3d77f8def4a6 100644
--- a/llvm/lib/Passes/PassRegistry.def
+++ b/llvm/lib/Passes/PassRegistry.def
@@ -234,6 +234,7 @@ CGSCC_PASS("attrib
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
@@ -0,0 +1,135 @@
+//===- CoroSplit.cpp - Converts a coroutine into a state machine
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+
+//===--===//
+
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
+
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Instruction.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/Transforms/Utils/CallGraphUpdater.h"
+
+#include
+
+using namespace llvm;
+
+#define DEBUG_TYPE "coro-annotation-elide"
+
+#define CORO_MUST_ELIDE_ANNOTATION "coro_must_elide"
+
+static Instruction *getFirstNonAllocaInTheEntryBlock(Function *F) {
+ for (Instruction &I : F->getEntryBlock())
+if (!isa(&I))
+ return &I;
+ llvm_unreachable("no terminator in the entry block");
+}
+
+static Value *allocateFrameInCaller(Function *Caller, uint64_t FrameSize,
+Align FrameAlign) {
+ LLVMContext &C = Caller->getContext();
+ BasicBlock::iterator InsertPt =
+ getFirstNonAllocaInTheEntryBlock(Caller)->getIterator();
+ const DataLayout &DL = Caller->getDataLayout();
+ auto FrameTy = ArrayType::get(Type::getInt8Ty(C), FrameSize);
+ auto *Frame = new AllocaInst(FrameTy, DL.getAllocaAddrSpace(), "", InsertPt);
+ Frame->setAlignment(FrameAlign);
+ return new BitCastInst(Frame, PointerType::getUnqual(C), "vFrame", InsertPt);
yuxuanchen1997 wrote:
This is never done for the legacy CoroElide. Inspecting the elided frame is no
different from inspecting a heap allocated frame from the debugger.
https://github.com/llvm/llvm-project/pull/99285
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [clang] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -1967,22 +2047,13 @@ splitCoroutine(Function &F, SmallVectorImpl
&Clones,
for (DbgVariableRecord *DVR : DbgVariableRecords)
coro::salvageDebugInfo(ArgToAllocaMap, *DVR, Shape.OptimizeFrame,
false /*UseEntryValue*/);
- return Shape;
-}
-/// Remove calls to llvm.coro.end in the original function.
-static void removeCoroEndsFromRampFunction(const coro::Shape &Shape) {
- if (Shape.ABI != coro::ABI::Switch) {
-for (auto *End : Shape.CoroEnds) {
- replaceCoroEnd(End, Shape, Shape.FramePtr, /*in resume*/ false, nullptr);
-}
- } else {
-for (llvm::AnyCoroEndInst *End : Shape.CoroEnds) {
- auto &Context = End->getContext();
- End->replaceAllUsesWith(ConstantInt::getFalse(Context));
- End->eraseFromParent();
-}
+ removeCoroEndsFromRampFunction(Shape);
+
+ if (!isNoSuspendCoroutine && Shape.ABI == coro::ABI::Switch) {
yuxuanchen1997 wrote:
> Yes, I would be in favor of adding a second attribute. What do you & others
> think? CC @ChuanqiXu9
I can put up another PR for that. Wdyt?
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From f1b66d6376ad890119449b93e20e9cdc36c770a6 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] Implement noalloc in CoroSplit
---
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 121 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 2 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 7 +
9 files changed, 140 insertions(+), 33 deletions(-)
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index 5716fd0ea4ab9..d91cccd99a703 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 9e4da5f8ca961..f78c03e9687c9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1179,6 +1180,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1194,10 +1203,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1455,6 +1462,62 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
+SmallVectorImpl &Clones) {
+auto *OrigFnTy = F.getFunctionType();
+auto OldParams = OrigFnTy->params();
+
+SmallVector NewParams;
+NewParams.reserve(OldParams.size() + 1);
+NewParams.append(OldParams.begin(), OldParams.end());
+NewParams.push_back(PointerType::getUnqual(Shape.FrameTy));
+
+auto *NewFnTy = FunctionType::get(OrigFnTy->getReturnType(), NewParams,
+ OrigFnTy->isVarArg());
+Function *NoAllocF =
+Function::Create(NewFnTy, F.getLinkage(), F.getName() + ".noalloc");
+ValueToValueMapTy VMap;
+unsigned int Idx = 0;
+for (const auto &I : F.args()) {
+ VMap[&I] = NoAllocF->getArg(Idx++);
+}
+SmallVector Returns;
+CloneFunctionInto(NoAllocF, &F, VMap,
+ CloneFunctionChangeType::LocalChangesOnly, Returns);
+
+if (Shape.CoroBegin) {
+ auto *NewCoroBegin =
+ cast_if_present(VMap[Shape.CoroBegin]);
+ auto *NewCoroId = cast(NewCoroBegin->getId());
+ coro::replaceCoroFree(NewCoroId, /*Elide=*/true);
+ coro::suppressCoroAllocs(NewCoroId);
+ NewCoroBegin->replaceAllUsesWith(NoAllocF->getArg(Idx));
+ NewCoroBegin->eraseFromParent();
+}
+
+Module *M = F.getParent();
+M->getFunctionList().insert(M->end(), NoAllocF);
+
+removeUnreachableBlocks(*NoAllocF);
+auto NewAttrs = NoAllocF->getAttributes();
+// We just appended the
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 3421b3f7ba207c864d8e71ad9adf5bebc91f3f2d Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] add CoroAnnotationElidePass
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: https://phabricator.intern.facebook.com/D60250514
---
.../Coroutines/CoroAnnotationElide.h | 34 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 136 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 ++
11 files changed, 261 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 0..dae1cc0c689a3
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,34 @@
+//===- CoroAnnotationElide.h - Optimizing a coro_must_elide call
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_must_elide" to call the `.noalloc` variant of coroutine instead.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 5dbb1e2f49871..8349ea728520f 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -135,6 +135,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index a6118726945e8..eb26ecdb9c4a7 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -32,6 +32,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -979,8 +980,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1022,9 +1025,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MPM.addPass(createM
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From 2c3902bbcc80bbd546b3a47c7d20a0fcf0c2c127 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] Implement noalloc in CoroSplit
---
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 121 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 2 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 7 +
9 files changed, 140 insertions(+), 33 deletions(-)
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index 5716fd0ea4ab9..d91cccd99a703 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 9e4da5f8ca961..f78c03e9687c9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1179,6 +1180,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1194,10 +1203,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1455,6 +1462,62 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
+SmallVectorImpl &Clones) {
+auto *OrigFnTy = F.getFunctionType();
+auto OldParams = OrigFnTy->params();
+
+SmallVector NewParams;
+NewParams.reserve(OldParams.size() + 1);
+NewParams.append(OldParams.begin(), OldParams.end());
+NewParams.push_back(PointerType::getUnqual(Shape.FrameTy));
+
+auto *NewFnTy = FunctionType::get(OrigFnTy->getReturnType(), NewParams,
+ OrigFnTy->isVarArg());
+Function *NoAllocF =
+Function::Create(NewFnTy, F.getLinkage(), F.getName() + ".noalloc");
+ValueToValueMapTy VMap;
+unsigned int Idx = 0;
+for (const auto &I : F.args()) {
+ VMap[&I] = NoAllocF->getArg(Idx++);
+}
+SmallVector Returns;
+CloneFunctionInto(NoAllocF, &F, VMap,
+ CloneFunctionChangeType::LocalChangesOnly, Returns);
+
+if (Shape.CoroBegin) {
+ auto *NewCoroBegin =
+ cast_if_present(VMap[Shape.CoroBegin]);
+ auto *NewCoroId = cast(NewCoroBegin->getId());
+ coro::replaceCoroFree(NewCoroId, /*Elide=*/true);
+ coro::suppressCoroAllocs(NewCoroId);
+ NewCoroBegin->replaceAllUsesWith(NoAllocF->getArg(Idx));
+ NewCoroBegin->eraseFromParent();
+}
+
+Module *M = F.getParent();
+M->getFunctionList().insert(M->end(), NoAllocF);
+
+removeUnreachableBlocks(*NoAllocF);
+auto NewAttrs = NoAllocF->getAttributes();
+// We just appended the
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 828463c9951daff698db417d1f2e2b8fe24d6243 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] add CoroAnnotationElidePass
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: https://phabricator.intern.facebook.com/D60250514
---
.../Coroutines/CoroAnnotationElide.h | 34 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 136 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 ++
11 files changed, 261 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 0..dae1cc0c689a3
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,34 @@
+//===- CoroAnnotationElide.h - Optimizing a coro_must_elide call
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_must_elide" to call the `.noalloc` variant of coroutine instead.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 5dbb1e2f49871..8349ea728520f 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -135,6 +135,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index a6118726945e8..eb26ecdb9c4a7 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -32,6 +32,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -979,8 +980,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1022,9 +1025,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MPM.addPass(createM
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From f8bb0f44f026540c7ae620e6b5bb7583e1078c67 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] Implement noalloc in CoroSplit
---
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 121 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 2 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 7 +
9 files changed, 140 insertions(+), 33 deletions(-)
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index 5716fd0ea4ab9..d91cccd99a703 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 9e4da5f8ca961..f78c03e9687c9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1179,6 +1180,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1194,10 +1203,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1455,6 +1462,62 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
+SmallVectorImpl &Clones) {
+auto *OrigFnTy = F.getFunctionType();
+auto OldParams = OrigFnTy->params();
+
+SmallVector NewParams;
+NewParams.reserve(OldParams.size() + 1);
+NewParams.append(OldParams.begin(), OldParams.end());
+NewParams.push_back(PointerType::getUnqual(Shape.FrameTy));
+
+auto *NewFnTy = FunctionType::get(OrigFnTy->getReturnType(), NewParams,
+ OrigFnTy->isVarArg());
+Function *NoAllocF =
+Function::Create(NewFnTy, F.getLinkage(), F.getName() + ".noalloc");
+ValueToValueMapTy VMap;
+unsigned int Idx = 0;
+for (const auto &I : F.args()) {
+ VMap[&I] = NoAllocF->getArg(Idx++);
+}
+SmallVector Returns;
+CloneFunctionInto(NoAllocF, &F, VMap,
+ CloneFunctionChangeType::LocalChangesOnly, Returns);
+
+if (Shape.CoroBegin) {
+ auto *NewCoroBegin =
+ cast_if_present(VMap[Shape.CoroBegin]);
+ auto *NewCoroId = cast(NewCoroBegin->getId());
+ coro::replaceCoroFree(NewCoroId, /*Elide=*/true);
+ coro::suppressCoroAllocs(NewCoroId);
+ NewCoroBegin->replaceAllUsesWith(NoAllocF->getArg(Idx));
+ NewCoroBegin->eraseFromParent();
+}
+
+Module *M = F.getParent();
+M->getFunctionList().insert(M->end(), NoAllocF);
+
+removeUnreachableBlocks(*NoAllocF);
+auto NewAttrs = NoAllocF->getAttributes();
+// We just appended the
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From d2fac21c4e9e5302fa2bee5365ab8be01dd024ea Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] add CoroAnnotationElidePass
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: https://phabricator.intern.facebook.com/D60250514
---
.../Coroutines/CoroAnnotationElide.h | 34 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 136 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 ++
11 files changed, 261 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 0..dae1cc0c689a3
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,34 @@
+//===- CoroAnnotationElide.h - Optimizing a coro_must_elide call
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_must_elide" to call the `.noalloc` variant of coroutine instead.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 5dbb1e2f49871..8349ea728520f 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -135,6 +135,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index a6118726945e8..eb26ecdb9c4a7 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -32,6 +32,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -979,8 +980,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1022,9 +1025,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MPM.addPass(createM
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -1967,22 +2047,13 @@ splitCoroutine(Function &F, SmallVectorImpl
&Clones,
for (DbgVariableRecord *DVR : DbgVariableRecords)
coro::salvageDebugInfo(ArgToAllocaMap, *DVR, Shape.OptimizeFrame,
false /*UseEntryValue*/);
- return Shape;
-}
-/// Remove calls to llvm.coro.end in the original function.
-static void removeCoroEndsFromRampFunction(const coro::Shape &Shape) {
- if (Shape.ABI != coro::ABI::Switch) {
-for (auto *End : Shape.CoroEnds) {
- replaceCoroEnd(End, Shape, Shape.FramePtr, /*in resume*/ false, nullptr);
-}
- } else {
-for (llvm::AnyCoroEndInst *End : Shape.CoroEnds) {
- auto &Context = End->getContext();
- End->replaceAllUsesWith(ConstantInt::getFalse(Context));
- End->eraseFromParent();
-}
+ removeCoroEndsFromRampFunction(Shape);
+
+ if (!isNoSuspendCoroutine && Shape.ABI == coro::ABI::Switch) {
yuxuanchen1997 wrote:
For sure!
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From dcd98a941d3edb3e5b96d7feeca50f99ea8309d2 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] Implement noalloc in CoroSplit
---
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 121 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 2 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 7 +
9 files changed, 140 insertions(+), 33 deletions(-)
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index 5716fd0ea4ab9..d91cccd99a703 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 9e4da5f8ca961..f78c03e9687c9 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1179,6 +1180,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1194,10 +1203,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1455,6 +1462,62 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
+SmallVectorImpl &Clones) {
+auto *OrigFnTy = F.getFunctionType();
+auto OldParams = OrigFnTy->params();
+
+SmallVector NewParams;
+NewParams.reserve(OldParams.size() + 1);
+NewParams.append(OldParams.begin(), OldParams.end());
+NewParams.push_back(PointerType::getUnqual(Shape.FrameTy));
+
+auto *NewFnTy = FunctionType::get(OrigFnTy->getReturnType(), NewParams,
+ OrigFnTy->isVarArg());
+Function *NoAllocF =
+Function::Create(NewFnTy, F.getLinkage(), F.getName() + ".noalloc");
+ValueToValueMapTy VMap;
+unsigned int Idx = 0;
+for (const auto &I : F.args()) {
+ VMap[&I] = NoAllocF->getArg(Idx++);
+}
+SmallVector Returns;
+CloneFunctionInto(NoAllocF, &F, VMap,
+ CloneFunctionChangeType::LocalChangesOnly, Returns);
+
+if (Shape.CoroBegin) {
+ auto *NewCoroBegin =
+ cast_if_present(VMap[Shape.CoroBegin]);
+ auto *NewCoroId = cast(NewCoroBegin->getId());
+ coro::replaceCoroFree(NewCoroId, /*Elide=*/true);
+ coro::suppressCoroAllocs(NewCoroId);
+ NewCoroBegin->replaceAllUsesWith(NoAllocF->getArg(Idx));
+ NewCoroBegin->eraseFromParent();
+}
+
+Module *M = F.getParent();
+M->getFunctionList().insert(M->end(), NoAllocF);
+
+removeUnreachableBlocks(*NoAllocF);
+auto NewAttrs = NoAllocF->getAttributes();
+// We just appended the
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 30f8875990935c24b7a3e3111a46d05bc9b902ab Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] add CoroAnnotationElidePass
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: https://phabricator.intern.facebook.com/D60250514
---
.../Coroutines/CoroAnnotationElide.h | 34 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 136 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 ++
11 files changed, 261 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 0..dae1cc0c689a3
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,34 @@
+//===- CoroAnnotationElide.h - Optimizing a coro_must_elide call
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_must_elide" to call the `.noalloc` variant of coroutine instead.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 5dbb1e2f49871..8349ea728520f 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -135,6 +135,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index a6118726945e8..eb26ecdb9c4a7 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -32,6 +32,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -979,8 +980,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1022,9 +1025,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MPM.addPass(createM
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -1455,6 +1462,62 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
+SmallVectorImpl &Clones) {
+auto *OrigFnTy = F.getFunctionType();
yuxuanchen1997 wrote:
Didn't get that?
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -1455,6 +1462,64 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
+SmallVectorImpl &Clones) {
+auto *OrigFnTy = F.getFunctionType();
+auto OldParams = OrigFnTy->params();
+
+SmallVector NewParams;
+NewParams.reserve(OldParams.size() + 1);
+for (Type *T : OldParams) {
+ NewParams.push_back(T);
+}
+NewParams.push_back(PointerType::getUnqual(Shape.FrameTy));
+
+auto *NewFnTy = FunctionType::get(OrigFnTy->getReturnType(), NewParams,
+ OrigFnTy->isVarArg());
+Function *NoAllocF =
+Function::Create(NewFnTy, F.getLinkage(), F.getName() + ".noalloc");
yuxuanchen1997 wrote:
This pass is now configured to not run at all during LTO pre-link. So internal
linkage SGTM.
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -5,7 +5,7 @@
define nonnull ptr @f(i32 %n) presplitcoroutine {
; CHECK-LABEL: @f(
; CHECK-NEXT: entry:
-; CHECK-NEXT:[[ID:%.*]] = call token @llvm.coro.id(i32 0, ptr null, ptr
null, ptr @f.resumers)
+; CHECK-NEXT:[[ID:%.*]] = call token @llvm.coro.id(i32 0, ptr null, ptr
null, ptr @{{.*}})
yuxuanchen1997 wrote:
Ah, the `noalloc` variant was appended to `f.resumers`, causing the name to be
changed to `f.resumers.1`.
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -5,7 +5,7 @@
define nonnull ptr @f(i32 %n) presplitcoroutine {
; CHECK-LABEL: @f(
; CHECK-NEXT: entry:
-; CHECK-NEXT:[[ID:%.*]] = call token @llvm.coro.id(i32 0, ptr null, ptr
null, ptr @f.resumers)
+; CHECK-NEXT:[[ID:%.*]] = call token @llvm.coro.id(i32 0, ptr null, ptr
null, ptr @{{.*}})
yuxuanchen1997 wrote:
Other than the legacy `CoroElide`, the main purpose of this variable is to keep
connection on the graph between the original copy and the `.resume`,
`.destroy`, `.cleanup`, `.noalloc` splits. I would suggest just changing the
name to read "f.splits" instead of `f.resumers`. It's a requirement for CGSCC
call graph updating.
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 closed https://github.com/llvm/llvm-project/pull/99283 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 reopened https://github.com/llvm/llvm-project/pull/99283 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From d3b68e89d672f5eef0f6cca011787f18ffcc70d6 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] add CoroAnnotationElidePass
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: https://phabricator.intern.facebook.com/D60250514
---
.../Coroutines/CoroAnnotationElide.h | 34 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 136 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 ++
11 files changed, 261 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 0..dae1cc0c689a3
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,34 @@
+//===- CoroAnnotationElide.h - Optimizing a coro_must_elide call
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_must_elide" to call the `.noalloc` variant of coroutine instead.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index bcc69d5ac3db6..9a7997c2682a0 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -136,6 +136,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index c175ee8980984..fc82025f81fb3 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -32,6 +32,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -983,8 +984,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1026,9 +1029,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MPM.addPass(createM
[llvm-branch-commits] [clang] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -1967,22 +2047,13 @@ splitCoroutine(Function &F, SmallVectorImpl
&Clones,
for (DbgVariableRecord *DVR : DbgVariableRecords)
coro::salvageDebugInfo(ArgToAllocaMap, *DVR, Shape.OptimizeFrame,
false /*UseEntryValue*/);
- return Shape;
-}
-/// Remove calls to llvm.coro.end in the original function.
-static void removeCoroEndsFromRampFunction(const coro::Shape &Shape) {
- if (Shape.ABI != coro::ABI::Switch) {
-for (auto *End : Shape.CoroEnds) {
- replaceCoroEnd(End, Shape, Shape.FramePtr, /*in resume*/ false, nullptr);
-}
- } else {
-for (llvm::AnyCoroEndInst *End : Shape.CoroEnds) {
- auto &Context = End->getContext();
- End->replaceAllUsesWith(ConstantInt::getFalse(Context));
- End->eraseFromParent();
-}
+ removeCoroEndsFromRampFunction(Shape);
+
+ if (!isNoSuspendCoroutine && Shape.ABI == coro::ABI::Switch) {
yuxuanchen1997 wrote:
This turned out to be easy. I am addressing this with the next push for this
patch.
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [clang] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -1455,6 +1462,62 @@ struct SwitchCoroutineSplitter {
setCoroInfo(F, Shape, Clones);
}
+ static Function *createNoAllocVariant(Function &F, coro::Shape &Shape,
yuxuanchen1997 wrote:
This is done. Thanks.
https://github.com/llvm/llvm-project/pull/99283
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From be91ecd53679df7536616132b3492d53a0642ef4 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Tue, 4 Jun 2024 23:22:00 -0700
Subject: [PATCH] [Clang] Introduce [[clang::coro_await_elidable]]
---
llvm/docs/Coroutines.rst | 22 +++
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 142 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 6 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 9 +-
10 files changed, 187 insertions(+), 35 deletions(-)
diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index 36092325e536fb..13cb2d768a3bf8 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -2022,6 +2022,12 @@ The pass CoroSplit builds coroutine frame and outlines
resume and destroy parts
into separate functions. This pass also lowers `coro.await.suspend.void`_,
`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.
+CoroAnnotationElide
+---
+This pass finds all usages of coroutines that are "must elide" and replaces
+`coro.begin` intrinsic with an address of a coroutine frame placed on its
caller
+and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`
+respectively to remove the deallocation code.
CoroElide
-
@@ -2049,6 +2055,22 @@ the coroutine must reach the final suspend point when it
get destroyed.
This attribute only works for switched-resume coroutines now.
+coro_must_elide
+---
+
+When a Call or Invoke instruction is marked with `coro_must_elide`,
+CoroAnnotationElidePass performs heap elision when possible. Note that for
+recursive or mutually recursive functions this elision is usually not possible.
+
+
+coro_gen_noalloc_ramp
+-
+
+This attribute hints CoroSplitPass to generate a `f.noalloc` ramp function for
+a given coroutine `f`. For any call or invoke instruction that calls `f` and
+attributed as `coro_must_elide`, CoroAnnotationElidePass is able to redirect
+the call to use the `.noalloc` variant.
+
Metadata
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index d535ad7f85d74a..760c0bf894c9e0 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 40bc932c3e0eef..111ebf6d5163d6 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1177,6 +1178,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1192,10 +1201,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -145
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 7ca8d8b7d1dfd1d901721dd45f83f861068f9ea0 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] add CoroAnnotationElidePass
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: https://phabricator.intern.facebook.com/D60250514
---
.../Coroutines/CoroAnnotationElide.h | 36 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 143 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 ++
11 files changed, 270 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..2d6e84bdd66423
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Optimizing a coro_must_elide call
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_must_elide" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 17eed97fd950c9..c2b99a0d1f8cea 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -138,6 +138,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 1184123c7710f0..992b4fca8a6919 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -984,8 +985,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1027,9 +1030,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PT
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
yuxuanchen1997 wrote: @ChuanqiXu9 I have changed this patch to only conditionally create the `.noalloc` variant based on an attribute (which is controlled by FE). Let me know if this is good to go. https://github.com/llvm/llvm-project/pull/99283 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
@@ -968,8 +969,8 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level, // it's been modified since. MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor( RequireAnalysisPass())); - MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0)); + MainCGPipeline.addPass(CoroAnnotationElidePass()); yuxuanchen1997 wrote: I applied this suggestion. However, looking at the `buildModuleInlinerPipeline`. It looks like it uses an adapter that runs single CGSCC pass on every function in the module. This won't work well for `CoroAnnotationElidePass` actually. It depends on the callee to be split, but not the caller. Thinking about this, this is actually the same condition as the old `CoroElidePass`. Maybe the right thing to do here is to make this pass a function pass instead and use `createCGSCCToFunctionPassAdaptor`. What do you think? https://github.com/llvm/llvm-project/pull/99285 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -2049,6 +2055,22 @@ the coroutine must reach the final suspend point when it get destroyed. This attribute only works for switched-resume coroutines now. +coro_must_elide +--- + +When a Call or Invoke instruction is marked with `coro_must_elide`, +CoroAnnotationElidePass performs heap elision when possible. Note that for yuxuanchen1997 wrote: What about `coro_elide_safe`? https://github.com/llvm/llvm-project/pull/99283 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From 05529de18e825c0357a4fdfbaeead577fc20cb91 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Tue, 4 Jun 2024 23:22:00 -0700
Subject: [PATCH] [Clang] Introduce [[clang::coro_await_elidable]]
---
llvm/docs/Coroutines.rst | 21 +++
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 142 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 6 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 9 +-
10 files changed, 186 insertions(+), 35 deletions(-)
diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index 36092325e536fb..23fec4d0a556fc 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -2022,6 +2022,12 @@ The pass CoroSplit builds coroutine frame and outlines
resume and destroy parts
into separate functions. This pass also lowers `coro.await.suspend.void`_,
`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.
+CoroAnnotationElide
+---
+This pass finds all usages of coroutines that are "must elide" and replaces
+`coro.begin` intrinsic with an address of a coroutine frame placed on its
caller
+and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`
+respectively to remove the deallocation code.
CoroElide
-
@@ -2049,6 +2055,21 @@ the coroutine must reach the final suspend point when it
get destroyed.
This attribute only works for switched-resume coroutines now.
+coro_elide_safe
+---
+
+When a Call or Invoke instruction is marked with `coro_elide_safe`,
+CoroAnnotationElidePass performs heap elision when possible. Note that for
+recursive or mutually recursive functions this elision is usually not possible.
+
+coro_gen_noalloc_ramp
+-
+
+This attribute hints CoroSplitPass to generate a `f.noalloc` ramp function for
+a given coroutine `f`. For any call or invoke instruction that calls `f` and
+attributed as `coro_must_elide`, CoroAnnotationElidePass is able to redirect
+the call to use the `.noalloc` variant.
+
Metadata
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index d535ad7f85d74a..760c0bf894c9e0 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 40bc932c3e0eef..111ebf6d5163d6 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1177,6 +1178,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1192,10 +1201,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1453,
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From 99d925c0aa2414bb791e94a2a8824e887dc2f367 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Tue, 4 Jun 2024 23:22:00 -0700
Subject: [PATCH] [Clang] Introduce [[clang::coro_await_elidable]]
---
llvm/docs/Coroutines.rst | 21 +++
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 142 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 6 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 9 +-
10 files changed, 186 insertions(+), 35 deletions(-)
diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index 36092325e536fb..534ae66d2ed1d7 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -2022,6 +2022,12 @@ The pass CoroSplit builds coroutine frame and outlines
resume and destroy parts
into separate functions. This pass also lowers `coro.await.suspend.void`_,
`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.
+CoroAnnotationElide
+---
+This pass finds all usages of coroutines that are "must elide" and replaces
+`coro.begin` intrinsic with an address of a coroutine frame placed on its
caller
+and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`
+respectively to remove the deallocation code.
CoroElide
-
@@ -2049,6 +2055,21 @@ the coroutine must reach the final suspend point when it
get destroyed.
This attribute only works for switched-resume coroutines now.
+coro_elide_safe
+---
+
+When a Call or Invoke instruction is marked with `coro_elide_safe`,
+CoroAnnotationElidePass performs heap elision when possible. Note that for
+recursive or mutually recursive functions this elision is usually not possible.
+
+coro_gen_noalloc_ramp
+-
+
+This attribute hints CoroSplitPass to generate a `f.noalloc` ramp function for
+a given coroutine `f`. For any call or invoke instruction that calls `f` and
+attributed as `coro_elide_safe`, CoroAnnotationElidePass is able to redirect
+the call to use the `.noalloc` variant.
+
Metadata
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index d535ad7f85d74a..760c0bf894c9e0 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 40bc932c3e0eef..111ebf6d5163d6 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1177,6 +1178,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1192,10 +1201,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1453,
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_must_elide" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 17f5922762c0f5203cccda6f3d09ac1b2623078f Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] add CoroAnnotationElidePass
Summary:
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: https://phabricator.intern.facebook.com/D60250514
---
.../Coroutines/CoroAnnotationElide.h | 36 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 141 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 ++
11 files changed, 268 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..2b8885c074635d
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Optimizing a coro_elide_safe call
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 17eed97fd950c9..c2b99a0d1f8cea 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -138,6 +138,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 1184123c7710f0..992b4fca8a6919 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -984,8 +985,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1027,9 +1030,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PT
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -2049,6 +2055,22 @@ the coroutine must reach the final suspend point when it get destroyed. This attribute only works for switched-resume coroutines now. +coro_must_elide +--- + +When a Call or Invoke instruction is marked with `coro_must_elide`, +CoroAnnotationElidePass performs heap elision when possible. Note that for yuxuanchen1997 wrote: Changed. Closing this one. https://github.com/llvm/llvm-project/pull/99283 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 edited https://github.com/llvm/llvm-project/pull/99285 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 edited https://github.com/llvm/llvm-project/pull/99285 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 edited https://github.com/llvm/llvm-project/pull/99283 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 0b3fc71e900e5bff749699411d3a76b2395b1202 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] [LLVM][Coroutines] Transform "coro_elide_safe" calls to
switch ABI coroutines to the `noalloc` variant
---
.../Coroutines/CoroAnnotationElide.h | 36 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 147 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 +
11 files changed, 274 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..352c9e14526697
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Elide attributed safe coroutine calls
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 17eed97fd950c9..c2b99a0d1f8cea 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -138,6 +138,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 1184123c7710f0..992b4fca8a6919 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -984,8 +985,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1027,9 +1030,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLT
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From 4084de77bf05708e9a38b328af1f9717d28b202c Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Tue, 4 Jun 2024 23:22:00 -0700
Subject: [PATCH] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI
coroutine ramp functions during CoroSplit
---
llvm/docs/Coroutines.rst | 21 +++
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 141 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 6 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 9 +-
10 files changed, 185 insertions(+), 35 deletions(-)
diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index 36092325e536fb..534ae66d2ed1d7 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -2022,6 +2022,12 @@ The pass CoroSplit builds coroutine frame and outlines
resume and destroy parts
into separate functions. This pass also lowers `coro.await.suspend.void`_,
`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.
+CoroAnnotationElide
+---
+This pass finds all usages of coroutines that are "must elide" and replaces
+`coro.begin` intrinsic with an address of a coroutine frame placed on its
caller
+and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`
+respectively to remove the deallocation code.
CoroElide
-
@@ -2049,6 +2055,21 @@ the coroutine must reach the final suspend point when it
get destroyed.
This attribute only works for switched-resume coroutines now.
+coro_elide_safe
+---
+
+When a Call or Invoke instruction is marked with `coro_elide_safe`,
+CoroAnnotationElidePass performs heap elision when possible. Note that for
+recursive or mutually recursive functions this elision is usually not possible.
+
+coro_gen_noalloc_ramp
+-
+
+This attribute hints CoroSplitPass to generate a `f.noalloc` ramp function for
+a given coroutine `f`. For any call or invoke instruction that calls `f` and
+attributed as `coro_elide_safe`, CoroAnnotationElidePass is able to redirect
+the call to use the `.noalloc` variant.
+
Metadata
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index 5716fd0ea4ab96..d91cccd99a7032 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 8eceaef59a1e1f..0f9a60a2b8388b 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1179,6 +1180,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1194,10 +1203,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 0b3fc71e900e5bff749699411d3a76b2395b1202 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] [LLVM][Coroutines] Transform "coro_elide_safe" calls to
switch ABI coroutines to the `noalloc` variant
---
.../Coroutines/CoroAnnotationElide.h | 36 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 147 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 +
11 files changed, 274 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..352c9e14526697
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Elide attributed safe coroutine calls
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 17eed97fd950c9..c2b99a0d1f8cea 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -138,6 +138,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 1184123c7710f0..992b4fca8a6919 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -984,8 +985,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1027,9 +1030,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLT
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From 4084de77bf05708e9a38b328af1f9717d28b202c Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Tue, 4 Jun 2024 23:22:00 -0700
Subject: [PATCH] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI
coroutine ramp functions during CoroSplit
---
llvm/docs/Coroutines.rst | 21 +++
llvm/lib/Transforms/Coroutines/CoroInternal.h | 4 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 141 ++
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
llvm/test/Transforms/Coroutines/ArgAddr.ll| 6 +-
.../Transforms/Coroutines/coro-alloca-07.ll | 2 +-
.../coro-alloca-loop-carried-address.ll | 2 +-
.../Coroutines/coro-lifetime-end.ll | 6 +-
.../Coroutines/coro-spill-after-phi.ll| 2 +-
.../Transforms/Coroutines/coro-split-00.ll| 9 +-
10 files changed, 185 insertions(+), 35 deletions(-)
diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index 36092325e536fb..534ae66d2ed1d7 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -2022,6 +2022,12 @@ The pass CoroSplit builds coroutine frame and outlines
resume and destroy parts
into separate functions. This pass also lowers `coro.await.suspend.void`_,
`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.
+CoroAnnotationElide
+---
+This pass finds all usages of coroutines that are "must elide" and replaces
+`coro.begin` intrinsic with an address of a coroutine frame placed on its
caller
+and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`
+respectively to remove the deallocation code.
CoroElide
-
@@ -2049,6 +2055,21 @@ the coroutine must reach the final suspend point when it
get destroyed.
This attribute only works for switched-resume coroutines now.
+coro_elide_safe
+---
+
+When a Call or Invoke instruction is marked with `coro_elide_safe`,
+CoroAnnotationElidePass performs heap elision when possible. Note that for
+recursive or mutually recursive functions this elision is usually not possible.
+
+coro_gen_noalloc_ramp
+-
+
+This attribute hints CoroSplitPass to generate a `f.noalloc` ramp function for
+a given coroutine `f`. For any call or invoke instruction that calls `f` and
+attributed as `coro_elide_safe`, CoroAnnotationElidePass is able to redirect
+the call to use the `.noalloc` variant.
+
Metadata
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index 5716fd0ea4ab96..d91cccd99a7032 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,10 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+void suppressCoroAllocs(CoroIdInst *CoroId);
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 8eceaef59a1e1f..0f9a60a2b8388b 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1179,6 +1180,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1194,10 +1203,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
@@ -0,0 +1,147 @@
+//===- CoroAnnotationElide.cpp - Elide attributed safe coroutine calls
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
+
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Instruction.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/Transforms/Utils/CallGraphUpdater.h"
+
+#include
+
+using namespace llvm;
+
+#define DEBUG_TYPE "coro-annotation-elide"
+
+static Instruction *getFirstNonAllocaInTheEntryBlock(Function *F) {
+ for (Instruction &I : F->getEntryBlock())
+if (!isa(&I))
+ return &I;
+ llvm_unreachable("no terminator in the entry block");
+}
+
+// Create an alloca in the caller, using FrameSize and FrameAlign as the callee
+// coroutine's activation frame.
+static Value *allocateFrameInCaller(Function *Caller, uint64_t FrameSize,
+Align FrameAlign) {
+ LLVMContext &C = Caller->getContext();
+ BasicBlock::iterator InsertPt =
+ getFirstNonAllocaInTheEntryBlock(Caller)->getIterator();
+ const DataLayout &DL = Caller->getDataLayout();
+ auto FrameTy = ArrayType::get(Type::getInt8Ty(C), FrameSize);
+ auto *Frame = new AllocaInst(FrameTy, DL.getAllocaAddrSpace(), "", InsertPt);
+ Frame->setAlignment(FrameAlign);
+ return new BitCastInst(Frame, PointerType::getUnqual(C), "vFrame", InsertPt);
+}
+
+// Given a call or invoke instruction to the elide safe coroutine, this
function
+// does the following:
+// - Allocate a frame for the callee coroutine in the caller using alloca.
+// - Replace the old CB with a new Call or Invoke to `NewCallee`, with the
+//pointer to the frame as an additional argument to NewCallee.
+static void processCall(CallBase *CB, Function *Caller, Function *NewCallee,
+uint64_t FrameSize, Align FrameAlign) {
+ auto *FramePtr = allocateFrameInCaller(Caller, FrameSize, FrameAlign);
yuxuanchen1997 wrote:
The old CoroElide didn't have it and just right out of my mind I don't see a
clear path for allowing this in the LLVM Coroutine semantics.
In C++ semantics this is doable (lifetime of the coroutine ended at the full
expression after `co_await`.) Maybe introduce this from FE?
But sure leave a todo here for another day.
https://github.com/llvm/llvm-project/pull/99285
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
@@ -0,0 +1,147 @@
+//===- CoroAnnotationElide.cpp - Elide attributed safe coroutine calls
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
+
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Instruction.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/Transforms/Utils/CallGraphUpdater.h"
+
+#include
+
+using namespace llvm;
+
+#define DEBUG_TYPE "coro-annotation-elide"
+
+static Instruction *getFirstNonAllocaInTheEntryBlock(Function *F) {
+ for (Instruction &I : F->getEntryBlock())
+if (!isa(&I))
+ return &I;
+ llvm_unreachable("no terminator in the entry block");
+}
+
+// Create an alloca in the caller, using FrameSize and FrameAlign as the callee
+// coroutine's activation frame.
+static Value *allocateFrameInCaller(Function *Caller, uint64_t FrameSize,
+Align FrameAlign) {
+ LLVMContext &C = Caller->getContext();
+ BasicBlock::iterator InsertPt =
+ getFirstNonAllocaInTheEntryBlock(Caller)->getIterator();
+ const DataLayout &DL = Caller->getDataLayout();
+ auto FrameTy = ArrayType::get(Type::getInt8Ty(C), FrameSize);
+ auto *Frame = new AllocaInst(FrameTy, DL.getAllocaAddrSpace(), "", InsertPt);
+ Frame->setAlignment(FrameAlign);
+ return new BitCastInst(Frame, PointerType::getUnqual(C), "vFrame", InsertPt);
yuxuanchen1997 wrote:
This is the same procedure as in `CoroElide`. Let's remove the bitcast I guess?
https://github.com/llvm/llvm-project/pull/99285
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
@@ -0,0 +1,147 @@
+//===- CoroAnnotationElide.cpp - Elide attributed safe coroutine calls
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
+
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/IRBuilder.h"
+#include "llvm/IR/InstIterator.h"
+#include "llvm/IR/Instruction.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/Transforms/Utils/CallGraphUpdater.h"
+
+#include
+
+using namespace llvm;
+
+#define DEBUG_TYPE "coro-annotation-elide"
+
+static Instruction *getFirstNonAllocaInTheEntryBlock(Function *F) {
+ for (Instruction &I : F->getEntryBlock())
+if (!isa(&I))
+ return &I;
+ llvm_unreachable("no terminator in the entry block");
+}
+
+// Create an alloca in the caller, using FrameSize and FrameAlign as the callee
+// coroutine's activation frame.
+static Value *allocateFrameInCaller(Function *Caller, uint64_t FrameSize,
+Align FrameAlign) {
+ LLVMContext &C = Caller->getContext();
+ BasicBlock::iterator InsertPt =
+ getFirstNonAllocaInTheEntryBlock(Caller)->getIterator();
+ const DataLayout &DL = Caller->getDataLayout();
+ auto FrameTy = ArrayType::get(Type::getInt8Ty(C), FrameSize);
+ auto *Frame = new AllocaInst(FrameTy, DL.getAllocaAddrSpace(), "", InsertPt);
+ Frame->setAlignment(FrameAlign);
+ return new BitCastInst(Frame, PointerType::getUnqual(C), "vFrame", InsertPt);
+}
+
+// Given a call or invoke instruction to the elide safe coroutine, this
function
+// does the following:
+// - Allocate a frame for the callee coroutine in the caller using alloca.
+// - Replace the old CB with a new Call or Invoke to `NewCallee`, with the
+//pointer to the frame as an additional argument to NewCallee.
+static void processCall(CallBase *CB, Function *Caller, Function *NewCallee,
+uint64_t FrameSize, Align FrameAlign) {
+ auto *FramePtr = allocateFrameInCaller(Caller, FrameSize, FrameAlign);
+ auto NewCBInsertPt = CB->getIterator();
+ llvm::CallBase *NewCB = nullptr;
+ SmallVector NewArgs;
+ NewArgs.append(CB->arg_begin(), CB->arg_end());
+ NewArgs.push_back(FramePtr);
+
+ if (auto *CI = dyn_cast(CB)) {
+auto *NewCI = CallInst::Create(NewCallee->getFunctionType(), NewCallee,
+ NewArgs, "", NewCBInsertPt);
+NewCI->setTailCallKind(CI->getTailCallKind());
+NewCB = NewCI;
yuxuanchen1997 wrote:
`setTailCallKind` is on `CallInst` not `CallBase`. This `NewCB = NewCI` is
upcasting the pointer.
https://github.com/llvm/llvm-project/pull/99285
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
@@ -2049,6 +2055,21 @@ the coroutine must reach the final suspend point when it get destroyed. This attribute only works for switched-resume coroutines now. +coro_elide_safe +--- + +When a Call or Invoke instruction is marked with `coro_elide_safe`, +CoroAnnotationElidePass performs heap elision when possible. Note that for +recursive or mutually recursive functions this elision is usually not possible. + +coro_gen_noalloc_ramp +- + +This attribute hints CoroSplitPass to generate a `f.noalloc` ramp function for yuxuanchen1997 wrote: This attribute is deleted while addressing your feedback in https://github.com/llvm/llvm-project/pull/99282#pullrequestreview-2265588601 I can add a clarification in the documentation for coro_safe_elide. https://github.com/llvm/llvm-project/pull/99283 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From d6f2e78230c0907db95568e5b920d574ce6b4758 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] [LLVM][Coroutines] Transform "coro_elide_safe" calls to
switch ABI coroutines to the `noalloc` variant
---
.../Coroutines/CoroAnnotationElide.h | 36 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 152 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 +
11 files changed, 279 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..352c9e14526697
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Elide attributed safe coroutine calls
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 17eed97fd950c9..c2b99a0d1f8cea 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -138,6 +138,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 1184123c7710f0..992b4fca8a6919 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -984,8 +985,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1027,9 +1030,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLT
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From e2a6027dd2af62f4fbfa92795873f0489fd35cfd Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Tue, 4 Jun 2024 23:22:00 -0700
Subject: [PATCH] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI
coroutine ramp functions during CoroSplit
---
llvm/docs/Coroutines.rst | 18 +++
llvm/lib/Transforms/Coroutines/CoroInternal.h | 7 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 150 +++---
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
.../Transforms/Coroutines/coro-split-00.ll| 15 ++
5 files changed, 191 insertions(+), 26 deletions(-)
diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index 36092325e536fb..5679aefcb421d8 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -2022,6 +2022,12 @@ The pass CoroSplit builds coroutine frame and outlines
resume and destroy parts
into separate functions. This pass also lowers `coro.await.suspend.void`_,
`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.
+CoroAnnotationElide
+---
+This pass finds all usages of coroutines that are "must elide" and replaces
+`coro.begin` intrinsic with an address of a coroutine frame placed on its
caller
+and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`
+respectively to remove the deallocation code.
CoroElide
-
@@ -2049,6 +2055,18 @@ the coroutine must reach the final suspend point when it
get destroyed.
This attribute only works for switched-resume coroutines now.
+coro_elide_safe
+---
+
+When a Call or Invoke instruction to switch ABI coroutine `f` is marked with
+`coro_elide_safe`, CoroSplitPass generates a `f.noalloc` ramp function.
+`f.noalloc` has one more argument than its original ramp function `f`, which is
+the pointer to the allocated frame. `f.noalloc` also suppressed any allocations
+or deallocations that may be guarded by `@llvm.coro.alloc` and
`@llvm.coro.free`.
+
+CoroAnnotationElidePass performs the heap elision when possible. Note that for
+recursive or mutually recursive functions this elision is usually not possible.
+
Metadata
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index d535ad7f85d74a..be86f96525b677 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,13 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+/// Replaces all @llvm.coro.alloc intrinsics calls associated with a given
+/// call @llvm.coro.id instruction with boolean value false.
+void suppressCoroAllocs(CoroIdInst *CoroId);
+/// Replaces CoroAllocs with boolean value false.
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 6bf3c75b95113e..494c4d632de95f 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1177,6 +1178,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1192,10 +1201,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1452,6 +1459,75 @@ struct SwitchCorou
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From e2a6027dd2af62f4fbfa92795873f0489fd35cfd Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Tue, 4 Jun 2024 23:22:00 -0700
Subject: [PATCH] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI
coroutine ramp functions during CoroSplit
---
llvm/docs/Coroutines.rst | 18 +++
llvm/lib/Transforms/Coroutines/CoroInternal.h | 7 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 150 +++---
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
.../Transforms/Coroutines/coro-split-00.ll| 15 ++
5 files changed, 191 insertions(+), 26 deletions(-)
diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index 36092325e536fb..5679aefcb421d8 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -2022,6 +2022,12 @@ The pass CoroSplit builds coroutine frame and outlines
resume and destroy parts
into separate functions. This pass also lowers `coro.await.suspend.void`_,
`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.
+CoroAnnotationElide
+---
+This pass finds all usages of coroutines that are "must elide" and replaces
+`coro.begin` intrinsic with an address of a coroutine frame placed on its
caller
+and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`
+respectively to remove the deallocation code.
CoroElide
-
@@ -2049,6 +2055,18 @@ the coroutine must reach the final suspend point when it
get destroyed.
This attribute only works for switched-resume coroutines now.
+coro_elide_safe
+---
+
+When a Call or Invoke instruction to switch ABI coroutine `f` is marked with
+`coro_elide_safe`, CoroSplitPass generates a `f.noalloc` ramp function.
+`f.noalloc` has one more argument than its original ramp function `f`, which is
+the pointer to the allocated frame. `f.noalloc` also suppressed any allocations
+or deallocations that may be guarded by `@llvm.coro.alloc` and
`@llvm.coro.free`.
+
+CoroAnnotationElidePass performs the heap elision when possible. Note that for
+recursive or mutually recursive functions this elision is usually not possible.
+
Metadata
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index d535ad7f85d74a..be86f96525b677 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,13 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+/// Replaces all @llvm.coro.alloc intrinsics calls associated with a given
+/// call @llvm.coro.id instruction with boolean value false.
+void suppressCoroAllocs(CoroIdInst *CoroId);
+/// Replaces CoroAllocs with boolean value false.
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 6bf3c75b95113e..494c4d632de95f 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1177,6 +1178,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1192,10 +1201,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1452,6 +1459,75 @@ struct SwitchCorou
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From d6f2e78230c0907db95568e5b920d574ce6b4758 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] [LLVM][Coroutines] Transform "coro_elide_safe" calls to
switch ABI coroutines to the `noalloc` variant
---
.../Coroutines/CoroAnnotationElide.h | 36 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 152 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 +
11 files changed, 279 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..352c9e14526697
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Elide attributed safe coroutine calls
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 17eed97fd950c9..c2b99a0d1f8cea 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -138,6 +138,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 1184123c7710f0..992b4fca8a6919 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -984,8 +985,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1027,9 +1030,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLT
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 68a410d159fdb96e7580a7f3fe035df00b893f3c Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] [LLVM][Coroutines] Transform "coro_elide_safe" calls to
switch ABI coroutines to the `noalloc` variant
---
.../Coroutines/CoroAnnotationElide.h | 36 +
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 152 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 76 +
11 files changed, 279 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..352c9e14526697
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Elide attributed safe coroutine calls
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 17eed97fd950c9..c2b99a0d1f8cea 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -138,6 +138,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 1184123c7710f0..992b4fca8a6919 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -984,8 +985,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1027,9 +1030,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLT
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 5b18641d2b59adf11810f71fe5ab3204a94a7a56 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] [LLVM][Coroutines] Transform "coro_elide_safe" calls to
switch ABI coroutines to the `noalloc` variant
---
.../Coroutines/CoroAnnotationElide.h | 36
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 155 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 75 +
11 files changed, 281 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..352c9e14526697
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Elide attributed safe coroutine calls
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 17eed97fd950c9..c2b99a0d1f8cea 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -138,6 +138,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 1184123c7710f0..992b4fca8a6919 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -984,8 +985,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1027,9 +1030,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLTO
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From 0c712a2fbc5b44e892b37085dbace8ba974c1238 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Tue, 4 Jun 2024 23:22:00 -0700
Subject: [PATCH] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI
coroutine ramp functions during CoroSplit
---
llvm/docs/Coroutines.rst | 18 +++
llvm/lib/Transforms/Coroutines/CoroInternal.h | 7 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 150 +++---
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
.../Transforms/Coroutines/coro-split-00.ll| 15 ++
5 files changed, 191 insertions(+), 26 deletions(-)
diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index 36092325e536fb..5679aefcb421d8 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -2022,6 +2022,12 @@ The pass CoroSplit builds coroutine frame and outlines
resume and destroy parts
into separate functions. This pass also lowers `coro.await.suspend.void`_,
`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.
+CoroAnnotationElide
+---
+This pass finds all usages of coroutines that are "must elide" and replaces
+`coro.begin` intrinsic with an address of a coroutine frame placed on its
caller
+and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`
+respectively to remove the deallocation code.
CoroElide
-
@@ -2049,6 +2055,18 @@ the coroutine must reach the final suspend point when it
get destroyed.
This attribute only works for switched-resume coroutines now.
+coro_elide_safe
+---
+
+When a Call or Invoke instruction to switch ABI coroutine `f` is marked with
+`coro_elide_safe`, CoroSplitPass generates a `f.noalloc` ramp function.
+`f.noalloc` has one more argument than its original ramp function `f`, which is
+the pointer to the allocated frame. `f.noalloc` also suppressed any allocations
+or deallocations that may be guarded by `@llvm.coro.alloc` and
`@llvm.coro.free`.
+
+CoroAnnotationElidePass performs the heap elision when possible. Note that for
+recursive or mutually recursive functions this elision is usually not possible.
+
Metadata
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index d535ad7f85d74a..be86f96525b677 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,13 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+/// Replaces all @llvm.coro.alloc intrinsics calls associated with a given
+/// call @llvm.coro.id instruction with boolean value false.
+void suppressCoroAllocs(CoroIdInst *CoroId);
+/// Replaces CoroAllocs with boolean value false.
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 6bf3c75b95113e..494c4d632de95f 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1177,6 +1178,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1192,10 +1201,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1452,6 +1459,75 @@ struct SwitchCorou
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 201bca06d4e75bc4fa24ac269ad7b9750f24616f Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] [LLVM][Coroutines] Transform "coro_elide_safe" calls to
switch ABI coroutines to the `noalloc` variant
---
.../Coroutines/CoroAnnotationElide.h | 36
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 155 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 75 +
11 files changed, 281 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..352c9e14526697
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Elide attributed safe coroutine calls
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 83c1a6712bf4d9..c34f9148cce58b 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -139,6 +139,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 7f9e1362e7ef23..4e8e3dcdff4428 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -973,8 +974,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1016,9 +1019,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLTO
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI coroutine ramp functions during CoroSplit (PR #99283)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99283
>From 0c712a2fbc5b44e892b37085dbace8ba974c1238 Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Tue, 4 Jun 2024 23:22:00 -0700
Subject: [PATCH] [LLVM][Coroutines] Create `.noalloc` variant of switch ABI
coroutine ramp functions during CoroSplit
---
llvm/docs/Coroutines.rst | 18 +++
llvm/lib/Transforms/Coroutines/CoroInternal.h | 7 +
llvm/lib/Transforms/Coroutines/CoroSplit.cpp | 150 +++---
llvm/lib/Transforms/Coroutines/Coroutines.cpp | 27
.../Transforms/Coroutines/coro-split-00.ll| 15 ++
5 files changed, 191 insertions(+), 26 deletions(-)
diff --git a/llvm/docs/Coroutines.rst b/llvm/docs/Coroutines.rst
index 36092325e536fb..5679aefcb421d8 100644
--- a/llvm/docs/Coroutines.rst
+++ b/llvm/docs/Coroutines.rst
@@ -2022,6 +2022,12 @@ The pass CoroSplit builds coroutine frame and outlines
resume and destroy parts
into separate functions. This pass also lowers `coro.await.suspend.void`_,
`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.
+CoroAnnotationElide
+---
+This pass finds all usages of coroutines that are "must elide" and replaces
+`coro.begin` intrinsic with an address of a coroutine frame placed on its
caller
+and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`
+respectively to remove the deallocation code.
CoroElide
-
@@ -2049,6 +2055,18 @@ the coroutine must reach the final suspend point when it
get destroyed.
This attribute only works for switched-resume coroutines now.
+coro_elide_safe
+---
+
+When a Call or Invoke instruction to switch ABI coroutine `f` is marked with
+`coro_elide_safe`, CoroSplitPass generates a `f.noalloc` ramp function.
+`f.noalloc` has one more argument than its original ramp function `f`, which is
+the pointer to the allocated frame. `f.noalloc` also suppressed any allocations
+or deallocations that may be guarded by `@llvm.coro.alloc` and
`@llvm.coro.free`.
+
+CoroAnnotationElidePass performs the heap elision when possible. Note that for
+recursive or mutually recursive functions this elision is usually not possible.
+
Metadata
diff --git a/llvm/lib/Transforms/Coroutines/CoroInternal.h
b/llvm/lib/Transforms/Coroutines/CoroInternal.h
index d535ad7f85d74a..be86f96525b677 100644
--- a/llvm/lib/Transforms/Coroutines/CoroInternal.h
+++ b/llvm/lib/Transforms/Coroutines/CoroInternal.h
@@ -26,6 +26,13 @@ bool declaresIntrinsics(const Module &M,
const std::initializer_list);
void replaceCoroFree(CoroIdInst *CoroId, bool Elide);
+/// Replaces all @llvm.coro.alloc intrinsics calls associated with a given
+/// call @llvm.coro.id instruction with boolean value false.
+void suppressCoroAllocs(CoroIdInst *CoroId);
+/// Replaces CoroAllocs with boolean value false.
+void suppressCoroAllocs(LLVMContext &Context,
+ArrayRef CoroAllocs);
+
/// Attempts to rewrite the location operand of debug intrinsics in terms of
/// the coroutine frame pointer, folding pointer offsets into the DIExpression
/// of the intrinsic.
diff --git a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
index 6bf3c75b95113e..494c4d632de95f 100644
--- a/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
+++ b/llvm/lib/Transforms/Coroutines/CoroSplit.cpp
@@ -25,6 +25,7 @@
#include "llvm/ADT/PriorityWorklist.h"
#include "llvm/ADT/SmallPtrSet.h"
#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringExtras.h"
#include "llvm/ADT/StringRef.h"
#include "llvm/ADT/Twine.h"
#include "llvm/Analysis/CFG.h"
@@ -1177,6 +1178,14 @@ static void
updateAsyncFuncPointerContextSize(coro::Shape &Shape) {
Shape.AsyncLowering.AsyncFuncPointer->setInitializer(NewFuncPtrStruct);
}
+static TypeSize getFrameSizeForShape(coro::Shape &Shape) {
+ // In the same function all coro.sizes should have the same result type.
+ auto *SizeIntrin = Shape.CoroSizes.back();
+ Module *M = SizeIntrin->getModule();
+ const DataLayout &DL = M->getDataLayout();
+ return DL.getTypeAllocSize(Shape.FrameTy);
+}
+
static void replaceFrameSizeAndAlignment(coro::Shape &Shape) {
if (Shape.ABI == coro::ABI::Async)
updateAsyncFuncPointerContextSize(Shape);
@@ -1192,10 +1201,8 @@ static void replaceFrameSizeAndAlignment(coro::Shape
&Shape) {
// In the same function all coro.sizes should have the same result type.
auto *SizeIntrin = Shape.CoroSizes.back();
- Module *M = SizeIntrin->getModule();
- const DataLayout &DL = M->getDataLayout();
- auto Size = DL.getTypeAllocSize(Shape.FrameTy);
- auto *SizeConstant = ConstantInt::get(SizeIntrin->getType(), Size);
+ auto *SizeConstant =
+ ConstantInt::get(SizeIntrin->getType(), getFrameSizeForShape(Shape));
for (CoroSizeInst *CS : Shape.CoroSizes) {
CS->replaceAllUsesWith(SizeConstant);
@@ -1452,6 +1459,75 @@ struct SwitchCorou
[llvm-branch-commits] [llvm] [LLVM][Coroutines] Transform "coro_elide_safe" calls to switch ABI coroutines to the `noalloc` variant (PR #99285)
https://github.com/yuxuanchen1997 updated
https://github.com/llvm/llvm-project/pull/99285
>From 201bca06d4e75bc4fa24ac269ad7b9750f24616f Mon Sep 17 00:00:00 2001
From: Yuxuan Chen
Date: Mon, 15 Jul 2024 15:01:39 -0700
Subject: [PATCH] [LLVM][Coroutines] Transform "coro_elide_safe" calls to
switch ABI coroutines to the `noalloc` variant
---
.../Coroutines/CoroAnnotationElide.h | 36
llvm/lib/Passes/PassBuilder.cpp | 1 +
llvm/lib/Passes/PassBuilderPipelines.cpp | 10 +-
llvm/lib/Passes/PassRegistry.def | 1 +
llvm/lib/Transforms/Coroutines/CMakeLists.txt | 1 +
.../Coroutines/CoroAnnotationElide.cpp| 155 ++
llvm/test/Other/new-pm-defaults.ll| 1 +
.../Other/new-pm-thinlto-postlink-defaults.ll | 1 +
.../new-pm-thinlto-postlink-pgo-defaults.ll | 1 +
...-pm-thinlto-postlink-samplepgo-defaults.ll | 1 +
.../Coroutines/coro-transform-must-elide.ll | 75 +
11 files changed, 281 insertions(+), 2 deletions(-)
create mode 100644
llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
create mode 100644 llvm/lib/Transforms/Coroutines/CoroAnnotationElide.cpp
create mode 100644 llvm/test/Transforms/Coroutines/coro-transform-must-elide.ll
diff --git a/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
new file mode 100644
index 00..352c9e14526697
--- /dev/null
+++ b/llvm/include/llvm/Transforms/Coroutines/CoroAnnotationElide.h
@@ -0,0 +1,36 @@
+//===- CoroAnnotationElide.h - Elide attributed safe coroutine calls
--===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// \file
+// This pass transforms all Call or Invoke instructions that are annotated
+// "coro_elide_safe" to call the `.noalloc` variant of coroutine instead.
+// The frame of the callee coroutine is allocated inside the caller. A pointer
+// to the allocated frame will be passed into the `.noalloc` ramp function.
+//
+//===--===//
+
+#ifndef LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+#define LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
+
+#include "llvm/Analysis/CGSCCPassManager.h"
+#include "llvm/Analysis/LazyCallGraph.h"
+#include "llvm/IR/PassManager.h"
+
+namespace llvm {
+
+struct CoroAnnotationElidePass : PassInfoMixin {
+ CoroAnnotationElidePass() {}
+
+ PreservedAnalyses run(LazyCallGraph::SCC &C, CGSCCAnalysisManager &AM,
+LazyCallGraph &CG, CGSCCUpdateResult &UR);
+
+ static bool isRequired() { return false; }
+};
+} // end namespace llvm
+
+#endif // LLVM_TRANSFORMS_COROUTINES_COROANNOTATIONELIDE_H
diff --git a/llvm/lib/Passes/PassBuilder.cpp b/llvm/lib/Passes/PassBuilder.cpp
index 83c1a6712bf4d9..c34f9148cce58b 100644
--- a/llvm/lib/Passes/PassBuilder.cpp
+++ b/llvm/lib/Passes/PassBuilder.cpp
@@ -139,6 +139,7 @@
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
#include "llvm/Transforms/CFGuard.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
diff --git a/llvm/lib/Passes/PassBuilderPipelines.cpp
b/llvm/lib/Passes/PassBuilderPipelines.cpp
index 7f9e1362e7ef23..4e8e3dcdff4428 100644
--- a/llvm/lib/Passes/PassBuilderPipelines.cpp
+++ b/llvm/lib/Passes/PassBuilderPipelines.cpp
@@ -33,6 +33,7 @@
#include "llvm/Support/VirtualFileSystem.h"
#include "llvm/Target/TargetMachine.h"
#include "llvm/Transforms/AggressiveInstCombine/AggressiveInstCombine.h"
+#include "llvm/Transforms/Coroutines/CoroAnnotationElide.h"
#include "llvm/Transforms/Coroutines/CoroCleanup.h"
#include "llvm/Transforms/Coroutines/CoroConditionalWrapper.h"
#include "llvm/Transforms/Coroutines/CoroEarly.h"
@@ -973,8 +974,10 @@ PassBuilder::buildInlinerPipeline(OptimizationLevel Level,
MainCGPipeline.addPass(createCGSCCToFunctionPassAdaptor(
RequireAnalysisPass()));
- if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink)
+ if (Phase != ThinOrFullLTOPhase::ThinLTOPreLink) {
MainCGPipeline.addPass(CoroSplitPass(Level != OptimizationLevel::O0));
+MainCGPipeline.addPass(CoroAnnotationElidePass());
+ }
// Make sure we don't affect potential future NoRerun CGSCC adaptors.
MIWP.addLateModulePass(createModuleToFunctionPassAdaptor(
@@ -1016,9 +1019,12 @@
PassBuilder::buildModuleInlinerPipeline(OptimizationLevel Level,
buildFunctionSimplificationPipeline(Level, Phase),
PTO.EagerlyInvalidateAnalyses));
- if (Phase != ThinOrFullLTO
[llvm-branch-commits] [clang] [llvm] release/20.x: [fatlto] Add coroutine passes when using FatLTO with ThinLTO (#134434) (PR #134711)
yuxuanchen1997 wrote: cc @apolloww https://github.com/llvm/llvm-project/pull/134711 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
