https://github.com/spavloff updated https://github.com/llvm/llvm-project/pull/135658
>From 287340535219cd5bc31de3a27cde1b279db0eb09 Mon Sep 17 00:00:00 2001 From: Serge Pavlov <sepavl...@gmail.com> Date: Mon, 14 Apr 2025 12:51:43 +0700 Subject: [PATCH 1/6] Minimal support of floating-point operand bundles This is a lite version of https://github.com/llvm/llvm-project/pull/109798, where code changes are minimized to facilitate discussion about the implementation. The motivations and ideas behind the new floating-point operation support are described in that PR and in the discussion https://discourse.llvm.org/t/rfc-change-of-strict-fp-operation-representation-in-ir/85021. There are concerns that the proposed changes are too invasive and a new approach is required to make the transition smoother. This implementation is essentially a subset of PR109798, where everything beyond the minimum is removed. It tries to build eventually the same implementation as that PR but in different steps. The patch does not attempt to modify the existing implementation based on the constrained intrinsics. Instead it introduces a new one using operand bundles. This new implementation initially has very limited functionality, which latter will be extended and finally can replace the existing one. This PR introduces the notion of floating-point operation, this is an intrinsic, that is listed in the file "FloatingPointOps.def". These have two additional properties: 1. In the strict environment (a function with strictfp attribute) calls to these operations acquire side effect, now it is read/write access to inaccessible memory, just as constrained intrinsics do. 2. Calls to these operations may have floating-point operand bundles. There are two kinds of such bundles, tagged with "fp.control" and "fp.except", which are used to carry additional information about control modes and exception handling. Initially the set of control modes consists of rounding mode only. The set of operations enlisted in "FloatingPointOps.def" and in "ConstrainedOps.def" are completely independent, an intrinsic may be in one list or in both. The set of floating-point operations is expected to grow and finally all FP intrinsics will be available in the new implementation. In this patch set of intrinsics in "FloatingPointOps.def" is minimum necessary for tests. --- llvm/docs/LangRef.rst | 51 ++++- llvm/docs/ReleaseNotes.md | 1 + llvm/include/llvm/IR/FPEnv.h | 18 ++ llvm/include/llvm/IR/FloatingPointOps.def | 24 +++ llvm/include/llvm/IR/IRBuilder.h | 45 ++-- llvm/include/llvm/IR/InstrTypes.h | 16 ++ llvm/include/llvm/IR/IntrinsicInst.h | 8 + llvm/include/llvm/IR/LLVMContext.h | 2 + llvm/include/llvm/Support/ModRef.h | 5 + llvm/lib/IR/FPEnv.cpp | 64 ++++++ llvm/lib/IR/IRBuilder.cpp | 75 +++++++ llvm/lib/IR/Instructions.cpp | 83 +++++++- llvm/lib/IR/IntrinsicInst.cpp | 10 + llvm/lib/IR/LLVMContext.cpp | 14 ++ llvm/lib/IR/Verifier.cpp | 31 ++- .../Bitcode/operand-bundles-bc-analyzer.ll | 2 + llvm/test/Verifier/fp-intrinsics.ll | 91 ++++++++ llvm/unittests/IR/IRBuilderTest.cpp | 197 ++++++++++++++++++ 18 files changed, 716 insertions(+), 21 deletions(-) create mode 100644 llvm/include/llvm/IR/FloatingPointOps.def diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst index 769003a90f959..9145c4c9092f1 100644 --- a/llvm/docs/LangRef.rst +++ b/llvm/docs/LangRef.rst @@ -3071,6 +3071,51 @@ A "convergencectrl" operand bundle is only valid on a ``convergent`` operation. When present, the operand bundle must contain exactly one value of token type. See the :doc:`ConvergentOperations` document for details. +.. _ob_fp: + +Floating-point Operand Bundles +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +These operand bundles are used for calls that involve floating-point +operations and interact with :ref:`floating-point environment <floatenv>` or +depend on floating-point options, such as rounding mode, denormal modes, etc. +There are two kinds of such operand bundles, which represent the value of +floating-point control modes and the treatment of status bits respectively. + +An operand bundle tagged with "fp.control" contains information about the +control modes used for the operation execution. Operands specified in this +bundle represent particular options. Currently, only rounding mode is supported. +It is represented by a metadata string value, which specifies the rounding mode +to be used for the operation evaluation. Possible values are: + +:: + + "rtz" - toward zero + "rte" - to nearest, ties to even + "rtp" - toward positive infinity + "rtn" - toward negative infinity + "rmm" - to nearest, ties away from zero + "dyn" - rounding mode is taken from control register + +If "fp.control" is absent, the default rounding rounding mode is taken from the +control register (dynamic rounding). In the particular case of +:ref:`default floating-point environment <floatenv>`, it must be rounding to +nearest, ties to even. + +An operand bundle tagged with "fp.except" may be associated with operations +that can read or write floating-point exception flags. It contains a single +metadata string value, which can have one of the following values: + +:: + + "ignore" + "strict" + "maytrap" + +It has the same meaning as the corresponding argument in +:ref:`constrained intrinsics <constrainedfp>`. + + .. _moduleasm: Module-Level Inline Assembly @@ -3776,9 +3821,9 @@ round-to-nearest rounding mode, and subnormals are assumed to be preserved. Running LLVM code in an environment where these assumptions are not met typically leads to undefined behavior. The ``strictfp`` and ``denormal-fp-math`` attributes as well as :ref:`Constrained Floating-Point Intrinsics -<constrainedfp>` can be used to weaken LLVM's assumptions and ensure defined -behavior in non-default floating-point environments; see their respective -documentation for details. +<constrainedfp>` or :ref:`floating-point operand bundles<ob_fp>` can be used to +weaken LLVM's assumptions and ensure defined behavior in non-default +floating-point environments; see their respective documentation for details. .. _floatnan: diff --git a/llvm/docs/ReleaseNotes.md b/llvm/docs/ReleaseNotes.md index 526d6b4002bba..c5b06e110dcd6 100644 --- a/llvm/docs/ReleaseNotes.md +++ b/llvm/docs/ReleaseNotes.md @@ -64,6 +64,7 @@ Changes to the LLVM IR * Updated semantics of `llvm.type.checked.load.relative` to match that of `llvm.load.relative`. +* Floating-point operand bundles have been added. Changes to LLVM infrastructure ------------------------------ diff --git a/llvm/include/llvm/IR/FPEnv.h b/llvm/include/llvm/IR/FPEnv.h index a0197377759da..2574eea962f69 100644 --- a/llvm/include/llvm/IR/FPEnv.h +++ b/llvm/include/llvm/IR/FPEnv.h @@ -48,18 +48,36 @@ enum ExceptionBehavior : uint8_t { /// metadata. std::optional<RoundingMode> convertStrToRoundingMode(StringRef); +/// Returns a valid RoundingMode enumerator given a string that is used as +/// rounding mode specifier in operand bundles. +std::optional<RoundingMode> convertBundleToRoundingMode(StringRef); + /// For any RoundingMode enumerator, returns a string valid as input in /// constrained intrinsic rounding mode metadata. std::optional<StringRef> convertRoundingModeToStr(RoundingMode); +/// For any RoundingMode enumerator, returns a string to be used in operand +/// bundles. +std::optional<StringRef> convertRoundingModeToBundle(RoundingMode); + /// Returns a valid ExceptionBehavior enumerator when given a string /// valid as input in constrained intrinsic exception behavior metadata. std::optional<fp::ExceptionBehavior> convertStrToExceptionBehavior(StringRef); +/// Returns a valid ExceptionBehavior enumerator given a string from the operand +/// bundle argument. +std::optional<fp::ExceptionBehavior> + convertBundleToExceptionBehavior(StringRef); + /// For any ExceptionBehavior enumerator, returns a string valid as /// input in constrained intrinsic exception behavior metadata. std::optional<StringRef> convertExceptionBehaviorToStr(fp::ExceptionBehavior); +/// Return string representing the given exception behavior for use in operand +/// bundles +std::optional<StringRef> + convertExceptionBehaviorToBundle(fp::ExceptionBehavior); + /// Returns true if the exception handling behavior and rounding mode /// match what is used in the default floating point environment. inline bool isDefaultFPEnvironment(fp::ExceptionBehavior EB, RoundingMode RM) { diff --git a/llvm/include/llvm/IR/FloatingPointOps.def b/llvm/include/llvm/IR/FloatingPointOps.def new file mode 100644 index 0000000000000..8567b5dbac302 --- /dev/null +++ b/llvm/include/llvm/IR/FloatingPointOps.def @@ -0,0 +1,24 @@ +//===- llvm/IR/FloatingPointOps.def - FP intrinsics -------------*- C++ -*-===// +// +// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +// See https://llvm.org/LICENSE.txt for license information. +// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +// +//===----------------------------------------------------------------------===// +// +// Defines set of intrinsics, which are classified as floating-point operations. +// +//===----------------------------------------------------------------------===// + +#ifndef FUNCTION +#define FUNCTION(N,D) +#endif + +// Arguments of the entries are: +// - intrinsic function name, +// - DAG node corresponding to the intrinsic. + +FUNCTION(nearbyint, FNEARBYINT) +FUNCTION(trunc, FTRUNC) + +#undef FUNCTION diff --git a/llvm/include/llvm/IR/IRBuilder.h b/llvm/include/llvm/IR/IRBuilder.h index 0e68ffadc6939..eeb9981999d36 100644 --- a/llvm/include/llvm/IR/IRBuilder.h +++ b/llvm/include/llvm/IR/IRBuilder.h @@ -1004,6 +1004,16 @@ class IRBuilderBase { ArrayRef<Value *> Args, FMFSource FMFSource = {}, const Twine &Name = ""); + /// Create a call to intrinsic \p ID with \p Args, mangled using \p Types and + /// with operand bundles. + /// If \p FMFSource is provided, copy fast-math-flags from that instruction to + /// the intrinsic. + CallInst *CreateIntrinsic(Intrinsic::ID ID, ArrayRef<Type *> Types, + ArrayRef<Value *> Args, + ArrayRef<OperandBundleDef> OpBundles, + Instruction *FMFSource = nullptr, + const Twine &Name = ""); + /// Create a call to non-overloaded intrinsic \p ID with \p Args. If /// \p FMFSource is provided, copy fast-math-flags from that instruction to /// the intrinsic. @@ -2492,24 +2502,13 @@ class IRBuilderBase { CallInst *CreateCall(FunctionType *FTy, Value *Callee, ArrayRef<Value *> Args = {}, const Twine &Name = "", MDNode *FPMathTag = nullptr) { - CallInst *CI = CallInst::Create(FTy, Callee, Args, DefaultOperandBundles); - if (IsFPConstrained) - setConstrainedFPCallAttr(CI); - if (isa<FPMathOperator>(CI)) - setFPAttrs(CI, FPMathTag, FMF); - return Insert(CI, Name); + return CreateCall(FTy, Callee, Args, DefaultOperandBundles, Name, + FPMathTag); } CallInst *CreateCall(FunctionType *FTy, Value *Callee, ArrayRef<Value *> Args, ArrayRef<OperandBundleDef> OpBundles, - const Twine &Name = "", MDNode *FPMathTag = nullptr) { - CallInst *CI = CallInst::Create(FTy, Callee, Args, OpBundles); - if (IsFPConstrained) - setConstrainedFPCallAttr(CI); - if (isa<FPMathOperator>(CI)) - setFPAttrs(CI, FPMathTag, FMF); - return Insert(CI, Name); - } + const Twine &Name = "", MDNode *FPMathTag = nullptr); CallInst *CreateCall(FunctionCallee Callee, ArrayRef<Value *> Args = {}, const Twine &Name = "", MDNode *FPMathTag = nullptr) { @@ -2731,6 +2730,24 @@ class IRBuilderBase { /// Create an assume intrinsic call that represents an dereferencable /// assumption on the provided pointer. CallInst *CreateDereferenceableAssumption(Value *PtrValue, Value *SizeValue); + + /// Create an operand bundle in the provided bundle set to represent given FP + /// rounding mode. + /// + /// If the rounding mode is not defined, adds the default rounding mode, + /// stored in this builder object. + void + createFPRoundingBundle(SmallVectorImpl<OperandBundleDef> &Bundles, + std::optional<RoundingMode> Rounding = std::nullopt); + + /// Create an operand bundle in the provided bundle set to represent FP + /// exception behavior. + /// + /// If the exception behavior is not defined, adds the default behavior, + /// stored in this builder object. + void createFPExceptionBundle( + SmallVectorImpl<OperandBundleDef> &Bundles, + std::optional<fp::ExceptionBehavior> Except = std::nullopt); }; /// This provides a uniform API for creating instructions and inserting diff --git a/llvm/include/llvm/IR/InstrTypes.h b/llvm/include/llvm/IR/InstrTypes.h index 8e47e3c7b3a7c..adf6806216c83 100644 --- a/llvm/include/llvm/IR/InstrTypes.h +++ b/llvm/include/llvm/IR/InstrTypes.h @@ -25,6 +25,7 @@ #include "llvm/IR/CallingConv.h" #include "llvm/IR/DerivedTypes.h" #include "llvm/IR/FMF.h" +#include "llvm/IR/FPEnv.h" #include "llvm/IR/Function.h" #include "llvm/IR/Instruction.h" #include "llvm/IR/LLVMContext.h" @@ -1091,6 +1092,13 @@ template <typename InputTy> class OperandBundleDefT { using OperandBundleDef = OperandBundleDefT<Value *>; using ConstOperandBundleDef = OperandBundleDefT<const Value *>; +void addFPRoundingBundle(LLVMContext &Ctx, + SmallVectorImpl<OperandBundleDef> &Bundles, + RoundingMode Rounding); +void addFPExceptionBundle(LLVMContext &Ctx, + SmallVectorImpl<OperandBundleDef> &Bundles, + fp::ExceptionBehavior Except); + //===----------------------------------------------------------------------===// // CallBase Class //===----------------------------------------------------------------------===// @@ -1150,6 +1158,8 @@ class CallBase : public Instruction { /// number of extra operands. unsigned getNumSubclassExtraOperandsDynamic() const; + MemoryEffects getFloatingPointMemoryEffects() const; + public: using Instruction::getContext; @@ -2155,6 +2165,12 @@ class CallBase : public Instruction { return false; } + /// Return rounding mode specified by operand bundles. + RoundingMode getRoundingMode() const; + + /// Return exception behavior specified by operand bundles. + std::optional<fp::ExceptionBehavior> getExceptionBehavior() const; + /// Used to keep track of an operand bundle. See the main comment on /// OperandBundleUser above. struct BundleOpInfo { diff --git a/llvm/include/llvm/IR/IntrinsicInst.h b/llvm/include/llvm/IR/IntrinsicInst.h index 93750d6e3845e..7db4e25de722a 100644 --- a/llvm/include/llvm/IR/IntrinsicInst.h +++ b/llvm/include/llvm/IR/IntrinsicInst.h @@ -128,6 +128,14 @@ class IntrinsicInst : public CallInst { /// course of IR transformations static bool mayLowerToFunctionCall(Intrinsic::ID IID); + /// Check if \p ID represents a function that may access FP environment and + /// may have FP operand bundles. + /// + /// Access to FP environment means that in the strict FP environment the + /// function has read/write memory effect, which is used to maintain proper + /// instructions ordering. + static bool isFloatingPointOperation(Intrinsic::ID IID); + /// Methods for support type inquiry through isa, cast, and dyn_cast: static bool classof(const CallInst *I) { if (const Function *CF = I->getCalledFunction()) diff --git a/llvm/include/llvm/IR/LLVMContext.h b/llvm/include/llvm/IR/LLVMContext.h index bbd125fd38cf1..b2af2b2bd9148 100644 --- a/llvm/include/llvm/IR/LLVMContext.h +++ b/llvm/include/llvm/IR/LLVMContext.h @@ -96,6 +96,8 @@ class LLVMContext { OB_ptrauth = 7, // "ptrauth" OB_kcfi = 8, // "kcfi" OB_convergencectrl = 9, // "convergencectrl" + OB_fp_control = 10, // "fp.control" + OB_fp_except = 11, // "fp.except" }; /// getMDKindID - Return a unique non-zero ID for the specified metadata kind. diff --git a/llvm/include/llvm/Support/ModRef.h b/llvm/include/llvm/Support/ModRef.h index 677c0a2f5d379..69798e08bc923 100644 --- a/llvm/include/llvm/Support/ModRef.h +++ b/llvm/include/llvm/Support/ModRef.h @@ -234,6 +234,11 @@ template <typename LocationEnum> class MemoryEffectsBase { return getWithoutLoc(Location::InaccessibleMem).doesNotAccessMemory(); } + /// Whether this function accesses inaccessible memory. + bool doesAccessInaccessibleMem() const { + return isModOrRefSet(getModRef(Location::InaccessibleMem)); + } + /// Whether this function only (at most) accesses errno memory. bool onlyAccessesErrnoMem() const { return getWithoutLoc(Location::ErrnoMem).doesNotAccessMemory(); diff --git a/llvm/lib/IR/FPEnv.cpp b/llvm/lib/IR/FPEnv.cpp index 67f21d3756e93..d48bebeb6d2b7 100644 --- a/llvm/lib/IR/FPEnv.cpp +++ b/llvm/lib/IR/FPEnv.cpp @@ -34,6 +34,17 @@ std::optional<RoundingMode> convertStrToRoundingMode(StringRef RoundingArg) { .Default(std::nullopt); } +std::optional<RoundingMode> convertBundleToRoundingMode(StringRef RoundingArg) { + return StringSwitch<std::optional<RoundingMode>>(RoundingArg) + .Case("dyn", RoundingMode::Dynamic) + .Case("rte", RoundingMode::NearestTiesToEven) + .Case("rmm", RoundingMode::NearestTiesToAway) + .Case("rtn", RoundingMode::TowardNegative) + .Case("rtp", RoundingMode::TowardPositive) + .Case("rtz", RoundingMode::TowardZero) + .Default(std::nullopt); +} + std::optional<StringRef> convertRoundingModeToStr(RoundingMode UseRounding) { std::optional<StringRef> RoundingStr; switch (UseRounding) { @@ -61,6 +72,33 @@ std::optional<StringRef> convertRoundingModeToStr(RoundingMode UseRounding) { return RoundingStr; } +std::optional<StringRef> convertRoundingModeToBundle(RoundingMode UseRounding) { + std::optional<StringRef> RoundingStr; + switch (UseRounding) { + case RoundingMode::Dynamic: + RoundingStr = "dyn"; + break; + case RoundingMode::NearestTiesToEven: + RoundingStr = "rte"; + break; + case RoundingMode::NearestTiesToAway: + RoundingStr = "rmm"; + break; + case RoundingMode::TowardNegative: + RoundingStr = "rtn"; + break; + case RoundingMode::TowardPositive: + RoundingStr = "rtp"; + break; + case RoundingMode::TowardZero: + RoundingStr = "rtz"; + break; + default: + break; + } + return RoundingStr; +} + std::optional<fp::ExceptionBehavior> convertStrToExceptionBehavior(StringRef ExceptionArg) { return StringSwitch<std::optional<fp::ExceptionBehavior>>(ExceptionArg) @@ -70,6 +108,15 @@ convertStrToExceptionBehavior(StringRef ExceptionArg) { .Default(std::nullopt); } +std::optional<fp::ExceptionBehavior> +convertBundleToExceptionBehavior(StringRef ExceptionArg) { + return StringSwitch<std::optional<fp::ExceptionBehavior>>(ExceptionArg) + .Case("ignore", fp::ebIgnore) + .Case("maytrap", fp::ebMayTrap) + .Case("strict", fp::ebStrict) + .Default(std::nullopt); +} + std::optional<StringRef> convertExceptionBehaviorToStr(fp::ExceptionBehavior UseExcept) { std::optional<StringRef> ExceptStr; @@ -87,6 +134,23 @@ convertExceptionBehaviorToStr(fp::ExceptionBehavior UseExcept) { return ExceptStr; } +std::optional<StringRef> +convertExceptionBehaviorToBundle(fp::ExceptionBehavior UseExcept) { + std::optional<StringRef> ExceptStr; + switch (UseExcept) { + case fp::ebStrict: + ExceptStr = "strict"; + break; + case fp::ebIgnore: + ExceptStr = "ignore"; + break; + case fp::ebMayTrap: + ExceptStr = "maytrap"; + break; + } + return ExceptStr; +} + Intrinsic::ID getConstrainedIntrinsicID(const Instruction &Instr) { Intrinsic::ID IID = Intrinsic::not_intrinsic; switch (Instr.getOpcode()) { diff --git a/llvm/lib/IR/IRBuilder.cpp b/llvm/lib/IR/IRBuilder.cpp index e5a2f08c393c9..e09c1e635720a 100644 --- a/llvm/lib/IR/IRBuilder.cpp +++ b/llvm/lib/IR/IRBuilder.cpp @@ -120,6 +120,67 @@ IRBuilderBase::createCallHelper(Function *Callee, ArrayRef<Value *> Ops, return CI; } +CallInst *IRBuilderBase::CreateCall(FunctionType *FTy, Value *Callee, + ArrayRef<Value *> Args, + ArrayRef<OperandBundleDef> OpBundles, + const Twine &Name, MDNode *FPMathTag) { + assert(std::count_if(OpBundles.begin(), OpBundles.end(), + [](const OperandBundleDef &Item) { + return Item.getTag() == "fp.control"; + }) <= 1); + assert(std::count_if(OpBundles.begin(), OpBundles.end(), + [](const OperandBundleDef &Item) { + return Item.getTag() == "fp.except"; + }) <= 1); + + ArrayRef<OperandBundleDef> ActualBundlesRef = OpBundles; + SmallVector<OperandBundleDef, 2> ActualBundles; + + // If the builder is in strictfp mode and has non-default options (like + // non-dynamic rounding), add corresponding operand bundle. If such bundle is + // already present, assume it overwrites defaults. + bool doesTheCallAccessFPEnv = false; + if (IsFPConstrained) { + if (const auto *Func = dyn_cast<Function>(Callee)) { + if (Intrinsic::ID ID = Func->getIntrinsicID()) { + if (IntrinsicInst::isFloatingPointOperation(ID)) { + doesTheCallAccessFPEnv = true; + bool NeedRound = DefaultConstrainedRounding != RoundingMode::Dynamic; + bool NeedExcept = DefaultConstrainedExcept != fp::ebStrict; + for (const auto &Item : OpBundles) { + if (NeedRound && Item.getTag() == "fp.control") + NeedRound = false; + else if (NeedExcept && Item.getTag() == "fp.except") + NeedExcept = false; + ActualBundles.push_back(Item); + } + if (NeedRound) + createFPRoundingBundle(ActualBundles, DefaultConstrainedRounding); + if (NeedExcept) + createFPExceptionBundle(ActualBundles, DefaultConstrainedExcept); + ActualBundlesRef = ActualBundles; + } + } + } + } + + // If the call accesses FPE, update memory effects accordingly. + CallInst *CI = CallInst::Create(FTy, Callee, Args, OpBundles); + if (doesTheCallAccessFPEnv) { + MemoryEffects ME = MemoryEffects::inaccessibleMemOnly(); + if (CI->getAttributes().hasFnAttr(Attribute::Memory)) + ME |= CI->getAttributes().getMemoryEffects(); + auto A = Attribute::getWithMemoryEffects(getContext(), ME); + CI->addFnAttr(A); + } + + if (IsFPConstrained) + setConstrainedFPCallAttr(CI); + if (isa<FPMathOperator>(CI)) + setFPAttrs(CI, FPMathTag, FMF); + return Insert(CI, Name); +} + Value *IRBuilderBase::CreateVScale(Constant *Scaling, const Twine &Name) { assert(isa<ConstantInt>(Scaling) && "Expected constant integer"); if (cast<ConstantInt>(Scaling)->isZero()) @@ -1348,6 +1409,20 @@ CallInst *IRBuilderBase::CreateDereferenceableAssumption(Value *PtrValue, {DereferenceableOpB}); } +void IRBuilderBase::createFPRoundingBundle( + SmallVectorImpl<OperandBundleDef> &Bundles, + std::optional<RoundingMode> Rounding) { + addFPRoundingBundle(Context, Bundles, + Rounding.value_or(DefaultConstrainedRounding)); +} + +void IRBuilderBase::createFPExceptionBundle( + SmallVectorImpl<OperandBundleDef> &Bundles, + std::optional<fp::ExceptionBehavior> Except) { + addFPExceptionBundle(Context, Bundles, + Except.value_or(DefaultConstrainedExcept)); +} + IRBuilderDefaultInserter::~IRBuilderDefaultInserter() = default; IRBuilderCallbackInserter::~IRBuilderCallbackInserter() = default; IRBuilderFolder::~IRBuilderFolder() = default; diff --git a/llvm/lib/IR/Instructions.cpp b/llvm/lib/IR/Instructions.cpp index 18109bf107858..a940dc5af2a14 100644 --- a/llvm/lib/IR/Instructions.cpp +++ b/llvm/lib/IR/Instructions.cpp @@ -609,9 +609,10 @@ bool CallBase::hasReadingOperandBundles() const { // Implementation note: this is a conservative implementation of operand // bundle semantics, where *any* non-assume operand bundle (other than // ptrauth) forces a callsite to be at least readonly. - return hasOperandBundlesOtherThan({LLVMContext::OB_ptrauth, - LLVMContext::OB_kcfi, - LLVMContext::OB_convergencectrl}) && + return hasOperandBundlesOtherThan( + {LLVMContext::OB_ptrauth, LLVMContext::OB_kcfi, + LLVMContext::OB_fp_control, LLVMContext::OB_fp_except, + LLVMContext::OB_convergencectrl}) && getIntrinsicID() != Intrinsic::assume; } @@ -619,14 +620,70 @@ bool CallBase::hasClobberingOperandBundles() const { return hasOperandBundlesOtherThan( {LLVMContext::OB_deopt, LLVMContext::OB_funclet, LLVMContext::OB_ptrauth, LLVMContext::OB_kcfi, + LLVMContext::OB_fp_control, LLVMContext::OB_fp_except, LLVMContext::OB_convergencectrl}) && getIntrinsicID() != Intrinsic::assume; } +RoundingMode CallBase::getRoundingMode() const { + // Try reading rounding mode from FP bundle. + std::optional<RoundingMode> RM; + if (auto RoundingBundle = getOperandBundle(LLVMContext::OB_fp_control)) { + Value *V = RoundingBundle->Inputs.front(); + Metadata *MD = cast<MetadataAsValue>(V)->getMetadata(); + RM = convertBundleToRoundingMode(cast<MDString>(MD)->getString()); + } + if (RM) + return *RM; + + // No FP bundle, try to guess from the current mode. + if (getParent()) + if (auto *F = getFunction(); F) + return F->getAttributes().hasFnAttr(Attribute::StrictFP) + ? RoundingMode::Dynamic + : RoundingMode::NearestTiesToEven; + + // Isolated call. Assume default environment. + return RoundingMode::NearestTiesToEven; +} + +std::optional<fp::ExceptionBehavior> CallBase::getExceptionBehavior() const { + // Try determining exception behavior from FP bundle. + std::optional<fp::ExceptionBehavior> EB; + if (auto ExceptionBundle = getOperandBundle(LLVMContext::OB_fp_except)) { + Value *V = ExceptionBundle->Inputs.front(); + Metadata *MD = cast<MetadataAsValue>(V)->getMetadata(); + EB = convertBundleToExceptionBehavior(cast<MDString>(MD)->getString()); + } + if (EB) + return *EB; + + // No FP bundle, try to guess from the current mode. + if (getParent()) + if (auto *F = getFunction(); F) + return F->getAttributes().hasFnAttr(Attribute::StrictFP) ? fp::ebStrict + : fp::ebIgnore; + + // Isolated call. Assume default environment. + return fp::ebIgnore; +} + +MemoryEffects CallBase::getFloatingPointMemoryEffects() const { + if (Intrinsic::ID IntrID = getIntrinsicID()) + if (const BasicBlock *BB = getParent()) + if (const Function *F = BB->getParent()) + if (F->hasFnAttribute(Attribute::StrictFP)) + if (IntrinsicInst::isFloatingPointOperation(IntrID)) { + return MemoryEffects::inaccessibleMemOnly(); + } + return MemoryEffects::none(); +} + MemoryEffects CallBase::getMemoryEffects() const { MemoryEffects ME = getAttributes().getMemoryEffects(); if (auto *Fn = dyn_cast<Function>(getCalledOperand())) { MemoryEffects FnME = Fn->getMemoryEffects(); + FnME |= getFloatingPointMemoryEffects(); if (hasOperandBundles()) { // TODO: Add a method to get memory effects for operand bundles instead. if (hasReadingOperandBundles()) @@ -727,6 +784,26 @@ bool CallBase::hasArgumentWithAdditionalReturnCaptureComponents() const { return false; } +void llvm::addFPRoundingBundle(LLVMContext &Ctx, + SmallVectorImpl<OperandBundleDef> &Bundles, + RoundingMode Rounding) { + std::optional<StringRef> RndStr = convertRoundingModeToBundle(Rounding); + assert(RndStr && "Garbage rounding mode!"); + auto *RoundingMDS = MDString::get(Ctx, *RndStr); + auto *RM = MetadataAsValue::get(Ctx, RoundingMDS); + Bundles.emplace_back("fp.control", RM); +} + +void llvm::addFPExceptionBundle(LLVMContext &Ctx, + SmallVectorImpl<OperandBundleDef> &Bundles, + fp::ExceptionBehavior Except) { + std::optional<StringRef> ExcStr = convertExceptionBehaviorToBundle(Except); + assert(ExcStr && "Garbage exception behavior!"); + auto *ExceptMDS = MDString::get(Ctx, *ExcStr); + auto *EB = MetadataAsValue::get(Ctx, ExceptMDS); + Bundles.emplace_back("fp.except", EB); +} + //===----------------------------------------------------------------------===// // CallInst Implementation //===----------------------------------------------------------------------===// diff --git a/llvm/lib/IR/IntrinsicInst.cpp b/llvm/lib/IR/IntrinsicInst.cpp index 256bce1abe71f..44a7479fbea48 100644 --- a/llvm/lib/IR/IntrinsicInst.cpp +++ b/llvm/lib/IR/IntrinsicInst.cpp @@ -66,6 +66,16 @@ bool IntrinsicInst::mayLowerToFunctionCall(Intrinsic::ID IID) { } } +bool IntrinsicInst::isFloatingPointOperation(Intrinsic::ID IID) { + switch (IID) { +#define FUNCTION(NAME, D) case Intrinsic::NAME: +#include "llvm/IR/FloatingPointOps.def" + return true; + default: + return false; + } +} + //===----------------------------------------------------------------------===// /// DbgVariableIntrinsic - This is the common base class for debug info /// intrinsics for variables. diff --git a/llvm/lib/IR/LLVMContext.cpp b/llvm/lib/IR/LLVMContext.cpp index 447e5d92e0b99..92dcb5858fb99 100644 --- a/llvm/lib/IR/LLVMContext.cpp +++ b/llvm/lib/IR/LLVMContext.cpp @@ -53,6 +53,10 @@ static StringRef knownBundleName(unsigned BundleTagID) { return "kcfi"; case LLVMContext::OB_convergencectrl: return "convergencectrl"; + case LLVMContext::OB_fp_control: + return "fp.control"; + case LLVMContext::OB_fp_except: + return "fp.except"; default: llvm_unreachable("unknown bundle id"); } @@ -82,6 +86,16 @@ LLVMContext::LLVMContext() : pImpl(new LLVMContextImpl(*this)) { assert(Entry->second == BundleTagID && "operand bundle id drifted!"); } + auto *RoundingEntry = pImpl->getOrInsertBundleTag("fp.control"); + assert(RoundingEntry->second == LLVMContext::OB_fp_control && + "fp.control operand bundle id drifted!"); + (void)RoundingEntry; + + auto *ExceptionEntry = pImpl->getOrInsertBundleTag("fp.except"); + assert(ExceptionEntry->second == LLVMContext::OB_fp_except && + "fp.except operand bundle id drifted!"); + (void)ExceptionEntry; + SyncScope::ID SingleThreadSSID = pImpl->getOrInsertSyncScopeID("singlethread"); assert(SingleThreadSSID == SyncScope::SingleThread && diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp index e3f6c1ad5a65b..747cc5b9ca552 100644 --- a/llvm/lib/IR/Verifier.cpp +++ b/llvm/lib/IR/Verifier.cpp @@ -3765,7 +3765,8 @@ void Verifier::visitCallBase(CallBase &Call) { FoundGCTransitionBundle = false, FoundCFGuardTargetBundle = false, FoundPreallocatedBundle = false, FoundGCLiveBundle = false, FoundPtrauthBundle = false, FoundKCFIBundle = false, - FoundAttachedCallBundle = false; + FoundAttachedCallBundle = false, FoundFpeControlBundle = false, + FoundFpeExceptBundle = false; for (unsigned i = 0, e = Call.getNumOperandBundles(); i < e; ++i) { OperandBundleUse BU = Call.getOperandBundleAt(i); uint32_t Tag = BU.getTagID(); @@ -3828,6 +3829,34 @@ void Verifier::visitCallBase(CallBase &Call) { "Multiple \"clang.arc.attachedcall\" operand bundles", Call); FoundAttachedCallBundle = true; verifyAttachedCallBundle(Call, BU); + } else if (Tag == LLVMContext::OB_fp_control) { + Check(!FoundFpeControlBundle, "Multiple fp.control operand bundles", + Call); + Check(BU.Inputs.size() == 1, + "Expected exactly one fp.control bundle operand", Call); + auto *V = dyn_cast<MetadataAsValue>(BU.Inputs.front()); + Check(V, "Value of fp.control bundle operand must be a metadata", Call); + auto *MDS = dyn_cast<MDString>(V->getMetadata()); + Check(MDS, "Value of fp.control bundle operand must be a string", Call); + auto RM = convertBundleToRoundingMode(MDS->getString()); + Check(RM.has_value(), + "Value of fp.control bundle operand is not a correct rounding mode", + Call); + FoundFpeControlBundle = true; + } else if (Tag == LLVMContext::OB_fp_except) { + Check(!FoundFpeExceptBundle, "Multiple fp.except operand bundles", Call); + Check(BU.Inputs.size() == 1, + "Expected exactly one fp.except bundle operand", Call); + auto *V = dyn_cast<MetadataAsValue>(BU.Inputs.front()); + Check(V, "Value of fp.except bundle operand must be a metadata", Call); + auto *MDS = dyn_cast<MDString>(V->getMetadata()); + Check(MDS, "Value of fp.except bundle operand must be a string", Call); + auto EB = convertBundleToExceptionBehavior(MDS->getString()); + Check(EB.has_value(), + "Value of fp.except bundle operand is not a correct exception " + "behavior", + Call); + FoundFpeExceptBundle = true; } } diff --git a/llvm/test/Bitcode/operand-bundles-bc-analyzer.ll b/llvm/test/Bitcode/operand-bundles-bc-analyzer.ll index d860104b9cb3d..01e5b3f6673ae 100644 --- a/llvm/test/Bitcode/operand-bundles-bc-analyzer.ll +++ b/llvm/test/Bitcode/operand-bundles-bc-analyzer.ll @@ -13,6 +13,8 @@ ; CHECK-NEXT: <OPERAND_BUNDLE_TAG ; CHECK-NEXT: <OPERAND_BUNDLE_TAG ; CHECK-NEXT: <OPERAND_BUNDLE_TAG +; CHECK-NEXT: <OPERAND_BUNDLE_TAG +; CHECK-NEXT: <OPERAND_BUNDLE_TAG ; CHECK-NEXT: </OPERAND_BUNDLE_TAGS_BLOCK ; CHECK: <FUNCTION_BLOCK diff --git a/llvm/test/Verifier/fp-intrinsics.ll b/llvm/test/Verifier/fp-intrinsics.ll index 4934843d5a2ed..12b0fca97e5af 100644 --- a/llvm/test/Verifier/fp-intrinsics.ll +++ b/llvm/test/Verifier/fp-intrinsics.ll @@ -51,4 +51,95 @@ entry: ret double %fadd } +; Test multiple fp.control bundles. +; CHECK-NEXT: Multiple fp.control operand bundles +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz"), "fp.control"(metadata !"rtz") ] +define double @f6(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !"rtz"), "fp.control"(metadata !"rtz") ] + ret double %ftrunc +} + +; Test fp.control bundle that has more than one operands. +; CHECK-NEXT: Expected exactly one fp.control bundle operand +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz", metadata !"rte") ] +define double @f7(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !"rtz", metadata !"rte") ] + ret double %ftrunc +} + +; Test fp.control bundle that has non-metadata operand. +; CHECK-NEXT: Value of fp.control bundle operand must be a metadata +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(i32 0) ] +define double @f8(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(i32 0) ] + ret double %ftrunc +} + +; Test fp.control bundle that has non-string operand. +; CHECK-NEXT: Value of fp.control bundle operand must be a string +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata i64 3) ] +define double @f9(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !{i64 3}) ] + ret double %ftrunc +} + +; Test fp.control bundle that specifies incorrect value. +; CHECK-NEXT: Value of fp.control bundle operand is not a correct rounding mode +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"qqq") ] +define double @f10(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.control"(metadata !"qqq") ] + ret double %ftrunc +} + +; Test multiple fp.except bundles. +; CHECK-NEXT: Multiple fp.except operand bundles +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"strict"), "fp.except"(metadata !"strict") ] +define double @f11(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(metadata !"strict"), "fp.except"(metadata !"strict") ] + ret double %ftrunc +} + +; Test fp.except bundle that has more than one operands. +; CHECK-NEXT: Expected exactly one fp.except bundle operand +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"strict", metadata !"strict") ] +define double @f12(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(metadata !"strict", metadata !"strict") ] + ret double %ftrunc +} + +; Test fp.except bundle that has non-metadata operand. +; CHECK-NEXT: Value of fp.except bundle operand must be a metadata +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(i32 0) ] +define double @f13(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(i32 0) ] + ret double %ftrunc +} + +; Test fp.except bundle that has non-string operand. +; CHECK-NEXT: Value of fp.except bundle operand must be a string +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata i64 3) ] +define double @f14(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(metadata !{i64 3}) ] + ret double %ftrunc +} + +; Test fp.except bundle that specifies incorrect value. +; CHECK-NEXT: Value of fp.except bundle operand is not a correct exception behavior +; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"qqq") ] +define double @f15(double %a) #0 { +entry: + %ftrunc = call double @llvm.trunc.f64(double %a) #0 [ "fp.except"(metadata !"qqq") ] + ret double %ftrunc +} + + attributes #0 = { strictfp } diff --git a/llvm/unittests/IR/IRBuilderTest.cpp b/llvm/unittests/IR/IRBuilderTest.cpp index b7eb0af728331..48a76c9cd586d 100644 --- a/llvm/unittests/IR/IRBuilderTest.cpp +++ b/llvm/unittests/IR/IRBuilderTest.cpp @@ -465,6 +465,203 @@ TEST_F(IRBuilderTest, ConstrainedFPFunctionCall) { EXPECT_FALSE(verifyModule(*M)); } +TEST_F(IRBuilderTest, FPBundlesDefault) { + IRBuilder<> Builder(BB); + GlobalVariable *GVDouble = new GlobalVariable( + *M, Type::getDoubleTy(Ctx), true, GlobalValue::ExternalLinkage, nullptr); + Value *FnArg = Builder.CreateLoad(GVDouble->getValueType(), GVDouble); + Function *Fn = Intrinsic::getOrInsertDeclaration( + M.get(), Intrinsic::nearbyint, {Type::getDoubleTy(Ctx)}); + + // A floating-point operation does not have side effects in default + // environment even. + { + Value *V = Builder.CreateCall(Fn, {FnArg}); + auto *I = cast<IntrinsicInst>(V); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode()); + EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesNotAccessMemory()); + } + + // Check call with FP bundles, rounding is set to default value. + // nearbyint(%x) [ "fp.control" (metadata !"rte") ] + { + SmallVector<OperandBundleDef, 1> Bundles; + llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven); + Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles); + auto *I = cast<IntrinsicInst>(V); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode()); + EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesNotAccessMemory()); + } + + // Check call with FP bundles, exception behavior is set to default value. + // nearbyint(%x) [ "fp.except" (metadata !"ignore") ] + { + SmallVector<OperandBundleDef, 1> Bundles; + llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore); + Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles); + auto *I = cast<IntrinsicInst>(V); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode()); + EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesNotAccessMemory()); + } + + // Check call with FP bundles, both rounding mode and exception behavior are + // set. + // nearbyint(%x) [ "fp.except" (metadata !"ignore") ] + { + SmallVector<OperandBundleDef, 1> Bundles; + llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven); + llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore); + Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles); + auto *I = cast<IntrinsicInst>(V); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode()); + EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesNotAccessMemory()); + } +} + +TEST_F(IRBuilderTest, FPBundlesStrict) { + F->addFnAttr(Attribute::StrictFP); + + IRBuilder<> Builder(BB); + Builder.setDefaultConstrainedExcept(fp::ebStrict); + Builder.setDefaultConstrainedRounding(RoundingMode::TowardZero); + Builder.setIsFPConstrained(true); + + GlobalVariable *GVDouble = new GlobalVariable( + *M, Type::getDoubleTy(Ctx), true, GlobalValue::ExternalLinkage, nullptr); + Value *FnArg = Builder.CreateLoad(GVDouble->getValueType(), GVDouble); + Function *Fn = Intrinsic::getOrInsertDeclaration( + M.get(), Intrinsic::nearbyint, {Type::getDoubleTy(Ctx)}); + + // A floating-point operation has side effects in strictfp environment even + // if it has no FP bundles. + { + Value *V = Builder.CreateCall(Fn, {FnArg}); + auto *I = cast<IntrinsicInst>(V); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode()); + EXPECT_EQ(fp::ebStrict, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesAccessInaccessibleMem()); + } + + // Check call with FP bundles, with default (dynamic) rounding mode + // nearbyint(%x) [ "fp.control" (metadata !"dyn") ] + { + SmallVector<OperandBundleDef, 1> Bundles; + llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::Dynamic); + Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles); + auto *I = cast<IntrinsicInst>(V); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode()); + EXPECT_EQ(fp::ebStrict, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesAccessInaccessibleMem()); + } + + // Check call with FP bundles, with specific rounding mode + // nearbyint(%x) [ "fp.control" (metadata !"rtz") ] + { + SmallVector<OperandBundleDef, 1> Bundles; + llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::TowardZero); + Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles); + auto *I = cast<IntrinsicInst>(V); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::TowardZero, I->getRoundingMode()); + EXPECT_EQ(fp::ebStrict, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesAccessInaccessibleMem()); + } + + // Check call with FP bundles, exception behavior is set to default value. + // nearbyint(%x) [ "fp.except" (metadata !"strict") ] + { + SmallVector<OperandBundleDef, 1> Bundles; + llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebStrict); + Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles); + auto *I = cast<IntrinsicInst>(V); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode()); + EXPECT_EQ(fp::ebStrict, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesAccessInaccessibleMem()); + } + + // Check call with FP bundles, exception behavior is set to specific value. + // nearbyint(%x) [ "fp.except" (metadata !"ignore") ] + { + SmallVector<OperandBundleDef, 1> Bundles; + llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore); + Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles); + auto *I = cast<IntrinsicInst>(V); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::Dynamic, I->getRoundingMode()); + EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesAccessInaccessibleMem()); + } + + // Check call with both FP bundles. + // nearbyint(%x) [ "fp.control" (metadata !"rtz"), + // "fp.except" (metadata !"ignore") ] + { + SmallVector<OperandBundleDef, 1> Bundles; + llvm::addFPRoundingBundle(Ctx, Bundles, RoundingMode::NearestTiesToEven); + llvm::addFPExceptionBundle(Ctx, Bundles, fp::ebIgnore); + Value *V = Builder.CreateCall(Fn, {FnArg}, Bundles); + auto *I = cast<IntrinsicInst>(V); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + EXPECT_TRUE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_EQ(Intrinsic::nearbyint, I->getIntrinsicID()); + EXPECT_EQ(RoundingMode::NearestTiesToEven, I->getRoundingMode()); + EXPECT_EQ(fp::ebIgnore, I->getExceptionBehavior()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesAccessInaccessibleMem()); + } + + // Function calls, that do not depend on FP options, does not have + // memory effects. + { + Function *Fn = Intrinsic::getOrInsertDeclaration(M.get(), Intrinsic::fabs, + {Type::getDoubleTy(Ctx)}); + Value *V = Builder.CreateCall(Fn, {FnArg}); + auto *I = cast<IntrinsicInst>(V); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_except).has_value()); + EXPECT_FALSE(I->getOperandBundle(LLVMContext::OB_fp_control).has_value()); + MemoryEffects ME = I->getMemoryEffects(); + EXPECT_TRUE(ME.doesNotAccessMemory()); + } +} + TEST_F(IRBuilderTest, Lifetime) { IRBuilder<> Builder(BB); AllocaInst *Var1 = Builder.CreateAlloca(Builder.getInt8Ty()); >From 089909dcd10a841b4504bbcbc1e991ecbac426c2 Mon Sep 17 00:00:00 2001 From: Serge Pavlov <sepavl...@gmail.com> Date: Wed, 16 Apr 2025 00:03:49 +0700 Subject: [PATCH 2/6] Fix test --- clang/test/CodeGen/strictfp-elementwise-builtins.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/clang/test/CodeGen/strictfp-elementwise-builtins.cpp b/clang/test/CodeGen/strictfp-elementwise-builtins.cpp index b250512efc5c7..532686512b0d4 100644 --- a/clang/test/CodeGen/strictfp-elementwise-builtins.cpp +++ b/clang/test/CodeGen/strictfp-elementwise-builtins.cpp @@ -210,7 +210,7 @@ float4 strict_elementwise_rint(float4 a) { // CHECK-LABEL: define dso_local noundef <4 x float> @_Z28strict_elementwise_nearbyintDv4_f // CHECK-SAME: (<4 x float> noundef [[A:%.*]]) local_unnamed_addr #[[ATTR2]] { // CHECK-NEXT: entry: -// CHECK-NEXT: [[ELT_NEARBYINT:%.*]] = tail call <4 x float> @llvm.nearbyint.v4f32(<4 x float> [[A]]) #[[ATTR4]] +// CHECK-NEXT: [[ELT_NEARBYINT:%.*]] = tail call <4 x float> @llvm.nearbyint.v4f32(<4 x float> [[A]]) #[[ATTR5:[0-9]+]] // CHECK-NEXT: ret <4 x float> [[ELT_NEARBYINT]] // float4 strict_elementwise_nearbyint(float4 a) { @@ -300,7 +300,7 @@ float4 strict_elementwise_atan2(float4 a, float4 b) { // CHECK-LABEL: define dso_local noundef <4 x float> @_Z24strict_elementwise_truncDv4_f // CHECK-SAME: (<4 x float> noundef [[A:%.*]]) local_unnamed_addr #[[ATTR2]] { // CHECK-NEXT: entry: -// CHECK-NEXT: [[ELT_TRUNC:%.*]] = tail call <4 x float> @llvm.trunc.v4f32(<4 x float> [[A]]) #[[ATTR4]] +// CHECK-NEXT: [[ELT_TRUNC:%.*]] = tail call <4 x float> @llvm.trunc.v4f32(<4 x float> [[A]]) #[[ATTR5]] // CHECK-NEXT: ret <4 x float> [[ELT_TRUNC]] // float4 strict_elementwise_trunc(float4 a) { >From 1a300f9783b9c0dcdeb76913d004f6c0f73a41a4 Mon Sep 17 00:00:00 2001 From: Serge Pavlov <sepavl...@gmail.com> Date: Thu, 17 Apr 2025 14:00:55 +0700 Subject: [PATCH 3/6] Set MemoryEffects of call only if called function is nomem It make call close to constrained calls, where memory effects of the calls do not define memory effects. It helps migration to operand bundles. --- llvm/lib/IR/IRBuilder.cpp | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/llvm/lib/IR/IRBuilder.cpp b/llvm/lib/IR/IRBuilder.cpp index e09c1e635720a..6ed01c31c0617 100644 --- a/llvm/lib/IR/IRBuilder.cpp +++ b/llvm/lib/IR/IRBuilder.cpp @@ -139,12 +139,13 @@ CallInst *IRBuilderBase::CreateCall(FunctionType *FTy, Value *Callee, // If the builder is in strictfp mode and has non-default options (like // non-dynamic rounding), add corresponding operand bundle. If such bundle is // already present, assume it overwrites defaults. - bool doesTheCallAccessFPEnv = false; + bool NeedUpdateMemoryEffects = false; if (IsFPConstrained) { if (const auto *Func = dyn_cast<Function>(Callee)) { if (Intrinsic::ID ID = Func->getIntrinsicID()) { if (IntrinsicInst::isFloatingPointOperation(ID)) { - doesTheCallAccessFPEnv = true; + MemoryEffects FME = Func->getMemoryEffects(); + NeedUpdateMemoryEffects = !FME.doesAccessInaccessibleMem(); bool NeedRound = DefaultConstrainedRounding != RoundingMode::Dynamic; bool NeedExcept = DefaultConstrainedExcept != fp::ebStrict; for (const auto &Item : OpBundles) { @@ -166,10 +167,8 @@ CallInst *IRBuilderBase::CreateCall(FunctionType *FTy, Value *Callee, // If the call accesses FPE, update memory effects accordingly. CallInst *CI = CallInst::Create(FTy, Callee, Args, OpBundles); - if (doesTheCallAccessFPEnv) { + if (NeedUpdateMemoryEffects) { MemoryEffects ME = MemoryEffects::inaccessibleMemOnly(); - if (CI->getAttributes().hasFnAttr(Attribute::Memory)) - ME |= CI->getAttributes().getMemoryEffects(); auto A = Attribute::getWithMemoryEffects(getContext(), ME); CI->addFnAttr(A); } >From f9180b3c9d4f304772802107569eb53e4a91ac77 Mon Sep 17 00:00:00 2001 From: Serge Pavlov <sepavl...@gmail.com> Date: Thu, 17 Apr 2025 15:30:17 +0700 Subject: [PATCH 4/6] Fix method signature --- llvm/include/llvm/IR/InstrTypes.h | 6 +++--- llvm/lib/IR/Instructions.cpp | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/llvm/include/llvm/IR/InstrTypes.h b/llvm/include/llvm/IR/InstrTypes.h index adf6806216c83..8425243e5efe9 100644 --- a/llvm/include/llvm/IR/InstrTypes.h +++ b/llvm/include/llvm/IR/InstrTypes.h @@ -2165,11 +2165,11 @@ class CallBase : public Instruction { return false; } - /// Return rounding mode specified by operand bundles. + /// Return rounding mode specified for this call. RoundingMode getRoundingMode() const; - /// Return exception behavior specified by operand bundles. - std::optional<fp::ExceptionBehavior> getExceptionBehavior() const; + /// Return exception behavior specified for this call. + fp::ExceptionBehavior getExceptionBehavior() const; /// Used to keep track of an operand bundle. See the main comment on /// OperandBundleUser above. diff --git a/llvm/lib/IR/Instructions.cpp b/llvm/lib/IR/Instructions.cpp index a940dc5af2a14..0e362d79493c1 100644 --- a/llvm/lib/IR/Instructions.cpp +++ b/llvm/lib/IR/Instructions.cpp @@ -647,7 +647,7 @@ RoundingMode CallBase::getRoundingMode() const { return RoundingMode::NearestTiesToEven; } -std::optional<fp::ExceptionBehavior> CallBase::getExceptionBehavior() const { +fp::ExceptionBehavior CallBase::getExceptionBehavior() const { // Try determining exception behavior from FP bundle. std::optional<fp::ExceptionBehavior> EB; if (auto ExceptionBundle = getOperandBundle(LLVMContext::OB_fp_except)) { >From add26fef01a93ddf81cd3410b13b1b2c66a5f9e9 Mon Sep 17 00:00:00 2001 From: Serge Pavlov <sepavl...@gmail.com> Date: Fri, 18 Apr 2025 01:55:35 +0700 Subject: [PATCH 5/6] Organize checks in Verifier --- llvm/lib/IR/Verifier.cpp | 52 ++++++++++++++++++----------- llvm/test/Verifier/fp-intrinsics.ll | 22 ++++++------ 2 files changed, 44 insertions(+), 30 deletions(-) diff --git a/llvm/lib/IR/Verifier.cpp b/llvm/lib/IR/Verifier.cpp index 747cc5b9ca552..6b7644fef9e71 100644 --- a/llvm/lib/IR/Verifier.cpp +++ b/llvm/lib/IR/Verifier.cpp @@ -3830,32 +3830,46 @@ void Verifier::visitCallBase(CallBase &Call) { FoundAttachedCallBundle = true; verifyAttachedCallBundle(Call, BU); } else if (Tag == LLVMContext::OB_fp_control) { - Check(!FoundFpeControlBundle, "Multiple fp.control operand bundles", - Call); - Check(BU.Inputs.size() == 1, - "Expected exactly one fp.control bundle operand", Call); - auto *V = dyn_cast<MetadataAsValue>(BU.Inputs.front()); - Check(V, "Value of fp.control bundle operand must be a metadata", Call); - auto *MDS = dyn_cast<MDString>(V->getMetadata()); - Check(MDS, "Value of fp.control bundle operand must be a string", Call); - auto RM = convertBundleToRoundingMode(MDS->getString()); - Check(RM.has_value(), - "Value of fp.control bundle operand is not a correct rounding mode", + Check(!FoundFpeControlBundle, "Multiple \"fp.control\" operand bundles", Call); + bool FoundRoundingMode = false; + for (auto &U : BU.Inputs) { + Value *V = U.get(); + Check(isa<MetadataAsValue>(V), + "Value of a \"fp.control\" bundle operand must be a metadata", + Call); + Metadata *MD = cast<MetadataAsValue>(V)->getMetadata(); + Check(isa<MDString>(MD), + "Value of a \"fp.control\" bundle operand must be a string", + Call); + StringRef Item = cast<MDString>(MD)->getString(); + if (convertBundleToRoundingMode(Item)) { + Check(!FoundRoundingMode, "Rounding mode is specified more that once", + Call); + FoundRoundingMode = true; + } else { + CheckFailed("Unrecognized value in \"fp.control\" bundle operand", + Call); + } + } FoundFpeControlBundle = true; } else if (Tag == LLVMContext::OB_fp_except) { - Check(!FoundFpeExceptBundle, "Multiple fp.except operand bundles", Call); + Check(!FoundFpeExceptBundle, "Multiple \"fp.except\" operand bundles", + Call); Check(BU.Inputs.size() == 1, - "Expected exactly one fp.except bundle operand", Call); + "Expected exactly one \"fp.except\" bundle operand", Call); auto *V = dyn_cast<MetadataAsValue>(BU.Inputs.front()); - Check(V, "Value of fp.except bundle operand must be a metadata", Call); + Check(V, "Value of a \"fp.except\" bundle operand must be a metadata", + Call); auto *MDS = dyn_cast<MDString>(V->getMetadata()); - Check(MDS, "Value of fp.except bundle operand must be a string", Call); - auto EB = convertBundleToExceptionBehavior(MDS->getString()); - Check(EB.has_value(), - "Value of fp.except bundle operand is not a correct exception " - "behavior", + Check(MDS, "Value of a \"fp.except\" bundle operand must be a string", Call); + auto EB = convertBundleToExceptionBehavior(MDS->getString()); + Check( + EB.has_value(), + "Value of a \"fp.except\" bundle operand is not a correct exception " + "behavior", + Call); FoundFpeExceptBundle = true; } } diff --git a/llvm/test/Verifier/fp-intrinsics.ll b/llvm/test/Verifier/fp-intrinsics.ll index 12b0fca97e5af..1e1df1b0da96a 100644 --- a/llvm/test/Verifier/fp-intrinsics.ll +++ b/llvm/test/Verifier/fp-intrinsics.ll @@ -52,7 +52,7 @@ entry: } ; Test multiple fp.control bundles. -; CHECK-NEXT: Multiple fp.control operand bundles +; CHECK-NEXT: Multiple "fp.control" operand bundles ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz"), "fp.control"(metadata !"rtz") ] define double @f6(double %a) #0 { entry: @@ -60,8 +60,8 @@ entry: ret double %ftrunc } -; Test fp.control bundle that has more than one operands. -; CHECK-NEXT: Expected exactly one fp.control bundle operand +; Test fp.control bundle that has more than one rounding mode specification. +; CHECK-NEXT: Rounding mode is specified more that once ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"rtz", metadata !"rte") ] define double @f7(double %a) #0 { entry: @@ -70,7 +70,7 @@ entry: } ; Test fp.control bundle that has non-metadata operand. -; CHECK-NEXT: Value of fp.control bundle operand must be a metadata +; CHECK-NEXT: Value of a "fp.control" bundle operand must be a metadata ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(i32 0) ] define double @f8(double %a) #0 { entry: @@ -79,7 +79,7 @@ entry: } ; Test fp.control bundle that has non-string operand. -; CHECK-NEXT: Value of fp.control bundle operand must be a string +; CHECK-NEXT: Value of a "fp.control" bundle operand must be a string ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata i64 3) ] define double @f9(double %a) #0 { entry: @@ -88,7 +88,7 @@ entry: } ; Test fp.control bundle that specifies incorrect value. -; CHECK-NEXT: Value of fp.control bundle operand is not a correct rounding mode +; CHECK-NEXT: Unrecognized value in "fp.control" bundle operand ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.control"(metadata !"qqq") ] define double @f10(double %a) #0 { entry: @@ -97,7 +97,7 @@ entry: } ; Test multiple fp.except bundles. -; CHECK-NEXT: Multiple fp.except operand bundles +; CHECK-NEXT: Multiple "fp.except" operand bundles ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"strict"), "fp.except"(metadata !"strict") ] define double @f11(double %a) #0 { entry: @@ -106,7 +106,7 @@ entry: } ; Test fp.except bundle that has more than one operands. -; CHECK-NEXT: Expected exactly one fp.except bundle operand +; CHECK-NEXT: Expected exactly one "fp.except" bundle operand ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"strict", metadata !"strict") ] define double @f12(double %a) #0 { entry: @@ -115,7 +115,7 @@ entry: } ; Test fp.except bundle that has non-metadata operand. -; CHECK-NEXT: Value of fp.except bundle operand must be a metadata +; CHECK-NEXT: Value of a "fp.except" bundle operand must be a metadata ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(i32 0) ] define double @f13(double %a) #0 { entry: @@ -124,7 +124,7 @@ entry: } ; Test fp.except bundle that has non-string operand. -; CHECK-NEXT: Value of fp.except bundle operand must be a string +; CHECK-NEXT: Value of a "fp.except" bundle operand must be a string ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata i64 3) ] define double @f14(double %a) #0 { entry: @@ -133,7 +133,7 @@ entry: } ; Test fp.except bundle that specifies incorrect value. -; CHECK-NEXT: Value of fp.except bundle operand is not a correct exception behavior +; CHECK-NEXT: Value of a "fp.except" bundle operand is not a correct exception behavior ; CHECK-NEXT: %ftrunc = call double @llvm.trunc.f64(double %a) #{{[0-9]+}} [ "fp.except"(metadata !"qqq") ] define double @f15(double %a) #0 { entry: >From ad480e9de4015a9acc17b80d9f2d6ce5f39d13e5 Mon Sep 17 00:00:00 2001 From: Serge Pavlov <sepavl...@gmail.com> Date: Sat, 19 Apr 2025 02:05:28 +0700 Subject: [PATCH 6/6] Small changes in documentation --- llvm/docs/LangRef.rst | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/llvm/docs/LangRef.rst b/llvm/docs/LangRef.rst index 9145c4c9092f1..8252971aa8a58 100644 --- a/llvm/docs/LangRef.rst +++ b/llvm/docs/LangRef.rst @@ -3085,8 +3085,9 @@ floating-point control modes and the treatment of status bits respectively. An operand bundle tagged with "fp.control" contains information about the control modes used for the operation execution. Operands specified in this bundle represent particular options. Currently, only rounding mode is supported. -It is represented by a metadata string value, which specifies the rounding mode -to be used for the operation evaluation. Possible values are: + +Rounding mode is represented by a metadata string value, which specifies the +the mode used for the operation evaluation. Possible values are: :: @@ -3097,10 +3098,10 @@ to be used for the operation evaluation. Possible values are: "rmm" - to nearest, ties away from zero "dyn" - rounding mode is taken from control register -If "fp.control" is absent, the default rounding rounding mode is taken from the -control register (dynamic rounding). In the particular case of -:ref:`default floating-point environment <floatenv>`, it must be rounding to -nearest, ties to even. +Only one such value may be specified. If "fp.control" is absent, the default +rounding rounding mode is taken from the control register (dynamic rounding). +In the particular case of :ref:`default floating-point environment <floatenv>`, +the operation uses rounding to nearest, ties to even. An operand bundle tagged with "fp.except" may be associated with operations that can read or write floating-point exception flags. It contains a single _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
