[llvm-branch-commits] [llvm][IR] Extend BranchWeightMetadata to track provenance of weights (PR #86609)

2024-06-05 Thread David Li via llvm-branch-commits 


@@ -55,6 +55,17 @@ MDNode *getBranchWeightMDNode(const Instruction &I);
 /// Nullptr otherwise.
 MDNode *getValidBranchWeightMDNode(const Instruction &I);
 
+/// Check if Branch Weight Metadata has an "expected" field from an 
llvm.expect*
+/// intrinsic
+bool hasBranchWeightProvenance(const Instruction &I);

david-xl wrote:

Is it clearer to name it 'IsBranchWeightUserExpected'? 

https://github.com/llvm/llvm-project/pull/86609
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm][IR] Extend BranchWeightMetadata to track provenance of weights (PR #86609)

2024-06-05 Thread David Li via llvm-branch-commits 

https://github.com/david-xl edited 
https://github.com/llvm/llvm-project/pull/86609
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm][IR] Extend BranchWeightMetadata to track provenance of weights (PR #86609)

2024-06-05 Thread David Li via llvm-branch-commits 


@@ -123,6 +121,25 @@ bool hasValidBranchWeightMD(const Instruction &I) {
   return getValidBranchWeightMDNode(I);
 }
 
+bool hasBranchWeightProvenance(const Instruction &I) {
+  auto *ProfileData = I.getMetadata(LLVMContext::MD_prof);
+  return hasBranchWeightProvenance(ProfileData);
+}
+
+bool hasBranchWeightProvenance(const MDNode *ProfileData) {
+  if (!isBranchWeightMD(ProfileData))
+return false;
+  auto *ProfDataName = dyn_cast(ProfileData->getOperand(1));
+  // NOTE: if we ever have more types of branch weight provenance,
+  // we need to check the string value is "expected". For now, we
+  // supply a more generic API, and avoid the spurious comparisons.
+  return ProfDataName;

david-xl wrote:

add a debug assert for now?

https://github.com/llvm/llvm-project/pull/86609
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm][IR] Extend BranchWeightMetadata to track provenance of weights (PR #86609)

2024-06-06 Thread David Li via llvm-branch-commits 


@@ -55,6 +55,17 @@ MDNode *getBranchWeightMDNode(const Instruction &I);
 /// Nullptr otherwise.
 MDNode *getValidBranchWeightMDNode(const Instruction &I);
 
+/// Check if Branch Weight Metadata has an "expected" field from an 
llvm.expect*
+/// intrinsic
+bool hasBranchWeightProvenance(const Instruction &I);

david-xl wrote:

hasBranchWeightOrigin sounds good to me -- it is straightforward for the reader 
to understand.

https://github.com/llvm/llvm-project/pull/86609
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] [Inline][PGO] After inline, update profile for invoke instruction in both cloned instruction in the caller and original callee (PR #83809)

2024-03-04 Thread David Li via llvm-branch-commits 


@@ -918,6 +918,18 @@ LandingPadInst *InvokeInst::getLandingPadInst() const {
   return cast(getUnwindDest()->getFirstNonPHI());
 }
 
+void InvokeInst::updateProfWeight(uint64_t S, uint64_t T) {

david-xl wrote:

should this be moved to CallBase?

https://github.com/llvm/llvm-project/pull/83809
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] [Inline][PGO] After inline, update profile for invoke instruction in both cloned instruction in the caller and original callee (PR #83809)

2024-03-05 Thread David Li via llvm-branch-commits 

https://github.com/david-xl commented:

The invoke instruction can have 3 different kinds of prof data
1) call count (if a direct call)
2) VP profile data (if an indirect call)
3) branch weights for landing pad.

3) can coexist with  2) and does not need to be updated. Is there an existing 
test coverage for type 1) update?

https://github.com/llvm/llvm-project/pull/83809
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm][IR] Extend BranchWeightMetadata to track provenance of weights (PR #86609)

2024-04-05 Thread David Li via llvm-branch-commits 


@@ -1210,12 +1210,22 @@ Instruction *Instruction::cloneImpl() const {
 
 void Instruction::swapProfMetadata() {
   MDNode *ProfileData = getBranchWeightMDNode(*this);
-  if (!ProfileData || ProfileData->getNumOperands() != 3)
+  if (!isBranchWeightMD(ProfileData))

david-xl wrote:

Extract this and other similar refactoring change into a different patch?

https://github.com/llvm/llvm-project/pull/86609
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] [ctx_prof] Flattened profile lowering pass (PR #107329)

2024-09-05 Thread David Li via llvm-branch-commits 


@@ -0,0 +1,341 @@
+//===- PGOCtxProfFlattening.cpp - Contextual Instr. Flattening 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// Flattens the contextual profile and lowers it to MD_prof.
+// This should happen after all IPO (which is assumed to have maintained the
+// contextual profile) happened. Flattening consists of summing the values at
+// the same index of the counters belonging to all the contexts of a function.
+// The lowering consists of materializing the counter values to function
+// entrypoint counts and branch probabilities.
+//
+// This pass also removes contextual instrumentation, which has been kept 
around
+// to facilitate its functionality.
+//
+//===--===//
+
+#include "llvm/Transforms/Instrumentation/PGOCtxProfFlattening.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/Analysis/CtxProfAnalysis.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/CFG.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/IR/ProfileSummary.h"
+#include "llvm/ProfileData/ProfileCommon.h"
+#include "llvm/Transforms/Instrumentation/PGOInstrumentation.h"
+#include "llvm/Transforms/Scalar/DCE.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
+
+using namespace llvm;
+
+namespace {
+
+class ProfileAnnotator final {
+  class BBInfo;
+  struct EdgeInfo {
+BBInfo *const Src;
+BBInfo *const Dest;
+std::optional Count;
+
+explicit EdgeInfo(BBInfo &Src, BBInfo &Dest) : Src(&Src), Dest(&Dest) {}
+  };
+
+  class BBInfo {
+std::optional Count;
+SmallVector OutEdges;
+SmallVector InEdges;
+size_t UnknownCountOutEdges = 0;
+size_t UnknownCountInEdges = 0;
+
+uint64_t getEdgeSum(const SmallVector &Edges,
+bool AssumeAllKnown) const {
+  uint64_t Sum = 0;
+  for (const auto *E : Edges)
+if (E)

david-xl wrote:

Why can E be null?

https://github.com/llvm/llvm-project/pull/107329
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] [ctx_prof] Flattened profile lowering pass (PR #107329)

2024-09-05 Thread David Li via llvm-branch-commits 


@@ -0,0 +1,341 @@
+//===- PGOCtxProfFlattening.cpp - Contextual Instr. Flattening 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// Flattens the contextual profile and lowers it to MD_prof.
+// This should happen after all IPO (which is assumed to have maintained the
+// contextual profile) happened. Flattening consists of summing the values at
+// the same index of the counters belonging to all the contexts of a function.
+// The lowering consists of materializing the counter values to function
+// entrypoint counts and branch probabilities.
+//
+// This pass also removes contextual instrumentation, which has been kept 
around
+// to facilitate its functionality.
+//
+//===--===//
+
+#include "llvm/Transforms/Instrumentation/PGOCtxProfFlattening.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/Analysis/CtxProfAnalysis.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/CFG.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/IR/ProfileSummary.h"
+#include "llvm/ProfileData/ProfileCommon.h"
+#include "llvm/Transforms/Instrumentation/PGOInstrumentation.h"
+#include "llvm/Transforms/Scalar/DCE.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
+
+using namespace llvm;
+
+namespace {
+
+class ProfileAnnotator final {
+  class BBInfo;
+  struct EdgeInfo {
+BBInfo *const Src;
+BBInfo *const Dest;
+std::optional Count;
+
+explicit EdgeInfo(BBInfo &Src, BBInfo &Dest) : Src(&Src), Dest(&Dest) {}
+  };
+
+  class BBInfo {
+std::optional Count;
+SmallVector OutEdges;
+SmallVector InEdges;
+size_t UnknownCountOutEdges = 0;
+size_t UnknownCountInEdges = 0;
+
+uint64_t getEdgeSum(const SmallVector &Edges,
+bool AssumeAllKnown) const {
+  uint64_t Sum = 0;
+  for (const auto *E : Edges)
+if (E)
+  Sum += AssumeAllKnown ? *E->Count : E->Count.value_or(0U);
+  return Sum;
+}
+
+void takeCountFrom(const SmallVector &Edges) {

david-xl wrote:

nit: take-> compute

https://github.com/llvm/llvm-project/pull/107329
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] [ctx_prof] Flattened profile lowering pass (PR #107329)

2024-09-05 Thread David Li via llvm-branch-commits 


@@ -0,0 +1,333 @@
+//===- PGOCtxProfFlattening.cpp - Contextual Instr. Flattening 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// Flattens the contextual profile and lowers it to MD_prof.
+// This should happen after all IPO (which is assumed to have maintained the
+// contextual profile) happened. Flattening consists of summing the values at
+// the same index of the counters belonging to all the contexts of a function.
+// The lowering consists of materializing the counter values to function
+// entrypoint counts and branch probabilities.
+//
+// This pass also removes contextual instrumentation, which has been kept 
around
+// to facilitate its functionality.
+//
+//===--===//
+
+#include "llvm/Transforms/Instrumentation/PGOCtxProfFlattening.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Analysis/CtxProfAnalysis.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/CFG.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/IR/ProfileSummary.h"
+#include "llvm/ProfileData/ProfileCommon.h"
+#include "llvm/Transforms/Instrumentation/PGOInstrumentation.h"
+#include "llvm/Transforms/Scalar/DCE.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
+
+using namespace llvm;
+
+namespace {
+
+class ProfileAnnotator final {
+  class BBInfo;
+  struct EdgeInfo {
+BBInfo *const Src;
+BBInfo *const Dest;
+std::optional Count;
+
+explicit EdgeInfo(BBInfo &Src, BBInfo &Dest) : Src(&Src), Dest(&Dest) {}
+  };
+
+  class BBInfo {
+std::optional Count;
+SmallVector OutEdges;
+SmallVector InEdges;
+size_t UnknownCountOutEdges = 0;
+size_t UnknownCountInEdges = 0;
+
+uint64_t getEdgeSum(const SmallVector &Edges,
+bool AssumeAllKnown) const {
+  uint64_t Sum = 0;
+  for (const auto *E : Edges)
+if (E)
+  Sum += AssumeAllKnown ? *E->Count : E->Count.value_or(0U);
+  return Sum;
+}
+
+void takeCountFrom(const SmallVector &Edges) {
+  assert(!Count.has_value());
+  Count = getEdgeSum(Edges, true);
+}
+
+void setSingleUnknownEdgeCount(SmallVector &Edges) {
+  uint64_t KnownSum = getEdgeSum(Edges, false);
+  uint64_t EdgeVal = *Count > KnownSum ? *Count - KnownSum : 0U;
+  EdgeInfo *E = nullptr;
+  for (auto *I : Edges)
+if (I && !I->Count.has_value()) {
+  E = I;
+#ifdef NDEBUG
+  break;
+#else
+  assert((!E || E == I) &&
+ "Expected exactly one edge to have an unknown count, "
+ "found a second one");
+  continue;
+#endif
+}
+  assert(E && "Expected exactly one edge to have an unknown count");
+  assert(!E->Count.has_value());
+  E->Count = EdgeVal;
+  assert(E->Src->UnknownCountOutEdges > 0);
+  assert(E->Dest->UnknownCountInEdges > 0);
+  --E->Src->UnknownCountOutEdges;
+  --E->Dest->UnknownCountInEdges;
+}
+
+  public:
+BBInfo(size_t NumInEdges, size_t NumOutEdges, std::optional 
Count)
+: Count(Count) {
+  InEdges.reserve(NumInEdges);
+  OutEdges.resize(NumOutEdges);
+}
+
+bool tryTakeCountFromKnownOutEdges(const BasicBlock &BB) {
+  if (!succ_empty(&BB) && !UnknownCountOutEdges) {
+takeCountFrom(OutEdges);
+return true;
+  }
+  return false;
+}
+
+bool tryTakeCountFromKnownInEdges(const BasicBlock &BB) {
+  if (!BB.isEntryBlock() && !UnknownCountInEdges) {
+takeCountFrom(InEdges);
+return true;
+  }
+  return false;
+}
+
+void addInEdge(EdgeInfo *Info) {
+  InEdges.push_back(Info);
+  ++UnknownCountInEdges;
+}
+
+void addOutEdge(size_t Index, EdgeInfo *Info) {
+  OutEdges[Index] = Info;
+  ++UnknownCountOutEdges;
+}
+
+bool hasCount() const { return Count.has_value(); }
+
+bool trySetSingleUnknownInEdgeCount() {
+  if (UnknownCountInEdges == 1) {
+setSingleUnknownEdgeCount(InEdges);
+return true;
+  }
+  return false;
+}
+
+bool trySetSingleUnknownOutEdgeCount() {
+  if (UnknownCountOutEdges == 1) {
+setSingleUnknownEdgeCount(OutEdges);
+return true;
+  }
+  return false;
+}
+size_t getNumOutEdges() const { return OutEdges.size(); }
+
+uint64_t getEdgeCount(size_t Index) const {
+  if (auto *E = OutEdges[Index])
+return *E->Count;
+  return 0U;
+}
+  };
+
+  F

[llvm-branch-commits] [llvm] [ctx_prof] Flattened profile lowering pass (PR #107329)

2024-09-05 Thread David Li via llvm-branch-commits 


@@ -0,0 +1,341 @@
+//===- PGOCtxProfFlattening.cpp - Contextual Instr. Flattening 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// Flattens the contextual profile and lowers it to MD_prof.
+// This should happen after all IPO (which is assumed to have maintained the
+// contextual profile) happened. Flattening consists of summing the values at
+// the same index of the counters belonging to all the contexts of a function.
+// The lowering consists of materializing the counter values to function
+// entrypoint counts and branch probabilities.
+//
+// This pass also removes contextual instrumentation, which has been kept 
around
+// to facilitate its functionality.
+//
+//===--===//
+
+#include "llvm/Transforms/Instrumentation/PGOCtxProfFlattening.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/ADT/ScopeExit.h"
+#include "llvm/Analysis/CtxProfAnalysis.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/CFG.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/IR/ProfileSummary.h"
+#include "llvm/ProfileData/ProfileCommon.h"
+#include "llvm/Transforms/Instrumentation/PGOInstrumentation.h"
+#include "llvm/Transforms/Scalar/DCE.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
+
+using namespace llvm;
+
+namespace {
+
+class ProfileAnnotator final {
+  class BBInfo;
+  struct EdgeInfo {
+BBInfo *const Src;
+BBInfo *const Dest;
+std::optional Count;
+
+explicit EdgeInfo(BBInfo &Src, BBInfo &Dest) : Src(&Src), Dest(&Dest) {}
+  };
+
+  class BBInfo {
+std::optional Count;
+SmallVector OutEdges;
+SmallVector InEdges;
+size_t UnknownCountOutEdges = 0;
+size_t UnknownCountInEdges = 0;
+
+uint64_t getEdgeSum(const SmallVector &Edges,

david-xl wrote:

brief document when to assumeAllKnown.

https://github.com/llvm/llvm-project/pull/107329
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] [ctx_prof] Flattened profile lowering pass (PR #107329)

2024-09-05 Thread David Li via llvm-branch-commits 


@@ -0,0 +1,333 @@
+//===- PGOCtxProfFlattening.cpp - Contextual Instr. Flattening 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// Flattens the contextual profile and lowers it to MD_prof.
+// This should happen after all IPO (which is assumed to have maintained the
+// contextual profile) happened. Flattening consists of summing the values at
+// the same index of the counters belonging to all the contexts of a function.
+// The lowering consists of materializing the counter values to function
+// entrypoint counts and branch probabilities.
+//
+// This pass also removes contextual instrumentation, which has been kept 
around
+// to facilitate its functionality.
+//
+//===--===//
+
+#include "llvm/Transforms/Instrumentation/PGOCtxProfFlattening.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Analysis/CtxProfAnalysis.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/CFG.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/IR/ProfileSummary.h"
+#include "llvm/ProfileData/ProfileCommon.h"
+#include "llvm/Transforms/Instrumentation/PGOInstrumentation.h"
+#include "llvm/Transforms/Scalar/DCE.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
+
+using namespace llvm;
+
+namespace {
+
+class ProfileAnnotator final {
+  class BBInfo;
+  struct EdgeInfo {
+BBInfo *const Src;
+BBInfo *const Dest;
+std::optional Count;
+
+explicit EdgeInfo(BBInfo &Src, BBInfo &Dest) : Src(&Src), Dest(&Dest) {}
+  };
+
+  class BBInfo {
+std::optional Count;
+SmallVector OutEdges;
+SmallVector InEdges;
+size_t UnknownCountOutEdges = 0;
+size_t UnknownCountInEdges = 0;
+
+uint64_t getEdgeSum(const SmallVector &Edges,
+bool AssumeAllKnown) const {
+  uint64_t Sum = 0;
+  for (const auto *E : Edges)
+if (E)
+  Sum += AssumeAllKnown ? *E->Count : E->Count.value_or(0U);
+  return Sum;
+}
+
+void takeCountFrom(const SmallVector &Edges) {
+  assert(!Count.has_value());
+  Count = getEdgeSum(Edges, true);
+}
+
+void setSingleUnknownEdgeCount(SmallVector &Edges) {
+  uint64_t KnownSum = getEdgeSum(Edges, false);
+  uint64_t EdgeVal = *Count > KnownSum ? *Count - KnownSum : 0U;
+  EdgeInfo *E = nullptr;
+  for (auto *I : Edges)
+if (I && !I->Count.has_value()) {
+  E = I;
+#ifdef NDEBUG
+  break;
+#else
+  assert((!E || E == I) &&
+ "Expected exactly one edge to have an unknown count, "
+ "found a second one");
+  continue;
+#endif
+}
+  assert(E && "Expected exactly one edge to have an unknown count");
+  assert(!E->Count.has_value());
+  E->Count = EdgeVal;
+  assert(E->Src->UnknownCountOutEdges > 0);
+  assert(E->Dest->UnknownCountInEdges > 0);
+  --E->Src->UnknownCountOutEdges;
+  --E->Dest->UnknownCountInEdges;
+}
+
+  public:
+BBInfo(size_t NumInEdges, size_t NumOutEdges, std::optional 
Count)
+: Count(Count) {
+  InEdges.reserve(NumInEdges);
+  OutEdges.resize(NumOutEdges);
+}
+
+bool tryTakeCountFromKnownOutEdges(const BasicBlock &BB) {
+  if (!succ_empty(&BB) && !UnknownCountOutEdges) {
+takeCountFrom(OutEdges);
+return true;
+  }
+  return false;
+}
+
+bool tryTakeCountFromKnownInEdges(const BasicBlock &BB) {
+  if (!BB.isEntryBlock() && !UnknownCountInEdges) {
+takeCountFrom(InEdges);
+return true;
+  }
+  return false;
+}
+
+void addInEdge(EdgeInfo *Info) {
+  InEdges.push_back(Info);
+  ++UnknownCountInEdges;
+}
+
+void addOutEdge(size_t Index, EdgeInfo *Info) {
+  OutEdges[Index] = Info;
+  ++UnknownCountOutEdges;
+}
+
+bool hasCount() const { return Count.has_value(); }
+
+bool trySetSingleUnknownInEdgeCount() {
+  if (UnknownCountInEdges == 1) {
+setSingleUnknownEdgeCount(InEdges);
+return true;
+  }
+  return false;
+}
+
+bool trySetSingleUnknownOutEdgeCount() {
+  if (UnknownCountOutEdges == 1) {
+setSingleUnknownEdgeCount(OutEdges);
+return true;
+  }
+  return false;
+}
+size_t getNumOutEdges() const { return OutEdges.size(); }
+
+uint64_t getEdgeCount(size_t Index) const {
+  if (auto *E = OutEdges[Index])
+return *E->Count;
+  return 0U;
+}
+  };
+
+  F

[llvm-branch-commits] [llvm] [ctx_prof] Flattened profile lowering pass (PR #107329)

2024-09-05 Thread David Li via llvm-branch-commits 


@@ -0,0 +1,333 @@
+//===- PGOCtxProfFlattening.cpp - Contextual Instr. Flattening 
===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM 
Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===--===//
+//
+// Flattens the contextual profile and lowers it to MD_prof.
+// This should happen after all IPO (which is assumed to have maintained the
+// contextual profile) happened. Flattening consists of summing the values at
+// the same index of the counters belonging to all the contexts of a function.
+// The lowering consists of materializing the counter values to function
+// entrypoint counts and branch probabilities.
+//
+// This pass also removes contextual instrumentation, which has been kept 
around
+// to facilitate its functionality.
+//
+//===--===//
+
+#include "llvm/Transforms/Instrumentation/PGOCtxProfFlattening.h"
+#include "llvm/ADT/STLExtras.h"
+#include "llvm/Analysis/CtxProfAnalysis.h"
+#include "llvm/Analysis/OptimizationRemarkEmitter.h"
+#include "llvm/Analysis/ProfileSummaryInfo.h"
+#include "llvm/CodeGen/MachineBasicBlock.h"
+#include "llvm/IR/Analysis.h"
+#include "llvm/IR/CFG.h"
+#include "llvm/IR/Dominators.h"
+#include "llvm/IR/IntrinsicInst.h"
+#include "llvm/IR/Module.h"
+#include "llvm/IR/PassManager.h"
+#include "llvm/IR/ProfileSummary.h"
+#include "llvm/ProfileData/ProfileCommon.h"
+#include "llvm/Transforms/Instrumentation/PGOInstrumentation.h"
+#include "llvm/Transforms/Scalar/DCE.h"
+#include "llvm/Transforms/Utils/BasicBlockUtils.h"
+
+using namespace llvm;
+
+namespace {
+
+class ProfileAnnotator final {
+  class BBInfo;
+  struct EdgeInfo {
+BBInfo *const Src;
+BBInfo *const Dest;
+std::optional Count;
+
+explicit EdgeInfo(BBInfo &Src, BBInfo &Dest) : Src(&Src), Dest(&Dest) {}
+  };
+
+  class BBInfo {
+std::optional Count;
+SmallVector OutEdges;
+SmallVector InEdges;
+size_t UnknownCountOutEdges = 0;
+size_t UnknownCountInEdges = 0;
+
+uint64_t getEdgeSum(const SmallVector &Edges,
+bool AssumeAllKnown) const {
+  uint64_t Sum = 0;
+  for (const auto *E : Edges)
+if (E)
+  Sum += AssumeAllKnown ? *E->Count : E->Count.value_or(0U);
+  return Sum;
+}
+
+void takeCountFrom(const SmallVector &Edges) {
+  assert(!Count.has_value());
+  Count = getEdgeSum(Edges, true);
+}
+
+void setSingleUnknownEdgeCount(SmallVector &Edges) {
+  uint64_t KnownSum = getEdgeSum(Edges, false);
+  uint64_t EdgeVal = *Count > KnownSum ? *Count - KnownSum : 0U;
+  EdgeInfo *E = nullptr;
+  for (auto *I : Edges)
+if (I && !I->Count.has_value()) {
+  E = I;
+#ifdef NDEBUG
+  break;
+#else
+  assert((!E || E == I) &&
+ "Expected exactly one edge to have an unknown count, "
+ "found a second one");
+  continue;
+#endif
+}
+  assert(E && "Expected exactly one edge to have an unknown count");
+  assert(!E->Count.has_value());
+  E->Count = EdgeVal;
+  assert(E->Src->UnknownCountOutEdges > 0);
+  assert(E->Dest->UnknownCountInEdges > 0);
+  --E->Src->UnknownCountOutEdges;
+  --E->Dest->UnknownCountInEdges;
+}
+
+  public:
+BBInfo(size_t NumInEdges, size_t NumOutEdges, std::optional 
Count)
+: Count(Count) {
+  InEdges.reserve(NumInEdges);
+  OutEdges.resize(NumOutEdges);
+}
+
+bool tryTakeCountFromKnownOutEdges(const BasicBlock &BB) {
+  if (!succ_empty(&BB) && !UnknownCountOutEdges) {
+takeCountFrom(OutEdges);
+return true;
+  }
+  return false;
+}
+
+bool tryTakeCountFromKnownInEdges(const BasicBlock &BB) {
+  if (!BB.isEntryBlock() && !UnknownCountInEdges) {
+takeCountFrom(InEdges);
+return true;
+  }
+  return false;
+}
+
+void addInEdge(EdgeInfo *Info) {
+  InEdges.push_back(Info);
+  ++UnknownCountInEdges;
+}
+
+void addOutEdge(size_t Index, EdgeInfo *Info) {
+  OutEdges[Index] = Info;
+  ++UnknownCountOutEdges;
+}
+
+bool hasCount() const { return Count.has_value(); }
+
+bool trySetSingleUnknownInEdgeCount() {
+  if (UnknownCountInEdges == 1) {
+setSingleUnknownEdgeCount(InEdges);
+return true;
+  }
+  return false;
+}
+
+bool trySetSingleUnknownOutEdgeCount() {
+  if (UnknownCountOutEdges == 1) {
+setSingleUnknownEdgeCount(OutEdges);
+return true;
+  }
+  return false;
+}
+size_t getNumOutEdges() const { return OutEdges.size(); }
+
+uint64_t getEdgeCount(size_t Index) const {
+  if (auto *E = OutEdges[Index])
+return *E->Count;
+  return 0U;
+}
+  };
+
+  F

[llvm-branch-commits] [llvm] [ctx_prof] Flattened profile lowering pass (PR #107329)

2024-09-06 Thread David Li via llvm-branch-commits 

https://github.com/david-xl approved this pull request.


https://github.com/llvm/llvm-project/pull/107329
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] [CodeGen][StaticDataSplitter]Support constant pool partitioning (PR #129781)

2025-03-25 Thread David Li via llvm-branch-commits 

https://github.com/david-xl approved this pull request.


https://github.com/llvm/llvm-project/pull/129781
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits

[llvm-branch-commits] [llvm] [CodeGen][StaticDataSplitter]Support constant pool partitioning (PR #129781)

2025-03-07 Thread David Li via llvm-branch-commits 


@@ -386,6 +386,16 @@ MCSection *TargetLoweringObjectFile::getSectionForConstant(
   return DataSection;
 }
 
+MCSection *TargetLoweringObjectFile::getSectionForConstant(
+const DataLayout &DL, SectionKind Kind, const Constant *C, Align 
&Alignment,
+StringRef SectionPrefix) const {
+  // Fallback to `getSectionForConstant` without `SectionPrefix` parameter if 
it
+  // is empty.
+  if (SectionPrefix.empty())

david-xl wrote:

current implementation is fine. Perhaps improve the message in 
report_fatal_error like "Not implemented for the object format".

https://github.com/llvm/llvm-project/pull/129781
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits