[llvm-branch-commits] [mlir] [mlir][SCF] `ValueBoundsConstraintSet`: Support `scf.if` (branches) (PR #85895)
@@ -199,6 +199,28 @@ class ValueBoundsConstraintSet {
std::optional dim1 = std::nullopt,
std::optional dim2 = std::nullopt);
+ /// Traverse the IR starting from the given value/dim and add populate
dcaballe wrote:
typo: add constraints or populate constraints but not both?
https://github.com/llvm/llvm-project/pull/85895
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][SCF] `ValueBoundsConstraintSet`: Support `scf.if` (branches) (PR #85895)
@@ -575,6 +575,68 @@ ValueBoundsConstraintSet::computeConstantDelta(Value
value1, Value value2,
{{value1, dim1}, {value2, dim2}});
}
+void ValueBoundsConstraintSet::populateConstraints(Value value,
+ std::optional dim)
{
+ // `getExpr` pushes the value/dim onto the worklist (unless it was already
+ // analyzed).
+ (void)getExpr(value, dim);
+ // Process all values/dims on the worklist. This may traverse and analyze
+ // additional IR, depending the current stop function.
+ processWorklist();
+}
+
+bool ValueBoundsConstraintSet::compare(Value value1,
+ std::optional dim1,
+ ComparisonOperator cmp, Value value2,
+ std::optional dim2) {
+ // This function returns "true" if value1/dim1 CMP value2/dim2 is proved to
+ // hold.
+ //
+ // Example for ComparisonOperator::LE and index-typed values: We would like
to
+ // prove that value1 <= value2. Proof by contradiction: add the inverse
+ // relation (value1 > value2) to the constraint set and check if the
resulting
+ // constraint set is "empty" (i.e. has no solution). In that case,
+ // value1 > value2 must be incorrect and we can deduce that value1 <= value2
+ // holds.
+
+ // We cannot use prove anything if the constraint set is already empty.
+ if (cstr.isEmpty()) {
+LLVM_DEBUG(
+llvm::dbgs()
+<< "cannot compare value/dims: constraint system is already empty");
+return false;
+ }
+
+ // EQ can be expressed as LE and GE.
+ if (cmp == EQ)
+return compare(value1, dim1, ComparisonOperator::LE, value2, dim2) &&
+ compare(value1, dim1, ComparisonOperator::GE, value2, dim2);
+
+ // Construct inequality. For the above example: value1 > value2.
+ // `IntegerRelation` inequalities are expressed in the "flattened" form and
+ // with ">= 0". I.e., value1 - value2 - 1 >= 0.
+ SmallVector eq(cstr.getNumDimAndSymbolVars() + 1, 0);
+ if (cmp == LT || cmp == LE) {
+eq[getPos(value1, dim1)]++;
+eq[getPos(value2, dim2)]--;
+ } else if (cmp == GT || cmp == GE) {
+eq[getPos(value1, dim1)]--;
+eq[getPos(value2, dim2)]++;
dcaballe wrote:
pre-increments per coding guidelines
https://github.com/llvm/llvm-project/pull/85895
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][SCF] `ValueBoundsConstraintSet`: Support `scf.if` (branches) (PR #85895)
@@ -575,6 +575,68 @@ ValueBoundsConstraintSet::computeConstantDelta(Value
value1, Value value2,
{{value1, dim1}, {value2, dim2}});
}
+void ValueBoundsConstraintSet::populateConstraints(Value value,
+ std::optional dim)
{
+ // `getExpr` pushes the value/dim onto the worklist (unless it was already
+ // analyzed).
+ (void)getExpr(value, dim);
+ // Process all values/dims on the worklist. This may traverse and analyze
+ // additional IR, depending the current stop function.
+ processWorklist();
+}
+
+bool ValueBoundsConstraintSet::compare(Value value1,
+ std::optional dim1,
dcaballe wrote:
nit: replace 1 and 2 with Lhs and Rhs?
https://github.com/llvm/llvm-project/pull/85895
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][SCF] `ValueBoundsConstraintSet`: Support `scf.if` (branches) (PR #85895)
https://github.com/dcaballe approved this pull request. LGTM! Just minor nits that you can address before landing. Thanks! https://github.com/llvm/llvm-project/pull/85895 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][SCF] `ValueBoundsConstraintSet`: Support `scf.if` (branches) (PR #85895)
https://github.com/dcaballe edited https://github.com/llvm/llvm-project/pull/85895 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][SCF] `ValueBoundsConstraintSet`: Support `scf.if` (branches) (PR #85895)
@@ -575,6 +575,68 @@ ValueBoundsConstraintSet::computeConstantDelta(Value
value1, Value value2,
{{value1, dim1}, {value2, dim2}});
}
+void ValueBoundsConstraintSet::populateConstraints(Value value,
+ std::optional dim)
{
+ // `getExpr` pushes the value/dim onto the worklist (unless it was already
+ // analyzed).
+ (void)getExpr(value, dim);
+ // Process all values/dims on the worklist. This may traverse and analyze
+ // additional IR, depending the current stop function.
+ processWorklist();
+}
+
+bool ValueBoundsConstraintSet::compare(Value value1,
+ std::optional dim1,
+ ComparisonOperator cmp, Value value2,
+ std::optional dim2) {
+ // This function returns "true" if value1/dim1 CMP value2/dim2 is proved to
+ // hold.
+ //
+ // Example for ComparisonOperator::LE and index-typed values: We would like
to
+ // prove that value1 <= value2. Proof by contradiction: add the inverse
+ // relation (value1 > value2) to the constraint set and check if the
resulting
+ // constraint set is "empty" (i.e. has no solution). In that case,
+ // value1 > value2 must be incorrect and we can deduce that value1 <= value2
+ // holds.
+
+ // We cannot use prove anything if the constraint set is already empty.
dcaballe wrote:
typo: use prove?
https://github.com/llvm/llvm-project/pull/85895
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] c8fc5c0 - [mlir][Affine] Add support for multi-store producer fusion
Author: Diego Caballero
Date: 2021-01-25T20:31:17+02:00
New Revision: c8fc5c0385dbb47623c1cca5efa0b96d5e5f8151
URL:
https://github.com/llvm/llvm-project/commit/c8fc5c0385dbb47623c1cca5efa0b96d5e5f8151
DIFF:
https://github.com/llvm/llvm-project/commit/c8fc5c0385dbb47623c1cca5efa0b96d5e5f8151.diff
LOG: [mlir][Affine] Add support for multi-store producer fusion
This patch adds support for producer-consumer fusion scenarios with
multiple producer stores to the AffineLoopFusion pass. The patch
introduces some changes to the producer-consumer algorithm, including:
* For a given consumer loop, producer-consumer fusion iterates over its
producer candidates until a fixed point is reached.
* Producer candidates are gathered beforehand for each iteration of the
consumer loop and visited in reverse program order (not strictly guaranteed)
to maximize the number of loops fused per iteration.
In general, these changes were needed to simplify the multi-store producer
support and remove some of the workarounds that were introduced in the past
to support more fusion cases under the single-store producer limitation.
This patch also preserves the existing functionality of AffineLoopFusion with
one minor change in behavior. Producer-consumer fusion didn't fuse scenarios
with escaping memrefs and multiple outgoing edges (from a single store).
Multi-store producer scenarios will usually (always?) have multiple outgoing
edges so we couldn't fuse any with escaping memrefs, which would greatly limit
the applicability of this new feature. Therefore, the patch enables fusion for
these scenarios. Please, see modified tests for specific details.
Reviewed By: andydavis1, bondhugula
Differential Revision: https://reviews.llvm.org/D92876
Added:
Modified:
mlir/include/mlir/Analysis/AffineStructures.h
mlir/include/mlir/Analysis/Utils.h
mlir/include/mlir/Transforms/LoopFusionUtils.h
mlir/include/mlir/Transforms/Passes.td
mlir/lib/Analysis/AffineStructures.cpp
mlir/lib/Analysis/Utils.cpp
mlir/lib/Transforms/LoopFusion.cpp
mlir/lib/Transforms/Utils/LoopFusionUtils.cpp
mlir/test/Transforms/loop-fusion.mlir
Removed:
diff --git a/mlir/include/mlir/Analysis/AffineStructures.h
b/mlir/include/mlir/Analysis/AffineStructures.h
index fa42db500684..6aa0a38243b9 100644
--- a/mlir/include/mlir/Analysis/AffineStructures.h
+++ b/mlir/include/mlir/Analysis/AffineStructures.h
@@ -232,6 +232,21 @@ class FlatAffineConstraints {
// TODO: add support for non-unit strides.
LogicalResult addAffineForOpDomain(AffineForOp forOp);
+ /// Adds constraints (lower and upper bounds) for each loop in the loop nest
+ /// described by the bound maps 'lbMaps' and 'ubMaps' of a computation slice.
+ /// Every pair ('lbMaps[i]', 'ubMaps[i]') describes the bounds of a loop in
+ /// the nest, sorted outer-to-inner. 'operands' contains the bound operands
+ /// for a single bound map. All the bound maps will use the same bound
+ /// operands. Note that some loops described by a computation slice might not
+ /// exist yet in the IR so the Value attached to those dimension identifiers
+ /// might be empty. For that reason, this method doesn't perform Value
+ /// look-ups to retrieve the dimension identifier positions. Instead, it
+ /// assumes the position of the dim identifiers in the constraint system is
+ /// the same as the position of the loop in the loop nest.
+ LogicalResult addDomainFromSliceMaps(ArrayRef lbMaps,
+ ArrayRef ubMaps,
+ ArrayRef operands);
+
/// Adds constraints imposed by the `affine.if` operation. These constraints
/// are collected from the IntegerSet attached to the given `affine.if`
/// instance argument (`ifOp`). It is asserted that:
diff --git a/mlir/include/mlir/Analysis/Utils.h
b/mlir/include/mlir/Analysis/Utils.h
index 30b6272181f5..ee6f8095f25e 100644
--- a/mlir/include/mlir/Analysis/Utils.h
+++ b/mlir/include/mlir/Analysis/Utils.h
@@ -83,10 +83,25 @@ struct ComputationSliceState {
// Clears all bounds and operands in slice state.
void clearBounds();
- /// Return true if the computation slice is empty.
+ /// Returns true if the computation slice is empty.
bool isEmpty() const { return ivs.empty(); }
+ /// Returns true if the computation slice encloses all the iterations of the
+ /// sliced loop nest. Returns false if it does not. Returns llvm::None if it
+ /// cannot determine if the slice is maximal or not.
+ // TODO: Cache 'isMaximal' so that we don't recompute it when the slice
+ // information hasn't changed.
+ Optional isMaximal() const;
+
void dump() const;
+
+private:
+ /// Fast check to determine if the computation slice is maximal. Returns true
+ /// if each slice dimension maps to an existing dst dimension and both the
src
+ /// and the dst loops for those dimens
[llvm-branch-commits] [mlir] 7dd1988 - [mlir][Affine] Add support for multi-store producer fusion
Author: Diego Caballero
Date: 2021-01-20T19:03:07+02:00
New Revision: 7dd198852b4db52ae22242dfeda4eccda83aa8b2
URL:
https://github.com/llvm/llvm-project/commit/7dd198852b4db52ae22242dfeda4eccda83aa8b2
DIFF:
https://github.com/llvm/llvm-project/commit/7dd198852b4db52ae22242dfeda4eccda83aa8b2.diff
LOG: [mlir][Affine] Add support for multi-store producer fusion
This patch adds support for producer-consumer fusion scenarios with
multiple producer stores to the AffineLoopFusion pass. The patch
introduces some changes to the producer-consumer algorithm, including:
* For a given consumer loop, producer-consumer fusion iterates over its
producer candidates until a fixed point is reached.
* Producer candidates are gathered beforehand for each iteration of the
consumer loop and visited in reverse program order (not strictly guaranteed)
to maximize the number of loops fused per iteration.
In general, these changes were needed to simplify the multi-store producer
support and remove some of the workarounds that were introduced in the past
to support more fusion cases under the single-store producer limitation.
This patch also preserves the existing functionality of AffineLoopFusion with
one minor change in behavior. Producer-consumer fusion didn't fuse scenarios
with escaping memrefs and multiple outgoing edges (from a single store).
Multi-store producer scenarios will usually (always?) have multiple outgoing
edges so we couldn't fuse any with escaping memrefs, which would greatly limit
the applicability of this new feature. Therefore, the patch enables fusion for
these scenarios. Please, see modified tests for specific details.
Reviewed By: andydavis1, bondhugula
Differential Revision: https://reviews.llvm.org/D92876
Added:
Modified:
mlir/include/mlir/Analysis/AffineStructures.h
mlir/include/mlir/Analysis/Utils.h
mlir/include/mlir/Transforms/LoopFusionUtils.h
mlir/include/mlir/Transforms/Passes.td
mlir/lib/Analysis/AffineStructures.cpp
mlir/lib/Analysis/Utils.cpp
mlir/lib/Transforms/LoopFusion.cpp
mlir/lib/Transforms/Utils/LoopFusionUtils.cpp
mlir/test/Transforms/loop-fusion.mlir
Removed:
diff --git a/mlir/include/mlir/Analysis/AffineStructures.h
b/mlir/include/mlir/Analysis/AffineStructures.h
index fa80db7d4b63..893d4ea4ff46 100644
--- a/mlir/include/mlir/Analysis/AffineStructures.h
+++ b/mlir/include/mlir/Analysis/AffineStructures.h
@@ -234,6 +234,21 @@ class FlatAffineConstraints {
// TODO: add support for non-unit strides.
LogicalResult addAffineForOpDomain(AffineForOp forOp);
+ /// Adds constraints (lower and upper bounds) for each loop in the loop nest
+ /// described by the bound maps 'lbMaps' and 'ubMaps' of a computation slice.
+ /// Every pair ('lbMaps[i]', 'ubMaps[i]') describes the bounds of a loop in
+ /// the nest, sorted outer-to-inner. 'operands' contains the bound operands
+ /// for a single bound map. All the bound maps will use the same bound
+ /// operands. Note that some loops described by a computation slice might not
+ /// exist yet in the IR so the Value attached to those dimension identifiers
+ /// might be empty. For that reason, this method doesn't perform Value
+ /// look-ups to retrieve the dimension identifier positions. Instead, it
+ /// assumes the position of the dim identifiers in the constraint system is
+ /// the same as the position of the loop in the loop nest.
+ LogicalResult addDomainFromSliceMaps(ArrayRef lbMaps,
+ ArrayRef ubMaps,
+ ArrayRef operands);
+
/// Adds constraints imposed by the `affine.if` operation. These constraints
/// are collected from the IntegerSet attached to the given `affine.if`
/// instance argument (`ifOp`). It is asserted that:
diff --git a/mlir/include/mlir/Analysis/Utils.h
b/mlir/include/mlir/Analysis/Utils.h
index 30b6272181f5..ee6f8095f25e 100644
--- a/mlir/include/mlir/Analysis/Utils.h
+++ b/mlir/include/mlir/Analysis/Utils.h
@@ -83,10 +83,25 @@ struct ComputationSliceState {
// Clears all bounds and operands in slice state.
void clearBounds();
- /// Return true if the computation slice is empty.
+ /// Returns true if the computation slice is empty.
bool isEmpty() const { return ivs.empty(); }
+ /// Returns true if the computation slice encloses all the iterations of the
+ /// sliced loop nest. Returns false if it does not. Returns llvm::None if it
+ /// cannot determine if the slice is maximal or not.
+ // TODO: Cache 'isMaximal' so that we don't recompute it when the slice
+ // information hasn't changed.
+ Optional isMaximal() const;
+
void dump() const;
+
+private:
+ /// Fast check to determine if the computation slice is maximal. Returns true
+ /// if each slice dimension maps to an existing dst dimension and both the
src
+ /// and the dst loops for those dimens
[llvm-branch-commits] [mlir] 735a07f - Revert "[mlir][Affine] Add support for multi-store producer fusion"
Author: Diego Caballero
Date: 2021-01-21T00:37:23+02:00
New Revision: 735a07f0478566f6f7c60a8a98eb8884db574113
URL:
https://github.com/llvm/llvm-project/commit/735a07f0478566f6f7c60a8a98eb8884db574113
DIFF:
https://github.com/llvm/llvm-project/commit/735a07f0478566f6f7c60a8a98eb8884db574113.diff
LOG: Revert "[mlir][Affine] Add support for multi-store producer fusion"
This reverts commit 7dd198852b4db52ae22242dfeda4eccda83aa8b2.
ASAN issue.
Added:
Modified:
mlir/include/mlir/Analysis/AffineStructures.h
mlir/include/mlir/Analysis/Utils.h
mlir/include/mlir/Transforms/LoopFusionUtils.h
mlir/include/mlir/Transforms/Passes.td
mlir/lib/Analysis/AffineStructures.cpp
mlir/lib/Analysis/Utils.cpp
mlir/lib/Transforms/LoopFusion.cpp
mlir/lib/Transforms/Utils/LoopFusionUtils.cpp
mlir/test/Transforms/loop-fusion.mlir
Removed:
diff --git a/mlir/include/mlir/Analysis/AffineStructures.h
b/mlir/include/mlir/Analysis/AffineStructures.h
index 893d4ea4ff46..fa80db7d4b63 100644
--- a/mlir/include/mlir/Analysis/AffineStructures.h
+++ b/mlir/include/mlir/Analysis/AffineStructures.h
@@ -234,21 +234,6 @@ class FlatAffineConstraints {
// TODO: add support for non-unit strides.
LogicalResult addAffineForOpDomain(AffineForOp forOp);
- /// Adds constraints (lower and upper bounds) for each loop in the loop nest
- /// described by the bound maps 'lbMaps' and 'ubMaps' of a computation slice.
- /// Every pair ('lbMaps[i]', 'ubMaps[i]') describes the bounds of a loop in
- /// the nest, sorted outer-to-inner. 'operands' contains the bound operands
- /// for a single bound map. All the bound maps will use the same bound
- /// operands. Note that some loops described by a computation slice might not
- /// exist yet in the IR so the Value attached to those dimension identifiers
- /// might be empty. For that reason, this method doesn't perform Value
- /// look-ups to retrieve the dimension identifier positions. Instead, it
- /// assumes the position of the dim identifiers in the constraint system is
- /// the same as the position of the loop in the loop nest.
- LogicalResult addDomainFromSliceMaps(ArrayRef lbMaps,
- ArrayRef ubMaps,
- ArrayRef operands);
-
/// Adds constraints imposed by the `affine.if` operation. These constraints
/// are collected from the IntegerSet attached to the given `affine.if`
/// instance argument (`ifOp`). It is asserted that:
diff --git a/mlir/include/mlir/Analysis/Utils.h
b/mlir/include/mlir/Analysis/Utils.h
index ee6f8095f25e..30b6272181f5 100644
--- a/mlir/include/mlir/Analysis/Utils.h
+++ b/mlir/include/mlir/Analysis/Utils.h
@@ -83,25 +83,10 @@ struct ComputationSliceState {
// Clears all bounds and operands in slice state.
void clearBounds();
- /// Returns true if the computation slice is empty.
+ /// Return true if the computation slice is empty.
bool isEmpty() const { return ivs.empty(); }
- /// Returns true if the computation slice encloses all the iterations of the
- /// sliced loop nest. Returns false if it does not. Returns llvm::None if it
- /// cannot determine if the slice is maximal or not.
- // TODO: Cache 'isMaximal' so that we don't recompute it when the slice
- // information hasn't changed.
- Optional isMaximal() const;
-
void dump() const;
-
-private:
- /// Fast check to determine if the computation slice is maximal. Returns true
- /// if each slice dimension maps to an existing dst dimension and both the
src
- /// and the dst loops for those dimensions have the same bounds. Returns
false
- /// if both the src and the dst loops don't have the same bounds. Returns
- /// llvm::None if none of the above can be proven.
- Optional isSliceMaximalFastCheck() const;
};
/// Computes the computation slice loop bounds for one loop nest as affine maps
diff --git a/mlir/include/mlir/Transforms/LoopFusionUtils.h
b/mlir/include/mlir/Transforms/LoopFusionUtils.h
index 10d6b83d022f..eade565e0325 100644
--- a/mlir/include/mlir/Transforms/LoopFusionUtils.h
+++ b/mlir/include/mlir/Transforms/LoopFusionUtils.h
@@ -50,8 +50,7 @@ struct FusionResult {
// TODO: Generalize utilities so that producer-consumer and sibling fusion
// strategies can be used without the assumptions made in the AffineLoopFusion
// pass.
-class FusionStrategy {
-public:
+struct FusionStrategy {
enum StrategyEnum {
// Generic loop fusion: Arbitrary loops are considered for fusion. No
// assumptions about a specific fusion strategy from AffineLoopFusion pass
@@ -70,34 +69,13 @@ class FusionStrategy {
// implementation in AffineLoopFusion pass are made. See pass for specific
// details.
Sibling
- };
+ } strategy;
- /// Construct a generic or producer-consumer fusion strategy.
- FusionStrategy(StrategyEnum strategy) : strateg
[llvm-branch-commits] [mlir] c939331 - [mlir][linalg] Fix incorrect reduction detection in Vectorizer
Author: Diego Caballero
Date: 2023-01-12T23:15:29Z
New Revision: c939331e4d97ec667ebf6bd470ff069b42ea6fb4
URL:
https://github.com/llvm/llvm-project/commit/c939331e4d97ec667ebf6bd470ff069b42ea6fb4
DIFF:
https://github.com/llvm/llvm-project/commit/c939331e4d97ec667ebf6bd470ff069b42ea6fb4.diff
LOG: [mlir][linalg] Fix incorrect reduction detection in Vectorizer
When detecting reductions, make sure the block argument is from the linalg
generic op.
This fixes https://github.com/iree-org/iree/issues/11779.
Co-authored-by: Andrzej Warzynski
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D141413
Added:
Modified:
mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
mlir/test/Dialect/Linalg/vectorization.mlir
Removed:
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
b/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
index 4f3d55ffde2bc..1e8335012504d 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Vectorization.cpp
@@ -752,13 +752,14 @@ vectorizeOneOp(RewriterBase &rewriter, LinalgOp linalgOp,
Operation *op,
// 4 . Check if the operation is a reduction.
SmallVector> reductionOperands;
for (Value operand : op->getOperands()) {
-auto arg = operand.dyn_cast();
-if (!arg || arg.getArgNumber() < linalgOp.getNumDpsInputs())
+auto blockArg = operand.dyn_cast();
+if (!blockArg || blockArg.getOwner() != linalgOp.getBlock() ||
+blockArg.getArgNumber() < linalgOp.getNumDpsInputs())
continue;
SmallVector reductionOps;
Value reduceValue = matchReduction(
linalgOp.getRegionOutputArgs(),
-arg.getArgNumber() - linalgOp.getNumDpsInputs(), reductionOps);
+blockArg.getArgNumber() - linalgOp.getNumDpsInputs(), reductionOps);
if (!reduceValue)
continue;
reductionOperands.push_back(std::make_pair(reduceValue, operand));
diff --git a/mlir/test/Dialect/Linalg/vectorization.mlir
b/mlir/test/Dialect/Linalg/vectorization.mlir
index 2c7d34066a4bc..0ccd6c4b96733 100644
--- a/mlir/test/Dialect/Linalg/vectorization.mlir
+++ b/mlir/test/Dialect/Linalg/vectorization.mlir
@@ -1853,3 +1853,43 @@ transform.sequence failures(propagate) {
%0 = transform.structured.match ops{["func.func"]} in %arg0
%1 = transform.structured.vectorize %0
}
+
+// -
+
+// Regression test: %13 was incorrectly detected as a reduction and
+// vectorization failed.
+
+func.func @wrong_reduction_detection(%input: tensor<120x64xf32>) ->
tensor<120x64xf32> {
+ %c0 = arith.constant 0 : index
+ %c4 = arith.constant 4 : index
+ %c64 = arith.constant 64 : index
+ %cst_6 = arith.constant 4.00e+00 : f32
+ %1 = scf.for %arg0 = %c0 to %c64 step %c4 iter_args(%arg1 = %input) ->
(tensor<120x64xf32>) {
+%extracted_slice = tensor.extract_slice %arg1[%c0, %arg0] [1, 4] [1, 1] :
tensor<120x64xf32> to tensor<1x4xf32>
+%10 = linalg.fill {__internal_linalg_transform__ = "1"} ins(%cst_6 : f32)
outs(%extracted_slice : tensor<1x4xf32>) -> tensor<1x4xf32>
+%11 = linalg.generic {indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>],
iterator_types = ["parallel", "parallel"]} outs(%10 : tensor<1x4xf32>) {
+^bb0(%out: f32):
+ %12 = linalg.index 0 : index
+ %13 = arith.addi %arg0, %12 : index
+ %18 = arith.index_cast %13 : index to i32
+ %20 = arith.uitofp %18 : i32 to f32
+ %67 = arith.mulf %out, %20 : f32
+ linalg.yield %67 : f32
+} -> tensor<1x4xf32>
+%inserted_slice = tensor.insert_slice %11 into %arg1[%c0, %arg0] [1, 4]
[1, 1] : tensor<1x4xf32> into tensor<120x64xf32>
+scf.yield %inserted_slice : tensor<120x64xf32>
+ }
+ return %1 : tensor<120x64xf32>
+}
+
+transform.sequence failures(propagate) {
+^bb1(%arg1: !pdl.operation):
+ %0 = transform.structured.match ops{["linalg.generic"]} in %arg1
+ %1 = get_closest_isolated_parent %0 : (!pdl.operation) -> !pdl.operation
+ %2 = transform.structured.vectorize %1
+}
+
+// CHECK-LABEL: @wrong_reduction_detection
+// CHECK: vector.broadcast
+// CHECK: vector.transfer_write
+
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] 2d3b9fd - [mlir][Affine] Fix vectorizability check for multiple load/stores
Author: Sergei Grechanik
Date: 2020-12-09T12:19:34-08:00
New Revision: 2d3b9fdc193fa835313f09f526b51bf6de3a54ef
URL:
https://github.com/llvm/llvm-project/commit/2d3b9fdc193fa835313f09f526b51bf6de3a54ef
DIFF:
https://github.com/llvm/llvm-project/commit/2d3b9fdc193fa835313f09f526b51bf6de3a54ef.diff
LOG: [mlir][Affine] Fix vectorizability check for multiple load/stores
This patch fixes a bug that allowed vectorizing of loops with loads and
stores having indexing functions varying along different memory
dimensions.
Reviewed By: aartbik, dcaballe
Differential Revision: https://reviews.llvm.org/D92702
Added:
Modified:
mlir/lib/Analysis/LoopAnalysis.cpp
mlir/test/Dialect/Affine/SuperVectorize/vectorize_1d.mlir
Removed:
diff --git a/mlir/lib/Analysis/LoopAnalysis.cpp
b/mlir/lib/Analysis/LoopAnalysis.cpp
index 210b68eae3cb..932559cac197 100644
--- a/mlir/lib/Analysis/LoopAnalysis.cpp
+++ b/mlir/lib/Analysis/LoopAnalysis.cpp
@@ -327,11 +327,23 @@ isVectorizableLoopBodyWithOpCond(AffineForOp loop,
bool mlir::isVectorizableLoopBody(AffineForOp loop, int *memRefDim,
NestedPattern &vectorTransferMatcher) {
+ *memRefDim = -1;
VectorizableOpFun fun([memRefDim](AffineForOp loop, Operation &op) {
auto load = dyn_cast(op);
auto store = dyn_cast(op);
-return load ? isContiguousAccess(loop.getInductionVar(), load, memRefDim)
-: isContiguousAccess(loop.getInductionVar(), store, memRefDim);
+int thisOpMemRefDim = -1;
+bool isContiguous = load ? isContiguousAccess(loop.getInductionVar(), load,
+ &thisOpMemRefDim)
+ : isContiguousAccess(loop.getInductionVar(),
store,
+ &thisOpMemRefDim);
+if (thisOpMemRefDim != -1) {
+ // If memory accesses vary across
diff erent dimensions then the loop is
+ // not vectorizable.
+ if (*memRefDim != -1 && *memRefDim != thisOpMemRefDim)
+return false;
+ *memRefDim = thisOpMemRefDim;
+}
+return isContiguous;
});
return isVectorizableLoopBodyWithOpCond(loop, fun, vectorTransferMatcher);
}
diff --git a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_1d.mlir
b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_1d.mlir
index ca496b75432c..86749e2c7bab 100644
--- a/mlir/test/Dialect/Affine/SuperVectorize/vectorize_1d.mlir
+++ b/mlir/test/Dialect/Affine/SuperVectorize/vectorize_1d.mlir
@@ -397,6 +397,33 @@ func @vec_rejected_10(%A : memref, %B :
memref) {
return
}
+// CHECK-LABEL: func @vec_rejected_11
+func @vec_rejected_11(%A : memref, %B : memref) {
+ // CHECK-DAG: %[[C0:.*]] = constant 0 : index
+ // CHECK-DAG: %[[C1:.*]] = constant 1 : index
+ // CHECK-DAG: %[[C2:.*]] = constant 2 : index
+ // CHECK-DAG: [[ARG_M:%[0-9]+]] = dim %{{.*}}, %[[C0]] : memref
+ // CHECK-DAG: [[ARG_N:%[0-9]+]] = dim %{{.*}}, %[[C1]] : memref
+ // CHECK-DAG: [[ARG_P:%[0-9]+]] = dim %{{.*}}, %[[C2]] : memref
+ %c0 = constant 0 : index
+ %c1 = constant 1 : index
+ %c2 = constant 2 : index
+ %M = dim %A, %c0 : memref
+ %N = dim %A, %c1 : memref
+ %P = dim %B, %c2 : memref
+
+ // CHECK: for [[IV10:%[arg0-9]*]] = 0 to %{{[0-9]*}} {
+ // CHECK: for [[IV11:%[arg0-9]*]] = 0 to %{{[0-9]*}} {
+ // This is similar to vec_rejected_5, but the order of indices is
diff erent.
+ affine.for %i10 = 0 to %M { // not vectorized
+affine.for %i11 = 0 to %N { // not vectorized
+ %a11 = affine.load %A[%i11, %i10] : memref
+ affine.store %a11, %A[%i10, %i11] : memref
+}
+ }
+ return
+}
+
// This should not vectorize due to the sequential dependence in the scf.
// CHECK-LABEL: @vec_rejected_sequential
func @vec_rejected_sequential(%A : memref) {
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] a706885 - [mlir][Affine] Introduce affine.vector_load and affine.vector_store
Author: Diego Caballero
Date: 2020-05-14T13:09:39-07:00
New Revision: a706885bb2605d74d62476241cce4f5729db2da2
URL:
https://github.com/llvm/llvm-project/commit/a706885bb2605d74d62476241cce4f5729db2da2
DIFF:
https://github.com/llvm/llvm-project/commit/a706885bb2605d74d62476241cce4f5729db2da2.diff
LOG: [mlir][Affine] Introduce affine.vector_load and affine.vector_store
This patch adds `affine.vector_load` and `affine.vector_store` ops to
the Affine dialect and lowers them to `vector.transfer_read` and
`vector.transfer_write`, respectively, in the Vector dialect.
Reviewed By: bondhugula, nicolasvasilache
Differential Revision: https://reviews.llvm.org/D79658
Added:
mlir/test/Conversion/AffineToStandard/lower-affine-to-vector.mlir
Modified:
mlir/include/mlir/Conversion/AffineToStandard/AffineToStandard.h
mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
mlir/include/mlir/Dialect/Vector/VectorOps.td
mlir/lib/Conversion/AffineToStandard/AffineToStandard.cpp
mlir/lib/Dialect/Affine/IR/AffineOps.cpp
mlir/lib/Dialect/Vector/VectorOps.cpp
mlir/test/Dialect/Affine/invalid.mlir
mlir/test/Dialect/Affine/load-store.mlir
Removed:
diff --git a/mlir/include/mlir/Conversion/AffineToStandard/AffineToStandard.h
b/mlir/include/mlir/Conversion/AffineToStandard/AffineToStandard.h
index 967c74c5c23f..5d04f157b8ce 100644
--- a/mlir/include/mlir/Conversion/AffineToStandard/AffineToStandard.h
+++ b/mlir/include/mlir/Conversion/AffineToStandard/AffineToStandard.h
@@ -44,6 +44,11 @@ Optional> expandAffineMap(OpBuilder
&builder,
void populateAffineToStdConversionPatterns(OwningRewritePatternList &patterns,
MLIRContext *ctx);
+/// Collect a set of patterns to convert vector-related Affine ops to the
Vector
+/// dialect.
+void populateAffineToVectorConversionPatterns(
+OwningRewritePatternList &patterns, MLIRContext *ctx);
+
/// Emit code that computes the lower bound of the given affine loop using
/// standard arithmetic operations.
Value lowerAffineLowerBound(AffineForOp op, OpBuilder &builder);
diff --git a/mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
b/mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
index bb02f8b18c5d..8286d8f315bd 100644
--- a/mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
+++ b/mlir/include/mlir/Dialect/Affine/IR/AffineOps.td
@@ -370,7 +370,44 @@ def AffineIfOp : Affine_Op<"if",
let hasFolder = 1;
}
-def AffineLoadOp : Affine_Op<"load", []> {
+class AffineLoadOpBase traits = []> :
+Affine_Op {
+ let arguments = (ins Arg:$memref,
+ Variadic:$indices);
+
+ code extraClassDeclarationBase = [{
+/// Returns the operand index of the memref.
+unsigned getMemRefOperandIndex() { return 0; }
+
+/// Get memref operand.
+Value getMemRef() { return getOperand(getMemRefOperandIndex()); }
+void setMemRef(Value value) { setOperand(getMemRefOperandIndex(), value); }
+MemRefType getMemRefType() {
+ return getMemRef().getType().cast();
+}
+
+/// Get affine map operands.
+operand_range getMapOperands() { return llvm::drop_begin(getOperands(),
1); }
+
+/// Returns the affine map used to index the memref for this operation.
+AffineMap getAffineMap() { return getAffineMapAttr().getValue(); }
+AffineMapAttr getAffineMapAttr() {
+ return getAttr(getMapAttrName()).cast();
+}
+
+/// Returns the AffineMapAttr associated with 'memref'.
+NamedAttribute getAffineMapAttrForMemRef(Value memref) {
+ assert(memref == getMemRef());
+ return {Identifier::get(getMapAttrName(), getContext()),
+ getAffineMapAttr()};
+}
+
+static StringRef getMapAttrName() { return "map"; }
+ }];
+}
+
+def AffineLoadOp : AffineLoadOpBase<"load", []> {
let summary = "affine load operation";
let description = [{
The "affine.load" op reads an element from a memref, where the index
@@ -393,9 +430,6 @@ def AffineLoadOp : Affine_Op<"load", []> {
```
}];
- let arguments = (ins Arg:$memref,
- Variadic:$indices);
let results = (outs AnyType:$result);
let builders = [
@@ -410,35 +444,7 @@ def AffineLoadOp : Affine_Op<"load", []> {
"AffineMap map, ValueRange mapOperands">
];
- let extraClassDeclaration = [{
-/// Returns the operand index of the memref.
-unsigned getMemRefOperandIndex() { return 0; }
-
-/// Get memref operand.
-Value getMemRef() { return getOperand(getMemRefOperandIndex()); }
-void setMemRef(Value value) { setOperand(getMemRefOperandIndex(), value); }
-MemRefType getMemRefType() {
- return getMemRef().getType().cast();
-}
-
-/// Get affine map operands.
-operand_range getMapOperands() { return llvm::drop_begin(getOperands(),
1); }
-
-/// Returns the affine map used to index the memref for this operation.
-AffineMap getAffineMap() { retu
[llvm-branch-commits] [mlir] [mlir][LLVM] Delete `LLVMFixedVectorType` and `LLVMScalableVectorType` (PR #133286)
https://github.com/dcaballe edited https://github.com/llvm/llvm-project/pull/133286 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][LLVM] Delete `LLVMFixedVectorType` and `LLVMScalableVectorType` (PR #133286)
@@ -150,8 +150,7 @@ generatedTypePrinter(Type def, AsmPrinter &printer);
bool LLVMArrayType::isValidElementType(Type type) {
return !llvm::isa(
- type);
+LLVMFunctionType, LLVMTokenType>(type);
dcaballe wrote:
That makes sense but I don't remember if we did something about this for SME.
cc: @banach-space
https://github.com/llvm/llvm-project/pull/133286
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][LLVM] Delete `LLVMFixedVectorType` and `LLVMScalableVectorType` (PR #133286)
https://github.com/dcaballe approved this pull request. Fantastic! Awesome cleanup! Thanks! LGTM but please, leave enough time for people to comment https://github.com/llvm/llvm-project/pull/133286 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][LLVM] Delete `LLVMFixedVectorType` and `LLVMScalableVectorType` (PR #133286)
@@ -1033,7 +1033,7 @@ llvm.func @scalable_vector() -> i16 {
llvm.func @scalable_llvm_vector() -> i16 {
%0 = llvm.mlir.constant(1 : i32) : i32
// CHECK: llvm.alloca
- %1 = llvm.alloca %0 x !llvm.vec : (i32) -> !llvm.ptr
+ %1 = llvm.alloca %0 x vector<[4] x !llvm.ppc_fp128> : (i32) -> !llvm.ptr
dcaballe wrote:
Interesting... I didn't expect a scalable vector with this element type...
https://github.com/llvm/llvm-project/pull/133286
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [MLIR][ArmSVE] Add initial lowering of vector.contract to SVE `*MMLA` instructions (PR #135636)
https://github.com/dcaballe commented: Really excited to see this! I'll take a look in the next iteration. Thanks! https://github.com/llvm/llvm-project/pull/135636 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
