[llvm-branch-commits] [mlir] [mlir][sparse] Fix memory leaks (part 3) (PR #85184)
https://github.com/aartbik approved this pull request. https://github.com/llvm/llvm-project/pull/85184 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][sparse] Fix memory leaks (part 3) (PR #85184)
https://github.com/aartbik edited https://github.com/llvm/llvm-project/pull/85184 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][sparse] Fix memory leaks (part 3) (PR #85184)
@@ -167,12 +169,14 @@ module {
call @dump(%d31) : (tensor<2x3x4xf64>) -> ()
//
-// Release sparse tensors.
+// Release tensors.
aartbik wrote:
same
https://github.com/llvm/llvm-project/pull/85184
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][sparse] Fix memory leaks (part 3) (PR #85184)
@@ -114,12 +114,14 @@ module {
call @dump(%d31) : (tensor<2x3x4xf64>) -> ()
//
-// Release sparse tensors.
+// Release tensors.
aartbik wrote:
we usually say
// Release the resources.
so perhaps use that here too
https://github.com/llvm/llvm-project/pull/85184
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][sparse] Add `has_runtime_library` test op (PR #85355)
https://github.com/aartbik edited https://github.com/llvm/llvm-project/pull/85355 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][sparse] Add `has_runtime_library` test op (PR #85355)
https://github.com/aartbik approved this pull request. https://github.com/llvm/llvm-project/pull/85355 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][sparse] Add `has_runtime_library` test op (PR #85355)
@@ -140,10 +140,16 @@ module {
sparse_tensor.print %s1 : tensor<4x3x2xf32, #BatchedCSR>
sparse_tensor.print %s2 : tensor<4x3x2xf32, #CSRDense>
-// FIXME: doing this explicitly crashes runtime
-// bufferization.dealloc_tensor %s0 : tensor<4x3x2xf32, #CCC>
-// bufferization.dealloc_tensor %s1 : tensor<4x3x2xf32, #BatchedCSR>
-// bufferization.dealloc_tensor %s2 : tensor<4x3x2xf32, #CSRDense>
+%has_runtime = sparse_tensor.has_runtime_library
aartbik wrote:
Can you add this information in a TODO inside the test, so we remember later
https://github.com/llvm/llvm-project/pull/85355
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][sparse] refactoring sparse_tensor.iterate lowering pattern implementation. (PR #105566)
https://github.com/aartbik approved this pull request. https://github.com/llvm/llvm-project/pull/105566 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] [mlir][sparse] unify block arguments order between iterate/coiterate operations. (PR #105567)
https://github.com/aartbik approved this pull request. https://github.com/llvm/llvm-project/pull/105567 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] f4f158b - [mlir][sparse] add vectorization strategies to sparse compiler
Author: Aart Bik
Date: 2021-01-13T11:55:23-08:00
New Revision: f4f158b2f89e16ee7068d6292d2d46457d6932bb
URL:
https://github.com/llvm/llvm-project/commit/f4f158b2f89e16ee7068d6292d2d46457d6932bb
DIFF:
https://github.com/llvm/llvm-project/commit/f4f158b2f89e16ee7068d6292d2d46457d6932bb.diff
LOG: [mlir][sparse] add vectorization strategies to sparse compiler
Similar to the parallelization strategies, the vectorization strategies
provide control on what loops should be vectorize. Unlike the parallel
strategies, only innermost loops are considered, but including reductions,
with the control of vectorizing dense loops only or dense and sparse loops.
The vectorized loops are always controlled by a vector mask to avoid
overrunning the iterations, but subsequent vector operation folding removes
redundant masks and replaces the operations with more efficient counterparts.
Similarly, we will rely on subsequent loop optimizations to further optimize
masking, e.g. using an unconditional full vector loop and scalar cleanup loop.
The current strategy already demonstrates a nice interaction between the
sparse compiler and all prior optimizations that went into the vector dialect.
Ongoing discussion at:
https://llvm.discourse.group/t/mlir-support-for-sparse-tensors/2020/10
Reviewed By: penpornk
Differential Revision: https://reviews.llvm.org/D94551
Added:
mlir/test/Dialect/Linalg/sparse_vector.mlir
Modified:
mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
mlir/test/lib/Transforms/TestSparsification.cpp
Removed:
diff --git a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
index de1658f96a87..0effa2f45c20 100644
--- a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
@@ -867,7 +867,13 @@ struct SparsificationOptions {
SparseVectorizationStrategy v, unsigned vl,
SparseIntType pt, SparseIntType it)
: parallelizationStrategy(p), vectorizationStrategy(v), vectorLength(vl),
-ptrType(pt), indType(it) {}
+ptrType(pt), indType(it) {
+// TODO: remove restriction when vectors with index elements are supported
+assert((v != SparseVectorizationStrategy::kAnyStorageInnerLoop ||
+(ptrType != SparseIntType::kNative &&
+ indType != SparseIntType::kNative)) &&
+ "This combination requires support for vectors with index
elements");
+ }
SparsificationOptions()
: SparsificationOptions(SparseParallelizationStrategy::kNone,
SparseVectorizationStrategy::kNone, 1u,
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
index dbf8d5ffcb8c..7ba0a2f63071 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
@@ -46,6 +46,7 @@
#include "mlir/Dialect/Linalg/Utils/Utils.h"
#include "mlir/Dialect/SCF/SCF.h"
#include "mlir/Dialect/StandardOps/IR/Ops.h"
+#include "mlir/IR/Matchers.h"
using namespace mlir;
@@ -301,7 +302,8 @@ struct CodeGen {
indices(numTensors, std::vector(numLoops)),
highs(numTensors, std::vector(numLoops)),
pidxs(numTensors, std::vector(numLoops)),
-idxs(numTensors, std::vector(numLoops)), redExp(-1u), redVal()
{}
+idxs(numTensors, std::vector(numLoops)), redExp(-1u), redVal(),
+curVecLength(1), curVecMask() {}
/// Sparsification options.
linalg::SparsificationOptions options;
/// Universal dense indices and upper bounds (by index). The loops array
@@ -327,6 +329,9 @@ struct CodeGen {
// is most effective; we could generalize to more outer and while-loops.
unsigned redExp;
Value redVal;
+ // Current vector length and mask.
+ unsigned curVecLength;
+ Value curVecMask;
};
} // namespace
@@ -558,6 +563,71 @@ static void genBuffers(Merger &merger, CodeGen &codegen,
}
}
+/// Constructs vector type from pointer.
+static VectorType vectorType(CodeGen &codegen, Value ptr) {
+ Type etp = ptr.getType().cast().getElementType();
+ return VectorType::get(codegen.curVecLength, etp);
+}
+
+/// Constructs vector iteration mask.
+static Value genVectorMask(CodeGen &codegen, PatternRewriter &rewriter,
+ Value iv, Value lo, Value hi, Value step) {
+ Location loc = iv.getLoc();
+ VectorType mtp =
+ VectorType::get(codegen.curVecLength, rewriter.getIntegerType(1));
+ // Special case if the vector length evenly divides the trip count (for
+ // example, "for i = 0, 128, 16"). A constant all-true mask is generated
+ // so that all subsequent masked memory operations are immediately folded
+ // into unconditional memory op
[llvm-branch-commits] [mlir] 5508516 - [mlir][sparse] retry sparse-only for cyclic iteration graphs
Author: Aart Bik
Date: 2021-01-14T22:39:29-08:00
New Revision: 5508516b06633e95fb5c2d6a5e196e4dcaa72c8d
URL:
https://github.com/llvm/llvm-project/commit/5508516b06633e95fb5c2d6a5e196e4dcaa72c8d
DIFF:
https://github.com/llvm/llvm-project/commit/5508516b06633e95fb5c2d6a5e196e4dcaa72c8d.diff
LOG: [mlir][sparse] retry sparse-only for cyclic iteration graphs
This is a very minor improvement during iteration graph construction.
If the first attempt considering the dimension order of all tensors fails,
a second attempt is made using the constraints of sparse tensors only.
Dense tensors prefer dimension order (locality) but provide random access
if needed, enabling the compilation of more sparse kernels.
Reviewed By: penpornk
Differential Revision: https://reviews.llvm.org/D94709
Added:
mlir/test/Dialect/Linalg/sparse_nd.mlir
Modified:
mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
Removed:
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
index 7ba0a2f63071..84c71e84c42e 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
@@ -274,6 +274,11 @@ class Merger {
return false;
}
+ // Returns true if tensor has any sparse dimension.
+ bool isSparseTensor(unsigned t) const {
+return llvm::any_of(dims[t], [](Dim d) { return d == Dim::kSparse; });
+ }
+
// Setter
void setDim(unsigned t, unsigned i, Dim d) { dims[t][i] = d; }
@@ -382,17 +387,22 @@ static bool topSortDFS(unsigned i, std::vector
&visit,
/// for sparse storage formats since these only support access along fixed
/// dimensions. Even for dense storage formats, however, the natural index
/// order yields innermost unit-stride access with better spatial locality.
-static bool computeIterationGraph(linalg::GenericOp op,
- std::vector &topSort) {
+static bool computeIterationGraph(Merger &merger, linalg::GenericOp op,
+ std::vector &topSort,
+ bool sparseOnly) {
// Set up an n x n from/to adjacency matrix of the iteration graph
// for the implicit loop indices i_0 .. i_n-1.
unsigned n = op.getNumLoops();
std::vector> adjM(n, std::vector(n, false));
// Iterate over the indexing maps of every tensor in the tensor expression.
- for (auto imap : llvm::enumerate(op.indexing_maps())) {
-auto map = imap.value().template cast().getValue();
+ unsigned numTensors = op.getNumShapedOperands();
+ for (unsigned t = 0; t < numTensors; t++) {
+auto map = op.getIndexingMap(t);
assert(map.getNumDims() == n);
+// Skip dense tensor constraints when sparse only is requested.
+if (sparseOnly && !merger.isSparseTensor(t))
+ continue;
// At the moment, we take the index variables in the tensor access
// expression in the order in which they appear (conceptually a
// "row-major" layout of every tensor). So, a tensor access A_ijk
@@ -407,6 +417,7 @@ static bool computeIterationGraph(linalg::GenericOp op,
// Topologically sort the iteration graph to determine loop order.
// Report failure for a cyclic iteration graph.
+ topSort.clear();
topSort.reserve(n);
std::vector visit(n, 0);
for (unsigned i = 0; i < n; i++)
@@ -1207,10 +1218,9 @@ struct GenericOpSparsifier : public
OpRewritePattern {
// tensors are visited in natural index order. Fails on cycles.
// This assumes that higher-level passes have already put the
// tensors in each tensor expression in a feasible order.
-// TODO: try again without *dense* constraints on failure or
-// even try to insert sparse reorderings to resolve cycles
std::vector topSort;
-if (!computeIterationGraph(op, topSort))
+if (!computeIterationGraph(merger, op, topSort, /*sparseOnly=*/false) &&
+!computeIterationGraph(merger, op, topSort, /*sparseOnly=*/true))
return failure();
// Finds the terminating yield statement and builds the tensor
diff --git a/mlir/test/Dialect/Linalg/sparse_nd.mlir
b/mlir/test/Dialect/Linalg/sparse_nd.mlir
new file mode 100644
index ..2b0762b1bf37
--- /dev/null
+++ b/mlir/test/Dialect/Linalg/sparse_nd.mlir
@@ -0,0 +1,94 @@
+// NOTE: Assertions have been autogenerated by utils/generate-test-checks.py
+// RUN: mlir-opt %s -test-sparsification | FileCheck %s
+
+// Example with cyclic iteration graph with sparse and dense constraints,
+// but an acyclic iteration graph using sparse constraints only.
+#trait_mul = {
+ indexing_maps = [
+affine_map<(i,j,k,l,m,n,o,p) -> (i,j,k,l,m,n,o,p)>, // A
+affine_map<(i,j,k,l,m,n,o,p) -> (p,o,n,m,l,k,j,i)>, // B
+affine_map<(i,j,k,l,m,n,o,p) -> (i,j,k,l,m,n,o,p)> // X
+ ],
+ sparse = [
+[ "D", "D", "D", "D", "D", "D", "D", "D" ], // a
+
[llvm-branch-commits] [mlir] d8fc273 - [mlir][sparse] improved sparse runtime support library
Author: Aart Bik
Date: 2021-01-16T12:16:10-08:00
New Revision: d8fc27301d18f0935ba99ead7ac61aa6a53f16e4
URL:
https://github.com/llvm/llvm-project/commit/d8fc27301d18f0935ba99ead7ac61aa6a53f16e4
DIFF:
https://github.com/llvm/llvm-project/commit/d8fc27301d18f0935ba99ead7ac61aa6a53f16e4.diff
LOG: [mlir][sparse] improved sparse runtime support library
Added the ability to read (an extended version of) the FROSTT
file format, so that we can now read in sparse tensors of arbitrary
rank. Generalized the API to deal with more than two dimensions.
Also added the ability to sort the indices of sparse tensors
lexicographically. This is an important step towards supporting
auto gen of initialization code, since sparse storage formats
are easier to initialize if the indices are sorted. Since most
external formats don't enforce such properties, it is convenient
to have this ability in our runtime support library.
Lastly, the re-entrant problem of the original implementation
is fixed by passing an opaque object around (rather than having
a single static variable, ugh!).
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D94852
Added:
mlir/integration_test/Sparse/CPU/frostt-example.mlir
mlir/integration_test/data/test.tns
Modified:
mlir/include/mlir/ExecutionEngine/CRunnerUtils.h
mlir/integration_test/CMakeLists.txt
mlir/integration_test/Sparse/CPU/matrix-market-example.mlir
mlir/lib/ExecutionEngine/SparseUtils.cpp
Removed:
diff --git a/mlir/include/mlir/ExecutionEngine/CRunnerUtils.h
b/mlir/include/mlir/ExecutionEngine/CRunnerUtils.h
index edcd8e0dc545..2d0608a8656b 100644
--- a/mlir/include/mlir/ExecutionEngine/CRunnerUtils.h
+++ b/mlir/include/mlir/ExecutionEngine/CRunnerUtils.h
@@ -198,7 +198,7 @@ class DynamicMemRefType {
};
//===--===//
-// Small runtime support "lib" for vector.print lowering during codegen.
+// Small runtime support library for vector.print lowering during codegen.
//===--===//
extern "C" MLIR_CRUNNERUTILS_EXPORT void printI64(int64_t i);
extern "C" MLIR_CRUNNERUTILS_EXPORT void printU64(uint64_t u);
@@ -210,15 +210,13 @@ extern "C" MLIR_CRUNNERUTILS_EXPORT void printComma();
extern "C" MLIR_CRUNNERUTILS_EXPORT void printNewline();
//===--===//
-// Small runtime support for sparse tensors.
+// Small runtime support library for sparse tensors.
//===--===//
-extern "C" MLIR_CRUNNERUTILS_EXPORT void openMatrixC(char *filename,
- uint64_t *mdata,
- uint64_t *ndata,
- uint64_t *nnzdata);
+extern "C" MLIR_CRUNNERUTILS_EXPORT void *openTensorC(char *filename,
+ uint64_t *idata);
extern "C" MLIR_CRUNNERUTILS_EXPORT void
-readMatrixItemC(uint64_t *idata, uint64_t *jdata, double *ddata);
-extern "C" MLIR_CRUNNERUTILS_EXPORT void closeMatrix();
-extern "C" MLIR_CRUNNERUTILS_EXPORT char *getMatrix(uint64_t id);
+readTensorItemC(void *tensor, uint64_t *idata, double *ddata);
+extern "C" MLIR_CRUNNERUTILS_EXPORT void closeTensor(void *tensor);
+extern "C" MLIR_CRUNNERUTILS_EXPORT char *getTensorFilename(uint64_t id);
#endif // EXECUTIONENGINE_CRUNNERUTILS_H_
diff --git a/mlir/integration_test/CMakeLists.txt
b/mlir/integration_test/CMakeLists.txt
index bc5ad90e1253..fb2be5256dcc 100644
--- a/mlir/integration_test/CMakeLists.txt
+++ b/mlir/integration_test/CMakeLists.txt
@@ -31,4 +31,5 @@ add_lit_testsuites(MLIR_INTEGRATION
${CMAKE_CURRENT_SOURCE_DIR}
# Copy test data over.
file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/data/test.mtx
+ ${CMAKE_CURRENT_SOURCE_DIR}/data/test.tns
DESTINATION ${MLIR_INTEGRATION_TEST_DIR}/data/)
diff --git a/mlir/integration_test/Sparse/CPU/frostt-example.mlir
b/mlir/integration_test/Sparse/CPU/frostt-example.mlir
new file mode 100644
index ..8144270aa91f
--- /dev/null
+++ b/mlir/integration_test/Sparse/CPU/frostt-example.mlir
@@ -0,0 +1,149 @@
+// RUN: mlir-opt %s \
+// RUN: -convert-scf-to-std -convert-vector-to-scf \
+// RUN: -convert-linalg-to-llvm -convert-vector-to-llvm | \
+// RUN: TENSOR0="%mlir_integration_test_dir/data/test.tns" \
+// RUN: mlir-cpu-runner \
+// RUN: -e entry -entry-point-result=void \
+// RUN:
-shared-libs=%mlir_integration_test_dir/libmlir_c_runner_utils%shlibext | \
+// RUN: FileCheck %s
+
+module {
+ //
+ // Example of using the sparse runtime support library to read a sparse
tensor
+ // in the FROSTT file format (http://frostt.io/tensors/file-formats.html).
+ //
+ f
[llvm-branch-commits] [mlir] b5c542d - [mlir][sparse] add narrower choices for pointers/indices
Author: Aart Bik
Date: 2021-01-19T20:20:38-08:00
New Revision: b5c542d64b98b5a74d35dedad41051a0b00d7946
URL:
https://github.com/llvm/llvm-project/commit/b5c542d64b98b5a74d35dedad41051a0b00d7946
DIFF:
https://github.com/llvm/llvm-project/commit/b5c542d64b98b5a74d35dedad41051a0b00d7946.diff
LOG: [mlir][sparse] add narrower choices for pointers/indices
Use cases with 16- or even 8-bit pointer/index structures have been identified.
Reviewed By: penpornk
Differential Revision: https://reviews.llvm.org/D95015
Added:
Modified:
mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
mlir/test/Dialect/Linalg/sparse_storage.mlir
mlir/test/lib/Transforms/TestSparsification.cpp
Removed:
diff --git a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
index 0effa2f45c20..611ab6867372 100644
--- a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
@@ -853,13 +853,13 @@ enum class SparseVectorizationStrategy {
};
/// Defines a type for "pointer" and "index" storage in the sparse storage
-/// scheme, with a choice between the native platform-dependent index width,
-/// 64-bit integers, or 32-bit integers. A narrow width obviously reduces
+/// scheme, with a choice between the native platform-dependent index width
+/// or any of 64-/32-/16-/8-bit integers. A narrow width obviously reduces
/// the memory footprint of the sparse storage scheme, but the width should
/// suffice to define the total required range (viz. the maximum number of
/// stored entries per indirection level for the "pointers" and the maximum
/// value of each tensor index over all dimensions for the "indices").
-enum class SparseIntType { kNative, kI64, kI32 };
+enum class SparseIntType { kNative, kI64, kI32, kI16, kI8 };
/// Sparsification options.
struct SparsificationOptions {
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
index 898b15266072..cefcdcbed9ae 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
@@ -512,6 +512,10 @@ static Type genIntType(PatternRewriter &rewriter,
linalg::SparseIntType tp) {
return rewriter.getIntegerType(64);
case linalg::SparseIntType::kI32:
return rewriter.getIntegerType(32);
+ case linalg::SparseIntType::kI16:
+return rewriter.getIntegerType(16);
+ case linalg::SparseIntType::kI8:
+return rewriter.getIntegerType(8);
}
llvm_unreachable("unexpected SparseIntType");
}
diff --git a/mlir/test/Dialect/Linalg/sparse_storage.mlir
b/mlir/test/Dialect/Linalg/sparse_storage.mlir
index 69b8e1903d69..ef5dc0d766e3 100644
--- a/mlir/test/Dialect/Linalg/sparse_storage.mlir
+++ b/mlir/test/Dialect/Linalg/sparse_storage.mlir
@@ -6,6 +6,10 @@
// RUN: FileCheck %s --check-prefix=CHECK-TYPE2
// RUN: mlir-opt %s -test-sparsification="ptr-type=2 ind-type=2" | \
// RUN: FileCheck %s --check-prefix=CHECK-TYPE3
+// RUN: mlir-opt %s -test-sparsification="ptr-type=3 ind-type=3" | \
+// RUN: FileCheck %s --check-prefix=CHECK-TYPE4
+// RUN: mlir-opt %s -test-sparsification="ptr-type=4 ind-type=4" | \
+// RUN: FileCheck %s --check-prefix=CHECK-TYPE5
#trait_mul_1d = {
indexing_maps = [
@@ -86,6 +90,38 @@
// CHECK-TYPE3: store %[[MUL]], %{{.*}}[%[[INDC]]] : memref<32xf64>
// CHECK-TYPE3: }
+// CHECK-TYPE4-LABEL: func @mul_dd(
+// CHECK-TYPE4: %[[C0:.*]] = constant 0 : index
+// CHECK-TYPE4: %[[C1:.*]] = constant 1 : index
+// CHECK-TYPE4: %[[P0:.*]] = load %{{.*}}[%[[C0]]] : memref
+// CHECK-TYPE4: %[[B0:.*]] = index_cast %[[P0]] : i16 to index
+// CHECK-TYPE4: %[[P1:.*]] = load %{{.*}}[%[[C1]]] : memref
+// CHECK-TYPE4: %[[B1:.*]] = index_cast %[[P1]] : i16 to index
+// CHECK-TYPE4: scf.for %[[I:.*]] = %[[B0]] to %[[B1]] step %[[C1]] {
+// CHECK-TYPE4: %[[IND0:.*]] = load %{{.*}}[%[[I]]] : memref
+// CHECK-TYPE4: %[[INDC:.*]] = index_cast %[[IND0]] : i16 to index
+// CHECK-TYPE4: %[[VAL0:.*]] = load %{{.*}}[%[[I]]] : memref
+// CHECK-TYPE4: %[[VAL1:.*]] = load %{{.*}}[%[[INDC]]] : memref<32xf64>
+// CHECK-TYPE4: %[[MUL:.*]] = mulf %[[VAL0]], %[[VAL1]] : f64
+// CHECK-TYPE4: store %[[MUL]], %{{.*}}[%[[INDC]]] : memref<32xf64>
+// CHECK-TYPE4: }
+
+// CHECK-TYPE5-LABEL: func @mul_dd(
+// CHECK-TYPE5: %[[C0:.*]] = constant 0 : index
+// CHECK-TYPE5: %[[C1:.*]] = constant 1 : index
+// CHECK-TYPE5: %[[P0:.*]] = load %{{.*}}[%[[C0]]] : memref
+// CHECK-TYPE5: %[[B0:.*]] = index_cast %[[P0]] : i8 to index
+// CHECK-TYPE5: %[[P1:.*]] = load %{{.*}}[%[[C1]]] : memref
+// CHECK-TYPE5: %[[B1:.*]] = index_cast %[[P1]] : i8 to index
+// CHECK-TYPE5: scf.for %[[I:.*]] = %[[B0]] to %[[B1]] step %[[C1]] {
+// CHECK-TYPE5: %[
[llvm-branch-commits] [mlir] 5959c28 - [mlir][sparse] add asserts on reading in tensor data
Author: Aart Bik
Date: 2021-01-20T14:30:13-08:00
New Revision: 5959c28f24856f3d4a1db6b4743c66bdc6dcd735
URL:
https://github.com/llvm/llvm-project/commit/5959c28f24856f3d4a1db6b4743c66bdc6dcd735
DIFF:
https://github.com/llvm/llvm-project/commit/5959c28f24856f3d4a1db6b4743c66bdc6dcd735.diff
LOG: [mlir][sparse] add asserts on reading in tensor data
Rationale:
Since I made the argument that metadata helps with extra
verification checks, I better actually do that ;-)
Reviewed By: penpornk
Differential Revision: https://reviews.llvm.org/D95072
Added:
Modified:
mlir/lib/ExecutionEngine/SparseUtils.cpp
Removed:
diff --git a/mlir/lib/ExecutionEngine/SparseUtils.cpp
b/mlir/lib/ExecutionEngine/SparseUtils.cpp
index 376b989975b5..d1962661fe79 100644
--- a/mlir/lib/ExecutionEngine/SparseUtils.cpp
+++ b/mlir/lib/ExecutionEngine/SparseUtils.cpp
@@ -48,9 +48,9 @@ namespace {
/// and a rank-5 tensor element like
/// ({i,j,k,l,m}, a[i,j,k,l,m])
struct Element {
- Element(const std::vector &ind, double val)
+ Element(const std::vector &ind, double val)
: indices(ind), value(val){};
- std::vector indices;
+ std::vector indices;
double value;
};
@@ -61,9 +61,15 @@ struct Element {
/// formats require the elements to appear in lexicographic index order).
struct SparseTensor {
public:
- SparseTensor(int64_t capacity) : pos(0) { elements.reserve(capacity); }
+ SparseTensor(const std::vector &szs, uint64_t capacity)
+ : sizes(szs), pos(0) {
+elements.reserve(capacity);
+ }
// Add element as indices and value.
- void add(const std::vector &ind, double val) {
+ void add(const std::vector &ind, double val) {
+assert(sizes.size() == ind.size());
+for (int64_t r = 0, rank = sizes.size(); r < rank; r++)
+ assert(ind[r] < sizes[r]); // within bounds
elements.emplace_back(Element(ind, val));
}
// Sort elements lexicographically by index.
@@ -82,6 +88,8 @@ struct SparseTensor {
}
return false;
}
+
+ std::vector sizes; // per-rank dimension sizes
std::vector elements;
uint64_t pos;
};
@@ -225,20 +233,24 @@ extern "C" void *openTensorC(char *filename, uint64_t
*idata) {
fprintf(stderr, "Unknown format %s\n", filename);
exit(1);
}
- // Read all nonzero elements.
+ // Prepare sparse tensor object with per-rank dimension sizes
+ // and the number of nonzeros as initial capacity.
uint64_t rank = idata[0];
uint64_t nnz = idata[1];
- SparseTensor *tensor = new SparseTensor(nnz);
- std::vector indices(rank);
- double value;
+ std::vector indices(rank);
+ for (uint64_t r = 0; r < rank; r++)
+indices[r] = idata[2 + r];
+ SparseTensor *tensor = new SparseTensor(indices, nnz);
+ // Read all nonzero elements.
for (uint64_t k = 0; k < nnz; k++) {
for (uint64_t r = 0; r < rank; r++) {
- if (fscanf(file, "%" PRId64, &indices[r]) != 1) {
+ if (fscanf(file, "%" PRIu64, &indices[r]) != 1) {
fprintf(stderr, "Cannot find next index in %s\n", filename);
exit(1);
}
indices[r]--; // 0-based index
}
+double value;
if (fscanf(file, "%lg\n", &value) != 1) {
fprintf(stderr, "Cannot find next value in %s\n", filename);
exit(1);
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] 8b124c1 - [mlir][sparse] adjust output shape inference to new tensor abstraction
Author: Aart Bik
Date: 2021-01-05T15:31:39-08:00
New Revision: 8b124c19f52cb8ed0236b602df56787553e1e1b6
URL:
https://github.com/llvm/llvm-project/commit/8b124c19f52cb8ed0236b602df56787553e1e1b6
DIFF:
https://github.com/llvm/llvm-project/commit/8b124c19f52cb8ed0236b602df56787553e1e1b6.diff
LOG: [mlir][sparse] adjust output shape inference to new tensor abstraction
Nicolas changed the tensor abstraction so that every output has
its own shape definition. This simplifies the "inference" that
was used in the sparse compiler.
Reviewed By: penpornk
Differential Revision: https://reviews.llvm.org/D94119
Added:
Modified:
mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
mlir/test/Dialect/Linalg/sparse_2d.mlir
Removed:
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
index a6b7277e47e3..ed81d5e24805 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
@@ -538,15 +538,8 @@ static void genBuffers(Merger &merger, CodeGen &codegen,
// Find lower and upper bound in current dimension.
Value up;
if (shape[d] == TensorType::kDynamicSize) {
-// For the output tensor, we may need to infer the upper bound.
-// For all others, we look at the incoming argument.
-if (t == numInputs && !op.getNumInitTensors()) {
- up = codegen.sizes[i];
- assert(up); // TODO: what else?
-} else {
- Value arg = t < numInputs ? op.getInput(t) : op.getInitTensors()[0];
- up = rewriter.create(loc, arg, d);
-}
+Value arg = t < numInputs ? op.getInput(t) : op.getOutput(0);
+up = rewriter.create(loc, arg, d);
args.push_back(up);
} else {
up = rewriter.create(loc, shape[d]);
diff --git a/mlir/test/Dialect/Linalg/sparse_2d.mlir
b/mlir/test/Dialect/Linalg/sparse_2d.mlir
index 6612a723f23d..9bb68ca91089 100644
--- a/mlir/test/Dialect/Linalg/sparse_2d.mlir
+++ b/mlir/test/Dialect/Linalg/sparse_2d.mlir
@@ -1139,19 +1139,19 @@ func @sum_reduction(%arga: tensor<10x20xf32>, %argx:
tensor) -> tensor
// CHECK: %[[VAL_2:.*]] = constant 999 : index
// CHECK: %[[VAL_3:.*]] = constant 0 : index
// CHECK: %[[VAL_4:.*]] = constant 1 : index
-// CHECK: %[[VAL_5:.*]] = dim %[[VAL_0]], %[[VAL_3]] :
tensor
+// CHECK: %[[VAL_5:.*]] = alloca(%[[VAL_2]]) : memref
// CHECK: %[[VAL_6:.*]] = alloca(%[[VAL_2]]) : memref
-// CHECK: %[[VAL_7:.*]] = alloca(%[[VAL_2]]) : memref
-// CHECK: %[[VAL_8:.*]] = dim %[[VAL_0]], %[[VAL_4]] :
tensor
-// CHECK: %[[VAL_9:.*]] = alloca(%[[VAL_2]]) : memref
-// CHECK: %[[VAL_10:.*]] = alloca(%[[VAL_5]], %[[VAL_8]]) :
memref
-// CHECK: scf.for %[[VAL_11:.*]] = %[[VAL_3]] to %[[VAL_5]] step
%[[VAL_4]] {
-// CHECK: %[[VAL_12:.*]] = load %[[VAL_6]]{{\[}}%[[VAL_11]]] :
memref
+// CHECK: %[[VAL_7:.*]] = alloca(%[[VAL_2]]) : memref
+// CHECK: %[[VAL_8:.*]] = dim %[[VAL_0]], %[[VAL_3]] :
tensor
+// CHECK: %[[VAL_9:.*]] = dim %[[VAL_0]], %[[VAL_4]] :
tensor
+// CHECK: %[[VAL_10:.*]] = alloca(%[[VAL_8]], %[[VAL_9]]) :
memref
+// CHECK: scf.for %[[VAL_11:.*]] = %[[VAL_3]] to %[[VAL_8]] step
%[[VAL_4]] {
+// CHECK: %[[VAL_12:.*]] = load %[[VAL_5]]{{\[}}%[[VAL_11]]] :
memref
// CHECK: %[[VAL_13:.*]] = addi %[[VAL_11]], %[[VAL_4]] : index
-// CHECK: %[[VAL_14:.*]] = load %[[VAL_6]]{{\[}}%[[VAL_13]]] :
memref
+// CHECK: %[[VAL_14:.*]] = load %[[VAL_5]]{{\[}}%[[VAL_13]]] :
memref
// CHECK: scf.for %[[VAL_15:.*]] = %[[VAL_12]] to %[[VAL_14]] step
%[[VAL_4]] {
-// CHECK: %[[VAL_16:.*]] = load %[[VAL_7]]{{\[}}%[[VAL_15]]] :
memref
-// CHECK: %[[VAL_17:.*]] = load %[[VAL_9]]{{\[}}%[[VAL_15]]] :
memref
+// CHECK: %[[VAL_16:.*]] = load %[[VAL_6]]{{\[}}%[[VAL_15]]] :
memref
+// CHECK: %[[VAL_17:.*]] = load %[[VAL_7]]{{\[}}%[[VAL_15]]] :
memref
// CHECK: %[[VAL_18:.*]] = mulf %[[VAL_17]], %[[VAL_1]] : f64
// CHECK: store %[[VAL_18]], %[[VAL_10]]{{\[}}%[[VAL_11]],
%[[VAL_16]]] : memref
// CHECK: }
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] a57def3 - [mlir][vector] generalized masked l/s and compressed l/s with indices
Author: Aart Bik
Date: 2021-01-08T13:59:34-08:00
New Revision: a57def30f53990aafc3f64b9b7a0f60916cc7f61
URL:
https://github.com/llvm/llvm-project/commit/a57def30f53990aafc3f64b9b7a0f60916cc7f61
DIFF:
https://github.com/llvm/llvm-project/commit/a57def30f53990aafc3f64b9b7a0f60916cc7f61.diff
LOG: [mlir][vector] generalized masked l/s and compressed l/s with indices
Adding the ability to index the base address brings these operations closer
to the transfer read and write semantics (with lowering advantages), ensures
more consistent use in vector MLIR code (easier to read), and reduces the
amount of code duplication to lower memrefs into base addresses considerably
(making codegen less error-prone).
Reviewed By: ThomasRaoux
Differential Revision: https://reviews.llvm.org/D94278
Added:
Modified:
mlir/include/mlir/Dialect/Vector/VectorOps.td
mlir/integration_test/Dialect/Vector/CPU/test-compress.mlir
mlir/integration_test/Dialect/Vector/CPU/test-expand.mlir
mlir/integration_test/Dialect/Vector/CPU/test-maskedload.mlir
mlir/integration_test/Dialect/Vector/CPU/test-maskedstore.mlir
mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
mlir/lib/Dialect/Vector/VectorOps.cpp
mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
mlir/test/Dialect/Vector/invalid.mlir
mlir/test/Dialect/Vector/ops.mlir
mlir/test/Dialect/Vector/vector-mem-transforms.mlir
mlir/test/Dialect/Vector/vector-transforms.mlir
mlir/test/lib/Transforms/TestVectorTransforms.cpp
Removed:
diff --git a/mlir/include/mlir/Dialect/Vector/VectorOps.td
b/mlir/include/mlir/Dialect/Vector/VectorOps.td
index 0a98b9ffe996..0aa4950e0a9e 100644
--- a/mlir/include/mlir/Dialect/Vector/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/VectorOps.td
@@ -1317,6 +1317,7 @@ def Vector_TransferWriteOp :
def Vector_MaskedLoadOp :
Vector_Op<"maskedload">,
Arguments<(ins AnyMemRef:$base,
+ Variadic:$indices,
VectorOfRankAndType<[1], [I1]>:$mask,
VectorOfRank<[1]>:$pass_thru)>,
Results<(outs VectorOfRank<[1]>:$result)> {
@@ -1325,12 +1326,12 @@ def Vector_MaskedLoadOp :
let description = [{
The masked load reads elements from memory into a 1-D vector as defined
-by a base and a 1-D mask vector. When the mask is set, the element is read
-from memory. Otherwise, the corresponding element is taken from a 1-D
-pass-through vector. Informally the semantics are:
+by a base with indices and a 1-D mask vector. When the mask is set, the
+element is read from memory. Otherwise, the corresponding element is taken
+from a 1-D pass-through vector. Informally the semantics are:
```
-result[0] := mask[0] ? MEM[base+0] : pass_thru[0]
-result[1] := mask[1] ? MEM[base+1] : pass_thru[1]
+result[0] := mask[0] ? base[i+0] : pass_thru[0]
+result[1] := mask[1] ? base[i+1] : pass_thru[1]
etc.
```
The masked load can be used directly where applicable, or can be used
@@ -1342,7 +1343,7 @@ def Vector_MaskedLoadOp :
Example:
```mlir
-%0 = vector.maskedload %base, %mask, %pass_thru
+%0 = vector.maskedload %base[%i], %mask, %pass_thru
: memref, vector<8xi1>, vector<8xf32> into vector<8xf32>
```
}];
@@ -1360,7 +1361,7 @@ def Vector_MaskedLoadOp :
return result().getType().cast();
}
}];
- let assemblyFormat = "$base `,` $mask `,` $pass_thru attr-dict `:` "
+ let assemblyFormat = "$base `[` $indices `]` `,` $mask `,` $pass_thru
attr-dict `:` "
"type($base) `,` type($mask) `,` type($pass_thru) `into` type($result)";
let hasCanonicalizer = 1;
}
@@ -1368,6 +1369,7 @@ def Vector_MaskedLoadOp :
def Vector_MaskedStoreOp :
Vector_Op<"maskedstore">,
Arguments<(ins AnyMemRef:$base,
+ Variadic:$indices,
VectorOfRankAndType<[1], [I1]>:$mask,
VectorOfRank<[1]>:$value)> {
@@ -1375,12 +1377,12 @@ def Vector_MaskedStoreOp :
let description = [{
The masked store operation writes elements from a 1-D vector into memory
-as defined by a base and a 1-D mask vector. When the mask is set, the
-corresponding element from the vector is written to memory. Otherwise,
+as defined by a base with indices and a 1-D mask vector. When the mask is
+set, the corresponding element from the vector is written to memory.
Otherwise,
no action is taken for the element. Informally the semantics are:
```
-if (mask[0]) MEM[base+0] = value[0]
-if (mask[1]) MEM[base+1] = value[1]
+if (mask[0]) base[i+0] = value[0]
+if (mask[1]) base[i+1] = value[1]
etc.
```
The masked store can be used directly where applicable, or can be used
@@ -1392,7 +1394,7 @@ def Vector_MaskedStoreOp :
Example:
```mlir
-vector.maskedstore %base, %mask, %value
+vector.maskedstore %base[%
[llvm-branch-commits] [mlir] 6728af1 - [mlir][vector] modified scatter/gather syntax, pass_thru mandatory
Author: Aart Bik
Date: 2021-01-09T11:41:37-08:00
New Revision: 6728af16cf987df3cf051f3a1f9c92ed2b8fbc2d
URL:
https://github.com/llvm/llvm-project/commit/6728af16cf987df3cf051f3a1f9c92ed2b8fbc2d
DIFF:
https://github.com/llvm/llvm-project/commit/6728af16cf987df3cf051f3a1f9c92ed2b8fbc2d.diff
LOG: [mlir][vector] modified scatter/gather syntax, pass_thru mandatory
This change makes the scatter/gather syntax more consistent with
the syntax of all the other memory operations in the Vector dialect
(order of types, use of [] for index, etc.). This will make the MLIR
code easier to read. In addition, the pass_thru parameter of the
gather has been made mandatory (there is very little benefit in
using the implicit "undefined" values).
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D94352
Added:
Modified:
mlir/include/mlir/Dialect/Vector/VectorOps.td
mlir/integration_test/Dialect/Vector/CPU/test-gather.mlir
mlir/integration_test/Dialect/Vector/CPU/test-scatter.mlir
mlir/integration_test/Dialect/Vector/CPU/test-sparse-dot-matvec.mlir
mlir/integration_test/Dialect/Vector/CPU/test-sparse-saxpy-jagged-matvec.mlir
mlir/lib/Dialect/Vector/VectorOps.cpp
mlir/test/Conversion/VectorToLLVM/vector-to-llvm.mlir
mlir/test/Dialect/Vector/invalid.mlir
mlir/test/Dialect/Vector/ops.mlir
mlir/test/Dialect/Vector/vector-mem-transforms.mlir
Removed:
diff --git a/mlir/include/mlir/Dialect/Vector/VectorOps.td
b/mlir/include/mlir/Dialect/Vector/VectorOps.td
index 0aa4950e0a9e..7f57dcd77def 100644
--- a/mlir/include/mlir/Dialect/Vector/VectorOps.td
+++ b/mlir/include/mlir/Dialect/Vector/VectorOps.td
@@ -1419,7 +1419,7 @@ def Vector_GatherOp :
Arguments<(ins AnyMemRef:$base,
VectorOfRankAndType<[1], [AnyInteger]>:$indices,
VectorOfRankAndType<[1], [I1]>:$mask,
- Variadic>:$pass_thru)>,
+ VectorOfRank<[1]>:$pass_thru)>,
Results<(outs VectorOfRank<[1]>:$result)> {
let summary = "gathers elements from memory into a vector as defined by an
index vector and mask";
@@ -1428,10 +1428,8 @@ def Vector_GatherOp :
The gather operation gathers elements from memory into a 1-D vector as
defined by a base and a 1-D index vector, but only if the corresponding
bit is set in a 1-D mask vector. Otherwise, the element is taken from a
-1-D pass-through vector, if provided, or left undefined. Informally the
-semantics are:
+1-D pass-through vector. Informally the semantics are:
```
-if (!defined(pass_thru)) pass_thru = [undef, .., undef]
result[0] := mask[0] ? base[index[0]] : pass_thru[0]
result[1] := mask[1] ? base[index[1]] : pass_thru[1]
etc.
@@ -1447,8 +1445,8 @@ def Vector_GatherOp :
Example:
```mlir
-%g = vector.gather %base, %indices, %mask, %pass_thru
-: (memref, vector<16xi32>, vector<16xi1>, vector<16xf32>) ->
vector<16xf32>
+%g = vector.gather %base[%indices], %mask, %pass_thru
+ : memref, vector<16xi32>, vector<16xi1>, vector<16xf32> into
vector<16xf32>
```
}];
let extraClassDeclaration = [{
@@ -1462,15 +1460,14 @@ def Vector_GatherOp :
return mask().getType().cast();
}
VectorType getPassThruVectorType() {
- return (llvm::size(pass_thru()) == 0)
-? VectorType()
-: (*pass_thru().begin()).getType().cast();
+ return pass_thru().getType().cast();
}
VectorType getResultVectorType() {
return result().getType().cast();
}
}];
- let assemblyFormat = "operands attr-dict `:` functional-type(operands,
results)";
+ let assemblyFormat = "$base `[` $indices `]` `,` $mask `,` $pass_thru
attr-dict `:` "
+"type($base) `,` type($indices) `,` type($mask) `,` type($pass_thru)
`into` type($result)";
let hasCanonicalizer = 1;
}
@@ -1507,8 +1504,8 @@ def Vector_ScatterOp :
Example:
```mlir
-vector.scatter %base, %indices, %mask, %value
-: vector<16xi32>, vector<16xi1>, vector<16xf32> into memref
+vector.scatter %base[%indices], %mask, %value
+: memref, vector<16xi32>, vector<16xi1>, vector<16xf32>
```
}];
let extraClassDeclaration = [{
@@ -1525,8 +1522,8 @@ def Vector_ScatterOp :
return value().getType().cast();
}
}];
- let assemblyFormat = "$base `,` $indices `,` $mask `,` $value attr-dict `:` "
-"type($indices) `,` type($mask) `,` type($value) `into` type($base)";
+ let assemblyFormat = "$base `[` $indices `]` `,` $mask `,` $value attr-dict
`:` "
+"type($base) `,` type($indices) `,` type($mask) `,` type($value)";
let hasCanonicalizer = 1;
}
diff --git a/mlir/integration_test/Dialect/Vector/CPU/test-gather.mlir
b/mlir/integration_test/Dialect/Vector/CPU/test-gather.mlir
index 5ed8f3ee38f8..95df5aea06e4 100644
--- a/mlir/integration_test/Dialect/Vector/CPU/test-gath
[llvm-branch-commits] [mlir] 046612d - [mlir][vector] verify memref of vector memory ops
Author: Aart Bik
Date: 2021-01-11T13:32:39-08:00
New Revision: 046612d29d7894783e8fcecbc62ebd6b4a78499f
URL:
https://github.com/llvm/llvm-project/commit/046612d29d7894783e8fcecbc62ebd6b4a78499f
DIFF:
https://github.com/llvm/llvm-project/commit/046612d29d7894783e8fcecbc62ebd6b4a78499f.diff
LOG: [mlir][vector] verify memref of vector memory ops
This ensures the memref base + indices expression is well-formed
Reviewed By: ThomasRaoux, ftynse
Differential Revision: https://reviews.llvm.org/D94441
Added:
Modified:
mlir/lib/Dialect/Vector/VectorOps.cpp
mlir/test/Dialect/Vector/invalid.mlir
Removed:
diff --git a/mlir/lib/Dialect/Vector/VectorOps.cpp
b/mlir/lib/Dialect/Vector/VectorOps.cpp
index 731ddae85ead..54e5e008e56f 100644
--- a/mlir/lib/Dialect/Vector/VectorOps.cpp
+++ b/mlir/lib/Dialect/Vector/VectorOps.cpp
@@ -2365,10 +2365,12 @@ static LogicalResult verify(MaskedLoadOp op) {
VectorType maskVType = op.getMaskVectorType();
VectorType passVType = op.getPassThruVectorType();
VectorType resVType = op.getResultVectorType();
+ MemRefType memType = op.getMemRefType();
- if (resVType.getElementType() != op.getMemRefType().getElementType())
+ if (resVType.getElementType() != memType.getElementType())
return op.emitOpError("base and result element type should match");
-
+ if (llvm::size(op.indices()) != memType.getRank())
+return op.emitOpError("requires ") << memType.getRank() << " indices";
if (resVType.getDimSize(0) != maskVType.getDimSize(0))
return op.emitOpError("expected result dim to match mask dim");
if (resVType != passVType)
@@ -2410,10 +2412,12 @@ void MaskedLoadOp::getCanonicalizationPatterns(
static LogicalResult verify(MaskedStoreOp op) {
VectorType maskVType = op.getMaskVectorType();
VectorType valueVType = op.getValueVectorType();
+ MemRefType memType = op.getMemRefType();
- if (valueVType.getElementType() != op.getMemRefType().getElementType())
+ if (valueVType.getElementType() != memType.getElementType())
return op.emitOpError("base and value element type should match");
-
+ if (llvm::size(op.indices()) != memType.getRank())
+return op.emitOpError("requires ") << memType.getRank() << " indices";
if (valueVType.getDimSize(0) != maskVType.getDimSize(0))
return op.emitOpError("expected value dim to match mask dim");
return success();
@@ -2454,10 +2458,10 @@ static LogicalResult verify(GatherOp op) {
VectorType indicesVType = op.getIndicesVectorType();
VectorType maskVType = op.getMaskVectorType();
VectorType resVType = op.getResultVectorType();
+ MemRefType memType = op.getMemRefType();
- if (resVType.getElementType() != op.getMemRefType().getElementType())
+ if (resVType.getElementType() != memType.getElementType())
return op.emitOpError("base and result element type should match");
-
if (resVType.getDimSize(0) != indicesVType.getDimSize(0))
return op.emitOpError("expected result dim to match indices dim");
if (resVType.getDimSize(0) != maskVType.getDimSize(0))
@@ -2500,10 +2504,10 @@ static LogicalResult verify(ScatterOp op) {
VectorType indicesVType = op.getIndicesVectorType();
VectorType maskVType = op.getMaskVectorType();
VectorType valueVType = op.getValueVectorType();
+ MemRefType memType = op.getMemRefType();
- if (valueVType.getElementType() != op.getMemRefType().getElementType())
+ if (valueVType.getElementType() != memType.getElementType())
return op.emitOpError("base and value element type should match");
-
if (valueVType.getDimSize(0) != indicesVType.getDimSize(0))
return op.emitOpError("expected value dim to match indices dim");
if (valueVType.getDimSize(0) != maskVType.getDimSize(0))
@@ -2544,10 +2548,12 @@ static LogicalResult verify(ExpandLoadOp op) {
VectorType maskVType = op.getMaskVectorType();
VectorType passVType = op.getPassThruVectorType();
VectorType resVType = op.getResultVectorType();
+ MemRefType memType = op.getMemRefType();
- if (resVType.getElementType() != op.getMemRefType().getElementType())
+ if (resVType.getElementType() != memType.getElementType())
return op.emitOpError("base and result element type should match");
-
+ if (llvm::size(op.indices()) != memType.getRank())
+return op.emitOpError("requires ") << memType.getRank() << " indices";
if (resVType.getDimSize(0) != maskVType.getDimSize(0))
return op.emitOpError("expected result dim to match mask dim");
if (resVType != passVType)
@@ -2589,10 +2595,12 @@ void ExpandLoadOp::getCanonicalizationPatterns(
static LogicalResult verify(CompressStoreOp op) {
VectorType maskVType = op.getMaskVectorType();
VectorType valueVType = op.getValueVectorType();
+ MemRefType memType = op.getMemRefType();
- if (valueVType.getElementType() != op.getMemRefType().getElementType())
+ if (valueVType.getElementType() != memType.getElementType())
[llvm-branch-commits] [mlir] 106e66f - [mlir][ArmSVE] Add documentation generation
Author: Javier Setoain Date: 2020-12-17T12:22:48-08:00 New Revision: 106e66f3f555c8f887e82c5f04c3e77bdaf345e8 URL: https://github.com/llvm/llvm-project/commit/106e66f3f555c8f887e82c5f04c3e77bdaf345e8 DIFF: https://github.com/llvm/llvm-project/commit/106e66f3f555c8f887e82c5f04c3e77bdaf345e8.diff LOG: [mlir][ArmSVE] Add documentation generation Adds missing cmake command to generate documentation for ArmSVE Dialect. Reviewed By: aartbik Differential Revision: https://reviews.llvm.org/D93465 Added: Modified: mlir/include/mlir/Dialect/ArmSVE/CMakeLists.txt Removed: diff --git a/mlir/include/mlir/Dialect/ArmSVE/CMakeLists.txt b/mlir/include/mlir/Dialect/ArmSVE/CMakeLists.txt index fb50fac68f33..c7db56122a95 100644 --- a/mlir/include/mlir/Dialect/ArmSVE/CMakeLists.txt +++ b/mlir/include/mlir/Dialect/ArmSVE/CMakeLists.txt @@ -1 +1,2 @@ add_mlir_dialect(ArmSVE arm_sve ArmSVE) +add_mlir_doc(ArmSVE -gen-dialect-doc ArmSVE Dialects/) ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] 14da25b - [mlir][sparse] scalarize reductions in for-loops during sparse codegen
Author: Aart Bik
Date: 2020-12-17T16:12:21-08:00
New Revision: 14da25b4b2eedf8a16aae34edfefd7bcaa5ceae5
URL:
https://github.com/llvm/llvm-project/commit/14da25b4b2eedf8a16aae34edfefd7bcaa5ceae5
DIFF:
https://github.com/llvm/llvm-project/commit/14da25b4b2eedf8a16aae34edfefd7bcaa5ceae5.diff
LOG: [mlir][sparse] scalarize reductions in for-loops during sparse codegen
Reductions in innermost loops become harder for the backend to disambiguate
after bufferization into memrefs, resulting in less efficient load-update-store
cycles. By scalarizing innermost reductions, the backend is more likely to
assign
a register to perform the reduction (also prepares vectorization). Even though
we could scalarize reductions for more outer loops and while-loops as well,
currently scalarization is only done for chains of innermost for-loops, where
it matters most, to avoid complicating codegen unnecessary (viz. adding lots
of yield instructions).
This CL also refactors condition simplification into the merger class,
where it belongs, so that conditions are simplified only once per loop
nest and not repeatedly as was currently done. This CL also fixes a few
minor bugs, some layout issues, and comments.
Reviewed By: penpornk
Differential Revision: https://reviews.llvm.org/D93143
Added:
Modified:
mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
mlir/test/Dialect/Linalg/sparse_1d.mlir
mlir/test/Dialect/Linalg/sparse_2d.mlir
mlir/test/Dialect/Linalg/sparse_3d.mlir
Removed:
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
index cfdb371e3234..fed2eedd41a4 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
@@ -52,6 +52,7 @@ using namespace mlir;
namespace {
enum class Kind { kTensor, kInvariant, kMulF, kMulI, kAddF, kAddI };
+enum class Dim { kSparse, kDense, kUndef };
/// Tensor expression. Represents a MLIR expression in tensor index notation.
/// For tensors, e0 denotes the tensor index. For invariants, the IR value is
@@ -81,8 +82,13 @@ struct LatPoint {
bits.set(b);
}
LatPoint(const llvm::BitVector &b, unsigned e) : bits(b), exp(e) {}
- /// Conjunction of tensor loop indices as bitvector.
+ /// Conjunction of tensor loop indices as bitvector. This represents
+ /// all indices involved in the tensor expression
llvm::BitVector bits;
+ /// Simplified conjunction of tensor loop indices as bitvector. This
+ /// represents a simplified condition under which this tensor expression
+ /// must execute. Pre-computed during codegen to avoid repeated eval.
+ llvm::BitVector simple;
/// Index of the tensor expresssion.
unsigned exp;
};
@@ -93,8 +99,14 @@ struct LatPoint {
/// independently from the basic algorithm if bottlenecks are identified.
class Merger {
public:
+ /// Constructs a merger for the given number of tensors and loops. The
+ /// user supplies the number of tensors involved in the kernel, with the
+ /// last tensor in this set denoting the output tensor. The merger adds an
+ /// additional synthetic tensor at the end of this set to represent all
+ /// invariant expressions in the kernel.
Merger(unsigned t, unsigned l)
- : numTensors(t), numLoops(l), isSparse(t, std::vector(l, false)) {}
+ : outTensor(t - 1), numTensors(t + 1), numLoops(l),
+dims(t + 1, std::vector(l, Dim::kUndef)) {}
/// Adds a tensor expression. Returns its index.
unsigned addExp(Kind k, unsigned e0, unsigned e1 = -1u, Value v = Value()) {
@@ -132,8 +144,8 @@ class Merger {
return p;
}
- /// Conjunctive merge of L1 and L2 is conjunction of cartesian product.
- /// Returns the index of the new set.
+ /// Conjunctive merge of two lattice sets L0 and L1 is conjunction of
+ /// cartesian product. Returns the index of the new set.
unsigned takeConj(Kind kind, unsigned s0, unsigned s1) {
unsigned s = addSet();
for (unsigned p0 : latSets[s0])
@@ -142,7 +154,7 @@ class Merger {
return s;
}
- /// Disjunctive merge of L0 and L1 is (L0 /\_op L1, L0, L1).
+ /// Disjunctive merge of two lattice sets L0 and L1 is (L0 /\_op L1, L0, L1).
/// Returns the index of the new set.
unsigned takeDisj(Kind kind, unsigned s0, unsigned s1) {
unsigned s = takeConj(kind, s0, s1);
@@ -156,26 +168,27 @@ class Merger {
/// Optimizes the iteration lattice points in the given set. This
/// method should be called right before code generation to avoid
/// generating redundant loops and conditions.
- unsigned optimize(unsigned s0) {
+ unsigned optimizeSet(unsigned s0) {
unsigned s = addSet();
assert(latSets[s0].size() != 0);
unsigned p0 = latSets[s0][0];
for (unsigned p1 : latSets[s0]) {
bool add = true;
+ llvm::BitVector simple = simplifyCond(s0, p1);
[llvm-branch-commits] [mlir] 9a8cab8 - [mlir][sparse] adjust output tensor to synthetic tensor
Author: Aart Bik Date: 2020-12-21T14:13:54-08:00 New Revision: 9a8cab8bacc12d48d74249d868082effe132029e URL: https://github.com/llvm/llvm-project/commit/9a8cab8bacc12d48d74249d868082effe132029e DIFF: https://github.com/llvm/llvm-project/commit/9a8cab8bacc12d48d74249d868082effe132029e.diff LOG: [mlir][sparse] adjust output tensor to synthetic tensor Fixes a merge conflict with previous two CLs. Reviewed By: mravishankar Differential Revision: https://reviews.llvm.org/D93664 Added: Modified: mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp Removed: diff --git a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp index eb940d0f769b..a6b7277e47e3 100644 --- a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp +++ b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp @@ -466,8 +466,8 @@ static unsigned buildLattices(Merger &merger, linalg::GenericOp op, // set to the undefined index in that dimension. An invariant expression // is set to a synthetic tensor with undefined indices only. unsigned s = merger.addSet(); -unsigned t = kind == Kind::kTensor ? merger.exp(exp).e0 - : op.getNumShapedOperands() - 1; +unsigned t = +kind == Kind::kTensor ? merger.exp(exp).e0 : op.getNumShapedOperands(); merger.set(s).push_back(merger.addLat(t, idx, exp)); return s; } ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] 5c4e397 - [mlir][sparse] add parallelization strategies to sparse compiler
Author: Aart Bik
Date: 2020-11-24T17:17:13-08:00
New Revision: 5c4e397e6ce5c89d63f590857e5cb0e80237de62
URL:
https://github.com/llvm/llvm-project/commit/5c4e397e6ce5c89d63f590857e5cb0e80237de62
DIFF:
https://github.com/llvm/llvm-project/commit/5c4e397e6ce5c89d63f590857e5cb0e80237de62.diff
LOG: [mlir][sparse] add parallelization strategies to sparse compiler
This CL adds the ability to request different parallelization strategies
for the generate code. Every "parallel" loop is a candidate, and converted
to a parallel op if it is an actual for-loop (not a while) and the strategy
allows dense/sparse outer/inner parallelization.
This will connect directly with the work of @ezhulenev on parallel loops.
Still TBD: vectorization strategy
Reviewed By: penpornk
Differential Revision: https://reviews.llvm.org/D91978
Added:
mlir/test/Dialect/Linalg/sparse_parallel.mlir
Modified:
mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
mlir/test/lib/Transforms/TestSparsification.cpp
Removed:
diff --git a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
index d67e81ceab87..7a96a5ed1390 100644
--- a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
@@ -783,9 +783,62 @@ LogicalResult applyStagedPatterns(
//===--===//
// Support for sparse tensor code generation.
+//
+// The sparse compiler part of MLIR lowers a tensor expression formulated as a
+// Linalg operation into a sequence of loops depending on what dimensions of
the
+// tensors are marked dense or sparse. The generated code distinguishes
between:
+// (1) for-loops that iterate over a single dense dimension,
+// (2) for-loops that iterate over a single sparse dimension,
+// (3) while-loops that co-iterate over several sparse dimensions.
+// The for-loops may be subsequently optimized for parallel or vector
execution.
+//
+// For more details, the Dialect/Linalg/Transforms/Sparsification.cpp file.
//===--===//
-void populateSparsificationPatterns(MLIRContext *context,
-OwningRewritePatternList &patterns);
+
+/// Defines a parallelization strategy. Any implicit loop in the Linalg
+/// operation that is marked "parallel" (thus not "reduction") is a candidate
+/// for parallelization. The loop is made parallel if (1) allowed by the
+/// strategy (e.g., AnyStorageOuterLoop considers either a dense or sparse
+/// outermost loop only), and (2) the generated code is an actual for-loop
+/// (and not a co-iterating while-loop).
+enum class SparseParallelizationStrategy {
+ kNone,
+ kDenseOuterLoop,
+ kAnyStorageOuterLoop,
+ kDenseAnyLoop,
+ kAnyStorageAnyLoop
+ // TODO: support reduction parallelization too?
+};
+
+/// Defines a vectorization strategy. Any implicit inner loop in the Linalg
+/// operation is a candidate (full SIMD for "parallel" loops and horizontal
+/// SIMD for "reduction" loops). A loop is actually vectorized if (1) allowed
+/// by the strategy, and (2) the emitted code is an actual for-loop (and not
+/// a co-iterating while-loop).
+enum class SparseVectorizationStrategy {
+ kNone,
+ kDenseInnerLoop,
+ kAnyStorageInnerLoop
+};
+
+/// Sparsification options.
+struct SparsificationOptions {
+ SparsificationOptions(SparseParallelizationStrategy p,
+SparseVectorizationStrategy v, unsigned vl)
+ : parallelizationStrategy(p), vectorizationStrategy(v), vectorLength(vl)
{
+ }
+ SparsificationOptions()
+ : SparsificationOptions(SparseParallelizationStrategy::kNone,
+ SparseVectorizationStrategy::kNone, 1u) {}
+ SparseParallelizationStrategy parallelizationStrategy;
+ SparseVectorizationStrategy vectorizationStrategy;
+ unsigned vectorLength;
+};
+
+/// Set up sparsification rewriting rules with the given options.
+void populateSparsificationPatterns(
+MLIRContext *context, OwningRewritePatternList &patterns,
+const SparsificationOptions &options = SparsificationOptions());
} // namespace linalg
} // namespace mlir
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
index 69a4d7e5648e..729268393ed9 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
@@ -235,22 +235,30 @@ class Merger {
// Code generation.
struct CodeGen {
- CodeGen(unsigned numTensors, unsigned numLoops)
- : loops(numLoops), sizes(numLoops), buffers(numTensors),
+ CodeGen(linalg::SparsificationOptions o, unsigned numTensors,
+ unsigned numLoops)
+ : o
[llvm-branch-commits] [mlir] d5f0d0c - [mlir][sparse] add ability to select pointer/index storage type
Author: Aart Bik
Date: 2020-11-25T17:32:44-08:00
New Revision: d5f0d0c0c4117295d9e76bbafaf0597e01ef3c99
URL:
https://github.com/llvm/llvm-project/commit/d5f0d0c0c4117295d9e76bbafaf0597e01ef3c99
DIFF:
https://github.com/llvm/llvm-project/commit/d5f0d0c0c4117295d9e76bbafaf0597e01ef3c99.diff
LOG: [mlir][sparse] add ability to select pointer/index storage type
This change gives sparse compiler clients more control over selecting
individual types for the pointers and indices in the sparse storage schemes.
Narrower width obviously results in smaller memory footprints, but the
range should always suffice for the maximum number of entries or index value.
Reviewed By: penpornk
Differential Revision: https://reviews.llvm.org/D92126
Added:
mlir/test/Dialect/Linalg/sparse_storage.mlir
Modified:
mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
mlir/test/lib/Transforms/TestSparsification.cpp
Removed:
diff --git a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
index 7a96a5ed1390..b37a14f0eb7a 100644
--- a/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
+++ b/mlir/include/mlir/Dialect/Linalg/Transforms/Transforms.h
@@ -821,18 +821,31 @@ enum class SparseVectorizationStrategy {
kAnyStorageInnerLoop
};
+/// Defines a type for "pointer" and "index" storage in the sparse storage
+/// scheme, with a choice between the native platform-dependent index width,
+/// 64-bit integers, or 32-bit integers. A narrow width obviously reduces
+/// the memory footprint of the sparse storage scheme, but the width should
+/// suffice to define the total required range (viz. the maximum number of
+/// stored entries per indirection level for the "pointers" and the maximum
+/// value of each tensor index over all dimensions for the "indices").
+enum class SparseIntType { kNative, kI64, kI32 };
+
/// Sparsification options.
struct SparsificationOptions {
SparsificationOptions(SparseParallelizationStrategy p,
-SparseVectorizationStrategy v, unsigned vl)
- : parallelizationStrategy(p), vectorizationStrategy(v), vectorLength(vl)
{
- }
+SparseVectorizationStrategy v, unsigned vl,
+SparseIntType pt, SparseIntType it)
+ : parallelizationStrategy(p), vectorizationStrategy(v), vectorLength(vl),
+ptrType(pt), indType(it) {}
SparsificationOptions()
: SparsificationOptions(SparseParallelizationStrategy::kNone,
- SparseVectorizationStrategy::kNone, 1u) {}
+ SparseVectorizationStrategy::kNone, 1u,
+ SparseIntType::kNative, SparseIntType::kNative)
{}
SparseParallelizationStrategy parallelizationStrategy;
SparseVectorizationStrategy vectorizationStrategy;
unsigned vectorLength;
+ SparseIntType ptrType;
+ SparseIntType indType;
};
/// Set up sparsification rewriting rules with the given options.
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
index 729268393ed9..07a3e1569622 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
@@ -420,16 +420,27 @@ static unsigned buildLattices(Merger &merger,
linalg::GenericOp op,
}
}
+/// Maps sparse integer option to actual integral storage type.
+static Type genIntType(PatternRewriter &rewriter, linalg::SparseIntType tp) {
+ switch (tp) {
+ case linalg::SparseIntType::kNative:
+return rewriter.getIndexType();
+ case linalg::SparseIntType::kI64:
+return rewriter.getIntegerType(64);
+ case linalg::SparseIntType::kI32:
+return rewriter.getIntegerType(32);
+ }
+}
+
/// Local bufferization of all dense and sparse data structures.
/// This code enables testing the first prototype sparse compiler.
// TODO: replace this with a proliferated bufferization strategy
-void genBuffers(Merger &merger, CodeGen &codegen, PatternRewriter &rewriter,
-linalg::GenericOp op) {
+static void genBuffers(Merger &merger, CodeGen &codegen,
+ PatternRewriter &rewriter, linalg::GenericOp op) {
Location loc = op.getLoc();
unsigned numTensors = op.getNumInputsAndOutputs();
unsigned numInputs = op.getNumInputs();
assert(numTensors == numInputs + 1);
- Type indexType = rewriter.getIndexType();
// For now, set all unknown dimensions to 999.
// TODO: compute these values (using sparsity or by reading tensor)
@@ -450,9 +461,13 @@ void genBuffers(Merger &merger, CodeGen &codegen,
PatternRewriter &rewriter,
// Handle sparse storage schemes.
if (merger.isSparseAccess(t, i)) {
allDense = false;
-auto dynTp = MemRefType::get({ShapedT
[llvm-branch-commits] [mlir] c95acf0 - [mlir][vector][avx512] move avx512 lowering pass into general vector lowering
Author: Aart Bik
Date: 2020-12-03T17:23:46-08:00
New Revision: c95acf052b53e5c18e380b8632e7de24b5e65dbe
URL:
https://github.com/llvm/llvm-project/commit/c95acf052b53e5c18e380b8632e7de24b5e65dbe
DIFF:
https://github.com/llvm/llvm-project/commit/c95acf052b53e5c18e380b8632e7de24b5e65dbe.diff
LOG: [mlir][vector][avx512] move avx512 lowering pass into general vector
lowering
A separate AVX512 lowering pass does not compose well with the regular
vector lowering pass. As such, it is at risk of code duplication and
lowering inconsistencies. This change removes the separate AVX512 lowering
pass and makes it an "option" in the regular vector lowering pass
(viz. vector dialect "augmented" with AVX512 dialect).
Reviewed By: rriddle
Differential Revision: https://reviews.llvm.org/D92614
Added:
mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVMPass.cpp
Modified:
mlir/include/mlir/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.h
mlir/include/mlir/Conversion/Passes.h
mlir/include/mlir/Conversion/Passes.td
mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
mlir/lib/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.cpp
mlir/lib/Conversion/VectorToLLVM/CMakeLists.txt
mlir/lib/Conversion/VectorToLLVM/ConvertVectorToLLVM.cpp
mlir/lib/Dialect/AVX512/CMakeLists.txt
mlir/test/Conversion/AVX512ToLLVM/convert-to-llvm.mlir
Removed:
diff --git a/mlir/include/mlir/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.h
b/mlir/include/mlir/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.h
index aff5c4ca2c70..06f2958a2d5a 100644
--- a/mlir/include/mlir/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.h
+++ b/mlir/include/mlir/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.h
@@ -9,21 +9,15 @@
#ifndef MLIR_CONVERSION_AVX512TOLLVM_CONVERTAVX512TOLLVM_H_
#define MLIR_CONVERSION_AVX512TOLLVM_CONVERTAVX512TOLLVM_H_
-#include
-
namespace mlir {
+
class LLVMTypeConverter;
-class ModuleOp;
-template class OperationPass;
class OwningRewritePatternList;
/// Collect a set of patterns to convert from the AVX512 dialect to LLVM.
void populateAVX512ToLLVMConversionPatterns(LLVMTypeConverter &converter,
OwningRewritePatternList
&patterns);
-/// Create a pass to convert AVX512 operations to the LLVMIR dialect.
-std::unique_ptr> createConvertAVX512ToLLVMPass();
-
} // namespace mlir
#endif // MLIR_CONVERSION_AVX512TOLLVM_CONVERTAVX512TOLLVM_H_
diff --git a/mlir/include/mlir/Conversion/Passes.h
b/mlir/include/mlir/Conversion/Passes.h
index 0d2281f99581..cc4f59c12496 100644
--- a/mlir/include/mlir/Conversion/Passes.h
+++ b/mlir/include/mlir/Conversion/Passes.h
@@ -9,7 +9,6 @@
#ifndef MLIR_CONVERSION_PASSES_H
#define MLIR_CONVERSION_PASSES_H
-#include "mlir/Conversion/AVX512ToLLVM/ConvertAVX512ToLLVM.h"
#include "mlir/Conversion/AffineToStandard/AffineToStandard.h"
#include "mlir/Conversion/AsyncToLLVM/AsyncToLLVM.h"
#include "mlir/Conversion/GPUCommon/GPUCommonPass.h"
diff --git a/mlir/include/mlir/Conversion/Passes.td
b/mlir/include/mlir/Conversion/Passes.td
index fdf01b7f93d0..6c99d84ea30a 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -73,17 +73,6 @@ def ConvertAffineToStandard : Pass<"lower-affine"> {
];
}
-//===--===//
-// AVX512ToLLVM
-//===--===//
-
-def ConvertAVX512ToLLVM : Pass<"convert-avx512-to-llvm", "ModuleOp"> {
- let summary = "Convert the operations from the avx512 dialect into the LLVM "
-"dialect";
- let constructor = "mlir::createConvertAVX512ToLLVMPass()";
- let dependentDialects = ["LLVM::LLVMDialect", "LLVM::LLVMAVX512Dialect"];
-}
-
//===--===//
// AsyncToLLVM
//===--===//
@@ -401,15 +390,28 @@ def ConvertVectorToSCF :
FunctionPass<"convert-vector-to-scf"> {
def ConvertVectorToLLVM : Pass<"convert-vector-to-llvm", "ModuleOp"> {
let summary = "Lower the operations from the vector dialect into the LLVM "
"dialect";
+ let description = [{
+
+Convert operations from the vector dialect into the LLVM IR dialect
+operations. The lowering pass provides several options to control
+the kind of optimizations that are allowed. It also provides options
+that augment the architectural-neutral vector dialect with
+architectural-specific dialects (AVX512, Neon, etc.).
+
+ }];
let constructor = "mlir::createConvertVectorToLLVMPass()";
- let dependentDialects = ["LLVM::LLVMDialect"];
+ let dependentDialects = ["LLVM::LLVMDialect", "LLVM::LLVMAVX512Dialect"];
let options = [
Option<"reassociateFPReductions", "reassociate-fp-reductions",
[llvm-branch-commits] [mlir] fc7818f - [mlir][vector] rephrased description
Author: Aart Bik
Date: 2020-12-04T14:00:49-08:00
New Revision: fc7818f5d6906555cebad2c2e7c313a383b9cb82
URL:
https://github.com/llvm/llvm-project/commit/fc7818f5d6906555cebad2c2e7c313a383b9cb82
DIFF:
https://github.com/llvm/llvm-project/commit/fc7818f5d6906555cebad2c2e7c313a383b9cb82.diff
LOG: [mlir][vector] rephrased description
More carefully worded description. Added constructor to options.
Reviewed By: nicolasvasilache
Differential Revision: https://reviews.llvm.org/D92664
Added:
Modified:
mlir/include/mlir/Conversion/Passes.td
mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
Removed:
diff --git a/mlir/include/mlir/Conversion/Passes.td
b/mlir/include/mlir/Conversion/Passes.td
index 6c99d84ea30a..53158afa0530 100644
--- a/mlir/include/mlir/Conversion/Passes.td
+++ b/mlir/include/mlir/Conversion/Passes.td
@@ -394,9 +394,10 @@ def ConvertVectorToLLVM : Pass<"convert-vector-to-llvm",
"ModuleOp"> {
Convert operations from the vector dialect into the LLVM IR dialect
operations. The lowering pass provides several options to control
-the kind of optimizations that are allowed. It also provides options
-that augment the architectural-neutral vector dialect with
-architectural-specific dialects (AVX512, Neon, etc.).
+the kinds of optimizations that are allowed. It also provides options
+that enable the use of one or more architectural-specific dialects
+(AVX512, Neon, SVE, etc.) in combination with the architectural-neutral
+vector dialect lowering.
}];
let constructor = "mlir::createConvertVectorToLLVMPass()";
@@ -411,7 +412,8 @@ def ConvertVectorToLLVM : Pass<"convert-vector-to-llvm",
"ModuleOp"> {
"faster code">,
Option<"enableAVX512", "enable-avx512",
"bool", /*default=*/"false",
- "Augments the vector dialect with the AVX512 dialect during
lowering">
+ "Enables the use of AVX512 dialect while lowering the vector "
+ "dialect.">
];
}
diff --git a/mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
b/mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
index 435a148e2b10..e6a515aa4564 100644
--- a/mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
+++ b/mlir/include/mlir/Conversion/VectorToLLVM/ConvertVectorToLLVM.h
@@ -21,9 +21,10 @@ class OperationPass;
/// This should kept in sync with VectorToLLVM options defined for the
/// ConvertVectorToLLVM pass in include/mlir/Conversion/Passes.td
struct LowerVectorToLLVMOptions {
- bool reassociateFPReductions = false;
- bool enableIndexOptimizations = true;
- bool enableAVX512 = false;
+ LowerVectorToLLVMOptions()
+ : reassociateFPReductions(false), enableIndexOptimizations(true),
+enableAVX512(false) {}
+
LowerVectorToLLVMOptions &setReassociateFPReductions(bool b) {
reassociateFPReductions = b;
return *this;
@@ -36,6 +37,10 @@ struct LowerVectorToLLVMOptions {
enableAVX512 = b;
return *this;
}
+
+ bool reassociateFPReductions;
+ bool enableIndexOptimizations;
+ bool enableAVX512;
};
/// Collect a set of patterns to convert from Vector contractions to LLVM
Matrix
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [mlir] 74cd9e5 - [mlir][sparse] hoist loop invariant tensor loads in sparse compiler
Author: Aart Bik
Date: 2020-12-07T11:59:48-08:00
New Revision: 74cd9e587d80063381242006d0690231d756aa7a
URL:
https://github.com/llvm/llvm-project/commit/74cd9e587d80063381242006d0690231d756aa7a
DIFF:
https://github.com/llvm/llvm-project/commit/74cd9e587d80063381242006d0690231d756aa7a.diff
LOG: [mlir][sparse] hoist loop invariant tensor loads in sparse compiler
After bufferization, the backend has much more trouble hoisting loop invariant
loads from the loops generated by the sparse compiler. Therefore, this is done
during sparse code generation. Note that we don't bother hoisting derived
invariant expressions on SSA values, since the backend does that very well.
Still TBD: scalarize reductions to avoid load-add-store cycles
Reviewed By: penpornk
Differential Revision: https://reviews.llvm.org/D92534
Added:
Modified:
mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
mlir/test/Dialect/Linalg/sparse_2d.mlir
mlir/test/Dialect/Linalg/sparse_3d.mlir
Removed:
diff --git a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
index 07a3e1569622..cfdb371e3234 100644
--- a/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
+++ b/mlir/lib/Dialect/Linalg/Transforms/Sparsification.cpp
@@ -59,14 +59,21 @@ enum class Kind { kTensor, kInvariant, kMulF, kMulI, kAddF,
kAddI };
/// children tensor expressions.
struct TensorExp {
TensorExp(Kind k, unsigned x, unsigned y, Value v)
- : kind(k), e0(x), e1(y), val(v) {}
+ : kind(k), e0(x), e1(y), val(v) {
+assert((kind == Kind::kTensor && e0 != -1u && e1 == -1u && !val) ||
+ (kind == Kind::kInvariant && e0 == -1u && e1 == -1u && val) ||
+ (kind >= Kind::kMulF && e0 != -1u && e1 != -1u && !val));
+ }
Kind kind;
+ /// Indices of children expression(s).
unsigned e0;
unsigned e1;
+ /// Direct link to IR for an invariant. During code generation,
+ /// field is used to cache "hoisted" loop invariant tensor loads.
Value val;
};
-/// Lattice point. Each lattice point consist of a conjunction of tensor
+/// Lattice point. Each lattice point consists of a conjunction of tensor
/// loop indices (encoded in a bitvector) and the index of the corresponding
/// tensor expression.
struct LatPoint {
@@ -74,7 +81,9 @@ struct LatPoint {
bits.set(b);
}
LatPoint(const llvm::BitVector &b, unsigned e) : bits(b), exp(e) {}
+ /// Conjunction of tensor loop indices as bitvector.
llvm::BitVector bits;
+ /// Index of the tensor expresssion.
unsigned exp;
};
@@ -502,8 +511,16 @@ static void genBuffers(Merger &merger, CodeGen &codegen,
/// Generates a load on a dense or sparse tensor.
static Value genTensorLoad(Merger &merger, CodeGen &codegen,
PatternRewriter &rewriter, linalg::GenericOp op,
- unsigned tensor) {
+ unsigned exp) {
+ // Test if the load was hoisted to a higher loop nest.
+ Value val = merger.exp(exp).val;
+ if (val) {
+merger.exp(exp).val = Value(); // reset
+return val;
+ }
+ // Actual load.
SmallVector args;
+ unsigned tensor = merger.exp(exp).e0;
auto map = op.getIndexingMap(tensor);
bool sparse = false;
for (unsigned i = 0, m = map.getNumResults(); i < m; ++i) {
@@ -515,7 +532,9 @@ static Value genTensorLoad(Merger &merger, CodeGen &codegen,
args.push_back(codegen.pidxs[tensor][idx]); // position index
}
}
- return rewriter.create(op.getLoc(), codegen.buffers[tensor], args);
+ Location loc = op.getLoc();
+ Value ptr = codegen.buffers[tensor];
+ return rewriter.create(loc, ptr, args);
}
/// Generates a store on a dense tensor.
@@ -528,25 +547,33 @@ static void genTensorStore(Merger &merger, CodeGen
&codegen,
unsigned idx = map.getDimPosition(i);
args.push_back(codegen.loops[idx]); // universal dense index
}
- rewriter.create(op.getLoc(), rhs, codegen.buffers[tensor], args);
+ Location loc = op.getLoc();
+ Value ptr = codegen.buffers[tensor];
+ rewriter.create(loc, rhs, ptr, args);
}
/// Generates a pointer/index load from the sparse storage scheme.
-static Value genIntLoad(PatternRewriter &rewriter, Location loc, Value ptr,
-Value s) {
+static Value genLoad(PatternRewriter &rewriter, Location loc, Value ptr,
+ Value s) {
Value load = rewriter.create(loc, ptr, s);
return load.getType().isa()
? load
: rewriter.create(loc, load,
rewriter.getIndexType());
}
+/// Generates an invariant value.
+static Value genInvariantValue(Merger &merger, CodeGen &codegen,
+ PatternRewriter &rewriter, unsigned exp) {
+ return merger.exp(exp).val;
+}
+
/// Recursively generates tensor expression.
static Value genExp(Merger &merger, CodeGen &codegen, PatternRewriter
&rewriter,
[llvm-branch-commits] [flang] [mlir] [mlir][Transforms] Support 1:N mappings in `ConversionValueMapping` (PR #116524)
https://github.com/aartbik commented: +1 on the simplification for the sparse 1:N conversion! https://github.com/llvm/llvm-project/pull/116524 ___ llvm-branch-commits mailing list llvm-branch-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
