[PATCH] D144911: adding bf16 support to NVPTX

bkramer accepted this revision. bkramer added a comment. This revision is now accepted and ready to land. I don't have much knowledge about the PTX ABI, but everything here makes perfect sense to me. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D14

[PATCH] D144911: adding bf16 support to NVPTX

tra added a comment. @bkramer Ben, PTAL when you get a chance. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 ___ cfe-commits mailing list cfe-commits@lists.ll

[PATCH] D144911: adding bf16 support to NVPTX

tra planned changes to this revision. tra added a comment. We're still missing clang-side tests for the new builtins. Now that the intrinsics use `bfloat` we also need to change builtin signatures. Or change codegen to bitcast to/from bfloat to match the types. To be continued next week. Reposi

[PATCH] D144911: adding bf16 support to NVPTX

tra updated this revision to Diff 534130. tra added a comment. Fixed few missed places in bf16 lowering. Changed intrinsic types to use bfloat type. Auto-upgrade the old intrinsic variants. Updated broken tests. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.l

[PATCH] D144911: adding bf16 support to NVPTX

tra commandeered this revision. tra edited reviewers, added: kushanam; removed: tra. tra added a comment. This revision now requires review to proceed. Herald added a subscriber: bixia. I've got a few more fixes for the patch. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION ht

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 534050. kushanam added a comment. fixing the f16x2-instructions test Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/include/clang/Basic/BuiltinsNVPTX

[PATCH] D144911: adding bf16 support to NVPTX

tra added a comment. The latest patch revision still fails on a few LLVM tests: Failed Tests (3): LLVM :: CodeGen/NVPTX/bf16-instructions.ll LLVM :: CodeGen/NVPTX/f16x2-instructions.ll LLVM :: CodeGen/NVPTX/math-intrins-sm80-ptx70.ll Comment at: llvm/lib/Target/N

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 533694. kushanam added a comment. Updating the bf16 conversion arc requirements Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/include/clang/Basic/Bu

[PATCH] D144911: adding bf16 support to NVPTX

tra added inline comments. Comment at: llvm/lib/Target/NVPTX/NVPTXInstrInfo.td:559-568 -multiclass CVT_FROM_FLOAT_SM80 { -def _f32 : - NVPTXInst<(outs RC:$dst), -(ins Float32Regs:$src, CvtMode:$mode), -!strconcat("cvt${mode:base}${mode:rel

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 533051. kushanam added a comment. fixing bf16 test cases Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/include/clang/Basic/BuiltinsNVPTX.def llvm/

[PATCH] D144911: adding bf16 support to NVPTX

tra added inline comments. Comment at: llvm/lib/Target/NVPTX/NVPTXIntrinsics.td:1271-1287 -def : Pat<(int_nvvm_ff2f16x2_rn Float32Regs:$a, Float32Regs:$b), - (CVT_f16x2_f32 Float32Regs:$a, Float32Regs:$b, CvtRN)>; -def : Pat<(int_nvvm_ff2f16x2_rn_relu Float32Regs:$a, Flo

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 531089. kushanam added a comment. renaming useBF16Math to hasBF16Math Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/include/clang/Basic/BuiltinsNVPT

[PATCH] D144911: adding bf16 support to NVPTX

tra accepted this revision. tra added a comment. This revision is now accepted and ready to land. LGTM with few nits. Thank you for your patience with revising the patch. Comment at: llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp:629-631 + const bool IsBFP16FP16x2NegAvailable = S

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 531068. kushanam added a comment. add constantFP for bf16 Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/include/clang/Basic/BuiltinsNVPTX.def llvm

[PATCH] D144911: adding bf16 support to NVPTX

tra added a comment. Almost there. Just few cosmetic nits remaining. Comment at: llvm/lib/Target/NVPTX/MCTargetDesc/NVPTXInstPrinter.cpp:64-69 + case 9: OS << "%h"; break; case 8: + case 10: OS << "%hh"; tra wrote: > Looks like I've forgot t

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 530123. kushanam added a comment. Adding new bf16 tests and refactoring the code with new conversion Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/i

[PATCH] D144911: adding bf16 support to NVPTX

tra added inline comments. Comment at: llvm/lib/Target/NVPTX/NVPTXISelDAGToDAG.cpp:615 // need to deal with. if (Vector.getSimpleValueType() != MVT::v2f16) return false; This needs to be updated to include v2bf16 Repository: rG LLVM Github Monorepo

[PATCH] D144911: adding bf16 support to NVPTX

tra added a comment. Overall looks good with few minor nits and a couple of questions. Comment at: llvm/include/llvm/IR/IntrinsicsNVVM.td:604 def int_nvvm_f # operation # variant : ClangBuiltin, DefaultAttrsIntrinsic<[llvm_i16_ty], [llvm_i16_ty, llvm_i1

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 529515. kushanam added a comment. updating the bf16 resigter types Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/include/clang/Basic/BuiltinsNVPTX.d

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 529494. kushanam added a comment. rebasing and readding the first commit changes Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/include/clang/Basic/B

[PATCH] D144911: adding bf16 support to NVPTX

tra added inline comments. Comment at: llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp:615 setFP16OperationAction(Op, MVT::v2f16, Legal, Expand); - } - - for (const auto &Op : {ISD::FADD, ISD::FMUL, ISD::FSUB, ISD::FMA}) { setBF16OperationAction(Op, MVT::bf16, Legal, Prom

[PATCH] D144911: adding bf16 support to NVPTX

kushanam added inline comments. Comment at: llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp:615 setFP16OperationAction(Op, MVT::v2f16, Legal, Expand); - } - - for (const auto &Op : {ISD::FADD, ISD::FMUL, ISD::FSUB, ISD::FMA}) { setBF16OperationAction(Op, MVT::bf16, Legal,

[PATCH] D144911: adding bf16 support to NVPTX

tra added inline comments. Comment at: llvm/lib/Target/NVPTX/NVPTXISelLowering.cpp:615 setFP16OperationAction(Op, MVT::v2f16, Legal, Expand); - } - - for (const auto &Op : {ISD::FADD, ISD::FMUL, ISD::FSUB, ISD::FMA}) { setBF16OperationAction(Op, MVT::bf16, Legal, Prom

[PATCH] D144911: adding bf16 support to NVPTX

tra added a comment. FYI https://reviews.llvm.org/D151601 has landed in https://github.com/llvm/llvm-project/commit/dc90f42ea7b4f6d9e643f5ad2ba663eba2f9e421. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D14491

[PATCH] D144911: adding bf16 support to NVPTX

kushanam added a comment. In D144911#4389187 , @manishucsd wrote: > I tried this patch to resolve lowering `nvvm.mma.sync` on bf16 operands for > the PR 13858 > > nvvm.mma.sync A[%293, %295, %297, %299] B[

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 528526. kushanam added a comment. adding min and max for bf16 and refactoring the code Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: llvm/lib/Target/NVPTX

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 528475. kushanam added a comment. Rebasing the D144911 patch Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: llvm/lib/Tar

[PATCH] D144911: adding bf16 support to NVPTX

tra added a comment. In D144911#4389187 , @manishucsd wrote: > I fail to compile this patch. Please find the compilation error below: > > [build] ./llvm-project/llvm/lib/Target/NVPTX/NVPTXInstrInfo.td:1117:40: > error: Variable not defined: 'hasPTX70'

[PATCH] D144911: adding bf16 support to NVPTX

manishucsd added a comment. I tried this patch to resolve lowering `nvvm.mma.sync` on bf16 operands for the PR 13858 nvvm.mma.sync A[%293, %295, %297, %299] B[%392, %394] C[%484, %485, %487, %488] {layoutA = #nvvm.mma_layout, layoutB = #nvvm.mma

[PATCH] D144911: adding bf16 support to NVPTX

tra added a comment. Here's a rough proof-of-concept patch coalescing i16/f16/bf16 to use the same Int16Regs register class: https://reviews.llvm.org/D151601 The changes are largely mechanical, replacing `%h` -> `%rs` in the tests and eliminating special cases we previously had for Float16Regis

[PATCH] D144911: adding bf16 support to NVPTX

tra added inline comments. Comment at: llvm/include/llvm/IR/IntrinsicsNVVM.td:604 def int_nvvm_f # operation # variant : ClangBuiltin, DefaultAttrsIntrinsic<[llvm_i16_ty], [llvm_i16_ty, llvm_i16_ty], tra wrote: > Availability of these new

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 525896. kushanam added a comment. removing commented td entry Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/include/clang/Basic/BuiltinsNVPTX.def

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 525883. kushanam added a comment. rebasing the diff based on the branch HEAD Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/new/ https://reviews.llvm.org/D144911 Files: clang/include/clang/Basic/Built

[PATCH] D144911: adding bf16 support to NVPTX

tra added inline comments. Comment at: llvm/lib/Target/NVPTX/NVPTXAsmPrinter.cpp:315-318 -} else if (RC == &NVPTX::BFloat16RegsRegClass) { - Ret = (9 << 28); -} else if (RC == &NVPTX::BFloat16x2RegsRegClass) { - Ret = (10 << 28); There's still s

[PATCH] D144911: adding bf16 support to NVPTX

jpienaar added inline comments. Comment at: llvm/lib/Target/NVPTX/NVPTXInstrInfo.td:3352 - // Coalesce two bf16 registers into bf16x2 - def BuildBF16x2 : NVPTXInst<(outs BFloat16x2Regs:$dst), - (ins BFloat16Regs:$a, BFloat16Regs:$b), -

[PATCH] D144911: adding bf16 support to NVPTX

jpienaar added a comment. In D144911#4347029 , @kushanam wrote: > In D144911#4340097 , @jpienaar > wrote: > >> Thanks Artem for explaining, > > I think somehow, Phabricator pushes my changes to > https://reviews

[PATCH] D144911: adding bf16 support to NVPTX

kushanam updated this revision to Diff 523111. kushanam added a comment. Herald added a project: clang. Herald added a subscriber: cfe-commits. Adding clang directives and removing bf16 registers Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D144911/