[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
https://github.com/rzinsly created https://github.com/llvm/llvm-project/pull/117612 Enable `-fstack-clash-protection` for RISCV and stack probe for function prologues. We probe the stack by creating a loop that allocates and probe the stack in ProbeSize chunks. We emit an unrolled probe loop for small allocations and emit a variable length probe loop for bigger ones. >From d17432b5a6c6d7b5de662726c15919e5ad242e87 Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Mon, 25 Nov 2024 14:51:35 -0300 Subject: [PATCH 1/2] [RISCV] Add initial stack clash protection Enable `-fstack-clash-protection` for RISCV and stack probe for function prologues. We probe the stack by creating an unrolled loop that allocates and probe the stack in ProbeSize chunks, this is not ideal if the loop has many iterations. --- clang/lib/Driver/ToolChains/Clang.cpp | 3 +- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 93 ++- llvm/lib/Target/RISCV/RISCVFrameLowering.h| 4 +- llvm/lib/Target/RISCV/RISCVISelLowering.cpp | 22 + llvm/lib/Target/RISCV/RISCVISelLowering.h | 6 ++ .../Target/RISCV/RISCVMachineFunctionInfo.cpp | 30 ++ .../Target/RISCV/RISCVMachineFunctionInfo.h | 7 +- llvm/lib/Target/RISCV/RISCVTargetMachine.cpp | 4 +- .../RISCV/stack-clash-prologue-nounwind.ll| 47 ++ .../CodeGen/RISCV/stack-clash-prologue.ll | 78 10 files changed, 267 insertions(+), 27 deletions(-) create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue-nounwind.ll create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue.ll diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp index d3eec9fea0d498..48fc43a8f968bc 100644 --- a/clang/lib/Driver/ToolChains/Clang.cpp +++ b/clang/lib/Driver/ToolChains/Clang.cpp @@ -3774,7 +3774,8 @@ static void RenderSCPOptions(const ToolChain &TC, const ArgList &Args, return; if (!EffectiveTriple.isX86() && !EffectiveTriple.isSystemZ() && - !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64()) + !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64() && + !EffectiveTriple.isRISCV()) return; Args.addOptInFlag(CmdArgs, options::OPT_fstack_clash_protection, diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp index 1ff435b76ad68a..c2412b5c17174d 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp @@ -610,23 +610,74 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, - unsigned CFIIndex) const { + MachineFunction &MF, uint64_t Offset, + uint64_t RealStackSize, bool EmitCFI, + bool NeedProbe, + uint64_t ProbeSize) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); - RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, Offset, MachineInstr::FrameSetup, -getStackAlign()); + // Simply allocate the stack if it's not big enough to require a probe. + if (!NeedProbe || Offset <= ProbeSize) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-Offset), + MachineInstr::FrameSetup, getStackAlign()); - if (EmitCFI) { -// Emit ".cfi_def_cfa_offset StackSize" -BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) -.addCFIIndex(CFIIndex) -.setMIFlag(MachineInstr::FrameSetup); +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset RealStackSize" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); + BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) + .addCFIIndex(CFIIndex) + .setMIFlag(MachineInstr::FrameSetup); +} + +return; + } + + // Do an unrolled probe loop. + uint64_t CurrentOffset = 0; + bool IsRV64 = STI.hasFeature(RISCV::Feature64Bit); + while (CurrentOffset + ProbeSize <= Offset) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, + StackOffset::getFixed(-ProbeSize), MachineInstr::FrameSetup, + getStackAlign()); +// s[d|w] zero, 0(sp) +BuildMI(MBB, MBBI, DL, TII->get(IsRV64 ? RISCV::SD : RISCV::SW)) +.addReg(RISCV::X0) +.addReg(SPReg) +.addImm(0) +.setMIFlags(MachineInstr::FrameSetup); + +CurrentOffset += ProbeSize; +if (EmitCFI) { + // Emit
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
https://github.com/rzinsly updated https://github.com/llvm/llvm-project/pull/117612 >From ca1c76549c09fe1f46dfcd369648d145069ef1fc Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Mon, 25 Nov 2024 14:51:35 -0300 Subject: [PATCH 1/2] [RISCV] Add initial stack clash protection Enable `-fstack-clash-protection` for RISCV and stack probe for function prologues. We probe the stack by creating an unrolled loop that allocates and probe the stack in ProbeSize chunks, this is not ideal if the loop has many iterations. --- clang/lib/Driver/ToolChains/Clang.cpp | 3 +- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 93 ++- llvm/lib/Target/RISCV/RISCVFrameLowering.h| 4 +- llvm/lib/Target/RISCV/RISCVISelLowering.cpp | 22 + llvm/lib/Target/RISCV/RISCVISelLowering.h | 6 ++ .../Target/RISCV/RISCVMachineFunctionInfo.cpp | 30 ++ .../Target/RISCV/RISCVMachineFunctionInfo.h | 7 +- llvm/lib/Target/RISCV/RISCVTargetMachine.cpp | 4 +- .../RISCV/stack-clash-prologue-nounwind.ll| 47 ++ .../CodeGen/RISCV/stack-clash-prologue.ll | 78 10 files changed, 267 insertions(+), 27 deletions(-) create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue-nounwind.ll create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue.ll diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp index d3eec9fea0d498..48fc43a8f968bc 100644 --- a/clang/lib/Driver/ToolChains/Clang.cpp +++ b/clang/lib/Driver/ToolChains/Clang.cpp @@ -3774,7 +3774,8 @@ static void RenderSCPOptions(const ToolChain &TC, const ArgList &Args, return; if (!EffectiveTriple.isX86() && !EffectiveTriple.isSystemZ() && - !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64()) + !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64() && + !EffectiveTriple.isRISCV()) return; Args.addOptInFlag(CmdArgs, options::OPT_fstack_clash_protection, diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp index 1ff435b76ad68a..861f31b609bc81 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp @@ -610,23 +610,74 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, - unsigned CFIIndex) const { + MachineFunction &MF, uint64_t Offset, + uint64_t RealStackSize, bool EmitCFI, + bool NeedProbe, + uint64_t ProbeSize) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); - RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, Offset, MachineInstr::FrameSetup, -getStackAlign()); + // Simply allocate the stack if it's not big enough to require a probe. + if (!NeedProbe || Offset <= ProbeSize) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-Offset), + MachineInstr::FrameSetup, getStackAlign()); - if (EmitCFI) { -// Emit ".cfi_def_cfa_offset StackSize" -BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) -.addCFIIndex(CFIIndex) -.setMIFlag(MachineInstr::FrameSetup); +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset RealStackSize" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); + BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) + .addCFIIndex(CFIIndex) + .setMIFlag(MachineInstr::FrameSetup); +} + +return; + } + + // Do an unrolled probe loop. + uint64_t CurrentOffset = 0; + bool IsRV64 = STI.hasFeature(RISCV::Feature64Bit); + while (CurrentOffset + ProbeSize <= Offset) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, + StackOffset::getFixed(-ProbeSize), MachineInstr::FrameSetup, + getStackAlign()); +// s[d|w] zero, 0(sp) +BuildMI(MBB, MBBI, DL, TII->get(IsRV64 ? RISCV::SD : RISCV::SW)) +.addReg(RISCV::X0) +.addReg(SPReg) +.addImm(0) +.setMIFlags(MachineInstr::FrameSetup); + +CurrentOffset += ProbeSize; +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset CurrentOffset" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, CurrentOffset)); + BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) + .addCFIIndex(CFIIndex) + .setMIFlag(MachineInst
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
https://github.com/rzinsly updated https://github.com/llvm/llvm-project/pull/117612 >From f6bb44ca2242623399eb0ea946f38399fed3807c Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Tue, 26 Nov 2024 16:40:37 -0300 Subject: [PATCH 1/3] [NFC][RISCV] Remove CFIIndex argument from RISCVFrameLowering::allocateStack Calculates CFIIndex inside RISCVFrameLowering::allocateStack instead of sending it by argument. --- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 23 +--- llvm/lib/Target/RISCV/RISCVFrameLowering.h | 3 ++- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp index 1ff435b76ad68a..e2c9baa1b7b1f6 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp @@ -612,8 +612,9 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, - unsigned CFIIndex) const { + MachineFunction &MF, StackOffset Offset, + uint64_t RealStackSize, + bool EmitCFI) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); @@ -622,7 +623,9 @@ void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, getStackAlign()); if (EmitCFI) { -// Emit ".cfi_def_cfa_offset StackSize" +// Emit ".cfi_def_cfa_offset RealStackSize" +unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) .addCFIIndex(CFIIndex) .setMIFlag(MachineInstr::FrameSetup); @@ -745,10 +748,8 @@ void RISCVFrameLowering::emitPrologue(MachineFunction &MF, if (StackSize != 0) { // Allocate space on the stack if necessary. -unsigned CFIIndex = MF.addFrameInst( -MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); -allocateStack(MBB, MBBI, StackOffset::getFixed(-StackSize), - /*EmitCFI=*/ true, CFIIndex); +allocateStack(MBB, MBBI, MF, StackOffset::getFixed(-StackSize), + RealStackSize, /*EmitCFI=*/ true); } // The frame pointer is callee-saved, and code has been generated for us to @@ -790,12 +791,8 @@ void RISCVFrameLowering::emitPrologue(MachineFunction &MF, assert(SecondSPAdjustAmount > 0 && "SecondSPAdjustAmount should be greater than zero"); -// If we are using a frame-pointer, and thus emitted ".cfi_def_cfa fp, 0", -// don't emit an sp-based .cfi_def_cfa_offset -unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::cfiDefCfaOffset( -nullptr, getStackSizeWithRVVPadding(MF))); -allocateStack(MBB, MBBI, StackOffset::getFixed(-SecondSPAdjustAmount), - !hasFP(MF), CFIIndex); +allocateStack(MBB, MBBI, MF, StackOffset::getFixed(-SecondSPAdjustAmount), + getStackSizeWithRVVPadding(MF), !hasFP(MF)); } if (RVVStackSize) { diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.h b/llvm/lib/Target/RISCV/RISCVFrameLowering.h index 9aff4dc9e4089a..8c2e3f0f3e5638 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.h +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.h @@ -79,7 +79,8 @@ class RISCVFrameLowering : public TargetFrameLowering { } void allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, unsigned CFIIndex) const; + MachineFunction &MF, StackOffset Offset, + uint64_t RealStackSize, bool EmitCFI) const; protected: const RISCVSubtarget &STI; >From 6a09d85096bdf08d0dfce5c1cb8b2cb22ec25eba Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Mon, 25 Nov 2024 14:51:35 -0300 Subject: [PATCH 2/3] [RISCV] Add initial stack clash protection Enable `-fstack-clash-protection` for RISCV and stack probe for function prologues. We probe the stack by creating an unrolled loop that allocates and probe the stack in ProbeSize chunks, this is not ideal if the loop has many iterations. --- clang/lib/Driver/ToolChains/Clang.cpp | 3 +- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 90 +++ llvm/lib/Target/RISCV/RISCVFrameLowering.h| 5 +- llvm/lib/Target/RISCV/RISCVISelLowering.cpp | 22 + llvm/lib/Target/RISCV/RISCVISelLowering.h | 5 ++ .../Target/RISCV/RISCVMachineFunctionInfo.cpp | 30 +++ .../Target/RISCV/RISCVMachineFunctionInfo.h | 7 +- llvm/lib/Target/RISCV/RISCVTargetMachine.cpp | 4 +- .../RISCV/stack-clash-prologue-
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -610,22 +610,134 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, - unsigned CFIIndex) const { + MachineFunction &MF, uint64_t Offset, + uint64_t RealStackSize, bool EmitCFI, + bool NeedProbe, + uint64_t ProbeSize) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); - RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, Offset, MachineInstr::FrameSetup, + // Simply allocate the stack if it's not big enough to require a probe. + if (!NeedProbe || Offset <= ProbeSize) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-Offset), + MachineInstr::FrameSetup, getStackAlign()); + +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset RealStackSize" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); + BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) + .addCFIIndex(CFIIndex) + .setMIFlag(MachineInstr::FrameSetup); +} + +return; + } + + // Unroll the probe loop depending on the number of iterations. + if (Offset < ProbeSize * 5) { +uint64_t CurrentOffset = 0; +bool IsRV64 = STI.is64Bit(); +while (CurrentOffset + ProbeSize <= Offset) { + RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, +StackOffset::getFixed(-ProbeSize), MachineInstr::FrameSetup, +getStackAlign()); + // s[d|w] zero, 0(sp) + BuildMI(MBB, MBBI, DL, TII->get(IsRV64 ? RISCV::SD : RISCV::SW)) + .addReg(RISCV::X0) + .addReg(SPReg) + .addImm(0) + .setMIFlags(MachineInstr::FrameSetup); + + CurrentOffset += ProbeSize; + if (EmitCFI) { +// Emit ".cfi_def_cfa_offset CurrentOffset" +unsigned CFIIndex = MF.addFrameInst( +MCCFIInstruction::cfiDefCfaOffset(nullptr, CurrentOffset)); +BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) +.addCFIIndex(CFIIndex) +.setMIFlag(MachineInstr::FrameSetup); + } +} + +uint64_t Residual = Offset - CurrentOffset; +if (Residual) { + RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, +StackOffset::getFixed(-Residual), MachineInstr::FrameSetup, +getStackAlign()); + if (EmitCFI) { +// Emit ".cfi_def_cfa_offset Offset" +unsigned CFIIndex = +MF.addFrameInst(MCCFIInstruction::cfiDefCfaOffset(nullptr, Offset)); +BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) +.addCFIIndex(CFIIndex) +.setMIFlag(MachineInstr::FrameSetup); + } +} + +return; + } + + // Emit a variable-length allocation probing loop. + uint64_t RoundedSize = (Offset / ProbeSize) * ProbeSize; + uint64_t Residual = Offset - RoundedSize; + + Register TargetReg = RISCV::X6; + // SUB TargetReg, SP, RoundedSize + RI->adjustReg(MBB, MBBI, DL, TargetReg, SPReg, +StackOffset::getFixed(-RoundedSize), MachineInstr::FrameSetup, getStackAlign()); if (EmitCFI) { -// Emit ".cfi_def_cfa_offset StackSize" +// Set the CFA register to TargetReg. +unsigned Reg = STI.getRegisterInfo()->getDwarfRegNum(TargetReg, true); +unsigned CFIIndex = +MF.addFrameInst(MCCFIInstruction::cfiDefCfa(nullptr, Reg, RoundedSize)); BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) .addCFIIndex(CFIIndex) -.setMIFlag(MachineInstr::FrameSetup); +.setMIFlags(MachineInstr::FrameSetup); + } + + // It will be expanded to a probe loop in `inlineStackProbe`. + BuildMI(MBB, MBBI, DL, TII->get(RISCV::PROBED_STACKALLOC)) + .addReg(SPReg) + .addReg(TargetReg); + + if (EmitCFI) { +// Set the CFA register back to SP. +unsigned Reg = STI.getRegisterInfo()->getDwarfRegNum(SPReg, true); +unsigned CFIIndex = +MF.addFrameInst(MCCFIInstruction::createDefCfaRegister(nullptr, Reg)); +BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) +.addCFIIndex(CFIIndex) +.setMIFlags(MachineInstr::FrameSetup); + } + + if (Residual) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-Residual), + MachineInstr::FrameSetup, getStackAlign()); +if (Residual > ProbeSize) { + // s[d|w] zero, 0(sp) + bool IsRV64 = STI.hasFeature(
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -610,22 +610,134 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, - unsigned CFIIndex) const { rzinsly wrote: Ok, I created a opened a new one: https://github.com/llvm/llvm-project/pull/117871 https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
https://github.com/rzinsly updated https://github.com/llvm/llvm-project/pull/117612 >From f6bb44ca2242623399eb0ea946f38399fed3807c Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Tue, 26 Nov 2024 16:40:37 -0300 Subject: [PATCH 1/3] [NFC][RISCV] Remove CFIIndex argument from RISCVFrameLowering::allocateStack Calculates CFIIndex inside RISCVFrameLowering::allocateStack instead of sending it by argument. --- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 23 +--- llvm/lib/Target/RISCV/RISCVFrameLowering.h | 3 ++- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp index 1ff435b76ad68a..e2c9baa1b7b1f6 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp @@ -612,8 +612,9 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, - unsigned CFIIndex) const { + MachineFunction &MF, StackOffset Offset, + uint64_t RealStackSize, + bool EmitCFI) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); @@ -622,7 +623,9 @@ void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, getStackAlign()); if (EmitCFI) { -// Emit ".cfi_def_cfa_offset StackSize" +// Emit ".cfi_def_cfa_offset RealStackSize" +unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) .addCFIIndex(CFIIndex) .setMIFlag(MachineInstr::FrameSetup); @@ -745,10 +748,8 @@ void RISCVFrameLowering::emitPrologue(MachineFunction &MF, if (StackSize != 0) { // Allocate space on the stack if necessary. -unsigned CFIIndex = MF.addFrameInst( -MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); -allocateStack(MBB, MBBI, StackOffset::getFixed(-StackSize), - /*EmitCFI=*/ true, CFIIndex); +allocateStack(MBB, MBBI, MF, StackOffset::getFixed(-StackSize), + RealStackSize, /*EmitCFI=*/ true); } // The frame pointer is callee-saved, and code has been generated for us to @@ -790,12 +791,8 @@ void RISCVFrameLowering::emitPrologue(MachineFunction &MF, assert(SecondSPAdjustAmount > 0 && "SecondSPAdjustAmount should be greater than zero"); -// If we are using a frame-pointer, and thus emitted ".cfi_def_cfa fp, 0", -// don't emit an sp-based .cfi_def_cfa_offset -unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::cfiDefCfaOffset( -nullptr, getStackSizeWithRVVPadding(MF))); -allocateStack(MBB, MBBI, StackOffset::getFixed(-SecondSPAdjustAmount), - !hasFP(MF), CFIIndex); +allocateStack(MBB, MBBI, MF, StackOffset::getFixed(-SecondSPAdjustAmount), + getStackSizeWithRVVPadding(MF), !hasFP(MF)); } if (RVVStackSize) { diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.h b/llvm/lib/Target/RISCV/RISCVFrameLowering.h index 9aff4dc9e4089a..8c2e3f0f3e5638 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.h +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.h @@ -79,7 +79,8 @@ class RISCVFrameLowering : public TargetFrameLowering { } void allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, unsigned CFIIndex) const; + MachineFunction &MF, StackOffset Offset, + uint64_t RealStackSize, bool EmitCFI) const; protected: const RISCVSubtarget &STI; >From 132e8598b2d044463d4d9fb6ef391e6095b75f87 Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Mon, 25 Nov 2024 14:51:35 -0300 Subject: [PATCH 2/3] [RISCV] Add initial stack clash protection Enable `-fstack-clash-protection` for RISCV and stack probe for function prologues. We probe the stack by creating an unrolled loop that allocates and probe the stack in ProbeSize chunks, this is not ideal if the loop has many iterations. --- clang/lib/Driver/ToolChains/Clang.cpp | 3 +- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 90 +++ llvm/lib/Target/RISCV/RISCVFrameLowering.h| 5 +- llvm/lib/Target/RISCV/RISCVISelLowering.cpp | 22 + llvm/lib/Target/RISCV/RISCVISelLowering.h | 6 ++ .../Target/RISCV/RISCVMachineFunctionInfo.cpp | 30 +++ .../Target/RISCV/RISCVMachineFunctionInfo.h | 7 +- llvm/lib/Target/RISCV/RISCVTargetMachine.cpp | 4 +- .../RISCV/stack-clash-prologue-
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -0,0 +1,208 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py +; RUN: llc -mtriple=riscv64 -mattr=+m,+v -O2 < %s \ +; RUN: | FileCheck %s -check-prefix=RV64I rzinsly wrote: It was an oversight on my part, fixed now. https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -610,22 +610,134 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, - unsigned CFIIndex) const { rzinsly wrote: I separated in another commit, should it be in another PR? https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
rzinsly wrote: @topperc thanks for the review, I followed your suggestions. https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
https://github.com/rzinsly updated https://github.com/llvm/llvm-project/pull/117612 >From f6bb44ca2242623399eb0ea946f38399fed3807c Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Tue, 26 Nov 2024 16:40:37 -0300 Subject: [PATCH 1/3] [NFC][RISCV] Remove CFIIndex argument from RISCVFrameLowering::allocateStack Calculates CFIIndex inside RISCVFrameLowering::allocateStack instead of sending it by argument. --- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 23 +--- llvm/lib/Target/RISCV/RISCVFrameLowering.h | 3 ++- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp index 1ff435b76ad68a..e2c9baa1b7b1f6 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp @@ -612,8 +612,9 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, - unsigned CFIIndex) const { + MachineFunction &MF, StackOffset Offset, + uint64_t RealStackSize, + bool EmitCFI) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); @@ -622,7 +623,9 @@ void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, getStackAlign()); if (EmitCFI) { -// Emit ".cfi_def_cfa_offset StackSize" +// Emit ".cfi_def_cfa_offset RealStackSize" +unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) .addCFIIndex(CFIIndex) .setMIFlag(MachineInstr::FrameSetup); @@ -745,10 +748,8 @@ void RISCVFrameLowering::emitPrologue(MachineFunction &MF, if (StackSize != 0) { // Allocate space on the stack if necessary. -unsigned CFIIndex = MF.addFrameInst( -MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); -allocateStack(MBB, MBBI, StackOffset::getFixed(-StackSize), - /*EmitCFI=*/ true, CFIIndex); +allocateStack(MBB, MBBI, MF, StackOffset::getFixed(-StackSize), + RealStackSize, /*EmitCFI=*/ true); } // The frame pointer is callee-saved, and code has been generated for us to @@ -790,12 +791,8 @@ void RISCVFrameLowering::emitPrologue(MachineFunction &MF, assert(SecondSPAdjustAmount > 0 && "SecondSPAdjustAmount should be greater than zero"); -// If we are using a frame-pointer, and thus emitted ".cfi_def_cfa fp, 0", -// don't emit an sp-based .cfi_def_cfa_offset -unsigned CFIIndex = MF.addFrameInst(MCCFIInstruction::cfiDefCfaOffset( -nullptr, getStackSizeWithRVVPadding(MF))); -allocateStack(MBB, MBBI, StackOffset::getFixed(-SecondSPAdjustAmount), - !hasFP(MF), CFIIndex); +allocateStack(MBB, MBBI, MF, StackOffset::getFixed(-SecondSPAdjustAmount), + getStackSizeWithRVVPadding(MF), !hasFP(MF)); } if (RVVStackSize) { diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.h b/llvm/lib/Target/RISCV/RISCVFrameLowering.h index 9aff4dc9e4089a..8c2e3f0f3e5638 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.h +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.h @@ -79,7 +79,8 @@ class RISCVFrameLowering : public TargetFrameLowering { } void allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - StackOffset Offset, bool EmitCFI, unsigned CFIIndex) const; + MachineFunction &MF, StackOffset Offset, + uint64_t RealStackSize, bool EmitCFI) const; protected: const RISCVSubtarget &STI; >From 6a09d85096bdf08d0dfce5c1cb8b2cb22ec25eba Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Mon, 25 Nov 2024 14:51:35 -0300 Subject: [PATCH 2/3] [RISCV] Add initial stack clash protection Enable `-fstack-clash-protection` for RISCV and stack probe for function prologues. We probe the stack by creating an unrolled loop that allocates and probe the stack in ProbeSize chunks, this is not ideal if the loop has many iterations. --- clang/lib/Driver/ToolChains/Clang.cpp | 3 +- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 90 +++ llvm/lib/Target/RISCV/RISCVFrameLowering.h| 5 +- llvm/lib/Target/RISCV/RISCVISelLowering.cpp | 22 + llvm/lib/Target/RISCV/RISCVISelLowering.h | 5 ++ .../Target/RISCV/RISCVMachineFunctionInfo.cpp | 30 +++ .../Target/RISCV/RISCVMachineFunctionInfo.h | 7 +- llvm/lib/Target/RISCV/RISCVTargetMachine.cpp | 4 +- .../RISCV/stack-clash-prologue-
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
https://github.com/rzinsly edited https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -580,25 +580,134 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - MachineFunction &MF, StackOffset Offset, - uint64_t RealStackSize, - bool EmitCFI) const { + MachineFunction &MF, uint64_t Offset, + uint64_t RealStackSize, bool EmitCFI, + bool NeedProbe, + uint64_t ProbeSize) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); - RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, Offset, MachineInstr::FrameSetup, + // Simply allocate the stack if it's not big enough to require a probe. + if (!NeedProbe || Offset <= ProbeSize) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-Offset), + MachineInstr::FrameSetup, getStackAlign()); + +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset RealStackSize" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); + BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) + .addCFIIndex(CFIIndex) + .setMIFlag(MachineInstr::FrameSetup); +} + +return; + } + + // Unroll the probe loop depending on the number of iterations. + if (Offset < ProbeSize * 5) { +uint64_t CurrentOffset = 0; +bool IsRV64 = STI.is64Bit(); +while (CurrentOffset + ProbeSize <= Offset) { + RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, +StackOffset::getFixed(-ProbeSize), MachineInstr::FrameSetup, +getStackAlign()); + // s[d|w] zero, 0(sp) + BuildMI(MBB, MBBI, DL, TII->get(IsRV64 ? RISCV::SD : RISCV::SW)) + .addReg(RISCV::X0) + .addReg(SPReg) + .addImm(0) + .setMIFlags(MachineInstr::FrameSetup); + + CurrentOffset += ProbeSize; + if (EmitCFI) { +// Emit ".cfi_def_cfa_offset CurrentOffset" +unsigned CFIIndex = MF.addFrameInst( +MCCFIInstruction::cfiDefCfaOffset(nullptr, CurrentOffset)); +BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) +.addCFIIndex(CFIIndex) +.setMIFlag(MachineInstr::FrameSetup); + } +} + +uint64_t Residual = Offset - CurrentOffset; +if (Residual) { + RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, +StackOffset::getFixed(-Residual), MachineInstr::FrameSetup, +getStackAlign()); + if (EmitCFI) { +// Emit ".cfi_def_cfa_offset Offset" +unsigned CFIIndex = +MF.addFrameInst(MCCFIInstruction::cfiDefCfaOffset(nullptr, Offset)); +BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) +.addCFIIndex(CFIIndex) +.setMIFlag(MachineInstr::FrameSetup); + } +} + +return; + } + + // Emit a variable-length allocation probing loop. + uint64_t RoundedSize = (Offset / ProbeSize) * ProbeSize; + uint64_t Residual = Offset - RoundedSize; + + Register TargetReg = RISCV::X6; + // SUB TargetReg, SP, RoundedSize + RI->adjustReg(MBB, MBBI, DL, TargetReg, SPReg, +StackOffset::getFixed(-RoundedSize), MachineInstr::FrameSetup, getStackAlign()); if (EmitCFI) { -// Emit ".cfi_def_cfa_offset RealStackSize" -unsigned CFIIndex = MF.addFrameInst( -MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); +// Set the CFA register to TargetReg. +unsigned Reg = STI.getRegisterInfo()->getDwarfRegNum(TargetReg, true); +unsigned CFIIndex = +MF.addFrameInst(MCCFIInstruction::cfiDefCfa(nullptr, Reg, RoundedSize)); BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) .addCFIIndex(CFIIndex) -.setMIFlag(MachineInstr::FrameSetup); +.setMIFlags(MachineInstr::FrameSetup); + } + + // It will be expanded to a probe loop in `inlineStackProbe`. + BuildMI(MBB, MBBI, DL, TII->get(RISCV::PROBED_STACKALLOC)) + .addReg(SPReg) + .addReg(TargetReg); + + if (EmitCFI) { +// Set the CFA register back to SP. +unsigned Reg = STI.getRegisterInfo()->getDwarfRegNum(SPReg, true); +unsigned CFIIndex = +MF.addFrameInst(MCCFIInstruction::createDefCfaRegister(nullptr, Reg)); +BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) +.addCFIIndex(CFIIndex) +.setMIFlags(MachineInstr::FrameSetup); + } + + if (Residual) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::g
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -1910,3 +2023,72 @@ bool RISCVFrameLowering::isSupportedStackID(TargetStackID::Value ID) const { TargetStackID::Value RISCVFrameLowering::getStackIDForScalableVectors() const { return TargetStackID::ScalableVector; } + +// Synthesize the probe loop. +static void emitStackProbeInline(MachineFunction &MF, MachineBasicBlock &MBB, + MachineBasicBlock::iterator MBBI, + DebugLoc DL) { + + auto &Subtarget = MF.getSubtarget(); + const RISCVInstrInfo *TII = Subtarget.getInstrInfo(); + bool IsRV64 = Subtarget.is64Bit(); + Align StackAlign = Subtarget.getFrameLowering()->getStackAlign(); + const RISCVTargetLowering *TLI = Subtarget.getTargetLowering(); + uint64_t ProbeSize = TLI->getStackProbeSize(MF, StackAlign); + + MachineFunction::iterator MBBInsertPoint = std::next(MBB.getIterator()); + MachineBasicBlock *LoopTestMBB = + MF.CreateMachineBasicBlock(MBB.getBasicBlock()); + MF.insert(MBBInsertPoint, LoopTestMBB); + MachineBasicBlock *ExitMBB = MF.CreateMachineBasicBlock(MBB.getBasicBlock()); + MF.insert(MBBInsertPoint, ExitMBB); + MachineInstr::MIFlag Flags = MachineInstr::FrameSetup; + Register TargetReg = RISCV::X6; + Register ScratchReg = RISCV::X7; + + // ScratchReg = ProbeSize + TII->movImm(MBB, MBBI, DL, ScratchReg, ProbeSize, Flags); + + // LoopTest: + // SUB SP, SP, ProbeSize + BuildMI(*LoopTestMBB, LoopTestMBB->end(), DL, TII->get(RISCV::SUB), SPReg) + .addReg(SPReg) + .addReg(ScratchReg) + .setMIFlags(Flags); + + // s[d|w] zero, 0(sp) + BuildMI(*LoopTestMBB, LoopTestMBB->end(), DL, + TII->get(IsRV64 ? RISCV::SD : RISCV::SW)) + .addReg(RISCV::X0) + .addReg(SPReg) + .addImm(0) + .setMIFlags(Flags); + + // BNE SP, TargetReg, LoopTest + BuildMI(*LoopTestMBB, LoopTestMBB->end(), DL, TII->get(RISCV::BNE)) + .addReg(SPReg) + .addReg(TargetReg) + .addMBB(LoopTestMBB) + .setMIFlags(Flags); + + ExitMBB->splice(ExitMBB->end(), &MBB, std::next(MBBI), MBB.end()); + + LoopTestMBB->addSuccessor(ExitMBB); + LoopTestMBB->addSuccessor(LoopTestMBB); + MBB.addSuccessor(LoopTestMBB); +} + +void RISCVFrameLowering::inlineStackProbe(MachineFunction &MF, + MachineBasicBlock &MBB) const { + // Get the instructions that need to be replaced. We emit at most two of rzinsly wrote: This foreshadows stack probing for vector code, I removed the comment from this patch and will add it in the next. https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -157,6 +159,9 @@ class RISCVMachineFunctionInfo : public MachineFunctionInfo { bool isVectorCall() const { return IsVectorCall; } void setIsVectorCall() { IsVectorCall = true; } + + bool hasStackProbing() const { return StackProbeSize != 0; } rzinsly wrote: Yes, I removed them now. https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -580,25 +580,134 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - MachineFunction &MF, StackOffset Offset, - uint64_t RealStackSize, - bool EmitCFI) const { + MachineFunction &MF, uint64_t Offset, + uint64_t RealStackSize, bool EmitCFI, + bool NeedProbe, + uint64_t ProbeSize) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); - RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, Offset, MachineInstr::FrameSetup, + // Simply allocate the stack if it's not big enough to require a probe. + if (!NeedProbe || Offset <= ProbeSize) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-Offset), + MachineInstr::FrameSetup, getStackAlign()); + +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset RealStackSize" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); + BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) + .addCFIIndex(CFIIndex) + .setMIFlag(MachineInstr::FrameSetup); +} + +return; + } + + // Unroll the probe loop depending on the number of iterations. + if (Offset < ProbeSize * 5) { +uint64_t CurrentOffset = 0; +bool IsRV64 = STI.is64Bit(); +while (CurrentOffset + ProbeSize <= Offset) { + RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, +StackOffset::getFixed(-ProbeSize), MachineInstr::FrameSetup, +getStackAlign()); + // s[d|w] zero, 0(sp) + BuildMI(MBB, MBBI, DL, TII->get(IsRV64 ? RISCV::SD : RISCV::SW)) + .addReg(RISCV::X0) + .addReg(SPReg) + .addImm(0) + .setMIFlags(MachineInstr::FrameSetup); + + CurrentOffset += ProbeSize; + if (EmitCFI) { +// Emit ".cfi_def_cfa_offset CurrentOffset" +unsigned CFIIndex = MF.addFrameInst( +MCCFIInstruction::cfiDefCfaOffset(nullptr, CurrentOffset)); +BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) +.addCFIIndex(CFIIndex) +.setMIFlag(MachineInstr::FrameSetup); + } +} + +uint64_t Residual = Offset - CurrentOffset; +if (Residual) { + RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, +StackOffset::getFixed(-Residual), MachineInstr::FrameSetup, +getStackAlign()); + if (EmitCFI) { +// Emit ".cfi_def_cfa_offset Offset" +unsigned CFIIndex = +MF.addFrameInst(MCCFIInstruction::cfiDefCfaOffset(nullptr, Offset)); +BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) +.addCFIIndex(CFIIndex) +.setMIFlag(MachineInstr::FrameSetup); + } +} + +return; + } + + // Emit a variable-length allocation probing loop. + uint64_t RoundedSize = (Offset / ProbeSize) * ProbeSize; rzinsly wrote: Yes, just replaced it. https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
https://github.com/rzinsly updated https://github.com/llvm/llvm-project/pull/117612 >From 9872154335da3b6f80c0ceb2d5756a99fb78df08 Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Mon, 25 Nov 2024 14:51:35 -0300 Subject: [PATCH 1/2] [RISCV] Add initial stack clash protection Enable `-fstack-clash-protection` for RISCV and stack probe for function prologues. We probe the stack by creating an unrolled loop that allocates and probe the stack in ProbeSize chunks, this is not ideal if the loop has many iterations. --- clang/lib/Driver/ToolChains/Clang.cpp | 3 +- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 90 +++ llvm/lib/Target/RISCV/RISCVFrameLowering.h| 5 +- llvm/lib/Target/RISCV/RISCVISelLowering.cpp | 22 + llvm/lib/Target/RISCV/RISCVISelLowering.h | 5 ++ .../Target/RISCV/RISCVMachineFunctionInfo.cpp | 30 +++ .../Target/RISCV/RISCVMachineFunctionInfo.h | 4 +- llvm/lib/Target/RISCV/RISCVTargetMachine.cpp | 4 +- .../RISCV/stack-clash-prologue-nounwind.ll| 72 +++ .../CodeGen/RISCV/stack-clash-prologue.ll | 78 10 files changed, 288 insertions(+), 25 deletions(-) create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue-nounwind.ll create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue.ll diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp index 99a092d83d531a..09d1350b115d7f 100644 --- a/clang/lib/Driver/ToolChains/Clang.cpp +++ b/clang/lib/Driver/ToolChains/Clang.cpp @@ -3775,7 +3775,8 @@ static void RenderSCPOptions(const ToolChain &TC, const ArgList &Args, return; if (!EffectiveTriple.isX86() && !EffectiveTriple.isSystemZ() && - !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64()) + !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64() && + !EffectiveTriple.isRISCV()) return; Args.addOptInFlag(CmdArgs, options::OPT_fstack_clash_protection, diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp index deb0b627225c64..816f4f06879524 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp @@ -580,26 +580,74 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - MachineFunction &MF, StackOffset Offset, - uint64_t RealStackSize, - bool EmitCFI) const { + MachineFunction &MF, uint64_t Offset, + uint64_t RealStackSize, bool EmitCFI, + bool NeedProbe, + uint64_t ProbeSize) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); - RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, Offset, MachineInstr::FrameSetup, -getStackAlign()); + // Simply allocate the stack if it's not big enough to require a probe. + if (!NeedProbe || Offset <= ProbeSize) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-Offset), + MachineInstr::FrameSetup, getStackAlign()); - if (EmitCFI) { -// Emit ".cfi_def_cfa_offset RealStackSize" -unsigned CFIIndex = MF.addFrameInst( -MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); -BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) -.addCFIIndex(CFIIndex) -.setMIFlag(MachineInstr::FrameSetup); +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset RealStackSize" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); + BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) + .addCFIIndex(CFIIndex) + .setMIFlag(MachineInstr::FrameSetup); +} + +return; } + + // Do an unrolled probe loop. + uint64_t CurrentOffset = 0; + bool IsRV64 = STI.is64Bit(); + while (CurrentOffset + ProbeSize <= Offset) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, + StackOffset::getFixed(-ProbeSize), MachineInstr::FrameSetup, + getStackAlign()); +// s[d|w] zero, 0(sp) +BuildMI(MBB, MBBI, DL, TII->get(IsRV64 ? RISCV::SD : RISCV::SW)) +.addReg(RISCV::X0) +.addReg(SPReg) +.addImm(0) +.setMIFlags(MachineInstr::FrameSetup); + +CurrentOffset += ProbeSize; +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset CurrentOffset" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
https://github.com/rzinsly updated https://github.com/llvm/llvm-project/pull/117612 >From 708dce34f76fc361a2093459efd3b14705045f6f Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Mon, 25 Nov 2024 14:51:35 -0300 Subject: [PATCH 1/2] [RISCV] Add initial stack clash protection Enable `-fstack-clash-protection` for RISCV and stack probe for function prologues. We probe the stack by creating an unrolled loop that allocates and probe the stack in ProbeSize chunks, this is not ideal if the loop has many iterations. --- clang/lib/Driver/ToolChains/Clang.cpp | 3 +- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 90 +++ llvm/lib/Target/RISCV/RISCVFrameLowering.h| 5 +- llvm/lib/Target/RISCV/RISCVISelLowering.cpp | 22 + llvm/lib/Target/RISCV/RISCVISelLowering.h | 5 ++ .../Target/RISCV/RISCVMachineFunctionInfo.cpp | 30 +++ .../Target/RISCV/RISCVMachineFunctionInfo.h | 7 +- llvm/lib/Target/RISCV/RISCVTargetMachine.cpp | 4 +- .../RISCV/stack-clash-prologue-nounwind.ll| 72 +++ .../CodeGen/RISCV/stack-clash-prologue.ll | 78 10 files changed, 291 insertions(+), 25 deletions(-) create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue-nounwind.ll create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue.ll diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp index 99a092d83d531a..09d1350b115d7f 100644 --- a/clang/lib/Driver/ToolChains/Clang.cpp +++ b/clang/lib/Driver/ToolChains/Clang.cpp @@ -3775,7 +3775,8 @@ static void RenderSCPOptions(const ToolChain &TC, const ArgList &Args, return; if (!EffectiveTriple.isX86() && !EffectiveTriple.isSystemZ() && - !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64()) + !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64() && + !EffectiveTriple.isRISCV()) return; Args.addOptInFlag(CmdArgs, options::OPT_fstack_clash_protection, diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp index deb0b627225c64..816f4f06879524 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp @@ -580,26 +580,74 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - MachineFunction &MF, StackOffset Offset, - uint64_t RealStackSize, - bool EmitCFI) const { + MachineFunction &MF, uint64_t Offset, + uint64_t RealStackSize, bool EmitCFI, + bool NeedProbe, + uint64_t ProbeSize) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); - RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, Offset, MachineInstr::FrameSetup, -getStackAlign()); + // Simply allocate the stack if it's not big enough to require a probe. + if (!NeedProbe || Offset <= ProbeSize) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-Offset), + MachineInstr::FrameSetup, getStackAlign()); - if (EmitCFI) { -// Emit ".cfi_def_cfa_offset RealStackSize" -unsigned CFIIndex = MF.addFrameInst( -MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); -BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) -.addCFIIndex(CFIIndex) -.setMIFlag(MachineInstr::FrameSetup); +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset RealStackSize" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); + BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) + .addCFIIndex(CFIIndex) + .setMIFlag(MachineInstr::FrameSetup); +} + +return; } + + // Do an unrolled probe loop. + uint64_t CurrentOffset = 0; + bool IsRV64 = STI.is64Bit(); + while (CurrentOffset + ProbeSize <= Offset) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, + StackOffset::getFixed(-ProbeSize), MachineInstr::FrameSetup, + getStackAlign()); +// s[d|w] zero, 0(sp) +BuildMI(MBB, MBBI, DL, TII->get(IsRV64 ? RISCV::SD : RISCV::SW)) +.addReg(RISCV::X0) +.addReg(SPReg) +.addImm(0) +.setMIFlags(MachineInstr::FrameSetup); + +CurrentOffset += ProbeSize; +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset CurrentOffset" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
rzinsly wrote: Rebased to fix the merge conflicts. https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
https://github.com/rzinsly updated https://github.com/llvm/llvm-project/pull/117612 >From f9c0f9c25c1fea047a7c0cc7b22889ed7c45afba Mon Sep 17 00:00:00 2001 From: Raphael Moreira Zinsly Date: Mon, 25 Nov 2024 14:51:35 -0300 Subject: [PATCH 1/2] [RISCV] Add initial stack clash protection Enable `-fstack-clash-protection` for RISCV and stack probe for function prologues. We probe the stack by creating an unrolled loop that allocates and probe the stack in ProbeSize chunks, this is not ideal if the loop has many iterations. --- clang/lib/Driver/ToolChains/Clang.cpp | 3 +- llvm/lib/Target/RISCV/RISCVFrameLowering.cpp | 90 +++ llvm/lib/Target/RISCV/RISCVFrameLowering.h| 5 +- llvm/lib/Target/RISCV/RISCVISelLowering.cpp | 22 + llvm/lib/Target/RISCV/RISCVISelLowering.h | 5 ++ .../Target/RISCV/RISCVMachineFunctionInfo.cpp | 30 +++ .../Target/RISCV/RISCVMachineFunctionInfo.h | 4 +- llvm/lib/Target/RISCV/RISCVTargetMachine.cpp | 4 +- .../RISCV/stack-clash-prologue-nounwind.ll| 72 +++ .../CodeGen/RISCV/stack-clash-prologue.ll | 82 + 10 files changed, 292 insertions(+), 25 deletions(-) create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue-nounwind.ll create mode 100644 llvm/test/CodeGen/RISCV/stack-clash-prologue.ll diff --git a/clang/lib/Driver/ToolChains/Clang.cpp b/clang/lib/Driver/ToolChains/Clang.cpp index 99a092d83d531a..09d1350b115d7f 100644 --- a/clang/lib/Driver/ToolChains/Clang.cpp +++ b/clang/lib/Driver/ToolChains/Clang.cpp @@ -3775,7 +3775,8 @@ static void RenderSCPOptions(const ToolChain &TC, const ArgList &Args, return; if (!EffectiveTriple.isX86() && !EffectiveTriple.isSystemZ() && - !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64()) + !EffectiveTriple.isPPC64() && !EffectiveTriple.isAArch64() && + !EffectiveTriple.isRISCV()) return; Args.addOptInFlag(CmdArgs, options::OPT_fstack_clash_protection, diff --git a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp index deb0b627225c64..816f4f06879524 100644 --- a/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp +++ b/llvm/lib/Target/RISCV/RISCVFrameLowering.cpp @@ -580,26 +580,74 @@ static MCCFIInstruction createDefCFAOffset(const TargetRegisterInfo &TRI, Comment.str()); } +// Allocate stack space and probe it if necessary. void RISCVFrameLowering::allocateStack(MachineBasicBlock &MBB, MachineBasicBlock::iterator MBBI, - MachineFunction &MF, StackOffset Offset, - uint64_t RealStackSize, - bool EmitCFI) const { + MachineFunction &MF, uint64_t Offset, + uint64_t RealStackSize, bool EmitCFI, + bool NeedProbe, + uint64_t ProbeSize) const { DebugLoc DL; const RISCVRegisterInfo *RI = STI.getRegisterInfo(); const RISCVInstrInfo *TII = STI.getInstrInfo(); - RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, Offset, MachineInstr::FrameSetup, -getStackAlign()); + // Simply allocate the stack if it's not big enough to require a probe. + if (!NeedProbe || Offset <= ProbeSize) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, StackOffset::getFixed(-Offset), + MachineInstr::FrameSetup, getStackAlign()); - if (EmitCFI) { -// Emit ".cfi_def_cfa_offset RealStackSize" -unsigned CFIIndex = MF.addFrameInst( -MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); -BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) -.addCFIIndex(CFIIndex) -.setMIFlag(MachineInstr::FrameSetup); +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset RealStackSize" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffset(nullptr, RealStackSize)); + BuildMI(MBB, MBBI, DL, TII->get(TargetOpcode::CFI_INSTRUCTION)) + .addCFIIndex(CFIIndex) + .setMIFlag(MachineInstr::FrameSetup); +} + +return; } + + // Do an unrolled probe loop. + uint64_t CurrentOffset = 0; + bool IsRV64 = STI.is64Bit(); + while (CurrentOffset + ProbeSize <= Offset) { +RI->adjustReg(MBB, MBBI, DL, SPReg, SPReg, + StackOffset::getFixed(-ProbeSize), MachineInstr::FrameSetup, + getStackAlign()); +// s[d|w] zero, 0(sp) +BuildMI(MBB, MBBI, DL, TII->get(IsRV64 ? RISCV::SD : RISCV::SW)) +.addReg(RISCV::X0) +.addReg(SPReg) +.addImm(0) +.setMIFlags(MachineInstr::FrameSetup); + +CurrentOffset += ProbeSize; +if (EmitCFI) { + // Emit ".cfi_def_cfa_offset CurrentOffset" + unsigned CFIIndex = MF.addFrameInst( + MCCFIInstruction::cfiDefCfaOffse
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -0,0 +1,208 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py +; RUN: llc -mtriple=riscv64 -mattr=+m,+v -O2 < %s \ +; RUN: | FileCheck %s -check-prefix=RV64I rzinsly wrote: @lenary hi, is there anything else I should change? https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
rzinsly wrote: > > > @rzinsly do you need someone to commit this? > > > > > > Yes, please. > > Please turn off [Keep my email addresses > private](https://github.com/settings/emails) setting in your account. > Otherwise the commit will be from the github hidden email which we don't use > in LLVM. See [LLVM > Discourse](https://discourse.llvm.org/t/hidden-emails-on-github-should-we-do-something-about-it) > for more information. Ok, done! https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
rzinsly wrote: > @rzinsly do you need someone to commit this? Yes, please. https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
rzinsly wrote: @topperc I opened a PR to fix it: https://github.com/llvm/llvm-project/pull/119451 But go ahead and push yours. https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
[clang] [llvm] [RISCV] Add stack clash protection (PR #117612)
@@ -0,0 +1,537 @@ +; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py +; RUN: llc -mtriple=riscv64 -mattr=+m -O2 < %s \ +; RUN: | FileCheck %s -check-prefix=RV64I +; RUN: llc -mtriple=riscv32 -mattr=+m -O2 < %s \ +; RUN: | FileCheck %s -check-prefix=RV32I + +; Tests copied from PowerPC. + +; Free probe +define i8 @f0() #0 nounwind { rzinsly wrote: Done! https://github.com/llvm/llvm-project/pull/117612 ___ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits