From: Clément Léger <[email protected]> Register a handler for the local PMU overflow SSE event so that RISC-V perf can receive overflows even when normal S-mode interrupts are masked. Reuse the existing overflow handler and pass it the interrupted pt_regs rebuilt by the architecture SSE entry path.
Select the delivery mechanism once during PMU probe. Prefer SSE when its event can be registered and enabled. If the extension or PMU event is explicitly unsupported, use the ordinary PMU interrupt. Do not enable the interrupt after any other SSE setup failure or when a crash kernel may have inherited firmware state. Install the PMU enable and disable callbacks only after SSE delivery becomes active. Keep the local PMU SSE event disabled across CPU power management. On entry, the generic SSE notifier masks the hart before the lower-priority PMU notifier disables the event and stops the counters. On exit, the SSE notifier first unmasks the hart while the event remains disabled. The PMU notifier then restores counters and event userpage state before enabling the event. An unmask failure stops the notifier chain and leaves the counters stopped. Ordinary PMU interrupts retain their existing notifier ordering. An SSE overflow can arrive as soon as the event is enabled during probe. Publish the counter mask before SSE setup, so an early handler can stop the counter source even before perf starts admitting normal samples. After a real overflow, restart only counters whose perf state is still running. Honor a non-zero return from perf_event_overflow() and leave throttled events stopped. Guest attribution is not part of this version. Detect an interrupted guest from hstatus.SPV and skip its sample while still updating the period and counter state, rather than exposing guest state as a host sample. The perf PMU callbacks cannot return errors. If an SSE transition or interrupted-context read fails, latch the failure per CPU and stop its mapped events through the normal perf state transitions. Do not reset the firmware counter mapping behind perf, restart a failed event, or attempt a runtime switch to IRQ delivery. During cleanup, close callback admission and synchronously drain each CPU before disabling and unregistering the SSE event. If firmware refuses the cleanup, stop the counter source and transfer the event to the SSE core so later CPU hotplug or shutdown processing can retry without using freed PMU callback state. Signed-off-by: Clément Léger <[email protected]> Co-developed-by: Himanshu Chauhan <[email protected]> Signed-off-by: Himanshu Chauhan <[email protected]> Co-developed-by: Zhanpeng Zhang <[email protected]> Signed-off-by: Zhanpeng Zhang <[email protected]> --- Documentation/arch/riscv/index.rst | 1 + Documentation/arch/riscv/pmu-sse.rst | 55 +++ MAINTAINERS | 1 + drivers/firmware/riscv/riscv_sbi_sse.c | 9 + drivers/perf/Kconfig | 11 + drivers/perf/riscv_pmu.c | 14 +- drivers/perf/riscv_pmu_sbi.c | 544 ++++++++++++++++++++----- include/linux/perf/riscv_pmu.h | 20 +- include/linux/riscv_sbi_sse.h | 6 + 9 files changed, 566 insertions(+), 95 deletions(-) create mode 100644 Documentation/arch/riscv/pmu-sse.rst diff --git a/Documentation/arch/riscv/index.rst b/Documentation/arch/riscv/index.rst index ac535c52d509..5cb909c83a88 100644 --- a/Documentation/arch/riscv/index.rst +++ b/Documentation/arch/riscv/index.rst @@ -11,6 +11,7 @@ RISC-V architecture vm-layout hwprobe patch-acceptance + pmu-sse uabi vector cmodx diff --git a/Documentation/arch/riscv/pmu-sse.rst b/Documentation/arch/riscv/pmu-sse.rst new file mode 100644 index 000000000000..b7458a9d0116 --- /dev/null +++ b/Documentation/arch/riscv/pmu-sse.rst @@ -0,0 +1,55 @@ +.. SPDX-License-Identifier: GPL-2.0 + +======================================== +RISC-V PMU overflow delivery through SSE +======================================== + +When ``CONFIG_RISCV_PMU_SBI_SSE`` is enabled and firmware provides the local +PMU overflow event, the RISC-V SBI PMU driver uses Supervisor Software Events +(SSE) to deliver counter overflows. + +Delivery selection +================== + +The delivery mechanism is selected once while the PMU device is probed. The +driver first tries to register and enable the local PMU overflow SSE event. It +tries the ordinary PMU interrupt only when the SSE extension or the local PMU +event is explicitly unsupported. It does not change the delivery mechanism +after the PMU has been registered. + +Other SSE setup errors do not prove that firmware released the overflow route, +so the driver does not enable the ordinary PMU interrupt. The PMU remains +available for counting but not sampling. If setup retained an SSE event, the +PMU keeps quiescing that event around perf scheduling changes without rearming +it. This preserves callback ownership without creating two possible delivery +mechanisms for the same hardware overflow. + +Interrupted context +=================== + +SSE enters Linux through a supervisor handler context constructed by firmware. +The entry contract preserves the interrupted GPRs except ``a6`` and ``a7``, +which Linux reads from the interrupted-register event attributes. Linux then +reconstructs the interrupted ``pt_regs`` and publishes it while dispatching the +event. + +Perf uses that context for register samples and callchains. Kernel stack +walking verifies that the interrupted frame belongs to the current task stack +before dereferencing it. For sensitive entry paths and IRQ stacks, Linux +records the interrupted PC without walking a stack whose bounds cannot be +proved. User callchains use the existing no-fault user unwinder. A DWARF +raw user-stack copy from an SSE handler must not take a page fault, so it +walks the current task's page tables with fast-only GUP, copies each resident +page through its kernel mapping, and stops at the first non-resident page, +preserving perf's truncated-user-stack semantics. + +CPU power management +==================== + +The PMU and SSE CPU power-management callbacks are ordered according to the +selected delivery mechanism. With SSE delivery, SSE events are masked before +the lower-priority PMU callback disables the local PMU event and stops the +counters on entry. On exit, the hart is unmasked while the local PMU event is +still disabled. The PMU callback then restores the counters and enables the +event, so an unmask failure leaves the counters stopped. The ordinary +interrupt path retains the existing PMU ordering. diff --git a/MAINTAINERS b/MAINTAINERS index d11cd44d2449..91b4e32a1fb2 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -23392,6 +23392,7 @@ M: Atish Patra <[email protected]> R: Anup Patel <[email protected]> L: [email protected] S: Supported +F: Documentation/arch/riscv/pmu-sse.rst F: drivers/perf/riscv_pmu.c F: drivers/perf/riscv_pmu_legacy.c F: drivers/perf/riscv_pmu_sbi.c diff --git a/drivers/firmware/riscv/riscv_sbi_sse.c b/drivers/firmware/riscv/riscv_sbi_sse.c index e81cfb4b0c56..c7d2a2def00a 100644 --- a/drivers/firmware/riscv/riscv_sbi_sse.c +++ b/drivers/firmware/riscv/riscv_sbi_sse.c @@ -161,6 +161,15 @@ static int sse_sbi_event_func(struct sse_event *event, unsigned long func) return 0; } +bool sse_event_is_enabled_local(struct sse_event *event) +{ + if (!sse_event_is_global(event->evt_id)) + lockdep_assert_preemption_disabled(); + + return sse_get_reg_evt(event)->is_enabled; +} +EXPORT_SYMBOL_GPL(sse_event_is_enabled_local); + int sse_event_disable_local(struct sse_event *event) { if (!sse_event_is_global(event->evt_id)) diff --git a/drivers/perf/Kconfig b/drivers/perf/Kconfig index 245e7bb763b9..10b047856121 100644 --- a/drivers/perf/Kconfig +++ b/drivers/perf/Kconfig @@ -105,6 +105,17 @@ config RISCV_PMU_SBI full perf feature support i.e. counter overflow, privilege mode filtering, counter configuration. +config RISCV_PMU_SBI_SSE + depends on RISCV_PMU_SBI && RISCV_SBI_SSE + bool "RISC-V PMU SSE events" + default n + help + Say y if you want to use SSE events to deliver PMU interrupts. This + provides a way to profile the kernel at any level by using NMI-like + SSE events. SSE events being really intrusive, this option allows + to select it only if needed. See Documentation/arch/riscv/pmu-sse.rst + for interrupted-context limitations. + config STARFIVE_STARLINK_PMU depends on ARCH_STARFIVE || COMPILE_TEST depends on 64BIT diff --git a/drivers/perf/riscv_pmu.c b/drivers/perf/riscv_pmu.c index 8e3cd0f35336..1f65c640b693 100644 --- a/drivers/perf/riscv_pmu.c +++ b/drivers/perf/riscv_pmu.c @@ -13,6 +13,7 @@ #include <linux/irqdesc.h> #include <linux/perf/riscv_pmu.h> #include <linux/printk.h> +#include <linux/riscv_sbi_sse.h> #include <linux/smp.h> #include <linux/sched_clock.h> @@ -247,6 +248,11 @@ void riscv_pmu_start(struct perf_event *event, int flags) if (flags & PERF_EF_RELOAD) WARN_ON_ONCE(!(event->hw.state & PERF_HES_UPTODATE)); +#ifdef CONFIG_RISCV_PMU_SBI_SSE + if (unlikely(this_cpu_ptr(rvpmu->hw_events)->sse_failed)) + return; +#endif + hwc->state = 0; riscv_pmu_event_set_period(event); init_val = local64_read(&hwc->prev_count) & max_period; @@ -306,6 +312,7 @@ static int riscv_pmu_event_init(struct perf_event *event) struct hw_perf_event *hwc = &event->hw; struct riscv_pmu *rvpmu = to_riscv_pmu(event->pmu); int mapped_event; + int ret; u64 event_config = 0; uint64_t cmask; @@ -331,8 +338,11 @@ static int riscv_pmu_event_init(struct perf_event *event) hwc->idx = -1; hwc->event_base = mapped_event; - if (rvpmu->event_init) - rvpmu->event_init(event); + if (rvpmu->event_init) { + ret = rvpmu->event_init(event); + if (ret) + return ret; + } if (!is_sampling_event(event)) { /* diff --git a/drivers/perf/riscv_pmu_sbi.c b/drivers/perf/riscv_pmu_sbi.c index 50220f7b46d9..7e971b47730c 100644 --- a/drivers/perf/riscv_pmu_sbi.c +++ b/drivers/perf/riscv_pmu_sbi.c @@ -16,13 +16,16 @@ #include <linux/irqdomain.h> #include <linux/of_irq.h> #include <linux/of.h> +#include <linux/riscv_sbi_sse.h> #include <linux/cpu_pm.h> +#include <linux/crash_dump.h> #include <linux/sched/clock.h> #include <linux/soc/andes/irq.h> #include <linux/workqueue.h> #include <asm/errata_list.h> #include <asm/sbi.h> +#include <asm/sse.h> #include <asm/cpufeature.h> #include <asm/vendor_extensions.h> #include <asm/vendor_extensions/andes.h> @@ -95,7 +98,6 @@ static bool riscv_pmu_use_irq; static unsigned int riscv_pmu_irq_num; static unsigned int riscv_pmu_irq_mask; static unsigned int riscv_pmu_irq; - /* Cache the available counters in a bitmask */ static unsigned long cmask; @@ -896,14 +898,24 @@ static int pmu_sbi_get_ctrinfo(int nctr, unsigned long *mask) return 0; } -static inline void pmu_sbi_stop_all(struct riscv_pmu *pmu) +static inline void pmu_sbi_stop_all_mask(unsigned long ctr_mask) { /* * No need to check the error because we are disabling all the counters * which may include counters that are not enabled yet. */ sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_STOP, - 0, pmu->cmask, SBI_PMU_STOP_FLAG_RESET, 0, 0, 0); + 0, ctr_mask, SBI_PMU_STOP_FLAG_RESET, 0, 0, 0); +} + +static inline void pmu_sbi_stop_all(struct riscv_pmu *pmu) +{ + pmu_sbi_stop_all_mask(pmu->cmask); +} + +static void pmu_sbi_stop_all_cpu(void *info) +{ + pmu_sbi_stop_all(info); } static inline void pmu_sbi_stop_hw_ctrs(struct riscv_pmu *pmu) @@ -953,7 +965,7 @@ static inline void pmu_sbi_stop_hw_ctrs(struct riscv_pmu *pmu) static inline void pmu_sbi_start_ovf_ctrs_sbi(struct cpu_hw_events *cpu_hw_evt, u64 ctr_ovf_mask) { - int idx = 0, i; + int idx, i; struct perf_event *event; unsigned long flag = SBI_PMU_START_FLAG_SET_INIT_VALUE; unsigned long ctr_start_mask = 0; @@ -962,7 +974,19 @@ static inline void pmu_sbi_start_ovf_ctrs_sbi(struct cpu_hw_events *cpu_hw_evt, u64 init_val = 0; for (i = 0; i < BITS_TO_LONGS(RISCV_MAX_COUNTERS); i++) { - ctr_start_mask = cpu_hw_evt->used_hw_ctrs[i] & ~ctr_ovf_mask; + unsigned long ctr_ovf_mask_word; + int lidx; + + ctr_ovf_mask_word = ctr_ovf_mask >> (i * BITS_PER_LONG); + ctr_start_mask = 0; + for_each_set_bit(idx, &cpu_hw_evt->used_hw_ctrs[i], BITS_PER_LONG) { + lidx = idx + i * BITS_PER_LONG; + event = cpu_hw_evt->events[lidx]; + if (event && !(event->hw.state & PERF_HES_STOPPED)) + ctr_start_mask |= BIT(idx); + } + ctr_start_mask &= ~ctr_ovf_mask_word; + /* Start all the counters that did not overflow in a single shot */ if (ctr_start_mask) { sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, i * BITS_PER_LONG, @@ -971,23 +995,25 @@ static inline void pmu_sbi_start_ovf_ctrs_sbi(struct cpu_hw_events *cpu_hw_evt, } /* Reinitialize and start all the counter that overflowed */ - while (ctr_ovf_mask) { - if (ctr_ovf_mask & 0x01) { - event = cpu_hw_evt->events[idx]; - hwc = &event->hw; - max_period = riscv_pmu_ctr_get_width_mask(event); - init_val = local64_read(&hwc->prev_count) & max_period; + for (idx = 0; idx < RISCV_MAX_COUNTERS; idx++) { + if (!(ctr_ovf_mask & BIT_ULL(idx))) + continue; + + event = cpu_hw_evt->events[idx]; + if (!event || event->hw.state & PERF_HES_STOPPED) + continue; + + hwc = &event->hw; + max_period = riscv_pmu_ctr_get_width_mask(event); + init_val = local64_read(&hwc->prev_count) & max_period; #if defined(CONFIG_32BIT) - sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx, 1, - flag, init_val, init_val >> 32, 0); + sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx, 1, + flag, init_val, init_val >> 32, 0); #else - sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx, 1, - flag, init_val, 0, 0); + sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx, 1, + flag, init_val, 0, 0); #endif - perf_event_update_userpage(event); - } - ctr_ovf_mask = ctr_ovf_mask >> 1; - idx++; + perf_event_update_userpage(event); } } @@ -1002,7 +1028,7 @@ static inline void pmu_sbi_start_ovf_ctrs_snapshot(struct cpu_hw_events *cpu_hw_ struct riscv_pmu_snapshot_data *sdata = cpu_hw_evt->snapshot_addr; for_each_set_bit(idx, cpu_hw_evt->used_hw_ctrs, RISCV_MAX_COUNTERS) { - if (ctr_ovf_mask & BIT(idx)) { + if (ctr_ovf_mask & BIT_ULL(idx)) { event = cpu_hw_evt->events[idx]; hwc = &event->hw; max_period = riscv_pmu_ctr_get_width_mask(event); @@ -1016,31 +1042,111 @@ static inline void pmu_sbi_start_ovf_ctrs_snapshot(struct cpu_hw_events *cpu_hw_ } for (i = 0; i < BITS_TO_LONGS(RISCV_MAX_COUNTERS); i++) { + unsigned long ctr_start_mask = 0; + int lidx; + /* Restore the counter values to relative indices for used hw counters */ - for_each_set_bit(idx, &cpu_hw_evt->used_hw_ctrs[i], BITS_PER_LONG) - sdata->ctr_values[idx] = - cpu_hw_evt->snapshot_cval_shcopy[idx + i * BITS_PER_LONG]; + for_each_set_bit(idx, &cpu_hw_evt->used_hw_ctrs[i], BITS_PER_LONG) { + lidx = idx + i * BITS_PER_LONG; + event = cpu_hw_evt->events[lidx]; + if (event && !(event->hw.state & PERF_HES_STOPPED)) + ctr_start_mask |= BIT(idx); + + sdata->ctr_values[idx] = cpu_hw_evt->snapshot_cval_shcopy[lidx]; + } + /* Start all the counters in a single shot */ - sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, idx * BITS_PER_LONG, - cpu_hw_evt->used_hw_ctrs[i], flag, 0, 0, 0); + if (ctr_start_mask) + sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_START, + i * BITS_PER_LONG, ctr_start_mask, flag, 0, 0, 0); } } +static bool pmu_sbi_sse_failed(struct cpu_hw_events *cpu_hw_evt) +{ +#ifdef CONFIG_RISCV_PMU_SBI_SSE + return READ_ONCE(cpu_hw_evt->sse_failed); +#else + return false; +#endif +} + static void pmu_sbi_start_overflow_mask(struct riscv_pmu *pmu, u64 ctr_ovf_mask) { struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(pmu->hw_events); + if (unlikely(pmu_sbi_sse_failed(cpu_hw_evt))) + return; + if (sbi_pmu_snapshot_available()) pmu_sbi_start_ovf_ctrs_snapshot(cpu_hw_evt, ctr_ovf_mask); else pmu_sbi_start_ovf_ctrs_sbi(cpu_hw_evt, ctr_ovf_mask); } -static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) +#ifdef CONFIG_RISCV_PMU_SBI_SSE +/* + * A local SSE delivery failure makes the current PMU state unsafe to resume. + * Latch the failure before stopping mapped events so the SSE transition and + * overflow restart paths cannot undo the fail-safe while they are quiesced. + */ +static void pmu_sbi_fail_sse(struct riscv_pmu *pmu, const char *op, int ret) +{ + struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(pmu->hw_events); + struct perf_event *event; + int idx; + + if (READ_ONCE(cpu_hw_evt->sse_failed)) + return; + + WRITE_ONCE(cpu_hw_evt->sse_failed, true); + pr_err_ratelimited("failed to %s local PMU SSE event: %d; stopping counters\n", + op, ret); + + for (idx = 0; idx < RISCV_MAX_COUNTERS; idx++) { + event = cpu_hw_evt->events[idx]; + if (event) + riscv_pmu_stop(event, PERF_EF_UPDATE); + } +} + +static void pmu_sbi_sse_disable(struct pmu *pmu) +{ + struct riscv_pmu *rvpmu = to_riscv_pmu(pmu); + struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(rvpmu->hw_events); + int ret; + + if (!READ_ONCE(rvpmu->sse_active) || + READ_ONCE(cpu_hw_evt->sse_failed)) + return; + + ret = sse_event_disable_local(rvpmu->sse_evt); + if (unlikely(ret)) + pmu_sbi_fail_sse(rvpmu, "disable", ret); +} + +static void pmu_sbi_sse_enable(struct pmu *pmu) +{ + struct riscv_pmu *rvpmu = to_riscv_pmu(pmu); + struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(rvpmu->hw_events); + int ret; + + if (!READ_ONCE(rvpmu->sse_active) || + READ_ONCE(cpu_hw_evt->sse_failed)) + return; + + ret = sse_event_enable_local(rvpmu->sse_evt); + if (unlikely(ret)) + pmu_sbi_fail_sse(rvpmu, "enable", ret); +} +#endif + +static irqreturn_t pmu_sbi_ovf_handler(struct cpu_hw_events *cpu_hw_evt, + struct pt_regs *regs, bool from_sse, + bool from_guest) { struct perf_sample_data data; - struct pt_regs *regs; struct hw_perf_event *hw_evt; union sbi_pmu_ctr_info *info; int lidx, hidx, fidx; @@ -1048,28 +1154,38 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) struct perf_event *event; u64 overflow; u64 overflowed_ctrs = 0; - struct cpu_hw_events *cpu_hw_evt = dev; u64 start_clock = sched_clock(); struct riscv_pmu_snapshot_data *sdata = cpu_hw_evt->snapshot_addr; if (WARN_ON_ONCE(!cpu_hw_evt)) return IRQ_NONE; - /* Firmware counter don't support overflow yet */ + /* + * SSE can arrive before perf installs an event. The early exits below + * must stop the PMU source before firmware completes the SSE. + */ fidx = find_first_bit(cpu_hw_evt->used_hw_ctrs, RISCV_MAX_COUNTERS); if (fidx == RISCV_MAX_COUNTERS) { - csr_clear(CSR_SIP, BIT(riscv_pmu_irq_num)); + if (from_sse) + pmu_sbi_stop_all_mask(cmask); + else + csr_clear(CSR_SIP, BIT(riscv_pmu_irq_num)); return IRQ_NONE; } event = cpu_hw_evt->events[fidx]; if (!event) { - ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); + if (from_sse) + pmu_sbi_stop_all_mask(cmask); + else + ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); return IRQ_NONE; } pmu = to_riscv_pmu(event->pmu); pmu_sbi_stop_hw_ctrs(pmu); + if (unlikely(pmu_sbi_sse_failed(cpu_hw_evt))) + return IRQ_NONE; /* Overflow status register should only be read after counter are stopped */ if (sbi_pmu_snapshot_available()) @@ -1079,15 +1195,17 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) /* * Overflow interrupt pending bit should only be cleared after stopping - * all the counters to avoid any race condition. + * all the counters to avoid any race condition. When using SSE, + * interrupt is cleared when stopping counters. */ - ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); + if (!from_sse) + ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); /* No overflow bit is set */ - if (!overflow) + if (!overflow) { + pmu_sbi_start_overflow_mask(pmu, 0); return IRQ_NONE; - - regs = get_irq_regs(); + } for_each_set_bit(lidx, cpu_hw_evt->used_hw_ctrs, RISCV_MAX_COUNTERS) { struct perf_event *event = cpu_hw_evt->events[lidx]; @@ -1109,14 +1227,20 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) hidx = info->csr - CSR_CYCLE; /* check if the corresponding bit is set in scountovf or overflow mask in shmem */ - if (!(overflow & BIT(hidx))) + if (!(overflow & BIT_ULL(hidx))) continue; +#ifdef CONFIG_CPU_PM + /* Do not let CPU-PM resume override this overflow decision. */ + if (from_sse) + clear_bit(lidx, cpu_hw_evt->pm_resume_hw_ctrs); +#endif + /* * Keep a track of overflowed counters so that they can be started * with updated initial value. */ - overflowed_ctrs |= BIT(lidx); + overflowed_ctrs |= BIT_ULL(lidx); hw_evt = &event->hw; /* Update the event states here so that we know the state while reading */ hw_evt->state |= PERF_HES_STOPPED; @@ -1124,6 +1248,14 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) hw_evt->state |= PERF_HES_UPTODATE; perf_sample_data_init(&data, 0, hw_evt->last_period); if (riscv_pmu_event_set_period(event)) { + int overflow_ret; + + /* Guest attribution and guest stack sampling are not supported yet. */ + if (from_guest) { + hw_evt->state = 0; + continue; + } + /* * Unlike other ISAs, RISC-V don't have to disable interrupts * to avoid throttling here. As per the specification, the @@ -1132,128 +1264,291 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev) * TODO: We will need to stop the guest counters once * virtualization support is added. */ - perf_event_overflow(event, &data, regs); + overflow_ret = perf_event_overflow(event, &data, regs); + if (!overflow_ret) + hw_evt->state = 0; + } else { + hw_evt->state = 0; } - /* Reset the state as we are going to start the counter after the loop */ - hw_evt->state = 0; } pmu_sbi_start_overflow_mask(pmu, overflowed_ctrs); + perf_sample_event_took(sched_clock() - start_clock); return IRQ_HANDLED; } -static int pmu_sbi_starting_cpu(unsigned int cpu, struct hlist_node *node) +static irqreturn_t pmu_sbi_ovf_irq_handler(int irq, void *dev) { - struct riscv_pmu *pmu = hlist_entry_safe(node, struct riscv_pmu, node); - struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(pmu->hw_events); - - /* - * We keep enabling userspace access to CYCLE, TIME and INSTRET via the - * legacy option but that will be removed in the future. - */ - if (sysctl_perf_user_access == SYSCTL_LEGACY) - csr_write(CSR_SCOUNTEREN, 0x7); - else - csr_write(CSR_SCOUNTEREN, 0x2); + return pmu_sbi_ovf_handler(dev, get_irq_regs(), false, false); +} - /* Stop all the counters so that they can be enabled from perf */ - pmu_sbi_stop_all(pmu); +#ifdef CONFIG_RISCV_PMU_SBI_SSE +static int pmu_sbi_ovf_sse_handler(u32 evt, void *arg, struct pt_regs *regs) +{ + const struct riscv_sse_interrupted_context *context; + struct riscv_pmu *pmu = arg; + struct cpu_hw_events *hw_event = raw_cpu_ptr(pmu->hw_events); + bool from_guest; + + if (unlikely(!READ_ONCE(pmu->sse_active))) { + pmu_sbi_stop_all(pmu); + return -EIO; + } - if (riscv_pmu_use_irq) { - cpu_hw_evt->irq = riscv_pmu_irq; - ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); - enable_percpu_irq(riscv_pmu_irq, IRQ_TYPE_NONE); + if (unlikely(!regs)) { + pmu_sbi_fail_sse(pmu, "read interrupted context for", -EIO); + return -EIO; } - if (sbi_pmu_snapshot_available()) - return pmu_sbi_snapshot_setup(pmu, cpu); + context = riscv_sse_get_interrupted_context(); + from_guest = context && context->regs == regs && + (context->hstatus & HSTATUS_SPV); + pmu_sbi_ovf_handler(hw_event, regs, true, from_guest); return 0; } -static int pmu_sbi_dying_cpu(unsigned int cpu, struct hlist_node *node) +static int pmu_sbi_setup_sse(struct riscv_pmu *pmu) { - if (riscv_pmu_use_irq) { - disable_percpu_irq(riscv_pmu_irq); - } + int ret; + struct sse_event *evt; - /* Disable all counters access for user mode now */ - csr_write(CSR_SCOUNTEREN, 0x0); + evt = sse_event_register(SBI_SSE_EVENT_LOCAL_PMU_OVERFLOW, 0, + pmu_sbi_ovf_sse_handler, pmu); + if (IS_ERR(evt)) + return PTR_ERR(evt); + pmu->sse_evt = evt; - if (sbi_pmu_snapshot_available()) - return pmu_sbi_snapshot_disable(); + ret = sse_event_enable(evt); + if (ret) { + int cleanup_ret; - return 0; + cleanup_ret = sse_event_disable(evt); + if (cleanup_ret) { + pr_warn("failed to disable SSE event after setup error: %d\n", + cleanup_ret); + return cleanup_ret; + } + + cleanup_ret = sse_event_unregister(evt); + + if (cleanup_ret) { + pr_warn("failed to unregister SSE event: %d\n", + cleanup_ret); + } else { + pmu->sse_evt = NULL; + } + return cleanup_ret ?: ret; + } + + WRITE_ONCE(pmu->sse_active, true); + pr_info("using SSE for PMU event delivery\n"); + + return ret; } -static int pmu_sbi_setup_irqs(struct riscv_pmu *pmu, struct platform_device *pdev) +static void pmu_sbi_cleanup_sse(struct riscv_pmu *pmu) { + struct sse_event *sse_evt; int ret; - struct cpu_hw_events __percpu *hw_events = pmu->hw_events; - struct irq_domain *domain = NULL; + sse_evt = pmu->sse_evt; + if (!sse_evt) + return; + /* + * Close callback admission before draining each CPU. PMU callbacks run + * with local interrupts disabled, so the synchronous IPI cannot complete + * until a callback that observed the old state has returned. + */ + WRITE_ONCE(pmu->sse_active, false); + on_each_cpu(pmu_sbi_stop_all_cpu, pmu, 1); + + ret = sse_event_disable(sse_evt); + if (ret) { + pr_warn("failed to disable SSE event: %d\n", ret); + goto retain; + } + + ret = sse_event_unregister(sse_evt); + if (ret) { + pr_warn("failed to unregister SSE event: %d\n", ret); + goto retain; + } + + pmu->sse_evt = NULL; + return; + +retain: + sse_event_cleanup(sse_evt); + pmu->sse_evt = NULL; +} + +static bool pmu_sbi_sse_state_retained(struct riscv_pmu *pmu) +{ + return pmu->sse_evt; +} +#else +static int pmu_sbi_setup_sse(struct riscv_pmu *pmu) +{ + return -EOPNOTSUPP; +} + +static void pmu_sbi_cleanup_sse(struct riscv_pmu *pmu) {} + +static bool pmu_sbi_sse_state_retained(struct riscv_pmu *pmu) +{ + return false; +} +#endif + +static bool pmu_sbi_select_irq(void) +{ if (riscv_isa_extension_available(NULL, SSCOFPMF)) { riscv_pmu_irq_num = RV_IRQ_PMU; - riscv_pmu_use_irq = true; + return true; } else if (IS_ENABLED(CONFIG_ERRATA_THEAD_PMU) && riscv_cached_mvendorid(0) == THEAD_VENDOR_ID && riscv_cached_marchid(0) == 0 && riscv_cached_mimpid(0) == 0) { riscv_pmu_irq_num = THEAD_C9XX_RV_IRQ_PMU; - riscv_pmu_use_irq = true; + return true; } else if (riscv_has_vendor_extension_unlikely(ANDES_VENDOR_ID, RISCV_ISA_VENDOR_EXT_XANDESPMU) && IS_ENABLED(CONFIG_ANDES_CUSTOM_PMU)) { riscv_pmu_irq_num = ANDES_SLI_CAUSE_BASE + ANDES_RV_IRQ_PMOVI; - riscv_pmu_use_irq = true; + return true; } - riscv_pmu_irq_mask = BIT(riscv_pmu_irq_num % BITS_PER_LONG); + return false; +} - if (!riscv_pmu_use_irq) +static int pmu_sbi_setup_irq(struct riscv_pmu *pmu) +{ + struct cpu_hw_events __percpu *hw_events = pmu->hw_events; + struct irq_domain *domain; + int ret; + + if (!pmu_sbi_select_irq()) return -EOPNOTSUPP; + riscv_pmu_irq_mask = BIT(riscv_pmu_irq_num % BITS_PER_LONG); + domain = irq_find_matching_fwnode(riscv_get_intc_hwnode(), DOMAIN_BUS_ANY); if (!domain) { pr_err("Failed to find INTC IRQ root domain\n"); - ret = -ENODEV; - goto err; + return -ENODEV; } riscv_pmu_irq = irq_create_mapping(domain, riscv_pmu_irq_num); if (!riscv_pmu_irq) { pr_err("Failed to map PMU interrupt for node\n"); - ret = -ENODEV; - goto err; + return -ENODEV; } - ret = request_percpu_irq(riscv_pmu_irq, pmu_sbi_ovf_handler, "riscv-pmu", hw_events); + ret = request_percpu_irq(riscv_pmu_irq, pmu_sbi_ovf_irq_handler, + "riscv-pmu", hw_events); if (ret) { pr_err("registering percpu irq failed [%d]\n", ret); irq_dispose_mapping(riscv_pmu_irq); riscv_pmu_irq = 0; - goto err; + return ret; } return 0; -err: - riscv_pmu_use_irq = false; - return ret; +} + +static int pmu_sbi_starting_cpu(unsigned int cpu, struct hlist_node *node) +{ + struct riscv_pmu *pmu = hlist_entry_safe(node, struct riscv_pmu, node); + struct cpu_hw_events *cpu_hw_evt = this_cpu_ptr(pmu->hw_events); + + /* + * We keep enabling userspace access to CYCLE, TIME and INSTRET via the + * legacy option but that will be removed in the future. + */ + if (sysctl_perf_user_access == SYSCTL_LEGACY) + csr_write(CSR_SCOUNTEREN, 0x7); + else + csr_write(CSR_SCOUNTEREN, 0x2); + + /* Stop all the counters so that they can be enabled from perf */ + pmu_sbi_stop_all(pmu); + + if (riscv_pmu_use_irq) { + cpu_hw_evt->irq = riscv_pmu_irq; + ALT_SBI_PMU_OVF_CLEAR_PENDING(riscv_pmu_irq_mask); + enable_percpu_irq(riscv_pmu_irq, IRQ_TYPE_NONE); + } + + if (sbi_pmu_snapshot_available()) + return pmu_sbi_snapshot_setup(pmu, cpu); + + return 0; +} + +static int pmu_sbi_dying_cpu(unsigned int cpu, struct hlist_node *node) +{ + if (riscv_pmu_use_irq) + disable_percpu_irq(riscv_pmu_irq); + + /* Disable all counters access for user mode now */ + csr_write(CSR_SCOUNTEREN, 0x0); + + if (sbi_pmu_snapshot_available()) + return pmu_sbi_snapshot_disable(); + + return 0; +} + +static int pmu_sbi_setup_irqs(struct riscv_pmu *pmu, struct platform_device *pdev) +{ + int irq_ret; + int sse_ret; + + /* Do not claim an IRQ route while a crash kernel leaves SSE state intact. */ + if (is_kdump_kernel() && riscv_sse_available()) { + pr_warn("PMU delivery unavailable with retained crash-kernel SSE state\n"); + return -EUCLEAN; + } + + sse_ret = pmu_sbi_setup_sse(pmu); + if (!sse_ret) { + riscv_pmu_use_irq = false; + return 0; + } + if (pmu_sbi_sse_state_retained(pmu)) { + pr_err("PMU-SSE setup failed with firmware state retained: %d\n", + sse_ret); + return -EUCLEAN; + } + /* Only an explicitly unsupported SSE path proves IRQ fallback is safe. */ + if (sse_ret != -EOPNOTSUPP) { + pr_err("PMU-SSE setup failed: %d\n", sse_ret); + return sse_ret; + } + + irq_ret = pmu_sbi_setup_irq(pmu); + if (!irq_ret) { + riscv_pmu_use_irq = true; + return 0; + } + return irq_ret; } #ifdef CONFIG_CPU_PM -static int riscv_pm_pmu_notify(struct notifier_block *b, unsigned long cmd, - void *v) +static int riscv_pm_pmu_update(struct riscv_pmu *rvpmu, unsigned long cmd) { - struct riscv_pmu *rvpmu = container_of(b, struct riscv_pmu, riscv_pm_nb); struct cpu_hw_events *cpuc = this_cpu_ptr(rvpmu->hw_events); bool enabled = !bitmap_empty(cpuc->used_hw_ctrs, RISCV_MAX_COUNTERS); struct perf_event *event; int idx; + if (cmd == CPU_PM_ENTER) + bitmap_zero(cpuc->pm_resume_hw_ctrs, RISCV_MAX_COUNTERS); + if (!enabled) return NOTIFY_OK; @@ -1264,6 +1559,8 @@ static int riscv_pm_pmu_notify(struct notifier_block *b, unsigned long cmd, switch (cmd) { case CPU_PM_ENTER: + if (!(event->hw.state & PERF_HES_STOPPED)) + set_bit(idx, cpuc->pm_resume_hw_ctrs); /* * Stop and update the counter */ @@ -1271,6 +1568,8 @@ static int riscv_pm_pmu_notify(struct notifier_block *b, unsigned long cmd, break; case CPU_PM_EXIT: case CPU_PM_ENTER_FAILED: + if (!test_and_clear_bit(idx, cpuc->pm_resume_hw_ctrs)) + break; /* * Restore and enable the counter. */ @@ -1284,9 +1583,59 @@ static int riscv_pm_pmu_notify(struct notifier_block *b, unsigned long cmd, return NOTIFY_OK; } +static int riscv_pm_pmu_notify(struct notifier_block *b, + unsigned long cmd, void *v) +{ + struct riscv_pmu *rvpmu = container_of(b, struct riscv_pmu, + riscv_pm_nb); + +#ifdef CONFIG_RISCV_PMU_SBI_SSE + struct cpu_hw_events *cpuc = this_cpu_ptr(rvpmu->hw_events); + int ret; + + if (!riscv_pmu_use_irq && READ_ONCE(rvpmu->sse_active)) { + switch (cmd) { + case CPU_PM_ENTER: + cpuc->pm_resume_sse = + sse_event_is_enabled_local(rvpmu->sse_evt); + if (cpuc->pm_resume_sse) { + ret = sse_event_disable_local(rvpmu->sse_evt); + if (ret) { + cpuc->pm_resume_sse = false; + pmu_sbi_fail_sse(rvpmu, "disable for CPU PM", + ret); + return notifier_from_errno(ret); + } + } + break; + case CPU_PM_EXIT: + case CPU_PM_ENTER_FAILED: + ret = riscv_pm_pmu_update(rvpmu, cmd); + if (!cpuc->pm_resume_sse) + return ret; + + cpuc->pm_resume_sse = false; + ret = sse_event_enable_local(rvpmu->sse_evt); + if (ret) { + pmu_sbi_fail_sse(rvpmu, "enable after CPU PM", ret); + return notifier_from_errno(ret); + } + return NOTIFY_OK; + default: + break; + } + } +#endif + + return riscv_pm_pmu_update(rvpmu, cmd); +} + static int riscv_pm_pmu_register(struct riscv_pmu *pmu) { pmu->riscv_pm_nb.notifier_call = riscv_pm_pmu_notify; + /* Keep PMU-SSE disabled until counters and userpage state are restored. */ + pmu->riscv_pm_nb.priority = riscv_pmu_use_irq ? 1 : -1; + return cpu_pm_register_notifier(&pmu->riscv_pm_nb); } @@ -1301,6 +1650,8 @@ static inline void riscv_pm_pmu_unregister(struct riscv_pmu *pmu) { } static void riscv_pmu_destroy(struct riscv_pmu *pmu) { + pmu_sbi_cleanup_sse(pmu); + if (sbi_v2_available) { if (sbi_pmu_snapshot_available()) { pmu_sbi_snapshot_disable(); @@ -1312,7 +1663,7 @@ static void riscv_pmu_destroy(struct riscv_pmu *pmu) cpuhp_state_remove_instance(CPUHP_AP_PERF_RISCV_STARTING, &pmu->node); } -static void pmu_sbi_event_init(struct perf_event *event) +static int pmu_sbi_event_init(struct perf_event *event) { /* * The permissions are set at event_init so that we do not depend @@ -1324,6 +1675,8 @@ static void pmu_sbi_event_init(struct perf_event *event) event->hw.flags |= PERF_EVENT_FLAG_USER_ACCESS; else event->hw.flags |= PERF_EVENT_FLAG_LEGACY; + + return 0; } static void pmu_sbi_event_mapped(struct perf_event *event, struct mm_struct *mm) @@ -1453,6 +1806,7 @@ static int pmu_sbi_device_probe(struct platform_device *pdev) /* cache all the information about counters now */ if (pmu_sbi_get_ctrinfo(num_counters, &cmask)) goto out_free; + pmu->cmask = cmask; ret = pmu_sbi_setup_irqs(pmu, pdev); if (ret < 0) { @@ -1462,9 +1816,15 @@ static int pmu_sbi_device_probe(struct platform_device *pdev) } irq_requested = (ret == 0); +#ifdef CONFIG_RISCV_PMU_SBI_SSE + if (pmu->sse_active) { + pmu->pmu.pmu_enable = pmu_sbi_sse_enable; + pmu->pmu.pmu_disable = pmu_sbi_sse_disable; + } +#endif + pmu->pmu.attr_groups = riscv_pmu_attr_groups; pmu->pmu.parent = &pdev->dev; - pmu->cmask = cmask; pmu->ctr_start = pmu_sbi_ctr_start; pmu->ctr_stop = pmu_sbi_ctr_stop; pmu->event_map = pmu_sbi_event_map; diff --git a/include/linux/perf/riscv_pmu.h b/include/linux/perf/riscv_pmu.h index f82a28040594..fb913ce7f65e 100644 --- a/include/linux/perf/riscv_pmu.h +++ b/include/linux/perf/riscv_pmu.h @@ -28,6 +28,8 @@ #define RISCV_PMU_CONFIG1_GUEST_EVENTS 0x1 +struct sse_event; + struct cpu_hw_events { /* currently enabled events */ int n_events; @@ -39,6 +41,18 @@ struct cpu_hw_events { DECLARE_BITMAP(used_hw_ctrs, RISCV_MAX_COUNTERS); /* currently enabled firmware counters */ DECLARE_BITMAP(used_fw_ctrs, RISCV_MAX_COUNTERS); +#ifdef CONFIG_RISCV_PMU_SBI_SSE + /* Keep counters stopped after an unrecoverable SSE transition failure. */ + bool sse_failed; +#endif +#ifdef CONFIG_CPU_PM + /* Counters stopped by CPU PM and still waiting to be restored. */ + DECLARE_BITMAP(pm_resume_hw_ctrs, RISCV_MAX_COUNTERS); +#ifdef CONFIG_RISCV_PMU_SBI_SSE + /* Restore the local PMU SSE event after counters and userpage state. */ + bool pm_resume_sse; +#endif +#endif /* The virtual address of the shared memory where counter snapshot will be taken */ void *snapshot_addr; /* The physical address of the shared memory where counter snapshot will be taken */ @@ -54,6 +68,10 @@ struct riscv_pmu { char *name; irqreturn_t (*handle_irq)(int irq_num, void *dev); +#ifdef CONFIG_RISCV_PMU_SBI_SSE + struct sse_event *sse_evt; + bool sse_active; +#endif unsigned long cmask; u64 (*ctr_read)(struct perf_event *event); @@ -63,7 +81,7 @@ struct riscv_pmu { void (*ctr_start)(struct perf_event *event, u64 init_val); void (*ctr_stop)(struct perf_event *event, unsigned long flag); int (*event_map)(struct perf_event *event, u64 *config); - void (*event_init)(struct perf_event *event); + int (*event_init)(struct perf_event *event); void (*event_mapped)(struct perf_event *event, struct mm_struct *mm); void (*event_unmapped)(struct perf_event *event, struct mm_struct *mm); uint8_t (*csr_index)(struct perf_event *event); diff --git a/include/linux/riscv_sbi_sse.h b/include/linux/riscv_sbi_sse.h index 774a782e556d..f6b3881c8884 100644 --- a/include/linux/riscv_sbi_sse.h +++ b/include/linux/riscv_sbi_sse.h @@ -42,6 +42,7 @@ int sse_event_enable(struct sse_event *sse_evt); int sse_event_disable(struct sse_event *sse_evt); /* Local events require the caller to remain on the current CPU. */ +bool sse_event_is_enabled_local(struct sse_event *sse_evt); int sse_event_enable_local(struct sse_event *sse_evt); int sse_event_disable_local(struct sse_event *sse_evt); @@ -76,6 +77,11 @@ static inline int sse_event_disable(struct sse_event *sse_evt) return -EOPNOTSUPP; } +static inline bool sse_event_is_enabled_local(struct sse_event *sse_evt) +{ + return false; +} + static inline int sse_event_enable_local(struct sse_event *sse_evt) { return -EOPNOTSUPP; -- 2.50.1 (Apple Git-155)
