Re: NMI for ARC
On Tue, Sep 27, 2016 at 05:22:13PM -0700, Vineet Gupta wrote: > > Yeah, Sparc64 might be a better example, it more closely matches your > > hardware. See > > arch/sparc/include/asm/irqflags_64.h:arch_local_irq_save(). > > So I finally got around to doing this and as expected has turned out to be > quite > some fun. I have a couple of questions and would really appreciate your > inputs there. > > 1. Is it OK in general to short-circuit preemption off irq checks for NMI > style > interrupts. Yes. If the NMI returns to kernel space you must not attempt preemption for reasons you found :-), if the NMI returns to userspace you should do the normal return to user bits, I think. > 2. The low level return code, resume_user_mode_begin and/or resume_kernel_mode > require interrupt safety, does that need to be NMI safe as well. We ofcourse > want > the very late register restore parts to be non-interruptible, but is this > required > before we call prrempt_schedule_irq() off of asm code. Urgh, I'm never quite sure on the details here, I've Cc'ed Andy who might actually know this off the top of his head. I'll try and dig through x86 to see what it does. ___ linux-snps-arc mailing list linux-snps-arc@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Re: NMI for ARC
On 09/28/2016 12:16 AM, Peter Zijlstra wrote:
>> 1. Is it OK in general to short-circuit preemption off irq checks for NMI
>> style
>> > interrupts.
> Yes. If the NMI returns to kernel space you must not attempt preemption
> for reasons you found :-), if the NMI returns to userspace you should do
> the normal return to user bits, I think.
Just to add, eliding preemption check in return path might not be sufficient as
we
could still have timer intr leading to scheduler_tick() - whcih is what i found
as
well ;-) So scheduler nevertheless needs to be told to not reschedule in this
code
path.
So eliding preempt-off-irq just becomes an optimization for NMI code paths IMHO.
Now I was a stupid fool, fudging preemption counts in low level code, to achieve
this, whereas we have nice generic nmi_{enter,exit}() and in_nmi() helpers which
can be used.
Thx,
-Vineet
___
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Re: NMI for ARC
On Wed, Sep 28, 2016 at 12:16 AM, Peter Zijlstra wrote: > On Tue, Sep 27, 2016 at 05:22:13PM -0700, Vineet Gupta wrote: > >> > Yeah, Sparc64 might be a better example, it more closely matches your >> > hardware. See >> > arch/sparc/include/asm/irqflags_64.h:arch_local_irq_save(). >> >> So I finally got around to doing this and as expected has turned out to be >> quite >> some fun. I have a couple of questions and would really appreciate your >> inputs there. >> >> 1. Is it OK in general to short-circuit preemption off irq checks for NMI >> style >> interrupts. > > Yes. If the NMI returns to kernel space you must not attempt preemption > for reasons you found :-), Last time I looked at this, I decided that there was no reason that NMIs would ever need to handle preemption. Even if the NMI hit interruptible kernel code, anything that would cause preemption to be needed would either send an IPI (and thus cause preemption) right after the NMI fiinished. NMI handlers themselves have no business setting TIF_NEED_RESCHED or similar. > if the NMI returns to userspace you should do > the normal return to user bits, I think. x86 does this for simplicity. There was a really nasty corner case that I could only figure out how to solve by special casing NMIs from user space. I'm not sure that it's actually necessary from a non-arch-specific POV to handle all the usual return-to-userspace work on NMI. But maybe perf NMIs can send signals? x86's MCEs *do* need the full return-to-userspace handling for memory failure to work right. MCE is kind of like NMI... > >> 2. The low level return code, resume_user_mode_begin and/or >> resume_kernel_mode >> require interrupt safety, does that need to be NMI safe as well. We ofcourse >> want >> the very late register restore parts to be non-interruptible, but is this >> required >> before we call prrempt_schedule_irq() off of asm code. > > Urgh, I'm never quite sure on the details here, I've Cc'ed Andy who > might actually know this off the top of his head. I'll try and dig > through x86 to see what it does. On x86, it's quite simple. IRQs are *always* off during the final register restore, and we don't re-check for preemption there. x86 handles preemption after turning off IRQs, and IRQs are guaranteed to stay off until we actually return to userspace. The code is almost entirely in C in arch/x86/entry/common.c. There isn't anything particularly x86-speficic in there. ___ linux-snps-arc mailing list linux-snps-arc@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Re: NMI for ARC
On Wed, Sep 28, 2016 at 12:25:11PM -0700, Andy Lutomirski wrote: > > Yes. If the NMI returns to kernel space you must not attempt preemption > > for reasons you found :-), > > Last time I looked at this, I decided that there was no reason that > NMIs would ever need to handle preemption. Even if the NMI hit > interruptible kernel code, anything that would cause preemption to be > needed would either send an IPI (and thus cause preemption) right > after the NMI fiinished. NMI handlers themselves have no business > setting TIF_NEED_RESCHED or similar. Good point, they don't and therefore you need not bother. > > if the NMI returns to userspace you should do > > the normal return to user bits, I think. > > x86 does this for simplicity. There was a really nasty corner case > that I could only figure out how to solve by special casing NMIs from > user space. I'm not sure that it's actually necessary from a > non-arch-specific POV to handle all the usual return-to-userspace work > on NMI. But maybe perf NMIs can send signals? No it cannot. It uses irq_work (which sends a self-IPI) when it wants to do signals. > >> 2. The low level return code, resume_user_mode_begin and/or > >> resume_kernel_mode > >> require interrupt safety, does that need to be NMI safe as well. We > >> ofcourse want > >> the very late register restore parts to be non-interruptible, but is this > >> required > >> before we call prrempt_schedule_irq() off of asm code. > > > > Urgh, I'm never quite sure on the details here, I've Cc'ed Andy who > > might actually know this off the top of his head. I'll try and dig > > through x86 to see what it does. > > On x86, it's quite simple. IRQs are *always* off during the final > register restore, and we don't re-check for preemption there. x86 > handles preemption after turning off IRQs, and IRQs are guaranteed to > stay off until we actually return to userspace. > > The code is almost entirely in C in arch/x86/entry/common.c. There > isn't anything particularly x86-speficic in there. Right, so what I think Vineet is asking is if we need to disable NMIs as well, we cannot on x86 disable NMIs so no. ___ linux-snps-arc mailing list linux-snps-arc@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Re: NMI for ARC
On Sep 28, 2016 1:37 PM, "Peter Zijlstra" wrote: > > On Wed, Sep 28, 2016 at 12:25:11PM -0700, Andy Lutomirski wrote: > > > Yes. If the NMI returns to kernel space you must not attempt preemption > > > for reasons you found :-), > > > > Last time I looked at this, I decided that there was no reason that > > NMIs would ever need to handle preemption. Even if the NMI hit > > interruptible kernel code, anything that would cause preemption to be > > needed would either send an IPI (and thus cause preemption) right > > after the NMI fiinished. NMI handlers themselves have no business > > setting TIF_NEED_RESCHED or similar. > > Good point, they don't and therefore you need not bother. > > > > if the NMI returns to userspace you should do > > > the normal return to user bits, I think. > > > > x86 does this for simplicity. There was a really nasty corner case > > that I could only figure out how to solve by special casing NMIs from > > user space. I'm not sure that it's actually necessary from a > > non-arch-specific POV to handle all the usual return-to-userspace work > > on NMI. But maybe perf NMIs can send signals? > > No it cannot. It uses irq_work (which sends a self-IPI) when it wants to > do signals. > > > >> 2. The low level return code, resume_user_mode_begin and/or > > >> resume_kernel_mode > > >> require interrupt safety, does that need to be NMI safe as well. We > > >> ofcourse want > > >> the very late register restore parts to be non-interruptible, but is > > >> this required > > >> before we call prrempt_schedule_irq() off of asm code. > > > > > > Urgh, I'm never quite sure on the details here, I've Cc'ed Andy who > > > might actually know this off the top of his head. I'll try and dig > > > through x86 to see what it does. > > > > On x86, it's quite simple. IRQs are *always* off during the final > > register restore, and we don't re-check for preemption there. x86 > > handles preemption after turning off IRQs, and IRQs are guaranteed to > > stay off until we actually return to userspace. > > > > The code is almost entirely in C in arch/x86/entry/common.c. There > > isn't anything particularly x86-speficic in there. > > Right, so what I think Vineet is asking is if we need to disable NMIs as > well, we cannot on x86 disable NMIs so no. > The same argument works here, too: an NMI won't set TIF_NEED_RESCHED without sending an IPI, so we can't miss a wakeup. ___ linux-snps-arc mailing list linux-snps-arc@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Re: NMI for ARC
On 09/28/2016 03:26 PM, Andy Lutomirski wrote: > 2. The low level return code, resume_user_mode_begin and/or > resume_kernel_mode > > > >> require interrupt safety, does that need to be NMI safe as well. > > > >> We ofcourse want > > > >> the very late register restore parts to be non-interruptible, but > > > >> is this required > > > >> before we call prrempt_schedule_irq() off of asm code. > > > > > > Urgh, I'm never quite sure on the details here, I've Cc'ed Andy who > > > might actually know this off the top of his head. I'll try and dig > > > through x86 to see what it does. >>> > > >>> > > On x86, it's quite simple. IRQs are *always* off during the final >>> > > register restore, and we don't re-check for preemption there. x86 >>> > > handles preemption after turning off IRQs, and IRQs are guaranteed to >>> > > stay off until we actually return to userspace. >>> > > >>> > > The code is almost entirely in C in arch/x86/entry/common.c. There >>> > > isn't anything particularly x86-speficic in there. >> > >> > Right, so what I think Vineet is asking is if we need to disable NMIs as >> > well, we cannot on x86 disable NMIs so no. >> > > The same argument works here, too: an NMI won't set TIF_NEED_RESCHED > without sending an IPI, so we can't miss a wakeup. The case I saw was different: timer intr (normal prio) comes in - scheduler_tick() sets TIF_NEED_RESCHED and before this intr return, it gets interrupted by perf intr (higher prio) and we decide not to follow through on preemption because a nested intr can't return to userspace anyways. ___ linux-snps-arc mailing list linux-snps-arc@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Re: NMI for ARC
On Wed, Sep 28, 2016 at 3:44 PM, Vineet Gupta wrote: > On 09/28/2016 03:26 PM, Andy Lutomirski wrote: >> 2. The low level return code, resume_user_mode_begin and/or >> resume_kernel_mode >> > > >> require interrupt safety, does that need to be NMI safe as well. >> > > >> We ofcourse want >> > > >> the very late register restore parts to be non-interruptible, but >> > > >> is this required >> > > >> before we call prrempt_schedule_irq() off of asm code. > > > > > > > > Urgh, I'm never quite sure on the details here, I've Cc'ed Andy who > > > > might actually know this off the top of his head. I'll try and dig > > > > through x86 to see what it does. > > > > On x86, it's quite simple. IRQs are *always* off during the final > > register restore, and we don't re-check for preemption there. x86 > > handles preemption after turning off IRQs, and IRQs are guaranteed to > > stay off until we actually return to userspace. > > > > The code is almost entirely in C in arch/x86/entry/common.c. There > > isn't anything particularly x86-speficic in there. >>> > >>> > Right, so what I think Vineet is asking is if we need to disable NMIs as >>> > well, we cannot on x86 disable NMIs so no. >>> > >> The same argument works here, too: an NMI won't set TIF_NEED_RESCHED >> without sending an IPI, so we can't miss a wakeup. > > The case I saw was different: timer intr (normal prio) comes in - > scheduler_tick() > sets TIF_NEED_RESCHED and before this intr return, it gets interrupted by perf > intr (higher prio) and we decide not to follow through on preemption because a > nested intr can't return to userspace anyways. > > This shouldn't cause a problem. When the timer interrupt returns, it should be able to handle preemption. ___ linux-snps-arc mailing list linux-snps-arc@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Re: NMI for ARC
On 09/28/2016 03:26 PM, Andy Lutomirski wrote:
>> Right, so what I think Vineet is asking is if we need to disable NMIs as
>> > well, we cannot on x86 disable NMIs so no.
>> >
> The same argument works here, too: an NMI won't set TIF_NEED_RESCHED
> without sending an IPI, so we can't miss a wakeup.
But what exact wakeup miss are we taking about here. If intr were NOT disabled,
how could this happen. Just trying to understand the need for "irqs-disabled" in
resume_{user,kernel}_*
The intr disabled before reg file restore makes complete sense though.
___
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
Re: NMI for ARC
On Wed, Sep 28, 2016 at 06:20:29PM -0700, Vineet Gupta wrote:
> On 09/28/2016 03:26 PM, Andy Lutomirski wrote:
> >> Right, so what I think Vineet is asking is if we need to disable NMIs as
> >> > well, we cannot on x86 disable NMIs so no.
> >> >
> > The same argument works here, too: an NMI won't set TIF_NEED_RESCHED
> > without sending an IPI, so we can't miss a wakeup.
>
> But what exact wakeup miss are we taking about here. If intr were NOT
> disabled,
> how could this happen. Just trying to understand the need for "irqs-disabled"
> in
> resume_{user,kernel}_*
>
> The intr disabled before reg file restore makes complete sense though.
userirq nmi
|
|
`-> .
|
|
|
`-> .
|
|
. <-'
. <-'
|
|
So what Andy is saying is that NMI context never sets TIF_NEED_RESCHED,
this means that return from NMI never needs to check for preemption
etc..
Now your return from IRQ obviously should, the normal way. If the IRQ
return gets interrupted by the NMI nothing special should occur. The
return from NMI should simply resume the return from IRQ.
So I'm a little confused by your timer interrupt example, it _should_ do
the preemption, the nested interrupt (NMI) will return to the regular
interrupt which should resume its normal return preemption or not.
___
linux-snps-arc mailing list
linux-snps-arc@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-snps-arc
