On Thu, 2016-01-28 at 10:25 +0100, Jesper Dangaard Brouer wrote:
> Yes, that is exactly what I'm contemplating :-) That is idea "(1)".
>
> A natural extension to this work, which I expect Tom will love, is to
> also use the idea for RPS. Once we have a SKB list in stack/GRO-layer,
> then we could build a local sk_buff_head list for each remote CPU, by
> calling get_rps_cpu(). And then enqueue_list_to_backlog, by a
> skb_queue_splice_tail(&cpu_list, &cpu->sd->input_pkt_queue) call.
>
> This would amortize the cost of transferring packets to a remote CPU,
> which Eric AFAIK points out is costing approx ~133ns.
>
Jesper, RPS and RFS already defer sending the IPI and submit batches to
remote cpus.
See commits
e326bed2f47d0365da5a8faaf8ee93ed2d86325b ("rps: immediate send IPI in
process_backlog()")
88751275b8e867d756e4f86ae92afe0232de129f ("rps: shortcut
net_rps_action()")
And of course all the discussions we had to come up with
0a9627f2649a02bea165cfd529d7bcb625c2fcad ("rps: Receive Packet
Steering")
The current state :
net_rps_action_and_irq_enable() sends the IPI at the end of
net_rx_action() once all NAPI handlers have been called, and therefore
have accumulated packets and cook rps_ipi_list (via calls to
rps_ipi_queued() from enqueue_to_backlog())
Adding another stage in the pipeline would not help.
