On 11/6/18 4:52 PM, Gerd Rausch wrote:
This reverts commit 56012459310a1dbcc55c2dbf5500a9f7571402cb.
Subject: rds_ib_post_reg_frmr spins and spins and spins
RDS kept spinning inside function "rds_ib_post_reg_frmr", waiting for
"i_fastreg_wrs" to become incremented:
while (atomic_dec_return(&ibmr->ic->i_fastreg_wrs) <= 0) {
atomic_inc(&ibmr->ic->i_fastreg_wrs);
cpu_relax();
}
Looking at the original commit:
commit 56012459310a ("RDS: IB: split the mr registration and invalidation path")
In there, the "rds_ib_mr_cqe_handler" was changed in the following way:
@@ -289,9 +289,10 @@ void rds_ib_mr_cqe_handler(struct rds_ib_connection *ic,
struct ib_wc *wc)
if (frmr->fr_inv) {
frmr->fr_state = FRMR_IS_FREE;
frmr->fr_inv = false;
+ atomic_inc(&ic->i_fastreg_wrs);
+ } else {
+ atomic_inc(&ic->i_fastunreg_wrs);
}
-
- atomic_inc(&ic->i_fastreg_wrs);
}
It looks like it's got it exactly backwards:
Function "rds_ib_post_reg_frmr" keeps track of the outstanding requests
via "i_fastreg_wrs".
Function "rds_ib_post_inv" keeps track of the outstanding requests
via "i_fastunreg_wrs" (post original commit). It also sets:
frmr->fr_inv = true;
However the completion handler "rds_ib_mr_cqe_handler" adjusts "i_fastreg_wrs"
when "fr_inv" had been true, and adjusts "i_fastunreg_wrs" otherwise.
The original commit was done in the name of performance:
to remove the performance bottleneck
No performance benefit could be observed with a fixed-up version
of the original commit measured between two Oracle X7 servers,
both equipped with Mellanox Connect-X5 HCAs.
The prudent course of action is to revert this commit.
Signed-off-by: Gerd Rausch <gerd.rau...@oracle.com>
---
As mentioned in offline discussion, when we added this change, the test
server was equipped with Connect-X3 NIC running with FRWR mode and it
did show step improvements. Unfortunately I don't have that data
stored, so it is just from memory.
But looking at Connext-X5 numbers(thanks), am fine to back this out.
Acked-by: Santosh Shilimkar <santosh.shilim...@oracle.com>
