On Wed, 2021-02-03 at 13:26 -0800, Jakub Kicinski wrote: > On Wed, 03 Feb 2021 11:22:44 -0800 Saeed Mahameed wrote: > > On Wed, 2021-02-03 at 10:51 -0800, Jakub Kicinski wrote: > > > On Tue, 02 Feb 2021 20:13:48 -0800 Saeed Mahameed wrote: > > > > yes, user in this case is the admin, who controls the > > > > provisioned > > > > network function SF/VFs.. by turning off this knob it allows to > > > > create > > > > more of that resource in case the user/admin is limited by > > > > memory. > > > > > > Ah, so in case of the SmartNIC this extra memory is allocated on > > > the > > > control system, not where the function resides? > > > > most of the memeory are actually allocated from where the function > > resides, some are on the management system but it is not as > > critical. > > SFs for now can only be probed on the management system, so the > > main > > issue will be on the SmartNIC side for now. > > Why not leave the decision whether to allocate that memory or not to > the SF itself? If user never binds the RDMA driver to the SF they > clearly don't care for RDMA. No extra knobs needed. >
Sorry about the late response. But FW is already setup for RDMA weather SW wants it or not for this specific function. a system admin may want to deploy a leaner function to the client. we can disable RoCE in FW before we even instantiate this function. > > > My next question is regarding the behavior on the target system - > > > what > > > does "that user" see? Can we expect they will understand that the > > > limitation was imposed by the admin and not due to some > > > initialization > > > failure or SW incompatibility? > > > > the whole thing works with only real HW capabilities, there is no > > synthetic SW capabilities. > > > > when mlx5 instance driver loads, it doesn't assume anything about > > underlying HW, and it queries for the advertised FW capability > > according to the HW spec before it enables a feature. > > > > so this patch adds the ability for admin to enforce a specific HW > > cap > > "off" for a VF/SF hca slice. > > > > > > RAW eth QP, i think you already know this one, it is a very > > > > thin > > > > layer > > > > that doesn't require the whole rdma stack. > > > > > > Sorry for asking a leading question. You know how we'll feel > > > about > > > that one, do we need to talk this out or can we save ourselves > > > the > > > battle? :S > > > > I know, I know :/ > > > > So, there is no rdma bit/cap in HW.. to disable non-RoCE commands > > we > > will have to disable etherent capability. > > It's your driver, you can make it do what you need to. Why does > the RDMA driver bind successfully to a non-RoCE Ethernet device > in the first place? > because RDMA people are greedy :), and they can create QPs that are not RoCE.. this patch is only basic enable/disable RoCE feature on a specific device slice, how rdma driver initializes itself is not related to this patch at all, and it is just how rdma works, with or without this patch. > > The user interface here has no synthetic semantics, all knobs will > > eventually be mapped to real HW/FW capabilities to get disabled. > > > > the whole feature is about allowing admin to ship network functions > > with different capabilities that are actually enforced by FW/HW.. > > so the user of the VF will see, RDMA/ETH only cards or both. > > RDMA-only, ETH-only, RDMA+ETH makes sense to me. Having an ETH-only > device also exposed though rdma subsystem does not. But this has nothing to do with this patch. this is the rdma driver implementation.
