> From: Dan Williams <dan.j.willi...@intel.com> > On Fri, Mar 12, 2021 at 1:55 PM Chen, Mike Ximing > <mike.ximing.c...@intel.com> wrote: > > > > We support up to 16/32 VF/VDEVs (depending on version) with SRIOV and > > SIOV. Role of the kernel driver includes VDEV Composition (vdcm > > module), functional level reset, live migration, error handling, power > > management, and etc.. > > Need some more specificity here. What about those features requires the > kernel to get involved with a > DLB2 specific ABI to manage ports, queues, credits, sequence numbers, etc...?
Role of the dlb kernel driver: VDEV Composition For example writing 1024 to the VDEV_CREDITS[0] register will allocate 1024 credits to VDEV 0. In this way, VFs or VDEVs can be composed as mini-versions of the full device. VDEV composition will leverage vfio-mdev to create the VDEV devices while the KMD will implement the VDCM. Dynamic Composition Such composition can be dynamic – the PF/VF interface supports scenarios whereby, for example, an application may wish to boost its credit allocation – can I have 100 more credits? Functional Level Reset Much of the internal storage is RAM based and not resettable by hardware schemes. There are also internal SRAM based control structures (BCAM) that have to be flushed. The planned way to do this is, roughly: -- Kernel driver disables access from the associated ports (to prevent any SW access, the application should be deadso this is a precaution). -- Kernel masquerades as the application to drain all data from internal queues. It can poll some internal counters to verify everything is fully drained. -- Only at this point can the resources associated with the VDEV be returned to the pool of available resources for handing to another application/VDEV. Migration Requirement is fairly similar to FLR. A VDEV has to be manually drained and reconstituted on another server, Kernel driver is responsible on both sides. Error Handling Errors include “Credit Excursions” where a VDEV attempts to use more of the internal capacity (credits) than has been allocated. In such a case, the data is dropped and an interrupt generated. All such interrupts are directed to the PF driver, which may simply forward them to a VF (via the PF/VF comms mechanism). Power Management The kernel driver keeps the device in D3Hot when not in use. The driver transitions the device to D0 when the first device file is opened or a VF or VDEV is created, and keeps it in that state until there are no open device files, memory mappings, or VFs/VDEVs. Ioctl interface Kernel driver provides ioctl interface for user applications to setup and configure dlb domains, ports, queues, scheduling types, credits, sequence numbers, and links between ports and queues. Applications also use the interface to start, stop and inquire the dlb operations. Thanks Mike
