On Tue, 16 Oct 2018 10:49:25 +0100 "Dr. David Alan Gilbert" <dgilb...@redhat.com> wrote:
> * Alex Williamson (alex.william...@redhat.com) wrote: > > This email is already too long, but I also wonder whether we should > > consider additional vfio-pci interfaces to trigger a link retraining or > > allow virtualized access to the physical upstream port config space. > > Clearly we need to consider multi-function devices and whether there > > are useful configurations that could benefit from such access. Thanks > > for reading, please discuss, > > I'm not sure about the consequences of the actual link speeds, but I > worry we'll hit things looking for PCIe v3 features; in particular > AMD's ROCm code needs PCIe atomics: > > https://github.com/RadeonOpenCompute/ROCm_Documentation/blob/master/Installation_Guide/More-about-how-ROCm-uses-PCIe-Atomics.rst > > so it feels like getting that to work with passthrough would > need some negotiation of features. Taking a quick read through the AtomicOps ECN, I think this is somewhat orthogonal to the link speeds and widths. Support for acting as a completer or router of atomic ops is indicated through the DEVCAP2 register in the PCIe capability, it's optional and should not be assumed by other newer PCIe features. So I think we can tackle it separately, but indeed it does appear to be a difficult feature to implement correctly for a VM, at least if we attempt to do it automatically. We might need to burden the user with this sort of configuration unless AtomicOps support is so ubiquitous that we can correctly assume that it's available between arbitrary endpoints which might be assigned to the same VM. Probably a device option on the pcie-root-port device to expose atomic op support is possible in the short term, though more thorough reading of the spec is required. Thanks, Alex
