Changes since v4: https://lore.kernel.org/qemu-devel/[email protected]/ - Dropped initial refactor. Reason being that it is not static enough for caching at boot to be appropriate. Kept the spec update part of this. - Added a patch to make sure the new flip mode stuff is reported in the Get Physical Port State command. - Rebased Arpit's Physical Control Control FMAPI Command given drop of the major refactor from v4 patch 1.
I had taken this into my cxl staging tree thinking it only needed a few minor tweaks. Then whilst testing I realized it didn't work with hotplug and that made me consider if Arpit's original approach made sense. So this is my proposal on how to take this forwards. Note that this is not ready for merge (probably) just yet as I'd like Arpit to take a look at the changes. I went through a version that had major refactors to push the logic of building the records to the ports themselves, but in the end don't think that brings enough benefits for the complexity. Based on: https://lore.kernel.org/qemu-devel/[email protected]/T/#t (with all it's precursors, though hopefully they will all be in Michael's next pull request) I'll note this is an enormous foot gun as it lets you effectively trigger surprise device resets. However, it's only accessible if you are using a configuration with the fabric management parts of the CXL emulation. For now that is a switch-cci (MCTP over USB support is near the top of my list of things to get ready for upstram. I tested this with a custom driver and if nothing else it exposed some gaps in Linux's ability to rescan PCI buses after a perst event (even with drivers unbound etc to make it safe). Original cover letter: This patch series refactor existing support for Identify Switch Device and Get Physical Port State by utilizing physical ports (USP & DSP) information stored during enumeration. Additionally, it introduces new support for Physical Port Control of Physical Switch Command Set as per CXL spec r3.2 Section 7.6.7.1.3. It primarily constitutes two logic: -Assert-Deassert PERST: Assert PERST involves physical port to be in hold reset phase for minimum 100ms. No other physical port control request are entertained until Deassert PERST command for the given port is issued. -Reset PPB: cold reset of physical port (completing enter->hold->exit phases). Tested using libcxl-mi interface[1]: All active ports and all opcodes per active port is tested. Also, tested against possible edge cases manually since the interface currently dosen't support run time input. Typical Qemu topology (1 USP + 3 DSP's in a switch with 2 CXLType3 devices connected to the 2 DSP's): FM="-object memory-backend-file,id=cxl-mem1,mem-path=$TMP_DIR/t3_cxl1.raw,size=256M \ -object memory-backend-file,id=cxl-lsa1,mem-path=$TMP_DIR/t3_lsa1.raw,size=1M \ -object memory-backend-file,id=cxl-mem2,mem-path=$TMP_DIR/t3_cxl2.raw,size=512M \ -object memory-backend-file,id=cxl-lsa2,mem-path=$TMP_DIR/t3_lsa2.raw,size=512M \ -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \ -device cxl-rp,port=0,bus=cxl.1,id=cxl_rp_port0,chassis=0,slot=2 \ -device cxl-upstream,port=2,sn=1234,bus=cxl_rp_port0,id=us0,addr=0.0,multifunction=on, \ -device cxl-switch-mailbox-cci,bus=cxl_rp_port0,addr=0.1,target=us0 \ -device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \ -device cxl-downstream,port=1,bus=us0,id=swport1,chassis=0,slot=5 \ -device cxl-downstream,port=3,bus=us0,id=swport2,chassis=0,slot=6 \ -device cxl-type3,bus=swport0,memdev=cxl-mem1,id=cxl-pmem1,lsa=cxl-lsa1,sn=3 \ -device cxl-type3,bus=swport2,memdev=cxl-mem2,id=cxl-pmem2,lsa=cxl-lsa2,sn=4 \ -machine cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=1k \ -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=4,target=us0 \ -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=5,target=cxl-pmem1 \ -device i2c_mctp_cxl,bus=aspeed.i2c.bus.0,address=6,target=cxl-pmem2 \ -device virtio-rng-pci,bus=swport1" Tested multiple Qemu topologies: -without any devices connected to downstream ports. -with virtio-rng-pci devices connected to downstream ports. -with CXLType3 devices connected to downstream ports. -with different unique values of ports (both upstream and downstream). Changes from v3 (https://lore.kernel.org/qemu-devel/[email protected]/T/): -Namespaced the defines with cleaner prefix for Get Physical Port State Port Information Block members. -switch CCI implementation instead of switch FM interface as per Jonathan's review comments, hence moved perst members initializations from: cxl_initialize_usp_mctpcci() -> cxl_initialize_mailbox_swcci(). [1] https://github.com/computexpresslink/libcxlmi/commit/35fe68bd9a31469f832a87694d7b18d2d50be5b8 Arpit Kumar (2): hw/cxl: Physical Port Info FMAPI - update to current spec and add defines. hw/cxl: Add Physical Port Control FMAPI Command (Opcode 5102h) Jonathan Cameron (1): hw/cxl: Get Physical Port State - update for PCIe flit mode include/hw/cxl/cxl_port.h | 73 ++++++++ include/hw/pci-bridge/cxl_downstream_port.h | 12 ++ include/hw/pci-bridge/cxl_upstream_port.h | 2 + hw/cxl/cxl-mailbox-utils.c | 183 ++++++++++++++++++-- hw/pci-bridge/cxl_downstream.c | 9 + hw/pci-bridge/cxl_upstream.c | 1 + 6 files changed, 268 insertions(+), 12 deletions(-) create mode 100644 include/hw/cxl/cxl_port.h create mode 100644 include/hw/pci-bridge/cxl_downstream_port.h -- 2.51.0
