On 07/27/2018 11:19 AM, Cédric Le Goater wrote:
> On 07/27/2018 10:43 AM, Benjamin Herrenschmidt wrote:
>> On Fri, 2018-07-27 at 10:25 +0200, Cédric Le Goater wrote:
>>> Each PHB creates a pci-bridge device and the PCI bus that comes with it.
>>> It makes things easier to define PCI devices.
>>>
>>> It is still quite complex ... Here is a sample :
>>>
>>> qemu-system-ppc64 -m 2G -machine powernv \
>>> -cpu POWER8 -smp 2,cores=2,threads=1 -accel tcg,thread=multi \
>>> -kernel ./zImage.epapr -initrd ./rootfs.cpio.xz -bios ./skiboot.lid \
>>> \
>>> -device megasas,id=scsi0,bus=pci.0,addr=0x1 \
>>> -drive
>>> file=./rhel7-ppc64le.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none
>>> \
>>> -device
>>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2
>>> \
>>> \
>>> -device ich9-ahci,id=sata0,bus=pci.1,addr=0x1 \
>>> -drive
>>> file=./ubuntu-ppc64le.qcow2,if=none,id=drive0,format=qcow2,cache=none \
>>> -device ide-hd,bus=sata0.0,unit=0,drive=drive0,id=ide,bootindex=1 \
>>> -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.1,addr=0x2 \
>>> -netdev bridge,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=net0 \
>>> -device nec-usb-xhci,bus=pci.1,addr=0x7 \
>>
>> I don't understand why. That means you can't put emulated (or real)
>> PCIe device below it ?
>
> Well, skiboot does seem to find them. But that's not a good reason.
> I will dig in.
Nothing is wrong in skiboot, that was my misunderstanding of the overall
device layout.
Each PHB3 device exposes a single PCIE root port on which a *single* device
can be plugged. It can be a any adapter or a bridge/switch. The QEMU device
hierarchy is rather deep if you want to have multiple devices on a single
PHB3 :
dev: pnv-phb3
bus: phb3-root-bus
dev: phb3-root-port
bus: pcie
dev: pcie-bridge (only one device allowed)
bus: pcie
dev: e1000e
dev: ...
So it makes the command line rather heavy :/ or you need to have
more phb3s to define an adapter on each. How about 3 per chip ?
Here is a quick survey,
On OpenPower systems:
palmetto: 3 PHBs - one 16x, one 8x, one 4x backed by a switch
with the BMC devices
habanero: 3 PHBs - one 16x, one 8x, one 8x backed by a switch
with the BMC devices
firestone: socket 0 : 2 16x PHBs
socket 1 : 3 PHBs - one 16x, one 8x, one 8x backed
by a switch with the BMC devices
garrison: socket 0 : 1 16x PHB, 2 8x PHB linked to the GPUs, 1 8x PHB,
socket 1 : 1 16x PHB, 2 8x PHB linked to the GPUs, 1 8x PHB
backed by a switch with the BMC devices
On IBM systems :
tuletas: 2 16x PHB, 2 8x PHB, backed with a switch.
Thanks,
C.
