On Thu, Apr 27, 2017 at 1:32 PM, Kilian Cavalotti <kilian.cavalotti.w...@gmail.com> wrote: > Yup, you can have as many PCIe devices in a single root-complex with > the appropriate number of PCIe switches.
Yes, understood. It used to be that there was some odd issue with most BIOS chips though, in that you couldn't actually have >8 video cards in a system. i don't recall the exact issue. i know supermicro has 10, but that might just be a "hey it worked" rather then the bios was specifically designed to handle a large (meaning 16+) number of GPU's > Will it create contentions? Certainly, because your CPU only have so > many (40 typically) PCIe lanes. Single PCI root-complex is good for > internal device-to-device communication (provided a sane PCIe > switching architecture), but transferring data from the host memory to > the PCIe devices (and vice-versa) will go through those 40-ish lanes, > and that could quickly become a serious bottleneck. agreed. fortunately, some of the apps my users run, make almost zero data transfers between the host memory and the gpu memory. the just want to spin the cores. so even a heavily oversubscribed PCI-e network wouldn't really harm us too much. that being said it has to at least work... oversubb'ed is one thing, congestion death is another _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
