Actually 12 per rack. The reasoning was that there were 2 connections
per host to top of rack switch leaving 12 uplinkks to two tier0 switches
at 6 each.
For the IB cards they are some special flavored Mellanox which attach to
the PCIv4 sockets, 8 lanes each. And since 8 lanes of v4 == 16 lanes of
v3, we get full EDR to both CPU sockets.
Bill
On 10/10/19 12:57 PM, Scott Atchley wrote:
That is better than 80% peak, nice.
Is it three racks of 15 nodes? Or two racks of 18 and 9 in the third rack?
You went with a single-port HCA per socket and not the shared, dual-port
HCA in the shared PCIe slot?
On Thu, Oct 10, 2019 at 8:48 AM Bill Wichser <b...@princeton.edu
<mailto:b...@princeton.edu>> wrote:
Thanks for the kind words. Yes, we installed more like a mini-Sierra
machine which is air cooled. There are 46 nodes of the IBM AC922, two
socket, 4 V100 where each socket uses the SMT threading x4. So two 16
core chips, 32/node, 128 threads per node. The GPUs all use NVLink.
There are two EDR connections per host, each tied to a CPU, 1:1 per
rack
of 12 and 2:1 between racks. We have a 2P scratch filesystem running
GPFS. Each node also has a 3T NVMe card as well for local scratch.
And we're running Slurm as our scheduler.
We'll see if it makes the top500 in November. It fits there today but
who knows what else got on there since June. With the help of
nVidia we
managed to get 1.09PF across 45 nodes.
Bill
On 10/10/19 7:45 AM, Michael Di Domenico wrote:
> for those that may not have seen
>
>
https://insidehpc.com/2019/10/traverse-supercomputer-to-accelerate-fusion-research-at-princeton/
>
> Bill Wischer and Prentice Bisbal are frequent contributors to the
> list, Congrats on the acquisition. Its nice to see more HPC
expansion
> in our otherwise barren hometown... :)
>
> Maybe one of them will pass along some detail on the machine...
> _______________________________________________
> Beowulf mailing list, Beowulf@beowulf.org
<mailto:Beowulf@beowulf.org> sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org
<mailto:Beowulf@beowulf.org> sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
_______________________________________________
Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
