Problem solved. I have changed the QPI Snoop Mode on these servers from ClusterOnDIe Enabled to Disabled and they display what I take to be correct behaviour - ie
[root@comp006 ~]# numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 node 0 size: 32673 MB node 0 free: 31541 MB node 1 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 node 1 size: 32768 MB node 1 free: 31860 MB node distances: node 0 1 0: 10 21 1: 21 10 On 16 December 2016 at 14:51, John Hearns <hear...@googlemail.com> wrote: > hwloc is finding weirdness also. > I am going to find I have done something stupid, right? > > > > [johnh@comp006 ~]$ lstopo > ************************************************************ > **************** > * hwloc 1.11.3 has encountered what looks like an error from the operating > system. > * > * Package (P#1 cpuset 0x00fff000) intersects with NUMANode (P#0 cpuset > 0x00fc0fff) without inclusion! > * Error occurred in topology.c line 1046 > * > * The following FAQ entry in the hwloc documentation may help: > * What should I do when hwloc reports "operating system" warnings? > * Otherwise please report this error message to the hwloc user's mailing > list, > * along with the output+tarball generated by the hwloc-gather-topology > script. > ************************************************************ > **************** > Machine (64GB total) > NUMANode L#0 (P#0 32GB) > Package L#0 > L3 L#0 (15MB) > L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU > L#0 (P#0) > L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU > L#1 (P#1) > L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU > L#2 (P#2) > L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU > L#3 (P#3) > L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU > L#4 (P#4) > L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU > L#5 (P#5) > L3 L#1 (15MB) > L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU > L#6 (P#6) > L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU > L#7 (P#7) > L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU > L#8 (P#8) > L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU > L#9 (P#9) > L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + > PU L#10 (P#10) > L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + > PU L#11 (P#11) > L3 L#2 (15MB) > L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU > L#12 (P#18) > L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU > L#13 (P#19) > L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU > L#14 (P#20) > L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU > L#15 (P#21) > L2 L#16 (256KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16 + PU > L#16 (P#22) > L2 L#17 (256KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17 + PU > L#17 (P#23) > HostBridge L#0 > PCIBridge > PCI 8086:24f0 > PCI 8086:8d62 > PCIBridge > PCI 102b:0522 > GPU L#0 "card0" > GPU L#1 "controlD64" > PCIBridge > PCI 8086:1521 > Net L#2 "enp7s0f0" > PCI 8086:1521 > Net L#3 "enp7s0f1" > PCI 8086:8d02 > Block(Disk) L#4 "sda" > NUMANode L#1 (P#2 32GB) + L3 L#3 (15MB) > L2 L#18 (256KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18 + PU > L#18 (P#12) > L2 L#19 (256KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19 + PU > L#19 (P#13) > L2 L#20 (256KB) + L1d L#20 (32KB) + L1i L#20 (32KB) + Core L#20 + PU > L#20 (P#14) > L2 L#21 (256KB) + L1d L#21 (32KB) + L1i L#21 (32KB) + Core L#21 + PU > L#21 (P#15) > L2 L#22 (256KB) + L1d L#22 (32KB) + L1i L#22 (32KB) + Core L#22 + PU > L#22 (P#16) > L2 L#23 (256KB) + L1d L#23 (32KB) + L1i L#23 (32KB) + Core L#23 + PU > L#23 (P#17) > > > On 16 December 2016 at 14:36, John Hearns <hear...@googlemail.com> wrote: > >> This is in the context of Ominpath cards and the hfi1 driver. >> In the file pio.c there is a check on the NUMA zones being online >> >> >> >> * num_numa = num_online_nodes >> <http://lxr.free-electrons.com/ident?v=4.4;i=num_online_nodes>();* >> >> *1711* >> <http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1711>* >> /* enforce the expectation that the numas are compact */* >> >> *1712* >> <http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1712>* >> for (i <http://lxr.free-electrons.com/ident?v=4.4;i=i> = 0; i >> <http://lxr.free-electrons.com/ident?v=4.4;i=i> < num_numa; i >> <http://lxr.free-electrons.com/ident?v=4.4;i=i>++) {* >> >> *1713* >> <http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1713>* >> if (!node_online >> <http://lxr.free-electrons.com/ident?v=4.4;i=node_online>(i >> <http://lxr.free-electrons.com/ident?v=4.4;i=i>)) {* >> >> *1714* >> <http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1714>* >> dd_dev_err <http://lxr.free-electrons.com/ident?v=4.4;i=dd_dev_err>(dd >> <http://lxr.free-electrons.com/ident?v=4.4;i=dd>, "NUMA nodes are not >> compact\n");* >> >> *1715* >> <http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1715>* >> ret <http://lxr.free-electrons.com/ident?v=4.4;i=ret> = -EINVAL >> <http://lxr.free-electrons.com/ident?v=4.4;i=EINVAL>;* >> >> *1716* >> <http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1716>* >> goto done <http://lxr.free-electrons.com/ident?v=4.4;i=done>;* >> >> *1717* >> <http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1717>* >> }* >> >> *1718* >> <http://lxr.free-electrons.com/source/drivers/staging/rdma/hfi1/pio.c?v=4.4#L1718>* >> }* >> >> >> >> >> On some servers I have I see this weirdness with the NUMA zones: >> >> (2650-v4 processors, HT is off) >> >> [root@comp006 ~]# numactl --hardware >> >> available: 2 nodes (0,2) >> >> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 18 19 20 21 22 23 >> >> node 0 size: 32673 MB >> >> node 0 free: 29840 MB >> >> node 2 cpus: 12 13 14 15 16 17 >> >> node 2 size: 32768 MB >> >> node 2 free: 31753 MB >> >> node distances: >> >> node 0 2 >> >> 0: 10 20 >> >> 2: 20 10 >> >> >> >> Someone will be along in a minute to explain why. >> >> I am sure this is a BISO Setting, but which oen is not makign itself >> clear to me. >> >> >> >> >> >> >> >
_______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
