** Description changed: BugLink: https://bugs.launchpad.net/bugs/1882478 [Impact] Intel Skylake server processors and onward have a different Last Level Cache (LLC) topology than earlier processors, and such processors have a new feature called Sub-NUMA-Clustering (SNC) which is similar to the existing Cluster-On-Die (CoD) feature earlier server processors has. Sub-NUMA-Clustering divides the system into two "slices", each of which are allocated half the CPU cores, half the Last Level Cache and one memory controller. Each slice is enumerated as a NUMA node. The difference between Sub-NUMA-Clustering and Cluster-On-Die is how the Last Level Cache is exposed to each NUMA node. CoD had the same cache line present in each half of the LLC. In SNC, each cache line is only present in its respective slice. Because of this, the semantics around accessing LLC changes, with a process accessing NUMA-local memory only seeing half the LLC capacity. On systems with Sub-NUMA-Clustering enabled, on the Xenial 4.4 and Bionic 4.15 kernels we see the following oops during NUMA node enumeration: .... node #0, CPUs: #1 #2 #3 #4 #5 #6 .... node #1, CPUs: #7 sched: CPU #7's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency. WARNING: CPU: 7 PID: 0 at /build/linux-hwe-F5opqf/linux-hwe-4.15.0/arch/x86/kernel/smpboot.c:375 topology_sane.isra.4+0x6c/0x70 Modules linked in: CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-47-generic #50~16.04.1-Ubuntu Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/02/2018 RIP: 0010:topology_sane.isra.4+0x6c/0x70 Call Trace: set_cpu_sibling_map+0x153/0x540 start_secondary+0xb2/0x200 secondary_startup_64+0xa5/0xb0 #8 #9 #10 #11 #12 #13 .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20 .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27 smp: Brought up 2 nodes, 28 CPUs This was with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10. The oops happens because topology_sane() checks to see if the Last Level Cache line matches across different CPUs, which it no longer does. [Fix] The fix comes in the form of the following upstream commit, which landed in Linux 4.17: commit 1340ccfa9a9afefdbab90d7935d4ed19817e37c2 Author: Alison Schofield <alison.schofi...@intel.com> Date: Fri Apr 6 17:21:30 2018 -0700 Subject: x86,sched: Allow topologies where NUMA nodes share an LLC Link: https://github.com/torvalds/linux/commit/1340ccfa9a9afefdbab90d7935d4ed19817e37c2 The commit adds a check for this particular family of Intel processors, and if the CPU family matches, it simply skips the check to topology_sane(). The commit needs minor backports to Xenial 4.4 and Bionic 4.15, with the only remarks being re-arranging #includes and small context fixups. [Testcase] Unfortunately, this is hardware specific. To test this, you need a Intel Skylake server processor which supports Sub-NUMA-Clustering. We have a customer with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10 that has successfully tested the below test kernels, with good results. Xenial 4.4 ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-ga Xenial 4.15 HWE ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-hwe Running the test kernel, the oops does not reproduce: smp: Bringing up secondary CPUs ... x86: Booting SMP configuration: .... node #0, CPUs: #1 NMI watchdog: Enabled. Permanently consumes one hw-PMU counter. #2 #3 #4 #5 #6 .... node #1, CPUs: #7 #8 #9 #10 #11 #12 #13 .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20 .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27 smp: Brought up 2 nodes, 28 CPUs smpboot: Max logical packages: 1 smpboot: Total of 28 processors activated [Regression Potential] The commit modifies a small section of smpboot code, which every machine will execute on boot. The majority of the commit breaks up a large if statement into smaller blocks than it was previously, and adds an extra if statement to check for a specific processor family. If a regression were to occur, some machines would or would not make their calls to topology_sane(), which in the worst case, would result in a oops message and slightly degraded performance. The system would still function normally. The commit has been present since 4.17-rc2 and is present in Eoan and Focal. There are no fixup commits, and no additional processor families have been added since. Because of the small re-arrangement in logic, and the addition of a - processor family check, these changes a fairly minor, and I don't think - it will cause any regressions. + processor family check, these changes are fairly minor, and I don't + think it will cause any regressions.
-- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1882478 Title: smpboot: don't call topology_sane() when Sub-NUMA-Clustering is enabled Status in linux package in Ubuntu: Fix Released Status in linux source package in Xenial: In Progress Status in linux source package in Bionic: In Progress Bug description: BugLink: https://bugs.launchpad.net/bugs/1882478 [Impact] Intel Skylake server processors and onward have a different Last Level Cache (LLC) topology than earlier processors, and such processors have a new feature called Sub-NUMA-Clustering (SNC) which is similar to the existing Cluster-On-Die (CoD) feature earlier server processors has. Sub-NUMA-Clustering divides the system into two "slices", each of which are allocated half the CPU cores, half the Last Level Cache and one memory controller. Each slice is enumerated as a NUMA node. The difference between Sub-NUMA-Clustering and Cluster-On-Die is how the Last Level Cache is exposed to each NUMA node. CoD had the same cache line present in each half of the LLC. In SNC, each cache line is only present in its respective slice. Because of this, the semantics around accessing LLC changes, with a process accessing NUMA-local memory only seeing half the LLC capacity. On systems with Sub-NUMA-Clustering enabled, on the Xenial 4.4 and Bionic 4.15 kernels we see the following oops during NUMA node enumeration: .... node #0, CPUs: #1 #2 #3 #4 #5 #6 .... node #1, CPUs: #7 sched: CPU #7's llc-sibling CPU #0 is not on the same node! [node: 1 != 0]. Ignoring dependency. WARNING: CPU: 7 PID: 0 at /build/linux-hwe-F5opqf/linux-hwe-4.15.0/arch/x86/kernel/smpboot.c:375 topology_sane.isra.4+0x6c/0x70 Modules linked in: CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.15.0-47-generic #50~16.04.1-Ubuntu Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 10/02/2018 RIP: 0010:topology_sane.isra.4+0x6c/0x70 Call Trace: set_cpu_sibling_map+0x153/0x540 start_secondary+0xb2/0x200 secondary_startup_64+0xa5/0xb0 #8 #9 #10 #11 #12 #13 .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20 .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27 smp: Brought up 2 nodes, 28 CPUs This was with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10. The oops happens because topology_sane() checks to see if the Last Level Cache line matches across different CPUs, which it no longer does. [Fix] The fix comes in the form of the following upstream commit, which landed in Linux 4.17: commit 1340ccfa9a9afefdbab90d7935d4ed19817e37c2 Author: Alison Schofield <alison.schofi...@intel.com> Date: Fri Apr 6 17:21:30 2018 -0700 Subject: x86,sched: Allow topologies where NUMA nodes share an LLC Link: https://github.com/torvalds/linux/commit/1340ccfa9a9afefdbab90d7935d4ed19817e37c2 The commit adds a check for this particular family of Intel processors, and if the CPU family matches, it simply skips the check to topology_sane(). The commit needs minor backports to Xenial 4.4 and Bionic 4.15, with the only remarks being re-arranging #includes and small context fixups. [Testcase] Unfortunately, this is hardware specific. To test this, you need a Intel Skylake server processor which supports Sub-NUMA-Clustering. We have a customer with a Intel Xeon Gold 5120 CPU on a HP DL360 Gen10 that has successfully tested the below test kernels, with good results. Xenial 4.4 ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-ga Xenial 4.15 HWE ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf280048-test-hwe Running the test kernel, the oops does not reproduce: smp: Bringing up secondary CPUs ... x86: Booting SMP configuration: .... node #0, CPUs: #1 NMI watchdog: Enabled. Permanently consumes one hw-PMU counter. #2 #3 #4 #5 #6 .... node #1, CPUs: #7 #8 #9 #10 #11 #12 #13 .... node #0, CPUs: #14 #15 #16 #17 #18 #19 #20 .... node #1, CPUs: #21 #22 #23 #24 #25 #26 #27 smp: Brought up 2 nodes, 28 CPUs smpboot: Max logical packages: 1 smpboot: Total of 28 processors activated [Regression Potential] The commit modifies a small section of smpboot code, which every machine will execute on boot. The majority of the commit breaks up a large if statement into smaller blocks than it was previously, and adds an extra if statement to check for a specific processor family. If a regression were to occur, some machines would or would not make their calls to topology_sane(), which in the worst case, would result in a oops message and slightly degraded performance. The system would still function normally. The commit has been present since 4.17-rc2 and is present in Eoan and Focal. There are no fixup commits, and no additional processor families have been added since. Because of the small re-arrangement in logic, and the addition of a processor family check, these changes are fairly minor, and I don't think it will cause any regressions. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1882478/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
