** Description changed: - We're unable to run Turbostat on Turin 2P Volcano systems with core - count greater than 400 on Ubuntu 22.04. On Ubuntu 24.04 turbostat works - fine. + BugLink: https://bugs.launchpad.net/bugs/2069961 - The following commit fixes the turbostat problem : + [Impact] - https://git.kernel.org/pub/scm/linux/kernel/git/lenb/linux.git/commit/?h=turbostat&id=3ac1d14d0583a2de75d49a5234d767e2590384dd + On large systems, e.g. with 512 cpus or more, turbostat fails to run due + to exceeding the rlimit for number of files. 512 cpus requires 1028 file + descriptors, but the current limit is 999. + + $ lscpu + ... + CPU(s): 512 + On-line CPU(s) list: 0-511 + ... + + $ sudo turbostat + ... + turbostat: /sys/devices/system/cpu/cpu477/cpuidle/state0/usage: open failed: Too many open files + + There is no workaround, apart from maybe using powerstat instead. + + [Fix] + + The fix is to increase the rlimit to increase the amount of file + descriptors that turbostat can open to 2^15, which should be plenty for + some time to come. + + commit 3ac1d14d0583a2de75d49a5234d767e2590384dd + Author: Wyes Karny <wyes.ka...@amd.com> + Date: Tue Oct 3 05:07:51 2023 +0000 + Subject: tools/power turbostat: Increase the limit for fd opened + Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3ac1d14d0583a2de75d49a5234d767e2590384dd + + This landed in 6.9-rc2, and is a clean cherry pick to jammy and noble. + + [Testcase] + + Deploy a bare metal system with 512 or more cpus. + + Install linux-tools: + + $ sudo apt install linux-tools-$(uname -r) + + Run turbostat: + + $ sudo turbostat + ... + turbostat: /sys/devices/system/cpu/cpu477/cpuidle/state0/usage: open failed: Too many open files + + There are test kernels available in the following ppa: + + https://launchpad.net/~mruffell/+archive/ubuntu/sf388491-test + + If you install them, you should be able to see normal turbostat output + for all cpus installed in the system. + + [Where problems can occur] + + We are simply increasing the rlimit for file descriptors that turbostat + can open. This should have no impact on any existing systems. + + If a regression should occur, then turbostat functionality might not + work. Users could use powerstat instead as a workaround while things are + fixed.
** Summary changed: - turbostat and powerstat not working on 22.04 + turbostat fails with too many open files on large systems ** Description changed: BugLink: https://bugs.launchpad.net/bugs/2069961 [Impact] On large systems, e.g. with 512 cpus or more, turbostat fails to run due to exceeding the rlimit for number of files. 512 cpus requires 1028 file descriptors, but the current limit is 999. $ lscpu ... CPU(s): 512 - On-line CPU(s) list: 0-511 + On-line CPU(s) list: 0-511 ... $ sudo turbostat ... turbostat: /sys/devices/system/cpu/cpu477/cpuidle/state0/usage: open failed: Too many open files There is no workaround, apart from maybe using powerstat instead. [Fix] The fix is to increase the rlimit to increase the amount of file descriptors that turbostat can open to 2^15, which should be plenty for some time to come. commit 3ac1d14d0583a2de75d49a5234d767e2590384dd Author: Wyes Karny <wyes.ka...@amd.com> Date: Tue Oct 3 05:07:51 2023 +0000 Subject: tools/power turbostat: Increase the limit for fd opened Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3ac1d14d0583a2de75d49a5234d767e2590384dd - This landed in 6.9-rc2, and is a clean cherry pick to jammy and noble. + This landed in 6.9-rc4, and is a clean cherry pick to jammy and noble. [Testcase] Deploy a bare metal system with 512 or more cpus. Install linux-tools: $ sudo apt install linux-tools-$(uname -r) Run turbostat: $ sudo turbostat ... turbostat: /sys/devices/system/cpu/cpu477/cpuidle/state0/usage: open failed: Too many open files There are test kernels available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf388491-test If you install them, you should be able to see normal turbostat output for all cpus installed in the system. [Where problems can occur] We are simply increasing the rlimit for file descriptors that turbostat can open. This should have no impact on any existing systems. If a regression should occur, then turbostat functionality might not work. Users could use powerstat instead as a workaround while things are fixed. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2069961 Title: turbostat fails with too many open files on large systems Status in linux package in Ubuntu: Fix Released Status in linux source package in Jammy: In Progress Status in linux source package in Mantic: Won't Fix Status in linux source package in Noble: In Progress Bug description: BugLink: https://bugs.launchpad.net/bugs/2069961 [Impact] On large systems, e.g. with 512 cpus or more, turbostat fails to run due to exceeding the rlimit for number of files. 512 cpus requires 1028 file descriptors, but the current limit is 999. $ lscpu ... CPU(s): 512 On-line CPU(s) list: 0-511 ... $ sudo turbostat ... turbostat: /sys/devices/system/cpu/cpu477/cpuidle/state0/usage: open failed: Too many open files There is no workaround, apart from maybe using powerstat instead. [Fix] The fix is to increase the rlimit to increase the amount of file descriptors that turbostat can open to 2^15, which should be plenty for some time to come. commit 3ac1d14d0583a2de75d49a5234d767e2590384dd Author: Wyes Karny <wyes.ka...@amd.com> Date: Tue Oct 3 05:07:51 2023 +0000 Subject: tools/power turbostat: Increase the limit for fd opened Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3ac1d14d0583a2de75d49a5234d767e2590384dd This landed in 6.9-rc4, and is a clean cherry pick to jammy and noble. [Testcase] Deploy a bare metal system with 512 or more cpus. Install linux-tools: $ sudo apt install linux-tools-$(uname -r) Run turbostat: $ sudo turbostat ... turbostat: /sys/devices/system/cpu/cpu477/cpuidle/state0/usage: open failed: Too many open files There are test kernels available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf388491-test If you install them, you should be able to see normal turbostat output for all cpus installed in the system. [Where problems can occur] We are simply increasing the rlimit for file descriptors that turbostat can open. This should have no impact on any existing systems. If a regression should occur, then turbostat functionality might not work. Users could use powerstat instead as a workaround while things are fixed. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2069961/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
