Thank you. I guess the interesting question here is whether current SCHED_OTHER behavior makes sense for majority of usecases (or any, really).
I think current situation makes isolcpus unusable in at least one common case - when user want to isolate some critical multithreaded application e.g. benchmark. Usually it's not desirable to add some custom scheduling logic to such program (e.g. source code may not even be available). -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2116749 Title: Thread affinity broken under isolcpus Status in linux package in Ubuntu: Invalid Bug description: I'd like to report a bug with unexpected scheduling of programs run with `taskset` when target cores have been reserved via `isolcpus` setting (e.g. for benchmark stability purposes). When I try to run multi-threaded program on reserved cores: ``` $ taskset -c 1-3 ./a.out ``` OS will pin all of them to single core for the whole duration of lifetime. I would instead expect them to be distributed evenly across allowed cores (1-3 in this case). Threads are distributed correctly if program is run under `chrt -f 1`. The attached small test program illustrates issues with isolcpus and taskset (see below). Results were obtained for Ubuntu 24.04 running on 4-core VirtualBox VM (I've also reproduced them on real PC with Ubuntu 22.04). Bug has also been reported previously (by another user) in https://serverfault.com/questions/573025/taskset-not-working-over-a- range-of-cores-in-isolcpus Isolcpus is very useful for obtaining stable CPU benchmark results and effective lack of support for multi-threaded apps is unfortunate. Everything works as expected when isolcpus is not set: 1) without `taskset` threads run on all cores: ``` yugr@yugr-VirtualBox:~/tasks/isolcpus$ gcc -O2 test.c -pthread yugr@yugr-VirtualBox:~/tasks/isolcpus$ ./a.out Thread 7 was scheduled on 1 2 Thread 5 was scheduled on 2 3 Thread 11 was scheduled on 0 2 3 Thread 1 was scheduled on 1 2 3 Thread 2 was scheduled on 0 1 3 Thread 8 was scheduled on 0 1 Thread 6 was scheduled on 0 1 2 Thread 9 was scheduled on 0 1 2 3 Thread 4 was scheduled on 1 2 3 Thread 10 was scheduled on 0 2 3 Thread 0 was scheduled on 0 1 3 Thread 3 was scheduled on 0 1 2 3 ``` 2) with taskset threads run according to the mask: ``` yugr@yugr-VirtualBox:~/tasks/isolcpus$ taskset -c 2-3 ./a.out Thread 3 was scheduled on 2 Thread 4 was scheduled on 2 Thread 0 was scheduled on 2 Thread 5 was scheduled on 2 3 Thread 2 was scheduled on 2 3 Thread 10 was scheduled on 2 3 Thread 11 was scheduled on 2 3 Thread 1 was scheduled on 2 3 Thread 9 was scheduled on 2 3 Thread 8 was scheduled on 2 3 Thread 6 was scheduled on 3 Thread 7 was scheduled on 2 3 ``` And here are results after adding `isolcpus=1-3` to `GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub` and doing `sudo update-grub`. 1) without taskset results are expected (everything goes to core 0): ``` yugr@yugr-VirtualBox:~/tasks/isolcpus$ ./a.out Thread 11 was scheduled on 0 Thread 3 was scheduled on 0 Thread 0 was scheduled on 0 Thread 1 was scheduled on 0 Thread 6 was scheduled on 0 Thread 8 was scheduled on 0 Thread 4 was scheduled on 0 Thread 7 was scheduled on 0 Thread 5 was scheduled on 0 Thread 10 was scheduled on Thread 9 was scheduled on 0 Thread 2 was scheduled on 0 0 ``` 2) but with taskset for some reason everything goes to single core (rather than three, 1-3) as well: ``` yugr@yugr-VirtualBox:~/tasks/isolcpus$ taskset -c 1-3 ./a.out Thread 3 was scheduled on 3 Thread 2 was scheduled on 3 Thread 0 was scheduled on 3 Thread 9 was scheduled on 3 Thread 8 was scheduled on 3 Thread 1 was scheduled on Thread 6 was scheduled on Thread 5 was scheduled on 3 Thread 4 was scheduled on 3 3 3 Thread 11 was scheduled on Thread 7 was scheduled on Thread 10 was scheduled on 3 3 3 ``` Everything fixed with `SCHED_FIFO` policy: ``` yugr@yugr-VirtualBox:~/tasks/isolcpus$ chrt -f 1 taskset -c 1-3 ./a.out Thread 2 was scheduled on 1 Thread 1 was scheduled on 2 Thread 0 was scheduled on 3 Thread 5 was scheduled on 3 Thread 4 was scheduled on 2 Thread 3 was scheduled on 1 Thread 7 was scheduled on 2 Thread 6 was scheduled on 3 Thread 8 was scheduled on 1 Thread 9 was scheduled on 2 Thread 10 was scheduled on 3 Thread 11 was scheduled on 1 ``` To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2116749/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : [email protected] Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
