The solution suggested above was rejected on the kernel when I submitted the change where we expose the parent kernel abi https://lists.ubuntu.com/archives/kernel-team/2023-June/140090.html This led to a new proposal that would introduce a configure script used during build (or before) and it would automatically detect the signature changes in the linux header. This solution would remove the need for versioning. Since this change is a deviation from upstream, I sent it there but it was rejected. You can see the discussion here https://lore.kernel.org/lttng-dev/2c6e0b29-0e88-4dc7-6691-7a89a5428...@efficios.com/T/#t. They are not willing to change an approach that works most of the time for our particular use case because it would be difficult for them to maintain the new change, even though I volunteer to help them maintain the configure script...
I went back to our kernel mailing list and suggested to go back to my original proposal but it was still rejected. Discussion is here https://lists.ubuntu.com/archives/kernel-team/2023-July/140822.html Conclusion: Lttng-module upstream refused to accept a better proposal of dealing with changes in linux kernel headers and the solution where we expose a kernel abi is gonna add technical debt to our kernel and it's not worth doing it for a kernel that's gonna be EOL soon. And as for jammy and lunar kernels, they should be replaced soon by 6.2 kernels. I am not sure how I should try again convincing upstream, but I will at some point. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2017006 Title: fix build issue for v5.19 kernels (jammy-hwe, kinetic, lunar-riscv) Status in linux package in Ubuntu: Invalid Status in lttng-modules package in Ubuntu: Won't Fix Status in linux source package in Focal: Invalid Status in linux source package in Jammy: Invalid Status in lttng-modules source package in Jammy: Won't Fix Status in linux source package in Kinetic: Invalid Status in lttng-modules source package in Kinetic: Won't Fix Status in linux source package in Lunar: Invalid Status in lttng-modules source package in Lunar: Won't Fix Status in linux source package in Mantic: Invalid Bug description: SRU Justification [ Impact ] Upstream stable added a change in the format of jbd2 in 5.15.87 that was cherry-picked to kinetic:linux. This is incompatible with the current changes in the lttng-module kinetic. We previously encountered this issue for focal https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2004644 and we proactively released a new version for kinetic. The problem is the fix does not really work as expected. As shown in the logs below, The build error is triggered because some function declaration does not match the kernel's. In `include/instrumentation/events/jbd2.h` in lttng-modules, there is a conditional declaration based on the kernel version. 5.19 is missing there, therefore kinetic will fall back to the old declaration. The reason 5.19 is not there is probably because this version has reached end of life last year. Build logs: 35:35 DEBUG| [stdout] In file included from /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/lttng/define_trace.h:87, 860 02:35:35 DEBUG| [stdout] from /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/instrumentation/events/jbd2.h:293, 861 02:35:35 DEBUG| [stdout] from /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/lttng-probe-jbd2.c:29: 862 02:35:35 DEBUG| [stdout] /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/lttng/tracepoint-event-impl.h:133:6: error: conflicting types for ‘trace_jbd2_run_stats’; have ‘void(dev_t, long unsigned int, struct transaction_run_stats_s *)’ {aka ‘void(unsigned int, long unsigned int, struct transaction_run_stats_s *)’} 863 02:35:35 DEBUG| [stdout] 133 | void trace_##_name(_proto); 864 02:35:35 DEBUG| [stdout] | ^~~~~~ 865 02:35:35 DEBUG| [stdout] /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/lttng/tracepoint-event-impl.h:45:9: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP’ 866 02:35:35 DEBUG| [stdout] 45 | LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP(map, name, map, PARAMS(proto), PARAMS(args)) 867 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 868 02:35:35 DEBUG| [stdout] /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/lttng/tracepoint-event-impl.h:87:9: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT_MAP’ 869 02:35:35 DEBUG| [stdout] 87 | LTTNG_TRACEPOINT_EVENT_MAP(name, name, \ 870 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~~~~~~~~~~~~ 871 02:35:35 DEBUG| [stdout] /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/instrumentation/events/jbd2.h:180:1: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT’ 872 02:35:35 DEBUG| [stdout] 180 | LTTNG_TRACEPOINT_EVENT(jbd2_run_stats, 873 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~~~~~~~~ 874 02:35:35 DEBUG| [stdout] In file included from ./include/trace/events/jbd2.h:9, 875 02:35:35 DEBUG| [stdout] from /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/lttng-probe-jbd2.c:18: 876 02:35:35 DEBUG| [stdout] ./include/linux/tracepoint.h:245:28: note: previous definition of ‘trace_jbd2_run_stats’ with type ‘void(dev_t, tid_t, struct transaction_run_stats_s *)’ {aka ‘void(unsigned int, unsigned int, struct transaction_run_stats_s *)’} 877 02:35:35 DEBUG| [stdout] 245 | static inline void trace_##name(proto) \ 878 02:35:35 DEBUG| [stdout] | ^~~~~~ 879 02:35:35 DEBUG| [stdout] ./include/linux/tracepoint.h:419:9: note: in expansion of macro ‘__DECLARE_TRACE’ 880 02:35:35 DEBUG| [stdout] 419 | __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ 881 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~ 882 02:35:35 DEBUG| [stdout] ./include/linux/tracepoint.h:553:9: note: in expansion of macro ‘DECLARE_TRACE’ 883 02:35:35 DEBUG| [stdout] 553 | DECLARE_TRACE(name, PARAMS(proto), PARAMS(args)) 884 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~ 885 02:35:35 DEBUG| [stdout] ./include/trace/events/jbd2.h:248:1: note: in expansion of macro ‘TRACE_EVENT’ 886 02:35:35 DEBUG| [stdout] 248 | TRACE_EVENT(jbd2_run_stats, 887 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~ 888 02:35:35 DEBUG| [stdout] /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/lttng/tracepoint-event-impl.h:133:6: error: conflicting types for ‘trace_jbd2_checkpoint_stats’; have ‘void(dev_t, long unsigned int, struct transaction_chp_stats_s *)’ {aka ‘void(unsigned int, long unsigned int, struct transaction_chp_stats_s *)’} 889 02:35:35 DEBUG| [stdout] 133 | void trace_##_name(_proto); 890 02:35:35 DEBUG| [stdout] | ^~~~~~ 891 02:35:35 DEBUG| [stdout] /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/lttng/tracepoint-event-impl.h:45:9: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP’ 892 02:35:35 DEBUG| [stdout] 45 | LTTNG_TRACEPOINT_EVENT_INSTANCE_MAP(map, name, map, PARAMS(proto), PARAMS(args)) 893 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 894 02:35:35 DEBUG| [stdout] /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/lttng/tracepoint-event-impl.h:87:9: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT_MAP’ 895 02:35:35 DEBUG| [stdout] 87 | LTTNG_TRACEPOINT_EVENT_MAP(name, name, \ 896 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~~~~~~~~~~~~ 897 02:35:35 DEBUG| [stdout] /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/../../include/instrumentation/events/jbd2.h:200:1: note: in expansion of macro ‘LTTNG_TRACEPOINT_EVENT’ 898 02:35:35 DEBUG| [stdout] 200 | LTTNG_TRACEPOINT_EVENT(jbd2_checkpoint_stats, 899 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~~~~~~~~ 900 02:35:35 DEBUG| [stdout] ./include/linux/tracepoint.h:245:28: note: previous definition of ‘trace_jbd2_checkpoint_stats’ with type ‘void(dev_t, tid_t, struct transaction_chp_stats_s *)’ {aka ‘void(unsigned int, unsigned int, struct transaction_chp_stats_s *)’} 901 02:35:35 DEBUG| [stdout] 245 | static inline void trace_##name(proto) \ 902 02:35:35 DEBUG| [stdout] | ^~~~~~ 903 02:35:35 DEBUG| [stdout] ./include/linux/tracepoint.h:419:9: note: in expansion of macro ‘__DECLARE_TRACE’ 904 02:35:35 DEBUG| [stdout] 419 | __DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \ 905 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~~~ 906 02:35:35 DEBUG| [stdout] ./include/linux/tracepoint.h:553:9: note: in expansion of macro ‘DECLARE_TRACE’ 907 02:35:35 DEBUG| [stdout] 553 | DECLARE_TRACE(name, PARAMS(proto), PARAMS(args)) 908 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~~~ 909 02:35:35 DEBUG| [stdout] ./include/trace/events/jbd2.h:296:1: note: in expansion of macro ‘TRACE_EVENT’ 910 02:35:35 DEBUG| [stdout] 296 | TRACE_EVENT(jbd2_checkpoint_stats, 911 02:35:35 DEBUG| [stdout] | ^~~~~~~~~~~ 912 02:35:35 DEBUG| [stdout] make[3]: *** [scripts/Makefile.build:257: /var/lib/dkms/lttng-modules/2.13.8/build/src/probes/lttng-probe-jbd2.o] Error 1 913 02:35:35 DEBUG| [stdout] make[2]: *** [scripts/Makefile.build:474: /var/lib/dkms/lttng-modules/2.13.8/build/src/probes] Error 2 914 02:35:35 DEBUG| [stdout] make[1]: *** [scripts/Makefile.build:474: /var/lib/dkms/lttng-modules/2.13.8/build/src] Error 2 915 02:35:35 DEBUG| [stdout] make: *** [Makefile:1850: /var/lib/dkms/lttng-modules/2.13.8/build] Error 2 916 02:35:35 DEBUG| [stdout] make: Leaving directory '/usr/src/linux-headers-5.19.0-42-generic' 917 02:35:35 ERROR| Exception escaping from test: 918 Traceback (most recent call last): 919 File "/home/ubuntu/autotest/client/shared/test.py", line 378, in _exec 920 _cherry_pick_call(self.initialize, *args, **dargs) 921 File "/home/ubuntu/autotest/client/shared/test.py", line 738, in _cherry_pick_call 922 return func(*p_args, **p_dargs) 923 File "/home/ubuntu/autotest/client/tests/ubuntu_lttng_smoke_test/ubuntu_lttng_smoke_test.py", line 58, in initialize 924 raise error.TestError('DKMS failed to install') [ Fix ] Initially, we tried adding a new interval [5.19.0 - 5.19.19]. But because 5.19.0 is not maintained upstream and there are no stable fixes from there, the actual upstream version has remained unchanged since last year. It is currently 5.19.17. Therefore we can use this version to make a distinction between kernels that have this commit and the ones that do not have it and it won't be compatible with older versions. We expose the abi version of the kernel, but it alone will not solve this issue because we deliver many derivatives, each with its own version. Hence a base main abi version needs to be exposed in the headers. Therefore this requires 2 steps: 1. Expose UTS_UBUNTU_BASE_RELEASE_ABI variable equal with the parent abi in <generated/utsrelease.h>. For backports with multiple inheritance (like jammy:linux-aws-5.19) the version from debian.master (kinetic:linux) will be used. In case there is no inheritance, the actual kernel abi will be used. This is needed for jammy, kinetic and lunar but it may be needed in the future for similar use cases, so it will be pushed to all releases. 2. Use this variable in the lttng-module to construct LTTNG_UBUNTU_BASE_VERSION_CODE and then use it for versioning: LTTNG_UBUNTU_BASE_VERSION_CODE >= LTTNG_UBUNTU_KERNEL_VERSION(5,19,0,42) The lttng-module fix will be done only for lunar, kinetic, and jammy. Attached there is a patch with the proposed solution, debdiffs for kinetic, lunar and jammy will be attached separately. This requires a new release for kinetic and jammy because jammy 5.19 backports are affected as well. Also lunar, because some riscv kernels are still on 5.19 as well. [ Where problems would occur ] We may notice new failures in ubuntu_lttng_smoke_test once this compiles and run. But it was tested locally and the results were good, so the probability is very very low. Hopefully this time it is fully backward compatible. [ Test Plan ] Since we hit backward issues in the past, multiple kernel versions were used to make sure this is the proper fix. General test instructions: 1. Install kinetic (disired version) 2. sudo apt install lttng-modules-dkms 3. Depending on the version it may work or not 4. Additionally lttng-modules regression test should be triggered to confirm everything works file. Steps to run rt: # Install packages for autotest sudo apt-get install git python2 gdb python-yaml -y || sudo apt-get install git python gdb python-yaml -y # Enable deb-src sudo sed -i 's/^# deb-src/deb-src/' /etc/apt/sources.list sudo apt-get update git clone --depth=1 https://git.launchpad.net/~canonical-kernel-team/+git/autotest git clone --depth=1 https://git.launchpad.net/~canonical-kernel-team/+git/autotest-client-tests rm -fr autotest/client/tests ln -sf ~/autotest-client-tests autotest/client/tests AUTOTEST_PATH=/home/ubuntu/autotest sudo -E autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_lttng_smoke_test/control My testing: A. I built a kinetic kernel where I expose UTS_UBUNTU_BASE_RELEASE_ABI Results: ubuntu@kinetic:~/lttng-modules$ dpkg -l | grep lttng-modules ii lttng-modules-dkms 2.13.8-1~ubuntu22.10.2 all Linux Trace Toolkit (LTTng) kernel modules (DKMS) RT results: 09:32:05 INFO | Found 325223 dd and 390615 context switches 09:32:05 INFO | PASSED (simple system call tracing with babeltrace) 09:32:05 INFO | 09:32:05 INFO | Summary: 7 passed, 0 failed B. The break commit was introduced in 5.19.0.42. Versions between 42 and 45(hopefully this will land in the next release) will break. C. Tested with a kernel version that does not have the break commit: 5.19.0-41 to see if it's backwards compatible. ubuntu@kinetic:~$ uname -a Linux kinetic 5.19.0-41-generic #42-Ubuntu SMP PREEMPT_DYNAMIC Mon Apr 17 19:17:03 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ubuntu@kinetic:~/lttng-modules$ dpkg -l | grep lttng-modules ii lttng-modules-dkms 2.13.8-1~ubuntu22.10.2 all Linux Trace Toolkit (LTTng) kernel modules (DKMS) D. Tested with jammy-aws-5.19 that has the commit with the fix: ubuntu@jammmy-lttng:~/lttng$ uname -a Linux jammmy-lttng 5.19.0-1027-aws #28~22.04.1 SMP Mon Jun 5 08:27:20 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ~ Build is fine ubuntu@jammmy-lttng:~/lttng$ sudo dpkg -l | grep lttng ii lttng-modules-dkms 2.13.8-1~ubuntu22.04.2 all Linux Trace Toolkit (LTTng) kernel modules (DKMS) E. Tested with linux-aws-5.19.0-1024 to see backwards compatibility ubuntu@jammmy-lttng:~/lttng$ sudo dpkg -l | grep lttng ii lttng-modules-dkms 2.13.8-1~ubuntu22.04.2 all Linux Trace Toolkit (LTTng) kernel modules (DKMS) ubuntu@jammmy-lttng:~/lttng$ uname -a Linux jammmy-lttng 5.19.0-1024-aws #25~22.04.1-Ubuntu SMP Tue Apr 18 23:41:58 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2017006/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
