Public bug reported: [Impact]
From Kernel 6.5 and above, the serial driver now claims multiple UART ports by default, when the previous behaviour was it would only claim 1. Due to this the ite-cir driver can no longer load because the port it would assign itself is now being used in memory by the 16550/8250 driver. We have found a workaround by disabling the serial driver from consuming more than 1 port. Below is the steps we went to troubleshoot, investigate and the solution we’ve found. This is a regression from 6.4 behaviour. When installing 24.04 I noticed that the /dev/lirc0 device was not loaded on the system which is what is created when the driver is loaded. After reviewing the driver we use on 20.04 I then searched the kernel logs and found that the ite-cir driver failed to load due to error -16. ``` 2024-04-19T13:44:51.873060+00:00 host kernel: rc rc0: ITE8708 CIR transceiver as /devices/pnp0/00:03/rc/rc0 2024-04-19T13:44:51.873062+00:00 host kernel: rc rc0: lirc_dev: driver ite-cir registered at minor = 0, raw IR receiver, raw IR transmitter 2024-04-19T13:44:51.873070+00:00 host kernel: input: ITE8708 CIR transceiver as /devices/pnp0/00:03/rc/rc0/input6 2024-04-19T13:44:51.873072+00:00 host kernel: i2c i2c-2: 2/2 memory slots populated (from DMI) 2024-04-19T13:44:51.873076+00:00 host kernel: i2c i2c-2: Successfully instantiated SPD at 0x50 2024-04-19T13:44:51.873078+00:00 host kernel: ite-cir: probe of 00:03 failed with error -16 ``` I reviewed kernel error message and this error is due to the device/resource is busy or in use. My colleague and I reviewed the ite- cir driver source code and noticed the driver initialisation succeeds up until L1404 https://github.com/torvalds/linux/blob/master/drivers/media/rc/ite- cir.c#L1404 where it attempts to request_region, if it fails to do this it will unregister the device. The request region allocates a place in memory for the device to use, we believe it’s using the ioport.h’s request region function https://github.com/torvalds/linux/blob/master/include/linux/ioport.h#L278 When we added some debug lines to ite-cir we noticed that the memory reference it is trying to allocate is 02f8-02ff. We checked the ioports on 24.04 and noticed that the serial driver was taking up this reference in memory when in 20.04 (The OS we currently use) and 23.04 it would not. Ubuntu 24.04 ioports ``` user@host:~$ sudo cat /proc/ioports 0000-0cf7 : PCI Bus 0000:00 02f8-02ff : serial 03f8-03ff : serial ``` Ubuntu 20.04 ioports memory allocation ``` user@host:~$ sudo cat /proc/ioports 0000-0cf7 : PCI Bus 0000:00 02f8-02ff : ite-cir 03f8-03ff : serial ``` When we investigated any changes that might cause this, we came with two potential commits that were released in 6.5 that might have caused this issue: * https://github.com/torvalds/linux/commit/9d86719f8769244dc99b8cb6091c41eae3fd684f * https://github.com/torvalds/linux/commit/84a9582fd203063cd4d301204971ff2cd8327f1a The serial changes allow 16550/8250 driver to consume as many serial ports as they require initializing the driver, originally they only used 1. Due to this we noticed that when we checked the serial ports in use by the device that 8250 was consuming the place in memory that ite-cir should. Ubuntu 24.04 ``` user@host:~$ sudo setserial -g /dev/ttyS* /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4 /dev/ttyS1, UART: 8250, Port: 0x02f8, IRQ: 3 ``` Ubuntu 20.04 ``` user@host:~$ sudo setserial -g /dev/ttyS* /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4 /dev/ttyS1, UART: unknown, Port: 0x02f8, IRQ: 3 ``` [Regression] We believe that the regression started from 6.5 kernel onwards based on the commit changes that were done to the serial driver that shipped to those kernels. 23.04 that uses 6.2 works as expected and shows no issues with using the serial port for IR while 23.10 that uses 6.5 and 24.04 that uses 6.8 does not load the ite-cir driver. [Reproducible steps] * Use Ubuntu 23.10 + or use Kernel 6.5+ * Use IR hardware that requires the ite-cir driver to be loaded * Attempt to load ite-cir driver on boot of OS * Check /var/log/kern.log for ite-cir driver failure [Fix] The requested solution would be to either: * Set the serial driver to no longer consume more than 1 serial port * Have the serial driver check what memory reference ite-cir driver needs to load, ignore that reference and then consume another point in memory instead. We have found a workaround where we set the /etc/default/grub config to make the 8250 use only 1 UART port, this allows ite-cir to consume the port in memory like it use to, enabling us to use the driver again /etc/default/grub ``` GRUB_CMDLINE_LINUX_DEFAULT="8250.nr_uarts=1" ``` The alternative fix is to have a startup script that will remove 8250 from using /dev/ttyS1 and then remove and re-add the ite-cir driver ``` #!/bin/bash setserial /dev/ttyS1 uart none modprobe -r ite-cir ``` [Diagnostic information] version ``` Linux version 6.8.0-22-generic (buildd@lcy02-amd64-070) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-23ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #22-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 4 22:30:32 UTC 2024 ``` lsb_release ``` No LSB modules are available. Description: Ubuntu Noble Numbat (development branch) Release: 24.04 ``` Kernel version ``` linux-image-6.8.0-22-generic: Installed: 6.8.0-22.22 Candidate: 6.8.0-22.22 Version table: *** 6.8.0-22.22 100 100 /var/lib/dpkg/status ``` ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: 6.5 6.8 driver ite-cir kernel-bug ** Attachment added: "lspci-vnvn.log" https://bugs.launchpad.net/bugs/2063322/+attachment/5769983/+files/lspci-vnvn.log ** Description changed: [Impact] From Kernel 6.5 and above, the serial driver now claims multiple UART ports by default, when the previous behaviour was it would only claim 1. Due to this the ite-cir driver can no longer load because the port it would assign itself is now being used in memory by the 16550/8250 driver. We have found a workaround by disabling the serial driver from consuming more than 1 port. Below is the steps we went to troubleshoot, investigate and the solution we’ve found. This is a regression from 6.4 behaviour. When installing 24.04 I noticed that the /dev/lirc0 device was not loaded on the system which is what is created when the driver is loaded. After reviewing the driver we use on 20.04 I then searched the kernel logs and found that the ite-cir driver failed to load due to error -16. ``` - 2024-04-19T13:44:51.873060+00:00 harness kernel: rc rc0: ITE8708 CIR transceiver as /devices/pnp0/00:03/rc/rc0 - 2024-04-19T13:44:51.873062+00:00 harness kernel: rc rc0: lirc_dev: driver ite-cir registered at minor = 0, raw IR receiver, raw IR transmitter - 2024-04-19T13:44:51.873070+00:00 harness kernel: input: ITE8708 CIR transceiver as /devices/pnp0/00:03/rc/rc0/input6 - 2024-04-19T13:44:51.873072+00:00 harness kernel: i2c i2c-2: 2/2 memory slots populated (from DMI) - 2024-04-19T13:44:51.873076+00:00 harness kernel: i2c i2c-2: Successfully instantiated SPD at 0x50 - 2024-04-19T13:44:51.873078+00:00 harness kernel: ite-cir: probe of 00:03 failed with error -16 + 2024-04-19T13:44:51.873060+00:00 host kernel: rc rc0: ITE8708 CIR transceiver as /devices/pnp0/00:03/rc/rc0 + 2024-04-19T13:44:51.873062+00:00 host kernel: rc rc0: lirc_dev: driver ite-cir registered at minor = 0, raw IR receiver, raw IR transmitter + 2024-04-19T13:44:51.873070+00:00 host kernel: input: ITE8708 CIR transceiver as /devices/pnp0/00:03/rc/rc0/input6 + 2024-04-19T13:44:51.873072+00:00 host kernel: i2c i2c-2: 2/2 memory slots populated (from DMI) + 2024-04-19T13:44:51.873076+00:00 host kernel: i2c i2c-2: Successfully instantiated SPD at 0x50 + 2024-04-19T13:44:51.873078+00:00 host kernel: ite-cir: probe of 00:03 failed with error -16 ``` I reviewed kernel error message and this error is due to the device/resource is busy or in use. My colleague and I reviewed the ite- cir driver source code and noticed the driver initialisation succeeds up until L1404 https://github.com/torvalds/linux/blob/master/drivers/media/rc/ite- cir.c#L1404 where it attempts to request_region, if it fails to do this it will unregister the device. The request region allocates a place in memory for the device to use, we believe it’s using the ioport.h’s request region function https://github.com/torvalds/linux/blob/master/include/linux/ioport.h#L278 When we added some debug lines to ite-cir we noticed that the memory reference it is trying to allocate is 02f8-02ff. We checked the ioports on 24.04 and noticed that the serial driver was taking up this reference in memory when in 20.04 (The OS we currently use) and 23.04 it would not. Ubuntu 24.04 ioports ``` - user@host:~$ sudo cat /proc/ioports + user@host:~$ sudo cat /proc/ioports 0000-0cf7 : PCI Bus 0000:00 - 02f8-02ff : serial - 03f8-03ff : serial + 02f8-02ff : serial + 03f8-03ff : serial ``` Ubuntu 20.04 ioports memory allocation ``` - user@host:~$ sudo cat /proc/ioports - 0000-0cf7 : PCI Bus 0000:00 - 02f8-02ff : ite-cir - 03f8-03ff : serial + user@host:~$ sudo cat /proc/ioports + 0000-0cf7 : PCI Bus 0000:00 + 02f8-02ff : ite-cir + 03f8-03ff : serial ``` When we investigated any changes that might cause this, we came with two potential commits that were released in 6.5 that might have caused this issue: * https://github.com/torvalds/linux/commit/9d86719f8769244dc99b8cb6091c41eae3fd684f * https://github.com/torvalds/linux/commit/84a9582fd203063cd4d301204971ff2cd8327f1a The serial changes allow 16550/8250 driver to consume as many serial ports as they require initializing the driver, originally they only used 1. Due to this we noticed that when we checked the serial ports in use by the device that 8250 was consuming the place in memory that ite-cir should. Ubuntu 24.04 ``` user@host:~$ sudo setserial -g /dev/ttyS* /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4 /dev/ttyS1, UART: 8250, Port: 0x02f8, IRQ: 3 ``` Ubuntu 20.04 ``` user@host:~$ sudo setserial -g /dev/ttyS* /dev/ttyS0, UART: 16550A, Port: 0x03f8, IRQ: 4 /dev/ttyS1, UART: unknown, Port: 0x02f8, IRQ: 3 ``` [Regression] We believe that the regression started from 6.5 kernel onwards based on the commit changes that were done to the serial driver that shipped to those kernels. 23.04 that uses 6.2 works as expected and shows no issues with using the serial port for IR while 23.10 that uses 6.5 and 24.04 that uses 6.8 does not load the ite-cir driver. - [Reproducible steps] * Use Ubuntu 23.10 + or use Kernel 6.5+ * Use IR hardware that requires the ite-cir driver to be loaded * Attempt to load ite-cir driver on boot of OS - * Check /var/log/kern.log for ite-cir driver failure + * Check /var/log/kern.log for ite-cir driver failure [Fix] The requested solution would be to either: * Set the serial driver to no longer consume more than 1 serial port * Have the serial driver check what memory reference ite-cir driver needs to load, ignore that reference and then consume another point in memory instead. - - We have found a workaround where we set the /etc/default/grub config to make the 8250 use only 1 UART port, this allows ite-cir to consume the port in memory like it use to, enabling us to use the driver again + We have found a workaround where we set the /etc/default/grub config to + make the 8250 use only 1 UART port, this allows ite-cir to consume the + port in memory like it use to, enabling us to use the driver again /etc/default/grub ``` GRUB_CMDLINE_LINUX_DEFAULT="8250.nr_uarts=1" ``` The alternative fix is to have a startup script that will remove 8250 from using /dev/ttyS1 and then remove and re-add the ite-cir driver ``` #!/bin/bash setserial /dev/ttyS1 uart none modprobe -r ite-cir ``` [Diagnostic information] version ``` Linux version 6.8.0-22-generic (buildd@lcy02-amd64-070) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-23ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #22-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 4 22:30:32 UTC 2024 ``` lsb_release ``` No LSB modules are available. Description: Ubuntu Noble Numbat (development branch) Release: 24.04 ``` Kernel version ``` linux-image-6.8.0-22-generic: - Installed: 6.8.0-22.22 - Candidate: 6.8.0-22.22 - Version table: - *** 6.8.0-22.22 100 - 100 /var/lib/dpkg/status + Installed: 6.8.0-22.22 + Candidate: 6.8.0-22.22 + Version table: + *** 6.8.0-22.22 100 + 100 /var/lib/dpkg/status ``` ** Summary changed: - ite-cir driver failed to load due to a regression in linux kernel 6.5+ + ite-cir driver failed to load due to a regression in linux kernel 6.5+ serial driver change -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2063322 Title: ite-cir driver failed to load due to a regression in linux kernel 6.5+ serial driver change To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2063322/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
