On Sun, 2025-11-30 at 10:27 +0100, Salvatore Bonaccorso wrote: > Hi Benjmanin, > > On Sat, Nov 29, 2025 at 12:02:00AM +0100, Salvatore Bonaccorso wrote: > > Hi Benjamin, > > > > On Fri, Nov 28, 2025 at 10:53:17PM +0100, Benjamin Drung wrote: > > > On Wed, 2025-11-26 at 08:52 +0100, Salvatore Bonaccorso wrote: > > > > Control: tags -1 + moreinfo > > > > > > > > Hi, > > > > > > > > On Sat, Nov 22, 2025 at 04:27:53PM +0100, Benjamin Drung wrote: > > > > > Hi Uwe, > > > > > > > > > > On Fri, 2025-11-21 at 09:52 +0100, Uwe Kleine-König wrote: > > > > > > Hello Benjamin, > > > > > > > > > > > > On Thu, Nov 20, 2025 at 11:11:07AM +0100, Benjamin Drung wrote: > > > > > > > On Thu, 2025-11-20 at 08:58 +0100, Uwe Kleine-König wrote: > > > > > > > > On Mon, Nov 17, 2025 at 02:05:41AM +0100, Benjamin Drung wrote: > > > > > > > > > On Wed, 2025-11-05 at 20:14 +0100, Uwe Kleine-König wrote: > > > > > > > > > > Can you try adding > > > > > > > > > > > > > > > > > > > > earlyprintk=serial,0x7e215040 > > > > > > > > > > > > > > > > > > > > to the kernel commandline. Not sure this activates the > > > > > > > > > > right procedures, > > > > > > > > > > maybe try it with the working kernel first. > > > > > > > > > > > > > > > > > > I tried with the working kernel 6.12.43+deb13-rpi but these > > > > > > > > > were the > > > > > > > > > earliest lines: > > > > > > > > > > > > > > > > > > [ 4.116565] printk: legacy console [ttyS1] enabled > > > > > > > > > [ 4.129635] printk: legacy bootconsole [earlycon0] disabled > > > > > > > > > [ 4.147050] bcm2835-power bcm2835-power: Broadcom BCM2835 > > > > > > > > > power domains driver > > > > > > > > > [ 4.160972] mousedev: PS/2 mouse device common for all mice > > > > > > > > > [ 4.172622] i2c-bcm2835 20805000.i2c: Could not read > > > > > > > > > clock-frequency property > > > > > > > > > [ 4.186585] bcm2835-wdt bcm2835-wdt: Broadcom BCM2835 > > > > > > > > > watchdog timer > > > > > > > > > > > > > > > > > > I tried with 6.12.57+deb13-rpi but that failed to boot and the > > > > > > > > > earlyprintk line did not give any additional logs. > > > > > > > > > > > > > > > > Can you please provide a full kernel log of the working kernel? > > > > > > > > > > > > > > dmesg log is attached. > > > > > > > > > > > > > > > Also please provide the output of: > > > > > > > > > > > > > > > > for f in /proc/tty/driver/*; do echo $f; sed 's/^/ /' > > > > > > > > < $f; done > > > > > > > > > > > > > > /proc/tty/driver/serial > > > > > > > serinfo:1.0 driver revision: > > > > > > > 0: uart:unknown port:00000000 irq:0 > > > > > > > 1: uart:16550 mmio:0x20215040 irq:53 tx:15629 rx:0 RTS|CTS|DTR > > > > > > > 2: uart:unknown port:00000000 irq:0 > > > > > > > 3: uart:unknown port:00000000 irq:0 > > > > > > > /proc/tty/driver/ttyAMA > > > > > > > serinfo:1.0 driver revision: > > > > > > > 0: uart:PL011 rev2 mmio:0x20201000 irq:81 tx:452 rx:980 > > > > > > > RTS|CTS|DTR > > > > > > > > > > > > OK, is > > > > > > > > > > > > earlyprintk=serial,0x20215040 > > > > > > > > > > > > more communicative then? > > > > > > > > > > Nope, but I found > > > > > https://www.raspberrypi.com/documentation/computers/configuration.html#enabling-early-console-for-linux > > > > > and setting earlycon=uart8250,mmio32,0x20215040 worked. The kernel > > > > > 6.12.43+deb13-rpi booted with printing the early lines as well (see > > > > > dmesg of previous mail). > > > > > > > > > > With all other experiments cleaned, I tried the latest kernel: > > > > > > > > > > ``` > > > > > $ cat /etc/default/raspi-extra-cmdline > > > > > earlycon=uart8250,mmio32,0x20215040 > > > > > $ update-initramfs -u > > > > > ``` > > > > > > > > > > I got following logs before it got stuck: > > > > > > > > > > ``` > > > > > [ 0.000000] Booting Linux on physical CPU 0x0 > > > > > [ 0.000000] Linux version 6.12.57+deb13-rpi > > > > > ([email protected]) (arm-linux-gnueabi-gcc-14 (Debian > > > > > 14.2.0-19) 14.2.0, GNU ld (GNU Binutils for Debian) 2.44) #1 Debian > > > > > 6.12.57-1 (2025-11-05) > > > > > [ 0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 > > > > > (ARMv7), cr=00c5387d > > > > > [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT > > > > > nonaliasing instruction cache > > > > > [ 0.000000] OF: fdt: Machine model: Raspberry Pi Zero W Rev 1.1 > > > > > [ 0.000000] random: crng init done > > > > > [ 0.000000] earlycon: uart8250 at MMIO32 0x20215040 (options '') > > > > > [ 0.000000] printk: legacy bootconsole [uart8250] enabled > > > > > [ 0.000000] Memory policy: Data cache writeback > > > > > [ 0.000000] Reserved memory: bypass linux,cma node, using cmdline > > > > > CMA params instead > > > > > [ 0.000000] OF: reserved mem: node linux,cma compatible matching > > > > > fail > > > > > [ 0.000000] cma: Reserved 64 MiB at 0x17000000 on node -1 > > > > > ``` > > > > > > > > > > It looks like being cma related and I found a cma boot option in > > > > > /etc/default/raspi-firmware. So I modified /boot/firmware/cmdline.txt > > > > > and changed cma=64M to cma=0. Then the system booted without problems: > > > > > > > > > > ``` > > > > > [...] > > > > > [ 0.000000] Reserved memory: bypass linux,cma node, using cmdline > > > > > CMA params instead > > > > > [ 0.000000] OF: reserved mem: node linux,cma compatible matching > > > > > fail > > > > > [ 0.000000] Zone ranges: > > > > > [ 0.000000] Normal [mem 0x0000000000000000-0x000000001bffffff] > > > > > [ 0.000000] Movable zone start for each node > > > > > [ 0.000000] Early memory node ranges > > > > > [ 0.000000] node 0: [mem 0x0000000000000000-0x000000001bffffff] > > > > > [ 0.000000] Initmem setup node 0 [mem > > > > > 0x0000000000000000-0x000000001bffffff] > > > > > [ 0.000000] Kernel command line: dma.dmachans=0x7ff5 > > > > > bcm2708.boardrev=0x9000c1 bcm2708.serial=0x8bb1238 > > > > > bcm2708.uart_clock=48000000 bcm2708.disk_led_gpio=47 > > > > > smsc95xx.macaddr=B8:27:EB:BB:12:38 vc_mem.mem_base=0x1ec00000 > > > > > vc_mem.mem_size=0x20000000 console=tty0 console=ttyS1,115200 > > > > > root=/dev/mmcblk0p2 rw fsck.repair=yes net.ifnames=0 cma=0 rootwait > > > > > earlycon=uart8250,mmio32,0x20215040 > > > > > [...] > > > > > ``` > > > > > > > > > > Then I persisted this change by setting CMA=0 in > > > > > /etc/default/raspi-firmware: > > > > > > > > > > ``` > > > > > $ grep -Ev '^(#|$)' /etc/default/raspi-firmware > > > > > CMA=0 > > > > > $ update-initramfs -u > > > > > ``` > > > > > > > > > > This config removes the cma cmdline option completely. The system > > > > > still > > > > > boots: > > > > > > > > > > ``` > > > > > [...] > > > > > [ 0.000000] Memory policy: Data cache writeback > > > > > [ 0.000000] Reserved memory: created CMA memory pool at > > > > > 0x17000000, size 64 MiB > > > > > [ 0.000000] OF: reserved mem: initialized node linux,cma, > > > > > compatible id shared-dma-pool > > > > > [ 0.000000] OF: reserved mem: 0x17000000..0x1affffff (65536 KiB) > > > > > map reusable linux,cma > > > > > [ 0.000000] Zone ranges: > > > > > [ 0.000000] Normal [mem 0x0000000000000000-0x000000001bffffff] > > > > > [ 0.000000] Movable zone start for each node > > > > > [ 0.000000] Early memory node ranges > > > > > [ 0.000000] node 0: [mem 0x0000000000000000-0x000000001bffffff] > > > > > [ 0.000000] Initmem setup node 0 [mem > > > > > 0x0000000000000000-0x000000001bffffff] > > > > > [ 0.000000] Kernel command line: dma.dmachans=0x7ff5 > > > > > bcm2708.boardrev=0x9000c1 bcm2708.serial=0x8bb1238 > > > > > bcm2708.uart_clock=48000000 bcm2708.disk_led_gpio=47 > > > > > smsc95xx.macaddr=B8:27:EB:BB:12:38 vc_mem.mem_base=0x1ec00000 > > > > > vc_mem.mem_size=0x20000000 console=tty0 console=ttyS1,115200 > > > > > root=/dev/mmcblk0p2 rw fsck.repair=yes net.ifnames=0 rootwait > > > > > earlycon=uart8250,mmio32,0x20215040 > > > > > [...] > > > > > ``` > > > > > > > > > > That should hopefully narrow down the issue space. > > > > > > > > If I'm not wrong the only cma related change between 6.12.43 and > > > > 6.12.48 was the backport of 2c223f7239f3 ("of: reserved_mem: > > > > Restructure call site for dma_contiguous_early_fixup()"): > > > > > > > > commit 46efab01648a04082266115a8e917c3b26b97fa8 > > > > Author: Oreoluwa Babatunde <[email protected]> > > > > Date: Wed Aug 6 10:24:21 2025 -0700 > > > > > > > > of: reserved_mem: Restructure call site for > > > > dma_contiguous_early_fixup() > > > > > > > > [ Upstream commit 2c223f7239f376a90d71903ec474ba887cf21d94 ] > > > > > > > > Restructure the call site for dma_contiguous_early_fixup() > > > > to > > > > where the reserved_mem nodes are being parsed from the DT > > > > so that > > > > dma_mmu_remap[] is populated before dma_contiguous_remap() > > > > is called. > > > > > > > > Fixes: 8a6e02d0c00e ("of: reserved_mem: Restructure how the > > > > reserved memory regions are processed") > > > > Signed-off-by: Oreoluwa Babatunde > > > > <[email protected]> > > > > Tested-by: William Zhang <[email protected]> > > > > Signed-off-by: Marek Szyprowski <[email protected]> > > > > Link: > > > > https://lore.kernel.org/r/[email protected] > > > > Signed-off-by: Sasha Levin <[email protected]> > > > > > > > > Uwe, would that make sense? Benjamin can you try to make a build with > > > > that commit reverted, does it fix the problem? > > > > > > Building the kernel on the Pi Zero would probably take ages. I have a > > > Raspberry Pi 5 running arm64. That should be able to build the kernel in > > > a armel schroot, shouldn't it? Is there documentation for building the > > > kernel? > > > > I hope it is fine for Uwe that i respond here, he might have another > > more efficient way to reach our goal. > > > > But we have the simple-patching guideline here: > > https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#id-1.6.6.4 > > > > That should work within an armel schroot work well, e.g. on a > > porterbox amdahl.d.o i would first setup a schroot via: > > > > schroot -b -c chroot:trixie_armel-dchroot -n linux > > dd-schroot-cmd -c linux apt-get update > > dd-schroot-cmd -c linux apt-get dist-upgrade > > dd-schroot-cmd -c linux apt-get build-dep linux > > schroot -r -c linux > > (see https://dsa.debian.org/doc/schroot/) > > > > Then download the linux/6.12.57-1 source and make the revert patch > > available. > > > > apt-get source linux > > cd linux-* > > ./debian/bin/test-patches -f rpi > > ../0001-Revert-of-reserved_mem-Restructure-call-site-for-dma.patch > > In case you can/want trust unsigned packages (but I have put a > sha256sum along signed with my key in the DD keyring), you can test > the patch revert via the packages on: > https://people.debian.org/~carnil/tmp/linux/1116251/
Thanks for providing the kernel. I downloaded linux-image- 6.12+unreleased-rpi_6.12.57-1a~test_armel.deb and installed it. This kernel hangs on boot (see attached boot.log). It hangs there reproducible. -- Benjamin Drung Debian & Ubuntu Developer
[ 0.000000] Booting Linux on physical CPU 0x0 [ 0.000000] Linux version 4.14.98+ (dom@dom-XPS-13-9370) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1200 Tue Feb 12 20:11:02 GMT 2019 [ 0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache [ 0.000000] OF: fdt: Machine model: Raspberry Pi Zero W Rev 1.1 [ 0.000000] earlycon: uart8250 at MMIO32 0x20215040 (options '') [ 0.000000] bootconsole [uart8250] enabled [ 0.000000] Memory policy: Data cache writeback [ 0.000000] Reserved memory: created CMA memory pool at 0x17c00000, size 64 MiB [ 0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 113680 [ 0.000000] Kernel command line: dma.dmachans=0x7ff5 bcm2708.boardrev=0x9000c1 bcm2708.serial=0x8bb1238 bcm2708.uart_clock=48000000 bcm2708.disk_led_gpio=47 smsc95xx.macaddr=B8:27:EB:BB:12:38 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000 console=tty0 console=ttyS1,115200 root=/dev/mmcblk0p2 rw fsck.repair=yes net.ifnames=0 rootwait earlycon=uart8250,mmio32,0x20215040 [ 0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes) [ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes) [ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes) [ 0.000000] Memory: 377768K/458752K available (6451K kernel code, 589K rwdata, 1992K rodata, 440K init, 673K bss, 15448K reserved, 65536K cma-reserved) [ 0.000000] Virtual kernel memory layout: [ 0.000000] vector : 0xffff0000 - 0xffff1000 ( 4 kB) [ 0.000000] fixmap : 0xffc00000 - 0xfff00000 (3072 kB) [ 0.000000] vmalloc : 0xdc800000 - 0xff800000 ( 560 MB) [ 0.000000] lowmem : 0xc0000000 - 0xdc000000 ( 448 MB) [ 0.000000] modules : 0xbf000000 - 0xc0000000 ( 16 MB) [ 0.000000] .text : 0xc0008000 - 0xc0655050 (6453 kB) [ 0.000000] .init : 0xc08d4000 - 0xc0942000 ( 440 kB) [ 0.000000] .data : 0xc0942000 - 0xc09d54f8 ( 590 kB) [ 0.000000] .bss : 0xc09dadf0 - 0xc0a8338c ( 674 kB) [ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] ftrace: allocating 24128 entries in 71 pages [ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16 [ 0.000025] sched_clock: 32 bits at 1000kHz, resolution 1000ns, wraps every 2147483647500ns [ 0.008505] clocksource: timer: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns [ 0.018004] bcm2835: system timer (irq = 27) [ 0.022803] Console: colour dummy device 80x30 [ 0.027948] console [tty0] enabled [ 0.031490] Calibrating delay loop... 697.95 BogoMIPS (lpj=3489792) [ 0.097892] pid_max: default: 32768 minimum: 301 [ 0.103044] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes) [ 0.109774] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes) [ 0.118134] Disabling memory control group subsystem [ 0.123331] CPU: Testing write buffer coherency: ok [ 0.129330] Setting up static identity map for 0x8200 - 0x8238 [ 0.136388] devtmpfs: initialized [ 0.147883] random: get_random_u32 called from bucket_table_alloc+0x88/0x1c4 with crng_init=0 [ 0.157593] VFP support v0.3: implementor 41 architecture 1 part 20 variant b rev 5 [ 0.165755] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 0.175811] futex hash table entries: 256 (order: -1, 3072 bytes) [ 0.189630] pinctrl core: initialized pinctrl subsystem [ 0.196264] NET: Registered protocol family 16 [ 0.203703] DMA: preallocated 1024 KiB pool for atomic coherent allocations [ 0.217148] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers. [ 0.224145] hw-breakpoint: maximum watchpoint size is 4 bytes. [ 0.230177] Serial: AMBA PL011 UART driver [ 0.238639] bcm2835-mbox 2000b880.mailbox: mailbox enabled [ 0.245169] uart-pl011 20201000.serial: could not find pctldev for node /soc/gpio@7e200000/uart0-gpio32, deferring probe [ 0.296761] bcm2835-dma 20007000.dma-controller: DMA legacy API manager at dc80d000, dmachans=0x1 [ 0.308003] SCSI subsystem initialized [ 0.312044] usbcore: registered new interface driver usbfs [ 0.317778] usbcore: registered new interface driver hub [ 0.323321] usbcore: registered new device driver usb [ 0.338587] raspberrypi-firmware soc:firmware: Attached to firmware from 2024-04-17 17:29 [ 0.349059] clocksource: Switched to clocksource timer [ 0.437991] VFS: Disk quotas dquot_6.6.0 [ 0.442214] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) [ 0.449596] FS-Cache: Loaded [ 0.452823] CacheFiles: Loaded [ 0.470565] NET: Registered protocol family 2 [ 0.476090] TCP established hash table entries: 4096 (order: 2, 16384 bytes) [ 0.483423] TCP bind hash table entries: 4096 (order: 2, 16384 bytes) [ 0.490081] TCP: Hash tables configured (established 4096 bind 4096) [ 0.496674] UDP hash table entries: 256 (order: 0, 4096 bytes) [ 0.502667] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes) [ 0.509372] NET: Registered protocol family 1 [ 0.514451] RPC: Registered named UNIX socket transport module. [ 0.520549] RPC: Registered udp transport module. [ 0.525328] RPC: Registered tcp transport module. [ 0.530132] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.537991] hw perfevents: no irqs for PMU, sampling events not supported [ 0.545034] hw perfevents: enabled with armv6_1176 PMU driver, 3 counters available [ 0.556143] workingset: timestamp_bits=14 max_order=17 bucket_order=3 [ 0.574124] FS-Cache: Netfs 'nfs' registered for caching [ 0.580847] NFS: Registering the id_resolver key type [ 0.586038] Key type id_resolver registered [ 0.590448] Key type id_legacy registered [ 0.594554] nfs4filelayout_init: NFSv4 File Layout Driver Registering... [ 0.605385] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251) [ 0.613374] io scheduler noop registered [ 0.617376] io scheduler deadline registered [ 0.622271] io scheduler cfq registered (default) [ 0.627066] io scheduler mq-deadline registered [ 0.631724] io scheduler kyber registered [ 0.636242] gpiochip_add_data: GPIOs 0..53 (pinctrl-bcm2835) failed to register [ 0.643747] pinctrl-bcm2835 20200000.gpio: could not add GPIO chip [ 0.652443] bcm2835-aux-uart 20215040.serial: could not find pctldev for node /soc/gpio@7e200000/uart1-gpio14, deferring probe [ 0.665760] bcm2835-rng 20104000.rng: hwrng registered [ 0.671311] vc-mem: phys_addr:0x00000000 mem_base=0x1ec00000 mem_size:0x20000000(512 MiB) [ 0.696587] brd: module loaded [ 0.710487] loop: module loaded [ 0.713714] Loading iSCSI transport class v2.0-870. [ 0.719480] usbcore: registered new interface driver smsc95xx [ 0.725347] dwc_otg: version 3.00a 10-AUG-2012 (platform bus) [ 0.731832] usbcore: registered new interface driver usb-storage [ 0.738299] mousedev: PS/2 mouse device common for all mice [ 0.744070] IR NEC protocol handler initialized [ 0.748677] IR RC5(x/sz) protocol handler initialized [ 0.753840] IR RC6 protocol handler initialized [ 0.758441] IR JVC protocol handler initialized [ 0.763067] IR Sony protocol handler initialized [ 0.767756] IR SANYO protocol handler initialized [ 0.772554] IR Sharp protocol handler initialized [ 0.777330] IR MCE Keyboard/mouse protocol handler initialized [ 0.783273] IR XMP protocol handler initialized [ 0.789307] bcm2835-wdt 20100000.watchdog: Broadcom BCM2835 watchdog timer [ 0.796716] bcm2835-cpufreq: min=700000 max=1000000 [ 0.802444] sdhci: Secure Digital Host Controller Interface driver [ 0.808735] sdhci: Copyright(c) Pierre Ossman [ 0.813478] sdhost-bcm2835 20202000.mmc: could not find pctldev for node /soc/gpio@7e200000/sdhost-gpio48, deferring probe [ 0.824879] sdhci-pltfm: SDHCI platform and OF driver helper [ 0.830977] ledtrig-cpu: registered to indicate activity on CPUs [ 0.837194] hidraw: raw HID events driver (C) Jiri Kosina [ 0.842949] usbcore: registered new interface driver usbhid [ 0.848615] usbhid: USB HID core driver [ 0.852793] bcm2835_vchiq 2000b840.mailbox: Missing firmware node [ 0.859018] bcm2835_vchiq: probe of 2000b840.mailbox failed with error -2 [ 0.866463] Initializing XFRM netlink socket [ 0.870925] NET: Registered protocol family 17 [ 0.875565] Key type dns_resolver registered [ 0.881570] registered taskstats version 1 [ 0.889671] uart-pl011 20201000.serial: could not find pctldev for node /soc/gpio@7e200000/uart0-gpio32, deferring probe [ 0.901155] gpiochip_add_data: GPIOs 0..53 (pinctrl-bcm2835) failed to register [ 0.908603] pinctrl-bcm2835 20200000.gpio: could not add GPIO chip [ 0.915167] bcm2835-aux-uart 20215040.serial: could not find pctldev for node /soc/gpio@7e200000/uart1-gpio14, deferring probe [ 0.927175] sdhost-bcm2835 20202000.mmc: could not find pctldev for node /soc/gpio@7e200000/sdhost-gpio48, deferring probe [ 0.939763] of_cfs_init [ 0.942381] of_cfs_init: OK
