The issue seems to be a misconfiguration in the standard /lib/systemd/system/ptp4l@.service: apparently, it starts already when the network (hardware) is not fully up and running – at least on my Raspberry Pi5 is not sufficiently up (running current Debian bookworm from Raspberry with Network Manager (not(!) dhcpd-base), as is delivered in the standard image). Instead of just After=sys-subsystem-net-devices-%i.device it requires After=network-online.target Wants=network-online.target Thank you!
P.S.: Hat-tip to @jclark @ https://github.com/raspberrypi/linux/issues/5881#issuecomment-2567484885