I have just upgraded to Bullseye (and nvidia-driver 460.91.03-1) - and this problem is gone.
Feel free to close this bug. (I leave this to you, if you want to track this further, otherwise I am fine with you closing it). Thanks for packaging and maintaining the Nvidia drivers! -- Andreas Rönnquist gus...@debian.org