A few more details... This is triggered only after I connect to a remove server with gpg-agent's "ssh agent" mode, backed by a hardware token (a yubikey) which holds my Authentication key. Merely starting the gpg-agent.service unit is not enough.
Using the following override reduces the delay, in ~/.config/systemd/user/gpg-agent.service.d/override.conf: [Service] TimeoutStopSec=10s A. -- The good news about computers is that they do what you tell them to do. The bad news is that they do what you tell them to do. - Ted Nelson