I also had a similar issue as anarcat with gpg 2.1.18 and my yubikey neo with the same symptoms, although I'm running Exherbo, and not Debian. The disable-ccid workaround in scdaemon.conf worked for me.
-- cmr http://octayn.net/ +16038524272
signature.asc
Description: OpenPGP digital signature
