Description of problem:

In the last few days I've noticed that my ssh-agent that has had my YK smartcard added will start failing frequently. Re-inserting the YK seems to be a trigger, though perhaps not the only one.

When I stops working I get:
sign_and_send_pubkey: signing failed: agent refused operation

ssh-pkcs11-helper[26590]: error: C_Sign failed: 257
ssh-agent[26551]: error: process_sign_request2: sshkey_sign: error in libcrypto

If I try to remove to re-add the card:
$ ssh-add -e pkcs11:id=%05
Could not remove card "pkcs11:id=%05": agent refused operation

ssh-agent[26551]: error: process_remove_smartcard_key: pkcs11_del_provider failed

Version-Release number of selected component (if applicable):
openssh-8.0p1-17.el8_7.x86_64
pcsc-lite-1.9.5-1.el8.x86_64
opensc-0.20.0-4.el8.x86_64

On disconnect:
Feb 28 12:28:56 host kernel: usb 4-1: USB disconnect, device number 3
Feb 28 12:28:56 host pcscd[26362]: 99999999 ccid_usb.c:849:WriteUSB() write failed (4/3): -4 LIBUSB_ERROR_NO_DEVICE

On re-insertion:
Feb 28 13:34:04 host kernel: usb 4-1: Product: Yubikey 4 OTP+U2F+CCID
Feb 28 13:34:04 host pcscd[26362]: 99999999 ifdhandler.c:150:CreateChannelByNameOrChannel() failed
Feb 28 13:34:04 host pcscd[26362]: 00036674 ifdhandler.c:150:CreateChannelByNameOrChannel() failed

opensc debug log file attached.