A customer reported that after they configured AAD SSO for the OpenShift console, users were redirected back to the login page in a loop. A similar issue occurred when trying to log in via the CLI with a token, which resulted in an "error: token provided is invalid or expired." The issue was caused by orphaned entries in the etcd database that remained after the customer deleted resources like namespaces and configmaps. These orphaned entries prevented the kube-apiserver from starting properly.

What would cause there to be orphaned entries in the etcd database and is this expected behavior?

kube-apiserver   4.16.37   True   True   False   2y143d   NodeInstallerProgressing: 3 nodes are at revision 642; 0 nodes have achieved new revision 648  

oc logs -n openshift-kube-apiserver kube-apiserver-ip-10-128-40-151 -c kube-apiserver

informer-sync check failed: readyz [-]informer-sync failed: 2 informers not started yet: [*v1.Secret *v1.ConfigMap]

"failed to decrypt data" err="no matching key was found for the provided AES transformer"

failed to list *core.ConfigMap: unable to transform key "/kubernetes.io/configmaps/..."

failed to list *core.Secret: unable to transform key "/kubernetes.io/secrets/..."apiserver was unable to write a JSON response: http: Handler timeout 

    seen on 4.16.37

    did not reproduce

  1. encrypt etcd during cluster deployment
  2. delete a resource but entry is not removed from etcd

Resources is deleted and the subsequent entry is not removed from etcd.
kube-apiserver pods no longer functional due to `no matching key was found for the provided AES transformer'

Resource is deleted and the corresponding entry in etcd is removed

Similar OCP BUG: https://issues.redhat.com/browse/OCPBUGS-38598