Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-62074

kube-apiserver not crashinglooping due to etcd encryption issue

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 4.16.z
    • kube-apiserver
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      One of the kube-apiserver pods had a fatal startup error but the operator never entered a degraded state.
      
      ~~~~~~
      
      informer-sync check failed: readyz [-]informer-sync failed: 2 informers not started yet: [*v1.Secret *v1.ConfigMap]
      
      "failed to decrypt data" err="no matching key was found for the provided AES transformer"
      
      failed to list *core.ConfigMap: unable to transform key "/kubernetes.io/configmaps/..."
      
      failed to list *core.Secret: unable to transform key "/kubernetes.io/secrets/..."apiserver was unable to write a JSON response: http: Handler timeout 
      
      ~~~
      
      kube-apiserver   4.16.37   True   True   False   2y143d   NodeInstallerProgressing: 3 nodes are at revision 642; 0 nodes have achieved new revision 648 
      
      ~~~ 
      $ oc get pods kube-apiserver-ip-10-128-40-151.ca-central-1.compute.internal
      NAME                                                            READY   STATUS    RESTARTS   AGE
      kube-apiserver-ip-10-128-40-151.ca-central-1.compute.internal   4/5     Running   0          40h        45d
      ~~~
      
      
      

      Version-Release number of selected component (if applicable):

        seen on 4.16.37

      How reproducible:

          did not reproduce

      Steps to Reproduce:

        1. encrypt etcd during cluster deployment
        2. delete a resource but entry is not removed from etcd

      Actual results:

      Resources is deleted and the subsequent entry is not removed from etcd.
      kube-apiserver pods no longer functional due to `no matching key was found for the provided AES transformer' KAS operator reports healthy and stuck in a progressing state.    

      Expected results:

      Resources is deleted and the subsequent entry is not removed from etcd. kube-apiserver fails to start and operator enters a degraded state.    

      Additional info:

      KCS to unblock the issue https://access.redhat.com/solutions/6769801

              Unassigned Unassigned
              reedcort Cortney Reed
              None
              None
              Ke Wang Ke Wang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: