Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-34410

[Backport 4.14] Ingress operator attempts spurious deletes of the client CA configmap when deleting an IngressController that has a client TLS configured

XMLWordPrintable

    • Low
    • No
    • 1
    • Sprint 254
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, when an `IngressController` object was configured with client SSL/TLS, but did not have the `clientca-configmap` finalizer, the Ingress Operator tried to add the finalizer without checking whether the `IngressController` object was marked for deletion. Consequently, if an `IngressController` was configured with client SSL/TLS and was subsequently deleted, the Operator would correctly remove the finalizer. The Operator would then repeatedly, and erroneously, try and fail to update the `IngressController` object to add the finalizer back, resulting in error messages in the Operator's logs. With this update, the Ingress Operator no longer adds the `clientca-configmap` finalizer to an `IngressController` object that is marked for deletion. As a result, the Ingress Operator no longer tries to perform erroneous updates and no longer logs the associated errors. (link:https://issues.redhat.com/browse/OCPBUGS-34410[*OCPBUGS-34410*])
      Show
      * Previously, when an `IngressController` object was configured with client SSL/TLS, but did not have the `clientca-configmap` finalizer, the Ingress Operator tried to add the finalizer without checking whether the `IngressController` object was marked for deletion. Consequently, if an `IngressController` was configured with client SSL/TLS and was subsequently deleted, the Operator would correctly remove the finalizer. The Operator would then repeatedly, and erroneously, try and fail to update the `IngressController` object to add the finalizer back, resulting in error messages in the Operator's logs. With this update, the Ingress Operator no longer adds the `clientca-configmap` finalizer to an `IngressController` object that is marked for deletion. As a result, the Ingress Operator no longer tries to perform erroneous updates and no longer logs the associated errors. (link: https://issues.redhat.com/browse/OCPBUGS-34410 [* OCPBUGS-34410 *])
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-14994. The following is the description of the original issue:

      Description of problem

      When the ingress operator's clientca-configmap controller reconciles an IngressController, this controller attempts to add a finalizer to the IngressController if that finalizer is absent. This controller erroneously attempts to add the missing finalizer even if the IngressController is marked for deletion, which results in an error. This error causes the controller to retry the deletion and log the error multiple times.

      Version-Release number of selected component (if applicable)

      I observed this in CI for OCP 4.14 and was able to reproduce it on 4.11.37, and it probably affects earlier versions as well. The problematic code was added in https://github.com/openshift/cluster-ingress-operator/pull/450/commits/0f36470250c3089769867ebd72e25c413a29cda2 in OCP 4.9 to implement NE-323.

      How reproducible

      Easily.

      Steps to Reproduce

      1. Create a configmap in the "openshift-config" namespace (to reproduce this issue, it is not necessary that the configmap have a valid TLS certificate and key):

      oc -n openshift-config create configmap client-ca-cert
      

      2. Create an IngressController that specifies spec.clientTLS.clientCA.name to point to the configmap from the previous step:

      oc create -f - <<EOF
      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: test-client-ca-configmap
        namespace: openshift-ingress-operator
      spec:
        domain: example.xyz
        endpointPublishingStrategy:
          type: Private
        clientTLS:
          clientCA:
            name: client-ca-cert
          clientCertificatePolicy: Required
      EOF
      

      3. Delete the IngressController:

      oc -n openshift-ingress-operator delete ingresscontrollers/test-client-ca-configmap
      

      4. Check the ingress operator's logs:

      oc -n openshift-ingress-operator logs -c ingress-operator deployments/ingress-operator
      

      Actual results

      The ingress operator logs several attempts to add the finalizer to the IngressController after it has been marked for deletion:

      2023-06-15T02:17:12.419Z        ERROR   operator.init   controller/controller.go:273    Reconciler error        {"controller": "clientca_configmap_controller", "object": {"name":"test-client-ca-configmap","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "test-client-ca-configmap", "reconcileID": "2274f55e-e5bd-4fdb-973e-821a44cf2ebf", "error": "failed to add client-ca-configmap finalizer: IngressController.operator.openshift.io \"test-client-ca-configmap\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ingresscontroller.operator.openshift.io/finalizer-clientca-configmap\"}"}                                                                                                                                   
      

      The deletion does succeed, errors notwithstanding.

      Expected results

      The ingress operator should succeed in deleting the IngressController without attempting to re-add the finalizer to the IngressController after it has been marked for deletion.

              cholman@redhat.com Candace Holman
              openshift-crt-jira-prow OpenShift Prow Bot
              Shudi Li Shudi Li
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: