Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14994

Ingress operator attempts spurious deletes of the client CA configmap when deleting an IngressController that has a client TLS configured

XMLWordPrintable

    • Low
    • No
    • Sprint 237, Sprint 238, Sprint 239, Sprint 240, Sprint 241, Sprint 242, Sprint 243, Sprint 244
    • 8
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, when an IngressController was configured with client SSL/TLS, but did not have the `clientca-configmap` finalizer, the Ingress Operator would try to add the finalizer without checking whether the IngressController was marked for deletion. Consequently, if an IngressController was configured with client SSL/TLS and was subsequently deleted, the Operator would correctly remove the finalizer. It would then repeatedly, and erroneously, try and fail to update the IngressController to add the finalizer back, resulting in error messages in the Operator's logs.
      +
      With this update, the Ingress Operator no longer adds the `clientca-configmap` finalizer to an IngressController that is marked for deletion. As a result, the Ingress Operator no longer tries to perform erroneous updates, and no longer logs the associated errors. (link:https://issues.redhat.com/browse/OCPBUGS-14994[*OCPBUGS-14994*])

      Original CCFR:

      Cause: If an IngressController was configured with client TLS and didn't have the "clientca-configmap" finalizer, the Ingress Operator would try to add the finalizer without checking whether the IngressController was marked for deletion.

      Consequence: If an IngressController was configured with client TLS and subsequently deleted, the Operator would correctly remove the finalizer and then repeatedly erroneously try and fail to update the IngressController to re-add the finalizer, resulting in error messages in the Operator's logs.

      Fix: The Ingress Operator no longer adds the "clientca-configmap" finalizer to an IngressController that is marked for deletion.

      Result: The Operator no longer tries to perform these spurious updates and no longer logs the associated errors.
      Show
      * Previously, when an IngressController was configured with client SSL/TLS, but did not have the `clientca-configmap` finalizer, the Ingress Operator would try to add the finalizer without checking whether the IngressController was marked for deletion. Consequently, if an IngressController was configured with client SSL/TLS and was subsequently deleted, the Operator would correctly remove the finalizer. It would then repeatedly, and erroneously, try and fail to update the IngressController to add the finalizer back, resulting in error messages in the Operator's logs. + With this update, the Ingress Operator no longer adds the `clientca-configmap` finalizer to an IngressController that is marked for deletion. As a result, the Ingress Operator no longer tries to perform erroneous updates, and no longer logs the associated errors. (link: https://issues.redhat.com/browse/OCPBUGS-14994 [* OCPBUGS-14994 *]) Original CCFR: Cause: If an IngressController was configured with client TLS and didn't have the "clientca-configmap" finalizer, the Ingress Operator would try to add the finalizer without checking whether the IngressController was marked for deletion. Consequence: If an IngressController was configured with client TLS and subsequently deleted, the Operator would correctly remove the finalizer and then repeatedly erroneously try and fail to update the IngressController to re-add the finalizer, resulting in error messages in the Operator's logs. Fix: The Ingress Operator no longer adds the "clientca-configmap" finalizer to an IngressController that is marked for deletion. Result: The Operator no longer tries to perform these spurious updates and no longer logs the associated errors.
    • Bug Fix
    • Done

      Description of problem

      When the ingress operator's clientca-configmap controller reconciles an IngressController, this controller attempts to add a finalizer to the IngressController if that finalizer is absent. This controller erroneously attempts to add the missing finalizer even if the IngressController is marked for deletion, which results in an error. This error causes the controller to retry the deletion and log the error multiple times.

      Version-Release number of selected component (if applicable)

      I observed this in CI for OCP 4.14 and was able to reproduce it on 4.11.37, and it probably affects earlier versions as well. The problematic code was added in https://github.com/openshift/cluster-ingress-operator/pull/450/commits/0f36470250c3089769867ebd72e25c413a29cda2 in OCP 4.9 to implement NE-323.

      How reproducible

      Easily.

      Steps to Reproduce

      1. Create a configmap in the "openshift-config" namespace (to reproduce this issue, it is not necessary that the configmap have a valid TLS certificate and key):

      oc -n openshift-config create configmap client-ca-cert

      2. Create an IngressController that specifies spec.clientTLS.clientCA.name to point to the configmap from the previous step:

      oc create -f - <<EOF
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: test-client-ca-configmap
  namespace: openshift-ingress-operator
spec:
  domain: example.xyz
  endpointPublishingStrategy:
    type: Private
  clientTLS:
    clientCA:
      name: client-ca-cert
    clientCertificatePolicy: Required
EOF

      3. Delete the IngressController:

      oc -n openshift-ingress-operator delete ingresscontrollers/test-client-ca-configmap

      4. Check the ingress operator's logs:

      oc -n openshift-ingress-operator logs -c ingress-operator deployments/ingress-operator

      Actual results

      The ingress operator logs several attempts to add the finalizer to the IngressController after it has been marked for deletion:

      2023-06-15T02:17:12.419Z        ERROR   operator.init   controller/controller.go:273    Reconciler error        {"controller": "clientca_configmap_controller", "object": {"name":"test-client-ca-configmap","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "test-client-ca-configmap", "reconcileID": "2274f55e-e5bd-4fdb-973e-821a44cf2ebf", "error": "failed to add client-ca-configmap finalizer: IngressController.operator.openshift.io \"test-client-ca-configmap\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ingresscontroller.operator.openshift.io/finalizer-clientca-configmap\"}"}

      The deletion does succeed, errors notwithstanding.

      Expected results

      The ingress operator should succeed in deleting the IngressController without attempting to re-add the finalizer to the IngressController after it has been marked for deletion.

              mmasters1@redhat.com Miciah Masters
              mmasters1@redhat.com Miciah Masters
              Shudi Li Shudi Li
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: