Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-35027

[Backport 4.12] Ingress operator attempts spurious deletes of the client CA configmap when deleting an IngressController that has a client TLS configured

XMLWordPrintable

    • Low
    • No
    • 2
    • Sprint 254, NE Sprint 255
    • 2
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, when an IngressController was configured with client SSL/TLS, but did not have the `clientca-configmap` finalizer, the Ingress Operator would try to add the finalizer without checking whether the IngressController was marked for deletion. Consequently, if an IngressController was configured with client SSL/TLS and was subsequently deleted, the Operator would correctly remove the finalizer. It would then repeatedly, and erroneously, try and fail to update the IngressController to add the finalizer back, resulting in error messages in the Operator's logs.
      +
      With this update, the Ingress Operator no longer adds the `clientca-configmap` finalizer to an IngressController that is marked for deletion. As a result, the Ingress Operator no longer tries to perform erroneous updates, and no longer logs the associated errors. (link:https://issues.redhat.com/browse/OCPBUGS-34410[*OCPBUGS-34410*], backport to OCP 4.14)

      Original CCFR:

      Cause: If an IngressController was configured with client TLS and didn't have the "clientca-configmap" finalizer, the Ingress Operator would try to add the finalizer without checking whether the IngressController was marked for deletion.

      Consequence: If an IngressController was configured with client TLS and subsequently deleted, the Operator would correctly remove the finalizer and then repeatedly erroneously try and fail to update the IngressController to re-add the finalizer, resulting in error messages in the Operator's logs.

      Fix: The Ingress Operator no longer adds the "clientca-configmap" finalizer to an IngressController that is marked for deletion.

      Result: The Operator no longer tries to perform these spurious updates and no longer logs the associated errors.
      Show
      * Previously, when an IngressController was configured with client SSL/TLS, but did not have the `clientca-configmap` finalizer, the Ingress Operator would try to add the finalizer without checking whether the IngressController was marked for deletion. Consequently, if an IngressController was configured with client SSL/TLS and was subsequently deleted, the Operator would correctly remove the finalizer. It would then repeatedly, and erroneously, try and fail to update the IngressController to add the finalizer back, resulting in error messages in the Operator's logs. + With this update, the Ingress Operator no longer adds the `clientca-configmap` finalizer to an IngressController that is marked for deletion. As a result, the Ingress Operator no longer tries to perform erroneous updates, and no longer logs the associated errors. (link: https://issues.redhat.com/browse/OCPBUGS-34410 [* OCPBUGS-34410 *], backport to OCP 4.14) Original CCFR: Cause: If an IngressController was configured with client TLS and didn't have the "clientca-configmap" finalizer, the Ingress Operator would try to add the finalizer without checking whether the IngressController was marked for deletion. Consequence: If an IngressController was configured with client TLS and subsequently deleted, the Operator would correctly remove the finalizer and then repeatedly erroneously try and fail to update the IngressController to re-add the finalizer, resulting in error messages in the Operator's logs. Fix: The Ingress Operator no longer adds the "clientca-configmap" finalizer to an IngressController that is marked for deletion. Result: The Operator no longer tries to perform these spurious updates and no longer logs the associated errors.
    • Bug Fix
    • In Progress

      This is a clone of issue OCPBUGS-34765. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-34410. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-14994. The following is the description of the original issue:
      —

      Description of problem

      When the ingress operator's clientca-configmap controller reconciles an IngressController, this controller attempts to add a finalizer to the IngressController if that finalizer is absent. This controller erroneously attempts to add the missing finalizer even if the IngressController is marked for deletion, which results in an error. This error causes the controller to retry the deletion and log the error multiple times.

      Version-Release number of selected component (if applicable)

      I observed this in CI for OCP 4.14 and was able to reproduce it on 4.11.37, and it probably affects earlier versions as well. The problematic code was added in https://github.com/openshift/cluster-ingress-operator/pull/450/commits/0f36470250c3089769867ebd72e25c413a29cda2 in OCP 4.9 to implement NE-323.

      How reproducible

      Easily.

      Steps to Reproduce

      1. Create a configmap in the "openshift-config" namespace (to reproduce this issue, it is not necessary that the configmap have a valid TLS certificate and key):

      oc -n openshift-config create configmap client-ca-cert

      2. Create an IngressController that specifies spec.clientTLS.clientCA.name to point to the configmap from the previous step:

      oc create -f - <<EOF
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: test-client-ca-configmap
  namespace: openshift-ingress-operator
spec:
  domain: example.xyz
  endpointPublishingStrategy:
    type: Private
  clientTLS:
    clientCA:
      name: client-ca-cert
    clientCertificatePolicy: Required
EOF

      3. Delete the IngressController:

      oc -n openshift-ingress-operator delete ingresscontrollers/test-client-ca-configmap

      4. Check the ingress operator's logs:

      oc -n openshift-ingress-operator logs -c ingress-operator deployments/ingress-operator

      Actual results

      The ingress operator logs several attempts to add the finalizer to the IngressController after it has been marked for deletion:

      2023-06-15T02:17:12.419Z        ERROR   operator.init   controller/controller.go:273    Reconciler error        {"controller": "clientca_configmap_controller", "object": {"name":"test-client-ca-configmap","namespace":"openshift-ingress-operator"}, "namespace": "openshift-ingress-operator", "name": "test-client-ca-configmap", "reconcileID": "2274f55e-e5bd-4fdb-973e-821a44cf2ebf", "error": "failed to add client-ca-configmap finalizer: IngressController.operator.openshift.io \"test-client-ca-configmap\" is invalid: metadata.finalizers: Forbidden: no new finalizers can be added if the object is being deleted, found new finalizers []string{\"ingresscontroller.operator.openshift.io/finalizer-clientca-configmap\"}"}

      The deletion does succeed, errors notwithstanding.

      Expected results

      The ingress operator should succeed in deleting the IngressController without attempting to re-add the finalizer to the IngressController after it has been marked for deletion.

              cholman@redhat.com Candace Holman
              openshift-crt-jira-prow OpenShift Prow Bot
              Shudi Li Shudi Li
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: