Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-6126

Handle 409 Conflict when removing finalizer

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Undefined Undefined
    • OSSM 3.0-TP1
    • None
    • Project Sail
    • None

      If the timing is just right, the operator may try to remove the finalizer twice. The second operation will then fail with 409 Conflict, as evident from the following logs:

      2024-03-19T14:11:36Z	INFO	ctrlr.istiorev	Reconciliation done. Updating status.	{"IstioRevision": "default", "reconcileID": "ea26f13c-1532-4aa1-8e27-c3dc07d0ffda"}
      2024-03-19T14:11:36Z	INFO	ctrlr.istiorev	Installing components	{"IstioRevision": "default", "reconcileID": "7073626b-d73e-4dd7-8120-08b34f28f46f"}
      2024-03-19T14:11:37Z	INFO	ctrlr.istiorev	Reconciliation done. Updating status.	{"IstioRevision": "default", "reconcileID": "7073626b-d73e-4dd7-8120-08b34f28f46f"}
      2024-03-19T14:11:37Z	INFO	ctrlr.istiorev	Installing components	{"IstioRevision": "default", "reconcileID": "f4c061c0-4ebd-43a2-ad1e-d45aebc6fe3a"}
      2024-03-19T14:11:38Z	INFO	ctrlr.istiorev	Reconciliation done. Updating status.	{"IstioRevision": "default", "reconcileID": "f4c061c0-4ebd-43a2-ad1e-d45aebc6fe3a"}
      2024-03-19T14:11:38Z	INFO	ctrlr.istiorev	Installing components	{"IstioRevision": "default", "reconcileID": "466265ad-8855-469a-8ef4-fafa42f885d8"}
      2024-03-19T14:11:39Z	INFO	ctrlr.istiorev	Reconciliation done. Updating status.	{"IstioRevision": "default", "reconcileID": "466265ad-8855-469a-8ef4-fafa42f885d8"}
      2024-03-19T14:11:56Z	INFO	ctrlr.istiorev	Removing finalizer	{"IstioRevision": "default", "reconcileID": "7d92f83c-6df8-46ba-9647-c1f38433a625"}
      2024-03-19T14:11:56Z	INFO	ctrlr.istiorev	Removing finalizer	{"IstioRevision": "default", "reconcileID": "34e313c9-9c2e-471b-8632-278f3cd1f6fc"}
      2024-03-19T14:11:56Z	INFO	ctrlr.istiorev	failed to remove finalizer	{"IstioRevision": "default", "reconcileID": "34e313c9-9c2e-471b-8632-278f3cd1f6fc"}
      2024-03-19T14:11:56Z	ERROR	ctrlr.istiorev	Reconciler error	{"IstioRevision": "default", "reconcileID": "34e313c9-9c2e-471b-8632-278f3cd1f6fc", "error": "Could not remove finalizer from /default: Operation cannot be fulfilled on istiorevisions.operator.istio.io \"default\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/operator.istio.io/istiorevisions/default, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 0e0db596-d03e-4c52-9824-e7fad17b12d1, UID in object meta: ", "errorVerbose": "Operation cannot be fulfilled on istiorevisions.operator.istio.io \"default\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/operator.istio.io/istiorevisions/default, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 0e0db596-d03e-4c52-9824-e7fad17b12d1, UID in object meta: \nCould not remove finalizer from /default\ngithub.com/istio-ecosystem/sail-operator/pkg/kube.RemoveFinalizer\n\tgithub.com/istio-ecosystem/sail-operator/pkg/kube/finalizers.go:50\ngithub.com/istio-ecosystem/sail-operator/controllers/istiorevision.(*IstioRevisionReconciler).Reconcile\n\tgithub.com/istio-ecosystem/sail-operator/controllers/istiorevision/istiorevision_controller.go:120\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1650"}
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      	sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227
      

      This can happen if a watch event that triggers the reconciliation of the IstioRevision CR comes in at the same time as the operator is removing the finalizer on the same IstioRevision CR. This watch event is triggered by a change to one of the owned resources.

            mluksa@redhat.com Marko Luksa
            mluksa@redhat.com Marko Luksa
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: