Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60918

Conflict errors while reconciling resources

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Low
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      There are conflict errors logged by the OSUS Operator, like:

      1.7562250919357936e+09 ERROR controller_updateservice Failed to update Status {"Request.Namespace": "openshift-update-service", "Request.Name": "sample", "error": "Operation cannot be fulfilled on updateservices.updateservice.operator.openshift.io \"sample\": the object has been modified; please apply your changes to the latest version and try again"}
      github.com/openshift/cincinnati-operator/controllers.(*UpdateServiceReconciler).Reconcile
      /remote-source/app/controllers/updateservice_controller.go:200
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
      /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
      /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
      /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
      /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234
      

      This happens on other resources too, not just UpdateService:

      1.7562256206702862e+09 ERROR controller_updateservice UpdateDeploymentFailed {"Request.Namespace": "openshift-update-service", "Request.Name": "another-sample", "error": "Operation cannot be fulfilled on deployments.apps \"another-sample\": the object has been modified; please apply your changes to the latest version and try again"}
      

      Version-Release number of selected component (if applicable):

      5.0.3

      How reproducible:

      Seems often

      Steps to Reproduce:

      1. Deploy an UpdateService with the operator and let it run for some time
      2. Inspect the operator log

      Actual results:

      Conflict errors in the log

      Expected results:

      No conflict errors in the log

      Additional info:

      This is probably not that serious because in controller-runtime returning an error will cause the controller to retry, but I would not expect a healthy operator to emit error noise in its log (at minimum it gets in the way when troubleshooting). Some combination of SSA, using patch updates, status updates, limiting concurrent reconciles or using some kind of retry that does not spam the log with error should allow getting rid of this.

              Unassigned Unassigned
              afri@afri.cz Petr Muller
              None
              None
              Jia Liu Jia Liu
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: