Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-49428

VirtHandlerRestErrorsHigh alert entirely due to "the object has been modified" retries

XMLWordPrintable

    • 0.42
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • ---
    • ---
    • Medium
    • None

      Description of problem:

      
      

      Version-Release number of selected component (if applicable):

      OCP 4.15.22
      
      

      How reproducible:

      Uncertain. The customer is seeing this on this cluster consistently.
      
      

      There are no steps. The error happened on its own.

      Actual results:{code:none}
      The log has many error entries like 
      
      2024-09-25T20:45:07.922623295Z {"component":"virt-handler","kind":"","level":"error","msg":"Updating the VirtualMachineInstance status failed.","name":"rptr1","namespace":"prod-vms","pos":"vm.go:2062","reason":"Operation cannot be fulfilled on virtualmachineinstances.kubevirt.io \"rptr1\": the object has been modified; please apply your changes to the latest version and try again","timestamp":"2024-09-25T20:45:07.922479Z","uid":"400da48b-5f3a-43a1-a724-ca69d4c59355"}
      
      .
      2024-09-26T20:35:07.930839358Z {"component":"virt-handler","kind":"","level":"error","msg":"Updating the VirtualMachineInstance status failed.","name":"sso1","namespace":"prod-vms","pos":"vm.go:2062","reason":"Operation cannot be fulfilled on virtualmachineinstances.kubevirt.io \"sso1\": the object has been modified; please apply your changes to the latest version and try again","timestamp":"2024-09-26T20:35:07.930703Z","uid":"80ceec62-de6f-4b1f-b507-b0a75078540e"}
      
      .
      2024-09-26T20:40:43.587108677Z {"component":"virt-handler","kind":"","level":"error","msg":"Updating the VirtualMachineInstance status failed.","name":"mta1t","namespace":"prod-vms","pos":"vm.go:2062","reason":"Operation cannot be fulfilled on virtualmachineinstances.kubevirt.io \"mta1\": the object has been modified; please apply your changes to the latest version and try again","timestamp":"2024-09-26T20:40:43.586926Z","uid":"0c735481-de18-49b4-82a8-d54e3d90ecf2"}
      
      
      Each time, these log messages is followed by a re-enqueue, for example
      
      2024-09-25T20:45:07.922645426Z {"component":"virt-handler","level":"info","msg":"re-enqueuing VirtualMachineInstance prod-vms/rpt1","pos":"vm.go:1699","reason":"Operation cannot be fulfilled on virtualmachineinstances.kubevirt.io \"rptr1\": the object has been modified; please apply your changes to the latest version and try again","timestamp":"2024-09-25T20:45:07.922603Z"}
      
      The errors happen in roughly 5 minute increments, so for example, on log has 599 of these errors out of 30k lines.
      
      

      Expected results:

      No alert, and I would expect that "The object has been modified" would not be treated as an error.
      
      

      Additional info:

      
      

              mschatzm@redhat.com Matan Schatzman
              rhn-support-dguthrie David Guthrie
              Guohua Ouyang Guohua Ouyang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: