+++ This bug was initially created as a clone of Bug #2083039 +++

Description of problem:
Got a lot of "Reconciler error" in cdi-deployment log after adding custom DataImportCron to hco

Version-Release number of selected component (if applicable):
CNV-v4.10.1-101

How reproducible:
Always

Steps to Reproduce:
1. Add custom DataImportCron to HCO
eg:
-------------8<-----------------
spec:
certConfig:
ca:
duration: 48h0m0s
renewBefore: 24h0m0s
server:
duration: 24h0m0s
renewBefore: 12h0m0s
dataImportCronTemplates:

• metadata:
  annotations:
  cdi.kubevirt.io/storage.bind.immediate.requested: "true"
  name: data-import-test2
  spec:
  managedDataSource: data-source-test2
  retentionPolicy: None
  schedule: '* * * * *'
  template:
  spec:
  source:
  registry:
  pullMethod: node
  url: docker://quay.io/containerdisks/fedora:35
  storage:
  resources:
  requests:
  storage: 10Gi
  featureGates:
  enableCommonBootImageImport: true
  -------------8<-----------------

Actual results:
Got below error frequently till the import finished, log file is attached.

$ oc logs cdi-deployment-7bbb74d84d-hjkjs -n openshift-cnv | grep Reconciler | grep data-source-test2

{"level":"error","ts":1652077998.9596689,"logger":"controller.dataimportcron-controller","msg":"Reconciler error","name":"data-source-test2","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on datasources.cdi.kubevirt.io \"data-source-test2\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1652078009.948743,"logger":"controller.datavolume-controller","msg":"Unable to update datavolume","name":"data-source-test2-3b3fc310abea","error":"Operation cannot be fulfilled on datavolumes.cdi.kubevirt.io \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"kubevirt.io/containerized-data-importer/pkg/controller.(*DatavolumeReconciler).reconcileDataVolumeStatus\n\t/remote-source/app/pkg/controller/datavolume-controller.go:2144\nkubevirt.io/containerized-data-importer/pkg/controller.(*DatavolumeReconciler).Reconcile\n\t/remote-source/app/pkg/controller/datavolume-controller.go:526\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1652078009.9488642,"logger":"controller.datavolume-controller","msg":"Reconciler error","name":"data-source-test2-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on datavolumes.cdi.kubevirt.io \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1652078010.129169,"logger":"controller.import-controller","msg":"Reconciler error","name":"data-source-test-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on persistentvolumeclaims \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"} {"level":"error","ts":1652078010.1938965,"logger":"controller.import-controller","msg":"Reconciler error","name":"data-source-test2-3b3fc310abea","namespace":"openshift-virtualization-os-images","error":"Operation cannot be fulfilled on persistentvolumeclaims \"data-source-test2-3b3fc310abea\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}

$ oc get dv data-source-test2-3b3fc310abea
NAME PHASE PROGRESS RESTARTS AGE
data-source-test2-3b3fc310abea Succeeded 100.0% 18m

Expected results:
Better not to reduce the noise

Additional info:
log attached

— Additional comment from Yan Du on 2022-05-09 07:02:06 UTC —