Description of problem:

Regression from cnv 4.13

DataImportCron annotation
  cdi.kubevirt.io/storage.import.sourceDesiredDigest is not updated automatically

Arnon agilboa@redhat.com looked into it:
W/A: manually delete this line from the DIC:
cdi.kubevirt.io/storage.import.sourceDesiredDigest
It will trigger the reconcile, DIC will be updated correctly, and will import the new DV.

The issue is that the reconcile is not being triggered for the second tag.

Version-Release number of selected component (if applicable):
CNV-v4.14.0.rhel9-1341

How reproducible:
Always

Steps to Reproduce:

1. Create rhel8_image_stream with 'latest' tag

apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  name: rhel8-image-stream
  namespace: test-data-import-cron
spec:
  lookupPolicy:
    local: false
  tags:
  - from:
      kind: DockerImage
      name: registry.redhat.io/rhel8/rhel-guest-image
    importPolicy:
      scheduled: true
    name: latest
    referencePolicy:
      type: Source

2. Create data_import_cron with ImportsToKeep = 1 

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataImportCron
metadata:
  annotations:
    cdi.kubevirt.io/storage.bind.immediate.requested: 'true'
  name: rhel8-image-import-cron
  namespace: test-data-import-cron
spec:
  garbageCollect: Outdated
  importsToKeep: 1
  managedDataSource: rhel8
  schedule: '* * * * *'
  template:
    spec:
      source:
        registry:
          imageStream: rhel8-image-stream
          pullMethod: node
      storage:
        resources:
          requests:
            storage: 10Gi
        storageClassName: hostpath-csi-basic

3. Wait for the first DV to Succeed

4. Update rhel8_image_stream, add another tag
metadata:
  name: rhel8-image-stream
spec:
  tags:
  - from:
      kind: DockerImage
      name: registry.redhat.io/rhel8/rhel-guest-image@sha256:947541648d7f12fd56d2224d55ce708d369f76ffeb4938c8846b287197f30970
    name: 8.4.0-423

5. Wait for the second DV - it's not getting created (we expect it to be created)
And the first one is not getting deleted (we expect it to be deleted)

Actual results:
Second DV is not created, first DV is not deleted, sourceDesiredDigest is not updated

Expected results:
Second DV created, first DV deleted

Additional info:

[cloud-user@ocp-psi-executor ~]$ oc get dataImportCron -n test-data-import-cron rhel8-image-import-cron -oyaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataImportCron
metadata:
  annotations:
    cdi.kubevirt.io/storage.bind.immediate.requested: "true"
    cdi.kubevirt.io/storage.import.imageStreamDockerRef: registry.redhat.io/rhel8/rhel-guest-image@sha256:0da894200daafc9b86d777e8a1413b26960ed6f3930c0003fe3280d52b74c9ac
    cdi.kubevirt.io/storage.import.lastCronTime: "2023-07-27T08:59:01Z"
    cdi.kubevirt.io/storage.import.nextCronTime: "2023-07-27T09:00:00Z"
    cdi.kubevirt.io/storage.import.sourceDesiredDigest: sha256:0da894200daafc9b86d777e8a1413b26960ed6f3930c0003fe3280d52b74c9ac
    cdi.kubevirt.io/storage.import.storageClass: hostpath-csi-basic
  creationTimestamp: "2023-07-27T08:58:54Z"
  generation: 6
  labels:
    created-by-dynamic-class-creator: "Yes"
  name: rhel8-image-import-cron
  namespace: test-data-import-cron
  resourceVersion: "850278"
  uid: 5ee646b0-e752-4276-bbb8-a7245fad03e2
spec:
  garbageCollect: Outdated
  importsToKeep: 1
  managedDataSource: rhel8
  schedule: '* * * * *'
  template:
    metadata: {}
    spec:
      source:
        registry:
          imageStream: rhel8-image-stream
          pullMethod: node
      storage:
        resources:
          requests:
            storage: 10Gi
        storageClassName: hostpath-csi-basic
    status: {}
status:
  conditions:
  - lastHeartbeatTime: "2023-07-27T08:59:54Z"
    lastTransitionTime: "2023-07-27T08:59:54Z"
    message: No current import
    reason: NoImport
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2023-07-27T08:59:54Z"
    lastTransitionTime: "2023-07-27T08:59:54Z"
    message: Latest import is up to date
    reason: UpToDate
    status: "True"
    type: UpToDate
  currentImports:
  - DataVolumeName: rhel8-0da894200daa
    Digest: sha256:0da894200daafc9b86d777e8a1413b26960ed6f3930c0003fe3280d52b74c9ac
  lastExecutionTimestamp: "2023-07-27T08:59:01Z"
  lastImportTimestamp: "2023-07-27T08:59:54Z"
  lastImportedPVC:
    name: rhel8-0da894200daa
    namespace: test-data-import-cron

[cloud-user@ocp-psi-executor ~]$ oc get imagestream -n test-data-import-cron rhel8-image-stream -oyaml
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/image.dockerRepositoryCheck: "2023-07-27T08:59:55Z"
  creationTimestamp: "2023-07-27T08:58:54Z"
  generation: 4
  labels:
    created-by-dynamic-class-creator: "Yes"
  name: rhel8-image-stream
  namespace: test-data-import-cron
  resourceVersion: "850297"
  uid: c858b3a7-dea4-451e-afaa-ec21e47de21e
spec:
  lookupPolicy:
    local: false
  tags:
  - annotations: null
    from:
      kind: DockerImage
      name: registry.redhat.io/rhel8/rhel-guest-image@sha256:947541648d7f12fd56d2224d55ce708d369f76ffeb4938c8846b287197f30970
    generation: 4
    importPolicy:
      importMode: Legacy
    name: 8.4.0-423
    referencePolicy:
      type: Source
status:
  dockerImageRepository: image-registry.openshift-image-registry.svc:5000/test-data-import-cron/rhel8-image-stream
  tags:
  - items:
    - created: "2023-07-27T08:59:55Z"
      dockerImageReference: registry.redhat.io/rhel8/rhel-guest-image@sha256:947541648d7f12fd56d2224d55ce708d369f76ffeb4938c8846b287197f30970
      generation: 4
      image: sha256:947541648d7f12fd56d2224d55ce708d369f76ffeb4938c8846b287197f30970
    tag: 8.4.0-423
  - items:
    - created: "2023-07-27T08:58:55Z"
      dockerImageReference: registry.redhat.io/rhel8/rhel-guest-image@sha256:0da894200daafc9b86d777e8a1413b26960ed6f3930c0003fe3280d52b74c9ac
      generation: 2
      image: sha256:0da894200daafc9b86d777e8a1413b26960ed6f3930c0003fe3280d52b74c9ac
    tag: latest

$ oc logs -n openshift-cnv cdi-deployment-5fdf4bcb96-rb7fx | grep rhel8-image-import-cron

Many update failures like this:

{"level":"error","ts":"2023-07-27T08:58:55Z","msg":"Reconciler error","controller":"dataimportcron-controller","object":

{"name":"rhel8-image-import-cron","namespace":"test-data-import-cron"}

,"namespace":"test-data-import-cron","name":"rhel8-image-import-cron","reconcileID":"374c7f85-0693-44b4-a890-362215e23b58","error":"ImageStream rhel8-image-stream has no tags","errorVerbose":"ImageStream rhel8-image-stream has no tags\nkubevirt.io/containerized-data-importer/pkg/controller.getImageStreamDigest\n\t/remote-source/app/pkg/controller/dataimportcron-controller.go:230\nkubevirt.io/containerized-data-importer/pkg/controller.(*DataImportCronReconciler).updateImageStreamDesiredDigest\n\t/remote-source/app/pkg/controller/dataimportcron-controller.go:535\nkubevirt.io/containerized-data-importer/pkg/controller.(*DataImportCronReconciler).pollImageStreamDigest\n\t/remote-source/app/pkg/controller/dataimportcron-controller.go:269\nkubevirt.io/containerized-data-importer/pkg/controller.(*DataImportCronReconciler).update\n\t/remote-source/app/pkg/controller/dataimportcron-controller.go:400\nkubevirt.io/containerized-data-importer/pkg/controller.(*DataImportCronReconciler).Reconcile\n\t/remote-source/app/pkg/controller/dataimportcron-controller.go:140\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1598","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}

Reconcile for the first tag:

{"level":"info","ts":"2023-07-27T08:58:56Z","logger":"controller.dataimportcron-controller","msg":"Updating DataImportCron","name":"rhel8-image-import-cron","uid":"5ee646b0-e752-4276-bbb8-a7245fad03e2","digest":"sha256:0da894200daafc9b86d777e8a1413b26960ed6f3930c0003fe3280d52b74c9ac"}

{"level":"error","ts":"2023-07-27T08:59:54Z","msg":"Reconciler error","controller":"dataimportcron-controller","object":

{"name":"rhel8-image-import-cron","namespace":"test-data-import-cron"}

,"namespace":"test-data-import-cron","name":"rhel8-image-import-cron","reconcileID":"e4187230-0005-485d-826b-b9b5a838a4aa","error":"Operation cannot be fulfilled on dataimportcrons.cdi.kubevirt.io \"rhel8-image-import-cron\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}

After the 'cdi.kubevirt.io/storage.import.sourceDesiredDigest' deletion from the DIC:

{"level":"info","ts":"2023-07-27T09:53:57Z","logger":"controller.dataimportcron-controller","msg":"Updating DataImportCron","name":"rhel8-image-import-cron","uid":"5ee646b0-e752-4276-bbb8-a7245fad03e2","digest":"sha256:947541648d7f12fd56d2224d55ce708d369f76ffeb4938c8846b287197f30970"} {"level":"info","ts":"2023-07-27T09:53:57Z","logger":"controller.dataimportcron-controller","msg":"Updating DataImportCron","name":"rhel8-image-import-cron","uid":"5ee646b0-e752-4276-bbb8-a7245fad03e2","digest":"sha256:947541648d7f12fd56d2224d55ce708d369f76ffeb4938c8846b287197f30970"}

{"level":"error","ts":"2023-07-27T09:53:57Z","msg":"Reconciler error","controller":"dataimportcron-controller","object":

{"name":"rhel8-image-import-cron","namespace":"test-data-import-cron"}

,"namespace":"test-data-import-cron","name":"rhel8-image-import-cron","reconcileID":"3962aee6-f92b-496a-9d76-e64fa63d2b75","error":"Operation cannot be fulfilled on dataimportcrons.cdi.kubevirt.io \"rhel8-image-import-cron\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}