Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-4235

Consider removing reserved labels from objects during backup or prior to restore

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Critical Critical
    • OADP 1.4.0
    • OADP 1.4.0
    • velero
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • Critical
    • 10
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • No

      Description of problem:

      Created on

      OCP 4.16 rc3

      OpenShift-Virt

       

      registry-proxy.engineering.redhat.com/rh-osbs/iib:742249
      RHTAP FBC Commit: https://github.com/openshift-cnv/cnv-fbc/commit/3a980a9723807a2efe4c12dd4121f65151d43a70
      RHTAP FBC Snapshot: cnv-fbc-v4-16-lhf69
      

       

      Recreate steps:

      On the oadp-1.4 branch of oadp-operator

      make deploy-olm

      Prep the CI, note the TEST_VIRT variable

      export OADP_CRED_FILE=/home/whayutin/OPENSHIFT/KEYS/openshift_aws_credentials
      export OADP_BUCKET=cvpbucketuswest2
      export CI_CRED_FILE=/home/whayutin/OPENSHIFT/KEYS/openshift_aws_credentials
      export VSL_REGION=us-west-2
      export BSL_REGION=us-west-2
      export TEST_VIRT=true

      Execute:

      make test-e2e

       

      Restore Fails w/

       

       
      Name:         cirros-test-327edb20-24e7-11ef-a676-047bcb15f3ce
      Namespace:    openshift-adp
      Labels:       <none>
      Annotations:  <none>
      Phase:                       PartiallyFailed (run 'velero restore logs cirros-test-327edb20-24e7-11ef-a676-047bcb15f3ce' for more information)
      Total items to be restored:  27
      Items restored:              27
      Started:    2024-06-07 16:06:15 +0000 UTC
      Completed:  2024-06-07 16:08:00 +0000 UTC
      Warnings:
        Velero:     <none>
        Cluster:  could not restore, CustomResourceDefinition "datavolumes.cdi.kubevirt.io" already exists. Warning: the in-cluster version is different than the backed-up version
                  could not restore, CustomResourceDefinition "virtualmachineinstances.kubevirt.io" already exists. Warning: the in-cluster version is different than the backed-up version
                  could not restore, CustomResourceDefinition "virtualmachines.kubevirt.io" already exists. Warning: the in-cluster version is different than the backed-up version
        Namespaces:
          cirros-test:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                        could not restore, ConfigMap "openshift-service-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                        error getting resource from lister for rolebindings.authorization.openshift.io, cirros-test/system:deployers: rolebindings.authorization.openshift.io "system:deployers" not found
                        error getting resource from lister for rolebindings.authorization.openshift.io, cirros-test/system:image-builders: rolebindings.authorization.openshift.io "system:image-builders" not found
                        error getting resource from lister for rolebindings.authorization.openshift.io, cirros-test/system:image-pullers: rolebindings.authorization.openshift.io "system:image-pullers" not found
      Errors:
        Velero:     <none>
        Cluster:    <none>
        Namespaces:
          cirros-test:  error restoring virtualmachineinstances.kubevirt.io/cirros-test/cirros-test: admission webhook "virtualmachineinstances-create-validator.kubevirt.io" denied the request: creation of the following reserved kubevirt.io/ labels on a VMI object is prohibited
      Backup:  cirros-test-327ecee3-24e7-11ef-a676-047bcb15f3ce
      Namespaces:
        Included:  all namespaces found in the backup
        Excluded:  <none>
      Resources:
        Included:        *
        Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
        Cluster-scoped:  auto
      Namespace mappings:  <none>
      Label selector:  <none>
      Or label selector:  <none>
      Restore PVs:  auto
      CSI Snapshot Restores:
        cirros-test/cirros-test-disk:
          Data Movement:
            Operation ID: dd-bc521512-3eca-4dd4-8ead-322f31c6c9f3.f328cdda-95b5-4d5c022a3
            Data Mover: velero
            Uploader Type: kopia
      Existing Resource Policy:   <none>
      ItemOperationTimeout:       4h0m0s
      Preserve Service NodePorts:  auto
      Restore Item Operations:
        Operation for persistentvolumeclaims cirros-test/cirros-test-disk:
          Restore Item Action Plugin:  velero.io/csi-pvc-restorer
          Operation ID:                dd-bc521512-3eca-4dd4-8ead-322f31c6c9f3.f328cdda-95b5-4d5c022a3
          Phase:                       Completed
          Progress:                    1073741824 of 1073741824 complete (Bytes)
          Progress description:        Completed
          Created:                     2024-06-07 16:06:17 +0000 UTC
          Started:                     2024-06-07 16:06:45 +0000 UTC
          Updated:                     2024-06-07 16:07:39 +0000 UTC
      HooksAttempted:   0
      HooksFailed:      0
      Resource List:
        apiextensions.k8s.io/v1/CustomResourceDefinition:
          - datavolumes.cdi.kubevirt.io(failed)
          - virtualmachineinstances.kubevirt.io(failed)
          - virtualmachines.kubevirt.io(failed)
        apps/v1/ControllerRevision:
          - cirros-test/revision-start-vm-e1792c10-c5be-477f-9e5c-e3575ad3a48a-1(created)
        authorization.openshift.io/v1/RoleBinding:
          - cirros-test/system:deployers(failed)
          - cirros-test/system:image-builders(failed)
          - cirros-test/system:image-pullers(failed)
        cdi.kubevirt.io/v1beta1/DataVolume:
          - cirros-test/cirros-test-disk(created)
        kubevirt.io/v1/VirtualMachine:
          - cirros-test/cirros-test(created)
        kubevirt.io/v1/VirtualMachineInstance:
          - cirros-test/cirros-test(failed)
        policy/v1/PodDisruptionBudget:
          - cirros-test/kubevirt-disruption-budget-gl5dn(created)
        rbac.authorization.k8s.io/v1/RoleBinding:
          - cirros-test/system:deployers(created)
          - cirros-test/system:image-builders(created)
          - cirros-test/system:image-pullers(created)
        v1/ConfigMap:
          - cirros-test/kube-root-ca.crt(failed)
          - cirros-test/openshift-service-ca.crt(failed)
        v1/Namespace:
          - cirros-test(created)
        v1/PersistentVolume:
          - pvc-ff55f76e-0f69-43ac-92ac-b72240ae9453(skipped)
        v1/PersistentVolumeClaim:
          - cirros-test/cirros-test-disk(created)
        v1/Pod:
          - cirros-test/virt-launcher-cirros-test-fxd76(created)
        v1/Secret:
          - cirros-test/builder-dockercfg-d6nk9(created)
          - cirros-test/default-dockercfg-9625j(created)
          - cirros-test/deployer-dockercfg-twlm6(created)
        v1/ServiceAccount:
          - cirros-test/builder(skipped)
          - cirros-test/default(skipped)
          - cirros-test/deployer(skipped)
        velero.io/v2alpha1/DataUpload:
          - openshift-adp/cirros-test-327ecee3-24e7-11ef-a676-047bcb15f3ce-dxqqv(skipped)
      whayutin@thinkdoe:~$ 
      

      Potentially hitting:

      https://github.com/kubevirt/kubevirt/blob/main/pkg/virt-api/webhooks/validating-webhook/admitters/vmi-create-admitter.go#L1207

       

      Potential Fix:

      Remove any k8s or kubevirt reserved labels prior to the restore:

      https://kubernetes.io/docs/reference/labels-annotations-taints/#internal.config.kubernetes.io-reserved-wildcard

      && kubevirt.io/* 

       

            spampatt@redhat.com Shubham Pampattiwar
            wnstb Wes Hayutin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: