Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-605

[OCP 4.11] CSI restore fails with admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Normal
    • OADP 1.1.0
    • OADP 1.1.0
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • oadp-velero-container-1.1.0-23
    • Passed
    • No
    • 0
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • Rejected

    Description

      Description of problem: CSI Restore partially fails with the following error in the logs:

      time="2022-07-14T14:34:44Z" level=error msg="error restoring example-snapclass: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: default snapshot class: example-snapclass already exits for driver: pd.csi.storage.gke.io" logSource="pkg/restore/restore.go:1357" restore=openshift-adp/mysql-42edcb0a-0381-11ed-9e9d-902e163f806c
      time="2022-07-14T14:34:45Z" level=error msg="Cluster resource restore error: error restoring volumesnapshotclasses.snapshot.storage.k8s.io/example-snapclass: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: default snapshot class: example-snapclass already exits for driver: pd.csi.storage.gke.io" logSource="pkg/controller/restore_controller.go:500" restore=openshift-adp/mysql-42edcb0a-0381-11ed-9e9d-902e163f806c

      Version-Release number of selected component (if applicable):

      OADP 1.1 latest prestage build

      brew.registry.redhat.io/rh-osbs/iib:277714

      OCP 4.11

      How reproducible: 100% 

      Steps to Reproduce:

      Storageclass:

      allowVolumeExpansion: true
      apiVersion: storage.k8s.io/v1
      kind: StorageClass
      metadata:
        annotations:
          storageclass.kubernetes.io/is-default-class: "true"
        creationTimestamp: "2022-07-14T13:16:59Z"
        name: ssd-csi
        resourceVersion: "367469"
        uid: ccbab41f-2431-4200-b0f7-b527e36f69c4
      parameters:
        replication-type: none
        type: pd-ssd
      provisioner: pd.csi.storage.gke.io
      reclaimPolicy: Delete
      volumeBindingMode: WaitForFirstConsumer

      1. Create secret & DPA:

      [mperetz@mperetz oadp-e2e-qe]$ oc get dpa -n openshift-adp -o yaml
      apiVersion: v1
      items:
      - apiVersion: oadp.openshift.io/v1alpha1
        kind: DataProtectionApplication
        metadata:
          creationTimestamp: "2022-07-14T14:28:49Z"
          generation: 1
          name: ts-dpa
          namespace: openshift-adp
          resourceVersion: "445027"
          uid: 429540bf-ea2c-47ce-9a45-360b1ae2e51f
        spec:
          backupLocations:
          - velero:
              credential:
                key: cloud
                name: cloud-credentials-gcp
              default: true
              objectStorage:
                bucket: oadpbucket120690
                prefix: velero-e2e-40d52269-0381-11ed-9e9d-902e163f806c
              provider: gcp
          configuration:
            restic:
              podConfig:
                resourceAllocations: {}
            velero:
              defaultPlugins:
              - openshift
              - gcp
              - kubevirt
              - csi
          podDnsConfig: {}
          snapshotLocations: []
        status:
          conditions:
          - lastTransitionTime: "2022-07-14T14:47:19Z"
            message: Reconcile complete
            reason: Complete
            status: "True"
            type: Reconciled

      2. Create VSC:

      [mperetz@mperetz oadp-e2e-qe]$ oc get volumesnapshotclass example-snapclass -o yaml

      apiVersion: snapshot.storage.k8s.io/v1
      deletionPolicy: Retain
      driver: pd.csi.storage.gke.io
      kind: VolumeSnapshotClass
      metadata:
        annotations:
          snapshot.storage.kubernetes.io/is-default-class: "true"
        creationTimestamp: "2022-07-14T14:28:58Z"
        generation: 1
        labels:
          velero.io/csi-volumesnapshot-class: "true"
        name: example-snapclass
        resourceVersion: "367457"
        uid: 72ada9ae-2f8b-48c6-b4c2-fdeea2cfec3f

      3. Deploy app using the relevant storageclass for CSI snapshots:

      [mperetz@mperetz oadp-e2e-qe]$ oc get pvc -n mysql-persistent
      NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
      mysql   Bound    pvc-772ba182-1933-46a8-8071-5770d5fa68a4   2Gi        RWO            ssd-csi        20m
      [mperetz@mperetz oadp-e2e-qe]$ 

      4. Create backup:

      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        annotations:
          velero.io/source-cluster-k8s-gitversion: v1.24.0+9546431
          velero.io/source-cluster-k8s-major-version: "1"
          velero.io/source-cluster-k8s-minor-version: "24"
        creationTimestamp: "2022-07-14T14:30:17Z"
        generation: 5
        labels:
          velero.io/storage-location: ts-dpa-1
        name: mysql-42edcb0a-0381-11ed-9e9d-902e163f806c
        namespace: openshift-adp
        resourceVersion: "378132"
        uid: 2536b1f4-4044-489e-89f2-be4f939e9c40
      spec:
        defaultVolumesToRestic: false
        hooks: {}
        includedNamespaces:
        - mysql-persistent
        metadata: {}
        storageLocation: ts-dpa-1
        ttl: 720h0m0s
      status:
        completionTimestamp: "2022-07-14T14:31:36Z"
        csiVolumeSnapshotsAttempted: 1
        expiration: "2022-08-13T14:30:17Z"
        formatVersion: 1.1.0
        phase: Completed
        progress:
          itemsBackedUp: 46
          totalItems: 46
        startTimestamp: "2022-07-14T14:30:17Z"
        version: 1
       

      5. Delete the app project, and create restore after it's deleted:

      [mperetz@mperetz oadp-e2e-qe]$ oc get restore -n openshift-adp mysql-42edcb0a-0381-11ed-9e9d-902e163f806c -o yaml
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        creationTimestamp: "2022-07-14T14:34:27Z"
        generation: 7
        name: mysql-42edcb0a-0381-11ed-9e9d-902e163f806c
        namespace: openshift-adp
        resourceVersion: "392194"
        uid: ff279a92-05fa-40cf-8427-0d8f892c13dd
      spec:
        backupName: mysql-42edcb0a-0381-11ed-9e9d-902e163f806c
        excludedResources:
        - nodes
        - events
        - events.events.k8s.io
        - backups.velero.io
        - restores.velero.io
        - resticrepositories.velero.io
        hooks: {}
      status:
        completionTimestamp: "2022-07-14T14:34:45Z"
        errors: 1
        phase: PartiallyFailed
        progress:
          itemsRestored: 34
          totalItems: 34
        startTimestamp: "2022-07-14T14:34:27Z"
        warnings: 5

      Actual results:

       

      Expected results:

       

      Additional info: attached must-gather

      Attachments

        Activity

          People

            spampatt@redhat.com Shubham Pampattiwar
            mperetz@redhat.com Maya Peretz
            Prasad Joshi Prasad Joshi
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: