Loading...

Type: Bug
Resolution: Done
Priority: Normal
Fix Version/s: OADP 1.1.0
Affects Version/s: OADP 1.1.0
Component/s: None
Labels:
- csi
- triaged

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Fixed in Build:
oadp-velero-container-1.1.0-23
QEStatus:
Passed

Cost of Delay:
0
WSJF:
0
Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

Release Blocker:
Rejected

Regression:
No

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Description of problem: CSI Restore partially fails with the following error in the logs:

time="2022-07-14T14:34:44Z" level=error msg="error restoring example-snapclass: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: default snapshot class: example-snapclass already exits for driver: pd.csi.storage.gke.io" logSource="pkg/restore/restore.go:1357" restore=openshift-adp/mysql-42edcb0a-0381-11ed-9e9d-902e163f806c
time="2022-07-14T14:34:45Z" level=error msg="Cluster resource restore error: error restoring volumesnapshotclasses.snapshot.storage.k8s.io/example-snapclass: admission webhook \"volumesnapshotclasses.snapshot.storage.k8s.io\" denied the request: default snapshot class: example-snapclass already exits for driver: pd.csi.storage.gke.io" logSource="pkg/controller/restore_controller.go:500" restore=openshift-adp/mysql-42edcb0a-0381-11ed-9e9d-902e163f806c

Version-Release number of selected component (if applicable):

OADP 1.1 latest prestage build

brew.registry.redhat.io/rh-osbs/iib:277714

OCP 4.11

How reproducible: 100%

Steps to Reproduce:

Storageclass:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2022-07-14T13:16:59Z"
  name: ssd-csi
  resourceVersion: "367469"
  uid: ccbab41f-2431-4200-b0f7-b527e36f69c4
parameters:
  replication-type: none
  type: pd-ssd
provisioner: pd.csi.storage.gke.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

1. Create secret & DPA:

[mperetz@mperetz oadp-e2e-qe]$ oc get dpa -n openshift-adp -o yaml
apiVersion: v1
items:
- apiVersion: oadp.openshift.io/v1alpha1
  kind: DataProtectionApplication
  metadata:
    creationTimestamp: "2022-07-14T14:28:49Z"
    generation: 1
    name: ts-dpa
    namespace: openshift-adp
    resourceVersion: "445027"
    uid: 429540bf-ea2c-47ce-9a45-360b1ae2e51f
  spec:
    backupLocations:
    - velero:
        credential:
          key: cloud
          name: cloud-credentials-gcp
        default: true
        objectStorage:
          bucket: oadpbucket120690
          prefix: velero-e2e-40d52269-0381-11ed-9e9d-902e163f806c
        provider: gcp
    configuration:
      restic:
        podConfig:
          resourceAllocations: {}
      velero:
        defaultPlugins:
        - openshift
        - gcp
        - kubevirt
        - csi
    podDnsConfig: {}
    snapshotLocations: []
  status:
    conditions:
    - lastTransitionTime: "2022-07-14T14:47:19Z"
      message: Reconcile complete
      reason: Complete
      status: "True"
      type: Reconciled

2. Create VSC:

[mperetz@mperetz oadp-e2e-qe]$ oc get volumesnapshotclass example-snapclass -o yaml

apiVersion: snapshot.storage.k8s.io/v1
deletionPolicy: Retain
driver: pd.csi.storage.gke.io
kind: VolumeSnapshotClass
metadata:
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2022-07-14T14:28:58Z"
  generation: 1
  labels:
    velero.io/csi-volumesnapshot-class: "true"
  name: example-snapclass
  resourceVersion: "367457"
  uid: 72ada9ae-2f8b-48c6-b4c2-fdeea2cfec3f

3. Deploy app using the relevant storageclass for CSI snapshots:

[mperetz@mperetz oadp-e2e-qe]$ oc get pvc -n mysql-persistent
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
mysql   Bound    pvc-772ba182-1933-46a8-8071-5770d5fa68a4   2Gi        RWO            ssd-csi        20m
[mperetz@mperetz oadp-e2e-qe]$

4. Create backup:

apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    velero.io/source-cluster-k8s-gitversion: v1.24.0+9546431
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "24"
  creationTimestamp: "2022-07-14T14:30:17Z"
  generation: 5
  labels:
    velero.io/storage-location: ts-dpa-1
  name: mysql-42edcb0a-0381-11ed-9e9d-902e163f806c
  namespace: openshift-adp
  resourceVersion: "378132"
  uid: 2536b1f4-4044-489e-89f2-be4f939e9c40
spec:
  defaultVolumesToRestic: false
  hooks: {}
  includedNamespaces:
  - mysql-persistent
  metadata: {}
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
status:
  completionTimestamp: "2022-07-14T14:31:36Z"
  csiVolumeSnapshotsAttempted: 1
  expiration: "2022-08-13T14:30:17Z"
  formatVersion: 1.1.0
  phase: Completed
  progress:
    itemsBackedUp: 46
    totalItems: 46
  startTimestamp: "2022-07-14T14:30:17Z"
  version: 1

5. Delete the app project, and create restore after it's deleted:

[mperetz@mperetz oadp-e2e-qe]$ oc get restore -n openshift-adp mysql-42edcb0a-0381-11ed-9e9d-902e163f806c -o yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: "2022-07-14T14:34:27Z"
  generation: 7
  name: mysql-42edcb0a-0381-11ed-9e9d-902e163f806c
  namespace: openshift-adp
  resourceVersion: "392194"
  uid: ff279a92-05fa-40cf-8427-0d8f892c13dd
spec:
  backupName: mysql-42edcb0a-0381-11ed-9e9d-902e163f806c
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  hooks: {}
status:
  completionTimestamp: "2022-07-14T14:34:45Z"
  errors: 1
  phase: PartiallyFailed
  progress:
    itemsRestored: 34
    totalItems: 34
  startTimestamp: "2022-07-14T14:34:27Z"
  warnings: 5