Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-601

DataMover: CSI restore fails

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • No

      Description of problem: restore fails with the following errors in the volume-snapshot-mover pod:

      1.6575448472955258e+09    ERROR    controller.volumesnapshotbackup    unable to fetch VolumeSnapshotBackup CR    {"reconciler group": "datamover.oadp.openshift.io", "reconciler kind": "VolumeSnapshotBackup", "name": "vsb-velero-mysql-8kcd2", "namespace": "mysql-persistent", "vsb": "mysql-persistent/vsb-velero-mysql-8kcd2", "error": "VolumeSnapshotBackup.datamover.oadp.openshift.io \"vsb-velero-mysql-8kcd2\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
1.6575448472955942e+09    ERROR    controller.volumesnapshotbackup    Reconciler error    {"reconciler group": "datamover.oadp.openshift.io", "reconciler kind": "VolumeSnapshotBackup", "name": "vsb-velero-mysql-8kcd2", "namespace": "mysql-persistent", "error": "VolumeSnapshotBackup.datamover.oadp.openshift.io \"vsb-velero-mysql-8kcd2\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
E0711 13:08:07.245188       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:08:54.119666       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
1.657544937309999e+09    ERROR    controller.volumesnapshotrestore    Reconciler error    {"reconciler group": "datamover.oadp.openshift.io", "reconciler kind": "VolumeSnapshotRestore", "name": "vsr-mysql", "namespace": "mysql-persistent", "error": "VolumeSnapshotRestore CR volumeSnapshotMoverBackupref ResticRepository cannot be empty"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
E0711 13:09:28.175078       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
1.657545007199967e+09    ERROR    controller.volumesnapshotrestore    Reconciler error    {"reconciler group": "datamover.oadp.openshift.io", "reconciler kind": "VolumeSnapshotRestore", "name": "vsr-mysql", "namespace": "mysql-persistent", "error": "VolumeSnapshotRestore CR volumeSnapshotMoverBackupref ResticRepository cannot be empty"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
E0711 13:10:24.163117       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:11:05.281945       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:11:44.229617       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:12:41.834239       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:13:19.492248       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:14:13.127587       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:14:52.980701       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:15:23.560101       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:16:12.048894       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:16:43.664178       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:17:16.087624       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
E0711 13:17:53.586599       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)

      Version-Release number of selected component (if applicable): 1.1

       

      How reproducible: 100%

       

      Steps to Reproduce:
      1. Install Volsync from latest stable channel

      cat <<EOF | oc create -f -
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        generateName: oadp-
        name: oadp-operator
        namespace: openshift-operators
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: volsync-product
        source: prestage-operators
        sourceNamespace: openshift-marketplace
      EOF

      2. Create VSC:

      apiVersion: snapshot.storage.k8s.io/v1
      deletionPolicy: Retain
      driver: ebs.csi.aws.com
      kind: VolumeSnapshotClass
      metadata:
        annotations:
          snapshot.storage.kubernetes.io/is-default-class: "true"
        labels:
          velero.io/csi-volumesnapshot-class: "true"
        name: example-snapclass

      2. create restic-secret:

      apiVersion: v1
      data:
        AWS_ACCESS_KEY_ID: XXXXXXX
        AWS_SECRET_ACCESS_KEY: XXXXXXXXXXXXXXX
        RESTIC_PASSWORD: my-secure-restic-password
        RESTIC_REPOSITORY: s3:s3.amazonaws.com/oadpbucket119606
      kind: Secret
      metadata:
        name: restic-secret
        namespace: openshift-adp
      type: Opaque

      3. Set DPA CR:

      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        creationTimestamp: "2022-07-11T12:49:59Z"
        generation: 1
        name: ts-dpa
        namespace: openshift-adp
      spec:
        backupLocations:
        - velero:
            config:
              region: us-east-2
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: oadpbucket119606
              prefix: velero-e2e-f395364a-0117-11ed-ad74-902e163f806c
            provider: aws
        configuration:
          restic:
            enable: true
            podConfig:
              resourceAllocations: {}
          velero:
            defaultPlugins:
            - openshift
            - aws
            - kubevirt
            - csi
        features:
          enableDataMover: true
        podDnsConfig: {}
        snapshotLocations: []

      4. Create backup of an application (mysql in my case). Make sure VSB status is Completed

      5. Delete the app namespace; delete the volumesnapshotcontent

      6. Create restore:
      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        name: mysql-f5a0b773-0117-11ed-ad74-902e163f806c-20220711160833
        namespace: openshift-adp
      spec:
        backupName: mysql-f5a0b773-0117-11ed-ad74-902e163f806c
        excludedResources:
        - nodes
        - events
        - events.events.k8s.io
        - backups.velero.io
        - restores.velero.io
        - resticrepositories.velero.io
        hooks: {}
        includedNamespaces:
        - '*'
        restorePVs: true

      7. Check the  volume-snapshot-mover logs - note the error described above in "Description of problem"

      8. Check the VSR on the app namespace; note the status doesn't appear at any point.

      Actual results:

       

      Expected results:

       

      Additional info:

        1. must-gather.tar.gz
          1.81 MB

              emcmulla@redhat.com Emily McMullan
              mperetz@redhat.com Maya Peretz
              Maya Peretz Maya Peretz
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: