Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-607

DataMover: VSB is stuck on SnapshotBackupDone

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • oadp-operator-bundle-container-1.1.0-32
    • Passed
    • 0
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • Yes

      Description of problem:  VSB is stuck on SnapshotBackupDone.

      Looks like there is a permissions issue. It reads on the volume-snapshot-mover pod's logs this error:

       

      -mysql-484sf", "namespace": "mysql-persistent", "vsb": "mysql-persistent/vsb-velero-mysql-484sf"}
      E0715 08:45:07.833396       1 reflector.go:138] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:250: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io)
      1.6578747114058044e+09    ERROR    controller.volumesnapshotbackup    unable to delete VSB resource    {"reconciler group": "datamover.oadp.openshift.io", "reconciler kind": "VolumeSnapshotBackup", "name": "vsb-velero-mysql-484sf", "namespace": "mysql-persistent", "vsb": "mysql-persistent/vsb-velero-mysql-484sf", "error": "volumesnapshots.snapshot.storage.k8s.io is forbidden: User \"system:serviceaccount:openshift-adp:openshift-adp-controller-manager\" cannot deletecollection resource \"volumesnapshots\" in API group \"snapshot.storage.k8s.io\" in the namespace \"openshift-adp\""}
      github.com/konveyor/volume-snapshot-mover/controllers.ReconcileBatch
          /workspace/controllers/common.go:58
      github.com/konveyor/volume-snapshot-mover/controllers.(*VolumeSnapshotBackupReconciler).Reconcile
          /workspace/controllers/volumesnapshotbackup_controller.go:117
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
          /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
          /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
          /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
          /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
       

      The VSB's phase is stuck on `SnapshotBackupDone`.

      Version-Release number of selected component (if applicable):

      OADP 1.1.0

      Checked both on OCP 4.11 & OCP 4.10

      How reproducible: 100%

      Steps to Reproduce:
      1. Install Volsync from latest stable channel

      cat <<EOF | oc create -f -
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        generateName: oadp-
        name: oadp-operator
        namespace: openshift-operators
      spec:
        channel: stable
        installPlanApproval: Automatic
        name: volsync-product
        source: prestage-operators
        sourceNamespace: openshift-marketplace
      EOF

      2. Create VSC:

      apiVersion: snapshot.storage.k8s.io/v1
      deletionPolicy: Retain
      driver: ebs.csi.aws.com
      kind: VolumeSnapshotClass
      metadata:
        annotations:
          snapshot.storage.kubernetes.io/is-default-class: "true"
        labels:
          velero.io/csi-volumesnapshot-class: "true"
        name: example-snapclass

      3. create restic-secret:

      apiVersion: v1
      data:
        AWS_ACCESS_KEY_ID: XXXXXXX
        AWS_SECRET_ACCESS_KEY: XXXXXXXXXXXXXXX
        RESTIC_PASSWORD: my-secure-restic-password
        RESTIC_REPOSITORY: s3:s3.amazonaws.com/oadpbucket119606
      kind: Secret
      metadata:
        name: restic-secret
        namespace: openshift-adp
      type: Opaque

      4. Set DPA CR:

      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        creationTimestamp: "2022-07-11T12:49:59Z"
        generation: 1
        name: ts-dpa
        namespace: openshift-adp
      spec:
        backupLocations:
        - velero:
            config:
              region: us-east-2
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: oadpbucket119606
              prefix: velero-e2e-f395364a-0117-11ed-ad74-902e163f806c
            provider: aws
        configuration:
          restic:
            enable: true
            podConfig:
              resourceAllocations: {}
          velero:
            defaultPlugins:
            - openshift
            - aws
            - kubevirt
            - csi
        features:
          enableDataMover: true
        podDnsConfig: {}
        snapshotLocations: []

      5. Create backup. After VSB's phase becomes SnapshotBackupDone, read the volume-snapshot-mover pod's logs.

      Actual results:

       

      Expected results:

       

      Additional info:

              spampatt@redhat.com Shubham Pampattiwar
              mperetz@redhat.com Maya Peretz
              Maya Peretz Maya Peretz
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: