Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-2204

Cloudstorage API: DataMover backups fails with error secret not found

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Undefined Undefined
    • None
    • OADP 1.2.0, OADP 1.1.6
    • data-mover
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • No
    • 0
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown

      Description of problem: 

      DataMover backups are partially failing due to missing volsync restic secret resource. It seems the secret is not getting created when DPA is created with CloudStorage spec.

      Attached error log below. 

       

      $ oc logs velero-5bd69b9f8f-68jn4 | grep error
time="2023-07-06T09:35:15Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1)" controller=backup-storage-location logSource="/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:194"
time="2023-07-06T09:54:40Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/mysql-582391ed-1be0-11ee-b197-845cf3eff33a logSource="/remote-source/velero/app/pkg/backup/backup.go:431" name=mysql-66865fdf8c-c4rcc
time="2023-07-06T09:54:40Z" level=error msg="Error backing up item" backup=openshift-adp/mysql-582391ed-1be0-11ee-b197-845cf3eff33a error="error executing custom action (groupResource=volumesnapshotcontents.snapshot.storage.k8s.io, namespace=, name=snapcontent-7e9cbee7-912c-4d63-bb49-5483e9de78d4): rpc error: code = Unknown desc = secrets \"ts-dpa-1-volsync-restic\" not found" logSource="/remote-source/velero/app/pkg/backup/backup.go:435" name=mysql-66865fdf8c-c4rcc
time="2023-07-06T09:55:20Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/mysql-582391ed-1be0-11ee-b197-845cf3eff33a logSource="/remote-source/velero/app/pkg/backup/backup.go:431" name=mysql-1
time="2023-07-06T09:55:20Z" level=error msg="Error backing up item" backup=openshift-adp/mysql-582391ed-1be0-11ee-b197-845cf3eff33a error="error executing custom action (groupResource=volumesnapshotcontents.snapshot.storage.k8s.io, namespace=, name=snapcontent-66378699-d95e-4d28-a140-4c0d7759f9b6): rpc error: code = Unknown desc = secrets \"ts-dpa-1-volsync-restic\" not found" logSource="/remote-source/velero/app/pkg/backup/backup.go:435" name=mysql-1
      $ oc get secret | grep ts-dpa
$ 

      Version-Release number of selected component (if applicable):
      OADP 1.2.0 
      ROSA 4.13
      Bucket: AWS s3

       

      How reproducible:
      Always

       

      Steps to Reproduce:
      1. Deploy a STS enabled cluster. 
      2. Create a DPA using cloudStorage spec.

      apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: ts-dpa
  namespace: openshift-adp
spec:
  backupLocations:
  - bucket:
      cloudStorageRef:
        name: oadpbucketoc
      config:
        enableSharedConfig: "true"
        region: us-east-1
      credential:
        key: cloud
        name: cloud-credentials
      default: true
  configuration:
    restic:
      podConfig:
        resourceAllocations: {}
    velero:
      defaultPlugins:
      - openshift
      - aws
      - kubevirt
      - csi
      - vsm
  features:
    dataMover:
      enable: true
  podDnsConfig: {}
  snapshotLocations: []
status:
  conditions:
  - lastTransitionTime: "2023-07-06T09:34:49Z"
    message: Reconcile complete
    reason: Complete
    status: "True"
    type: Reconciled

      3. Deploy a stateful application.

      4. Execute backup

      oc get backup mysql-582391ed-1be0-11ee-b197-845cf3eff33a -o yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: mysql-582391ed-1be0-11ee-b197-845cf3eff33a
  namespace: openshift-adp
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  hooks: {}
  includedNamespaces:
  - mysql-persistent-with-datamover
  itemOperationTimeout: 1h0m0s
  metadata: {}
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
status:
  completionTimestamp: "2023-07-06T09:55:21Z"
  csiVolumeSnapshotsAttempted: 2
  csiVolumeSnapshotsCompleted: 2
  errors: 2
  expiration: "2023-08-05T09:54:09Z"
  formatVersion: 1.1.0
  phase: PartiallyFailed
  startTimestamp: "2023-07-06T09:54:09Z"
  version: 1

       

      Actual results:

      DataMover backups are partially failing due to missing volsync restic secret.

      $ oc logs velero-5bd69b9f8f-68jn4 | grep error
time="2023-07-06T09:35:15Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1)" controller=backup-storage-location logSource="/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:194"
time="2023-07-06T09:54:40Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/mysql-582391ed-1be0-11ee-b197-845cf3eff33a logSource="/remote-source/velero/app/pkg/backup/backup.go:431" name=mysql-66865fdf8c-c4rcc
time="2023-07-06T09:54:40Z" level=error msg="Error backing up item" backup=openshift-adp/mysql-582391ed-1be0-11ee-b197-845cf3eff33a error="error executing custom action (groupResource=volumesnapshotcontents.snapshot.storage.k8s.io, namespace=, name=snapcontent-7e9cbee7-912c-4d63-bb49-5483e9de78d4): rpc error: code = Unknown desc = secrets \"ts-dpa-1-volsync-restic\" not found" logSource="/remote-source/velero/app/pkg/backup/backup.go:435" name=mysql-66865fdf8c-c4rcc
time="2023-07-06T09:55:20Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/mysql-582391ed-1be0-11ee-b197-845cf3eff33a logSource="/remote-source/velero/app/pkg/backup/backup.go:431" name=mysql-1
time="2023-07-06T09:55:20Z" level=error msg="Error backing up item" backup=openshift-adp/mysql-582391ed-1be0-11ee-b197-845cf3eff33a error="error executing custom action (groupResource=volumesnapshotcontents.snapshot.storage.k8s.io, namespace=, name=snapcontent-66378699-d95e-4d28-a140-4c0d7759f9b6): rpc error: code = Unknown desc = secrets \"ts-dpa-1-volsync-restic\" not found" logSource="/remote-source/velero/app/pkg/backup/backup.go:435" name=mysql-1

       

      Expected results: 
      DataMover backups should be successful. 

       

      Additional info:

      $ oc get secret
NAME                                               TYPE                                  DATA   AGE
builder-dockercfg-tgnn4                            kubernetes.io/dockercfg               1      111m
builder-token-pn2ds                                kubernetes.io/service-account-token   4      111m
cloud-credentials                                  Opaque                                1      35m
default-dockercfg-fg8hb                            kubernetes.io/dockercfg               1      111m
default-token-kplj9                                kubernetes.io/service-account-token   4      111m
deployer-dockercfg-55kcq                           kubernetes.io/dockercfg               1      111m
deployer-token-wvpxv                               kubernetes.io/service-account-token   4      111m
dm-credential                                      Opaque                                1      35m
openshift-adp-controller-manager-dockercfg-47lff   kubernetes.io/dockercfg               1      111m
openshift-adp-controller-manager-token-lj4nf       kubernetes.io/service-account-token   4      111m
velero-dockercfg-szdn8                             kubernetes.io/dockercfg               1      110m
velero-repo-credentials                            Opaque                                1      85m
velero-token-5pdmf                                 kubernetes.io/service-account-token   4      110m

            wnstb Wes Hayutin
            rhn-support-prajoshi Prasad Joshi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: