Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-1152

Backup Using "Restic" Failed on single ns with 5k pods exit on : " Fatal: invalid id \"6f87e619\": no matching ID found for prefix" ( multiple IDs)

    XMLWordPrintable

Details

    • False
    • Hide

      None

      Show
      None
    • False
    • QE - Ack
    • oadp-operator-container-1.2.0-44
    • ToDo
    • Yes
    • 0
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown

    Description

      Backup using "restic"  failed on single namespace with 5000 pods pvc size 32MB
      on "ocs-storagecluster-ceph-rbd"
      adp version:  iib:387699  - v1.2.0 ,   velero   v1.9.2-OADP

      the error " 

      the main error :  "Fatal: invalid id \"${ID}\": no matching ID found for prefix"  appears multiple times for more then one ID 
      the full ID's list can be found on the attached file ID's.list 

      update ( 9-1-23) 
      reproduce again  , failed in total two  cycles out of two 

      exit status 1 stderr: Fatal: invalid id \"6f87e619\": no matching ID found for prefix \"6f87e619\" 

      the full error from the velero log command : 

       time="2023-01-05T17:02:21Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/single-ns-5000-pods-restic-rbd-backup2 logSource="/remote-source/velero/app/pkg/backup/ba
      ckup.go:421" name=busybox-perf-single-ns-5000-pods-587
      time="2023-01-05T17:02:21Z" level=error msg="Error backing up item" backup=openshift-adp/single-ns-5000-pods-restic-rbd-backup2 error="pod volume backup failed: running backup, stderr=error r
      unning restic backup command restic backup --repo=s3:https://s3-openshift-storage.apps.vlan608.rdu2.scalelab.redhat.com/oadp-bucket/velero/restic/busybox-perf-single-ns-5000-pods --password-f
      ile=/tmp/credentials/openshift-adp/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic . --tag=backup-uid=1a2b4416-23f5-423f-b095-ebb3e3d7b9c1 --tag=ns=busybox-perf
      -single-ns-5000-pods --tag=pod=busybox-perf-single-ns-5000-pods-587 --tag=pod-uid=2e9aa1d4-5f55-4781-91fe-ea0915612c7c --tag=pvc-uid=73fca5e8-2675-4019-b899-ea0764de099d --tag=volume=busybox-
      perf-single-ns-5000-pods-587 --tag=backup=single-ns-5000-pods-restic-rbd-backup2 --host=velero --json --insecure-tls=true --parent=6f87e619 with error: exit status 1 stderr: Fatal: invalid id
       \"6f87e619\": no matching ID found for prefix \"6f87e619\"\n: error running restic backup command restic backup --repo=s3:https://s3-openshift-storage.apps.vlan608.rdu2.scalelab.redhat.com/o
      adp-bucket/velero/restic/busybox-perf-single-ns-5000-pods --password-file=/tmp/credentials/openshift-adp/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic . --tag
      =backup-uid=1a2b4416-23f5-423f-b095-ebb3e3d7b9c1 --tag=ns=busybox-perf-single-ns-5000-pods --tag=pod=busybox-perf-single-ns-5000-pods-587 --tag=pod-uid=2e9aa1d4-5f55-4781-91fe-ea0915612c7c --
      tag=pvc-uid=73fca5e8-2675-4019-b899-ea0764de099d --tag=volume=busybox-perf-single-ns-5000-pods-587 --tag=backup=single-ns-5000-pods-restic-rbd-backup2 --host=velero --json --insecure-tls=true
       --parent=6f87e619 with error: exit status 1 stderr: Fatal: invalid id \"6f87e619\": no matching ID found for prefix \"6f87e619\"\n" error.file="/remote-source/velero/app/pkg/podvolume/backup
      per.go:223" error.function="github.com/vmware-tanzu/velero/pkg/podvolume.(*backupper).BackupPodVolumes" logSource="/remote-source/velero/app/pkg/backup/backup.go:425" name=busybox-perf-single
      -ns-5000-pods-587
      

      velero backup describe : 

       Name:         single-ns-5000-pods-restic-rbd-backup2
      Namespace:    openshift-adp
      Labels:       velero.io/storage-location=example-velero-1
      Annotations:  velero.io/source-cluster-k8s-gitversion=v1.24.0+3882f8f
                    velero.io/source-cluster-k8s-major-version=1
                    velero.io/source-cluster-k8s-minor-version=24Phase:  PartiallyFailed (run `velero backup logs single-ns-5000-pods-restic-rbd-backup2` for more information)Errors:    2365
      Warnings:  0Namespaces:
        Included:  busybox-perf-single-ns-5000-pods
        Excluded:  <none>Resources:
        Included:        *
        Excluded:        <none>
        Cluster-scoped:  autoLabel selector:  <none>Storage Location:  example-velero-1Velero-Native Snapshot PVs:  falseTTL:  720h0m0sCSISnapshotTimeout:  10m0sHooks:  <none>Backup Format Version:  1.1.0Started:    2023-01-05 08:26:05 +0000 UTC
      Completed:  2023-01-05 17:33:12 +0000 UTCExpiration:  2023-02-04 08:26:05 +0000 UTCTotal items to be backed up:  40022
      Items backed up:              40022Velero-Native Snapshots: <none included>restic Backups (specify --details for more information):
        Completed:  2635
        Failed:     2365
      

      Attachments

        1. adp-1152_velero.log
          58.51 MB
        2. IDs.list
          243 kB
        3. single-ns-5000-pods-restic-rbd-backup2.log
          69.91 MB
        4. velero-5bbdd484c9-7tsfw.log
          28.95 MB
        5. velero-67b85f9b48-g64st.log
          34 kB

        Activity

          People

            tkaovila@redhat.com Tiger Kaovilai
            tzahia Tzahi Ashkenazi
            Tzahi Ashkenazi Tzahi Ashkenazi
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: