Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-740

Restic failed to backup 200namespaces

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • No
    • 0
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown

      Description of problem:

      While try to backup 200namespace using Restic, the backup finished with "PartiallyFailed" status.

      Total "resticrepository" were created as the total namespaces.
      when running second cycle of the backup - The status is 'Completed'
      (No new "resticrepository" were created).

      restic backup with 100namespaces succeed.

      Version-Release number of selected component (if applicable):

      OCP: 4.10.21

      OADP: 1.1.0-64 (iib 293902)

      How reproducible:

      Each namespace includes:

      4 apps (4 deploy pods & 4 running pods)
      4 PVs
      4 PVCs

       

      Steps to Reproduce:
      1. Create 200 namespaces 
      2. Run restic backup with 200namespaces
      3.

      Actual results:

      restic backup finished with "PartiallyFailed" status

      Expected results:

      restic backup finished with "Completed" status at the 1st try

      Additional info:

      Logs: https://drive.google.com/drive/folders/1V7Cf_S6ZgQTZ-YZPTZCVulnsK1O9K00k?usp=sharing

       

      time="2022-08-17T09:06:49Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/backup-restic-ocs-cephfs-200ns-label logSource="pkg/backup/backup.go:413" name=mariadb-1-5d6hz
      time="2022-08-17T09:06:49Z" level=error msg="Error backing up item" backup=openshift-adp/backup-restic-ocs-cephfs-200ns-label error="timed out waiting for restic repository to become ready" error.file="/remote-source/src/github.com/vmware-tanzu/velero/pkg/restic/repository_ensurer.go:180" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*repositoryEnsurer).EnsureRepo" logSource="pkg/backup/backup.go:417" name=mariadb-1-5d6hz
      time="2022-08-17T09:06:49Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/backup-restic-ocs-cephfs-200ns-label logSource="pkg/backup/backup.go:413" name=mongodb-1-b6z9z
      time="2022-08-17T09:06:49Z" level=error msg="Error backing up item" backup=openshift-adp/backup-restic-ocs-cephfs-200ns-label error="restic repository is not ready: " error.file="/remote-source/src/github.com/vmware-tanzu/velero/pkg/restic/repository_ensurer.go:144" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*repositoryEnsurer).EnsureRepo" logSource="pkg/backup/backup.go:417" name=mongodb-1-b6z9z

            spampatt@redhat.com Shubham Pampattiwar
            dvaanunu@redhat.com David Vaanunu
            David Vaanunu David Vaanunu
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: