Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-2733

[Upstream] Kopia Backup failed on 5k pods on sn , exit on "failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • OADP 1.3.2
    • OADP 1.3.0
    • kopia
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • No
    • 0
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown

      Description of problem:  https://github.com/vmware-tanzu/velero/issues/682

      Tracking bug for upstream Velero 1.12

      Running kopia backup .
      using  SN with 5000 pods & PVC size 32MB storage class = cephrbd
      backup CR failed exit on : "PartiallyFailed"

      exit on 

                   name: /busybox-perf-single-ns-5000-pods-989 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-99 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-990 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-991 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-992 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-993 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-994 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-995 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-996 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-997 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-998 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
                   name: /busybox-perf-single-ns-5000-pods-999 error: /failed to list daemonset pods: client rate limiter Wait returned an error: context deadline exceeded
      

      Version-Release number of selected component (if applicable)

      OCP 4.11.43

      ODF  4.11.10

      Upstream Velero 1.12

      How reproducible:

       

      Steps to Reproduce:
      1. create sn with 5000 pods pvc size 32MB  using cephfs
      2. create backup using kopia 
      3. backup failed  "PartiallyFailed"

      Actual results:

       

      Expected results:

       

      Additional info:

            wnstb Wes Hayutin
            tzahia Tzahi Ashkenazi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: