Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-1167

Performance issues when restoring 30k resources

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • oadp-operator-bundle-container-1.3.0-131
    • ToDo
    • 0
    • 0
    • Very Likely
    • 0
    • Customer Escalated, Customer Facing
    • None
    • Unset
    • Unknown
    • No

      1. ACM 2.7 had finished perf tests for a hub managing 3510 SNOs clusters and the numbers for restore is very high compared with the backup
        Scenario: 
      • Backup all ACM resources, time to backup about 4min ( 28202 resources )
      • Run a restore on the same hub, restoring the above backup ( so all resources already exist, they are just updated and tagged with the restore label) : 1h12min - this time is too high.. especially since this restore is executed on a passive hub and would be re-run as soon as new backups are available

      To simplify the tests I run a simplified set of perf tests :
          - created ~32705 secrets

      • backup then restore them

      Results for restore are listed below

      1. Ask to update existing (existingResourcePolicy: update)

        • update existing
          apiVersion: velero.io/v1
          kind: Restore
          metadata:
           creationTimestamp: '2023-01-04T22:08:47Z'
           generation: 4895
           managedFields:
            - apiVersion: velero.io/v1
             fieldsType: FieldsV1
             fieldsV1:
              'f:spec':
               .: {}
               'f:backupName': {}
               'f:existingResourcePolicy': {}
             manager: Mozilla
             operation: Update
             time: '2023-01-04T22:08:47Z'
            - apiVersion: velero.io/v1
             fieldsType: FieldsV1
             fieldsV1:
              'f:spec':
               'f:excludedResources': {}
               'f:scheduleName': {}
              'f:status':
               .: {}
               'f:completionTimestamp': {}
               'f:phase': {}
               'f:progress':
                .: {}
                'f:itemsRestored': {}
                'f:totalItems': {}
               'f:startTimestamp': {}
             manager: velero-server
             operation: Update
             time: '2023-01-04T23:32:43Z'
           name: 1-update-restore-acm-credentials-schedule-20230104205535
           namespace: open-cluster-management-backup
           resourceVersion: '845146'
           uid: eaafc392-5ea7-4418-a51b-eabbea353ed8
          spec:
           backupName: acm-credentials-schedule-20230104205535
           excludedResources:
            - nodes
            - events
            - events.events.k8s.io
            - backups.velero.io
            - restores.velero.io
            - resticrepositories.velero.io
           existingResourcePolicy: update
           scheduleName: acm-credentials-schedule
          status:
           completionTimestamp: '2023-01-04T23:32:43Z'
           phase: Completed
           progress:
            itemsRestored: 32694
            totalItems: 32694
           startTimestamp: '2023-01-04T22:08:47Z'

      2.  * do not update existing

      apiVersion: velero.io/v1
      kind: Restore
      metadata:
       creationTimestamp: '2023-01-04T21!https://emoji.slack-edge.com/T030G10V24F/00/72db96a6f75c0f77.png!47Z'
       generation: 3267
       managedFields:
        - apiVersion: velero.io/v1
         fieldsType: FieldsV1
         fieldsV1:
          'f:spec':
           .: {}
           'f:backupName': {}
         manager: Mozilla
         operation: Update
         time: '2023-01-04T21!https://emoji.slack-edge.com/T030G10V24F/00/72db96a6f75c0f77.png!47Z'
        - apiVersion: velero.io/v1
         fieldsType: FieldsV1
         fieldsV1:
          'f:spec':
           'f:excludedResources': {}
           'f:scheduleName': {}
          'f:status':
           .: {}
           'f:completionTimestamp': {}
           'f:phase': {}
           'f:progress':
            .: {}
            'f:itemsRestored': {}
            'f:totalItems': {}
           'f:startTimestamp': {}
         manager: velero-server
         operation: Update
         time: '2023-01-04T21:56:49Z'
       name: 1-restore-acm-credentials-schedule-20230104205535
       namespace: open-cluster-management-backup
       resourceVersion: '679645'
       uid: 40c01523-a098-42b4-a204-c28e4f43450d
      spec:
       backupName: acm-credentials-schedule-20230104205535
       excludedResources:
        - nodes
        - events
        - events.events.k8s.io
        - backups.velero.io
        - restores.velero.io
        - resticrepositories.velero.io
       scheduleName: acm-credentials-schedule
      status:
       completionTimestamp: '2023-01-04T21:56:49Z'
       phase: Completed
       progress:
        itemsRestored: 32694
        totalItems: 32694
      startTimestamp: '2023-01-04T24: 00 : 47Z'

      3.  * clean restore ( no data on hub )

      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        creationTimestamp: '2023-01-05T00:18:29Z'
        generation: 1636
        managedFields:
          - apiVersion: velero.io/v1
            fieldsType: FieldsV1
            fieldsV1:
              'f:spec':
                .: {}
                'f:backupName': {}
                'f:existingResourcePolicy': {}
            manager: Mozilla
            operation: Update
            time: '2023-01-05T00:18:29Z'
          - apiVersion: velero.io/v1
            fieldsType: FieldsV1
            fieldsV1:
              'f:spec':
                'f:excludedResources': {}
                'f:scheduleName': {}
              'f:status':
                .: {}
                'f:completionTimestamp': {}
                'f:phase': {}
                'f:progress':
                  .: {}
                  'f:itemsRestored': {}
                  'f:totalItems': {}
                'f:startTimestamp': {}
            manager: velero-server
            operation: Update
            time: '2023-01-05T00!https://emoji.slack-edge.com/T030G10V24F/46/022dacef2450ff96.png!40Z'
        name: 1-clean-hub-restore-acm-credentials-schedule-20230104205535
        namespace: open-cluster-management-backup
        resourceVersion: '1010580'
        uid: 1fa43d2b-07a8-410f-9477-eafd3cecc0ea
      spec:
        backupName: acm-credentials-schedule-20230104205535
        excludedResources:
          - nodes
          - events
          - events.events.k8s.io
          - backups.velero.io
          - restores.velero.io
          - resticrepositories.velero.io
        existingResourcePolicy: update
        scheduleName: acm-credentials-schedule
      status
        completionTimestamp: '2023-01-05T00: 46 : 40Z'
        phase: Completed
        progress:
          itemsRestored: 32694
          totalItems: 32694
        startTimestamp: '2023-01-05T00:18:29Z'

              sseago Scott Seago
              vbirsan@redhat.com Valentina Birsan
              David Vaanunu David Vaanunu
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: