-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
False
-
-
False
-
ToDo
-
-
-
0
-
0
-
Very Likely
-
0
-
Customer Escalated, Customer Facing
-
None
-
Unset
-
Unknown
-
No
- ACM 2.7 had finished perf tests for a hub managing 3510 SNOs clusters and the numbers for restore is very high compared with the backup
Scenario:
- Backup all ACM resources, time to backup about 4min ( 28202 resources )
- Run a restore on the same hub, restoring the above backup ( so all resources already exist, they are just updated and tagged with the restore label) : 1h12min - this time is too high.. especially since this restore is executed on a passive hub and would be re-run as soon as new backups are available
To simplify the tests I run a simplified set of perf tests :
- created ~32705 secrets
- backup then restore them
Results for restore are listed below
1. Ask to update existing (existingResourcePolicy: update)
-
- update existing
apiVersion: velero.io/v1
kind: Restore
metadata:
creationTimestamp: '2023-01-04T22:08:47Z'
generation: 4895
managedFields:
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
'f:spec':
.: {}
'f:backupName': {}
'f:existingResourcePolicy': {}
manager: Mozilla
operation: Update
time: '2023-01-04T22:08:47Z'
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
'f:spec':
'f:excludedResources': {}
'f:scheduleName': {}
'f:status':
.: {}
'f:completionTimestamp': {}
'f:phase': {}
'f:progress':
.: {}
'f:itemsRestored': {}
'f:totalItems': {}
'f:startTimestamp': {}
manager: velero-server
operation: Update
time: '2023-01-04T23:32:43Z'
name: 1-update-restore-acm-credentials-schedule-20230104205535
namespace: open-cluster-management-backup
resourceVersion: '845146'
uid: eaafc392-5ea7-4418-a51b-eabbea353ed8
spec:
backupName: acm-credentials-schedule-20230104205535
excludedResources:
- nodes
- events
- events.events.k8s.io
- backups.velero.io
- restores.velero.io
- resticrepositories.velero.io
existingResourcePolicy: update
scheduleName: acm-credentials-schedule
status:
completionTimestamp: '2023-01-04T23:32:43Z'
phase: Completed
progress:
itemsRestored: 32694
totalItems: 32694
startTimestamp: '2023-01-04T22:08:47Z'
- update existing
2. * do not update existing
apiVersion: velero.io/v1
kind: Restore
metadata:
creationTimestamp: '2023-01-04T21!https://emoji.slack-edge.com/T030G10V24F/00/72db96a6f75c0f77.png!47Z'
generation: 3267
managedFields:
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
'f:spec':
.: {}
'f:backupName': {}
manager: Mozilla
operation: Update
time: '2023-01-04T21!https://emoji.slack-edge.com/T030G10V24F/00/72db96a6f75c0f77.png!47Z'
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
'f:spec':
'f:excludedResources': {}
'f:scheduleName': {}
'f:status':
.: {}
'f:completionTimestamp': {}
'f:phase': {}
'f:progress':
.: {}
'f:itemsRestored': {}
'f:totalItems': {}
'f:startTimestamp': {}
manager: velero-server
operation: Update
time: '2023-01-04T21:56:49Z'
name: 1-restore-acm-credentials-schedule-20230104205535
namespace: open-cluster-management-backup
resourceVersion: '679645'
uid: 40c01523-a098-42b4-a204-c28e4f43450d
spec:
backupName: acm-credentials-schedule-20230104205535
excludedResources:
- nodes
- events
- events.events.k8s.io
- backups.velero.io
- restores.velero.io
- resticrepositories.velero.io
scheduleName: acm-credentials-schedule
status:
completionTimestamp: '2023-01-04T21:56:49Z'
phase: Completed
progress:
itemsRestored: 32694
totalItems: 32694
startTimestamp: '2023-01-04T24: 00 : 47Z'
3. * clean restore ( no data on hub )
apiVersion: velero.io/v1
kind: Restore
metadata:
creationTimestamp: '2023-01-05T00:18:29Z'
generation: 1636
managedFields:
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
'f:spec':
.: {}
'f:backupName': {}
'f:existingResourcePolicy': {}
manager: Mozilla
operation: Update
time: '2023-01-05T00:18:29Z'
- apiVersion: velero.io/v1
fieldsType: FieldsV1
fieldsV1:
'f:spec':
'f:excludedResources': {}
'f:scheduleName': {}
'f:status':
.: {}
'f:completionTimestamp': {}
'f:phase': {}
'f:progress':
.: {}
'f:itemsRestored': {}
'f:totalItems': {}
'f:startTimestamp': {}
manager: velero-server
operation: Update
time: '2023-01-05T00!https://emoji.slack-edge.com/T030G10V24F/46/022dacef2450ff96.png!40Z'
name: 1-clean-hub-restore-acm-credentials-schedule-20230104205535
namespace: open-cluster-management-backup
resourceVersion: '1010580'
uid: 1fa43d2b-07a8-410f-9477-eafd3cecc0ea
spec:
backupName: acm-credentials-schedule-20230104205535
excludedResources:
- nodes
- events
- events.events.k8s.io
- backups.velero.io
- restores.velero.io
- resticrepositories.velero.io
existingResourcePolicy: update
scheduleName: acm-credentials-schedule
status
completionTimestamp: '2023-01-05T00: 46 : 40Z'
phase: Completed
progress:
itemsRestored: 32694
totalItems: 32694
startTimestamp: '2023-01-05T00:18:29Z'
- clones
-
OADP-1167 Performance issues when restoring 30k resources
- Closed