Based on the initial findings in https://access.redhat.com/support/cases/#/case/03826642

Engineering is requesting the following performance 2 scenarios.

1. OCP, ODF, OpenShift-Virt, OADP-1.4.x.

• A single namespace w/ the following
• 30-50 VM's w/ two volumes attached
  • ODF Volumes
  • Block Volume-1: 30-60 GB of densely populated data
  • Block Volume-2: 100-150 GB of less populated data
• ODF s3 endpoint on the same or remote cluster

2. OCP, ODF, OpenShift-Virt, OADP-1.4.x

• A single namespace w/ the following
• 30-50 VM's w/ two volumes attached
  • ODF Volumes
  • Block Volume-1: 30-60 GB of densely populated data
  • Block Volume-2: 100-150 GB of less populated data
• Remote S3 bucket that is not ODF, MCG, nooba.  Engineering can provide a bucket if needed.

Engineering is requesting performance data to including time to complete and tuning recommendations. 

FYI.. thus far the customer ALSO has issues on OADP-1.3.2.  Debug is in progress and no bugs have been opened yet.

Customer Related Details:

• Ally Bank has namespaces with more VM's than the above scenario
• Ally Bank backs up VM's off cluster daily in an 8 hour window
• Ally Bank is utilizing DataMover, not filesystem backups.
• Ally Bank IS or will be using multiple current datauploads via the following config the node-agent concurrency documentation: https://velero.io/docs/main/node-agent-concurrency/

Workflow requirements:

• Do not cleanup or remove backups during the interations.
• Collect the performance stats from the first  DM backup.
  • Size the data in the backup kopia repository (Example: aws s3 ls {}summarize{-}  -recursive --human-readable  s3://cvpbucketuswest2/velero/kopia/cirros-test/)
  • Calculate the time for dataupload to complete ( completionTimestamp - creationTimestamp)
• Add data to all the VM's 
  • perhaps 1GB
• Execute another DM backup
  • Collect the same size and timing 
• Add data to all the VM's
  • Perhaps another 1GB for consistency 
  • execute the DM backup again
  • Collect the same size and timing
• Add a much larger set of data 10GB
  • execute the DM backup again
  • Collect the same size and timing

• kopia https://hackmd.io/Jc7oHHwqQxKD6M1VSqCrrw
  • This may be interesting to the perf team, In this CUSTOM velero quay.io/sseago/velero:new-kopia-encryption
    • The following has been set. https://github.com/openshift/velero/pull/329
    • The patch was delivered to Ally in hopes of improving performance.