Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-459

Common backup plugin is causing performance problems

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • OADP 1.1.0
    • None
    • openshift-plugin
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • In Progress
    • OADP Sprint 218
    • 1
    • 0
    • 0
    • 0
    • None

      The common backup plugin is attempting to read the registry route from a configmap on every run. This is leading to significant performance problems at scale. From slack:

      We’re doing some performance analysis on OADP backups trying to figure out why our backups are taking so long. We’ve deployed 1000 instances of busybox, which resulted in 1000 deployments, 1000 replicasets, and 1000 pods. We tried backing up all those objects with OADP and it took about 30 minutes. Individually the different sets (by type) of objects took 10 minutes. We’ve traced it down to the backup plugin in the openshift set of plugins. That plugin queries the backup registry route for every single deployment, replicaset or pod. If we eliminate the openshift set of plugins the backup time is reduced to 20-30 seconds.

      We should investigate either

      1. Read the route from an annotation on the backup CR that a user would manually set
      2. Write the route to a file in memory and attempt to read from it in every plugin

      It's also worth noting if we deprecate the image backup/restore workflow this should resolve this issue.

      Reported in https://coreos.slack.com/archives/C0144ECKUJ0/p1650317587855379

              tkaovila@redhat.com Tiger Kaovilai
              rhn-engineering-dymurray Dylan Murray
              Mordechai Lehrer Mordechai Lehrer
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: