Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-4855

Kopia leaving cache on worker node

XMLWordPrintable

    • 3
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • Important
    • 8
    • 2.6666666666666665
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • None

      Description of problem:

      When 'ephemeral-storage' is configured and running filesystem restore,

      the "/var" partition on the worker increases. After the backup is completed the "cache" is left on the worker node and not deleted.

      Only node-agent restart clears the "cache"

      Version-Release number of selected component (if applicable):

      OCP - 4.16.9

      ODF - 4.16.1
      OADP - 1.4.1-28
      TC - Single namespace, Singe pod, PV size 1.2T , Total usage 1T (100 files x 10GB each, 2.4.1.9) 

      How reproducible:

       

      Steps to Reproduce:
      1. Create a namespace with 1 pod, large PV & usage
      2. Set DPA with "ephemeral-storage"
      3. Run backup & restore
      4. During restore monitor the worker node of node-agent (df -h /var)
      5. "/var" capacity increases during the restore
      6. Wait 30-60min , the "cache" is not deleted
      7. restart the node-agent pod - "cache" deleted.

      Actual results:

      "Cache" left on the worker node

      Expected results:

      "Cache" deleted after restore completed

      Additional info:

      DPA setup:
        configuration:
          nodeAgent:
            enable: true
            podConfig:
              resourceAllocations:
                limits:
                  ephemeral-storage: 400Gi
                requests:
                  ephemeral-storage: 400Gi
            uploaderType: kopia

      Worker node: (back to 6% after node-agent pod restart)
      /dev/sdb4       447G   93G  354G  21% /var
      /dev/sdb4       447G   94G  354G  21% /var
      /dev/sdb4       447G   94G  354G  21% /var
      /dev/sdb4       447G   94G  354G  21% /var
      /dev/sdb4       447G   94G  354G  21% /var
      /dev/sdb4       447G   94G  354G  21% /var
      /dev/sdb4       447G   94G  354G  21% /var
      /dev/sdb4       447G   94G  354G  21% /var
      /dev/sdb4       447G   93G  354G  21% /var
      /dev/sdb4       447G  138G  309G  31% /var
      /dev/sdb4       447G  178G  269G  40% /var
      /dev/sdb4       447G  210G  237G  47% /var
      /dev/sdb4       447G  241G  206G  54% /var
      /dev/sdb4       447G   25G  422G   6% /var
      /dev/sdb4       447G   25G  422G   6% /var
      /dev/sdb4       447G   25G  422G   6% /var
      /dev/sdb4       447G   25G  422G   6% /var
      /dev/sdb4       447G   25G  422G   6% /var
      /dev/sdb4       447G   25G  422G   6% /var
      /dev/sdb4       447G   25G  422G   6% /var
      /dev/sdb4       447G   25G  422G   6% /var
      /dev/sdb4       447G   52G  395G  12% /var
      /dev/sdb4       447G   95G  352G  22% /var
      /dev/sdb4       447G  130G  317G  30% /var
      /dev/sdb4       447G  165G  282G  37% /var
      /dev/sdb4       447G  202G  245G  46% /var
      /dev/sdb4       447G  237G  211G  53% /var
      /dev/sdb4       447G   25G  422G   6% /var

            wnstb Wes Hayutin
            dvaanunu@redhat.com David Vaanunu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: