Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-4817

Imagestream backups are partially failing when dpa is configured with caCert

XMLWordPrintable

    • 10
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • 0
    • 0.000
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • None

      Description of problem:

      Imagestreams backups are partially failing when the dpa is configured with caCert.  Specially with buildconfig related applications. 

       

      Version-Release number of selected component (if applicable):
      OADP 1.4.1-28

       

      How reproducible:
      Always 

       

      Steps to Reproduce:
      1. Create a dpa with caCert spec. 

      $ oc get dpa ts-dpa -o yaml
      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        creationTimestamp: "2024-09-09T07:20:03Z"
        generation: 5
        name: ts-dpa
        namespace: openshift-adp
        resourceVersion: "115349"
        uid: 56da090a-35e3-4a32-8ba4-7c48c76d3898
      spec:
        backupLocations:
        - velero:
            config:
              insecureSkipTLSVerify: "false"
              profile: noobaa
              region: us-east-1
              s3ForcePathStyle: "true"
              s3Url: https://s3-openshift-storage.apps.oadp-96031.qe.gcp.devcluster.openshift.com
              caCert: <cert>
            credential:
              key: cloud
              name: cloud-credentials
            default: true
            objectStorage:
              bucket: oadp96031c66l6
              prefix: velero-e2e-a2c534da-6e7b-11ef-95c5-845cf3eff33a
            provider: aws
        configuration:
          nodeAgent:
            enable: true
            podConfig:
              resourceAllocations: {}
            uploaderType: kopia
          velero:
            defaultPlugins:
            - openshift
            - aws
            - kubevirt
            logLevel: debug
        podDnsConfig: {}
        snapshotLocations: []
      status:
        conditions:
        - lastTransitionTime: "2024-09-09T07:20:03Z"
          message: Reconcile complete
          reason: Complete
          status: "True"
          type: Reconciled

      2. Deploy ocp-django application 

      $ oc get pod -n ocp-django
      NAME                              READY   STATUS      RESTARTS   AGE
      django-psql-persistent-1-build    0/1     Completed   0          141m
      django-psql-persistent-1-deploy   0/1     Completed   0          140m
      django-psql-persistent-1-lppbj    1/1     Running     0          140m
      postgresql-1-deploy               0/1     Completed   0          141m
      postgresql-1-f5hw8                1/1     Running     0          140m

      3. Trigger a backup 

      Actual results:

      Imagestream backup is partially failing if bsl is configured with CaCert. 

       

       

      $ oc get backup test-backup -o yaml
      apiVersion: velero.io/v1
      kind: Backup
      metadata:
        annotations:
          velero.io/resource-timeout: 10m0s
          velero.io/source-cluster-k8s-gitversion: v1.29.8+f10c92d
          velero.io/source-cluster-k8s-major-version: "1"
          velero.io/source-cluster-k8s-minor-version: "29"
        creationTimestamp: "2024-09-09T08:24:20Z"
        generation: 7
        labels:
          velero.io/storage-location: ts-dpa-1
        name: test-backup
        namespace: openshift-adp
        resourceVersion: "123047"
        uid: d74129fa-efa2-4cec-92a4-ec3a4b808411
      spec:
        csiSnapshotTimeout: 10m0s
        defaultVolumesToFsBackup: false
        includedNamespaces:
        - ocp-django
        itemOperationTimeout: 4h0m0s
        snapshotMoveData: false
        storageLocation: ts-dpa-1
        ttl: 720h0m0s
      status:
        completionTimestamp: "2024-09-09T08:26:13Z"
        errors: 1
        expiration: "2024-10-09T08:24:20Z"
        formatVersion: 1.1.0
        hookStatus: {}
        phase: PartiallyFailed
        progress:
          itemsBackedUp: 91
          totalItems: 91
        startTimestamp: "2024-09-09T08:24:20Z"
        version: 1 
      

       

      Expected results:

       

      Imagestream backup should be successful. 

       

      Additional info:

      Attached velero logs below:- 

      time="2024-09-09T08:26:10Z" level=debug msg="time=\"2024-09-09T08:26:10.803147562Z\" level=error msg=\"response completed with error\" environment=development err.code=unknown err.detail=\"s3aws: RequestError: send request failed\\ncaused by: Get \\\"https://s3-openshift-storage.apps.oadp-96031.qe.gcp.devcluster.openshift.com/oadp96031c66l6/docker/registry/v2/repositories/ocp-django/django-psql-persistent/_layers/sha256/a9644f686a26ba8fb5115ea69971272c42a1b776e4c63ae77667d7d59f5a4094/link\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\" err.message=\"unknown error\" go.version=\"go1.22.5 (Red Hat 1.22.5-1.el9) X:strictfipsruntime\" http.request.host= http.request.id=76ccf293-7610-4fc9-a945-a7884d22d96d http.request.method=HEAD http.request.remoteaddr= http.request.uri= http.request.useragent=\"containers/5.27.0 (github.com/containers/image)\" http.response.contenttype=application/json http.response.duration=373.38021ms http.response.status=500 http.response.written=104 instance.id=f7a6e6b6-99c0-49c0-856c-25a08d445a67 service=registry vars.digest=\"sha256:a9644f686a26ba8fb5115ea69971272c42a1b776e4c63ae77667d7d59f5a4094\" vars.name=ocp-django/django-psql-persistent version=v3.0.0+unknown" backup=openshift-adp/test-backup cmd=/plugins/velero-plugins logSource="/remote-source/velero/app/pkg/plugin/clientmgmt/process/logrus_adapter.go:75" pluginName=velero-plugins
      time="2024-09-09T08:26:10Z" level=error msg="Error backing up item" backup=openshift-adp/test-backup error="error executing custom action (groupResource=imagestreams.image.openshift.io, namespace=ocp-django, name=django-psql-persistent): rpc error: code = Unknown desc = trying to reuse blob sha256:4bb16177726caec64d3e9592403e6b642602b0b5b9c3ce3efeb9c00e117772ca at destination: failed to read from destination repository ocp-django/django-psql-persistent: 500 (Internal Server Error)" error.file="/remote-source/velero/app/pkg/backup/item_backupper.go:400" error.function="github.com/vmware-tanzu/velero/pkg/backup.(*itemBackupper).executeActions" logSource="/remote-source/velero/app/pkg/backup/backup.go:511" name=django-psql-persistent

              rhn-engineering-mpryc Michal Pryc
              rhn-support-prajoshi Prasad Joshi
              Prasad Joshi Prasad Joshi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: