-
Sub-task
-
Resolution: Unresolved
-
Undefined
-
None
-
None
-
None
-
False
-
-
False
-
ToDo
-
-
-
0
-
0
-
Very Likely
-
0
-
None
-
Unset
-
Unknown
Description of problem:
Imagestreams backups are partially failing when the dpa is configured with caCert. Specially with buildconfig related applications.
Version-Release number of selected component (if applicable):
OADP 1.4.1-28
How reproducible:
Always
Steps to Reproduce:
1. Create a dpa with caCert spec.
$ oc get dpa ts-dpa -o yaml apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: creationTimestamp: "2024-09-09T07:20:03Z" generation: 5 name: ts-dpa namespace: openshift-adp resourceVersion: "115349" uid: 56da090a-35e3-4a32-8ba4-7c48c76d3898 spec: backupLocations: - velero: config: insecureSkipTLSVerify: "false" profile: noobaa region: us-east-1 s3ForcePathStyle: "true" s3Url: https://s3-openshift-storage.apps.oadp-96031.qe.gcp.devcluster.openshift.com caCert: <cert> credential: key: cloud name: cloud-credentials default: true objectStorage: bucket: oadp96031c66l6 prefix: velero-e2e-a2c534da-6e7b-11ef-95c5-845cf3eff33a provider: aws configuration: nodeAgent: enable: true podConfig: resourceAllocations: {} uploaderType: kopia velero: defaultPlugins: - openshift - aws - kubevirt logLevel: debug podDnsConfig: {} snapshotLocations: [] status: conditions: - lastTransitionTime: "2024-09-09T07:20:03Z" message: Reconcile complete reason: Complete status: "True" type: Reconciled
2. Deploy ocp-django application
$ oc get pod -n ocp-django NAME READY STATUS RESTARTS AGE django-psql-persistent-1-build 0/1 Completed 0 141m django-psql-persistent-1-deploy 0/1 Completed 0 140m django-psql-persistent-1-lppbj 1/1 Running 0 140m postgresql-1-deploy 0/1 Completed 0 141m postgresql-1-f5hw8 1/1 Running 0 140m
3. Trigger a backup
Actual results:
Imagestream backup is partially failing if bsl is configured with CaCert.
$ oc get backup test-backup -o yaml apiVersion: velero.io/v1 kind: Backup metadata: annotations: velero.io/resource-timeout: 10m0s velero.io/source-cluster-k8s-gitversion: v1.29.8+f10c92d velero.io/source-cluster-k8s-major-version: "1" velero.io/source-cluster-k8s-minor-version: "29" creationTimestamp: "2024-09-09T08:24:20Z" generation: 7 labels: velero.io/storage-location: ts-dpa-1 name: test-backup namespace: openshift-adp resourceVersion: "123047" uid: d74129fa-efa2-4cec-92a4-ec3a4b808411 spec: csiSnapshotTimeout: 10m0s defaultVolumesToFsBackup: false includedNamespaces: - ocp-django itemOperationTimeout: 4h0m0s snapshotMoveData: false storageLocation: ts-dpa-1 ttl: 720h0m0s status: completionTimestamp: "2024-09-09T08:26:13Z" errors: 1 expiration: "2024-10-09T08:24:20Z" formatVersion: 1.1.0 hookStatus: {} phase: PartiallyFailed progress: itemsBackedUp: 91 totalItems: 91 startTimestamp: "2024-09-09T08:24:20Z" version: 1
Expected results:
Imagestream backup should be successful.
Additional info:
Attached velero logs below:-
time="2024-09-09T08:26:10Z" level=debug msg="time=\"2024-09-09T08:26:10.803147562Z\" level=error msg=\"response completed with error\" environment=development err.code=unknown err.detail=\"s3aws: RequestError: send request failed\\ncaused by: Get \\\"https://s3-openshift-storage.apps.oadp-96031.qe.gcp.devcluster.openshift.com/oadp96031c66l6/docker/registry/v2/repositories/ocp-django/django-psql-persistent/_layers/sha256/a9644f686a26ba8fb5115ea69971272c42a1b776e4c63ae77667d7d59f5a4094/link\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\" err.message=\"unknown error\" go.version=\"go1.22.5 (Red Hat 1.22.5-1.el9) X:strictfipsruntime\" http.request.host= http.request.id=76ccf293-7610-4fc9-a945-a7884d22d96d http.request.method=HEAD http.request.remoteaddr= http.request.uri= http.request.useragent=\"containers/5.27.0 (github.com/containers/image)\" http.response.contenttype=application/json http.response.duration=373.38021ms http.response.status=500 http.response.written=104 instance.id=f7a6e6b6-99c0-49c0-856c-25a08d445a67 service=registry vars.digest=\"sha256:a9644f686a26ba8fb5115ea69971272c42a1b776e4c63ae77667d7d59f5a4094\" vars.name=ocp-django/django-psql-persistent version=v3.0.0+unknown" backup=openshift-adp/test-backup cmd=/plugins/velero-plugins logSource="/remote-source/velero/app/pkg/plugin/clientmgmt/process/logrus_adapter.go:75" pluginName=velero-plugins time="2024-09-09T08:26:10Z" level=error msg="Error backing up item" backup=openshift-adp/test-backup error="error executing custom action (groupResource=imagestreams.image.openshift.io, namespace=ocp-django, name=django-psql-persistent): rpc error: code = Unknown desc = trying to reuse blob sha256:4bb16177726caec64d3e9592403e6b642602b0b5b9c3ce3efeb9c00e117772ca at destination: failed to read from destination repository ocp-django/django-psql-persistent: 500 (Internal Server Error)" error.file="/remote-source/velero/app/pkg/backup/item_backupper.go:400" error.function="github.com/vmware-tanzu/velero/pkg/backup.(*itemBackupper).executeActions" logSource="/remote-source/velero/app/pkg/backup/backup.go:511" name=django-psql-persistent