-
Sub-task
-
Resolution: Done
-
Undefined
-
None
-
None
-
False
-
-
False
-
ToDo
-
-
-
0
-
0
-
Very Likely
-
0
-
None
-
Unset
-
Unknown
Description of problem:
While performing backup of any application , keeping Noobaa as the backup location, and having s3Url prefix as https:// , Backup fails with
failureReason: 'rpc error: code = Unknown desc = error putting object velero/backups/b12/velero-backup.json:
operation error S3: PutObject, https response error StatusCode: 400, RequestID:
lxa7ym7t-960tv4-qn8, HostID: lxa7ym7t-960tv4-qn8, api error InvalidDigest: The
Content-MD5 you specified is not valid.'
Version-Release number of selected component (if applicable):
4.14 AWS-MCG + OADP 1.4.0
How reproducible:
Always
Steps to Reproduce:
DPA:
$ oc get dpa -o yaml
apiVersion: v1
items:
- apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
creationTimestamp: "2024-06-11T09:44:42Z"
generation: 2
name: ts-dpa
namespace: openshift-adp
resourceVersion: "283545"
uid: 560cddf5-b52c-4441-8fc0-937fbfcca016
spec:
backupImages: true
backupLocations:
- velero:
config:
insecureSkipTLSVerify: "true"
profile: noobaa
region: noobaa
s3ForcePathStyle: "true"
s3Url: https://s3-openshift-storage.apps.oadp-83511.qe.devcluster.openshift.com
credential:
key: cloud
name: cloud-credentials
default: true
objectStorage:
bucket: oadp83511lxpwk
prefix: velero
provider: aws
configuration:
velero:
defaultPlugins:
- openshift
- csi
- aws
snapshotLocations:
- velero:
config:
profile: default
region: us-east-2
provider: aws
status:
conditions:
- lastTransitionTime: "2024-06-11T09:44:42Z"
message: Reconcile complete
reason: Complete
status: "True"
type: Reconciled
kind: List
metadata:
resourceVersion: ""
1. Deploy application to backup
2. Perform any backup, CSI/Datamover/Restic/Kopia
Actual results:
$ oc get backup b12 -o yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
annotations:
velero.io/resource-timeout: 10m0s
velero.io/source-cluster-k8s-gitversion: v1.27.14+7852426
velero.io/source-cluster-k8s-major-version: "1"
velero.io/source-cluster-k8s-minor-version: "27"
creationTimestamp: "2024-06-11T09:48:06Z"
generation: 5
labels:
velero.io/storage-location: ts-dpa-1
name: b12
namespace: openshift-adp
resourceVersion: "283803"
uid: 3ef077ed-b9c3-4205-bbe5-8915e491e22f
spec:
csiSnapshotTimeout: 10m0s
defaultVolumesToFsBackup: false
includedNamespaces:
- mysql
itemOperationTimeout: 4h0m0s
snapshotMoveData: false
storageLocation: ts-dpa-1
ttl: 720h0m0s
volumeSnapshotLocations:
- ts-dpa-1
status:
backupItemOperationsAttempted: 2
backupItemOperationsCompleted: 1
csiVolumeSnapshotsAttempted: 2
expiration: "2024-07-11T09:48:06Z"
failureReason: 'rpc error: code = Unknown desc = error putting object velero/backups/b12/velero-backup.json:
operation error S3: PutObject, https response error StatusCode: 400, RequestID:
lxa7ym7t-960tv4-qn8, HostID: lxa7ym7t-960tv4-qn8, api error InvalidDigest: The
Content-MD5 you specified is not valid.'
formatVersion: 1.1.0
hookStatus: {}
phase: Failed
progress:
itemsBackedUp: 48
totalItems: 48
startTimestamp: "2024-06-11T09:48:06Z"
version: 1
Expected results:
Backup should be successful
Additional info:
velero logs
time="2024-06-11T09:48:17Z" level=error msg="Error uploading log file" backup=b12 bucket=oadp83511lxpwk error="rpc error: code = Unknown desc = error putting object velero/backups/b12/b12-logs.gz: operation error S3: PutObject, https response error StatusCode: 400, RequestID: lxa7ym7p-937ep1-i3d, HostID: lxa7ym7p-937ep1-i3d, api error InvalidDigest: The Content-MD5 you specified is not valid." error.file="/remote-source/app/velero-plugin-for-aws/object_store.go:280" error.function="main.(*ObjectStore).PutObject" logSource="/remote-source/velero/app/pkg/persistence/object_store.go:256" prefix=velero
