Loading...

XML

Word

Printable

Type: Bug
Resolution: Not a Bug
Priority: Major
Fix Version/s: None
Affects Version/s: OADP 1.3.1
Component/s: data-mover
Labels:
None

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
QEStatus:
ToDo
Regression:
No
Intelligence Requested:
Market:

Cost of Delay:
0
WSJF:
0
Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

SFDC Cases Links:
SFDC Cases Counter:

Issue Description:

During the backup process, we have observed that the bucket contains less data than expected, specifically less than the 2GB total capacity. Upon further investigation, we noticed that under the backup data folder on the bucket (velero/kopia/perf-datagen-case-0-0-4-cephrbd/), the data was incomplete.

Observations:

This behavior is inconsistent and has affected our operations, particularly during the restore process, where the operation finishes more quickly than expected due to the missing data on the bucket.
The issue was initially identified on OADP 1.3.1-42 across three different clusters on the perf team: cloud33, cloud10, and cloud20.
Even after upgrading cloud10 to the latest version of OADP 1.3.1-49, the issue persists.
The backup CR status for the missing data was reported as completed, indicating a discrepancy between the reported completion and the actual data backup.

Steps to Reproduce:

Run backup using Datamover on a single namespace with a total capacity of 2GB.
Verify the data in the backup bucket (velero/kopia/perf-datagen-case-0-0-4-cephrbd/) against the expected size.
Notice the inconsistency in data size and completion status of the backup CR.
this issue is not consistent!!
before each backup we have clean all the relevant objects for clean flow
bucket content , backup CR , backuprepo
backup command :

oc -n openshift-adp exec deployment/velero -c velero -it – ./velero backup create test-backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g --include-namespaces perf-datagen-case-0-0-4-cephrbd --snapshot-move-data=true

Impact:

Incomplete backup data affects the reliability and integrity of our backup and restore operations, leading to potential data loss or inconsistencies in the event of a restore.

Additional Information:

This issue has been consistently observed across multiple clusters and versions of OADP, indicating a potential systemic issue with the backup process using Datamover.
OADP version : 1.3.1-49
ODF : 4.14.5
OCP: 4.14.12

target namespace data capacity is 2GB

Pod: perf-datagen-case-0-0-4-cephrbd/deploy-perf-datagen-0-0-21-6gi-1-rbd-0-5c759fccff-qjmbr
(app-root) bash-4.2$ du -sh /opt/mounts/mnt1/python/
2.0G    /opt/mounts/mnt1/python/

some examples ( we expect 2GB data on the bucket ) :
missing data on the bucket using asw api
Total Objects: 22 Total Size: 135.0 KiB

aws s3 ls --endpoint-url http://10.1.41.217:9000 s3://network-limitation --recursive --human-readable --summarize

2024-03-24 11:30:47 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-csi-volumesnapshotclasses.json.gz
2024-03-24 11:30:47 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-csi-volumesnapshotcontents.json.gz
2024-03-24 11:30:48 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-csi-volumesnapshots.json.gz
2024-03-24 11:31:00 403 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-itemoperations.json.gz
2024-03-24 11:30:47 15.3 KiB velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-logs.gz
2024-03-24 11:30:47 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-podvolumebackups.json.gz
2024-03-24 11:30:48 543 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-resource-list.json.gz
2024-03-24 11:30:47 49 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-results.gz
2024-03-24 11:30:47 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-volumesnapshots.json.gz
2024-03-24 11:31:01 31.8 KiB velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g.tar.gz
2024-03-24 11:31:01 3.1 KiB velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/velero-backup.json
2024-03-24 11:30:58 742 Bytes velero/kopia/perf-datagen-case-0-0-4-cephrbd/_log_20240324113058_9734_1711279858_1711279858_1_641fe0171ab30563ab669339a8da203a
2024-03-24 11:30:59 2.4 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/_log_20240324113059_9775_1711279859_1711279859_1_fdddef04502f416c4e0cc83fd5baabb3
2024-03-24 11:30:57 30 Bytes velero/kopia/perf-datagen-case-0-0-4-cephrbd/kopia.blobcfg
2024-03-24 11:30:57 1.0 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/kopia.repository
2024-03-24 11:30:59 4.2 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/paae40c6702a461851498f8ae2807aeeb-sb3032e61d3f7de08126
2024-03-24 11:30:59 65.3 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/q3b792781fb5afe620fc818c856926836-sb3032e61d3f7de08126
2024-03-24 11:30:59 4.2 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/qc33a0c7cd94b23c3c07d308c34e6f0a6-se53446f41a701204126
2024-03-24 11:30:58 4.2 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/qc432d1797f0fc200bff4ad1949fa8e25-s9d2a0cd13536019b126
2024-03-24 11:30:58 143 Bytes velero/kopia/perf-datagen-case-0-0-4-cephrbd/xn0_69f92821b50514aceead9211c5fa04de-s9d2a0cd13536019b126-c1
2024-03-24 11:30:59 143 Bytes velero/kopia/perf-datagen-case-0-0-4-cephrbd/xn0_aaf801ba4e91bff8bab9620029970c5b-se53446f41a701204126-c1
2024-03-24 11:30:59 1.3 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/xn0_acd74f00d65903ed86b98bae9eb18321-sb3032e61d3f7de08126-c1
Total Objects: 22
Total Size: 135.0 KiB

the backup CR :

 apiVersion: velero.io/v1
kind: Backup
metadata:
annotations:
velero.io/resource-timeout: 10m0s
velero.io/source-cluster-k8s-gitversion: v1.27.10+28ed2d7
velero.io/source-cluster-k8s-major-version: "1"
velero.io/source-cluster-k8s-minor-version: "27"
creationTimestamp: "2024-03-24T11:30:31Z"
generation: 7
labels:
velero.io/storage-location: bucket
name: backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g
namespace: openshift-adp
resourceVersion: "135559768"
uid: 8f116c01-ba50-4db7-aa2d-e94b495ab025
spec:
csiSnapshotTimeout: 10m0s
defaultVolumesToFsBackup: false
hooks: {}
includedNamespaces:

perf-datagen-case-0-0-4-cephrbd
itemOperationTimeout: 4h0m0s
metadata: {}
snapshotMoveData: true
storageLocation: bucket
ttl: 720h0m0s
status:
backupItemOperationsAttempted: 1
backupItemOperationsCompleted: 1
completionTimestamp: "2024-03-24T11:31:01Z"
expiration: "2024-04-23T11:30:31Z"
formatVersion: 1.1.0
phase: Completed
progress:
itemsBackedUp: 37
totalItems: 37
startTimestamp: "2024-03-24T11:30:31Z"
version: 1

all the other object as :

backuprepositories 
backupstoragelocations
DPA
dataupload

are valid and on ready state

example for valid data on the bucket using asw api
Total Objects: 121 Total Size: 2.0 GiB

aws s3 ls --endpoint-url http://10.1.41.217:9000 s3://network-limitation --recursive --human-readable --summarize

2024-03-24 09:55:24 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-csi-volumesnapshotclasses.json.gz
2024-03-24 09:55:24 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-csi-volumesnapshotcontents.json.gz
2024-03-24 09:55:24 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-csi-volumesnapshots.json.gz
2024-03-24 09:56:00 423 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-itemoperations.json.gz
2024-03-24 09:55:24 15.4 KiB velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-logs.gz
2024-03-24 09:55:24 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-podvolumebackups.json.gz
2024-03-24 09:55:24 545 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-resource-list.json.gz
2024-03-24 09:55:24 49 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-results.gz
2024-03-24 09:55:24 29 Bytes velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-volumesnapshots.json.gz
2024-03-24 09:56:00 31.9 KiB velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g.tar.gz
2024-03-24 09:56:00 3.1 KiB velero/backups/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g/velero-backup.json
2024-03-24 09:55:38 744 Bytes velero/kopia/perf-datagen-case-0-0-4-cephrbd/_log_20240324095538_d53e_1711274138_1711274138_1_7b078c3bc419c309a3b754830d782d8d
2024-03-24 09:55:57 232.1 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/_log_20240324095539_6b92_1711274139_1711274157_1_c446e49a4150a3f990000c663f972bda
2024-03-24 09:55:38 30 Bytes velero/kopia/perf-datagen-case-0-0-4-cephrbd/kopia.blobcfg
2024-03-24 09:55:38 1.0 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/kopia.repository
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p058a73f5fd6550368b37274bdd3e9020-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p0761c75a3ccfeaaac57ce323fa8819e8-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p0847150de8de0ed42e07378ef622f400-scb345d04450e35ff126
2024-03-24 09:55:48 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p084a4e79274e606a3b7360158bdba8aa-scb345d04450e35ff126
2024-03-24 09:55:51 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p09aa9319d1ce7a83f5abdffb00fe3512-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p0c8264a1728d4dd839a641d9926ba2a7-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p0d9a1213ea266b2ba64949cd31da494b-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p11be9843ea1cb596c7dbf2da6d92b0cb-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p1242f91012c5090570cd337359b7c48f-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p1554b88bd3374af2665e534673b92479-scb345d04450e35ff126
2024-03-24 09:55:51 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p16b2ead120f0702b56d6b39491120cbd-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p1ef9f1fe757c9ce84adb41db76dbe6dc-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p1f89e73c0027bfb249ef416a2c79fbb6-scb345d04450e35ff126
2024-03-24 09:55:50 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p2181b39966e3876aca01170ce4c766af-scb345d04450e35ff126
2024-03-24 09:55:47 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p2246484b80b257c1bac554f6790a42ee-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p22f8e2ae21936554085e7d7cf367f711-scb345d04450e35ff126
2024-03-24 09:55:51 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p23adf9a41c616d4c183c7278d995fb92-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p2bdd7a327feec510ab53ef8d5f5101a6-scb345d04450e35ff126
2024-03-24 09:55:50 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p32550b2ce2ded09d4154c8988d652cc1-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p34e989346e948d646b7a86aa47fdb3e7-scb345d04450e35ff126
2024-03-24 09:55:56 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p39ab8aacd6a1deb822db41ebaea4f553-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p3fe5867b7809d52e4a972f0dfe02fb1a-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p4255b4103951913e6a569339f4110bba-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p43d26ce9caa19b5ab1282c7796907cc3-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p4403bc6bd2fa3542b4187d0b55bc6053-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p44e9a41b0d666dd35245e8edb90c301e-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p4c799027a597542d9b2f7b274cf3b8db-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p4cf7470a3bafa6330d6f97a3273f59ea-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p4f3caef5d1f25bcc0c4f8d18b1bc9b41-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p502c94dabc910cb82825d6c768945719-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p54c0c0d8bb4b4c416391798307722900-scb345d04450e35ff126
2024-03-24 09:55:50 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p55dfc63ee3e23a35a7f60ad553741c7a-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p575d3a868ed7e7ed8d9437dfb75defee-scb345d04450e35ff126
2024-03-24 09:55:50 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p5ece65f35313bcfb56a9b73638e9f784-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p5f0d744cdd707700bf3b0d5cfcd24131-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p5f5b0d18dabb802f2c74f11cfadad1e7-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p61b40ca64d6973f7176866687d3c578e-scb345d04450e35ff126
2024-03-24 09:55:50 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p61ba3250d995618418895601fc23421d-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p660ce6733fa241b02e970667e6528fd3-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p6a1a67bfc5f7c9078b2a2582e97e9012-scb345d04450e35ff126
2024-03-24 09:55:48 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p6bd0b2b9069171280f55c30ab931cc4b-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p70398cc52ade356aa910ff090b9ea12d-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p7590a046008c051d6eb6b993f0e1d940-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p7a95032b1a3244ff9c224424006a14a3-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p83137fd2cc2dbd650b25d20991f2999f-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p83b7d5722201aa529227acbd7e29412a-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p8446b5d2111bb847f98cb5f09f6adbc4-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p869ccbcf2290de270c5dfcc2df2bd03a-scb345d04450e35ff126
2024-03-24 09:55:48 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p874f3f86da611fe29b3386c2d7201ae0-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p8a5c68e63c80dc865b18b31d8de1084d-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p8bf218d4ea9b2479a9eed2a4b3a5a586-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p8f4f7b163327e2999808d2a277c1269b-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p9185daf3f483294d3397cf06579b6956-scb345d04450e35ff126
2024-03-24 09:55:57 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p929355652ce0fb1f5d7406ea69f59a46-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p933e2638712c9467c6f31414b7636b87-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p9cc2cce0ef865acb88d9d54dc07ee0ec-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/p9f0969d265b883a4c29618dbc0050b25-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pa24a12c6ba554a7af1ad424273b96197-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pa48183750a55aa48bb72b38ff5a0a1ce-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pa5c977204cff15671830fb6bdbd26f95-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pa7120739a33996ffce33052d64d42161-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pa8c9985dd3800ff3c03492be907519ba-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/paadc478601c5841c77905ad36e6bd499-scb345d04450e35ff126
2024-03-24 09:55:47 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pab61750238904d21b3c21f33c1bcf389-scb345d04450e35ff126
2024-03-24 09:55:57 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/paeadea8c914c6ca71120a0677142e688-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/paec149abc1ca3cdc271b3c65fba1e1d5-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pb1692a0b6380fb1a5c9ea44d3f1d68f2-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pb3414228eecf80b88f45a9ea060e6ddb-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pb3cf1acb74ee3dcd8d4f15ad86e69406-scb345d04450e35ff126
2024-03-24 09:55:47 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pbad56a842143b36f23bb55c688022d07-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pbdd66772e73df55fc00af89a2aff2f96-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pc0f996cb48a8195c40b9bbf83af6af1a-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pc1ed17ab2304e601810f8ddd76497a30-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pc24142cf413371658e2be285437a5f69-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pc2c8e6b1c3da271fa30146b13347c8ee-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pc4e5b7107f858b97a633999b3c79a81f-scb345d04450e35ff126
2024-03-24 09:55:48 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pc60a3ad31204c50e71006c87d719faa5-scb345d04450e35ff126
2024-03-24 09:55:48 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pcd8b5850b60cdae0704634a44d5262f5-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pd05b5bb9c4acf728c0430a82433fb9ef-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pd0a28ff05df1ac5191eeb6222ec4e718-scb345d04450e35ff126
2024-03-24 09:55:39 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pd0b8e8539f181e9ca6e57ea82c4964da-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pd0cda8ef45615741509b144132098279-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pd7b4252378c603e22586df9e85f30da0-scb345d04450e35ff126
2024-03-24 09:55:48 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pddba67d912065f27458b40d2017988e6-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pe23964f435262f2824c853c01ffd1786-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pe4a20af0726d505faa29ba70f7485ff5-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pe74c911e4e858504e03a5c517bcead27-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pe9b8c57b06ad9eb4cd54ddb50741d77a-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/peb42dd4ffffb23331651ee0769d78f76-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pec3d570d061a1ee057a8a285030b1ad6-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pec4cebd71d8e64895616c5d3f3a2b6b5-scb345d04450e35ff126
2024-03-24 09:55:41 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/ped936103a37ad190198f40e6c286ab53-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pedb6514b5ee3a5879322d6db092ceb42-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pef94ae8e54280ddd28f1b483c1c3b19f-scb345d04450e35ff126
2024-03-24 09:55:49 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pf075787df07f69f4c2a1933b56fd9960-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pf2e24ad850f2ba35dbd393dd1d174b61-scb345d04450e35ff126
2024-03-24 09:55:50 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pf5497da22384af0d61386b723e7d52de-scb345d04450e35ff126
2024-03-24 09:55:51 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pf5e632d02a324ef189d884a92b86a9bd-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pf869273105b543b9a4aee88e609f5889-scb345d04450e35ff126
2024-03-24 09:55:40 20.2 MiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/pfad119a4e975b9445c233dc05c72a81c-scb345d04450e35ff126
2024-03-24 09:55:38 4.2 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/q350bb2cd13a007d625b13bbc82418378-s731a0dcf24448914126
2024-03-24 09:55:39 4.2 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/q86ce74b7b3c6dd820735613d916ebc8b-s5f60eb2bd9a8d675126
2024-03-24 09:55:57 185.3 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/qeb070d41596cbf24c9beb14664179b1b-scb345d04450e35ff126
2024-03-24 09:55:39 143 Bytes velero/kopia/perf-datagen-case-0-0-4-cephrbd/xn0_06fa03ea826fa9e01efc4e10e16fa7b1-s5f60eb2bd9a8d675126-c1
2024-03-24 09:55:38 143 Bytes velero/kopia/perf-datagen-case-0-0-4-cephrbd/xn0_522c7e22eb645fc9698a91146ac3f720-s731a0dcf24448914126-c1
2024-03-24 09:55:57 236.1 KiB velero/kopia/perf-datagen-case-0-0-4-cephrbd/xn0_b2f05559b509bd44fd1464eb1d6f40e9-scb345d04450e35ff126-c1
Total Objects: 121
Total Size: 2.0 GiB

on the node-agent log on missing data there are no errors or any indicates for the issue :

Starting backup" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/uploader/provider/kopia.go:144" parentSnapshot= path="/host_pods/556c2157-130f-4093-bcec-29808b1c7740/volumes/kubernetes.io~csi/pvc-20713c04-cb5b-43ab-909d-92d059ff5569/mount" realSource=perf-datagen-case-0-0-4-cephrbd/pvc-perf-datagen-0-0-21-6gi-1-rbd-0

time="2024-03-24T09:27:01Z" level=info msg="Data upload is in progress" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/controller/data_upload_controller.go:292"

time="2024-03-24T09:27:01Z" level=info msg="Start to snapshot..." controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/uploader/kopia/snapshot.go:230" parentSnapshot= path="/host_pods/556c2157-130f-4093-bcec-29808b1c7740/volumes/kubernetes.io~csi/pvc-20713c04-cb5b-43ab-909d-92d059ff5569/mount" realSource=perf-datagen-case-0-0-4-cephrbd/pvc-perf-datagen-0-0-21-6gi-1-rbd-0

time="2024-03-24T09:27:01Z" level=info msg="Searching for parent snapshot" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/uploader/kopia/snapshot.go:245" parentSnapshot= path="/host_pods/556c2157-130f-4093-bcec-29808b1c7740/volumes/kubernetes.io~csi/pvc-20713c04-cb5b-43ab-909d-92d059ff5569/mount" realSource=perf-datagen-case-0-0-4-cephrbd/pvc-perf-datagen-0-0-21-6gi-1-rbd-0

time="2024-03-24T09:27:01Z" level=info msg="Reconcile backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/controller/data_upload_controller.go:113"

time="2024-03-24T09:27:01Z" level=info msg="Data upload is in progress" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/controller/data_upload_controller.go:292"

time="2024-03-24T09:27:02Z" level=info msg="Created snapshot with root k19a5deba038224f328a2f83b58a6c146 and ID 7ec0515da1b61b1c8d8f0ffc7b7097a5 in 1s" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/uploader/kopia/snapshot.go:289" parentSnapshot= path="/host_pods/556c2157-130f-4093-bcec-29808b1c7740/volumes/kubernetes.io~csi/pvc-20713c04-cb5b-43ab-909d-92d059ff5569/mount" realSource=perf-datagen-case-0-0-4-cephrbd/pvc-perf-datagen-0-0-21-6gi-1-rbd-0

time="2024-03-24T09:27:02Z" level=info msg="Async fs backup data path completed" dataupload=backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/controller/data_upload_controller.go:359"

time="2024-03-24T09:27:02Z" level=info msg="Action finished" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/uploader/provider/kopia.go:91"

time="2024-03-24T09:27:02Z" level=info msg="Reconcile backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/controller/data_upload_controller.go:113"

time="2024-03-24T09:27:02Z" level=info msg="Data upload is in progress" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/controller/data_upload_controller.go:292"

time="2024-03-24T09:27:02Z" level=info msg="Data upload completed" dataupload=backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/controller/data_upload_controller.go:393"

time="2024-03-24T09:27:02Z" level=info msg="Reconcile backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/controller/data_upload_controller.go:113"

time="2024-03-24T09:27:02Z" level=info msg="FileSystemBR is closed" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/datapath/file_system.go:129" user=backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m

time="2024-03-24T09:27:02Z" level=info msg="Reconcile backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m" controller=dataupload dataupload=openshift-adp/backup-1pod-vbd-pvc-util-0-0-21-cephrbd-6g-jkq9m logSource="/remote-source/velero/app/pkg/controller/data_upload_controller.go:113"

when i have run restore from the non-valid backup
the duration was faster 18sec insted of 38sec as expected
and the total capacity on the target pod was 368K instead of 2GB

(app-root) bash-4.2$ du -sh /opt/mounts/mnt1/python/
368K    /opt/mounts/mnt1/python/

the full logs folder for comparison for valid and non valid content can be found here :

[~~OADP-3766~~|https://drive.google.com/drive/folders/1ZrjxR2vu9azsrxIaCdXTH8aG0Row_swG?usp=sharing]

Assignee:: Wes Hayutin

Reporter:: Tzahi Ashkenazi

QA Contact:: Tzahi Ashkenazi

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: 2024/03/24 12:03 PM

Updated:: 2024/04/01 12:10 PM

Resolved:: 2024/04/01 12:10 PM

Details

Description

Attachments

Activity

People

Dates