-
Bug
-
Resolution: Done
-
Blocker
-
OADP 1.1.0
-
False
-
-
False
-
oadp-velero-container-1.1.0-24
-
Passed
-
0
-
0
-
Very Likely
-
0
-
None
-
Unset
-
Unknown
-
Approved
-
Yes
Description of problem: restic-wait container continuously fails with the error message "Not found: /restores/<pod-volume>/.velero/<restore-UID>" and the pod is stuck on Init status.
The restore's status however is "Completed", and there is no indication that the restic volume backup has failed.
The PVR is not created for that volume.
Version-Release number of selected component (if applicable): happens since 1.1.0-62, telling by the failures on Report Portal.
How reproducible: 100%. Fails on all platforms, OCP 4.9-4.11 and for all restic tests cases.
Steps to Reproduce:
1. Create a stateful app.
2. Create a restic backup of that stateful app.
3. Delete the app namespace.
4. Create a restore of that app.
Actual results:
[mperetz@mperetz oadp-qe-automation]$ oc get pods -ntodolist-mariadb NAME READY STATUS RESTARTS AGE mysql-7c6bdd58b5-v2jfs 0/1 Init:0/1 0 59m todolist-564d56c7b9-pjf4c 0/1 CrashLoopBackOff 11 (113s ago) 59m [mperetz@mperetz oadp-qe-automation]$ oc logs mysql-7c6bdd58b5-v2jfs -n todolist-mariadb -c restic-wait | tail Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 Not found: /restores/mysql-data/.velero/bec6932f-8a4c-4e2d-8d9b-40eda3602fc7 [mperetz@mperetz oadp-qe-automation]$ velero restore get -nopenshift-adp NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c Completed 2022-08-15 12:34:38 +0300 IDT 2022-08-15 12:34:58 +0300 IDT 0 13 2022-08-15 12:34:38 +0300 IDT <none> [mperetz@mperetz oadp-qe-automation]$ [mperetz@mperetz oadp-qe-automation]$ oc get podvolumerestore -n openshift-adp No resources found in openshift-adp namespace. [mperetz@mperetz oadp-qe-automation]$ [mperetz@mperetz oadp-qe-automation]$ oc logs deploy/velero -n openshift-adp | grep warn time="2022-08-15T09:34:58Z" level=warning msg="Cluster resource restore warning: could not restore, CustomResourceDefinition \"clusterserviceversions.operators.coreos.com\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:511" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Cluster resource restore warning: could not restore, SecurityContextConstraints \"todolist-mariadb-scc\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:511" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, Pod \"mysql-7c6bdd58b5-v2jfs\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, Pod \"todolist-564d56c7b9-pjf4c\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, Deployment \"mysql\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, Deployment \"todolist\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, RoleBinding \"admin\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, RoleBinding \"system:deployers\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, RoleBinding \"system:image-builders\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, RoleBinding \"system:image-pullers\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, Route \"todolist-route\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, Service \"mysql\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c time="2022-08-15T09:34:58Z" level=warning msg="Namespace todolist-mariadb, resource restore warning: could not restore, Service \"todolist\" already exists. Warning: the in-cluster version is different than the backed-up version." logSource="pkg/controller/restore_controller.go:515" restore=openshift-adp/todolist-backup-04e2bee9-1c7d-11ed-9545-902e163f806c
Expected results:
Additional info:
