-
Bug
-
Resolution: Done
-
Undefined
-
None
-
False
-
False
-
Passed
-
-
0
-
0
-
0
-
Untriaged
Description of problem:
After creating a Restic backup on namespace X, clearing the bucket and creating another restic backup of namespace X, new backup fails to restore the data (PartiallyFailed) with this error:
time="2021-12-29T18:29:14Z" level=info msg="1 errors encountered backup up item" backup=velero/backup65 logSource="pkg/backup/backup.go:431" name=mysql-7d99fc949-qbkds time="2021-12-29T18:29:14Z" level=error msg="Error backing up item" backup=velero/backup65 error="pod volume backup failed: error running restic backup, stderr=Fatal: unable to open config file: Stat: The specified key does not exist.\nIs there a repository at the following location?\ns3:http://minio-minio.apps.mayap-oadp-veleo-1234.qe.devcluster.openshift.com/mayapvelerooadp2/velero1/restic/mysql-persistent\n: exit status 1" error.file="/remote-source/src/github.com/vmware-tanzu/velero/pkg/restic/backupper.go:184" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:435" name=mysql-7d99fc949-qbkds
velero backup describe of the backup output:
(mtc-e2e-venv) [mperetz@mperetz velero-examples]$ velero backup describe backup71 --details Name: backup71 Namespace: velero Labels: velero.io/storage-location=example-velero-1 Annotations: velero.io/source-cluster-k8s-gitversion=v1.22.3+e790d7f velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=22Phase: PartiallyFailed (run `velero backup logs backup71` for more information)Errors: 1 Warnings: 0Namespaces: Included: mysql-persistent Excluded: <none>Resources: Included: * Excluded: <none> Cluster-scoped: autoLabel selector: <none>Storage Location: example-velero-1Velero-Native Snapshot PVs: falseTTL: 720h0m0sHooks: <none>Backup Format Version: 1.1.0Started: 2021-12-30 11:26:13 +0200 IST Completed: 2021-12-30 11:26:33 +0200 ISTExpiration: 2022-01-29 11:26:13 +0200 ISTTotal items to be backed up: 56 Items backed up: 56Resource List: apps/v1/Deployment: - mysql-persistent/mysql apps/v1/ReplicaSet: - mysql-persistent/mysql-7d99fc949 authorization.openshift.io/v1/RoleBinding: - mysql-persistent/system:deployers - mysql-persistent/system:image-builders - mysql-persistent/system:image-pullers discovery.k8s.io/v1/EndpointSlice: - mysql-persistent/mysql-56bct rbac.authorization.k8s.io/v1/RoleBinding: - mysql-persistent/system:deployers - mysql-persistent/system:image-builders - mysql-persistent/system:image-pullers v1/ConfigMap: - mysql-persistent/kube-root-ca.crt - mysql-persistent/openshift-service-ca.crt v1/Endpoints: - mysql-persistent/mysql v1/Event: - mysql-persistent/mysql-7d99fc949-4srlf.16c57a7f5437e39e - mysql-persistent/mysql-7d99fc949-4srlf.16c57a7ff53ad7cc - mysql-persistent/mysql-7d99fc949-4srlf.16c57a81cc5870e0 - mysql-persistent/mysql-7d99fc949-4srlf.16c57a81ce8d173b - mysql-persistent/mysql-7d99fc949-4srlf.16c57a830895114f - mysql-persistent/mysql-7d99fc949-4srlf.16c57a831124de2c - mysql-persistent/mysql-7d99fc949-4srlf.16c57a83128a6bc0 - mysql-persistent/mysql-7d99fc949-4srlf.16c57a8b67a977a2 - mysql-persistent/mysql-7d99fc949-4srlf.16c57b9d05fd3c96 - mysql-persistent/mysql-7d99fc949-j78q4.16c57b9df7219c6c - mysql-persistent/mysql-7d99fc949-j78q4.16c57b9e88c7dc58 - mysql-persistent/mysql-7d99fc949-j78q4.16c57ba21fd52621 - mysql-persistent/mysql-7d99fc949-j78q4.16c57ba22259d33c - mysql-persistent/mysql-7d99fc949-j78q4.16c57ba361de25bf - mysql-persistent/mysql-7d99fc949-j78q4.16c57ba36a839f48 - mysql-persistent/mysql-7d99fc949-j78q4.16c57ba36c1113e3 - mysql-persistent/mysql-7d99fc949-j78q4.16c57babc099d27e - mysql-persistent/mysql-7d99fc949.16c57a7dedb1f289 - mysql-persistent/mysql-7d99fc949.16c57b9d07a807a5 - mysql-persistent/mysql.16c57a7de087e06d - mysql-persistent/mysql.16c57a7dec070072 - mysql-persistent/mysql.16c57a7f3468983f - mysql-persistent/mysql.16c57b977cc712e6 - mysql-persistent/mysql.16c57b9d08aa3ace - mysql-persistent/mysql.16c57b9d08abe70b - mysql-persistent/mysql.16c57b9dd253ca50 v1/Namespace: - mysql-persistent v1/PersistentVolume: - pvc-96ad5bc0-f5a3-428d-92e9-9cebbb83749a v1/PersistentVolumeClaim: - mysql-persistent/mysql v1/Pod: - mysql-persistent/mysql-7d99fc949-j78q4 v1/Secret: - mysql-persistent/builder-dockercfg-rdrs2 - mysql-persistent/builder-token-k9vn9 - mysql-persistent/builder-token-zzk48 - mysql-persistent/default-dockercfg-gc8pj - mysql-persistent/default-token-56k5l - mysql-persistent/default-token-mxl6f - mysql-persistent/deployer-dockercfg-l69j5 - mysql-persistent/deployer-token-qg768 - mysql-persistent/deployer-token-zk7nn - mysql-persistent/mysql v1/Service: - mysql-persistent/mysql v1/ServiceAccount: - mysql-persistent/builder - mysql-persistent/default - mysql-persistent/deployerVelero-Native Snapshots: <none included>Restic Backups: Failed: mysql-persistent/mysql-7d99fc949-j78q4: mysql-data
WA: delete all restic repositories manually from CLI(oc delete resticrepositories -n velero –all)
Version-Release number of selected component (if applicable): 0.5.3
How reproducible:
Always
Steps to reproduce:
- Create a Velero backup of stateful app (namespaced) with Restic for volume backup
- Empty the bucket and wait till the backup CR is deleted by itself
- Create another Velero backup of the same stateful app with Restic (namespaced)
- Notice that the backup partially fails (fails to backup the data)
Expected results: expecting Velero to sync between Restic repositories and backups (delete the repositories if backup is deleted/recreate the repository in case of new backup of the same namespace)
