Normal Hi,
I have a number of customers currently using Azure Red Hat OpenShift (ARO) who are interested in using OADP.
As a Solution Architect, I've been trying to get backup/restore working with ARO in order to demo the functionality for these different Government departments.
So far, backups "seem" to work, but I have not been able to get PVCs restored. Any restored pod that requires a PVC gets stuck in "pending" because the PVC is never created properly.
I've been going back and forth with rhn-engineering-dymurray and tkaovila@redhat.com on Slack to try to debug the issue, but so far we haven't had any luck. Can you please provide explicit details on how to successfully backup and restore namespaces with PVCs in Azure Red Hat OpenShift.
The process I used:
- Create a SPN with "Contributor" access to my Subscription.
- Create the required "credentials-velero" file and secret with the SPN details (subscription ID, tenant ID, client ID, secret, backup RG, etc...)
- Create a DataProtectionApplictaion (below)
- Create a Backup (below)
- Attempt a Restore (below)
The backup completes, and the restore partially completes - but always stalls/fails on the PVC restore.
Please advise on the proper config (dpa/backup and SPN config), as this is an important feature for my customers.
Thanks!
DPA:
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
name: dpa
namespace: openshift-adp
spec:
configuration:
velero:
defaultPlugins:
- azure
- openshift
restic:
enable: true
backupLocations:
- velero:
config:
resourceGroup: Velero_Backups
storageAccount: velerodxxxxxxx
subscriptionId: ea271b60-xxxx-xxxxx-xxxx-422408e476ba
storageAccountKeyEnvVar: AZURE_STORAGE_ACCOUNT_ACCESS_KEY
credential:
key: cloud
name: cloud-credentials-azure
provider: azure
default: true
objectStorage:
bucket: velero
prefix: velero
snapshotLocations:
- velero:
config:
resourceGroup: Velero_Backups
subscriptionId: ea271b60-xxxxxx-xxxxx-422408e476ba
incremental: "true"
provider: azure
Backup:
apiVersion: velero.io/v1
kind: Backup
metadata:
name: a-backup
labels:
velero.io/storage-location: default
namespace: openshift-adp
spec: hooks: {}
includedNamespaces:
- testproj
includeClusterResources: false
storageLocation: dpa-1
ttl: 24h0m0s
Restore:
apiVersion: velero.io/v1 kind: Restore metadata: name: a-restore namespace: openshift-adp spec: backupName: a-backup restorePVs: true
Velero error message on restore:
time="2022-04-06T19:36:29Z" level=info msg="Getting client for /v1, Kind=PersistentVolumeClaim" logSource="pkg/restore/restore.go:878" restore=openshift-adp/a-restore time="2022-04-06T19:36:29Z" level=info msg="Executing item action for persistentvolumeclaims" logSource="pkg/restore/restore.go:1159" restore=openshift-adp/a-restore time="2022-04-06T19:36:29Z" level=info msg="Executing AddPVFromPVCAction" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:44" pluginName=velero restore=openshift-adp/a-restore time="2022-04-06T19:36:29Z" level=info msg="Adding PV pvc-ab62c854-112a-44a5-bb73-05da0fa6f091 as an additional item to restore" cmd=/velero logSource="pkg/restore/add_pv_from_pvc_action.go:66" pluginName=velero restore=openshift-adp/a-restore time="2022-04-06T19:36:29Z" level=warning msg="unable to restore additional item" additionalResource=persistentvolumes additionalResourceName=pvc-ab62c854-112a-44a5-bb73-05da0fa6f091 additionalResourceNamespace= error="stat /tmp/718783699/resources/persistentvolumes/cluster/pvc-ab62c854-112a-44a5-bb73-05da0fa6f091.json: no such file or directory" logSource="pkg/restore/restore.go:1191" restore=openshift-adp/a-restore
