Loading...

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: OADP 1.5.3
Affects Version/s: OADP 1.3.0
Component/s: volumesnapshotlocation
Labels:

Activity Type:
Quality / Stability / Reliability
Story Points:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
QEStatus:
ToDo
Intelligence Requested:
Market:

WSJF:
0
Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

Regression:
No

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem:

VSL backup PartiallyFailed with GCP Workload Identity configured.

Version-Release number of selected component (if applicable):

oadp-operator-bundle-container-1.3.0-156

How reproducible:

100%

Steps to Reproduce:
1. Install DPA with snapshotLocations.
2. install stateful application.
3. Run VSL backup.

Actual results:

Backup finished with PartiallyFailed phase.

Expected results:

Backup should finish successfully with Completed phase.

Additional info:

DPA:

oc get dpa -n openshift-adp gcp-sts-dpa -o yaml
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  annotations:
    meta.helm.sh/release-name: gcp-sts
    meta.helm.sh/release-namespace: openshift-adp
  creationTimestamp: "2023-11-19T13:59:49Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: gcp-sts-dpa
  namespace: openshift-adp
  resourceVersion: "148782"
  uid: 8873acf3-7254-450b-b447-cc82d6b1d8a5
spec:
  backupLocations:
  - name: main-bsl
    velero:
      credential:
        key: cloud
        name: cloud-credentials-gcp
      default: true
      objectStorage:
        bucket: oadpbucket247358
        prefix: velero
      provider: gcp
  configuration:
    velero:
      defaultPlugins:
      - openshift
      - gcp
  snapshotLocations:
  - velero:
      config:
        project: openshift-qe
        snapshotLocation: us-central1
      provider: gcp
status:
  conditions:
  - lastTransitionTime: "2023-11-19T13:59:50Z"
    message: Reconcile complete
    reason: Complete
    status: "True"
    type: Reconciled

Backup:

oc get backup b9 -n openshift-adp -o yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    helm.sh/resource-policy: keep
    meta.helm.sh/release-name: gcp-sts
    meta.helm.sh/release-namespace: openshift-adp
    velero.io/resource-timeout: 10m0s
    velero.io/source-cluster-k8s-gitversion: v1.27.6+b49f9d1
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "27"
  creationTimestamp: "2023-11-19T14:02:14Z"
  generation: 6
  labels:
    app.kubernetes.io/managed-by: Helm
    velero.io/storage-location: main-bsl
  name: b9
  namespace: openshift-adp
  resourceVersion: "150172"
  uid: 17a1991d-d4a0-4a08-9dca-dbad51e4310f
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  includedNamespaces:
  - a1
  itemOperationTimeout: 4h0m0s
  snapshotMoveData: false
  storageLocation: main-bsl
  ttl: 720h0m0s
  volumeSnapshotLocations:
  - gcp-sts-dpa-1
status:
  completionTimestamp: "2023-11-19T14:02:21Z"
  errors: 2
  expiration: "2023-12-19T14:02:14Z"
  formatVersion: 1.1.0
  phase: PartiallyFailed
  progress:
    itemsBackedUp: 43
    totalItems: 43
  startTimestamp: "2023-11-19T14:02:14Z"
  version: 1

Error logs:

time="2023-11-19T14:02:18Z" level=info msg="label \"topology.kubernetes.io/zone\" is not present on PersistentVolume, checking deprecated label..." backup=openshift-adp/b9 logSource="/remote-source/velero/app/pkg/backup/item_backupper.go
:552" name=pvc-c748a9a9-79c2-47d3-b6da-8297903388ca namespace= persistentVolume=pvc-c748a9a9-79c2-47d3-b6da-8297903388ca resource=persistentvolumes
time="2023-11-19T14:02:18Z" level=info msg="label \"failure-domain.beta.kubernetes.io/zone\" is not present on PersistentVolume" backup=openshift-adp/b9 logSource="/remote-source/velero/app/pkg/backup/item_backupper.go:556" name=pvc-c748
a9a9-79c2-47d3-b6da-8297903388ca namespace= persistentVolume=pvc-c748a9a9-79c2-47d3-b6da-8297903388ca resource=persistentvolumes
time="2023-11-19T14:02:18Z" level=info msg="zone info from nodeAffinity requirements: us-central1-c, key: topology.gke.io/zone" backup=openshift-adp/b9 logSource="/remote-source/velero/app/pkg/backup/item_backupper.go:559" name=pvc-c748a
9a9-79c2-47d3-b6da-8297903388ca namespace= persistentVolume=pvc-c748a9a9-79c2-47d3-b6da-8297903388ca resource=persistentvolumes
time="2023-11-19T14:02:18Z" level=info msg="Got volume ID for persistent volume" backup=openshift-adp/b9 logSource="/remote-source/velero/app/pkg/backup/item_backupper.go:589" name=pvc-c748a9a9-79c2-47d3-b6da-8297903388ca namespace= pers
istentVolume=pvc-c748a9a9-79c2-47d3-b6da-8297903388ca resource=persistentvolumes volumeSnapshotLocation=gcp-sts-dpa-1
time="2023-11-19T14:02:18Z" level=info msg="Getting volume information" backup=openshift-adp/b9 logSource="/remote-source/velero/app/pkg/backup/item_backupper.go:612" name=pvc-c748a9a9-79c2-47d3-b6da-8297903388ca namespace= persistentVol
ume=pvc-c748a9a9-79c2-47d3-b6da-8297903388ca resource=persistentvolumes volumeID=pvc-c748a9a9-79c2-47d3-b6da-8297903388ca
time="2023-11-19T14:02:18Z" level=info msg="1 errors encountered backup up item" backup=openshift-adp/b9 logSource="/remote-source/velero/app/pkg/backup/backup.go:444" name=mysql-1
time="2023-11-19T14:02:18Z" level=error msg="Error backing up item" backup=openshift-adp/b9 error="error getting volume info: rpc error: code = Unknown desc = googleapi: Error 400: Invalid resource field value in the request.\nDetails:\n
[\n  {\n    \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n    \"domain\": \"googleapis.com\",\n    \"metadatas\": {\n      \"method\": \"compute.v1.DisksService.Get\",\n      \"service\": \"compute.googleapis.com\"\n    },\n 
   \"reason\": \"RESOURCE_PROJECT_INVALID\"\n  }\n]\n, invalidParameter" logSource="/remote-source/velero/app/pkg/backup/backup.go:448" name=mysql-1

oc get pv -n a1 -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    annotations:
      pv.kubernetes.io/provisioned-by: pd.csi.storage.gke.io
      volume.kubernetes.io/provisioner-deletion-secret-name: ""
      volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
    creationTimestamp: "2023-11-20T09:13:49Z"
    finalizers:
    - kubernetes.io/pv-protection
    - external-attacher/pd-csi-storage-gke-io
    name: pvc-33b119d1-59f5-42b8-9840-cc69cafcef62
    resourceVersion: "81660"
    uid: 32c362e6-1c11-4410-bd84-4f16b4519723
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 2Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: mysql-1
      namespace: a1
      resourceVersion: "81586"
      uid: 33b119d1-59f5-42b8-9840-cc69cafcef62
    csi:
      driver: pd.csi.storage.gke.io
      fsType: ext4
      volumeAttributes:
        storage.kubernetes.io/csiProvisionerIdentity: 1700464484937-8010-pd.csi.storage.gke.io
      volumeHandle: projects/openshift-qe/zones/us-central1-a/disks/pvc-33b119d1-59f5-42b8-9840-cc69cafcef62
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.gke.io/zone
            operator: In
            values:
            - us-central1-a
    persistentVolumeReclaimPolicy: Delete
    storageClassName: standard-csi
    volumeMode: Filesystem
  status:
    phase: Bound
- apiVersion: v1
  kind: PersistentVolume
  metadata:
    annotations:
      pv.kubernetes.io/provisioned-by: pd.csi.storage.gke.io
      volume.kubernetes.io/provisioner-deletion-secret-name: ""
      volume.kubernetes.io/provisioner-deletion-secret-namespace: ""
    creationTimestamp: "2023-11-20T09:13:49Z"
    finalizers:
    - kubernetes.io/pv-protection
    - external-attacher/pd-csi-storage-gke-io
    name: pvc-a9b77c51-9ae6-4b5d-8dbd-c456e179d07c
    resourceVersion: "81659"
    uid: b39d6c34-700c-4242-9bd7-21304b2ba048
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 2Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: mysql
      namespace: a1
      resourceVersion: "81585"
      uid: a9b77c51-9ae6-4b5d-8dbd-c456e179d07c
    csi:
      driver: pd.csi.storage.gke.io
      fsType: ext4
      volumeAttributes:
        storage.kubernetes.io/csiProvisionerIdentity: 1700464484937-8010-pd.csi.storage.gke.io
      volumeHandle: projects/openshift-qe/zones/us-central1-a/disks/pvc-a9b77c51-9ae6-4b5d-8dbd-c456e179d07c
    nodeAffinity:
      required:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.gke.io/zone
            operator: In
            values:
            - us-central1-a
    persistentVolumeReclaimPolicy: Delete
    storageClassName: standard-csi
    volumeMode: Filesystem
  status:
    phase: Bound
kind: List
metadata:
  resourceVersion: ""

Credentials for WIF are in openshift-qe project

There are no Sub-Tasks for this issue.

Details

Description

Description of problem:

VSL backup PartiallyFailed with GCP Workload Identity configured.

oadp-operator-bundle-container-1.3.0-156

100%

Actual results:

Backup finished with PartiallyFailed phase.

Backup should finish successfully with Completed phase.

Attachments

Easy Agile Planning Poker

Sub-Tasks

Activity

People

Dates