Uploaded image for project: 'OpenShift API for Data Protection'
  1. OpenShift API for Data Protection
  2. OADP-4230

fail to patch dynamic PV, err: context deadline exceeded

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • OADP 1.4.1
    • None
    • kubevirt
    • False
    • Hide

      None

      Show
      None
    • False
    • ToDo
    • 0
    • 0
    • Very Likely
    • 0
    • None
    • Unset
    • Unknown
    • No

      Description of problem:

      This behavior is observed when restoring data volumes  and a VMs that the are NOT running

      Restore Partially Failed:

          velero container contains "level=error" in line#2363: time="2024-06-05T15:18:47Z" level=error msg="err patching dynamic PV using volume info" error="fail to patch dynamic PV, err: context deadline exceeded, PVC: test-vm-dv, PV: pvc-1a29c3e7-572e-4430-b78a-d5033d9ca343" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/restore_finalizer_controller.go:351" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*finalizerContext).patchDynamicPVWithVolumeInfo.func1" logSource="pkg/controller/restore_finalizer_controller.go:351" restore=openshift-adp/ocp-kubevirt-f85bddcb-234c-11ef-b64a-0a58ac1c09d3 

       

      Version-Release number of selected component (if applicable):
      oadp-1.4 w/ cnv 4.16 ocp 4.16

       

      How reproducible: 100%

       

      Steps to Reproduce:
      1. Deploy velero

      apiVersion: oadp.openshift.io/v1alpha1
      kind: DataProtectionApplication
      metadata:
        creationTimestamp: '2024-06-05T19:34:30Z'
        generation: 1
        managedFields:
          - apiVersion: oadp.openshift.io/v1alpha1
            fieldsType: FieldsV1
            fieldsV1:
              'f:spec':
                .: {}
                'f:backupLocations': {}
                'f:configuration':
                  .: {}
                  'f:velero':
                    .: {}
                    'f:defaultPlugins': {}
                'f:podDnsConfig': {}
                'f:snapshotLocations': {}
            manager: __debug_bin
            operation: Update
            time: '2024-06-05T19:34:30Z'
          - apiVersion: oadp.openshift.io/v1alpha1
            fieldsType: FieldsV1
            fieldsV1:
              'f:status':
                .: {}
                'f:conditions': {}
            manager: manager
            operation: Update
            subresource: status
            time: '2024-06-05T19:34:30Z'
        name: ts-dpa
        namespace: openshift-adp
        resourceVersion: '290690'
        uid: 2581f7ba-fed1-4fed-b461-82e4272b0e5d
      spec:
        backupLocations:
          - velero:
              credential:
                key: cloud
                name: cloud-credentials-gcp
              default: true
              objectStorage:
                bucket: oadp826614nf7l
                prefix: kubevirt
              provider: gcp
        configuration:
          velero:
            defaultPlugins:
              - openshift
              - gcp
              - kubevirt
              - csi
        podDnsConfig: {}
        snapshotLocations: []
      status:
        conditions:
          - lastTransitionTime: '2024-06-05T19:34:30Z'
            message: Reconcile complete
            reason: Complete
            status: 'True'
            type: Reconciled 

      2. Deploy a datavolume in some ns

       oc get dv/test-dv -n ocp-datavolume  -oyaml
      apiVersion: cdi.kubevirt.io/v1beta1
      kind: DataVolume
      metadata:
        annotations:
          cdi.kubevirt.io/storage.bind.immediate.requested: "true"
          cdi.kubevirt.io/storage.deleteAfterCompletion: "false"
          cdi.kubevirt.io/storage.usePopulator: "true"
        creationTimestamp: "2024-06-05T19:05:41Z"
        generation: 1
        name: test-dv
        namespace: ocp-datavolume
        resourceVersion: "271547"
        uid: 03f70c18-b8e2-409d-a044-79c4795058ca
      spec:
        pvc:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 100Mi
        source:
          blank: {}
      status:
        claimName: test-dv
        conditions:
        - lastHeartbeatTime: "2024-06-05T19:05:59Z"
          lastTransitionTime: "2024-06-05T19:05:59Z"
          message: PVC test-dv Bound
          reason: Bound
          status: "True"
          type: Bound
        - lastHeartbeatTime: "2024-06-05T19:05:59Z"
          lastTransitionTime: "2024-06-05T19:05:59Z"
          status: "True"
          type: Ready
        - lastHeartbeatTime: "2024-06-05T19:05:58Z"
          lastTransitionTime: "2024-06-05T19:05:58Z"
          message: Import Complete
          reason: Completed
          status: "False"
          type: Running
        phase: Succeeded
        progress: 100.0% 

      3. Run a backup

       apiVersion: velero.io/v1
      kind: Backup
      metadata:
        annotations:
          velero.io/resource-timeout: 10m0s
          velero.io/source-cluster-k8s-gitversion: v1.29.5+87992f4
          velero.io/source-cluster-k8s-major-version: "1"
          velero.io/source-cluster-k8s-minor-version: "29"
        creationTimestamp: "2024-06-05T18:00:32Z"
        generation: 7
        labels:
          velero.io/storage-location: ts-dpa-1
        name: ocp-datavolume-5c2a737e-2365-11ef-9c89-0a58ac1c0390
        namespace: openshift-adp
        resourceVersion: "226032"
        uid: 2b8ec996-6ae2-48b7-86fe-28ddb3d92f73
      spec:
        csiSnapshotTimeout: 10m0s
        defaultVolumesToFsBackup: false
        hooks: {}
        includedNamespaces:
        - test-oadp-187
        itemOperationTimeout: 4h0m0s
        metadata: {}
        snapshotMoveData: false
        storageLocation: ts-dpa-1
        ttl: 720h0m0s
      status:
        backupItemOperationsAttempted: 1
        backupItemOperationsCompleted: 1
        completionTimestamp: "2024-06-05T18:01:14Z"
        csiVolumeSnapshotsAttempted: 1
        csiVolumeSnapshotsCompleted: 1
        expiration: "2024-07-05T18:00:32Z"
        formatVersion: 1.1.0
        hookStatus: {}
        phase: Completed
        progress:
          itemsBackedUp: 45
          totalItems: 45
        startTimestamp: "2024-06-05T18:00:32Z"
        version: 1

      4. Remove the datavolume's namespae

      5. Run a restore

      apiVersion: velero.io/v1
      kind: Restore
      metadata:
        creationTimestamp: "2024-06-05T18:01:51Z"
        finalizers:
        - restores.velero.io/external-resources-finalizer
        generation: 7
        name: ocp-datavolume-5c2a737e-2365-11ef-9c89-0a58ac1c0390
        namespace: openshift-adp
        resourceVersion: "234219"
        uid: 3803d412-11be-41df-9c29-246829fdfc19
      spec:
        backupName: ocp-datavolume-5c2a737e-2365-11ef-9c89-0a58ac1c0390
        excludedResources:
        - nodes
        - events
        - events.events.k8s.io
        - backups.velero.io
        - restores.velero.io
        - resticrepositories.velero.io
        - csinodes.storage.k8s.io
        - volumeattachments.storage.k8s.io
        - backuprepositories.velero.io
        hooks: {}
        itemOperationTimeout: 4h0m0s 

      Actual results:

      status:
        completionTimestamp: "2024-06-05T18:11:56Z"
        errors: 1
        hookStatus: {}
        phase: PartiallyFailed
        progress:
          itemsRestored: 24
          totalItems: 24
        startTimestamp: "2024-06-05T18:01:52Z"
        warnings: 8
      

       

      Expected results:

       

      Additional info:

              wnstb Wes Hayutin
              amastbau Amos Mastbaum
              Amos Mastbaum Amos Mastbaum
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: