Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: OADP 1.1.0
Affects Version/s: OADP 1.1.0
Component/s: velero
Labels:
- velero

Activity Type:
Quality / Stability / Reliability
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Fixed in Build:
oadp-velero-container-1.1.0-24
QEStatus:
Passed

WSJF:
0
Risk Probability:
Very Likely
Risk Score:
0

Workstream:

None

Root Cause:
Unset
Failure Category:
Unknown

Release Blocker:
Approved

Regression:
Yes

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem: second restore of a namespace partially fails

Version-Release number of selected component (if applicable): looks like since 1.1.0-62 the automated incremental restore tests (which are not using restic backup-restore) started to fail with the following error:

[mperetz@mperetz oadp-qe-automation]$ oc logs deploy/velero -n openshift-adp | grep error
time="2022-08-15T11:24:14Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:173"
time="2022-08-15T11:31:24Z" level=error msg="patch attempt failed for /v1, Kind=PersistentVolumeClaim todolist-mariadb/mysql: PersistentVolumeClaim \"mysql\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims\n  core.PersistentVolumeClaimSpec{\n  \tAccessModes:      {\"ReadWriteOnce\"},\n  \tSelector:         nil,\n  \tResources:        {Requests: {s\"storage\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}},\n- \tVolumeName:       \"pvc-783e5c06-f54b-4ca7-9f3c-e6c3807a5ea2\",\n+ \tVolumeName:       \"\",\n  \tStorageClassName: &\"ocs-storagecluster-ceph-rbd\",\n  \tVolumeMode:       &\"Filesystem\",\n  \t... // 2 identical fields\n  }\n" logSource="pkg/restore/restore.go:2045" restore=openshift-adp/todolist-backup-3d163197-1c8d-11ed-9f6d-902e163f806c
time="2022-08-15T11:31:24Z" level=error msg="backup/restore label patch attempt failed for /v1, Kind=Secret todolist-mariadb/default-token-vmxpq: secrets \"default-token-vmxpq\" not found" logSource="pkg/restore/restore.go:2013" restore=openshift-adp/todolist-backup-3d163197-1c8d-11ed-9f6d-902e163f806c
time="2022-08-15T11:31:27Z" level=error msg="Namespace todolist-mariadb, resource restore error: secrets \"default-token-vmxpq\" not found" logSource="pkg/controller/restore_controller.go:504" restore=openshift-adp/todolist-backup-3d163197-1c8d-11ed-9f6d-902e163f806c

So it looks like the issue was introduced on that build, but need to further investigate.

How reproducible: looks like 100%

Steps to Reproduce:
1. Follow https://polarion.engineering.redhat.com/polarion/redirect/project/OADP/workitem?id=OADP-165 steps
2. Note that the second restore partially fails
3.

Actual results:

Second restore partially fails.

DPA:

apiVersion: v1
items:
- apiVersion: oadp.openshift.io/v1alpha1
  kind: DataProtectionApplication
  metadata:
    creationTimestamp: "2022-08-15T13:19:50Z"
    generation: 1
    name: ts-dpa
    namespace: openshift-adp
    resourceVersion: "862832"
    uid: f6c77bda-7288-436a-bdc9-02d179a060dc
  spec:
    backupLocations:
    - velero:
        config:
          insecureSkipTLSVerify: "true"
          profile: noobaa
          region: noobaa
          s3ForcePathStyle: "true"
          s3Url: http://s3.openshift-storage.svc.cluster.local:80
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: oadpbucket129866
          prefix: velero-e2e-eab10ead-1c9c-11ed-8a82-902e163f806c
        provider: aws
    configuration:
      restic:
        podConfig:
          resourceAllocations: {}
      velero:
        defaultPlugins:
        - openshift
        - aws
        - kubevirt
    podDnsConfig: {}
  status:
    conditions:
    - lastTransitionTime: "2022-08-15T13:19:50Z"
      message: Reconcile complete
      reason: Complete
      status: "True"
      type: Reconciled
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

first backup CR:

 apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    velero.io/source-cluster-k8s-gitversion: v1.24.0+9546431
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "24"
  creationTimestamp: "2022-08-15T14:04:00Z"
  generation: 5
  labels:
    velero.io/storage-location: ts-dpa-1
  managedFields:
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:hooks: {}
        f:includedNamespaces: {}
        f:metadata: {}
      f:status: {}
    manager: __debug_bin2391485877
    operation: Update
    time: "2022-08-15T14:04:00Z"
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:velero.io/source-cluster-k8s-gitversion: {}
          f:velero.io/source-cluster-k8s-major-version: {}
          f:velero.io/source-cluster-k8s-minor-version: {}
        f:labels:
          .: {}
          f:velero.io/storage-location: {}
      f:spec:
        f:csiSnapshotTimeout: {}
        f:defaultVolumesToRestic: {}
        f:storageLocation: {}
        f:ttl: {}
      f:status:
        f:completionTimestamp: {}
        f:expiration: {}
        f:formatVersion: {}
        f:phase: {}
        f:progress:
          .: {}
          f:itemsBackedUp: {}
          f:totalItems: {}
        f:startTimestamp: {}
        f:version: {}
    manager: velero-server
    operation: Update
    time: "2022-08-15T14:04:41Z"
  name: todolist-backup-1cb3bd70-1ca3-11ed-bb31-902e163f806c
  namespace: openshift-adp
  resourceVersion: "970220"
  uid: d7d4e930-5f02-4ea9-8a80-d31f2be727d1
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToRestic: false
  hooks: {}
  includedNamespaces:
  - todolist-mariadb
  metadata: {}
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
status:
  completionTimestamp: "2022-08-15T14:04:39Z"
  expiration: "2022-09-14T14:04:00Z"
  formatVersion: 1.1.0
  phase: Completed
  progress:
    itemsBackedUp: 64
    totalItems: 64
  startTimestamp: "2022-08-15T14:04:00Z"
  version: 1

second backup CR:

 apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    velero.io/source-cluster-k8s-gitversion: v1.24.0+9546431
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "24"
  creationTimestamp: "2022-08-15T14:04:58Z"
  generation: 5
  labels:
    velero.io/storage-location: ts-dpa-1
  managedFields:
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:hooks: {}
        f:includedNamespaces: {}
        f:metadata: {}
      f:status: {}
    manager: __debug_bin2391485877
    operation: Update
    time: "2022-08-15T14:04:58Z"
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:velero.io/source-cluster-k8s-gitversion: {}
          f:velero.io/source-cluster-k8s-major-version: {}
          f:velero.io/source-cluster-k8s-minor-version: {}
        f:labels:
          .: {}
          f:velero.io/storage-location: {}
      f:spec:
        f:csiSnapshotTimeout: {}
        f:defaultVolumesToRestic: {}
        f:storageLocation: {}
        f:ttl: {}
      f:status:
        f:completionTimestamp: {}
        f:expiration: {}
        f:formatVersion: {}
        f:phase: {}
        f:progress:
          .: {}
          f:itemsBackedUp: {}
          f:totalItems: {}
        f:startTimestamp: {}
        f:version: {}
    manager: velero-server
    operation: Update
    time: "2022-08-15T14:05:38Z"
  name: todolist-backup-3f6c1467-1ca3-11ed-bb31-902e163f806c
  namespace: openshift-adp
  resourceVersion: "972257"
  uid: 334ff4c0-2ea8-46aa-a498-d170008170a3
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToRestic: false
  hooks: {}
  includedNamespaces:
  - todolist-mariadb
  metadata: {}
  storageLocation: ts-dpa-1
  ttl: 720h0m0s
status:
  completionTimestamp: "2022-08-15T14:05:37Z"
  expiration: "2022-09-14T14:04:58Z"
  formatVersion: 1.1.0
  phase: Completed
  progress:
    itemsBackedUp: 76
    totalItems: 76
  startTimestamp: "2022-08-15T14:04:58Z"
  version: 1

first restore CR:

 apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: "2022-08-15T14:06:37Z"
  generation: 8
  managedFields:
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:backupName: {}
        f:hooks: {}
      f:status: {}
    manager: __debug_bin2391485877
    operation: Update
    time: "2022-08-15T14:06:37Z"
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:excludedResources: {}
      f:status:
        f:completionTimestamp: {}
        f:phase: {}
        f:progress:
          .: {}
          f:itemsRestored: {}
          f:totalItems: {}
        f:startTimestamp: {}
        f:warnings: {}
    manager: velero-server
    operation: Update
    time: "2022-08-15T14:06:57Z"
  name: todolist-backup-1cb3bd70-1ca3-11ed-bb31-902e163f806c
  namespace: openshift-adp
  resourceVersion: "974992"
  uid: 2b6842ce-6a79-4094-a1e4-f8637041436f
spec:
  backupName: todolist-backup-1cb3bd70-1ca3-11ed-bb31-902e163f806c
  excludedResources:
  - volumesnapshotclasses.snapshot.storage.k8s.io
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  hooks: {}
status:
  completionTimestamp: "2022-08-15T14:06:57Z"
  phase: Completed
  progress:
    itemsRestored: 41
    totalItems: 41
  startTimestamp: "2022-08-15T14:06:37Z"
  warnings: 11

second restore:
**

apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: "2022-08-15T14:09:14Z"
  generation: 10
  managedFields:
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:backupName: {}
        f:existingResourcePolicy: {}
        f:hooks: {}
      f:status: {}
    manager: __debug_bin2391485877
    operation: Update
    time: "2022-08-15T14:09:14Z"
  - apiVersion: velero.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:excludedResources: {}
      f:status:
        f:completionTimestamp: {}
        f:errors: {}
        f:phase: {}
        f:progress:
          .: {}
          f:itemsRestored: {}
          f:totalItems: {}
        f:startTimestamp: {}
        f:warnings: {}
    manager: velero-server
    operation: Update
    time: "2022-08-15T14:09:36Z"
  name: todolist-backup-3f6c1467-1ca3-11ed-bb31-902e163f806c
  namespace: openshift-adp
  resourceVersion: "984453"
  uid: c7a56325-1108-42ac-a42e-84a95f225397
spec:
  backupName: todolist-backup-3f6c1467-1ca3-11ed-bb31-902e163f806c
  excludedResources:
  - volumesnapshotclasses.snapshot.storage.k8s.io
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  existingResourcePolicy: update
  hooks: {}
status:
  completionTimestamp: "2022-08-15T14:09:36Z"
  errors: 2
  phase: PartiallyFailed
  progress:
    itemsRestored: 41
    totalItems: 41
  startTimestamp: "2022-08-15T14:09:14Z"
  warnings: 1

Expected results:

Additional info:

links to

openshift/velero#189: OADP-716: Update Condition to fetch obj from cluster and change label patch log failure from error to warn

openshift/velero#190: [konveyor-1.9.1] OADP-716: Update Condition to fetch obj from cluster and change label patch log failure from error to warn

openshift/velero#191: [oadp-1.1] OADP-716: Update Condition to fetch obj from cluster and change label patch log failure from error to warn

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates