-

Adding annotations to velero and node-agent Pods
-

What is the nature and description of the request?

We found an issue that has already been fixed in upcoming Velero 1.17.

Velero 1.16/OADP 1.5 and earlier datamover controllers, both upload and download. Will cancel datauploads and datadownloads on node-agent startup in the Accepted phase.

This is fixed in Velero 1.17-dev. https://github.com/vmware-tanzu/velero/pull/8952

This has an unpleasant interaction with cluster-autoscalers.

Creating datamover pods with set resource requests can trigger the cluster-autoscaler to add additional nodes.

The node-agent on startup check for DataUploads and DataDownloads without reservation to assigned nodes for objects in phase Accepted and cancels them. 

Daemonsets run one per node unless nodeSelectors in the DataProtectionApplication is used to limit node assignment to mitigate the issue. Combined with Velero node-agent-config loadAffinity can be used to mitigate the issue. But will cause Restores to fail due to lack of loadAffinity on DataDownload datamover spawning (also being fixed in Velero 1.17 as I recall seeing).

The new nodes spawn new node-agent Pods. The startup of the new node-agent Pods can find DataUpload and DataDownload state Accepted phase and cancel them. With no retry mechanism, this causes these operations to fail.

The cluster-autoscaler with the customer we're working with can control their behavior by setting annotations on Pods. But the existing DPA object lacks the ability to set annotations on nodeAgent annotations. Velero datamovers inherit labels and annotations in Velero 1.16 from their node-agent parent.

Why does the customer need this? (List the business requirements here)

• Adding ability to add annotations will allow for mitigating this issue with control over the cluster-autoscalers the customer uses.

Optional: List affected component/s.

oadp-operator