Description of problem:

Restore is partially failing on OCP 4.15 with OADP 1.3.1, verified this with build 35 which is the latest and with older builds as well.

Velero logs contains following error.

oc logs velero-57596cdc85-846lj -n openshift-adp  | grep error

Defaulted container "velero" out of: velero, openshift-velero-plugin (init), velero-plugin-for-aws (init), velero-plugin-for-csi (init)

time="2024-02-28T05:43:35Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1)" controller=backup-storage-location logSource="/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:194"

time="2024-02-28T06:38:53Z" level=warning msg="active indexes [] deletion watermark 0001-01-01 00:00:00 +0000 UTC" logModule=kopia/kopia/format logSource="/remote-source/velero/app/pkg/kopia/kopia_log.go:101" sublevel=error

time="2024-02-28T06:38:54Z" level=warning msg="active indexes [] deletion watermark 0001-01-01 00:00:00 +0000 UTC" logModule=kopia/kopia/format logSource="/remote-source/velero/app/pkg/kopia/kopia_log.go:101" sublevel=error

time="2024-02-28T06:38:57Z" level=warning msg="active indexes [xn0_f613e240834400bc22c5e0da10f3b8ec-sb03a425b07387896126-c1] deletion watermark 0001-01-01 00:00:00 +0000 UTC" logModule=kopia/kopia/format logSource="/remote-source/velero/app/pkg/kopia/kopia_log.go:101" sublevel=error

time="2024-02-28T06:46:31Z" level=error msg="Namespace mysql-persistent, resource restore error: error preparing pods/mysql-persistent/mysql-6b49bd67c7-x4nwr: rpc error: code = Unknown desc = Secret is not getting created" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:567" restore=openshift-adp/restore-mysql

However, the application restore completes and all the resources appears to be in good state.

[root@m1320001 1.3]# oc get all -n mysql-persistent
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                         READY   STATUS    RESTARTS   AGE
pod/mysql-6b49bd67c7-x5kb7   1/1     Running   0          35m


NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/mysql   ClusterIP   172.30.187.202   <none>        3306/TCP   35m


NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql   1/1     1            1           35m


NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-6b49bd67c7   1         1         1       35m
[root@m1320001 1.3]# oc get secret -n mysql-persistent
NAME    TYPE     DATA   AGE
mysql   Opaque   4      42m
[root@m1320001 1.3]#  

Version-Release number of selected component (if applicable):

OADP 1.3.1-35

[root@m1320001 1.3]# oc version
Client Version: 4.15.0-rc.7
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: 4.15.0-rc.7
Kubernetes Version: v1.28.6+f1618d5 

How reproducible:

Always

Steps to Reproduce:
1. Deploy mysql app
2. Take backup using FileSystemBackup (Kopia) 
3. Restore it either on same cluster after deleting the mysql project or restore it on another cluster

Actual results:

oc get restore  -n openshift-adp restore-mysql -o jsonpath='{.status.phase}'

PartiallyFailed

Expected results:

Restore should complete fine.

Additional info:
Dev notes:

• slack: https://redhat-internal.slack.com/archives/C0144ECKUJ0/p1709567060841249
• tracing back how this secret is created
  https://github.com/openshift/openshift-controller-manager/blob/master/pkg/cmd/openshift-controller-manager/controller_manager.go#L140
  https://github.com/openshift/openshift-controller-manager/blob/master/pkg/cmd/controller/config.go#L12
  https://github.com/openshift/openshift-controller-manager/blob/master/pkg/cmd/controller/pullsecrets.go#L26
  https://github.com/openshift/openshift-controller-manager/blob/3369227f301f7bc98e3f364c9bfb6fff788302d6/pkg/serviceaccounts/controllers/create_dockercfg_secrets.go#L115
  https://github.com/openshift/openshift-controller-manager/blob/3369227f301f7bc98e3f364c9bfb6fff788302d6/pkg/serviceaccounts/controllers/create_dockercfg_secrets.go#L354
  reveals that docker-cfg- secret is created when OpenShiftServiceAccountPullSecretsController is enabled.

asking imageregistry in https://redhat-internal.slack.com/archives/C013VBYBJQH/p1709952439633439