Description of problem:

OCP Upgrade failing

Version-Release number of the following components:

oc version
Client Version: 4.8.0-202108312109.p0.git.0d10c3f.assembly.stream-0d10c3f
Server Version: 4.10.13
Kubernetes Version: v1.23.5+b463d71

How reproducible: Always

Steps to Reproduce:
1. Create the following SCC (that has `with readOnlyRootFilesystem: true`):
~~~
cat << EOF | oc create -f -
allowHostDirVolumePlugin: true
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowedCapabilities: []
apiVersion: security.openshift.io/v1
defaultAddCapabilities: []
fsGroup:
type: MustRunAs
groups: []
kind: SecurityContextConstraints
metadata:
annotations:
meta.helm.sh/release-name: azure-arc
meta.helm.sh/release-namespace: default
labels:
app.kubernetes.io/managed-by: Helm
name: kube-aad-proxy-scc
priority: null
readOnlyRootFilesystem: true
requiredDropCapabilities: []
runAsUser:
type: RunAsAny
seLinuxContext:
type: MustRunAs
supplementalGroups:
type: RunAsAny
users:

• system:serviceaccount:azure-arc:azure-arc-kube-aad-proxy-sa
  volumes:
• configMap
• hostPath
• secret
  EOF
  ~~~

2. oc adm upgrade --to=4.10.20

Actual results:

SCC kube-aad-proxy-scc, which has readOnlyRootFilesystem is injected inside the pod version-4.10.20-smvt9-6vqwc, causing it to fail.
~~~

1. oc get po -n openshift-cluster-version
   NAME READY STATUS RESTARTS AGE
   cluster-version-operator-6b5c8ff5c8-4bmxx 1/1 Running 0 33m
   version-4.10.20-smvt9-6vqwc 0/1 Error 0 10s
2. oc logs version-4.10.20-smvt9-6vqwc -n openshift-cluster-version
   oc logs version-4.10.20-smvt9-6vqwc
   mv: cannot remove '/manifests/0000_00_cluster-version-operator_00_namespace.yaml': Read-only file system
   mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_adminack_configmap.yaml': Read-only file system
   mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_admingate_configmap.yaml': Read-only file system
   mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_clusteroperator.crd.yaml': Read-only file system
   mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_clusterversion.crd.yaml': Read-only file system
   mv: cannot remove '/manifests/0000_00_cluster-version-operator_02_roles.yaml': Read-only file system
   mv: cannot remove '/manifests/0000_00_cluster-version-operator_03_deployment.yaml': Read-only file system
   mv: cannot remove '/manifests/0000_90_cluster-version-operator_00_prometheusrole.yaml': Read-only file system
   mv: cannot remove '/manifests/0000_90_cluster-version-operator_01_prometheusrolebinding.yaml': Read-only file system
   mv: cannot remove '/manifests/0000_90_cluster-version-operator_02_servicemonitor.yaml': Read-only file system
   mv: cannot remove '/manifests/0001_00_cluster-version-operator_03_service.yaml': Read-only file system
   ~~~

Expected results:

Pod version-4.10.20-smvt9-6vqwc should run fine

Additional info:

I don't know why, but SCC kube-aad-proxy-scc is injected inside pod version-4.10.20-smvt9-6vqwc:
~~~
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.129.0.70"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status: |-
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.129.0.70"
],
"default": true,
"dns": {}
}]
openshift.io/scc: kube-aad-proxy-scc ### HERE
creationTimestamp: "2022-07-25T16:47:39Z"
generateName: version-4.10.20-5xqtv-
labels:
controller-uid: ba707bbe-1825-4f80-89ce-f6bf2301a812
job-name: version-4.10.20-5xqtv
name: version-4.10.20-5xqtv-9gcwk
namespace: openshift-cluster-version
ownerReferences:

• apiVersion: batch/v1
  blockOwnerDeletion: true
  controller: true
  kind: Job
  name: version-4.10.20-5xqtv
  uid: ba707bbe-1825-4f80-89ce-f6bf2301a812
  resourceVersion: "40040"
  uid: 0d668d3d-7452-463f-a421-4dfee9c89c23
  spec:
  containers:
• args:
• -c
• mkdir -p /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww && mv /manifests /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww/manifests
  && mkdir -p /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww && mv /release-manifests
  /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww/release-manifests
  command:
• /bin/sh
  image: quay.io/openshift-release-dev/ocp-release@sha256:b89ada9261a1b257012469e90d7d4839d0d2f99654f5ce76394fa3f06522b600
  imagePullPolicy: IfNotPresent
  name: payload
  resources:
  requests:
  cpu: 10m
  ephemeral-storage: 2Mi
  memory: 50Mi
  securityContext:
  privileged: true
  readOnlyRootFilesystem: true
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
• mountPath: /etc/cvo/updatepayloads
  name: payloads
• mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  name: kube-api-access-fwblb
  readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
• name: default-dockercfg-smmf4
  nodeName: ip-10-0-215-206.eu-central-1.compute.internal
  nodeSelector:
  node-role.kubernetes.io/master: ""
  preemptionPolicy: PreemptLowerPriority
  priority: 1000000000
  priorityClassName: openshift-user-critical
  restartPolicy: OnFailure
  schedulerName: default-scheduler
  securityContext:
  fsGroup: 1000030000
  seLinuxOptions:
  level: s0:c6,c0
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
• key: node-role.kubernetes.io/master
• effect: NoExecute
  key: node.kubernetes.io/not-ready
  operator: Exists
  tolerationSeconds: 300
• effect: NoExecute
  key: node.kubernetes.io/unreachable
  operator: Exists
  tolerationSeconds: 300
• effect: NoSchedule
  key: node.kubernetes.io/memory-pressure
  operator: Exists
  volumes:
• hostPath:
  path: /etc/cvo/updatepayloads
  type: ""
  name: payloads
• name: kube-api-access-fwblb
  projected:
  defaultMode: 420
  sources:
• serviceAccountToken:
  expirationSeconds: 3607
  path: token
• configMap:
  items:
• key: ca.crt
  path: ca.crt
  name: kube-root-ca.crt
• downwardAPI:
  items:
• fieldRef:
  apiVersion: v1
  fieldPath: metadata.namespace
  path: namespace
• configMap:
  items:
• key: service-ca.crt
  path: service-ca.crt
  name: openshift-service-ca.crt
  status:
  conditions:
• lastProbeTime: null
  lastTransitionTime: "2022-07-25T16:47:39Z"
  status: "True"
  type: Initialized
• lastProbeTime: null
  lastTransitionTime: "2022-07-25T16:47:39Z"
  message: 'containers with unready status: [payload]'
  reason: ContainersNotReady
  status: "False"
  type: Ready
• lastProbeTime: null
  lastTransitionTime: "2022-07-25T16:47:39Z"
  message: 'containers with unready status: [payload]'
  reason: ContainersNotReady
  status: "False"
  type: ContainersReady
• lastProbeTime: null
  lastTransitionTime: "2022-07-25T16:47:39Z"
  status: "True"
  type: PodScheduled
  containerStatuses:
• containerID: cri-o://ac6f6a5d8925620f1a2835a50fe26ea02d35e3a5c2d033015f38fde5206daf8c
  image: quay.io/openshift-release-dev/ocp-release@sha256:b89ada9261a1b257012469e90d7d4839d0d2f99654f5ce76394fa3f06522b600
  imageID: quay.io/openshift-release-dev/ocp-release@sha256:b89ada9261a1b257012469e90d7d4839d0d2f99654f5ce76394fa3f06522b600
  lastState:
  terminated:
  containerID: cri-o://fdac85e975eb00a3abd08e18061ae3673a857769ddfc87ca94a3527a8c7b83f3
  exitCode: 1
  finishedAt: "2022-07-25T16:47:42Z"
  reason: Error
  startedAt: "2022-07-25T16:47:42Z"
  name: payload
  ready: false
  restartCount: 2
  started: false
  state:
  terminated:
  containerID: cri-o://ac6f6a5d8925620f1a2835a50fe26ea02d35e3a5c2d033015f38fde5206daf8c
  exitCode: 1
  finishedAt: "2022-07-25T16:47:56Z"
  reason: Error
  startedAt: "2022-07-25T16:47:56Z"
  hostIP: 10.0.215.206
  phase: Running
  podIP: 10.129.0.70
  podIPs:
• ip: 10.129.0.70
  qosClass: Burstable
  startTime: "2022-07-25T16:47:39Z"
  ~~~