Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-233

Upgrade failing because restrictive scc is injected into version pod

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Undefined
    • Resolution: Done
    • 4.10
    • None
    • None
    • Moderate
    • OTA 223
    • 1
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      OCP Upgrade failing

      Version-Release number of the following components:

      oc version
      Client Version: 4.8.0-202108312109.p0.git.0d10c3f.assembly.stream-0d10c3f
      Server Version: 4.10.13
      Kubernetes Version: v1.23.5+b463d71

      How reproducible: Always

      Steps to Reproduce:
      1. Create the following SCC (that has `with readOnlyRootFilesystem: true`):
      ~~~
      cat << EOF | oc create -f -
      allowHostDirVolumePlugin: true
      allowHostIPC: false
      allowHostNetwork: false
      allowHostPID: false
      allowHostPorts: false
      allowPrivilegeEscalation: true
      allowPrivilegedContainer: true
      allowedCapabilities: []
      apiVersion: security.openshift.io/v1
      defaultAddCapabilities: []
      fsGroup:
      type: MustRunAs
      groups: []
      kind: SecurityContextConstraints
      metadata:
      annotations:
      meta.helm.sh/release-name: azure-arc
      meta.helm.sh/release-namespace: default
      labels:
      app.kubernetes.io/managed-by: Helm
      name: kube-aad-proxy-scc
      priority: null
      readOnlyRootFilesystem: true
      requiredDropCapabilities: []
      runAsUser:
      type: RunAsAny
      seLinuxContext:
      type: MustRunAs
      supplementalGroups:
      type: RunAsAny
      users:

      • system:serviceaccount:azure-arc:azure-arc-kube-aad-proxy-sa
        volumes:
      • configMap
      • hostPath
      • secret
        EOF
        ~~~

      2. oc adm upgrade --to=4.10.20

      Actual results:

      SCC kube-aad-proxy-scc, which has readOnlyRootFilesystem is injected inside the pod version-4.10.20-smvt9-6vqwc, causing it to fail.
      ~~~

      1. oc get po -n openshift-cluster-version
        NAME READY STATUS RESTARTS AGE
        cluster-version-operator-6b5c8ff5c8-4bmxx 1/1 Running 0 33m
        version-4.10.20-smvt9-6vqwc 0/1 Error 0 10s
      2. oc logs version-4.10.20-smvt9-6vqwc -n openshift-cluster-version
        oc logs version-4.10.20-smvt9-6vqwc
        mv: cannot remove '/manifests/0000_00_cluster-version-operator_00_namespace.yaml': Read-only file system
        mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_adminack_configmap.yaml': Read-only file system
        mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_admingate_configmap.yaml': Read-only file system
        mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_clusteroperator.crd.yaml': Read-only file system
        mv: cannot remove '/manifests/0000_00_cluster-version-operator_01_clusterversion.crd.yaml': Read-only file system
        mv: cannot remove '/manifests/0000_00_cluster-version-operator_02_roles.yaml': Read-only file system
        mv: cannot remove '/manifests/0000_00_cluster-version-operator_03_deployment.yaml': Read-only file system
        mv: cannot remove '/manifests/0000_90_cluster-version-operator_00_prometheusrole.yaml': Read-only file system
        mv: cannot remove '/manifests/0000_90_cluster-version-operator_01_prometheusrolebinding.yaml': Read-only file system
        mv: cannot remove '/manifests/0000_90_cluster-version-operator_02_servicemonitor.yaml': Read-only file system
        mv: cannot remove '/manifests/0001_00_cluster-version-operator_03_service.yaml': Read-only file system
        ~~~

      Expected results:

      Pod version-4.10.20-smvt9-6vqwc should run fine

      Additional info:

      I don't know why, but SCC kube-aad-proxy-scc is injected inside pod version-4.10.20-smvt9-6vqwc:
      ~~~
      apiVersion: v1
      kind: Pod
      metadata:
      annotations:
      k8s.v1.cni.cncf.io/network-status: |-
      [{
      "name": "openshift-sdn",
      "interface": "eth0",
      "ips": [
      "10.129.0.70"
      ],
      "default": true,
      "dns": {}
      }]
      k8s.v1.cni.cncf.io/networks-status: |-
      [{
      "name": "openshift-sdn",
      "interface": "eth0",
      "ips": [
      "10.129.0.70"
      ],
      "default": true,
      "dns": {}
      }]
      openshift.io/scc: kube-aad-proxy-scc ### HERE
      creationTimestamp: "2022-07-25T16:47:39Z"
      generateName: version-4.10.20-5xqtv-
      labels:
      controller-uid: ba707bbe-1825-4f80-89ce-f6bf2301a812
      job-name: version-4.10.20-5xqtv
      name: version-4.10.20-5xqtv-9gcwk
      namespace: openshift-cluster-version
      ownerReferences:

      • apiVersion: batch/v1
        blockOwnerDeletion: true
        controller: true
        kind: Job
        name: version-4.10.20-5xqtv
        uid: ba707bbe-1825-4f80-89ce-f6bf2301a812
        resourceVersion: "40040"
        uid: 0d668d3d-7452-463f-a421-4dfee9c89c23
        spec:
        containers:
      • args:
      • -c
      • mkdir -p /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww && mv /manifests /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww/manifests
        && mkdir -p /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww && mv /release-manifests
        /etc/cvo/updatepayloads/KsrCX7X9QbtoXkW3TkPcww/release-manifests
        command:
      • /bin/sh
        image: quay.io/openshift-release-dev/ocp-release@sha256:b89ada9261a1b257012469e90d7d4839d0d2f99654f5ce76394fa3f06522b600
        imagePullPolicy: IfNotPresent
        name: payload
        resources:
        requests:
        cpu: 10m
        ephemeral-storage: 2Mi
        memory: 50Mi
        securityContext:
        privileged: true
        readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
      • mountPath: /etc/cvo/updatepayloads
        name: payloads
      • mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        name: kube-api-access-fwblb
        readOnly: true
        dnsPolicy: ClusterFirst
        enableServiceLinks: true
        imagePullSecrets:
      • name: default-dockercfg-smmf4
        nodeName: ip-10-0-215-206.eu-central-1.compute.internal
        nodeSelector:
        node-role.kubernetes.io/master: ""
        preemptionPolicy: PreemptLowerPriority
        priority: 1000000000
        priorityClassName: openshift-user-critical
        restartPolicy: OnFailure
        schedulerName: default-scheduler
        securityContext:
        fsGroup: 1000030000
        seLinuxOptions:
        level: s0:c6,c0
        serviceAccount: default
        serviceAccountName: default
        terminationGracePeriodSeconds: 30
        tolerations:
      • key: node-role.kubernetes.io/master
      • effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 300
      • effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 300
      • effect: NoSchedule
        key: node.kubernetes.io/memory-pressure
        operator: Exists
        volumes:
      • hostPath:
        path: /etc/cvo/updatepayloads
        type: ""
        name: payloads
      • name: kube-api-access-fwblb
        projected:
        defaultMode: 420
        sources:
      • serviceAccountToken:
        expirationSeconds: 3607
        path: token
      • configMap:
        items:
      • key: ca.crt
        path: ca.crt
        name: kube-root-ca.crt
      • downwardAPI:
        items:
      • fieldRef:
        apiVersion: v1
        fieldPath: metadata.namespace
        path: namespace
      • configMap:
        items:
      • key: service-ca.crt
        path: service-ca.crt
        name: openshift-service-ca.crt
        status:
        conditions:
      • lastProbeTime: null
        lastTransitionTime: "2022-07-25T16:47:39Z"
        status: "True"
        type: Initialized
      • lastProbeTime: null
        lastTransitionTime: "2022-07-25T16:47:39Z"
        message: 'containers with unready status: [payload]'
        reason: ContainersNotReady
        status: "False"
        type: Ready
      • lastProbeTime: null
        lastTransitionTime: "2022-07-25T16:47:39Z"
        message: 'containers with unready status: [payload]'
        reason: ContainersNotReady
        status: "False"
        type: ContainersReady
      • lastProbeTime: null
        lastTransitionTime: "2022-07-25T16:47:39Z"
        status: "True"
        type: PodScheduled
        containerStatuses:
      • containerID: cri-o://ac6f6a5d8925620f1a2835a50fe26ea02d35e3a5c2d033015f38fde5206daf8c
        image: quay.io/openshift-release-dev/ocp-release@sha256:b89ada9261a1b257012469e90d7d4839d0d2f99654f5ce76394fa3f06522b600
        imageID: quay.io/openshift-release-dev/ocp-release@sha256:b89ada9261a1b257012469e90d7d4839d0d2f99654f5ce76394fa3f06522b600
        lastState:
        terminated:
        containerID: cri-o://fdac85e975eb00a3abd08e18061ae3673a857769ddfc87ca94a3527a8c7b83f3
        exitCode: 1
        finishedAt: "2022-07-25T16:47:42Z"
        reason: Error
        startedAt: "2022-07-25T16:47:42Z"
        name: payload
        ready: false
        restartCount: 2
        started: false
        state:
        terminated:
        containerID: cri-o://ac6f6a5d8925620f1a2835a50fe26ea02d35e3a5c2d033015f38fde5206daf8c
        exitCode: 1
        finishedAt: "2022-07-25T16:47:56Z"
        reason: Error
        startedAt: "2022-07-25T16:47:56Z"
        hostIP: 10.0.215.206
        phase: Running
        podIP: 10.129.0.70
        podIPs:
      • ip: 10.129.0.70
        qosClass: Burstable
        startTime: "2022-07-25T16:47:39Z"
        ~~~

      Attachments

        Issue Links

          Activity

            People

              lmohanty@redhat.com Lalatendu Mohanty
              trking W. Trevor King
              Yang Yang Yang Yang
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: