Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-31462

Upgrade failing because custom scc in version pod

XMLWordPrintable

    • Important
    • No
    • 3
    • OTA 262, OTA 263
    • 2
    • False
    • Hide

      None

      Show
      None
    • Hide
      *Cause*: A custom Security Context Constraint (SCC) affects a Pod generated by the Cluster Version Operator.
      *Consequence*: A cluster version upgrade may fail.
      *Fix*: An OpenShift default SCC is pinned to the Pod. A custom SCC will have no impact on the Pod.
      *Result*: Bug doesn’t present anymore.
      Show
      *Cause*: A custom Security Context Constraint (SCC) affects a Pod generated by the Cluster Version Operator. *Consequence*: A cluster version upgrade may fail. *Fix*: An OpenShift default SCC is pinned to the Pod. A custom SCC will have no impact on the Pod. *Result*: Bug doesn’t present anymore.
    • Bug Fix
    • In Progress

      Description of problem:

      Openshift cluster upgrade from 4.12.10 to 4.12.30 failing because pod version-4.12.30-xxx is in CreateContainerConfigError. Also tested in 4.14

      Steps to Reproduce:

      • Deploy new 4.12.10 cluster
      $ oc get clusterversion
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.12.10   True        False         12m     Cluster version is 4.12.10
      
      • Create the following SCC
      ---
      allowHostDirVolumePlugin: true
      allowHostIPC: false
      allowHostNetwork: false
      allowHostPID: false
      allowHostPorts: false
      allowPrivilegeEscalation: true
      allowPrivilegedContainer: true
      allowedCapabilities: null
      apiVersion: security.openshift.io/v1
      defaultAddCapabilities: null
      fsGroup:
        type: RunAsAny
      groups: []
      kind: SecurityContextConstraints
      metadata:
        name: scc-hostpath-cnf-cat-1
      priority: null
      readOnlyRootFilesystem: false
      requiredDropCapabilities:
      - KILL
      - MKNOD
      - SETUID
      - SETGID
      runAsUser:
        type: MustRunAsNonRoot
      seLinuxContext:
        type: MustRunAs
      supplementalGroups:
        type: RunAsAny
      users: []
      volumes:
      - configMap
      - downwardAPI
      - emptyDir
      - hostPath
      - persistentVolumeClaim
      - projected
      - secret
      
      • Upgrade to 4.12.30
      $ oc adm upgrade --to=4.12.30
      $ oc get pod -n openshift-cluster-version
      NAME                                        READY   STATUS                            RESTARTS   AGE
      cluster-version-operator-85db98885c-jt25z   1/1     Running                           0          41m
      version-4.12.30-vw4pm-l2nng                 0/1     Init:CreateContainerConfigError   0          42s
      
      $ oc get events | grep Failed
      10s         Warning   Failed                  pod/version-4.12.30-p6k4r-nmn6m                  Error: container has runAsNonRoot and image will run as root (pod: "version-4.12.30-p6k4r-nmn6m_openshift-cluster-version(4d1704d9-ca34-4aa3-86e1-1742e8cead0c)", container: cleanup)
      
      $ oc get pod version-4.12.30-97nbr-88mxp -o yaml  |grep scc
          openshift.io/scc: scc-hostpath-cnf-cat-1
      

      As a workaround, we can remove the scc "scc-hostpath-cnf-cat-1" and the pod version-xxx and the upgrade worked. Customer has created custom scc for use of applications.

      $ oc get pod version-4.12.30-nmskz-d5x2c -o yaml | grep scc
          openshift.io/scc: node-exporter
      
      $ oc get pod
      NAME                                        READY   STATUS      RESTARTS   AGE
      cluster-version-operator-6cb5557f8f-v65vb   1/1     Running     0          54s
      version-4.12.30-nmskz-d5x2c                 0/1     Completed   0          67s
      

      There's an old bug https://issues.redhat.com/browse/OCPBUGSM-47192 which was fixed setting readOnlyRootFilesystem to false, but in this case the scc is still failing.

      https://github.com/openshift/cluster-version-operator/blob/release-4.12/pkg/cvo/updatepayload.go#L206

      ---
      container.SecurityContext = &corev1.SecurityContext{
      	Privileged:             pointer.BoolPtr(true),
      	ReadOnlyRootFilesystem: pointer.BoolPtr(false),
      }
      ---
      

              dhurta@redhat.com David Hurta
              rhn-support-jclaretm Jorge Claret Membrado
              Dinesh Kumar S Dinesh Kumar S
              Votes:
              1 Vote for this issue
              Watchers:
              17 Start watching this issue

                Created:
                Updated: