Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-31462

Upgrade failing because custom scc in version pod

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.12.z, 4.14.z
    • None
    • Important
    • No
    • 3
    • OTA 262
    • 1
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Openshift cluster upgrade from 4.12.10 to 4.12.30 failing because pod version-4.12.30-xxx is in CreateContainerConfigError. Also tested in 4.14

      Steps to Reproduce:

      • Deploy new 4.12.10 cluster
      $ oc get clusterversion
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.12.10   True        False         12m     Cluster version is 4.12.10
      
      • Create the following SCC
      ---
      allowHostDirVolumePlugin: true
      allowHostIPC: false
      allowHostNetwork: false
      allowHostPID: false
      allowHostPorts: false
      allowPrivilegeEscalation: true
      allowPrivilegedContainer: true
      allowedCapabilities: null
      apiVersion: security.openshift.io/v1
      defaultAddCapabilities: null
      fsGroup:
        type: RunAsAny
      groups: []
      kind: SecurityContextConstraints
      metadata:
        name: scc-hostpath-cnf-cat-1
      priority: null
      readOnlyRootFilesystem: false
      requiredDropCapabilities:
      - KILL
      - MKNOD
      - SETUID
      - SETGID
      runAsUser:
        type: MustRunAsNonRoot
      seLinuxContext:
        type: MustRunAs
      supplementalGroups:
        type: RunAsAny
      users: []
      volumes:
      - configMap
      - downwardAPI
      - emptyDir
      - hostPath
      - persistentVolumeClaim
      - projected
      - secret
      
      • Upgrade to 4.12.30
      $ oc adm upgrade --to=4.12.30
      $ oc get pod -n openshift-cluster-version
      NAME                                        READY   STATUS                            RESTARTS   AGE
      cluster-version-operator-85db98885c-jt25z   1/1     Running                           0          41m
      version-4.12.30-vw4pm-l2nng                 0/1     Init:CreateContainerConfigError   0          42s
      
      $ oc get events | grep Failed
      10s         Warning   Failed                  pod/version-4.12.30-p6k4r-nmn6m                  Error: container has runAsNonRoot and image will run as root (pod: "version-4.12.30-p6k4r-nmn6m_openshift-cluster-version(4d1704d9-ca34-4aa3-86e1-1742e8cead0c)", container: cleanup)
      
      $ oc get pod version-4.12.30-97nbr-88mxp -o yaml  |grep scc
          openshift.io/scc: scc-hostpath-cnf-cat-1
      

      As a workaround, we can remove the scc "scc-hostpath-cnf-cat-1" and the pod version-xxx and the upgrade worked. Customer has created custom scc for use of applications.

      $ oc get pod version-4.12.30-nmskz-d5x2c -o yaml | grep scc
          openshift.io/scc: node-exporter
      
      $ oc get pod
      NAME                                        READY   STATUS      RESTARTS   AGE
      cluster-version-operator-6cb5557f8f-v65vb   1/1     Running     0          54s
      version-4.12.30-nmskz-d5x2c                 0/1     Completed   0          67s
      

      There's an old bug https://issues.redhat.com/browse/OCPBUGSM-47192 which was fixed setting readOnlyRootFilesystem to false, but in this case the scc is still failing.

      https://github.com/openshift/cluster-version-operator/blob/release-4.12/pkg/cvo/updatepayload.go#L206

      ---
      container.SecurityContext = &corev1.SecurityContext{
      	Privileged:             pointer.BoolPtr(true),
      	ReadOnlyRootFilesystem: pointer.BoolPtr(false),
      }
      ---
      

              dhurta@redhat.com David Hurta
              rhn-support-jclaretm Jorge Claret Membrado
              Rahul Gangwar Rahul Gangwar
              Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

                Created:
                Updated: