Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14050

SPO installation failed when a cluster scaled up rhel worker nodes

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.14.0
    • None
    • Important
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

       

      For a gcp-ipi-workers-rhel8-fips cluster(master nodes are rhcos nodes, worker nodes are rhel8 nodes), SPO installation failed. The spod pods failed to deploy on the master nodes due to "error parsing semanage configuration file: syntax error".
      Details seen from below.
      $ oc get node
      NAME                                                          STATUS   ROLES                  AGE    VERSION
      ci-op-jixzl7m3-b996e-cntz5-master-0.c.openshift-qe.internal   Ready    control-plane,master   103m   v1.27.1+38c64ac
      ci-op-jixzl7m3-b996e-cntz5-master-1.c.openshift-qe.internal   Ready    control-plane,master   102m   v1.27.1+38c64ac
      ci-op-jixzl7m3-b996e-cntz5-master-2.c.openshift-qe.internal   Ready    control-plane,master   102m   v1.27.1+38c64ac
      ci-op-jixzl7m3-b996e-cntz5-rhel-1                             Ready    worker                 52m    v1.27.2+fa7eb19
      ci-op-jixzl7m3-b996e-cntz5-rhel-2                             Ready    worker                 52m    v1.27.2+fa7eb19
      $ oc get pod
      NAME                                                  READY   STATUS                  RESTARTS        AGE
      security-profiles-operator-57cc486f79-826ss           1/1     Running                 0               43m
      security-profiles-operator-57cc486f79-bjj9v           1/1     Running                 0               43m
      security-profiles-operator-57cc486f79-c2xr7           1/1     Running                 0               43m
      security-profiles-operator-webhook-6757b54bcb-7nvqz   1/1     Running                 0               43m
      security-profiles-operator-webhook-6757b54bcb-dvhh8   1/1     Running                 0               43m
      security-profiles-operator-webhook-6757b54bcb-wqqlf   1/1     Running                 0               43m
      spod-2z9hv                                            0/4     Init:CrashLoopBackOff   13 (113s ago)   43m
      spod-57mn9                                            4/4     Running                 0               43m
      spod-gw5nk                                            0/4     Init:CrashLoopBackOff   13 (2m2s ago)   43m
      spod-ttgmg                                            0/4     Init:CrashLoopBackOff   13 (105s ago)   43m
      spod-zlgqm                                            4/4     Running                 0               43m
      $ oc logs pod/spod-2z9hv --all-containers
      2023-05-25T04:34:45.713566204Z + semodule -i /usr/share/selinuxd/templates/base_container.cil /usr/share/selinuxd/templates/config_container.cil /usr/share/selinuxd/templates/home_container.cil /usr/share/selinuxd/templates/log_container.cil /usr/share/selinuxd/templates/net_container.cil /usr/share/selinuxd/templates/tmp_container.cil /usr/share/selinuxd/templates/tty_container.cil /usr/share/selinuxd/templates/virt_container.cil /usr/share/selinuxd/templates/x_container.cil
      2023-05-25T04:34:45.716680671Z error parsing semanage configuration file: syntax error
      2023-05-25T04:34:45.716791667Z semodule:  Could not create semanage handle
      2023-05-25T04:34:45.717167638Z + semodule -i /opt/spo-profiles/selinuxd.cil
      2023-05-25T04:34:45.719836493Z error parsing semanage configuration file: syntax error
      2023-05-25T04:34:45.719836493Z semodule:  Could not create semanage handle
      2023-05-25T04:34:45.720187066Z + semodule -i /opt/spo-profiles/selinuxrecording.cil
      2023-05-25T04:34:45.722516873Z error parsing semanage configuration file: syntax error
      2023-05-25T04:34:45.722613459Z semodule:  Could not create semanage handle
      

       

      Version-Release number of selected component (if applicable):

      4.14.0-0.nightly-2023-05-23-103225 + Security-profiles-operator.v0.7.1

      How reproducible:

      Always

      Steps to Reproduce:

      1. Install a gcp-ipi-workers-rhel8-fips cluster
      2. Install Security Profiles Operator 

      Actual results:

      Security Profiles Operator installation failed. Details seen from the description.

      Expected results:

      Security Profiles Operator installation succeeded

      Additional info:

      There is no such issue on a fips enable, rhcos nodes only(no rhel nodes) cluster.
      But there is similar issues when a cluster have both rhcos workers and rhel workers
      $ oc get pod -o wide
      NAME                                                  READY   STATUS       RESTARTS      AGE    IP             NODE                                                       NOMINATED NODE   READINESS GATES
      security-profiles-operator-57cc486f79-7pqq9           1/1     Running      0             102s   10.x.x.x    jiwei-0525a-gfn9m-worker-c-rmf8q.c.openshift-qe.internal   <none>           <none>
      security-profiles-operator-57cc486f79-qf2v7           1/1     Running      0             102s   10.x.x.x    jiwei-0525a-gfn9m-worker-a-hw2cd.c.openshift-qe.internal   <none>           <none>
      security-profiles-operator-57cc486f79-wbb5r           1/1     Running      0             102s   10.x.x.x    jiwei-0525a-gfn9m-worker-b-7k675.c.openshift-qe.internal   <none>           <none>
      security-profiles-operator-webhook-6757b54bcb-52vdm   1/1     Running      0             99s    10.x.x.x    jiwei-0525a-gfn9m-worker-c-rmf8q.c.openshift-qe.internal   <none>           <none>
      security-profiles-operator-webhook-6757b54bcb-5dv2v   1/1     Running      0             99s    10.x.x.x    jiwei-0525a-gfn9m-worker-b-7k675.c.openshift-qe.internal   <none>           <none>
      security-profiles-operator-webhook-6757b54bcb-q72tk   1/1     Running      0             99s    10.x.x.x    jiwei-0525a-gfn9m-worker-a-hw2cd.c.openshift-qe.internal   <none>           <none>
      spod-2z9dn                                            4/4     Running      0             99s     10.x.x.x    jiwei-0525a-gfn9m-worker-b-7k675.c.openshift-qe.internal   <none>           <none>
      spod-4gr26                                            4/4     Running      0             99s    10.x.x.x    jiwei-0525a-gfn9m-worker-a-hw2cd.c.openshift-qe.internal   <none>           <none>
      spod-5gsnl                                            4/4     Running      0             99s    10.x.x.x    jiwei-0525a-gfn9m-worker-c-rmf8q.c.openshift-qe.internal   <none>           <none>
      spod-62mws                                            0/4     Init:Error   3 (38s ago)   99s    10.x.x.x     jiwei-0525a-gfn9m-w-a-l-rhel-0                             <none>           <none>
      spod-6hwtp                                            4/4     Running      0             99s    10.x.x.x    jiwei-0525a-gfn9m-master-1.c.openshift-qe.internal         <none>           <none>
      spod-cvth4                                            4/4     Running      0             99s    10.x.x.x    jiwei-0525a-gfn9m-master-0.c.openshift-qe.internal         <none>           <none>
      spod-tx5br                                            0/4     Init:1/2     3 (32s ago)   99s    10.x.x.x     jiwei-0525a-gfn9m-w-a-l-rhel-1                             <none>           <none>
      spod-xbhnb                                            4/4     Running      0             99s    10.x.x.x   jiwei-0525a-gfn9m-master-2.c.openshift-qe.internal         <none>           <none>
      $ oc logs pod/spod-62mws --all-containers
      ...
      + semodule -i /usr/share/selinuxd/templates/base_container.cil /usr/share/selinuxd/templates/config_container.cil /usr/share/selinuxd/templates/home_container.cil /usr/share/selinuxd/templates/log_container.cil /usr/share/selinuxd/templates/net_container.cil /usr/share/selinuxd/templates/tmp_container.cil /usr/share/selinuxd/templates/tty_container.cil /usr/share/selinuxd/templates/virt_container.cil /usr/share/selinuxd/templates/x_container.cil
      Failed to resolve permission audit_write
      Failed to resolve allow statement at /var/lib/selinux/targeted/tmp/modules/400/openvswitch-custom/cil:75
      Failed to resolve AST
      semodule:  Failed!
      + semodule -i /opt/spo-profiles/selinuxd.cil
      Failed to resolve blockinherit statement at /var/lib/selinux/targeted/tmp/modules/400/selinuxd/cil:2
      Failed to resolve AST
      semodule:  Failed!
      + semodule -i /opt/spo-profiles/selinuxrecording.cil
      Failed to resolve blockinherit statement at /var/lib/selinux/targeted/tmp/modules/400/selinuxrecording/cil:2
      Failed to resolve AST
      semodule:  Failed!
      Error from server (BadRequest): container "security-profiles-operator" in pod "spod-62mws" is waiting to start: PodInitializing
       

            wenshen@redhat.com Vincent Shen
            xiyuan@redhat.com Xiaojie Yuan
            Xiaojie Yuan Xiaojie Yuan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: