Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-1882

runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524

    XMLWordPrintable

Details

    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      network operator stuck in PROGRESSING due to 'DaemonSet "/openshift-network-diagnostics/network-check-target" update is rolling out (2 out of 5 updated)'.
      
      [2022-09-22T16:16:16.880Z] network 4.12.0-0.nightly-arm64-2022-09-22-050711 True True False 5h1m DaemonSet "/openshift-network-diagnostics/network-check-target" update is rolling out (2 out of 5 updated)
      
      The error of the DaemonSet "/openshift-network-diagnostics/network-check-target is as below:
      
      Error: container create failed: time="2022-09-22T14:15:36Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"

      Version-Release number of selected component (if applicable):

      Upgrade 4.12.0-0.nightly-arm64-2022-09-22-014351 ->  4.12.0-0.nightly-arm64-2022-09-22-050711

      How reproducible:

      I met this the first time in 4.12 upgrade ci watching. 

      Steps to Reproduce:

      1. Install cluster with profile 02_aarch64_UPI_on_Baremetal-packet_OVN-dual-stack_Disk-encryption_Disk-mirroring_Etcd-encryption
      2. Upgrade to the latest nightly build
      

      Actual results:

      During upgrade network operator stuck in PROGRESSING status

      Expected results:

      Network operator roll out successsfully

      Additional info:

      Logs: http://virt-openshift-05.lab.eng.nay.redhat.com/buildcorp/ocp_upgrade/22839.html 

      must-gather: http://file.nay.redhat.com/~qili/OCPBUGS-708/must-gather-140827-909463746.tar.gz 

       

      Get events from openshift-network-diagnostics namespace, I saw the error in network-check-target pods: "Error: container create failed: time="2022-09-22T14:15:36Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524""

       

      [2022-09-22T16:16:23.705Z] oc get event -n openshift-network-diagnostics
      [2022-09-22T16:16:24.005Z] LAST SEEN   TYPE      REASON                       OBJECT                                                    MESSAGE
      [2022-09-22T16:16:24.005Z] 169m        Normal    Killing                      pod/network-check-source-545dd5f569-bj9j5                 Stopping container check-endpoints
      [2022-09-22T16:16:24.005Z] 163m        Normal    Scheduled                    pod/network-check-source-545dd5f569-tdhrg                 Successfully assigned openshift-network-diagnostics/network-check-source-545dd5f569-tdhrg to worker-00.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
      [2022-09-22T16:16:24.005Z] 163m        Normal    AddedInterface               pod/network-check-source-545dd5f569-tdhrg                 Add eth0 [fd01:0:0:4::102/64] from ovn-kubernetes
      [2022-09-22T16:16:24.005Z] 163m        Normal    Started                      pod/network-check-source-545dd5f569-tdhrg                 Started container check-endpoints
      [2022-09-22T16:16:24.005Z] 122m        Normal    Killing                      pod/network-check-source-545dd5f569-tdhrg                 Stopping container check-endpoints
      [2022-09-22T16:16:24.005Z] 169m        Normal    Scheduled                    pod/network-check-source-545dd5f569-zst8h                 Successfully assigned openshift-network-diagnostics/network-check-source-545dd5f569-zst8h to worker-01.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
      [2022-09-22T16:16:24.005Z] 169m        Normal    AddedInterface               pod/network-check-source-545dd5f569-zst8h                 Add eth0 [fd01:0:0:5::e8/64] from ovn-kubernetes
      [2022-09-22T16:16:24.005Z] 163m        Normal    Killing                      pod/network-check-source-545dd5f569-zst8h                 Stopping container check-endpoints
      [2022-09-22T16:16:24.005Z] 169m        Normal    SuccessfulCreate             replicaset/network-check-source-545dd5f569                Created pod: network-check-source-545dd5f569-zst8h
      [2022-09-22T16:16:24.005Z] 163m        Normal    SuccessfulCreate             replicaset/network-check-source-545dd5f569                Created pod: network-check-source-545dd5f569-tdhrg
      [2022-09-22T16:16:24.005Z] 122m        Normal    SuccessfulDelete             replicaset/network-check-source-545dd5f569                Deleted pod: network-check-source-545dd5f569-tdhrg
      [2022-09-22T16:16:24.005Z] 122m        Normal    Scheduled                    pod/network-check-source-7cf7c7db57-jtjkf                 Successfully assigned openshift-network-diagnostics/network-check-source-7cf7c7db57-jtjkf to worker-01.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
      [2022-09-22T16:16:24.005Z] 122m        Normal    AddedInterface               pod/network-check-source-7cf7c7db57-jtjkf                 Add eth0 [fd01:0:0:5::10c/64] from ovn-kubernetes
      [2022-09-22T16:16:24.005Z] 122m        Normal    Pulling                      pod/network-check-source-7cf7c7db57-jtjkf                 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01"
      [2022-09-22T16:16:24.005Z] 122m        Normal    Pulled                       pod/network-check-source-7cf7c7db57-jtjkf                 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01" in 3.594069675s
      [2022-09-22T16:16:24.005Z] 122m        Normal    Created                      pod/network-check-source-7cf7c7db57-jtjkf                 Created container check-endpoints
      [2022-09-22T16:16:24.005Z] 122m        Normal    Started                      pod/network-check-source-7cf7c7db57-jtjkf                 Started container check-endpoints
      [2022-09-22T16:16:24.005Z] 122m        Normal    SuccessfulCreate             replicaset/network-check-source-7cf7c7db57                Created pod: network-check-source-7cf7c7db57-jtjkf
      [2022-09-22T16:16:24.005Z] 122m        Normal    ScalingReplicaSet            deployment/network-check-source                           Scaled down replica set network-check-source-545dd5f569 to 0
      [2022-09-22T16:16:24.005Z] 122m        Normal    ScalingReplicaSet            deployment/network-check-source                           Scaled up replica set network-check-source-7cf7c7db57 to 1
      [2022-09-22T16:16:24.005Z] 167m        Warning   NodeNotReady                 pod/network-check-target-5xbgk                            Node is not ready
      [2022-09-22T16:16:24.005Z] 163m        Normal    AddedInterface               pod/network-check-target-5xbgk                            Add eth0 [fd01:0:0:4::4/64] from ovn-kubernetes
      [2022-09-22T16:16:24.005Z] 163m        Normal    Pulled                       pod/network-check-target-5xbgk                            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d0b9b488bd6c952361d98cba096560d095c036557f4f7872b0ee4fb6ed4afb7" already present on machine
      [2022-09-22T16:16:24.005Z] 163m        Normal    Created                      pod/network-check-target-5xbgk                            Created container network-check-target-container
      [2022-09-22T16:16:24.005Z] 163m        Normal    Started                      pod/network-check-target-5xbgk                            Started container network-check-target-container
      [2022-09-22T16:16:24.005Z] 121m        Normal    Killing                      pod/network-check-target-5xbgk                            Stopping container network-check-target-container
      [2022-09-22T16:16:24.005Z] 121m        Warning   ProbeError                   pod/network-check-target-5xbgk                            Readiness probe error: Get "http://[fd01:0:0:4::4]:8080/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...
      [2022-09-22T16:16:24.005Z] 121m        Warning   Unhealthy                    pod/network-check-target-5xbgk                            Readiness probe failed: Get "http://[fd01:0:0:4::4]:8080/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
      [2022-09-22T16:16:24.005Z] 122m        Normal    Scheduled                    pod/network-check-target-g6fk4                            Successfully assigned openshift-network-diagnostics/network-check-target-g6fk4 to master-02.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
      [2022-09-22T16:16:24.005Z] 122m        Normal    AddedInterface               pod/network-check-target-g6fk4                            Add eth0 [fd01:0:0:2::8f/64] from ovn-kubernetes
      [2022-09-22T16:16:24.005Z] 122m        Normal    Pulling                      pod/network-check-target-g6fk4                            Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01"
      [2022-09-22T16:16:24.005Z] 122m        Normal    Pulled                       pod/network-check-target-g6fk4                            Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01" in 3.673534833s
      [2022-09-22T16:16:24.005Z] 122m        Normal    Created                      pod/network-check-target-g6fk4                            Created container network-check-target-container
      [2022-09-22T16:16:24.005Z] 122m        Normal    Started                      pod/network-check-target-g6fk4                            Started container network-check-target-container
      [2022-09-22T16:16:24.005Z] 161m        Warning   NodeNotReady                 pod/network-check-target-lqmh4                            Node is not ready
      [2022-09-22T16:16:24.005Z] 157m        Normal    AddedInterface               pod/network-check-target-lqmh4                            Add eth0 [fd01:0:0:5::4/64] from ovn-kubernetes
      [2022-09-22T16:16:24.005Z] 157m        Normal    Pulled                       pod/network-check-target-lqmh4                            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d0b9b488bd6c952361d98cba096560d095c036557f4f7872b0ee4fb6ed4afb7" already present on machine
      [2022-09-22T16:16:24.005Z] 157m        Normal    Created                      pod/network-check-target-lqmh4                            Created container network-check-target-container
      [2022-09-22T16:16:24.005Z] 157m        Normal    Started                      pod/network-check-target-lqmh4                            Started container network-check-target-container
      [2022-09-22T16:16:24.005Z] 121m        Normal    Scheduled                    pod/network-check-target-npb5p                            Successfully assigned openshift-network-diagnostics/network-check-target-npb5p to worker-00.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
      [2022-09-22T16:16:24.005Z] 121m        Normal    AddedInterface               pod/network-check-target-npb5p                            Add eth0 [fd01:0:0:4::12b/64] from ovn-kubernetes
      [2022-09-22T16:16:24.005Z] 121m        Normal    Pulling                      pod/network-check-target-npb5p                            Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01"
      [2022-09-22T16:16:24.005Z] 121m        Normal    Pulled                       pod/network-check-target-npb5p                            Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01" in 5.656398579s
      [2022-09-22T16:16:24.005Z] 121m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:14:54Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.005Z] 90s         Normal    Pulled                       pod/network-check-target-npb5p                            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01" already present on machine
      [2022-09-22T16:16:24.005Z] 121m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:14:56Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.006Z] 121m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:15:10Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.006Z] 121m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:15:22Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.006Z] 120m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:15:36Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.006Z] 120m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:15:51Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.006Z] 120m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:16:05Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.006Z] 120m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:16:19Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.006Z] 119m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:16:34Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.006Z] 16m         Warning   Failed                       pod/network-check-target-npb5p                            (combined from similar events): Error: container create failed: time="2022-09-22T15:59:47Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      [2022-09-22T16:16:24.006Z] 122m        Normal    Killing                      pod/network-check-target-sxbjt                            Stopping container network-check-target-container
      [2022-09-22T16:16:24.006Z] 122m        Normal    SuccessfulDelete             daemonset/network-check-target                            Deleted pod: network-check-target-sxbjt
      [2022-09-22T16:16:24.006Z] 122m        Normal    SuccessfulCreate             daemonset/network-check-target                            Created pod: network-check-target-g6fk4
      [2022-09-22T16:16:24.006Z] 121m        Normal    SuccessfulDelete             daemonset/network-check-target                            Deleted pod: network-check-target-5xbgk
      [2022-09-22T16:16:24.006Z] 121m        Normal    SuccessfulCreate             daemonset/network-check-target                            Created pod: network-check-target-npb5p
      [2022-09-22T16:16:24.006Z] 163m        Warning   FastControllerResync         node/worker-00.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsTimeToStart" resync interval is set to 0s which might lead to client request throttling
      [2022-09-22T16:16:24.006Z] 163m        Warning   FastControllerResync         node/worker-00.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsStop" resync interval is set to 0s which might lead to client request throttling
      [2022-09-22T16:16:24.006Z] 156m        Warning   ConnectivityOutageDetected   node/worker-00.newugd-22839.qe.devcluster.openshift.com   Connectivity outage detected: network-check-target-worker-01: failed to establish a TCP connection to [fd01:0:0:5::4]:8080: dial tcp [fd01:0:0:5::4]:8080: i/o timeout
      [2022-09-22T16:16:24.006Z] 156m        Normal    ConnectivityRestored         node/worker-00.newugd-22839.qe.devcluster.openshift.com   Connectivity restored after 3m59.99984513s: network-check-target-worker-01: tcp connection to [fd01:0:0:5::4]:8080 succeeded
      [2022-09-22T16:16:24.006Z] 169m        Warning   FastControllerResync         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsTimeToStart" resync interval is set to 0s which might lead to client request throttling
      [2022-09-22T16:16:24.006Z] 169m        Warning   FastControllerResync         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsStop" resync interval is set to 0s which might lead to client request throttling
      [2022-09-22T16:16:24.006Z] 122m        Warning   FastControllerResync         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsTimeToStart" resync interval is set to 0s which might lead to client request throttling
      [2022-09-22T16:16:24.006Z] 122m        Warning   FastControllerResync         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsStop" resync interval is set to 0s which might lead to client request throttling
      [2022-09-22T16:16:24.006Z] 82m         Normal    ConnectivityRestored         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Connectivity restored after 2h15m23.190386736s: openshift-apiserver-endpoint-master-00: tcp connection to [fd01:0:0:1::62]:8443 succeeded 

      From must-gather event-filter.html:

      Time
       
      Namespace
       
      Component
       
      RelatedObject
       
      Reason
       
      Message
       
      14:14:55 openshift-network-diagnostics kubelet network-check-target-npb5p Failed Error: container create failed: time="2022-09-22T14:14:54Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      14:14:57 openshift-network-diagnostics kubelet network-check-target-npb5p Failed Error: container create failed: time="2022-09-22T14:14:56Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      14:15:10 openshift-network-diagnostics kubelet network-check-target-npb5p Failed Error: container create failed: time="2022-09-22T14:15:10Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      14:15:23 openshift-network-diagnostics kubelet network-check-target-npb5p Failed Error: container create failed: time="2022-09-22T14:15:22Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      14:15:37 openshift-network-diagnostics kubelet network-check-target-npb5p Failed Error: container create failed: time="2022-09-22T14:15:36Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      14:15:52 openshift-network-diagnostics kubelet network-check-target-npb5p Failed Error: container create failed: time="2022-09-22T14:15:51Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      14:16:05 openshift-network-diagnostics kubelet network-check-target-npb5p Failed Error: container create failed: time="2022-09-22T14:16:05Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      14:16:19 openshift-network-diagnostics kubelet network-check-target-npb5p Failed Error: container create failed: time="2022-09-22T14:16:19Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      14:16:34 openshift-network-diagnostics kubelet network-check-target-npb5p Failed Error: container create failed: time="2022-09-22T14:16:34Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
      15:59:47 (x421) openshift-network-diagnostics kubelet network-check-target-npb5p Failed (combined from similar events): Error: container create failed: time="2022-09-22T15:59:47Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"

       

       

      Attachments

        Issue Links

          Activity

            People

              kolyshkin Kirill Kolyshkin
              rhn-support-qili Qiujie Li
              Sunil Choudhary Sunil Choudhary
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: