Loading...

Type: Bug
Resolution: Duplicate
Priority: Normal
Fix Version/s: None
Affects Version/s: 4.12
Component/s: Multi-Arch / ARM, Node / Kubelet
Labels:

Regression:
None
Blocked:
False
Blocked Reason:

Hide

None

Show
None

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Description of problem:

network operator stuck in PROGRESSING due to 'DaemonSet "/openshift-network-diagnostics/network-check-target" update is rolling out (2 out of 5 updated)'.

[2022-09-22T16:16:16.880Z] network 4.12.0-0.nightly-arm64-2022-09-22-050711 True True False 5h1m DaemonSet "/openshift-network-diagnostics/network-check-target" update is rolling out (2 out of 5 updated)

The error of the DaemonSet "/openshift-network-diagnostics/network-check-target is as below:

Error: container create failed: time="2022-09-22T14:15:36Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"

Version-Release number of selected component (if applicable):

Upgrade 4.12.0-0.nightly-arm64-2022-09-22-014351 ->  4.12.0-0.nightly-arm64-2022-09-22-050711

How reproducible:

I met this the first time in 4.12 upgrade ci watching.

Steps to Reproduce:

1. Install cluster with profile 02_aarch64_UPI_on_Baremetal-packet_OVN-dual-stack_Disk-encryption_Disk-mirroring_Etcd-encryption
2. Upgrade to the latest nightly build

Actual results:

During upgrade network operator stuck in PROGRESSING status

Expected results:

Network operator roll out successsfully

Additional info:

Logs: http://virt-openshift-05.lab.eng.nay.redhat.com/buildcorp/ocp_upgrade/22839.html

must-gather: http://file.nay.redhat.com/~qili/OCPBUGS-708/must-gather-140827-909463746.tar.gz

Get events from openshift-network-diagnostics namespace, I saw the error in network-check-target pods: "Error: container create failed: time="2022-09-22T14:15:36Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524""

[2022-09-22T16:16:23.705Z] oc get event -n openshift-network-diagnostics
[2022-09-22T16:16:24.005Z] LAST SEEN   TYPE      REASON                       OBJECT                                                    MESSAGE
[2022-09-22T16:16:24.005Z] 169m        Normal    Killing                      pod/network-check-source-545dd5f569-bj9j5                 Stopping container check-endpoints
[2022-09-22T16:16:24.005Z] 163m        Normal    Scheduled                    pod/network-check-source-545dd5f569-tdhrg                 Successfully assigned openshift-network-diagnostics/network-check-source-545dd5f569-tdhrg to worker-00.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
[2022-09-22T16:16:24.005Z] 163m        Normal    AddedInterface               pod/network-check-source-545dd5f569-tdhrg                 Add eth0 [fd01:0:0:4::102/64] from ovn-kubernetes
[2022-09-22T16:16:24.005Z] 163m        Normal    Started                      pod/network-check-source-545dd5f569-tdhrg                 Started container check-endpoints
[2022-09-22T16:16:24.005Z] 122m        Normal    Killing                      pod/network-check-source-545dd5f569-tdhrg                 Stopping container check-endpoints
[2022-09-22T16:16:24.005Z] 169m        Normal    Scheduled                    pod/network-check-source-545dd5f569-zst8h                 Successfully assigned openshift-network-diagnostics/network-check-source-545dd5f569-zst8h to worker-01.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
[2022-09-22T16:16:24.005Z] 169m        Normal    AddedInterface               pod/network-check-source-545dd5f569-zst8h                 Add eth0 [fd01:0:0:5::e8/64] from ovn-kubernetes
[2022-09-22T16:16:24.005Z] 163m        Normal    Killing                      pod/network-check-source-545dd5f569-zst8h                 Stopping container check-endpoints
[2022-09-22T16:16:24.005Z] 169m        Normal    SuccessfulCreate             replicaset/network-check-source-545dd5f569                Created pod: network-check-source-545dd5f569-zst8h
[2022-09-22T16:16:24.005Z] 163m        Normal    SuccessfulCreate             replicaset/network-check-source-545dd5f569                Created pod: network-check-source-545dd5f569-tdhrg
[2022-09-22T16:16:24.005Z] 122m        Normal    SuccessfulDelete             replicaset/network-check-source-545dd5f569                Deleted pod: network-check-source-545dd5f569-tdhrg
[2022-09-22T16:16:24.005Z] 122m        Normal    Scheduled                    pod/network-check-source-7cf7c7db57-jtjkf                 Successfully assigned openshift-network-diagnostics/network-check-source-7cf7c7db57-jtjkf to worker-01.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
[2022-09-22T16:16:24.005Z] 122m        Normal    AddedInterface               pod/network-check-source-7cf7c7db57-jtjkf                 Add eth0 [fd01:0:0:5::10c/64] from ovn-kubernetes
[2022-09-22T16:16:24.005Z] 122m        Normal    Pulling                      pod/network-check-source-7cf7c7db57-jtjkf                 Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01"
[2022-09-22T16:16:24.005Z] 122m        Normal    Pulled                       pod/network-check-source-7cf7c7db57-jtjkf                 Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01" in 3.594069675s
[2022-09-22T16:16:24.005Z] 122m        Normal    Created                      pod/network-check-source-7cf7c7db57-jtjkf                 Created container check-endpoints
[2022-09-22T16:16:24.005Z] 122m        Normal    Started                      pod/network-check-source-7cf7c7db57-jtjkf                 Started container check-endpoints
[2022-09-22T16:16:24.005Z] 122m        Normal    SuccessfulCreate             replicaset/network-check-source-7cf7c7db57                Created pod: network-check-source-7cf7c7db57-jtjkf
[2022-09-22T16:16:24.005Z] 122m        Normal    ScalingReplicaSet            deployment/network-check-source                           Scaled down replica set network-check-source-545dd5f569 to 0
[2022-09-22T16:16:24.005Z] 122m        Normal    ScalingReplicaSet            deployment/network-check-source                           Scaled up replica set network-check-source-7cf7c7db57 to 1
[2022-09-22T16:16:24.005Z] 167m        Warning   NodeNotReady                 pod/network-check-target-5xbgk                            Node is not ready
[2022-09-22T16:16:24.005Z] 163m        Normal    AddedInterface               pod/network-check-target-5xbgk                            Add eth0 [fd01:0:0:4::4/64] from ovn-kubernetes
[2022-09-22T16:16:24.005Z] 163m        Normal    Pulled                       pod/network-check-target-5xbgk                            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d0b9b488bd6c952361d98cba096560d095c036557f4f7872b0ee4fb6ed4afb7" already present on machine
[2022-09-22T16:16:24.005Z] 163m        Normal    Created                      pod/network-check-target-5xbgk                            Created container network-check-target-container
[2022-09-22T16:16:24.005Z] 163m        Normal    Started                      pod/network-check-target-5xbgk                            Started container network-check-target-container
[2022-09-22T16:16:24.005Z] 121m        Normal    Killing                      pod/network-check-target-5xbgk                            Stopping container network-check-target-container
[2022-09-22T16:16:24.005Z] 121m        Warning   ProbeError                   pod/network-check-target-5xbgk                            Readiness probe error: Get "http://[fd01:0:0:4::4]:8080/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)...
[2022-09-22T16:16:24.005Z] 121m        Warning   Unhealthy                    pod/network-check-target-5xbgk                            Readiness probe failed: Get "http://[fd01:0:0:4::4]:8080/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[2022-09-22T16:16:24.005Z] 122m        Normal    Scheduled                    pod/network-check-target-g6fk4                            Successfully assigned openshift-network-diagnostics/network-check-target-g6fk4 to master-02.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
[2022-09-22T16:16:24.005Z] 122m        Normal    AddedInterface               pod/network-check-target-g6fk4                            Add eth0 [fd01:0:0:2::8f/64] from ovn-kubernetes
[2022-09-22T16:16:24.005Z] 122m        Normal    Pulling                      pod/network-check-target-g6fk4                            Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01"
[2022-09-22T16:16:24.005Z] 122m        Normal    Pulled                       pod/network-check-target-g6fk4                            Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01" in 3.673534833s
[2022-09-22T16:16:24.005Z] 122m        Normal    Created                      pod/network-check-target-g6fk4                            Created container network-check-target-container
[2022-09-22T16:16:24.005Z] 122m        Normal    Started                      pod/network-check-target-g6fk4                            Started container network-check-target-container
[2022-09-22T16:16:24.005Z] 161m        Warning   NodeNotReady                 pod/network-check-target-lqmh4                            Node is not ready
[2022-09-22T16:16:24.005Z] 157m        Normal    AddedInterface               pod/network-check-target-lqmh4                            Add eth0 [fd01:0:0:5::4/64] from ovn-kubernetes
[2022-09-22T16:16:24.005Z] 157m        Normal    Pulled                       pod/network-check-target-lqmh4                            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d0b9b488bd6c952361d98cba096560d095c036557f4f7872b0ee4fb6ed4afb7" already present on machine
[2022-09-22T16:16:24.005Z] 157m        Normal    Created                      pod/network-check-target-lqmh4                            Created container network-check-target-container
[2022-09-22T16:16:24.005Z] 157m        Normal    Started                      pod/network-check-target-lqmh4                            Started container network-check-target-container
[2022-09-22T16:16:24.005Z] 121m        Normal    Scheduled                    pod/network-check-target-npb5p                            Successfully assigned openshift-network-diagnostics/network-check-target-npb5p to worker-00.newugd-22839.qe.devcluster.openshift.com by master-01.newugd-22839.qe.devcluster.openshift.com
[2022-09-22T16:16:24.005Z] 121m        Normal    AddedInterface               pod/network-check-target-npb5p                            Add eth0 [fd01:0:0:4::12b/64] from ovn-kubernetes
[2022-09-22T16:16:24.005Z] 121m        Normal    Pulling                      pod/network-check-target-npb5p                            Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01"
[2022-09-22T16:16:24.005Z] 121m        Normal    Pulled                       pod/network-check-target-npb5p                            Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01" in 5.656398579s
[2022-09-22T16:16:24.005Z] 121m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:14:54Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.005Z] 90s         Normal    Pulled                       pod/network-check-target-npb5p                            Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db1fd2344ca1fc168080ec16d4540c7a1fe5b6f68c7e5e27f9da9f16c16acc01" already present on machine
[2022-09-22T16:16:24.005Z] 121m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:14:56Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.006Z] 121m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:15:10Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.006Z] 121m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:15:22Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.006Z] 120m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:15:36Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.006Z] 120m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:15:51Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.006Z] 120m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:16:05Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.006Z] 120m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:16:19Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.006Z] 119m        Warning   Failed                       pod/network-check-target-npb5p                            Error: container create failed: time="2022-09-22T14:16:34Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.006Z] 16m         Warning   Failed                       pod/network-check-target-npb5p                            (combined from similar events): Error: container create failed: time="2022-09-22T15:59:47Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
[2022-09-22T16:16:24.006Z] 122m        Normal    Killing                      pod/network-check-target-sxbjt                            Stopping container network-check-target-container
[2022-09-22T16:16:24.006Z] 122m        Normal    SuccessfulDelete             daemonset/network-check-target                            Deleted pod: network-check-target-sxbjt
[2022-09-22T16:16:24.006Z] 122m        Normal    SuccessfulCreate             daemonset/network-check-target                            Created pod: network-check-target-g6fk4
[2022-09-22T16:16:24.006Z] 121m        Normal    SuccessfulDelete             daemonset/network-check-target                            Deleted pod: network-check-target-5xbgk
[2022-09-22T16:16:24.006Z] 121m        Normal    SuccessfulCreate             daemonset/network-check-target                            Created pod: network-check-target-npb5p
[2022-09-22T16:16:24.006Z] 163m        Warning   FastControllerResync         node/worker-00.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsTimeToStart" resync interval is set to 0s which might lead to client request throttling
[2022-09-22T16:16:24.006Z] 163m        Warning   FastControllerResync         node/worker-00.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsStop" resync interval is set to 0s which might lead to client request throttling
[2022-09-22T16:16:24.006Z] 156m        Warning   ConnectivityOutageDetected   node/worker-00.newugd-22839.qe.devcluster.openshift.com   Connectivity outage detected: network-check-target-worker-01: failed to establish a TCP connection to [fd01:0:0:5::4]:8080: dial tcp [fd01:0:0:5::4]:8080: i/o timeout
[2022-09-22T16:16:24.006Z] 156m        Normal    ConnectivityRestored         node/worker-00.newugd-22839.qe.devcluster.openshift.com   Connectivity restored after 3m59.99984513s: network-check-target-worker-01: tcp connection to [fd01:0:0:5::4]:8080 succeeded
[2022-09-22T16:16:24.006Z] 169m        Warning   FastControllerResync         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsTimeToStart" resync interval is set to 0s which might lead to client request throttling
[2022-09-22T16:16:24.006Z] 169m        Warning   FastControllerResync         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsStop" resync interval is set to 0s which might lead to client request throttling
[2022-09-22T16:16:24.006Z] 122m        Warning   FastControllerResync         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsTimeToStart" resync interval is set to 0s which might lead to client request throttling
[2022-09-22T16:16:24.006Z] 122m        Warning   FastControllerResync         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Controller "CheckEndpointsStop" resync interval is set to 0s which might lead to client request throttling
[2022-09-22T16:16:24.006Z] 82m         Normal    ConnectivityRestored         node/worker-01.newugd-22839.qe.devcluster.openshift.com   Connectivity restored after 2h15m23.190386736s: openshift-apiserver-endpoint-master-00: tcp connection to [fd01:0:0:1::62]:8443 succeeded

From must-gather event-filter.html:

Time	Namespace	Component	RelatedObject	Reason	Message
14:14:55	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	Error: container create failed: time="2022-09-22T14:14:54Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
14:14:57	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	Error: container create failed: time="2022-09-22T14:14:56Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
14:15:10	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	Error: container create failed: time="2022-09-22T14:15:10Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
14:15:23	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	Error: container create failed: time="2022-09-22T14:15:22Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
14:15:37	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	Error: container create failed: time="2022-09-22T14:15:36Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
14:15:52	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	Error: container create failed: time="2022-09-22T14:15:51Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
14:16:05	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	Error: container create failed: time="2022-09-22T14:16:05Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
14:16:19	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	Error: container create failed: time="2022-09-22T14:16:19Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
14:16:34	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	Error: container create failed: time="2022-09-22T14:16:34Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"
15:59:47 (x421)	openshift-network-diagnostics	kubelet	network-check-target-npb5p	Failed	(combined from similar events): Error: container create failed: time="2022-09-22T15:59:47Z" level=error msg="runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524"

duplicates

OCPBUGS-2637 [ARM64][4.11.0+] Containers are stuck in CreateError with 'error loading seccomp filter: errno 524'

Closed

is related to

OCPBUGS-708 UpdatingKubeStateMetricsFailed before Upgrade

Closed

OCPBUGS-2302 4.11 upgrade to 4.12, prometheus-operator-admission-webhook pod is failed to start up due to "error loading seccomp filter into kernel: error loading seccomp filter: errno 524"

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates