Gathering artifacts ... Gathering data for ns/openshift-config... Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+ Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice Gathering data for ns/openshift-config-managed... Gathering data for ns/openshift-authentication... Gathering data for ns/openshift-authentication-operator... Gathering data for ns/openshift-ingress... Gathering data for ns/openshift-oauth-apiserver... Gathering data for ns/openshift-machine-api... Gathering data for ns/openshift-cloud-controller-manager-operator... Gathering data for ns/openshift-cloud-controller-manager... Gathering data for ns/openshift-cloud-credential-operator... Gathering data for ns/openshift-config-operator... Gathering data for ns/openshift-monitoring... Gathering data for ns/openshift-network-console... Gathering data for ns/openshift-console-operator... Gathering data for ns/openshift-console... Gathering data for ns/openshift-cluster-storage-operator... Gathering data for ns/openshift-dns-operator... Gathering data for ns/openshift-dns... Gathering data for ns/openshift-etcd-operator... Gathering data for ns/openshift-etcd... Gathering data for ns/openshift-image-registry... Gathering data for ns/openshift-ingress-operator... Gathering data for ns/openshift-ingress-canary... Gathering data for ns/openshift-insights... Gathering data for ns/openshift-cluster-node-tuning-operator... Gathering data for ns/openshift-kube-apiserver-operator... Gathering data for ns/openshift-kube-apiserver... Gathering data for ns/openshift-catalogd... Gathering data for ns/openshift-multus... Gathering data for ns/openshift-kube-controller-manager... Gathering data for ns/openshift-kube-controller-manager-operator... Gathering data for ns/kube-system... Gathering data for ns/openshift-kube-scheduler... Gathering data for ns/openshift-kube-scheduler-operator... Gathering data for ns/openshift-kube-storage-version-migrator... Gathering data for ns/openshift-kube-storage-version-migrator-operator... Gathering data for ns/openshift-cluster-machine-approver... Gathering data for ns/openshift-machine-config-operator... Gathering data for ns/openshift-kni-infra... Gathering data for ns/openshift-openstack-infra... Gathering data for ns/openshift-ovirt-infra... Gathering data for ns/openshift-vsphere-infra... Gathering data for ns/openshift-nutanix-infra... Gathering data for ns/openshift-cloud-platform-infra... Gathering data for ns/openshift-marketplace... Gathering data for ns/openshift-user-workload-monitoring... Gathering data for ns/openshift-ovn-kubernetes... Gathering data for ns/openshift-host-network... Gathering data for ns/openshift-network-diagnostics... Gathering data for ns/openshift-network-node-identity... Gathering data for ns/openshift-network-operator... Gathering data for ns/openshift-cloud-network-config-controller... Gathering data for ns/openshift-operator-controller... Gathering data for ns/openshift-cluster-olm-operator... Gathering data for ns/openshift-apiserver-operator... Gathering data for ns/openshift-apiserver... Gathering data for ns/openshift-controller-manager-operator... Gathering data for ns/openshift-controller-manager... Gathering data for ns/openshift-route-controller-manager... Gathering data for ns/openshift-cluster-samples-operator... Gathering data for ns/openshift-operator-lifecycle-manager... Gathering data for ns/openshift-service-ca-operator... Gathering data for ns/openshift-service-ca... Gathering data for ns/openshift-cluster-csi-drivers... Wrote inspect data to /logs/artifacts/inspect. error: inspection completed with the errors occurred while gathering data: [skipping gathering secrets/support due to error: secrets "support" not found, skipping gathering endpoints/host-etcd-2 due to error: endpoints "host-etcd-2" not found] oc --insecure-skip-tls-verify --request-timeout=5s get apiserver.config.openshift.io,authentication.config.openshift.io,build.config.openshift.io,console.config.openshift.io,dns.config.openshift.io,featuregate.config.openshift.io,image.config.openshift.io,infrastructure.config.openshift.io,ingress.config.openshift.io,network.config.openshift.io,oauth.config.openshift.io,project.config.openshift.io,scheduler.config.openshift.io -o json oc --insecure-skip-tls-verify --request-timeout=5s get apiservices -o json oc --insecure-skip-tls-verify --request-timeout=5s get apiservices oc --insecure-skip-tls-verify --request-timeout=5s get clusteroperators -o json oc --insecure-skip-tls-verify --request-timeout=5s get clusteroperators oc --insecure-skip-tls-verify --request-timeout=5s get clusterversion -o json oc --insecure-skip-tls-verify --request-timeout=5s get clusterversion oc --insecure-skip-tls-verify --request-timeout=5s get configmaps --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get configmaps --all-namespaces oc --insecure-skip-tls-verify --request-timeout=5s get credentialsrequests --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get credentialsrequests --all-namespaces oc --insecure-skip-tls-verify --request-timeout=5s get csr -o json oc --insecure-skip-tls-verify --request-timeout=5s get endpoints --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get endpoints --all-namespaces oc --insecure-skip-tls-verify --request-timeout=5s get deployments --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get deployments --all-namespaces -o wide oc --insecure-skip-tls-verify --request-timeout=5s get daemonsets --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get daemonsets --all-namespaces -o wide oc --insecure-skip-tls-verify --request-timeout=5s get jobs.batch --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get events --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get events --all-namespaces oc --insecure-skip-tls-verify --request-timeout=5s get featuregate -o json oc --insecure-skip-tls-verify --request-timeout=5s get featuregate oc --insecure-skip-tls-verify --request-timeout=5s get kubeapiserver -o json Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice Unable to connect to the server: dial tcp: lookup api.ci-op-c652krw7-d1c68.cspilp.interop.ccitredhat.com on 172.30.0.10:53: no such host Unable to connect to the server: dial tcp: lookup api.ci-op-c652krw7-d1c68.cspilp.interop.ccitredhat.com on 172.30.0.10:53: no such host oc --insecure-skip-tls-verify --request-timeout=5s get kubeapiserver oc --insecure-skip-tls-verify --request-timeout=5s get kubecontrollermanager -o json oc --insecure-skip-tls-verify --request-timeout=5s get kubecontrollermanager Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice oc --insecure-skip-tls-verify --request-timeout=5s get machineconfigpools -o json oc --insecure-skip-tls-verify --request-timeout=5s get machineconfigpools oc --insecure-skip-tls-verify --request-timeout=5s get machineconfigs -o json oc --insecure-skip-tls-verify --request-timeout=5s get machineconfigs oc --insecure-skip-tls-verify --request-timeout=5s get controlplanemachinesets -A -o json oc --insecure-skip-tls-verify --request-timeout=5s get controlplanemachinesets -A oc --insecure-skip-tls-verify --request-timeout=5s get machinesets.machine.openshift.io -A -o json oc --insecure-skip-tls-verify --request-timeout=5s get machinesets.machine.openshift.io -A oc --insecure-skip-tls-verify --request-timeout=5s get machinesets.cluster.x-k8s.io -A -o json oc --insecure-skip-tls-verify --request-timeout=5s get machines.machine.openshift.io -A -o json oc --insecure-skip-tls-verify --request-timeout=5s get machines.machine.openshift.io -A -o wide oc --insecure-skip-tls-verify --request-timeout=5s get machines.cluster.x-k8s.io -A -o json oc --insecure-skip-tls-verify --request-timeout=5s get awsclusters.infrastructure.cluster.x-k8s.io -A -o json oc --insecure-skip-tls-verify --request-timeout=5s get awsmachines.infrastructure.cluster.x-k8s.io -A -o json oc --insecure-skip-tls-verify --request-timeout=5s get awsmachinetemplates.infrastructure.cluster.x-k8s.io -A -o json Unable to connect to the server: dial tcp: lookup api.ci-op-c652krw7-d1c68.cspilp.interop.ccitredhat.com on 172.30.0.10:53: no such host oc --insecure-skip-tls-verify --request-timeout=5s get namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get namespaces oc --insecure-skip-tls-verify --request-timeout=5s get nodes -o json oc --insecure-skip-tls-verify --request-timeout=5s get nodes -o wide oc --insecure-skip-tls-verify --request-timeout=5s get openshiftapiserver -o json oc --insecure-skip-tls-verify --request-timeout=5s get openshiftapiserver oc --insecure-skip-tls-verify --request-timeout=5s get pods --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get pods --all-namespaces -o wide oc --insecure-skip-tls-verify --request-timeout=5s get persistentvolumes --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get persistentvolumes --all-namespaces -o wide oc --insecure-skip-tls-verify --request-timeout=5s get persistentvolumeclaims --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get persistentvolumeclaims --all-namespaces -o wide oc --insecure-skip-tls-verify --request-timeout=5s get replicasets --all-namespaces -o json error: the server doesn't have a resource type "machinesets" oc --insecure-skip-tls-verify --request-timeout=5s get replicasets --all-namespaces -o wide oc --insecure-skip-tls-verify --request-timeout=5s get rolebindings --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get rolebindings --all-namespaces oc --insecure-skip-tls-verify --request-timeout=5s get roles --all-namespaces -o json error: the server doesn't have a resource type "awsmachines" oc --insecure-skip-tls-verify --request-timeout=5s get roles --all-namespaces oc --insecure-skip-tls-verify --request-timeout=5s get services --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get services --all-namespaces error: the server doesn't have a resource type "awsclusters" oc --insecure-skip-tls-verify --request-timeout=5s get statefulsets --all-namespaces -o json error: the server doesn't have a resource type "awsmachinetemplates" Unable to connect to the server: dial tcp: lookup api.ci-op-c652krw7-d1c68.cspilp.interop.ccitredhat.com on 172.30.0.10:53: no such host oc --insecure-skip-tls-verify --request-timeout=5s get statefulsets --all-namespaces error: the server doesn't have a resource type "machines" oc --insecure-skip-tls-verify --request-timeout=5s get routes --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get routes --all-namespaces oc --insecure-skip-tls-verify --request-timeout=5s get subscriptions --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get subscriptions --all-namespaces oc --insecure-skip-tls-verify --request-timeout=5s get clusterserviceversions --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get clusterserviceversions --all-namespaces oc --insecure-skip-tls-verify --request-timeout=5s adm release info -o json oc --insecure-skip-tls-verify --request-timeout=5s get clusterrolebindings --all-namespaces -o json oc --insecure-skip-tls-verify --request-timeout=5s get --raw /openapi/v2 oc --insecure-skip-tls-verify get --request-timeout=20s --raw /api/v1/nodes/ip-10-0-111-17.ec2.internal/proxy/debug/pprof/heap oc --insecure-skip-tls-verify adm node-logs ip-10-0-111-17.ec2.internal --unify=false oc --insecure-skip-tls-verify adm node-logs ip-10-0-111-17.ec2.internal --unify=false --path=audit/audit.log oc --insecure-skip-tls-verify get --request-timeout=20s --raw /api/v1/nodes/ip-10-0-126-23.ec2.internal/proxy/debug/pprof/heap oc --insecure-skip-tls-verify adm node-logs ip-10-0-126-23.ec2.internal --unify=false oc --insecure-skip-tls-verify adm node-logs ip-10-0-126-23.ec2.internal --unify=false --path=audit/audit.log oc --insecure-skip-tls-verify get --request-timeout=20s --raw /api/v1/nodes/ip-10-0-27-5.ec2.internal/proxy/debug/pprof/heap oc --insecure-skip-tls-verify adm node-logs ip-10-0-27-5.ec2.internal --unify=false oc --insecure-skip-tls-verify adm node-logs ip-10-0-27-5.ec2.internal --unify=false --path=audit/audit.log oc --insecure-skip-tls-verify get --request-timeout=20s --raw /api/v1/nodes/ip-10-0-30-70.ec2.internal/proxy/debug/pprof/heap oc --insecure-skip-tls-verify adm node-logs ip-10-0-30-70.ec2.internal --unify=false error: Get "https://api.ci-op-c652krw7-d1c68.cspilp.interop.ccitredhat.com:6443/api/v1/nodes/ip-10-0-111-17.ec2.internal": dial tcp: lookup api.ci-op-c652krw7-d1c68.cspilp.interop.ccitredhat.com on 172.30.0.10:53: no such host oc --insecure-skip-tls-verify adm node-logs ip-10-0-30-70.ec2.internal --unify=false --path=audit/audit.log oc --insecure-skip-tls-verify get --request-timeout=20s --raw /api/v1/nodes/ip-10-0-78-147.ec2.internal/proxy/debug/pprof/heap oc --insecure-skip-tls-verify adm node-logs ip-10-0-78-147.ec2.internal --unify=false oc --insecure-skip-tls-verify adm node-logs ip-10-0-78-147.ec2.internal --unify=false --path=audit/audit.log oc --insecure-skip-tls-verify get --request-timeout=20s --raw /api/v1/nodes/ip-10-0-88-145.ec2.internal/proxy/debug/pprof/heap oc --insecure-skip-tls-verify adm node-logs ip-10-0-88-145.ec2.internal --unify=false oc --insecure-skip-tls-verify adm node-logs ip-10-0-88-145.ec2.internal --unify=false --path=audit/audit.log INFO: gathering the audit logs for each master ip-10-0-111-17.ec2.internal audit.log ip-10-0-27-5.ec2.internal audit.log ip-10-0-88-145.ec2.internal audit.log INFO: Queueing download/gzip of openshift-apiserver/audit.log from ip-10-0-111-17.ec2.internal INFO: gziping to /logs/artifacts/audit_logs/openshift-apiserver/ip-10-0-111-17.ec2.internal-audit.log.gz oc --insecure-skip-tls-verify adm node-logs ip-10-0-111-17.ec2.internal --path=openshift-apiserver/audit.log INFO: Queueing download/gzip of openshift-apiserver/audit.log from ip-10-0-27-5.ec2.internal INFO: gziping to /logs/artifacts/audit_logs/openshift-apiserver/ip-10-0-27-5.ec2.internal-audit.log.gz oc --insecure-skip-tls-verify adm node-logs ip-10-0-27-5.ec2.internal --path=openshift-apiserver/audit.log INFO: Queueing download/gzip of openshift-apiserver/audit.log from ip-10-0-88-145.ec2.internal INFO: gziping to /logs/artifacts/audit_logs/openshift-apiserver/ip-10-0-88-145.ec2.internal-audit.log.gz oc --insecure-skip-tls-verify adm node-logs ip-10-0-88-145.ec2.internal --path=openshift-apiserver/audit.log error: Get "https://api.ci-op-c652krw7-d1c68.cspilp.interop.ccitredhat.com:6443/api/v1/nodes?labelSelector=node-role.kubernetes.io%2Fmaster": dial tcp: lookup api.ci-op-c652krw7-d1c68.cspilp.interop.ccitredhat.com on 172.30.0.10:53: no such host ip-10-0-111-17.ec2.internal audit.log ip-10-0-27-5.ec2.internal audit.log ip-10-0-88-145.ec2.internal audit.log INFO: Queueing download/gzip of oauth-apiserver/audit.log from ip-10-0-111-17.ec2.internal INFO: gziping to /logs/artifacts/audit_logs/oauth-apiserver/ip-10-0-111-17.ec2.internal-audit.log.gz oc --insecure-skip-tls-verify adm node-logs ip-10-0-111-17.ec2.internal --path=oauth-apiserver/audit.log INFO: Queueing download/gzip of oauth-apiserver/audit.log from ip-10-0-27-5.ec2.internal INFO: gziping to /logs/artifacts/audit_logs/oauth-apiserver/ip-10-0-27-5.ec2.internal-audit.log.gz oc --insecure-skip-tls-verify adm node-logs ip-10-0-27-5.ec2.internal --path=oauth-apiserver/audit.log INFO: Queueing download/gzip of oauth-apiserver/audit.log from ip-10-0-88-145.ec2.internal INFO: gziping to /logs/artifacts/audit_logs/oauth-apiserver/ip-10-0-88-145.ec2.internal-audit.log.gz oc --insecure-skip-tls-verify adm node-logs ip-10-0-88-145.ec2.internal --path=oauth-apiserver/audit.log /logs/artifacts/network/multus_logs / [must-gather ] OUT 2025-11-19T04:19:14.229425506Z Using must-gather plug-in image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:de004e9eb349ddd2a2051e0a2e17e30de637f7138c1aa8e78620123dbff54897 W1119 04:19:14.229534 1754 mustgather.go:425] volume percentage greater than or equal to 90 might cause filling up the disk space and have an impact on other components running on master When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 50ac7af9-c33a-4c1f-8d4e-f3fdd2621c54 ClientVersion: 4.21.0-202511121752.p2.g0c09391.assembly.stream.el9-0c09391 ClusterVersion: Stable at "4.21.0-0.nightly-2025-11-13-042845" ClusterOperators: All healthy and stable [must-gather ] OUT 2025-11-19T04:19:14.255681101Z namespace/openshift-must-gather-zmfrn created [must-gather ] OUT 2025-11-19T04:19:14.269509253Z clusterrolebinding.rbac.authorization.k8s.io/must-gather-ppdp5 created [must-gather ] OUT 2025-11-19T04:19:14.298343939Z pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:de004e9eb349ddd2a2051e0a2e17e30de637f7138c1aa8e78620123dbff54897 created [must-gather-qcm9v] POD 2025-11-19T04:19:15.171716128Z [disk usage checker] Started [must-gather-qcm9v] POD 2025-11-19T04:19:15.176890299Z [disk usage checker] Volume usage percentage: current = 19 ; allowed = 100 [must-gather-qcm9v] POD 2025-11-19T04:19:15.176913849Z bash: line 1: /usr/bin/gather_multus_logs: No such file or directory [must-gather-qcm9v] POD 2025-11-19T04:19:15.195671978Z Caches written to disk [must-gather-qcm9v] OUT 2025-11-19T04:19:24.313148973Z waiting for gather to complete [must-gather-qcm9v] OUT 2025-11-19T04:19:24.316231474Z downloading gather output [must-gather-qcm9v] OUT 2025-11-19T04:19:24.523472743Z receiving incremental file list [must-gather-qcm9v] OUT 2025-11-19T04:19:24.525459222Z ./ [must-gather-qcm9v] OUT 2025-11-19T04:19:24.526403691Z [must-gather-qcm9v] OUT 2025-11-19T04:19:24.526448262Z sent 27 bytes received 42 bytes 138.00 bytes/sec [must-gather-qcm9v] OUT 2025-11-19T04:19:24.526456912Z total size is 0 speedup is 0.00 [must-gather ] OUT 2025-11-19T04:19:24.553762529Z namespace/openshift-must-gather-zmfrn deleted Reprinting Cluster State: When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information: ClusterID: 50ac7af9-c33a-4c1f-8d4e-f3fdd2621c54 ClientVersion: 4.21.0-202511121752.p2.g0c09391.assembly.stream.el9-0c09391 ClusterVersion: Stable at "4.21.0-0.nightly-2025-11-13-042845" ClusterOperators: All healthy and stable / INFO: gathering tcpdump information if present error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found INFO: gathering conntrackdump information if present error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found error: the server could not find the requested resource 404 page not found INFO: Fetching debug info from etcd pods if present pod/etcd-ip-10-0-111-17.ec2.internal error: dial tcp: lookup api.ci-op-c652krw7-d1c68.cspilp.interop.ccitredhat.com on 172.30.0.10:53: no such host