Details
-
Bug
-
Resolution: Done
-
Normal
-
None
-
None
-
None
Description
job link
glancing through the log snippet from e2e log I see a few things that stand out. The container "router" did not start in the
"router-haproxy-cfgmgr" pod. I see an "error: etcdserver: request timed out":
[BeforeEach] [sig-network][Feature:Router] github.com/openshift/origin/test/extended/router/config_manager.go:46 Apr 8 17:55:06.392: INFO: Running 'oc --namespace=e2e-test-router-config-manager-sz2l9 --kubeconfig=/tmp/kubeconfig-3151948817 new-app -f /tmp/fixture-testdata-dir2334740690/test/extended/testdata/router/router-config-manager.yaml -p IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f396be70b93743a0417e764fc8e42b10d14a58a7a51cb4d3ee4be91e7f24a02c' Apr 8 17:55:17.495: INFO: Error running /usr/bin/oc --namespace=e2e-test-router-config-manager-sz2l9 --kubeconfig=/tmp/kubeconfig-3151948817 new-app -f /tmp/fixture-testdata-dir2334740690/test/extended/testdata/router/router-config-manager.yaml -p IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f396be70b93743a0417e764fc8e42b10d14a58a7a51cb4d3ee4be91e7f24a02c: StdOut> W0408 17:55:06.457336 85234 shim_kubectl.go:58] Using non-groupfied API resources is deprecated and will be removed in a future release, update apiVersion to "template.openshift.io/v1" for your resource --> Deploying template "e2e-test-router-config-manager-sz2l9/" for "/tmp/fixture-testdata-dir2334740690/test/extended/testdata/router/router-config-manager.yaml" to project e2e-test-router-config-manager-sz2l9 * With parameters: * IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f396be70b93743a0417e764fc8e42b10d14a58a7a51cb4d3ee4be91e7f24a02c --> Creating resources ... error: etcdserver: request timed out rolebinding.authorization.openshift.io "system-router" created route.route.openshift.io "edge-blueprint" created route.route.openshift.io "reencrypt-blueprint" created route.route.openshift.io "passthrough-blueprint" created configmap "serving-cert" created pod "insecure-endpoint" created pod "secure-endpoint" created service "insecure-service" created service "secure-service" created route.route.openshift.io "insecure-route" created route.route.openshift.io "edge-allow-http-route" created route.route.openshift.io "reencrypt-route" created route.route.openshift.io "passthrough-route" created --> Failed StdErr> W0408 17:55:06.457336 85234 shim_kubectl.go:58] Using non-groupfied API resources is deprecated and will be removed in a future release, update apiVersion to "template.openshift.io/v1" for your resource --> Deploying template "e2e-test-router-config-manager-sz2l9/" for "/tmp/fixture-testdata-dir2334740690/test/extended/testdata/router/router-config-manager.yaml" to project e2e-test-router-config-manager-sz2l9 * With parameters: * IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f396be70b93743a0417e764fc8e42b10d14a58a7a51cb4d3ee4be91e7f24a02c --> Creating resources ... error: etcdserver: request timed out rolebinding.authorization.openshift.io "system-router" created route.route.openshift.io "edge-blueprint" created route.route.openshift.io "reencrypt-blueprint" created route.route.openshift.io "passthrough-blueprint" created configmap "serving-cert" created pod "insecure-endpoint" created pod "secure-endpoint" created service "insecure-service" created service "secure-service" created route.route.openshift.io "insecure-route" created route.route.openshift.io "edge-allow-http-route" created route.route.openshift.io "reencrypt-route" created route.route.openshift.io "passthrough-route" created --> Failed W0408 17:55:06.457336 85234 shim_kubectl.go:58] Using non-groupfied API resources is deprecated and will be removed in a future release, update apiVersion to "template.openshift.io/v1" for your resource --> Deploying template "e2e-test-router-config-manager-sz2l9/" for "/tmp/fixture-testdata-dir2334740690/test/extended/testdata/router/router-config-manager.yaml" to project e2e-test-router-config-manager-sz2l9 * With parameters: * IMAGE=quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f396be70b93743a0417e764fc8e42b10d14a58a7a51cb4d3ee4be91e7f24a02c --> Creating resources ... error: etcdserver: request timed out rolebinding.authorization.openshift.io "system-router" created route.route.openshift.io "edge-blueprint" created route.route.openshift.io "reencrypt-blueprint" created route.route.openshift.io "passthrough-blueprint" created configmap "serving-cert" created pod "insecure-endpoint" created pod "secure-endpoint" created service "insecure-service" created service "secure-service" created route.route.openshift.io "insecure-route" created route.route.openshift.io "edge-allow-http-route" created route.route.openshift.io "reencrypt-route" created route.route.openshift.io "passthrough-route" created --> Failed [AfterEach] [sig-network][Feature:Router] github.com/openshift/origin/test/extended/router/config_manager.go:34 Apr 8 17:55:17.525: INFO: Routes: NAME ROUTER HOST LAST TRANSITION edge-allow-http-route default edge.allow.hapcm.test 2022-04-08 17:55:17 +0000 UTC edge-blueprint default edge.blueprint.hapcm.test 2022-04-08 17:55:15 +0000 UTC insecure-route default insecure.hapcm.test 2022-04-08 17:55:17 +0000 UTC passthrough-blueprint default passthrough.blueprint.hapcm.test 2022-04-08 17:55:16 +0000 UTC passthrough-route default passthrough.hapcm.test 2022-04-08 17:55:17 +0000 UTC reencrypt-blueprint default reencrypt.blueprint.hapcm.test 2022-04-08 17:55:16 +0000 UTC reencrypt-route default reencrypt.hapcm.test 2022-04-08 17:55:17 +0000 UTC Apr 8 17:55:17.553: INFO: Running 'oc --namespace=e2e-test-router-config-manager-sz2l9 --kubeconfig=/tmp/kubeconfig-3151948817 describe pod/router-haproxy-cfgmgr -n e2e-test-router-config-manager-sz2l9' Apr 8 17:55:17.761: INFO: Describing pod "router-haproxy-cfgmgr" Name: router-haproxy-cfgmgr Namespace: e2e-test-router-config-manager-sz2l9 Priority: 0 Node: ip-10-0-157-243.ec2.internal/10.0.157.243 Start Time: Fri, 08 Apr 2022 17:55:06 +0000 Labels: test=router-haproxy-cfgmgr Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.129.3.87/23"],"mac_address":"0a:58:0a:81:03:57","gateway_ips":["10.129.2.1"],"ip_address":"10.129.3.87/23"... openshift.io/generated-by: OpenShiftNewApp openshift.io/scc: privileged Status: Pending IP: IPs: <none> Containers: router: Container ID: Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f396be70b93743a0417e764fc8e42b10d14a58a7a51cb4d3ee4be91e7f24a02c Image ID: Ports: 80/TCP, 443/TCP, 1936/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Args: --namespace=$(POD_NAMESPACE) -v=4 --haproxy-config-manager=true --blueprint-route-labels=select=hapcm-blueprint --labels=select=haproxy-cfgmgr --stats-password=password --stats-port=1936 --stats-user=admin State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: POD_NAMESPACE: e2e-test-router-config-manager-sz2l9 (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6n9zq (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-6n9zq: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 11s default-scheduler Successfully assigned e2e-test-router-config-manager-sz2l9/router-haproxy-cfgmgr to ip-10-0-157-243.ec2.internal Warning ErrorAddingLogicalPort 11s controlplane failed to ensurePod e2e-test-router-config-manager-sz2l9/router-haproxy-cfgmgr since it is not yet scheduled Apr 8 17:55:17.761: INFO: Running 'oc --namespace=e2e-test-router-config-manager-sz2l9 --kubeconfig=/tmp/kubeconfig-3151948817 logs pod/router-haproxy-cfgmgr -c router -n e2e-test-router-config-manager-sz2l9' Apr 8 17:55:18.020: INFO: Error running /usr/bin/oc --namespace=e2e-test-router-config-manager-sz2l9 --kubeconfig=/tmp/kubeconfig-3151948817 logs pod/router-haproxy-cfgmgr -c router -n e2e-test-router-config-manager-sz2l9: StdOut> Error from server (BadRequest): container "router" in pod "router-haproxy-cfgmgr" is waiting to start: ContainerCreating StdErr> Error from server (BadRequest): container "router" in pod "router-haproxy-cfgmgr" is waiting to start: ContainerCreating Apr 8 17:55:18.020: INFO: Error retrieving logs for pod "router-haproxy-cfgmgr"/"router": exit status 1 [AfterEach] [sig-network][Feature:Router] github.com/openshift/origin/test/extended/util/client.go:138 STEP: Collecting events from namespace "e2e-test-router-config-manager-sz2l9". STEP: Found 6 events. Apr 8 17:55:18.059: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for insecure-endpoint: { } Scheduled: Successfully assigned e2e-test-router-config-manager-sz2l9/insecure-endpoint to ip-10-0-220-60.ec2.internal Apr 8 17:55:18.059: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for router-haproxy-cfgmgr: { } Scheduled: Successfully assigned e2e-test-router-config-manager-sz2l9/router-haproxy-cfgmgr to ip-10-0-157-243.ec2.internal Apr 8 17:55:18.059: INFO: At 0001-01-01 00:00:00 +0000 UTC - event for secure-endpoint: { } Scheduled: Successfully assigned e2e-test-router-config-manager-sz2l9/secure-endpoint to ip-10-0-157-243.ec2.internal Apr 8 17:55:18.059: INFO: At 2022-04-08 17:55:06 +0000 UTC - event for router-haproxy-cfgmgr: {controlplane } ErrorAddingLogicalPort: failed to ensurePod e2e-test-router-config-manager-sz2l9/router-haproxy-cfgmgr since it is not yet scheduled Apr 8 17:55:18.059: INFO: At 2022-04-08 17:55:16 +0000 UTC - event for insecure-endpoint: {controlplane } ErrorAddingLogicalPort: failed to ensurePod e2e-test-router-config-manager-sz2l9/insecure-endpoint since it is not yet scheduled Apr 8 17:55:18.059: INFO: At 2022-04-08 17:55:17 +0000 UTC - event for secure-endpoint: {controlplane } ErrorAddingLogicalPort: failed to ensurePod e2e-test-router-config-manager-sz2l9/secure-endpoint since it is not yet scheduled Apr 8 17:55:18.111: INFO: POD NODE PHASE GRACE CONDITIONS Apr 8 17:55:18.112: INFO: insecure-endpoint ip-10-0-220-60.ec2.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:16 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:16 +0000 UTC ContainersNotReady containers with unready status: [test]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:16 +0000 UTC ContainersNotReady containers with unready status: [test]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:16 +0000 UTC }] Apr 8 17:55:18.112: INFO: router-haproxy-cfgmgr ip-10-0-157-243.ec2.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:06 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:06 +0000 UTC ContainersNotReady containers with unready status: [router]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:06 +0000 UTC ContainersNotReady containers with unready status: [router]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:06 +0000 UTC }] Apr 8 17:55:18.112: INFO: secure-endpoint ip-10-0-157-243.ec2.internal Pending [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:17 +0000 UTC } {Ready False 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:17 +0000 UTC ContainersNotReady containers with unready status: [serve]} {ContainersReady False 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:17 +0000 UTC ContainersNotReady containers with unready status: [serve]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2022-04-08 17:55:17 +0000 UTC }] Apr 8 17:55:18.112: INFO: Apr 8 17:55:18.247: INFO: Unable to fetch e2e-test-router-config-manager-sz2l9/insecure-endpoint/test logs: the server rejected our request for an unknown reason (get pods insecure-endpoint) Apr 8 17:55:18.276: INFO: Unable to fetch e2e-test-router-config-manager-sz2l9/router-haproxy-cfgmgr/router logs: the server rejected our request for an unknown reason (get pods router-haproxy-cfgmgr) Apr 8 17:55:18.312: INFO: Unable to fetch e2e-test-router-config-manager-sz2l9/secure-endpoint/serve logs: the server rejected our request for an unknown reason (get pods secure-endpoint) Apr 8 17:55:18.358: INFO: skipping dumping cluster info - cluster too large Apr 8 17:55:18.410: INFO: Deleted {user.openshift.io/v1, Resource=users e2e-test-router-config-manager-sz2l9-user}, err: <nil> Apr 8 17:55:18.440: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthclients e2e-client-e2e-test-router-config-manager-sz2l9}, err: <nil> Apr 8 17:55:18.466: INFO: Deleted {oauth.openshift.io/v1, Resource=oauthaccesstokens sha256~a-aU7B0_VE0AMFTp3vtWjDJHMNHcrszgTXNZKCGIJdw}, err: <nil> [AfterEach] [sig-network][Feature:Router] github.com/openshift/origin/test/extended/util/client.go:139 STEP: Destroying namespace "e2e-test-router-config-manager-sz2l9" for this suite. fail [github.com/openshift/origin/test/extended/router/config_manager.go:53]: Unexpected error: <*exec.ExitError | 0xc001e1e220>: { ProcessState: { pid: 85234, status: 256, rusage: { Utime: {Sec: 0, Usec: 159334}, Stime: {Sec: 0, Usec: 57528}, Maxrss: 162832, Ixrss: 0, Idrss: 0, Isrss: 0, Minflt: 3643, Majflt: 0, Nswap: 0, Inblock: 0, Oublock: 0, Msgsnd: 0, Msgrcv: 0, Nsignals: 0, Nvcsw: 1178, Nivcsw: 10, }, }, Stderr: nil, } exit status 1 occurred failed: (14.7s) 2022-04-08T17:55:18 "[sig-network][Feature:Router] The HAProxy router should serve the correct routes when running with the haproxy config manager [Suite:openshift/conformance/parallel]"
link to this job's testgrid for reference.