-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.18
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
Hypershift Sprint 261, Hypershift Sprint 262, Hypershift Sprint 263
-
3
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
The issue was found when testing the fix for https://issues.redhat.com/browse/OCPBUGS-38709
Version-Release number of selected component (if applicable):
How reproducible:
always
Steps to Reproduce:
1. Create a cluster with proxy, no port is set in the proxy 2. 3.
Actual results:
ingress-operator degraded due to the issue with konnectivity-proxy container
Expected results:
ingress-operator is ready
Additional info:
jiezhao-mac:hypershift jiezhao$ oc get hc/jie-test -n clusters NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE jie-test jie-test-admin-kubeconfig Partial True False The hosted control plane is available - lastTransitionTime: "2024-10-02T19:39:12Z" message: 'Unable to apply 4.18.0-0.nightly-2024-10-02-081004: Cluster operator ingress is degraded' observedGeneration: 3 reason: ClusterOperatorDegraded status: "True" type: ClusterVersionProgressing jiezhao-mac:hypershift jiezhao$ oc get hc/jie-test -n clusters -ojsonpath='{.spec.configuration}' | jq { "proxy": { "httpProxy": "http://10.0.130.219", "httpsProxy": "http://10.0.130.219", "trustedCA": { "name": "" } } } jiezhao-mac:hypershift jiezhao$ oc get pods -n clusters-jie-test NAME READY STATUS RESTARTS AGE aws-cloud-controller-manager-6749ff6789-2ctbs 2/2 Running 0 14m aws-ebs-csi-driver-controller-fc49b56f9-jzrxp 12/12 Running 0 13m aws-ebs-csi-driver-operator-76b4f5cf49-cs624 1/1 Running 0 13m capi-provider-55799fdf64-l5pjt 2/2 Running 0 15m catalog-operator-75885d5cb-2tspl 2/2 Running 2 (13m ago) 14m certified-operators-catalog-8598d6469d-xrqsx 1/1 Running 0 13m cloud-credential-operator-587697f789-94g8l 1/1 Running 0 14m cloud-network-config-controller-67f8854f67-5mmjd 3/3 Running 0 13m cluster-api-d8976d868-wprms 1/1 Running 0 15m cluster-image-registry-operator-df9898775-mjgjf 3/3 Running 0 14m cluster-network-operator-86554d9899-fmm2l 2/2 Running 0 14m cluster-node-tuning-operator-5645ccf976-k4dj7 1/1 Running 0 14m cluster-policy-controller-57bf56c64f-2jwln 1/1 Running 0 14m cluster-storage-operator-bdc8d848b-l6rqs 1/1 Running 0 14m cluster-version-operator-868b8b795c-vpmtd 1/1 Running 0 14m community-operators-catalog-5444c9554b-ddc8r 1/1 Running 0 13m control-plane-operator-8594c978c5-ztkpb 2/2 Running 0 15m control-plane-pki-operator-c97cf55f8-ppg6z 1/1 Running 0 15m csi-snapshot-controller-56cfb7869b-rgcnv 1/1 Running 0 13m csi-snapshot-controller-operator-6456dc95ff-n8vbp 1/1 Running 0 14m csi-snapshot-webhook-78ff7945fc-rhlvf 1/1 Running 0 13m dns-operator-79f97559cc-x7zkq 1/1 Running 0 14m etcd-0 3/3 Running 0 15m hosted-cluster-config-operator-78f9956c55-f6prl 1/1 Running 0 14m ignition-server-7565b8664-bkv8p 1/1 Running 0 13m ignition-server-proxy-6bbb98dd9f-kxvpf 1/1 Running 0 13m ingress-operator-6f455f6d96-nhv8n 3/3 Running 0 14m konnectivity-agent-6646b7bd9-kqksm 1/1 Running 0 14m kube-apiserver-55577987-dwbsd 5/5 Running 0 14m kube-controller-manager-667944d444-8sgg9 1/1 Running 0 6m59s kube-scheduler-6cfffc95df-9g5gl 1/1 Running 0 14m machine-approver-5ff59547b6-zmp9q 1/1 Running 0 14m multus-admission-controller-649c85c6c-r45fd 2/2 Running 0 12m network-node-identity-7cf9bbcd4d-v5d9g 3/3 Running 0 12m oauth-openshift-696bb5f667-2m4gp 4/4 Running 0 13m olm-operator-7796cb468b-xr5m8 2/2 Running 0 14m openshift-apiserver-8cc964fd6-xmz8x 3/3 Running 0 6m59s openshift-controller-manager-786ffc677-42ztb 1/1 Running 0 14m openshift-oauth-apiserver-8559696b55-8zw78 2/2 Running 0 14m openshift-route-controller-manager-6ff85499b4-5vm42 1/1 Running 0 14m ovnkube-control-plane-55b797567-dqgb9 3/3 Running 0 12m packageserver-6bdd89b6b7-kbjwc 2/2 Running 0 14m redhat-marketplace-catalog-697ddbcc8c-tf4ws 1/1 Running 0 13m redhat-operators-catalog-77bbd44d99-nwb9m 1/1 Running 0 13m router-7f575556c7-mdvnx 1/1 Running 0 14m jiezhao-mac:hypershift jiezhao$ jiezhao-mac:hypershift jiezhao$ ./bin/hypershift create kubeconfig > hostedcluster.kubeconfig jiezhao-mac:hypershift jiezhao$ export KUBECONFIG=hostedcluster.kubeconfig jiezhao-mac:hypershift jiezhao$ oc config use-context clusters-jie-test Switched to context "clusters-jie-test". jiezhao-mac:hypershift jiezhao$ jiezhao-mac:hypershift jiezhao$ oc get nodes NAME STATUS ROLES AGE VERSION ip-10-0-136-216.us-east-2.compute.internal Ready worker 10m v1.30.3 ip-10-0-137-243.us-east-2.compute.internal Ready worker 10m v1.30.3 jiezhao-mac:hypershift jiezhao$ jiezhao-mac:hypershift jiezhao$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE console 4.18.0-0.nightly-2024-10-02-081004 True False False 8m9s csi-snapshot-controller 4.18.0-0.nightly-2024-10-02-081004 True False False 15m dns 4.18.0-0.nightly-2024-10-02-081004 True False False 8m22s image-registry 4.18.0-0.nightly-2024-10-02-081004 True False False 8m27s ingress 4.18.0-0.nightly-2024-10-02-081004 True False True 8m6s The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:... insights 4.18.0-0.nightly-2024-10-02-081004 True False False 9m2s kube-apiserver 4.18.0-0.nightly-2024-10-02-081004 True False False 15m kube-controller-manager 4.18.0-0.nightly-2024-10-02-081004 True False False 15m kube-scheduler 4.18.0-0.nightly-2024-10-02-081004 True False False 15m kube-storage-version-migrator 4.18.0-0.nightly-2024-10-02-081004 True False False 8m59s monitoring 4.18.0-0.nightly-2024-10-02-081004 True False False 6m33s network 4.18.0-0.nightly-2024-10-02-081004 True False False 15m node-tuning 4.18.0-0.nightly-2024-10-02-081004 True True False 9m56s Waiting for 2/2 Profiles to be applied openshift-apiserver 4.18.0-0.nightly-2024-10-02-081004 True False False 15m openshift-controller-manager 4.18.0-0.nightly-2024-10-02-081004 True False False 15m openshift-samples 4.18.0-0.nightly-2024-10-02-081004 True False False 8m17s operator-lifecycle-manager 4.18.0-0.nightly-2024-10-02-081004 True False False 15m operator-lifecycle-manager-catalog 4.18.0-0.nightly-2024-10-02-081004 True False False 15m operator-lifecycle-manager-packageserver 4.18.0-0.nightly-2024-10-02-081004 True False False 15m service-ca 4.18.0-0.nightly-2024-10-02-081004 True False False 8m59s storage 4.18.0-0.nightly-2024-10-02-081004 True False False 10m jiezhao-mac:hypershift jiezhao$ jiezhao-mac:hypershift jiezhao$ oc get pods -n kube-system NAME READY STATUS RESTARTS AGE konnectivity-agent-m9c9x 1/1 Running 0 11m konnectivity-agent-qk8g6 1/1 Running 0 11m kube-apiserver-proxy-ip-10-0-136-216.us-east-2.compute.internal 1/1 Running 0 11m kube-apiserver-proxy-ip-10-0-137-243.us-east-2.compute.internal 1/1 Running 0 11m jiezhao-mac:hypershift jiezhao$ jiezhao-mac:hypershift jiezhao$ oc get pods/kube-apiserver-proxy-ip-10-0-136-216.us-east-2.compute.internal -n kube-system -o yaml apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/config.hash: 2d1ad2ba5129d58e0c1c4ffcc21212a6 kubernetes.io/config.mirror: 2d1ad2ba5129d58e0c1c4ffcc21212a6 kubernetes.io/config.seen: "2024-10-02T19:44:20.077131462Z" kubernetes.io/config.source: file creationTimestamp: "2024-10-02T19:44:20Z" labels: k8s-app: kube-apiserver-proxy name: kube-apiserver-proxy-ip-10-0-136-216.us-east-2.compute.internal namespace: kube-system ownerReferences: - apiVersion: v1 controller: true kind: Node name: ip-10-0-136-216.us-east-2.compute.internal uid: 20492180-ba1b-443f-b9a4-8f703df040d6 resourceVersion: "6601" uid: d1e9aba9-539f-4187-bec6-7247e3144978 spec: containers: - command: - control-plane-operator - kubernetes-default-proxy - --listen-addr=172.20.0.1:6443 - --proxy-addr=10.0.130.219:80 - --apiserver-addr=api-jie-test.jiezhao-hypershift.qe.devcluster.openshift.com:443 image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7742cb977182e5cb3672b37852ec23e5d1bcc7b0b436b03aab8fefad3bb40f35 imagePullPolicy: IfNotPresent name: kubernetes-default-proxy resources: requests: cpu: 13m memory: 16Mi securityContext: runAsUser: 1001 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst enableServiceLinks: true hostNetwork: true nodeName: ip-10-0-136-216.us-east-2.compute.internal preemptionPolicy: PreemptLowerPriority priority: 2000001000 priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute operator: Exists - effect: NoSchedule key: node.kubernetes.io/memory-pressure operator: Exists status: conditions: - lastProbeTime: null lastTransitionTime: "2024-10-02T19:44:50Z" status: "True" type: PodReadyToStartContainers - lastProbeTime: null lastTransitionTime: "2024-10-02T19:44:20Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2024-10-02T19:45:12Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2024-10-02T19:45:12Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2024-10-02T19:44:20Z" status: "True" type: PodScheduled containerStatuses: - containerID: cri-o://7234d33a80a77c77a4e72fb5fad3eb13bfe5512a22a52d655f31a4d5d41d3b48 image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7742cb977182e5cb3672b37852ec23e5d1bcc7b0b436b03aab8fefad3bb40f35 imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7742cb977182e5cb3672b37852ec23e5d1bcc7b0b436b03aab8fefad3bb40f35 lastState: {} name: kubernetes-default-proxy ready: true restartCount: 0 started: true state: running: startedAt: "2024-10-02T19:45:11Z" hostIP: 10.0.136.216 hostIPs: - ip: 10.0.136.216 phase: Running podIP: 10.0.136.216 podIPs: - ip: 10.0.136.216 qosClass: Burstable startTime: "2024-10-02T19:44:20Z" jiezhao-mac:hypershift jiezhao$ jiezhao-mac:hypershift jiezhao$ export KUBECONFIG=kubeconfig jiezhao-mac:hypershift jiezhao$ jiezhao-mac:hypershift jiezhao$ oc get pods -n clusters-jie-test | grep ingress ingress-operator-6f455f6d96-nhv8n 3/3 Running 0 18m jiezhao-mac:hypershift jiezhao$ jiezhao-mac:hypershift jiezhao$ oc logs ingress-operator-6f455f6d96-nhv8n -n clusters-jie-test Defaulted container "ingress-operator" out of: ingress-operator, konnectivity-proxy, token-minter, availability-prober (init) 2024-10-02T19:56:44.956Z ERROR operator.ingress_controller controller/controller.go:114 got retryable error; requeueing {"after": "1m0s", "error": "IngressController is degraded: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:\nerror sending canary HTTP request to \"canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com\": Get \"https://canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com\": Bad Gateway (x11 over 10m0s))"} jiezhao-mac:hypershift jiezhao$ oc logs ingress-operator-6f455f6d96-nhv8n -c konnectivity-proxy -n clusters-jie-test {"level":"info","ts":"2024-10-02T19:39:33Z","logger":"konnectivity-https-proxy","msg":"Starting proxy","version":"openshift/hypershift: 2d754dae4b6375bbb6e7bfb772580927f2e32fac. Latest supported OCP: 4.18.0"} Error: failed to split proxy URL host port (10.0.130.219): address 10.0.130.219: missing port in address2024/10/02 19:39:47 [001] INFO: Running 0 CONNECT handlers 2024/10/02 19:39:47 [001] INFO: Accepting CONNECT to sts.us-east-2.amazonaws.com:443 2024/10/02 19:39:47 [002] INFO: Running 0 CONNECT handlers 2024/10/02 19:39:47 [002] INFO: Accepting CONNECT to route53.amazonaws.com:443 2024/10/02 19:39:47 [003] INFO: Running 0 CONNECT handlers 2024/10/02 19:39:47 [003] INFO: Accepting CONNECT to elasticloadbalancing.us-east-2.amazonaws.com:443 2024/10/02 19:39:47 [004] INFO: Running 0 CONNECT handlers 2024/10/02 19:39:47 [004] INFO: Accepting CONNECT to tagging.us-east-1.amazonaws.com:443 2024/10/02 19:46:44 [005] INFO: Running 0 CONNECT handlers 2024/10/02 19:46:44 [005] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF 2024/10/02 19:47:44 [006] INFO: Running 0 CONNECT handlers 2024/10/02 19:47:44 [006] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF 2024/10/02 19:48:44 [007] INFO: Running 0 CONNECT handlers 2024/10/02 19:48:44 [007] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF 2024/10/02 19:49:44 [008] INFO: Running 0 CONNECT handlers 2024/10/02 19:49:44 [008] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF