-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.18
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
Hypershift Sprint 261, Hypershift Sprint 262, Hypershift Sprint 263
-
3
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
The issue was found when testing the fix for https://issues.redhat.com/browse/OCPBUGS-38709
Version-Release number of selected component (if applicable):
How reproducible:
always
Steps to Reproduce:
1. Create a cluster with proxy, no port is set in the proxy
2.
3.
Actual results:
ingress-operator degraded due to the issue with konnectivity-proxy container
Expected results:
ingress-operator is ready
Additional info:
jiezhao-mac:hypershift jiezhao$ oc get hc/jie-test -n clusters
NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE
jie-test jie-test-admin-kubeconfig Partial True False The hosted control plane is available
- lastTransitionTime: "2024-10-02T19:39:12Z"
message: 'Unable to apply 4.18.0-0.nightly-2024-10-02-081004: Cluster operator
ingress is degraded'
observedGeneration: 3
reason: ClusterOperatorDegraded
status: "True"
type: ClusterVersionProgressing
jiezhao-mac:hypershift jiezhao$ oc get hc/jie-test -n clusters -ojsonpath='{.spec.configuration}' | jq
{
"proxy": {
"httpProxy": "http://10.0.130.219",
"httpsProxy": "http://10.0.130.219",
"trustedCA": {
"name": ""
}
}
}
jiezhao-mac:hypershift jiezhao$ oc get pods -n clusters-jie-test
NAME READY STATUS RESTARTS AGE
aws-cloud-controller-manager-6749ff6789-2ctbs 2/2 Running 0 14m
aws-ebs-csi-driver-controller-fc49b56f9-jzrxp 12/12 Running 0 13m
aws-ebs-csi-driver-operator-76b4f5cf49-cs624 1/1 Running 0 13m
capi-provider-55799fdf64-l5pjt 2/2 Running 0 15m
catalog-operator-75885d5cb-2tspl 2/2 Running 2 (13m ago) 14m
certified-operators-catalog-8598d6469d-xrqsx 1/1 Running 0 13m
cloud-credential-operator-587697f789-94g8l 1/1 Running 0 14m
cloud-network-config-controller-67f8854f67-5mmjd 3/3 Running 0 13m
cluster-api-d8976d868-wprms 1/1 Running 0 15m
cluster-image-registry-operator-df9898775-mjgjf 3/3 Running 0 14m
cluster-network-operator-86554d9899-fmm2l 2/2 Running 0 14m
cluster-node-tuning-operator-5645ccf976-k4dj7 1/1 Running 0 14m
cluster-policy-controller-57bf56c64f-2jwln 1/1 Running 0 14m
cluster-storage-operator-bdc8d848b-l6rqs 1/1 Running 0 14m
cluster-version-operator-868b8b795c-vpmtd 1/1 Running 0 14m
community-operators-catalog-5444c9554b-ddc8r 1/1 Running 0 13m
control-plane-operator-8594c978c5-ztkpb 2/2 Running 0 15m
control-plane-pki-operator-c97cf55f8-ppg6z 1/1 Running 0 15m
csi-snapshot-controller-56cfb7869b-rgcnv 1/1 Running 0 13m
csi-snapshot-controller-operator-6456dc95ff-n8vbp 1/1 Running 0 14m
csi-snapshot-webhook-78ff7945fc-rhlvf 1/1 Running 0 13m
dns-operator-79f97559cc-x7zkq 1/1 Running 0 14m
etcd-0 3/3 Running 0 15m
hosted-cluster-config-operator-78f9956c55-f6prl 1/1 Running 0 14m
ignition-server-7565b8664-bkv8p 1/1 Running 0 13m
ignition-server-proxy-6bbb98dd9f-kxvpf 1/1 Running 0 13m
ingress-operator-6f455f6d96-nhv8n 3/3 Running 0 14m
konnectivity-agent-6646b7bd9-kqksm 1/1 Running 0 14m
kube-apiserver-55577987-dwbsd 5/5 Running 0 14m
kube-controller-manager-667944d444-8sgg9 1/1 Running 0 6m59s
kube-scheduler-6cfffc95df-9g5gl 1/1 Running 0 14m
machine-approver-5ff59547b6-zmp9q 1/1 Running 0 14m
multus-admission-controller-649c85c6c-r45fd 2/2 Running 0 12m
network-node-identity-7cf9bbcd4d-v5d9g 3/3 Running 0 12m
oauth-openshift-696bb5f667-2m4gp 4/4 Running 0 13m
olm-operator-7796cb468b-xr5m8 2/2 Running 0 14m
openshift-apiserver-8cc964fd6-xmz8x 3/3 Running 0 6m59s
openshift-controller-manager-786ffc677-42ztb 1/1 Running 0 14m
openshift-oauth-apiserver-8559696b55-8zw78 2/2 Running 0 14m
openshift-route-controller-manager-6ff85499b4-5vm42 1/1 Running 0 14m
ovnkube-control-plane-55b797567-dqgb9 3/3 Running 0 12m
packageserver-6bdd89b6b7-kbjwc 2/2 Running 0 14m
redhat-marketplace-catalog-697ddbcc8c-tf4ws 1/1 Running 0 13m
redhat-operators-catalog-77bbd44d99-nwb9m 1/1 Running 0 13m
router-7f575556c7-mdvnx 1/1 Running 0 14m
jiezhao-mac:hypershift jiezhao$
jiezhao-mac:hypershift jiezhao$ ./bin/hypershift create kubeconfig > hostedcluster.kubeconfig
jiezhao-mac:hypershift jiezhao$ export KUBECONFIG=hostedcluster.kubeconfig
jiezhao-mac:hypershift jiezhao$ oc config use-context clusters-jie-test
Switched to context "clusters-jie-test".
jiezhao-mac:hypershift jiezhao$
jiezhao-mac:hypershift jiezhao$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-136-216.us-east-2.compute.internal Ready worker 10m v1.30.3
ip-10-0-137-243.us-east-2.compute.internal Ready worker 10m v1.30.3
jiezhao-mac:hypershift jiezhao$
jiezhao-mac:hypershift jiezhao$ oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
console 4.18.0-0.nightly-2024-10-02-081004 True False False 8m9s
csi-snapshot-controller 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
dns 4.18.0-0.nightly-2024-10-02-081004 True False False 8m22s
image-registry 4.18.0-0.nightly-2024-10-02-081004 True False False 8m27s
ingress 4.18.0-0.nightly-2024-10-02-081004 True False True 8m6s The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:...
insights 4.18.0-0.nightly-2024-10-02-081004 True False False 9m2s
kube-apiserver 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
kube-controller-manager 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
kube-scheduler 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
kube-storage-version-migrator 4.18.0-0.nightly-2024-10-02-081004 True False False 8m59s
monitoring 4.18.0-0.nightly-2024-10-02-081004 True False False 6m33s
network 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
node-tuning 4.18.0-0.nightly-2024-10-02-081004 True True False 9m56s Waiting for 2/2 Profiles to be applied
openshift-apiserver 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
openshift-controller-manager 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
openshift-samples 4.18.0-0.nightly-2024-10-02-081004 True False False 8m17s
operator-lifecycle-manager 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
operator-lifecycle-manager-catalog 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
operator-lifecycle-manager-packageserver 4.18.0-0.nightly-2024-10-02-081004 True False False 15m
service-ca 4.18.0-0.nightly-2024-10-02-081004 True False False 8m59s
storage 4.18.0-0.nightly-2024-10-02-081004 True False False 10m
jiezhao-mac:hypershift jiezhao$
jiezhao-mac:hypershift jiezhao$ oc get pods -n kube-system
NAME READY STATUS RESTARTS AGE
konnectivity-agent-m9c9x 1/1 Running 0 11m
konnectivity-agent-qk8g6 1/1 Running 0 11m
kube-apiserver-proxy-ip-10-0-136-216.us-east-2.compute.internal 1/1 Running 0 11m
kube-apiserver-proxy-ip-10-0-137-243.us-east-2.compute.internal 1/1 Running 0 11m
jiezhao-mac:hypershift jiezhao$
jiezhao-mac:hypershift jiezhao$ oc get pods/kube-apiserver-proxy-ip-10-0-136-216.us-east-2.compute.internal -n kube-system -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/config.hash: 2d1ad2ba5129d58e0c1c4ffcc21212a6
kubernetes.io/config.mirror: 2d1ad2ba5129d58e0c1c4ffcc21212a6
kubernetes.io/config.seen: "2024-10-02T19:44:20.077131462Z"
kubernetes.io/config.source: file
creationTimestamp: "2024-10-02T19:44:20Z"
labels:
k8s-app: kube-apiserver-proxy
name: kube-apiserver-proxy-ip-10-0-136-216.us-east-2.compute.internal
namespace: kube-system
ownerReferences:
- apiVersion: v1
controller: true
kind: Node
name: ip-10-0-136-216.us-east-2.compute.internal
uid: 20492180-ba1b-443f-b9a4-8f703df040d6
resourceVersion: "6601"
uid: d1e9aba9-539f-4187-bec6-7247e3144978
spec:
containers:
- command:
- control-plane-operator
- kubernetes-default-proxy
- --listen-addr=172.20.0.1:6443
- --proxy-addr=10.0.130.219:80
- --apiserver-addr=api-jie-test.jiezhao-hypershift.qe.devcluster.openshift.com:443
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7742cb977182e5cb3672b37852ec23e5d1bcc7b0b436b03aab8fefad3bb40f35
imagePullPolicy: IfNotPresent
name: kubernetes-default-proxy
resources:
requests:
cpu: 13m
memory: 16Mi
securityContext:
runAsUser: 1001
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostNetwork: true
nodeName: ip-10-0-136-216.us-east-2.compute.internal
preemptionPolicy: PreemptLowerPriority
priority: 2000001000
priorityClassName: system-node-critical
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/memory-pressure
operator: Exists
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-10-02T19:44:50Z"
status: "True"
type: PodReadyToStartContainers
- lastProbeTime: null
lastTransitionTime: "2024-10-02T19:44:20Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-10-02T19:45:12Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-10-02T19:45:12Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-10-02T19:44:20Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: cri-o://7234d33a80a77c77a4e72fb5fad3eb13bfe5512a22a52d655f31a4d5d41d3b48
image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7742cb977182e5cb3672b37852ec23e5d1bcc7b0b436b03aab8fefad3bb40f35
imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7742cb977182e5cb3672b37852ec23e5d1bcc7b0b436b03aab8fefad3bb40f35
lastState: {}
name: kubernetes-default-proxy
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-10-02T19:45:11Z"
hostIP: 10.0.136.216
hostIPs:
- ip: 10.0.136.216
phase: Running
podIP: 10.0.136.216
podIPs:
- ip: 10.0.136.216
qosClass: Burstable
startTime: "2024-10-02T19:44:20Z"
jiezhao-mac:hypershift jiezhao$
jiezhao-mac:hypershift jiezhao$ export KUBECONFIG=kubeconfig
jiezhao-mac:hypershift jiezhao$
jiezhao-mac:hypershift jiezhao$ oc get pods -n clusters-jie-test | grep ingress
ingress-operator-6f455f6d96-nhv8n 3/3 Running 0 18m
jiezhao-mac:hypershift jiezhao$
jiezhao-mac:hypershift jiezhao$ oc logs ingress-operator-6f455f6d96-nhv8n -n clusters-jie-test
Defaulted container "ingress-operator" out of: ingress-operator, konnectivity-proxy, token-minter, availability-prober (init)
2024-10-02T19:56:44.956Z ERROR operator.ingress_controller controller/controller.go:114 got retryable error; requeueing {"after": "1m0s", "error": "IngressController is degraded: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:\nerror sending canary HTTP request to \"canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com\": Get \"https://canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com\": Bad Gateway (x11 over 10m0s))"}
jiezhao-mac:hypershift jiezhao$ oc logs ingress-operator-6f455f6d96-nhv8n -c konnectivity-proxy -n clusters-jie-test
{"level":"info","ts":"2024-10-02T19:39:33Z","logger":"konnectivity-https-proxy","msg":"Starting proxy","version":"openshift/hypershift: 2d754dae4b6375bbb6e7bfb772580927f2e32fac. Latest supported OCP: 4.18.0"}
Error: failed to split proxy URL host port (10.0.130.219): address 10.0.130.219: missing port in address2024/10/02 19:39:47 [001] INFO: Running 0 CONNECT handlers
2024/10/02 19:39:47 [001] INFO: Accepting CONNECT to sts.us-east-2.amazonaws.com:443
2024/10/02 19:39:47 [002] INFO: Running 0 CONNECT handlers
2024/10/02 19:39:47 [002] INFO: Accepting CONNECT to route53.amazonaws.com:443
2024/10/02 19:39:47 [003] INFO: Running 0 CONNECT handlers
2024/10/02 19:39:47 [003] INFO: Accepting CONNECT to elasticloadbalancing.us-east-2.amazonaws.com:443
2024/10/02 19:39:47 [004] INFO: Running 0 CONNECT handlers
2024/10/02 19:39:47 [004] INFO: Accepting CONNECT to tagging.us-east-1.amazonaws.com:443
2024/10/02 19:46:44 [005] INFO: Running 0 CONNECT handlers
2024/10/02 19:46:44 [005] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF
2024/10/02 19:47:44 [006] INFO: Running 0 CONNECT handlers
2024/10/02 19:47:44 [006] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF
2024/10/02 19:48:44 [007] INFO: Running 0 CONNECT handlers
2024/10/02 19:48:44 [007] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF
2024/10/02 19:49:44 [008] INFO: Running 0 CONNECT handlers
2024/10/02 19:49:44 [008] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF
