Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-42994

konnectivity-proxy error in ingress-operator when no port is set in the proxy

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.18
    • HyperShift
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • Hypershift Sprint 261, Hypershift Sprint 262, Hypershift Sprint 263
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

          The issue was found when testing the fix for https://issues.redhat.com/browse/OCPBUGS-38709

      Version-Release number of selected component (if applicable):

          

      How reproducible:

          always

      Steps to Reproduce:

          1. Create a cluster with proxy, no port is set in the proxy
          2.
          3.
          

      Actual results:

          ingress-operator degraded due to the issue with konnectivity-proxy container

      Expected results:

          ingress-operator is ready

      Additional info:

       
      jiezhao-mac:hypershift jiezhao$ oc get hc/jie-test -n clusters
      NAME       VERSION   KUBECONFIG                  PROGRESS   AVAILABLE   PROGRESSING   MESSAGE
      jie-test             jie-test-admin-kubeconfig   Partial    True        False         The hosted control plane is available
        - lastTransitionTime: "2024-10-02T19:39:12Z"
          message: 'Unable to apply 4.18.0-0.nightly-2024-10-02-081004: Cluster operator
            ingress is degraded'
          observedGeneration: 3
          reason: ClusterOperatorDegraded
          status: "True"
          type: ClusterVersionProgressing
      jiezhao-mac:hypershift jiezhao$ oc get hc/jie-test -n clusters -ojsonpath='{.spec.configuration}' | jq
      {
        "proxy": {
          "httpProxy": "http://10.0.130.219",
          "httpsProxy": "http://10.0.130.219",
          "trustedCA": {
            "name": ""
          }
        }
      }
      jiezhao-mac:hypershift jiezhao$ oc get pods -n clusters-jie-test
      NAME                                                  READY   STATUS    RESTARTS      AGE
      aws-cloud-controller-manager-6749ff6789-2ctbs         2/2     Running   0             14m
      aws-ebs-csi-driver-controller-fc49b56f9-jzrxp         12/12   Running   0             13m
      aws-ebs-csi-driver-operator-76b4f5cf49-cs624          1/1     Running   0             13m
      capi-provider-55799fdf64-l5pjt                        2/2     Running   0             15m
      catalog-operator-75885d5cb-2tspl                      2/2     Running   2 (13m ago)   14m
      certified-operators-catalog-8598d6469d-xrqsx          1/1     Running   0             13m
      cloud-credential-operator-587697f789-94g8l            1/1     Running   0             14m
      cloud-network-config-controller-67f8854f67-5mmjd      3/3     Running   0             13m
      cluster-api-d8976d868-wprms                           1/1     Running   0             15m
      cluster-image-registry-operator-df9898775-mjgjf       3/3     Running   0             14m
      cluster-network-operator-86554d9899-fmm2l             2/2     Running   0             14m
      cluster-node-tuning-operator-5645ccf976-k4dj7         1/1     Running   0             14m
      cluster-policy-controller-57bf56c64f-2jwln            1/1     Running   0             14m
      cluster-storage-operator-bdc8d848b-l6rqs              1/1     Running   0             14m
      cluster-version-operator-868b8b795c-vpmtd             1/1     Running   0             14m
      community-operators-catalog-5444c9554b-ddc8r          1/1     Running   0             13m
      control-plane-operator-8594c978c5-ztkpb               2/2     Running   0             15m
      control-plane-pki-operator-c97cf55f8-ppg6z            1/1     Running   0             15m
      csi-snapshot-controller-56cfb7869b-rgcnv              1/1     Running   0             13m
      csi-snapshot-controller-operator-6456dc95ff-n8vbp     1/1     Running   0             14m
      csi-snapshot-webhook-78ff7945fc-rhlvf                 1/1     Running   0             13m
      dns-operator-79f97559cc-x7zkq                         1/1     Running   0             14m
      etcd-0                                                3/3     Running   0             15m
      hosted-cluster-config-operator-78f9956c55-f6prl       1/1     Running   0             14m
      ignition-server-7565b8664-bkv8p                       1/1     Running   0             13m
      ignition-server-proxy-6bbb98dd9f-kxvpf                1/1     Running   0             13m
      ingress-operator-6f455f6d96-nhv8n                     3/3     Running   0             14m
      konnectivity-agent-6646b7bd9-kqksm                    1/1     Running   0             14m
      kube-apiserver-55577987-dwbsd                         5/5     Running   0             14m
      kube-controller-manager-667944d444-8sgg9              1/1     Running   0             6m59s
      kube-scheduler-6cfffc95df-9g5gl                       1/1     Running   0             14m
      machine-approver-5ff59547b6-zmp9q                     1/1     Running   0             14m
      multus-admission-controller-649c85c6c-r45fd           2/2     Running   0             12m
      network-node-identity-7cf9bbcd4d-v5d9g                3/3     Running   0             12m
      oauth-openshift-696bb5f667-2m4gp                      4/4     Running   0             13m
      olm-operator-7796cb468b-xr5m8                         2/2     Running   0             14m
      openshift-apiserver-8cc964fd6-xmz8x                   3/3     Running   0             6m59s
      openshift-controller-manager-786ffc677-42ztb          1/1     Running   0             14m
      openshift-oauth-apiserver-8559696b55-8zw78            2/2     Running   0             14m
      openshift-route-controller-manager-6ff85499b4-5vm42   1/1     Running   0             14m
      ovnkube-control-plane-55b797567-dqgb9                 3/3     Running   0             12m
      packageserver-6bdd89b6b7-kbjwc                        2/2     Running   0             14m
      redhat-marketplace-catalog-697ddbcc8c-tf4ws           1/1     Running   0             13m
      redhat-operators-catalog-77bbd44d99-nwb9m             1/1     Running   0             13m
      router-7f575556c7-mdvnx                               1/1     Running   0             14m
      jiezhao-mac:hypershift jiezhao$ 
      jiezhao-mac:hypershift jiezhao$ ./bin/hypershift create kubeconfig > hostedcluster.kubeconfig
      jiezhao-mac:hypershift jiezhao$ export KUBECONFIG=hostedcluster.kubeconfig
      jiezhao-mac:hypershift jiezhao$ oc config use-context clusters-jie-test
      Switched to context "clusters-jie-test".
      jiezhao-mac:hypershift jiezhao$ 
      jiezhao-mac:hypershift jiezhao$ oc get nodes
      NAME                                         STATUS   ROLES    AGE   VERSION
      ip-10-0-136-216.us-east-2.compute.internal   Ready    worker   10m   v1.30.3
      ip-10-0-137-243.us-east-2.compute.internal   Ready    worker   10m   v1.30.3
      jiezhao-mac:hypershift jiezhao$ 
      jiezhao-mac:hypershift jiezhao$ oc get co
      NAME                                       VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      console                                    4.18.0-0.nightly-2024-10-02-081004   True        False         False      8m9s    
      csi-snapshot-controller                    4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      dns                                        4.18.0-0.nightly-2024-10-02-081004   True        False         False      8m22s   
      image-registry                             4.18.0-0.nightly-2024-10-02-081004   True        False         False      8m27s   
      ingress                                    4.18.0-0.nightly-2024-10-02-081004   True        False         True       8m6s    The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:...
      insights                                   4.18.0-0.nightly-2024-10-02-081004   True        False         False      9m2s    
      kube-apiserver                             4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      kube-controller-manager                    4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      kube-scheduler                             4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      kube-storage-version-migrator              4.18.0-0.nightly-2024-10-02-081004   True        False         False      8m59s   
      monitoring                                 4.18.0-0.nightly-2024-10-02-081004   True        False         False      6m33s   
      network                                    4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      node-tuning                                4.18.0-0.nightly-2024-10-02-081004   True        True          False      9m56s   Waiting for 2/2 Profiles to be applied
      openshift-apiserver                        4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      openshift-controller-manager               4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      openshift-samples                          4.18.0-0.nightly-2024-10-02-081004   True        False         False      8m17s   
      operator-lifecycle-manager                 4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      operator-lifecycle-manager-catalog         4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      operator-lifecycle-manager-packageserver   4.18.0-0.nightly-2024-10-02-081004   True        False         False      15m     
      service-ca                                 4.18.0-0.nightly-2024-10-02-081004   True        False         False      8m59s   
      storage                                    4.18.0-0.nightly-2024-10-02-081004   True        False         False      10m     
      jiezhao-mac:hypershift jiezhao$ 
      jiezhao-mac:hypershift jiezhao$ oc get pods -n kube-system
      NAME                                                              READY   STATUS    RESTARTS   AGE
      konnectivity-agent-m9c9x                                          1/1     Running   0          11m
      konnectivity-agent-qk8g6                                          1/1     Running   0          11m
      kube-apiserver-proxy-ip-10-0-136-216.us-east-2.compute.internal   1/1     Running   0          11m
      kube-apiserver-proxy-ip-10-0-137-243.us-east-2.compute.internal   1/1     Running   0          11m
      jiezhao-mac:hypershift jiezhao$ 
      jiezhao-mac:hypershift jiezhao$ oc get pods/kube-apiserver-proxy-ip-10-0-136-216.us-east-2.compute.internal -n kube-system -o yaml
      apiVersion: v1
      kind: Pod
      metadata:
        annotations:
          kubernetes.io/config.hash: 2d1ad2ba5129d58e0c1c4ffcc21212a6
          kubernetes.io/config.mirror: 2d1ad2ba5129d58e0c1c4ffcc21212a6
          kubernetes.io/config.seen: "2024-10-02T19:44:20.077131462Z"
          kubernetes.io/config.source: file
        creationTimestamp: "2024-10-02T19:44:20Z"
        labels:
          k8s-app: kube-apiserver-proxy
        name: kube-apiserver-proxy-ip-10-0-136-216.us-east-2.compute.internal
        namespace: kube-system
        ownerReferences:
        - apiVersion: v1
          controller: true
          kind: Node
          name: ip-10-0-136-216.us-east-2.compute.internal
          uid: 20492180-ba1b-443f-b9a4-8f703df040d6
        resourceVersion: "6601"
        uid: d1e9aba9-539f-4187-bec6-7247e3144978
      spec:
        containers:
        - command:
          - control-plane-operator
          - kubernetes-default-proxy
          - --listen-addr=172.20.0.1:6443
          - --proxy-addr=10.0.130.219:80
          - --apiserver-addr=api-jie-test.jiezhao-hypershift.qe.devcluster.openshift.com:443
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7742cb977182e5cb3672b37852ec23e5d1bcc7b0b436b03aab8fefad3bb40f35
          imagePullPolicy: IfNotPresent
          name: kubernetes-default-proxy
          resources:
            requests:
              cpu: 13m
              memory: 16Mi
          securityContext:
            runAsUser: 1001
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
        dnsPolicy: ClusterFirst
        enableServiceLinks: true
        hostNetwork: true
        nodeName: ip-10-0-136-216.us-east-2.compute.internal
        preemptionPolicy: PreemptLowerPriority
        priority: 2000001000
        priorityClassName: system-node-critical
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        terminationGracePeriodSeconds: 30
        tolerations:
        - effect: NoExecute
          operator: Exists
        - effect: NoSchedule
          key: node.kubernetes.io/memory-pressure
          operator: Exists
      status:
        conditions:
        - lastProbeTime: null
          lastTransitionTime: "2024-10-02T19:44:50Z"
          status: "True"
          type: PodReadyToStartContainers
        - lastProbeTime: null
          lastTransitionTime: "2024-10-02T19:44:20Z"
          status: "True"
          type: Initialized
        - lastProbeTime: null
          lastTransitionTime: "2024-10-02T19:45:12Z"
          status: "True"
          type: Ready
        - lastProbeTime: null
          lastTransitionTime: "2024-10-02T19:45:12Z"
          status: "True"
          type: ContainersReady
        - lastProbeTime: null
          lastTransitionTime: "2024-10-02T19:44:20Z"
          status: "True"
          type: PodScheduled
        containerStatuses:
        - containerID: cri-o://7234d33a80a77c77a4e72fb5fad3eb13bfe5512a22a52d655f31a4d5d41d3b48
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7742cb977182e5cb3672b37852ec23e5d1bcc7b0b436b03aab8fefad3bb40f35
          imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7742cb977182e5cb3672b37852ec23e5d1bcc7b0b436b03aab8fefad3bb40f35
          lastState: {}
          name: kubernetes-default-proxy
          ready: true
          restartCount: 0
          started: true
          state:
            running:
              startedAt: "2024-10-02T19:45:11Z"
        hostIP: 10.0.136.216
        hostIPs:
        - ip: 10.0.136.216
        phase: Running
        podIP: 10.0.136.216
        podIPs:
        - ip: 10.0.136.216
        qosClass: Burstable
        startTime: "2024-10-02T19:44:20Z"
      jiezhao-mac:hypershift jiezhao$ 
      jiezhao-mac:hypershift jiezhao$ export KUBECONFIG=kubeconfig
      jiezhao-mac:hypershift jiezhao$ 
      jiezhao-mac:hypershift jiezhao$ oc get pods -n clusters-jie-test | grep ingress
      ingress-operator-6f455f6d96-nhv8n                     3/3     Running   0             18m
      jiezhao-mac:hypershift jiezhao$ 
      jiezhao-mac:hypershift jiezhao$ oc logs ingress-operator-6f455f6d96-nhv8n -n clusters-jie-test
      Defaulted container "ingress-operator" out of: ingress-operator, konnectivity-proxy, token-minter, availability-prober (init)
      2024-10-02T19:56:44.956Z    ERROR    operator.ingress_controller    controller/controller.go:114    got retryable error; requeueing    {"after": "1m0s", "error": "IngressController is degraded: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:\nerror sending canary HTTP request to \"canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com\": Get \"https://canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com\": Bad Gateway (x11 over 10m0s))"}
      jiezhao-mac:hypershift jiezhao$ oc logs ingress-operator-6f455f6d96-nhv8n -c konnectivity-proxy -n clusters-jie-test
      {"level":"info","ts":"2024-10-02T19:39:33Z","logger":"konnectivity-https-proxy","msg":"Starting proxy","version":"openshift/hypershift: 2d754dae4b6375bbb6e7bfb772580927f2e32fac. Latest supported OCP: 4.18.0"}
      Error: failed to split proxy URL host port (10.0.130.219): address 10.0.130.219: missing port in address2024/10/02 19:39:47 [001] INFO: Running 0 CONNECT handlers
      2024/10/02 19:39:47 [001] INFO: Accepting CONNECT to sts.us-east-2.amazonaws.com:443
      2024/10/02 19:39:47 [002] INFO: Running 0 CONNECT handlers
      2024/10/02 19:39:47 [002] INFO: Accepting CONNECT to route53.amazonaws.com:443
      2024/10/02 19:39:47 [003] INFO: Running 0 CONNECT handlers
      2024/10/02 19:39:47 [003] INFO: Accepting CONNECT to elasticloadbalancing.us-east-2.amazonaws.com:443
      2024/10/02 19:39:47 [004] INFO: Running 0 CONNECT handlers
      2024/10/02 19:39:47 [004] INFO: Accepting CONNECT to tagging.us-east-1.amazonaws.com:443
      2024/10/02 19:46:44 [005] INFO: Running 0 CONNECT handlers
      2024/10/02 19:46:44 [005] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF
      2024/10/02 19:47:44 [006] INFO: Running 0 CONNECT handlers
      2024/10/02 19:47:44 [006] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF
      2024/10/02 19:48:44 [007] INFO: Running 0 CONNECT handlers
      2024/10/02 19:48:44 [007] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF
      2024/10/02 19:49:44 [008] INFO: Running 0 CONNECT handlers
      2024/10/02 19:49:44 [008] WARN: Error dialing to canary-openshift-ingress-canary.apps.jie-test.jiezhao-dev.qe.devcluster.openshift.com:443: unexpected EOF

              cewong@redhat.com Cesar Wong
              rhn-support-jiezhao Jie Zhao
              None
              None
              Jie Zhao Jie Zhao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: