Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-4551

Backport 4.11: Guard Pod Hostnames Too Long and Truncated Down Into Collisions With Other Masters

    XMLWordPrintable

Details

    • Important
    • Rejected
    • False
    • Hide

      None

      Show
      None

    Description

      Discovered in the must gather kubelet_service.log from https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-sdn-upgrade/1586093220087992320

      It appears the guard pod names are too long, and being truncated down to where they will collide with those from the other masters.

      From kubelet logs in this run:

      ❯ grep openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste kubelet_service.log
Oct 28 23:58:55.693391 ci-op-3hj6pnwf-4f6ab-lv57z-master-1 kubenswrapper[1657]: E1028 23:58:55.693346    1657 kubelet_pods.go:413] "Hostname for pod was too long, truncated it" podName="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-master-1" hostnameMaxLen=63 truncatedHostname="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste"
Oct 28 23:59:03.735726 ci-op-3hj6pnwf-4f6ab-lv57z-master-0 kubenswrapper[1670]: E1028 23:59:03.735671    1670 kubelet_pods.go:413] "Hostname for pod was too long, truncated it" podName="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-master-0" hostnameMaxLen=63 truncatedHostname="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste"
Oct 28 23:59:11.168082 ci-op-3hj6pnwf-4f6ab-lv57z-master-2 kubenswrapper[1667]: E1028 23:59:11.168041    1667 kubelet_pods.go:413] "Hostname for pod was too long, truncated it" podName="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-master-2" hostnameMaxLen=63 truncatedHostname="openshift-kube-scheduler-guard-ci-op-3hj6pnwf-4f6ab-lv57z-maste"

      This also looks to be happening for openshift-kube-scheduler-guard, kube-controller-manager-guard, possibly others.

      Looks like they should be truncated further to make room for random suffixes in https://github.com/openshift/library-go/blame/bd9b0e19121022561dcd1d9823407cd58b2265d0/pkg/operator/staticpod/controller/guard/guard_controller.go#L97-L98

      Unsure of the implications here, it looks a little scary.

      Attachments

        Issue Links

          clones

          Bug - A problem which impairs or prevents the functions of the product. OCPBUGS-4478 Backport: Guard Pod Hostnames Too Long and Truncated Down Into Collisions With Other Masters

          • Undefined - Priority not specified.
          • Closed
          depends on

          Bug - A problem which impairs or prevents the functions of the product. OCPBUGS-4478 Backport: Guard Pod Hostnames Too Long and Truncated Down Into Collisions With Other Masters

          • Undefined - Priority not specified.
          • Closed
          is cloned by

          Bug - A problem which impairs or prevents the functions of the product. OCPBUGS-4552 Backport 4.10: Guard Pod Hostnames Too Long and Truncated Down Into Collisions With Other Masters

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          is depended on by

          Bug - A problem which impairs or prevents the functions of the product. OCPBUGS-4552 Backport 4.10: Guard Pod Hostnames Too Long and Truncated Down Into Collisions With Other Masters

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          links to

          GitHub openshift/cluster-kube-apiserver-operator#1429: OCPBUGS-4551: guard controller: set an explicit hostname to avoid name collisions

          GitHub openshift/cluster-kube-controller-manager-operator#690: OCPBUGS-4551: guard controller: set an explicit hostname to avoid name collisions

          GitHub openshift/cluster-kube-scheduler-operator#458: OCPBUGS-4551: Sync library go 4.11

          GitHub openshift/library-go#1455: OCPBUGS-4551: [release-4.11] guard controller: set an explicit hostname to avoid name collisions

          Activity

            People

              rh-ee-lseveroa Lucas Severo Alves
              rhn-engineering-dgoodwin Devan Goodwin
              Jitendar Singh Jitendar Singh
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: