Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-11449

[Backport-4.13] Excessive TopologyAwareHintsDisabled events due to service/dns-default with topology aware hints activated.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • 4.13.z
    • 4.13
    • Networking / DNS
    • None
    • Critical
    • No
    • Sprint 234, Sprint 235, Sprint 236, Sprint 237, Sprint 238, Sprint 239, Sprint 240, Sprint 241, Sprint 242, Sprint 243, Sprint 244, Sprint 245, Sprint 246, Sprint 247, Sprint 248, Sprint 249, Sprint 250, Sprint 251, Sprint 252, Sprint 254, NE Sprint 255, NE Sprint 256
    • 22
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the DNS Operator did not verify that a cluster had ready nodes with available CPU in at least two availability zones, and the DNS daemon set did not use surge for rolling updates. As a result, clusters in which all nodes were in the same availability zone would repeatedly emit TopologyAwareHintsDisabled events for the cluster DNS service. With this release, the TopologyAwareHintsDisabled events are no longer emitted on clusters that do not have nodes in multiple availability zones and the issue has been resolved. (link:https://issues.redhat.com/browse/OCPBUGS-11449[*OCPBUGS-11449*])
      Show
      * Previously, the DNS Operator did not verify that a cluster had ready nodes with available CPU in at least two availability zones, and the DNS daemon set did not use surge for rolling updates. As a result, clusters in which all nodes were in the same availability zone would repeatedly emit TopologyAwareHintsDisabled events for the cluster DNS service. With this release, the TopologyAwareHintsDisabled events are no longer emitted on clusters that do not have nodes in multiple availability zones and the issue has been resolved. (link: https://issues.redhat.com/browse/OCPBUGS-11449 [* OCPBUGS-11449 *])
    • Bug Fix
    • In Progress

      This is a clone of issue OCPBUGS-5943. The following is the description of the original issue:

      Kube 1.26 introduced the warning level TopologyAwareHintsDisabled event. TopologyAwareHintsDisabled is fired by the EndpointSliceController whenever reconciling a service that has activated topology aware hints via the service.kubernetes.io/topology-aware-hints annotation, but there is not enough information in the existing cluster resources (typically nodes) to apply the topology aware hints.

      When re-basing OpnShift onto Kube 1.26, are CI builds are failing (except on AWS), because these events are firing "pathologically", for example:

      : [sig-arch] events should not repeat pathologically
        events happened too frequently event happened 83 times, something is wrong: ns/openshift-dns service/dns-default - reason/TopologyAwareHintsDisabled Insufficient Node information: allocatable CPU or zone not specified on one or more nodes, addressType: IPv4 result=reject 

      AWS nodes seem to have the proper values in the nodes. GCP has the values also, but they are not "right" for the purposes of the EndpointSliceController:

      event happened 38 times, something is wrong: ns/openshift-dns service/dns-default - reason/TopologyAwareHintsDisabled Unable to allocate minimum required endpoints to each zone without exceeding overload threshold (5 endpoints, 3 zones), addressType: IPv4 result=reject }

      https://github.com/openshift/origin/pull/27666 will mask this problem (make it stop erroring in CI) but changes still need to be made in the product so end users are not subjected to these events.

       

              mmasters1@redhat.com Miciah Masters
              openshift-crt-jira-prow OpenShift Prow Bot
              Melvin Joseph Melvin Joseph
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: