Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-12359

Node Exporter scraping fails on TLS errors

XMLWordPrintable

    • 2
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • telemetry-operator-container-1.0.7-3, openstack-operator-container-1.0.7-3
    • None
    • Hide
      .TLS errors caused node exporter scraping issues

      This release of Red Hat OpenStack Services on OpenShift (RHOSO) fixes an issue with scraping metrics in specific dataplane configurations.
      Show
      .TLS errors caused node exporter scraping issues This release of Red Hat OpenStack Services on OpenShift (RHOSO) fixes an issue with scraping metrics in specific dataplane configurations.
    • Bug Fix
    • Done
    • CloudOps 2024 Sprint 26, CloudOps 2025 Sprint 5
    • 2
    • Important

      When inside the openstackdataplanenodeset CR a user doesn't set the ansibleHost to an IP address of each compute node and at the same time IP set doesn't exist, scraping of Node Exporter metrics fails with TLS errors:

      2024-11-18T21:05:52.734089145Z ts=2024-11-18T21:05:52.733996025Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from <IP address>:53884: EOF" 

      The reason of the issue is, that node exporter and kepler certificates are generated only for the compute node IP addresses, not for the compute node hostnames. When the ansibleHost isn't explicitly set in the openstackdataplanenodeset CR, the openstack-operator sets the ansible_host variable in the dataplane inventory CR to a hostname of the compute node. The ansible_host variable is then read by the telemetry_operator and used for ScrapeConfigs. So this way Prometheus gets configured to scrape the Node Exporter by using a hostname, while TLS certificates are created only for IP addresses.

      We should be generating Node Exporter and Kepler certificates so that they also work with hostnames.

      We need to add `- dnsnames` under https://github.com/openstack-k8s-operators/openstack-operator/blob/main/config/services/dataplane_v1beta1_openstackdataplaneservice_telemetry.yaml#L13
      ditto for kepler.

              lnatapov@redhat.com Leonid Natapov
              rh-ee-jwysogla Jaromir Wysoglad
              Leonid Natapov Leonid Natapov
              rhos-dfg-cloudops
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: