Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-10342

Observability - metrics collector would not work if the cert is updated

XMLWordPrintable

    • 1
    • False
    • None
    • False
    • No
    • RHOBS Sprint 20
    • Important

      Description of problem: When the cert 

      observability-server-ca-certs
      observability-server-certs
      observability-client-ca-certs

      updated, the metrics collector will report error and failed to forward the data. Have to manually delete secret observability-controller-open-cluster-management.io-observability-signer-client-cert to work around.

      level=warn caller=logger.go:55 ts=2024-03-08T06:13:14.34379071Z component=forwarder component=metricsclient msg="error: failed to forward request happened at time: 18.301015852s"
      level=warn caller=logger.go:55 ts=2024-03-08T06:13:32.664422183Z component=forwarder component=metricsclient msg="failed to forward request" err="Post \"https://observatorium-api-open-cluster-management-observability.apps.regional-hub-135.dev09.red-chesterfield.com/api/metrics/v1/default/api/v1/receive\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"observability-server-ca-certificate\")"

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:

      1. Manually delete secret 
        observability-server-ca-certs
        observability-server-certs
        observability-client-ca-certs
      2. check the metrics collector log, it reports above error
      3. hub's data would be not forward and display on the Grafana
      4. manually delete secret observability-controller-open-cluster-management.io-observability-signer-client-cert
      5. metrics collector back to work as expected

      Actual results:

      Expected results:

      Additional info:

            rh-ee-coquadro Coleen Iona Quadros
            cquredhat ChangLiang Qu
            Xiang Yin Xiang Yin
            ACM QE Team
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: