Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4891

[release-5.7] lokistack ruler Error while dialing IPv6

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, the LokiStack ruler pods would not format the IPv6 pod IP in HTTP URLs used for cross pod communication caused querying rules and alerts through the Prometheus-compatible API to fail. With this update, the LokiStack ruler pods encapsulate the IPv6 pod IP in square brackets calls resolves the issue and querying rules and alerts through the Prometheus-compatible API works as in IPv4 environments.
      Show
      Before this update, the LokiStack ruler pods would not format the IPv6 pod IP in HTTP URLs used for cross pod communication caused querying rules and alerts through the Prometheus-compatible API to fail. With this update, the LokiStack ruler pods encapsulate the IPv6 pod IP in square brackets calls resolves the issue and querying rules and alerts through the Prometheus-compatible API works as in IPv4 environments.
    • Known Issue
    • Log Storage - Sprint 246

    Description

      Description of problem:

      it seems the application(console plugin? loki?) should dial [fd01:0:0:5::95]:9095 rather than fd01:0:0:5::95:9095

      $oc logs lokistack-sample-ruler-0

      ...

      Error while dialing: dial tcp: address fd01:0:0:5::95:9095: too many colons in address\""

      level=warn ts=2023-10-23T10:16:59.074130143Z caller=logging.go:123 orgID=application msg="GET /prometheus/api/v1/rules (500) 841.674µs Response: \"{\\\"status\\\":\\\"error\\\",\\\"data\\\":null,\\\"errorType\\\":\\\"server_error\\\",\\\"error\\\":\\\"unable to retrieve rules from ruler fd01:0:0:5::95:9095: rpc error: code = Unavailable desc = connection error: desc = \\\\\\\"transport: Error while dialing: dial tcp: address fd01:0:0:5::95:9095: too many colons in address\\\\\\\"\\\"}\" ws: false; Accept: application/json; Accept-Encoding: gzip, deflate, br; Accept-Language: en-US,en;q=0.5; Forwarded: for=\"[2604:1380:4642:7e01:3d11:d50a:3b41:ef87]\";host=console-openshift-console.apps.anli623.qe.devcluster.openshift.com;proto=https; Referer: https://console-openshift-console.apps.anli623.qe.devcluster.openshift.com/monitoring/alerts?rowFilter-alert-state=firing&rowFilter-alert-source=platform; Sec-Fetch-Dest: empty; Sec-Fetch-Mode: cors; Sec-Fetch-Site: same-origin; User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0; X-Forwarded-For: 2604:1380:4642:7e01:3d11:d50a:3b41:ef87, fd01:0:0:4::2; X-Forwarded-Host: console-openshift-console.apps.anli623.qe.devcluster.openshift.com; X-Forwarded-Port: 443; X-Forwarded-Prefix: /api/logs/v1/application; X-Forwarded-Proto: https; X-Scope-Orgid: application; "

      ....

      Version-Release number of selected component (if applicable):

      4.14.0-0.nightly-2023-10-18-004928

      How reproducible:

      always

      Steps to Reproduce:

      1. Deploy lokistack on IPv6 single stack cluster

      ---
      apiVersion: loki.grafana.com/v1
      kind: LokiStack
      metadata:
        name: lokistack-sample
      spec:
        hashRing:
        memberlist:
          enableIPv6: true
        type: memberlist
        managementState: Managed
        size: 1x.demo
        storage:
          secret:
            name: s3-secret
            type: s3
        storageClassName: nfs
        tenants:
          mode: openshift-logging
        rules:
          enabled: true
          namespaceSelector:
            matchLabels:
              openshift.io/cluster-monitoring: "true"
          selector:
            matchLabels:
              openshift.io/cluster-monitoring: "true"

      2. Create alertingrules for project my-app-1

      oc new-project my-app-1
      oc new-app https://raw.githubusercontent.com/openshift/verification-tests/master/testdata/logging/loggen/container_json_log_template.json
      Note: Make sure to add openshift.io/cluster-monitoring: 'true' label on the app namespace.
      apiVersion: loki.grafana.com/v1
      kind: AlertingRule
      metadata:
        labels:
          openshift.io/cluster-monitoring: 'true'
        name: dev-workload-alerts
        namespace: my-app-1
      spec:
        groups:
          - interval: 1m
            name: devAppAlert
            rules:
              - alert: DevAppLogVolumeIsHigh
                annotations:
                  description: My application has high amount of logs.
                  summary: project "my-app-1" log volume is high.
                expr: >
                  count_over_time({kubernetes_namespace_name="my-app-1"}[2m])
                  > 10
                for: 5m
                labels:
                  severity: info
                  devApp: 'true'
      

       

       

      3. Check the loki ruler pods
      4. Check the alert rules in Webconsole

      Expected results:

       

      we can find dev-workload-alerts in Observe->Alerting->Alerting rules

      Actual results:

      we cannot see dev-workload-alerts.

      oc logs lokistack-sample-ruler-0

      ....
      Error while dialing: dial tcp: address fd01:0:0:5::95:9095: too many colons in address\""
      level=warn ts=2023-10-23T10:16:59.074130143Z caller=logging.go:123 orgID=application msg="GET /prometheus/api/v1/rules (500) 841.674µs Response: \"{\\\"status\\\":\\\"error\\\",\\\"data\\\":null,\\\"errorType\\\":\\\"server_error\\\",\\\"error\\\":\\\"unable to retrieve rules from ruler fd01:0:0:5::95:9095: rpc error: code = Unavailable desc = connection error: desc = \\\\\\\"transport: Error while dialing: dial tcp: address fd01:0:0:5::95:9095: too many colons in address\\\\\\\"\\\"}\" ws: false; Accept: application/json; Accept-Encoding: gzip, deflate, br; Accept-Language: en-US,en;q=0.5; Forwarded: for=\"[2604:1380:4642:7e01:3d11:d50a:3b41:ef87]\";host=console-openshift-console.apps.anli623.qe.devcluster.openshift.com;proto=https; Referer: https://console-openshift-console.apps.anli623.qe.devcluster.openshift.com/monitoring/alerts?rowFilter-alert-state=firing&rowFilter-alert-source=platform; Sec-Fetch-Dest: empty; Sec-Fetch-Mode: cors; Sec-Fetch-Site: same-origin; User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0; X-Forwarded-For: 2604:1380:4642:7e01:3d11:d50a:3b41:ef87, fd01:0:0:4::2; X-Forwarded-Host: console-openshift-console.apps.anli623.qe.devcluster.openshift.com; X-Forwarded-Port: 443; X-Forwarded-Prefix: /api/logs/v1/application; X-Forwarded-Proto: https; X-Scope-Orgid: application; "

       

      Additional info:

      Attachments

        Activity

          People

            ptsiraki@redhat.com Periklis Tsirakidis
            anli@redhat.com Anping Li
            Kabir Bharti Kabir Bharti
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: