Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2860

Error on LokiStack Components when forwarding logs to Loki on proxy cluster

    XMLWordPrintable

Details

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, loki did not have support to reach an external storage in a disconnected cluster. This update, makes provision to add proxy env variables and proxy trusted CA bundles in the container image so that secure connection can be established
      Show
      Before this update, loki did not have support to reach an external storage in a disconnected cluster. This update, makes provision to add proxy env variables and proxy trusted CA bundles in the container image so that secure connection can be established
    • Log Storage - Sprint 222, Log Storage - Sprint 223, Log Storage - Sprint 224, Log Storage - Sprint 225, Log Storage - Sprint 226, Log Storage - Sprint 227, Log Storage - Sprint 228

    Description

      Unable to query Loki logs through logcli with time out error. Also, see network/DNS errors on Loki Components.

      Gateway:

      level=error name=lokistack-gateway ts=2022-07-26T13:51:37.314957328Z caller=openshift.go:146 tenant=application msg="unable to auto discover OpenShift OAuth endpoints: unable to send request to oauth server: Get \"https://172.30.0.1/.well-known/oauth-authorization-server\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" level=error name=lokistack-gateway ts=2022-07-26T13:51:37.317124791Z caller=openshift.go:146 tenant=infrastructure msg="unable to auto discover OpenShift OAuth endpoints: unable to send request to oauth server: Get \"https://172.30.0.1/.well-known/oauth-authorization-server\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" level=error name=lokistack-gateway ts=2022-07-26T13:51:37.317263053Z caller=openshift.go:146 tenant=audit msg="unable to auto discover OpenShift OAuth endpoints: unable to send request to oauth server: Get \"https://172.30.0.1/.well-known/oauth-authorization-server\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

      Ingester:

      level=error ts=2022-07-26T14:37:04.802619358Z caller=flush.go:146 org_id=audit msg="failed to flush user" err="store put chunk: RequestError: send request failed\ncaused by: Put \"https://s3.us-east-1.amazonaws.com/vsphere-kbharti-loki/audit/b2a68385d7921ab0%3A1823ac0e361%3A1823acfb388%3A9b702641\": dial tcp 52.217.72.254:443: i/o timeout" level=error ts=2022-07-26T14:37:04.840920722Z caller=flush.go:146 org_id=audit msg="failed to flush user" err="store put chunk: RequestError: send request failed\ncaused by: Put \"https://s3.us-east-1.amazonaws.com/vsphere-kbharti-loki/audit/1a706534f6a6528f%3A1823ac054ba%3A1823acde5c5%3A5ab455b6\": dial tcp 52.217.72.254:443: i/o timeout" level=error ts=2022-07-26T14:37:04.864713696Z caller=flush.go:146 org_id=infrastructure msg="failed to flush user" err="store put chunk: RequestError: send request failed\ncaused by: Put \"https://s3.us-east-1.amazonaws.com/vsphere-kbharti-loki/infrastructure/a821a114b092d15d%3A1823a9a14e3%3A1823ad735ce%3Aa8c94853\": dial tcp 52.217.72.254:443: i/o timeout" level=error ts=2022-07-26T14:37:04.906220475Z caller=flush.go:146 org_id=infrastructure msg="failed to flush user" err="store put chunk: RequestError: send request failed\ncaused by: Put \"https://s3.us-east-1.amazonaws.com/vsphere-kbharti-loki/infrastructure/6f43c796dd089202%3A1823a99c188%3A1823ac2b736%3A247e00fe\": dial tcp 52.217.72.254:443: i/o timeout"

      Querier:

      level=warn ts=2022-07-26T13:51:51.057805228Z caller=dns_resolver.go:225 msg="failed DNS A record lookup" err="lookup lokistack-dev-query-frontend-grpc.openshift-logging.svc.cluster.local on 172.30.0.10:53: no such host"   level=warn ts=2022-07-26T13:51:54.070748127Z caller=dns_resolver.go:225 msg="failed DNS A record lookup" err="lookup lokistack-dev-query-frontend-grpc.openshift-logging.svc.cluster.local on 172.30.0.10:53: no such host"  
      level=info ts=2022-07-26T13:51:57.082108286Z caller=worker.go:209 msg="adding connection" addr=10.128.2.14:9095
      

      Cluster Profile: vsphere7 https-proxy cluster

      OCP version: 4.11

      Steps to reproduce:

      1) Deploy Loki and ClusterLogging Operators.

      2) Create LokiStack CR 

      spec:
        managementState: Managed
        replicationFactor: 1
        rules:
          enabled: true
          namespaceSelector:
            matchLabels:
              openshift.io/cluster-monitoring: 'true'
          selector:
            matchLabels:
              openshift.io/cluster-monitoring: 'true'
        size: 1x.extra-small
        storage:
          schemas:
            - effectiveDate: '2020-10-11'
              version: v11
          secret:
            name: test
            type: s3
        storageClassName: thin
        tenants:
          mode: openshift-logging

      How reproducible: Always

      Attachments

        Activity

          People

            spad09 Shweta Padubidri
            rhn-support-kbharti Kabir Bharti
            Anping Li Anping Li
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: