Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3280

Logs generated by lokistack pods are collected and sent to log store.

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before this update, when LokiStack was deployed as the log store, the logs generated by Loki pods were collected and sent to LokiStack.

      With this update, the logs generated by Loki are excluded from collection. As a result, they will not be stored.
      Show
      Before this update, when LokiStack was deployed as the log store, the logs generated by Loki pods were collected and sent to LokiStack. With this update, the logs generated by Loki are excluded from collection. As a result, they will not be stored.
    • Log Collection - Sprint 227, Log Collection - Sprint 228, Log Collection - Sprint 229, Log Collection - Sprint 230, Log Collection - Sprint 231, Log Collection - Sprint 232, Log Collection - Sprint 233

      Description of problem:

      When deploy lokistack as the log store, the logs generated by loki pods are collected and sent to lokistack.

      $ logcli -o raw --tls-skip-verify --bearer-token="${token}" --addr="https://${lokistack_route}/api/logs/v1/infrastructure" query '{kubernetes_namespace_name="openshift-logging"}' |jq
      Common labels: {kubernetes_namespace_name="openshift-logging", log_type="infrastructure"}
      {
        "@timestamp": "2022-11-10T08:06:58.015770082Z",
        "file": "/var/log/pods/openshift-logging_loki-53817-query-frontend-78bbd685c-6mt7h_b0682bd7-2bf5-4352-af81-a911fb5c7d16/loki-query-frontend/0.log",
        "hostname": "preserve-qitan-vzcbc-worker-0-kgtbb",
        "kubernetes": {
          "annotations": {
            "k8s.v1.cni.cncf.io/network-status": "[{\n    \"name\": \"openshift-sdn\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.131.0.50\"\n    ],\n    \"default\": true,\n    \"dns\": {}\n}]",
            "k8s.v1.cni.cncf.io/networks-status": "[{\n    \"name\": \"openshift-sdn\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.131.0.50\"\n    ],\n    \"default\": true,\n    \"dns\": {}\n}]",
            "loki.grafana.com/certRotationRequiredAt": "",
            "loki.grafana.com/config-hash": "b101b6b7a6494595583a176b1e8c549f673cc94c",
            "openshift.io/scc": "restricted-v2",
            "seccomp.security.alpha.kubernetes.io/pod": "runtime/default"
          },
          "container_id": "cri-o://fd47c3bb6adcd41af262da2b9c352e6c18a71ce8535cd36092030c350a9689e0",
          "container_image": "registry.redhat.io/openshift-logging/logging-loki-rhel8@sha256:3b8d81248c12c8f2a505190a0d33921c0bd4d061fb13d0eadae798a3c77374e1",
          "container_name": "loki-query-frontend",
          "labels": {
            "app.kubernetes.io/component": "query-frontend",
            "app.kubernetes.io/created-by": "lokistack-controller",
            "app.kubernetes.io/instance": "loki-53817",
            "app.kubernetes.io/managed-by": "lokistack-controller",
            "app.kubernetes.io/name": "lokistack",
            "loki.grafana.com/gossip": "true",
            "pod-template-hash": "78bbd685c"
          },
          "namespace_labels": {
            "kubernetes.io/metadata.name": "openshift-logging",
            "olm.operatorgroup.uid/5126c530-d4d8-422d-ad67-d7f4b127ac1b": "",
            "olm.operatorgroup.uid/f65225dc-32f7-47c3-96cd-5de5a1113355": "",
            "openshift.io/cluster-monitoring": "true",
            "pod-security.kubernetes.io/audit": "privileged",
            "pod-security.kubernetes.io/enforce": "privileged",
            "pod-security.kubernetes.io/warn": "privileged",
            "security.openshift.io/scc.podSecurityLabelSync": "false"
          },
          "namespace_name": "openshift-logging",
          "pod_id": "b0682bd7-2bf5-4352-af81-a911fb5c7d16",
          "pod_ip": "10.131.0.50",
          "pod_name": "loki-53817-query-frontend-78bbd685c-6mt7h",
          "pod_owner": "ReplicaSet/loki-53817-query-frontend-78bbd685c"
        },
        "level": "info",
        "log_type": "infrastructure",
        "message": "level=info ts=2022-11-10T08:06:57.969552353Z caller=metrics.go:143 component=frontend org_id=infrastructure latency=fast query=\"{kubernetes_namespace_name=\\\"openshift-logging\\\"}\" query_type=limited range_type=range length=1h0m0s start_delta=1h0m1.741892886s end_delta=1.741893074s step=14s duration=344.945268ms status=200 limit=30 returned_lines=0 throughput=1.1MB total_bytes=374kB total_entries=30 queue_time=0s subqueries=2 cache_chunk_req=0 cache_chunk_hit=0 cache_chunk_bytes_stored=0 cache_chunk_bytes_fetched=0 cache_index_req=0 cache_index_hit=0 cache_result_req=1 cache_result_hit=0",
        "openshift": {
          "cluster_id": "1d75ef55-bd8d-4686-a2cb-939f96e4dede"
        }
      }

      To compare with ES, when using ES as the log store, we don't collect logs in ES and Kibana.

      Version-Release number of selected component (if applicable):

      cluster-logging.v5.6.0

      loki-operator.v5.6.0

      How reproducible:

      Always

      Steps to Reproduce:

      1. deploy lokistack, then deploy clusterlogging, using lokistack as the default log store
      2. check data in lokistack

      Actual results:

      Logs generated by loki pods are collected

      Expected results:

      Shouldn't collect loki pods' logs

      Additional info:

       

            vparfono Vitalii Parfonov
            qitang@redhat.com Qiaoling Tang
            Qiaoling Tang Qiaoling Tang
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: