Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1222

ClusterLogForwarder namespace-specific log forwarding does not work as expected

    XMLWordPrintable

Details

    • False
    • False
    • NEW
    • NEW
    • Undefined
    • Logging (Core) - Sprint 198, Logging (Core) - Sprint 199, Logging (Core) - Sprint 200

    Description

      Description of problem:

      As per documentation[1], we can configure ClusterLogForwarder to send project-specific logs to external/internal log aggregators.

      [1] https://docs.openshift.com/container-platform/4.6/logging/cluster-logging-external.html#cluster-logging-collector-log-forwarding-about_cluster-logging-external

      If only single project logs need to be forwarded, the configuration works fine.
      But if we need to forward multiple project logs to different destinations, the fluentd configuration is messed up.

      For example, forwarding logs from two different projects to two different external elasticsearch instances, the logs from both the projects are forwarded to both the elasticsearch instances, due to the fluentd configuration getting messed up as below:

      ~~~
      cat /etc/fluentd/fluent.conf

      – OUTOUT_EMITTED---

      1. A log source matcher may be null if no pipeline wants that type of log.
        <match *default* *kube-** *openshift-** *openshift* journal.** system.var.log**>
        @type null
        </match>
        <match kubernetes.*dev-apple* kubernetes.*dev-ocp* >
        @type relabel
        @label @_APPLICATION <<<<< Here combined logs are labelled as APPLICATION logs
        </match>
        <match kubernetes.** >
        @type null
        </match>
        <match linux-audit.log** k8s-audit.log** openshift-audit.log**>
        @type null
        </match>

      <match **>
      @type stdout
      </match>

      </label>

      1. Relabel specific sources (e.g. logs.apps) to multiple pipelines <<< Then we have this section, I believe this section should be merged/corrected with above section
        <label @_APPLICATION>
        <match **>
        @type copy

      <store>
      @type relabel
      @label @LOGS_NAMESPACES_APPLE
      </store>
      <store>
      @type relabel
      @label @LOGS_NAMESPACES_OCP
      </store>

      -OUTPUT_EMITTED-
      ~~~

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:
      1. Deploy an OCP 4.6 cluster and install cluster-logging stack on it.
      2. Configure clusterlogforwarder to forward logs from two projects to different elasticsearch instance.
      3. Check the logs on external elasticsearch instances, it will contain logs from both the projects

      Actual results:

      Logs from both the project are available on both elasticsearch instances

      Expected results:

      Logs from one project to go to the first external elasticsearch instance and the second projects logs to the second elasticsearch instance.

      Additional info:

      [1] Example CLF configuration:

      ~~~
      apiVersion: "logging.openshift.io/v1"
      kind: ClusterLogForwarder
      metadata:
      name: instance
      namespace: openshift-logging
      spec:
      outputs:

      • name: elasticsearch-ocp
        type: "elasticsearch"
        url: http://<FQDN-ES-1>:9200
      • name: elasticsearch-apple
        type: "elasticsearch"
        url: http://<FQDN-ES-2>:9200
        inputs:
      • name: input-namespaces-ocp
        application:
        namespaces:
      • dev-ocp
      • name: input-namespaces-apple
        application:
        namespaces:
      • dev-apple
        pipelines:
      • name: logs-namespaces-ocp
        inputRefs:
      • input-namespaces-ocp
        outputRefs:
      • elasticsearch-ocp
        labels:
        datacenter: lab-ocpcart-01
      • name: logs-namespaces-apple
        inputRefs:
      • input-namespaces-apple
        outputRefs:
      • elasticsearch-apple
        labels:
        datacenter: lab-ocpcart-01
        ~~~

      Attachments

        Issue Links

          Activity

            People

              jcantril@redhat.com Jeffrey Cantrill
              jcantril@redhat.com Jeffrey Cantrill
              Anping Li Anping Li
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 1 week
                  1w
                  Remaining:
                  0m
                  Logged:
                  Time Not Required
                  Not Specified