Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-1209

Using legacy Log Forwarding is not sending logs to the internal Elasticsearch

    XMLWordPrintable

Details

    • Logging (Core) - Sprint 198
    • Passed
    • NEW
    • VERIFIED

    Description

      +++ This bug was initially created as a clone of Bug #1911477 +++

      [Description of problem]
      In previous version, for example 3.x and until 4.3. When using secure_forward following the documentation [1] the logs were sent to the internal Elasticsearch and to the external instance.

      Now, in 4.5 it's only sending to the external instance and it doesn't send more to the internal Elasticsearch.

      Verifying the documentation, it has not changed [2]. Then, it's expected that it works like it did in the past:

      • Sending logs to the internal Elasticsearch
      • Sending logs to the external instance configured in the secure-forward configmap

      One thing has changed in the configuration generated for fluentd. In 4.3 the fluentd configuration after configuring secure_forward following the documentation is like this:

      ~~~
      $ oc rsh <fluentd pod> cat /etc/fluent/fluent.conf
      ...
      <label @_LOGS_APP>
      <match **>
      @type copy

      <store>
      @type relabel
      @label @CLO_DEFAULT_APP_PIPELINE
      </store>

      <store>
      @type relabel
      @label @_LEGACY_SECUREFORWARD
      </store>

      </match>
      <label @_LOGS_INFRA>
      <match **>
      @type copy

      <store>
      @type relabel
      @label @CLO_DEFAULT_INFRA_PIPELINE
      </store>

      <store>
      @type relabel
      @label @_LEGACY_SECUREFORWARD
      </store>

      </match>
      </label>

      1. Relabel specific pipelines to multiple, outputs (e.g. ES, kafka stores)

      <label @CLO_DEFAULT_APP_PIPELINE>
      <match **>
      @type copy

      <store>
      @type relabel
      @label @CLO_DEFAULT_OUTPUT_ES
      </store>
      </match>
      </label>

      <label @CLO_DEFAULT_INFRA_PIPELINE>
      <match **>
      @type copy

      <store>
      @type relabel
      @label @CLO_DEFAULT_OUTPUT_ES
      </store>
      </match>
      </label>
      ...
      ~~~

      As we can see above, it's sending to the CLO_DEFAULT and to the LEGACY_SECUREFORWARD, but, the configuration in OCP 4.5 generated after configuring the secure forward is like this:

      ~~~
      $ oc rsh <fluentd pod> cat /etc/fluent/fluent.conf
      ...
      <label @_LOGS_APP>
      <match **>
      @type copy

      <store>
      @type relabel
      @label @_LEGACY_SECUREFORWARD
      </store>

      </match>
      </label>
      <label @_LOGS_AUDIT>
      <match **>
      @type copy

      <store>
      @type relabel
      @label @_LEGACY_SECUREFORWARD
      </store>

      </match>
      </label>
      <label @_LOGS_INFRA>
      <match **>
      @type copy

      <store>
      @type relabel
      @label @_LEGACY_SECUREFORWARD
      </store>

      </match>
      </label>
      ...
      ~~~

      As we can see, it's only relabeling like "_LEGACY_SECUREFORWARD", but it's not possible to see the relabeling to CLO_DEFAULT_XXX.

      [Version-Release number of selected component (if applicable):]

      Version used for OCP 4.5
      ~~~
      $ oc version
      Client Version: 4.5.23
      Server Version: 4.5.23
      $ oc get csv -n openshift-logging
      NAME DISPLAY VERSION REPLACES PHASE
      clusterlogging.4.5.0-202012120433.p0 Cluster Logging 4.5.0-202012120433.p0 Failed
      elasticsearch-operator.4.5.0-202012120433.p0 Elasticsearch Operator 4.5.0-202012120433.p0 Succeeded
      ~~~

      Version used for OCP 4.3:
      ~~~
      $ oc version
      Client Version: 4.3.38
      Server Version: 4.3.40
      Kubernetes Version: v1.16.2+853223d
      $ oc get csv -n openshift-logging
      NAME DISPLAY VERSION REPLACES PHASE
      clusterlogging.4.3.40-202010141211.p0 Cluster Logging 4.3.40-202010141211.p0 Succeeded
      elasticsearch-operator.4.3.40-202010141211.p0 Elasticsearch Operator 4.3.40-202010141211.p0 Succeeded
      ~~~

      [How reproducible]
      Always

      Steps to Reproduce:
      1. Install Cluster Logging
      2. Configure secure_forward
      3. Logs are not sent to the internal Elasticsearch

      [Actual results]
      Logs are not sent to the internal Elasticsearch

      [Expected results]
      Logs should be sent to the internal Elasticsearch at the same time that to the external instance configured in the secure-forward configmap

      We are aware that this is deprecated, but in 4.3 the documentation is saying the same that in 4.4 and 4.5 and it was working in 4.3 and previous versions, the same that in 3.x. Then, it's expected that it continues working and the logs are sent in parallel to the internal Elasticsearch, to the external instance configured.

      [1] https://docs.openshift.com/container-platform/4.3/logging/config/cluster-logging-external.html
      [2] https://docs.openshift.com/container-platform/4.3/logging/config/cluster-logging-external.html#cluster-logging-collector-fluentd_cluster-logging-external

      Attachments

        Issue Links

          Activity

            People

              jcantril@redhat.com Jeffrey Cantrill
              jcantril@redhat.com Jeffrey Cantrill
              Anping Li Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 4 hours
                  4h
                  Remaining:
                  0m
                  Logged:
                  Time Not Required
                  Not Specified