Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2525

The container's logs are not sent to separate index if the annotation is added after the pod is ready.

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide

      1. deploy logging stack with fluentd collector

      2. create a CLF with:

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        outputDefaults:
          elasticsearch:
            enableStructuredContainerLogs: true
            structuredTypeKey: kubernetes.namespace_name
            structuredTypeName: test-name
        pipelines:
        - inputRefs:
          - application
          name: forward
          outputRefs:
          - default
          parse: json

      3. create some pods to generate json format logs, then add annotation to pods, for example:

      oc annotate pod centos-logtest-gbt8m containerType.logging.openshift.io/centos-logtest=container-log-centos-logtest -n test-1

      4. check indices in ES, the index name is app-$project-name, not app-container-log-centos-logtest:

      $ oc exec elasticsearch-cdm-slb4l0jd-1-749984d987-rknr8  -- indices
      Defaulted container "elasticsearch" out of: elasticsearch, proxy
      Sun Apr 24 08:30:47 UTC 2022
      health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
      green  open   infra-000001      Uit3BxfJT6SnaNA-MtDciQ   3   1          0            0          0              0
      green  open   app-test-1-000001 YMjkNaTlT7OFQ39vCW9M4A   3   1        130            0          0              0
      green  open   .security         9BwDwv_5TSGVm6gITq8b6w   1   1          6            1          0              0
      green  open   .kibana_1         kYAdIIvMRtGU15pjxzatcg   1   1          0            0          0              0
      green  open   audit-000001      7EaYqfpeRRSrlJBNezLpzw   3   1          0            0          0              0
      green  open   app-000001        bZo9_oxvSCqVDK_fjD0HqQ   3   1          0            0          0              0

      5. remove all the collector pods and wait for new collector pods to be created, check indices again, then the index app-container-log-centos-logtest can be created:

      $ oc exec elasticsearch-cdm-slb4l0jd-1-749984d987-rknr8  -- indices
      Defaulted container "elasticsearch" out of: elasticsearch, proxy
      Sun Apr 24 08:32:12 UTC 2022
      health status index                                   uuid                   pri rep docs.count docs.deleted store.size pri.store.size
      green  open   audit-000001                            7EaYqfpeRRSrlJBNezLpzw   3   1          0            0          0              0
      green  open   app-000001                              bZo9_oxvSCqVDK_fjD0HqQ   3   1          0            0          0              0
      green  open   infra-000001                            Uit3BxfJT6SnaNA-MtDciQ   3   1          0            0          0              0
      green  open   app-test-1-000001                       YMjkNaTlT7OFQ39vCW9M4A   3   1        163            0          0              0
      green  open   app-container-log-centos-logtest-000001 GchIkq9MRmukYPWKlozM_Q   3   1         53            0          0              0
      green  open   .security                               9BwDwv_5TSGVm6gITq8b6w   1   1          6            1          0              0
      green  open   .kibana_1                               kYAdIIvMRtGU15pjxzatcg   1   1          0            0          0              0 
      Show
      1. deploy logging stack with fluentd collector 2. create a CLF with: apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata:   name: instance   namespace: openshift-logging spec:   outputDefaults:     elasticsearch:       enableStructuredContainerLogs: true       structuredTypeKey: kubernetes.namespace_name       structuredTypeName: test-name   pipelines:   - inputRefs:     - application     name: forward     outputRefs:     - default     parse: json 3. create some pods to generate json format logs, then add annotation to pods, for example: oc annotate pod centos-logtest-gbt8m containerType.logging.openshift.io/centos-logtest=container-log-centos-logtest -n test-1 4. check indices in ES, the index name is app-$project-name, not app-container-log-centos-logtest: $ oc exec elasticsearch-cdm-slb4l0jd-1-749984d987-rknr8  -- indices Defaulted container "elasticsearch" out of: elasticsearch, proxy Sun Apr 24 08:30:47 UTC 2022 health status index             uuid                   pri rep docs.count docs.deleted store.size pri.store.size green  open   infra-000001      Uit3BxfJT6SnaNA-MtDciQ   3   1          0            0          0              0 green  open   app-test-1-000001 YMjkNaTlT7OFQ39vCW9M4A   3   1        130            0          0              0 green  open   .security         9BwDwv_5TSGVm6gITq8b6w   1   1          6            1          0              0 green  open   .kibana_1         kYAdIIvMRtGU15pjxzatcg   1   1          0            0          0              0 green  open   audit-000001      7EaYqfpeRRSrlJBNezLpzw   3   1          0            0          0              0 green  open   app-000001        bZo9_oxvSCqVDK_fjD0HqQ   3   1          0            0          0              0 5. remove all the collector pods and wait for new collector pods to be created, check indices again, then the index app-container-log-centos-logtest can be created: $ oc exec elasticsearch-cdm-slb4l0jd-1-749984d987-rknr8  -- indices Defaulted container "elasticsearch" out of: elasticsearch, proxy Sun Apr 24 08:32:12 UTC 2022 health status index                                   uuid                   pri rep docs.count docs.deleted store.size pri.store.size green  open   audit-000001                            7EaYqfpeRRSrlJBNezLpzw   3   1          0            0          0              0 green  open   app-000001                              bZo9_oxvSCqVDK_fjD0HqQ   3   1          0            0          0              0 green  open   infra-000001                            Uit3BxfJT6SnaNA-MtDciQ   3   1          0            0          0              0 green  open   app-test-1-000001                       YMjkNaTlT7OFQ39vCW9M4A   3   1        163            0          0              0 green  open   app-container-log-centos-logtest-000001 GchIkq9MRmukYPWKlozM_Q   3   1         53            0          0              0 green  open   .security                               9BwDwv_5TSGVm6gITq8b6w   1   1          6            1          0              0 green  open   .kibana_1                               kYAdIIvMRtGU15pjxzatcg   1   1          0            0          0              0
    • Logging (Core) - Sprint 218

      When set `enableStructuredContainerLogs: true` in CLF, the container's logs are not sent to separate index if the annotation `containerType.logging.openshift.io/centos-logtest: xxxxxx` is added after the pod is ready.

      Workaround: restarting the collector pods.

      Test images:

      quay.io/openshift-logging/cluster-logging-operator@sha256:c15a6183d57be88e70e4eae9c1bfb3c0510b400e2845ad7bcbbe6f4b97a3a2fa

      quay.io/openshift-logging/fluentd@sha256:aef0241ee0ac0b82a32c09b33b121883a7ce3ee3f50e174275337105a7b35d38

            jcantril@redhat.com Jeffrey Cantrill
            qitang@redhat.com Qiaoling Tang
            Qiaoling Tang Qiaoling Tang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: