Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4441

No error when using parse json with default without setting structuredTypeKey or structuredTypeName in clusterlogforwarder

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • VERIFIED
    • Hide
      Before these changes, CLO did not inform the user about an invalid configuration when configuring JSON log parser without setting 'structuredTypeKey' or 'structuredTypeName'. With this update, this situation will be checked, and an invalid condition will be set for CLO, alerting the user about the configuration issue.
      Show
      Before these changes, CLO did not inform the user about an invalid configuration when configuring JSON log parser without setting 'structuredTypeKey' or 'structuredTypeName'. With this update, this situation will be checked, and an invalid condition will be set for CLO, alerting the user about the configuration issue.
    • Bug Fix
    • Log Collection - Sprint 243, Log Collection - Sprint 244
    • Moderate

      Description of problem:

      When in the clusterLogForwarder is set parse: json using the default outputRefs and without setting outputDefaults.elasticsearch.structuredTypeKey or outputDefaults.elasticsearch.structuredTypeName, then, any change done in the clusterlogging CR is not applied and no an error is visible in the clusterLogForwarder. It's only visible in the clusterLogging pod something like:

      $ clo=$(oc get pod -l name=cluster-logging-operator -n openshift-logging -o name)
      $ oc logs $clo -n openshift-logging  |grep -i "structuredTypeKey or structuredTypeName must be defined for Elasticsearch output"  |tail -1 
      {"_ts":"2023-08-14T14:27:27.865948363Z","_level":"0","_component":"cluster-logging-operator","_message":"Error reconciling clusterlogging instance","_error":{"msg":"structuredTypeKey or structuredTypeName must be defined for Elasticsearch output named \"default\" when JSON parsing is enabled on pipeline \"container-logs\" that references it"}}

      This could lead that the change applied is correct since no error in the clusterLogForwarder visible:

      $ oc get clusterlogforwarder instance -o yaml -n openshift-logging
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        creationTimestamp: "2023-08-14T14:13:39Z"
        generation: 2
        name: instance
        namespace: openshift-logging
        resourceVersion: "335236"
        uid: 066a36a2-b104-43a5-a4d1-7c8fe39184c1
      spec:
        pipelines:
        - inputRefs:
          - application
          - infrastructure
          - audit
          name: container-logs
          outputRefs:
          - default
          parse: json
      status:
        conditions:
        - lastTransitionTime: "2023-08-14T14:27:01Z"
          status: "True"
          type: Ready
        outputs:
          default:
          - lastTransitionTime: "2023-08-14T14:27:01Z"
            status: "True"
            type: Ready
        pipelines:
          container-logs:
          - lastTransitionTime: "2023-08-14T14:27:01Z"
            status: "True"
            type: Ready 

      It should be expected that in the the CLF status is showed an error indicating that the pipeline or configuration created is not correct as it shows in other situations or even not allowing to apply a wrong configuration.

      Version-Release number of selected component (if applicable):

      $ oc get csv
      NAME                                       DISPLAY                                          VERSION                   REPLACES                        PHASE
      cluster-logging.v5.7.4                     Red Hat OpenShift Logging                        5.7.4                     cluster-logging.v5.7.3          Succeeded
      elasticsearch-operator.v5.7.4              OpenShift Elasticsearch Operator                 5.7.4                     elasticsearch-operator.v5.7.3   Succeeded 

      How reproducible:

      Always

      Steps to Reproduce:

      1. Create or edit the clusterLogforwarder using a configuration similar to [1]
      2. Review the logs from the clusterlogging operator pod and check that any posterior changes applied to the clusterLogging instance won't be applied since the clusterLogforwarder configuration is wrong and showed the error structuredTypeKey or structuredTypeName must be defined for Elasticsearch output named
      3. Verify that in the clusterlogforwader instance status not errors are visible [3]
      4. Apply any change to the clusterLogging instance and observe is not applied

      Actual results:

      The clusterLogForwarder is not showing any error and also allowing to create a wrong configuration.

      Expected results:

      The clusterlogforwarder shows an error indicating that the configuration is wrong. Added to this, for improving the user experience, really, applying this configuration should be given an error and not allowing it

      Additional info:

       

       [1]

      $ cat clf.yaml 
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        name: instance 
        namespace: openshift-logging 
      spec:
        pipelines: 
         - name: container-logs
           inputRefs:
           - application
           - infrastructure
           - audit
           outputRefs:
           - default
           parse: json
       $ oc create -f clf.yaml 

       [2]

      $ oc logs $clo -n openshift-logging  |grep -i "structuredTypeKey or structuredTypeName must be defined for Elasticsearch output"  |tail -1  {"_ts":"2023-08-14T14:27:27.865948363Z","_level":"0","_component":"cluster-logging-operator","_message":"Error reconciling clusterlogging instance","_error":{"msg":"structuredTypeKey or structuredTypeName must be defined for Elasticsearch output named \"default\" when JSON parsing is enabled on pipeline \"container-logs\" that references it"}}

       [3]

      $ oc get clusterlogforwarder instance  -o yaml 
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        creationTimestamp: "2023-08-14T14:32:26Z"
        generation: 1
        name: instance
        namespace: openshift-logging
        resourceVersion: "340727"
        uid: d6b48f9c-270f-46ed-b6ab-1a10bac5439b
      spec:
        pipelines:
        - inputRefs:
          - application
          - infrastructure
          - audit
          name: container-logs
          outputRefs:
          - default
          parse: json
      status:
        conditions:
        - lastTransitionTime: "2023-08-14T14:38:05Z"
          status: "True"
          type: Ready
        outputs:
          default:
          - lastTransitionTime: "2023-08-14T14:38:05Z"
            status: "True"
            type: Ready
        pipelines:
          container-logs:
          - lastTransitionTime: "2023-08-14T14:38:05Z"
            status: "True"
            type: Ready
      

      [4]

      $ oc get clusterlogging instance -o yaml 
      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance"
        namespace: "openshift-logging"
      spec:   collection:
          logs:
            type: "fluentd"
            fluentd:
              resources: {}
              nodeSelector: 
                test: failure

       

            vparfono Vitalii Parfonov
            rhn-support-ocasalsa Oscar Casal Sanchez
            Kabir Bharti Kabir Bharti
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: