Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-2972

ElasticsearchError error="400 - Rejected by Elasticsearch" when adding some labels in application namespaces

    XMLWordPrintable

Details

    • False
    • None
    • False
    • Logging
    • NEW
    • VERIFIED
    • Before this change, Elasticsearch index templates defined the fields for labels with the wrong types. This change updates those templates to match the expected types forwarded by the log collector
    • Medium
    • Hide

      1) Deploy OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator with 5.4.4 version

      2) Create namespace and add labels, we tested with these labels:
      2.1) Adding two labels to the namespace that begin with "app"

      $ oc label namespace <namespace> "app=config-server"
      $ oc label namespace <namespace> "app.kubernetes.io/instance=config-server-uat"
      

      2.2)The same as 2.1 but with other two labels

      $ oc label namespace <namespace> "app.test=test"
      $ oc label namespace <namespace> "application.test=test"
      

      Remarkable that it does not happen with all the labels, these tests worked properly:
      2.3)

      $ oc label namespace <namespace> "test=test"
      

      2.4)

      $ oc label namespace <namespace> "kubernetes.io/metadata.name=test"
      

      2.5)

      $ oc label namespace <namespace> "app_.test=test"
      

      3) Deploy an application inside the <namespace>, in my case:

      $ oc new-app rails-postgresql-example
      

      4) error="400 - Rejected by Elasticsearch" is seen from collector pod that belong to the node where the application in <namespace> is deployed.

       

      Show
      1) Deploy OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator with 5.4.4 version 2) Create namespace and add labels, we tested with these labels: 2.1) Adding two labels to the namespace that begin with "app" $ oc label namespace <namespace> "app=config-server" $ oc label namespace <namespace> "app.kubernetes.io/instance=config-server-uat" 2.2)The same as 2.1 but with other two labels $ oc label namespace <namespace> "app.test=test" $ oc label namespace <namespace> "application.test=test" Remarkable that it does not happen with all the labels, these tests worked properly: 2.3) $ oc label namespace <namespace> "test=test" 2.4) $ oc label namespace <namespace> "kubernetes.io/metadata.name=test" 2.5) $ oc label namespace <namespace> "app_.test=test" 3) Deploy an application inside the <namespace>, in my case: $ oc new -app rails-postgresql-example 4) error="400 - Rejected by Elasticsearch" is seen from collector pod that belong to the node where the application in <namespace> is deployed.  
    • Log Collection - Sprint 223

    Description

      -RHOL VERSION 5.4.4

      ElasticsearchError error="400 - Rejected by Elasticsearch" appeared in collector pods for application logs when an application namespace has some labels, the main point is that it is not happening with all labels, it seems to be specific with some labels.

      Some examples are in "Steps to Reproduce"

      2022-08-19 08:35:41 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.pods.test_rails-postgresql-example-1-build_7418d148-3eda-4078-b074-c26d53fa01c3.sti-build.0.log" time=2022-08-19 08:35:39.733921100 +0000 record={"@timestamp"=>"2022-08-19T08:35:39.733921100+00:00", "message"=>"Fetching sassc 2.4.0", "docker"=>
      {"container_id"=>"f43bb3760f88d040b286f6d6bbdafab2dd05df66d64a6966aef7b0ff8870f3a8"}
      , "kubernetes"=>{"container_name"=>"sti-build", "namespace_name"=>"test", "pod_name"=>"rails-postgresql-example-1-build", "container_image"=>"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aecd1cb82ff1d78d0c8363a5bbdfaf7a19652e9f4e31c48e63d18a933624655f", "container_image_id"=>"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aecd1cb82ff1d78d0c8363a5bbdfaf7a19652e9f4e31c48e63d18a933624655f", "pod_id"=>"7418d148-3eda-4078-b074-c26d53fa01c3", "pod_ip"=>"10.131.0.100", "host"=>"worker-0.adricluster.lab.pnq2.cee.redhat.com", "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"7a116e5e-a434-49fa-86f4-eb4af2d4c860", "namespace_labels"=>
      {"app"=>"config-server", "app.kubernetes.io/instance"=>"config-server-uat", "kubernetes.io/metadata.name"=>"test"}
      , "flat_labels"=>["openshift.io/build.name=rails-postgresql-example-1"]}, "level"=>"unknown", "hostname"=>"worker-0.adricluster.lab.pnq2.cee.redhat.com", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.74.176.82", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2022-08-19T08:35:39.735747+00:00", "version"=>"1.14.6 1.6.0"}}, "openshift"=>
      {"sequence"=>23904}
      , "viaq_msg_id"=>"ODAwY2NhY2YtOTU5ZC00NmM2LWJjZjctNzFlMzg3MTlhNzAx", "log_type"=>"application", "viaq_index_name"=>"app-write"}
      

       

      Please take into consideration that also a customer started having this problem after the update from RHOL 5.4.3 to RHOL 5.4.4

      Attachments

        Activity

          People

            jcantril@redhat.com Jeffrey Cantrill
            acandelp Adrian Candel
            Qiaoling Tang Qiaoling Tang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: