Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3341

ElasticsearchError error="400 - Rejected by Elasticsearch" when adding some labels in application namespaces

XMLWordPrintable

    • False
    • None
    • False
    • Logging
    • NEW
    • VERIFIED
    • Before this update, records written to Elasticsearch would fail if multiple label keys had the same prefix and some keys included included dots. With this update, underscores replace dots in label keys, resolving the issue.
    • Medium
    • Hide

      1) Create a new project

      $ oc create project adri

      2) Label the namespace

      $ oc label namespace adri "app=config-server"
$ oc label namespace adri "app.kubernetes.io/instance=config-server-uat"
$ oc label namespace adri "app.test=test"

      3) Deploy an application

      $ oc new-app rails-postgresql-example

      4) Check collector logs

      2022-11-18 11:25:08 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.pods.adri_rails-postgresql-example-1-build_bc8495af-acd6-4da3-bede-3d4b47ff99fb.sti-build.0.log" time=2022-11-18 11:25:08.027204286 +0000 record={"@timestamp"=>"2022-11-18T11:25:08.027204286+00:00", "message"=>"Push successful", "docker"=>{"container_id"=>"f822a2fada8223e39e4a6d26dca41ec21e52c10ce39a3146ec9a6e7b68b860ae"}, "kubernetes"=>{"container_name"=>"sti-build", "namespace_name"=>"adri", "pod_name"=>"rails-postgresql-example-1-build", "container_image"=>"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c043c4d68753120a32044f476a8e1977101139cbab3694761ef523da5be8fb0e", "container_image_id"=>"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c043c4d68753120a32044f476a8e1977101139cbab3694761ef523da5be8fb0e", "pod_id"=>"bc8495af-acd6-4da3-bede-3d4b47ff99fb", "pod_ip"=>"10.129.2.80", "host"=>"worker-2.adricluster.lab.psi.pnq2.redhat.com", "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"b2c972e1-05ea-412d-a2e9-8db4bb800c94", "namespace_labels"=>{"app"=>"config-server", "app.kubernetes.io/instance"=>"config-server-uat", "app.test"=>"test", "kubernetes.io/metadata.name"=>"adri", "pod-security.kubernetes.io/audit"=>"restricted", "pod-security.kubernetes.io/audit-version"=>"v1.24", "pod-security.kubernetes.io/warn"=>"restricted", "pod-security.kubernetes.io/warn-version"=>"v1.24"}, "flat_labels"=>["openshift.io/build.name=rails-postgresql-example-1"]}, "level"=>"unknown", "hostname"=>"worker-2.adricluster.lab.psi.pnq2.redhat.com", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.74.215.95", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2022-11-18T11:25:08.028075+00:00", "version"=>"1.14.6 1.6.0"}}, "openshift"=>{"sequence"=>2598}, "viaq_msg_id"=>"NGY0YzNlMmMtNWNjZS00ODZlLWJkNWItYWQ3MzE0OWNkYmRl", "log_type"=>"application", "viaq_index_name"=>"app-write"}

       

      Show
      1) Create a new project $ oc create project adri 2) Label the namespace $ oc label namespace adri "app=config-server" $ oc label namespace adri "app.kubernetes.io/instance=config-server-uat" $ oc label namespace adri "app.test=test" 3) Deploy an application $ oc new -app rails-postgresql-example 4) Check collector logs 2022-11-18 11:25:08 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error= "400 - Rejected by Elasticsearch" location=nil tag= "kubernetes. var .log.pods.adri_rails-postgresql-example-1-build_bc8495af-acd6-4da3-bede-3d4b47ff99fb.sti-build.0.log" time=2022-11-18 11:25:08.027204286 +0000 record={ "@timestamp" => "2022-11-18T11:25:08.027204286+00:00" , "message" => "Push successful" , "docker" =>{ "container_id" => "f822a2fada8223e39e4a6d26dca41ec21e52c10ce39a3146ec9a6e7b68b860ae" }, "kubernetes" =>{ "container_name" => "sti-build" , "namespace_name" => "adri" , "pod_name" => "rails-postgresql-example-1-build" , "container_image" => "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c043c4d68753120a32044f476a8e1977101139cbab3694761ef523da5be8fb0e" , "container_image_id" => "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c043c4d68753120a32044f476a8e1977101139cbab3694761ef523da5be8fb0e" , "pod_id" => "bc8495af-acd6-4da3-bede-3d4b47ff99fb" , "pod_ip" => "10.129.2.80" , "host" => "worker-2.adricluster.lab.psi.pnq2.redhat.com" , "master_url" => "https: //kubernetes. default .svc" , "namespace_id" => "b2c972e1-05ea-412d-a2e9-8db4bb800c94" , "namespace_labels" =>{ "app" => "config-server" , "app.kubernetes.io/instance" => "config-server-uat" , "app.test" => "test" , "kubernetes.io/metadata.name" => "adri" , "pod-security.kubernetes.io/audit" => "restricted" , "pod-security.kubernetes.io/audit-version" => "v1.24" , "pod-security.kubernetes.io/warn" => "restricted" , "pod-security.kubernetes.io/warn-version" => "v1.24" }, "flat_labels" =>[ "openshift.io/build.name=rails-postgresql-example-1" ]}, "level" => "unknown" , "hostname" => "worker-2.adricluster.lab.psi.pnq2.redhat.com" , "pipeline_metadata" =>{ "collector" =>{ "ipaddr4" => "10.74.215.95" , "inputname" => "fluent-plugin-systemd" , "name" => "fluentd" , "received_at" => "2022-11-18T11:25:08.028075+00:00" , "version" => "1.14.6 1.6.0" }}, "openshift" =>{ "sequence" =>2598}, "viaq_msg_id" => "NGY0YzNlMmMtNWNjZS00ODZlLWJkNWItYWQ3MzE0OWNkYmRl" , "log_type" => "application" , "viaq_index_name" => "app-write" }  
    • Log Collection - Sprint 228, Log Collection - Sprint 229, Log Collection - Sprint 230

      Description of problem:

      -RHOL VERSION 5.5.4

      ElasticsearchError error="400 - Rejected by Elasticsearch" appeared in collector pods for application logs when an application namespace has some labels, the main point is that it is not happening with all labels, it seems to be specific with some labels.

      The behavior of this issue is the same to--> https://issues.redhat.com/browse/LOG-2972

       

       

            [LOG-3341] ElasticsearchError error="400 - Rejected by Elasticsearch" when adding some labels in application namespaces

            Ishwar Kanse added a comment -

            Verified fix on cluster-logging.5.5.6 elasticsearch-operator.5.5.6

            Ishwar Kanse added a comment - Verified fix on cluster-logging.5.5.6 elasticsearch-operator.5.5.6

            GitLab CEE Bot added a comment -

            CPaaS Service Account mentioned this issue in merge request !563 of openshift-logging / Log Collection Midstream on branch openshift-logging-5.5-rhel-8_upstream_a59ce66fab17f58ffd92c7e9a4baf644:

            Updated US source to: bab3d7b Merge pull request #1799 from jcantrill/log3341

            GitLab CEE Bot added a comment - CPaaS Service Account mentioned this issue in merge request !563 of openshift-logging / Log Collection Midstream on branch openshift-logging-5.5-rhel-8_ upstream _a59ce66fab17f58ffd92c7e9a4baf644 : Updated US source to: bab3d7b Merge pull request #1799 from jcantrill/log3341

            Jeffrey Cantrill added a comment -

            This workaround isnt viable since labels are the core communication in Kubernetes, so removing them can be dangerous

            I would disagree. Either they want working logging or they don't. How important are their logs? I would further offer that it is my understanding they should be namespacing their labels; I believe this is a recommended kubernetes practice. It is already part of the recommended common labels.

            The issue is a limitation of how Elasticsearch ingests dot delimited fields. Until the fix lands which will replace dots with underscores for namespace labels, users will continue to experience this issue. There are several workarounds identified, the easiest of which are to:

            • remove the offending label
            • add a namespace to the label

            Jeffrey Cantrill added a comment - This workaround isnt viable since labels are the core communication in Kubernetes, so removing them can be dangerous I would disagree. Either they want working logging or they don't. How important are their logs? I would further offer that it is my understanding they should be namespacing their labels; I believe this is a recommended kubernetes practice. It is already part of the recommended common labels. The issue is a limitation of how Elasticsearch ingests dot delimited fields. Until the fix lands which will replace dots with underscores for namespace labels, users will continue to experience this issue. There are several workarounds identified, the easiest of which are to: remove the offending label add a namespace to the label

            Hevellyn Gomes added a comment -

            Hi team,

            This workaround isnt viable since labels are the core communication in Kubernetes, so removing them can be dangerous. Specially when it can happens to any other label/namespace, like in my original issue it was to a different label than "app".  Let me know if any additional data is needed to a more stable workaround.

            Hevellyn Gomes added a comment - Hi team, This workaround isnt viable since labels are the core communication in Kubernetes, so removing them can be dangerous. Specially when it can happens to any other label/namespace, like in my original issue it was to a different label than "app".  Let me know if any additional data is needed to a more stable workaround.

            Hevellyn Gomes added a comment -

            jcantril@redhat.com  i see you closed LOG-3405 as duplicated of this one. I assume you dont need the data asked anymore? Eitherway, let me know if you do.

            Hevellyn Gomes added a comment - jcantril@redhat.com   i see you closed LOG-3405 as duplicated of this one. I assume you dont need the data asked anymore? Eitherway, let me know if you do.

              jcantril@redhat.com Jeffrey Cantrill
              acandelp Adrian Candel
              Ishwar Kanse Ishwar Kanse
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: