Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4096

error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch"

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Logging
    • NEW
    • NEW
    • Bug Fix
    • Log Collection - Sprint 238, Log Collection - Sprint 239
    • Important

      Description of problem: 

      1) Collector pods are throwing below warning messages.

      2023-05-09T13:00:16.051491905Z 2023-05-09 13:00:16 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.pods.di-XXXXXXXXXX-sit_ol155-XXXXXXXXXX-validate-pin-complexity-XXXXXXXXXX-46jdp_6e39fb53-e139-49ba-8775-f5a8b15375cc.api.0.log" time=2023-05-09 13:00:00.179386825 +0000 record={"@timestamp"=>"2023-05-09T13:00:00.179386825+00:00", "message"=>"
      
      {\"@timestamp\":\"2023-05-09T13:00:00.175Z\",\"message\":\"end of error processing. Status: 400, errorCode: E0002, description: RequestID is missing\",\"logger_name\":\"net.atos.ol.service.routes.MainRoute\",\"thread_name\":\"XNIO-1 task-1\",\"level\":\"INFO\",\"ol.apiRequestOriginator\":\"heartbeat-monitoring\",\"camel.exchangeId\":\"ID-ol155-XXXXXXXXXX-validate-pin-complexity-XXXXXXXXXX-46jdp-XXXXXXXXXX-0-8164\",\"camel.contextId\":\"ol155-ptii36-validate-pin-complexity\",\"camel.messageId\":\"ID-ol155-ptii36-validate-pin-complexity-XXXXXXXXXX-46jdp-XXXXXXXXXX-0-8165\",\"ol.correlation-id\":\"XXXXXXXXXX-320f-473d-87f8-XXXXXXXXXX\",\"camel.breadcrumbId\":\"ID-ol155-ptii36-validate-pin-complexity-XXXXXXXXXX-XXXXXXXXXX-1681187660807-0-8164\",\"camel.routeId\":\"route1\",\"ol.marker\":\"API_FINISHED_WITH_ERROR\",\"ol.responseHttpStatus\":400,\"ol.errorCode\":\"E0002\",\"ol.errorDescription\":\"RequestID is missing\"}
      
      ", "docker"=>\{"container_id"=>"cc25f31bc363630c0eXXXXXXXXXXf4766a3f03e2d364bc4a3XXXXXXXXXX73"}, "kubernetes"=>{"container_name"=>"api", "namespace_name"=>"di-jbsb-sit", "pod_name"=>"ol155-XXXXXXXXXX-validate-pin-complexity-XXXXXXXXXX-46jdp", "container_image"=>"image-registry.openshift-image-registry.svc:5000/jbsb-ci/ol155-ptii36-validate-pin-complexity:1.0.2", "container_image_id"=>"ec2-35-167-9-42.us-west-2.compute.amazonaws.com:5000/ol155-ptii36-validate-pin-complexity@sha256:0c3760XXXXXXXXXXad5c36335ef2c9e20XXXXXXXXXXcb522d32993108722ca80528", "pod_id"=>"6e39fb53-e139-XXXXXXXXXX-8775-XXXXXXXXXX", "pod_ip"=>"10.128.3.107", "host"=>"ip-10-23-218-198.eu-west-2.compute.internal", "labels"=>
      
      {"app.kubernetes.io/instance"=>"orchestration-layer", "app.kubernetes.io/name"=>"ol155-ptii36-validate-pin-complexity"}
      
      , "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"XXXXXXXXXX-5b4e-45f7-XXXXXXXXXX-7d6ce443aa07", "namespace_labels"=>\{"kubernetes_io/metadata_name"=>"di-jbsb-sit", "managed"=>"retail", "openshift-pipelines_tekton_dev/namespace-reconcile-version"=>"1.8.2"}, "flat_labels"=>["app.kubernetes.io/instance=orchestration-layer", "app.kubernetes.io/name=ol155-ptii36-validate-pin-complexity", "pod-template-hash=6dXXXXXXXXXXddbf"]}, "level"=>"info", "hostname"=>"ip-10-XX-218-198.eu-west-2.compute.internal", "pipeline_metadata"=>\{"collector"=>{"ipaddr4"=>"10.XX.218.198", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2023-XX-09T13:00:01.089500+00:00", "version"=>"1.xx.6 1.6.0"}}, "openshift"=>\{"sequence"=>1273699}, "viaq_msg_id"=>"NjI2MjU1XXXXXXXXXX00NTczLTg1NWUtMjkXXXXXXXXXXlOTM4", "log_type"=>"application", "viaq_index_name"=>"app-write"}
      

      2) Current cluster details are below

      • RHCOP Version: 4.10.53
      • Openshift cluster-logging. 5.5.7
      • Openshift elasticsearch-operator.5.5.7

      3) The behavior of this issue is the same to–> https://issues.redhat.com/browse/LOG-3341   --> Which was fixed in Logging 5.5.6

       

      Version-Release number of selected component (if applicable):

      • Logging 5.5.7

      Expected results: 

      • Mentioned error is the description, should not present under collector pods logs.

        1. clf.yaml
          3 kB
          Prithviraj Patil
        2. collectorcm.yaml
          37 kB
          Prithviraj Patil
        3. cr.yaml
          6 kB
          Prithviraj Patil
        4. Rejected_Elasticsearch_logs_13_Jun_23.txt
          6.95 MB
          Prithviraj Patil
        5. ElasticSearch.json
          11 kB
          Prithviraj Patil

              jcantril@redhat.com Jeffrey Cantrill
              rhn-support-pripatil Prithviraj Patil
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: