Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4096

error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch"

XMLWordPrintable

    • False
    • None
    • False
    • Logging
    • NEW
    • NEW
    • Bug Fix
    • Log Collection - Sprint 238, Log Collection - Sprint 239
    • Important

      Description of problem: 

      1) Collector pods are throwing below warning messages.

      2023-05-09T13:00:16.051491905Z 2023-05-09 13:00:16 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.pods.di-jbsb-sit_ol155-ptii36-validate-pin-complexity-6d77d9ddbf-46jdp_6e39fb53-e139-49ba-8775-f5a8b15375cc.api.0.log" time=2023-05-09 13:00:00.179386825 +0000 record={"@timestamp"=>"2023-05-09T13:00:00.179386825+00:00", "message"=>"
      
      {\"@timestamp\":\"2023-05-09T13:00:00.175Z\",\"message\":\"end of error processing. Status: 400, errorCode: E0002, description: RequestID is missing\",\"logger_name\":\"net.atos.ol.service.routes.MainRoute\",\"thread_name\":\"XNIO-1 task-1\",\"level\":\"INFO\",\"ol.apiRequestOriginator\":\"heartbeat-monitoring\",\"camel.exchangeId\":\"ID-ol155-ptii36-validate-pin-complexity-6d77d9ddbf-46jdp-1681187660807-0-8164\",\"camel.contextId\":\"ol155-ptii36-validate-pin-complexity\",\"camel.messageId\":\"ID-ol155-ptii36-validate-pin-complexity-6d77d9ddbf-46jdp-1681187660807-0-8165\",\"ol.correlation-id\":\"dfe2c638-320f-473d-87f8-73dd58d7ea6d\",\"camel.breadcrumbId\":\"ID-ol155-ptii36-validate-pin-complexity-6d77d9ddbf-46jdp-1681187660807-0-8164\",\"camel.routeId\":\"route1\",\"ol.marker\":\"API_FINISHED_WITH_ERROR\",\"ol.responseHttpStatus\":400,\"ol.errorCode\":\"E0002\",\"ol.errorDescription\":\"RequestID is missing\"}
      
      ", "docker"=>\{"container_id"=>"cc25f31bc363630c0ebda0218b19e7b9965f4766a3f03e2d364bc4a3206df673"}, "kubernetes"=>{"container_name"=>"api", "namespace_name"=>"di-jbsb-sit", "pod_name"=>"ol155-ptii36-validate-pin-complexity-6d77d9ddbf-46jdp", "container_image"=>"image-registry.openshift-image-registry.svc:5000/jbsb-ci/ol155-ptii36-validate-pin-complexity:1.0.2", "container_image_id"=>"ec2-35-167-9-42.us-west-2.compute.amazonaws.com:5000/ol155-ptii36-validate-pin-complexity@sha256:0c37603859c77adad5c36335ef2c9e204633560ccb522d32993108722ca80528", "pod_id"=>"6e39fb53-e139-49ba-8775-f5a8b15375cc", "pod_ip"=>"10.128.3.107", "host"=>"ip-10-23-218-198.eu-west-2.compute.internal", "labels"=>
      
      {"app.kubernetes.io/instance"=>"orchestration-layer", "app.kubernetes.io/name"=>"ol155-ptii36-validate-pin-complexity"}
      
      , "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"2444949c-5b4e-45f7-b3a2-7d6ce443aa07", "namespace_labels"=>\{"kubernetes_io/metadata_name"=>"di-jbsb-sit", "managed"=>"retail", "openshift-pipelines_tekton_dev/namespace-reconcile-version"=>"1.8.2"}, "flat_labels"=>["app.kubernetes.io/instance=orchestration-layer", "app.kubernetes.io/name=ol155-ptii36-validate-pin-complexity", "pod-template-hash=6d77d9ddbf"]}, "level"=>"info", "hostname"=>"ip-10-23-218-198.eu-west-2.compute.internal", "pipeline_metadata"=>\{"collector"=>{"ipaddr4"=>"10.23.218.198", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2023-05-09T13:00:01.089500+00:00", "version"=>"1.14.6 1.6.0"}}, "openshift"=>\{"sequence"=>1273699}, "viaq_msg_id"=>"NjI2MjU1ZTgtODJjMi00NTczLTg1NWUtMjk3MTA5MzllOTM4", "log_type"=>"application", "viaq_index_name"=>"app-write"}
      

      2) Current cluster details are below

      • RHCOP Version: 4.10.53
      • Openshift cluster-logging. 5.5.7
      • Openshift elasticsearch-operator.5.5.7

      3) The behavior of this issue is the same to–> https://issues.redhat.com/browse/LOG-3341   --> Which was fixed in Logging 5.5.6

       

      Version-Release number of selected component (if applicable):

      • Logging 5.5.7

      Expected results: 

      • Mentioned error is the description, should not present under collector pods logs.

        1. clf.yaml
          3 kB
        2. collectorcm.yaml
          37 kB
        3. cr.yaml
          6 kB
        4. ElasticSearch.json
          11 kB
        5. must-gather_06_06_2023.tar.gz
          9.88 MB
        6. must-gather_29_05_2023.tar.gz
          28.97 MB
        7. Rejected_Elasticsearch_logs_13_Jun_23.txt
          6.95 MB

            jcantril@redhat.com Jeffrey Cantrill
            rhn-support-pripatil Prithviraj Patil
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: