Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3405

Can't merge a non object mapping [...] with an object mapping

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Normal Normal
    • None
    • None
    • Log Collection
    • False
    • Hide

      None

      Show
      None
    • False
    • NEW
    • NEW
    • Log Collection - Sprint 229

      Description of problem:

       Hi team, customer is facing issues in their ElasticSearch while trying to ingest some logs, error messages can also be seen in the collector pods.

      It happens for different namespaces, being one of them: redhat-rhoam-cloud-operator.

      The type received for the mapping does not match what is there, looks like a bug in the parsing of fluentd logic. Let me know if you need any other information.

      The logging must-gather is attached.

      Important snip from the ES error message:

      java.lang.IllegalArgumentException: Can't merge a non object mapping [kubernetes.namespace_labels.integreatly] with an object mapping [kubernetes.namespace_labels.integreatly] 
      • ES full error message
      [2022-12-08T16:32:33,986][DEBUG][o.e.a.b.TransportShardBulkAction] [elasticsearch-cdm-e3dpz9sp-2] [app-000993][1] failed to execute bulk item (create) index {[app-write][_doc][YjE0YTZiZTktODY1ZS00NmJiLTg0OWItYWEwOTEzYjY3NmEx], source[{"kubernetes":{"container_image_id":"quay.io/integreatly/cloud-resource-operator@sha256:876548ee8714cd1622c99117c648cdd36155d2ac273f0eccb753c28b5a0c003c","container_name":"cloud-resource-operator","namespace_id":"7784ed63-cd08-4626-b361-db3dfba49253","flat_labels":["name=cloud-resource-operator","pod-template-hash=6cb5f7b854"],"pod_ip":"10.243.12.23","host":"ip-10-191-120-151.eu-west-3.compute.internal","master_url":"https://kubernetes.default.svc","pod_id":"034a73ad-9589-4e73-9376-8f4ff2ad2c93","namespace_labels":{"openshift.io/user-monitoring":"false","integreatly.org/installation-uid":"bba2818b-7d76-4354-ba0f-3bb0ddebe3d0","monitoring-key":"middleware","integreatly":"true","openshift-pipelines.tekton.dev/namespace-reconcile-version":"v1.5","kubernetes.io/metadata.name":"redhat-rhoam-cloud-resources-operator"},"container_image":"quay.io/integreatly/cloud-resource-operator:v0.41.0","namespace_name":"redhat-rhoam-cloud-resources-operator","pod_name":"cloud-resource-operator-6cb5f7b854-7cqch"},"viaq_msg_id":"YjE0YTZiZTktODY1ZS00NmJiLTg0OWItYWEwOTEzYjY3NmEx","level":"info","openshift":{"sequence":138001249},"message":"time=\"2022-12-08T16:32:32Z\" level=info msg=\"tags were added successfully to the rds instance spbtskwqredhatrhoamoperatorrhssopos-usce\" action=TagRDSPostgres controller=controller_postgres provider=aws-rds","docker":{"container_id":"3b1b0011bded5d294ffe1c405a5152e9695cff53754c18869ff2f61d4cc70da0"},"hostname":"ip-10-191-120-151.eu-west-3.compute.internal","log_type":"application","@timestamp":"2022-12-08T16:32:32.570097923+00:00","pipeline_metadata":{"collector":{"received_at":"2022-12-08T16:32:32.570517+00:00","name":"fluentd","inputname":"fluent-plugin-systemd","version":"1.14.6 1.6.0","ipaddr4":"10.191.120.151"}}}]}
      java.lang.IllegalArgumentException: Can't merge a non object mapping [kubernetes.namespace_labels.integreatly] with an object mapping [kubernetes.namespace_labels.integreatly] 
      •  Collector error message
      2022-12-08 16:21:56 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.pods.redhat-rhoam-cloud-resources-operator_cloud-resource-operator-6cb5f7b854-7cqch_034a73ad-9589-4e73-9376-8f4ff2ad2c93.cloud-resource-operator.0.log" time=2022-12-08 16:21:55.318488209 +0000 record={"@timestamp"=>"2022-12-08T16:21:55.318488209+00:00", "message"=>"time=\"2022-12-08T16:21:55Z\" level=info msg=\"reconciling Redis\" controller=controller_redis", "docker"=>{"container_id"=>"3b1b0011bded5d294ffe1c405a5152e9695cff53754c18869ff2f61d4cc70da0"}, "kubernetes"=>{"container_name"=>"cloud-resource-operator", "namespace_name"=>"redhat-rhoam-cloud-resources-operator", "pod_name"=>"cloud-resource-operator-6cb5f7b854-7cqch", "container_image"=>"quay.io/integreatly/cloud-resource-operator:v0.41.0", "container_image_id"=>"quay.io/integreatly/cloud-resource-operator@sha256:876548ee8714cd1622c99117c648cdd36155d2ac273f0eccb753c28b5a0c003c", "pod_id"=>"034a73ad-9589-4e73-9376-8f4ff2ad2c93", "pod_ip"=>"10.243.12.23", "host"=>"ip-10-191-120-151.eu-west-3.compute.internal", "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"7784ed63-cd08-4626-b361-db3dfba49253", "namespace_labels"=>{"integreatly"=>"true", "integreatly.org/installation-uid"=>"bba2818b-7d76-4354-ba0f-3bb0ddebe3d0", "kubernetes.io/metadata.name"=>"redhat-rhoam-cloud-resources-operator", "monitoring-key"=>"middleware", "openshift-pipelines.tekton.dev/namespace-reconcile-version"=>"v1.5", "openshift.io/user-monitoring"=>"false"}, "flat_labels"=>["name=cloud-resource-operator", "pod-template-hash=6cb5f7b854"]}, "level"=>"info", "hostname"=>"ip-10-191-120-151.eu-west-3.compute.internal", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.191.120.151", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2022-12-08T16:21:55.319105+00:00", "version"=>"1.14.6 1.6.0"}}, "openshift"=>{"sequence"=>137965487}, "viaq_msg_id"=>"NTYyNjg4ZTItNjU3OC00MDY4LWFhNGUtZjY3YjM2YTE5ZjM1", "log_type"=>"application", "viaq_index_name"=>"app-write"} 

      Version-Release number of selected component (if applicable):

      5.5.4

      Actual results:

      Expected results:

      Additional info:

        1. must-gather.spb-tskwq.2022-12-08T18 42Z.tar.gz
          34.85 MB
          Hevellyn Gomes
        2. mapping.yml
          24 kB
          Hevellyn Gomes
        3. index000993-search.txt
          31 kB
          Hevellyn Gomes

              jcantril@redhat.com Jeffrey Cantrill
              rhn-support-hgomes Hevellyn Gomes
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: