Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: Logging 5.4.5
Affects Version/s: Logging 5.4.4
Component/s: Log Collection
Labels:
- devel_ack+

Blocked:
False
Blocked Reason:
None
Ready:
False
Backlogs:

Logging
Docs QE Status:
NEW
QE Status:
VERIFIED
Release Note Text:
Before this change, Elasticsearch index templates defined the fields for labels with the wrong types. This change updates those templates to match the expected types forwarded by the log collector
Risk Impact Level:
Medium
Steps to Reproduce:
Hide

1) Deploy OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator with 5.4.4 version

2) Create namespace and add labels, we tested with these labels:
2.1) Adding two labels to the namespace that begin with "app"

$ oc label namespace <namespace> "app=config-server" $ oc label namespace <namespace> "app.kubernetes.io/instance=config-server-uat"

2.2)The same as 2.1 but with other two labels

$ oc label namespace <namespace> "app.test=test" $ oc label namespace <namespace> "application.test=test"

Remarkable that it does not happen with all the labels, these tests worked properly:
2.3)

$ oc label namespace <namespace> "test=test"

2.4)

$ oc label namespace <namespace> "kubernetes.io/metadata.name=test"

2.5)

$ oc label namespace <namespace> "app_.test=test"

3) Deploy an application inside the <namespace>, in my case:

$ oc new-app rails-postgresql-example

4) error="400 - Rejected by Elasticsearch" is seen from collector pod that belong to the node where the application in <namespace> is deployed.
Show
1) Deploy OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator with 5.4.4 version 2) Create namespace and add labels, we tested with these labels: 2.1) Adding two labels to the namespace that begin with "app" $ oc label namespace <namespace> "app=config-server" $ oc label namespace <namespace> "app.kubernetes.io/instance=config-server-uat" 2.2)The same as 2.1 but with other two labels $ oc label namespace <namespace> "app.test=test" $ oc label namespace <namespace> "application.test=test" Remarkable that it does not happen with all the labels, these tests worked properly: 2.3) $ oc label namespace <namespace> "test=test" 2.4) $ oc label namespace <namespace> "kubernetes.io/metadata.name=test" 2.5) $ oc label namespace <namespace> "app_.test=test" 3) Deploy an application inside the <namespace>, in my case: $ oc new -app rails-postgresql-example 4) error="400 - Rejected by Elasticsearch" is seen from collector pod that belong to the node where the application in <namespace> is deployed.

Sprint:
Log Collection - Sprint 223

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

-RHOL VERSION 5.4.4

ElasticsearchError error="400 - Rejected by Elasticsearch" appeared in collector pods for application logs when an application namespace has some labels, the main point is that it is not happening with all labels, it seems to be specific with some labels.

Some examples are in "Steps to Reproduce"

2022-08-19 08:35:41 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.pods.test_rails-postgresql-example-1-build_7418d148-3eda-4078-b074-c26d53fa01c3.sti-build.0.log" time=2022-08-19 08:35:39.733921100 +0000 record={"@timestamp"=>"2022-08-19T08:35:39.733921100+00:00", "message"=>"Fetching sassc 2.4.0", "docker"=>
{"container_id"=>"f43bb3760f88d040b286f6d6bbdafab2dd05df66d64a6966aef7b0ff8870f3a8"}
, "kubernetes"=>{"container_name"=>"sti-build", "namespace_name"=>"test", "pod_name"=>"rails-postgresql-example-1-build", "container_image"=>"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aecd1cb82ff1d78d0c8363a5bbdfaf7a19652e9f4e31c48e63d18a933624655f", "container_image_id"=>"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aecd1cb82ff1d78d0c8363a5bbdfaf7a19652e9f4e31c48e63d18a933624655f", "pod_id"=>"7418d148-3eda-4078-b074-c26d53fa01c3", "pod_ip"=>"10.131.0.100", "host"=>"worker-0.adricluster.lab.pnq2.cee.redhat.com", "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"7a116e5e-a434-49fa-86f4-eb4af2d4c860", "namespace_labels"=>
{"app"=>"config-server", "app.kubernetes.io/instance"=>"config-server-uat", "kubernetes.io/metadata.name"=>"test"}
, "flat_labels"=>["openshift.io/build.name=rails-postgresql-example-1"]}, "level"=>"unknown", "hostname"=>"worker-0.adricluster.lab.pnq2.cee.redhat.com", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.74.176.82", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2022-08-19T08:35:39.735747+00:00", "version"=>"1.14.6 1.6.0"}}, "openshift"=>
{"sequence"=>23904}
, "viaq_msg_id"=>"ODAwY2NhY2YtOTU5ZC00NmM2LWJjZjctNzFlMzg3MTlhNzAx", "log_type"=>"application", "viaq_index_name"=>"app-write"}