[2022-09-05 07:53:49,959][INFO ][container.run ] Begin Elasticsearch startup script [2022-09-05 07:53:49,963][INFO ][container.run ] Comparing the specified RAM to the maximum recommended for Elasticsearch... [2022-09-05 07:53:49,965][INFO ][container.run ] Inspecting cgroup version... [2022-09-05 07:53:49,966][INFO ][container.run ] Detected cgroup v1 [2022-09-05 07:53:49,968][INFO ][container.run ] Inspecting the maximum RAM available... [2022-09-05 07:53:49,971][INFO ][container.run ] ES_JAVA_OPTS: ' -Xms1024m -Xmx1024m' [2022-09-05 07:53:49,973][INFO ][container.run ] Copying certs from /etc/openshift/elasticsearch/secret to /etc/elasticsearch//secret [2022-09-05 07:53:49,978][INFO ][container.run ] Building required p12 files and truststore Picked up _JAVA_OPTIONS: -Dcom.redhat.fips=false Certificate was added to keystore Picked up _JAVA_OPTIONS: -Dcom.redhat.fips=false Certificate was added to keystore [2022-09-05 07:53:56,273][INFO ][container.run ] Removing unnecessary JKS files [2022-09-05 07:53:56,280][INFO ][container.run ] Setting heap dump location /elasticsearch/persistent/heapdump.hprof [2022-09-05 07:53:56,280][INFO ][container.run ] Checking if Elasticsearch is ready [2022-09-05 07:53:56,282][INFO ][container.run ] ES_JAVA_OPTS: ' -Xms1024m -Xmx1024m -XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof -Xlog:gc*,gc+age=trace,safepoint:file=/elasticsearch/persistent/elasticsearch/logs/gc.log:utctime,pid,tags:filecount=8,filesize=64m -XX:ErrorFile=/elasticsearch/persistent/elasticsearch/logs/error.log -Djdk.tls.ephemeralDHKeySize=2048 -Des.cgroups.hierarchy.override=/ -Dlog4j2.configurationFile=/usr/share/java/elasticsearch/config/log4j2.properties' Picked up _JAVA_OPTIONS: -Dcom.redhat.fips=false Picked up _JAVA_OPTIONS: -Dcom.redhat.fips=false Picked up _JAVA_OPTIONS: -Dcom.redhat.fips=false Picked up _JAVA_OPTIONS: -Dcom.redhat.fips=false OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. OpenJDK 64-Bit Server VM warning: Option MaxRAMFraction was deprecated in version 10.0 and will likely be removed in a future release. OpenJDK 64-Bit Server VM warning: Option InitialRAMFraction was deprecated in version 10.0 and will likely be removed in a future release. OpenJDK 64-Bit Server VM warning: Option MinRAMFraction was deprecated in version 10.0 and will likely be removed in a future release. [2022-09-05T07:53:57,393][WARN ][o.e.c.l.LogConfigurator ] [elasticsearch-cdm-d7ug886s-2] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations: /etc/elasticsearch/log4j2.properties [2022-09-05T07:53:57,586][INFO ][o.e.e.NodeEnvironment ] [elasticsearch-cdm-d7ug886s-2] using [1] data paths, mounts [[/elasticsearch/persistent (/dev/nvme0n1p4)]], net usable_space [110.9gb], net total_space [119.4gb], types [xfs] [2022-09-05T07:53:57,586][INFO ][o.e.e.NodeEnvironment ] [elasticsearch-cdm-d7ug886s-2] heap size [1015.6mb], compressed ordinary object pointers [true] [2022-09-05T07:53:57,588][INFO ][o.e.n.Node ] [elasticsearch-cdm-d7ug886s-2] node name [elasticsearch-cdm-d7ug886s-2], node ID [22rAWvaWTOa2GF463Y7n9g] [2022-09-05T07:53:57,588][INFO ][o.e.n.Node ] [elasticsearch-cdm-d7ug886s-2] version[6.8.1], pid[1], build[oss/zip/3208b92/2022-04-07T06:22:50.624952Z], OS[Linux/4.18.0-305.49.1.el8_4.x86_64/amd64], JVM[Red Hat, Inc./OpenJDK 64-Bit Server VM/11.0.16/11.0.16+8-LTS] [2022-09-05T07:53:57,588][INFO ][o.e.n.Node ] [elasticsearch-cdm-d7ug886s-2] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-12270780524778968772, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -XX:+UnlockExperimentalVMOptions, -XX:MaxRAMFraction=2, -XX:InitialRAMFraction=2, -XX:MinRAMFraction=2, -Xms1024m, -Xmx1024m, -XX:HeapDumpPath=/elasticsearch/persistent/heapdump.hprof, -Xlog:gc*,gc+age=trace,safepoint:file=/elasticsearch/persistent/elasticsearch/logs/gc.log:utctime,pid,tags:filecount=8,filesize=64m, -XX:ErrorFile=/elasticsearch/persistent/elasticsearch/logs/error.log, -Djdk.tls.ephemeralDHKeySize=2048, -Des.cgroups.hierarchy.override=/, -Dlog4j2.configurationFile=/usr/share/java/elasticsearch/config/log4j2.properties, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=oss, -Des.distribution.type=zip, -Dcom.redhat.fips=false] [2022-09-05T07:53:58,201][INFO ][o.e.p.p.PrometheusExporterPlugin] [elasticsearch-cdm-d7ug886s-2] starting Prometheus exporter plugin [2022-09-05T07:53:58,385][INFO ][c.a.o.s.s.t.OpenDistroSSLConfig] [elasticsearch-cdm-d7ug886s-2] SSL dual mode is disabled [2022-09-05T07:53:58,385][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-d7ug886s-2] ES Config path is /etc/elasticsearch [2022-09-05T07:53:58,535][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-d7ug886s-2] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSLContext [2022-09-05T07:53:58,600][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-d7ug886s-2] JVM supports TLSv1.3 [2022-09-05T07:53:58,601][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-d7ug886s-2] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively [2022-09-05T07:53:58,736][INFO ][c.a.o.s.s.u.SSLCertificateHelper] [elasticsearch-cdm-d7ug886s-2] No alias given, use the first one: elasticsearch [2022-09-05T07:53:58,736][WARN ][c.a.o.s.s.u.SSLCertificateHelper] [elasticsearch-cdm-d7ug886s-2] Certificate chain for alias elasticsearch contains a root certificate [2022-09-05T07:53:58,886][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-d7ug886s-2] HTTPS client auth mode OPTIONAL [2022-09-05T07:53:58,929][INFO ][c.a.o.s.s.u.SSLCertificateHelper] [elasticsearch-cdm-d7ug886s-2] No alias given, use the first one: logging-es [2022-09-05T07:53:58,929][WARN ][c.a.o.s.s.u.SSLCertificateHelper] [elasticsearch-cdm-d7ug886s-2] Certificate chain for alias logging-es contains a root certificate [2022-09-05T07:53:59,005][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-d7ug886s-2] TLS Transport Client Provider : JDK [2022-09-05T07:53:59,006][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-d7ug886s-2] TLS Transport Server Provider : JDK [2022-09-05T07:53:59,006][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-d7ug886s-2] TLS HTTP Provider : JDK [2022-09-05T07:53:59,006][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-d7ug886s-2] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2] [2022-09-05T07:53:59,006][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [elasticsearch-cdm-d7ug886s-2] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2] [2022-09-05T07:53:59,097][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-d7ug886s-2] Clustername: elasticsearch [2022-09-05T07:53:59,101][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-d7ug886s-2] Directory /etc/elasticsearch has insecure file permissions (should be 0700) [2022-09-05T07:53:59,101][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-d7ug886s-2] Directory /etc/elasticsearch/scripts has insecure file permissions (should be 0700) [2022-09-05T07:53:59,135][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [aggs-matrix-stats] [2022-09-05T07:53:59,135][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [analysis-common] [2022-09-05T07:53:59,135][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [ingest-common] [2022-09-05T07:53:59,135][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [ingest-user-agent] [2022-09-05T07:53:59,135][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [lang-expression] [2022-09-05T07:53:59,135][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [lang-mustache] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [lang-painless] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [mapper-extras] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [parent-join] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [percolator] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [rank-eval] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [reindex] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [repository-url] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [transport-netty4] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded module [tribe] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded plugin [opendistro_security] [2022-09-05T07:53:59,136][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded plugin [openshift-ingest-plugin] [2022-09-05T07:53:59,137][INFO ][o.e.p.PluginsService ] [elasticsearch-cdm-d7ug886s-2] loaded plugin [prometheus-exporter] [2022-09-05T07:53:59,150][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-d7ug886s-2] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml [2022-09-05T07:54:00,847][WARN ][c.a.o.s.c.Salt ] [elasticsearch-cdm-d7ug886s-2] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes [2022-09-05T07:54:00,861][ERROR][c.a.o.s.a.s.SinkProvider ] [elasticsearch-cdm-d7ug886s-2] Default endpoint could not be created, auditlog will not work properly. [2022-09-05T07:54:00,862][WARN ][c.a.o.s.a.r.AuditMessageRouter] [elasticsearch-cdm-d7ug886s-2] No default storage available, audit log may not work properly. Please check configuration. [2022-09-05T07:54:00,862][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Message routing enabled: false [2022-09-05T07:54:00,918][INFO ][c.a.o.s.f.OpenDistroSecurityFilter] [elasticsearch-cdm-d7ug886s-2] indices are made immutable. [2022-09-05T07:54:00,991][DEBUG][o.e.a.ActionModule ] [elasticsearch-cdm-d7ug886s-2] Using REST wrapper from plugin com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin [2022-09-05T07:54:01,046][INFO ][o.e.d.DiscoveryModule ] [elasticsearch-cdm-d7ug886s-2] using discovery type [zen] and host providers [settings] Registering Handler [2022-09-05T07:54:01,468][INFO ][o.e.n.Node ] [elasticsearch-cdm-d7ug886s-2] initialized [2022-09-05T07:54:01,468][INFO ][o.e.n.Node ] [elasticsearch-cdm-d7ug886s-2] starting ... [2022-09-05T07:54:01,603][INFO ][o.e.t.TransportService ] [elasticsearch-cdm-d7ug886s-2] publish_address {10.131.2.13:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {10.131.2.13:9300} [2022-09-05T07:54:01,610][INFO ][o.e.b.BootstrapChecks ] [elasticsearch-cdm-d7ug886s-2] bound or publishing to a non-loopback address, enforcing bootstrap checks [2022-09-05T07:54:01,619][INFO ][c.a.o.s.c.IndexBaseConfigurationRepository] [elasticsearch-cdm-d7ug886s-2] Will not attempt to create index .security and default configs if they are absent. Use securityadmin to initialize cluster [2022-09-05T07:54:01,625][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [elasticsearch-cdm-d7ug886s-2] no known master node, scheduling a retry [2022-09-05T07:54:05,373][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-cdm-d7ug886s-2] detected_master {elasticsearch-cdm-d7ug886s-1}{CtlM22IbSTiLhsshHJwYPA}{8oZQaxcPTVqtF8V0OC8rEQ}{10.128.4.11}{10.128.4.11:9300}, added {{elasticsearch-cdm-d7ug886s-1}{CtlM22IbSTiLhsshHJwYPA}{8oZQaxcPTVqtF8V0OC8rEQ}{10.128.4.11}{10.128.4.11:9300},{elasticsearch-cdm-d7ug886s-3}{nCbE1yZ1TSKatMKGn37a5A}{-JoiKmYeRMyjTKwhMG492w}{10.130.2.28}{10.130.2.28:9300},}, reason: apply cluster state (from master [master {elasticsearch-cdm-d7ug886s-1}{CtlM22IbSTiLhsshHJwYPA}{8oZQaxcPTVqtF8V0OC8rEQ}{10.128.4.11}{10.128.4.11:9300} committed version [2]]) [2022-09-05T07:54:05,678][INFO ][o.e.h.n.Netty4HttpServerTransport] [elasticsearch-cdm-d7ug886s-2] publish_address {10.131.2.13:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {10.131.2.13:9200} [2022-09-05T07:54:05,678][INFO ][o.e.n.Node ] [elasticsearch-cdm-d7ug886s-2] started [2022-09-05T07:54:05,679][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-cdm-d7ug886s-2] 3 Open Distro Security modules loaded so far: [Module [type=MULTITENANCY, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.amazon.opendistroforelasticsearch.security.dlic.rest.api.OpenDistroSecurityRestApiActions], Module [type=AUDITLOG, implementing class=com.amazon.opendistroforelasticsearch.security.auditlog.impl.AuditLogImpl]] [2022-09-05 07:54:06,463][INFO ][container.run ] Elasticsearch is ready and listening /usr/share/elasticsearch/init ~ [2022-09-05 07:54:06,472][INFO ][container.run ] Starting init script: 0001-jaeger [2022-09-05 07:54:06,475][INFO ][container.run ] Completed init script: 0001-jaeger [2022-09-05 07:54:06,526][INFO ][container.run ] Forcing the seeding of ACL documents [2022-09-05 07:54:06,605][INFO ][container.run ] Seeding the security ACL index. Will wait up to 604800 seconds. [2022-09-05 07:54:06,607][INFO ][container.run ] Seeding the security ACL index. Will wait up to 604800 seconds. Picked up _JAVA_OPTIONS: -Dcom.redhat.fips=false Open Distro Security Admin v6 Will connect to localhost:9300 ... done Elasticsearch Version: 6.8.1 Open Distro Security Version: 0.10.1.2 Connected as CN=system.admin,OU=OpenShift,O=Logging Contacting elasticsearch cluster 'elasticsearch' ... Clustername: elasticsearch Clusterstate: GREEN Number of nodes: 3 Number of data nodes: 3 .security index does not exists, attempt to create it ... ERR: An unexpected ResourceAlreadyExistsException occured: index [.security/sZgCQMAZS2Ck0-43pvsd9g] already exists Trace: [.security/sZgCQMAZS2Ck0-43pvsd9g] ResourceAlreadyExistsException[index [.security/sZgCQMAZS2Ck0-43pvsd9g] already exists] at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexName(MetaDataCreateIndexService.java:150) at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validate(MetaDataCreateIndexService.java:598) at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:289) at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:270) at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:200) at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:135) at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:829) [2022-09-05T07:54:09,640][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for config while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:09,641][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for roles while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:09,641][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for rolesmapping while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:09,641][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for internalusers while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:09,641][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for actiongroups while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:09,641][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for nodesdn while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:09,641][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for audit while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) 0 [2022-09-05 07:54:09,662][WARN ][container.run ] Error seeding the security ACL index... retrying in 10 seconds - 0 retries so far [2022-09-05 07:54:09,664][WARN ][container.run ] Seeding will continue to fail until the cluster status is YELLOW [2022-09-05 07:54:09,724][INFO ][container.run ] Remaining red indices: 0 [2022-09-05T07:54:14,648][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for audit while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:14,666][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on REST API is enabled. [2022-09-05T07:54:14,666][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on Transport API is enabled. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of request body is enabled. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Bulk requests resolution is disabled during request auditing. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Index resolution is enabled during request auditing. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Sensitive headers auditing is enabled. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing requests from kibanaserver users is disabled. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of external configuration is disabled. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of internal configuration is disabled. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for read request is disabled. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch {} for read requests. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing read operation requests from kibanaserver users is disabled. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for write request is disabled. [2022-09-05T07:54:14,667][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing diffs for write requests is disabled. [2022-09-05T07:54:14,668][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing write operation requests from kibanaserver users is disabled. [2022-09-05T07:54:14,668][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch for write requests. [2022-09-05T07:54:14,668][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] .security is used as internal security index. [2022-09-05T07:54:14,668][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Internal index used for posting audit logs is null [2022-09-05T07:54:14,681][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for audit while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on REST API is enabled. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on Transport API is enabled. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of request body is enabled. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Bulk requests resolution is disabled during request auditing. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Index resolution is enabled during request auditing. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Sensitive headers auditing is enabled. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing requests from kibanaserver users is disabled. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of external configuration is disabled. [2022-09-05T07:54:14,688][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of internal configuration is disabled. [2022-09-05T07:54:14,689][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for read request is disabled. [2022-09-05T07:54:14,689][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch {} for read requests. [2022-09-05T07:54:14,689][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing read operation requests from kibanaserver users is disabled. [2022-09-05T07:54:14,689][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for write request is disabled. [2022-09-05T07:54:14,689][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing diffs for write requests is disabled. [2022-09-05T07:54:14,689][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing write operation requests from kibanaserver users is disabled. [2022-09-05T07:54:14,689][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch for write requests. [2022-09-05T07:54:14,689][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] .security is used as internal security index. [2022-09-05T07:54:14,689][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Internal index used for posting audit logs is null [2022-09-05T07:54:17,644][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for audit while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on REST API is enabled. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on Transport API is enabled. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of request body is enabled. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Bulk requests resolution is disabled during request auditing. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Index resolution is enabled during request auditing. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Sensitive headers auditing is enabled. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing requests from kibanaserver users is disabled. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of external configuration is disabled. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of internal configuration is disabled. [2022-09-05T07:54:17,649][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for read request is disabled. [2022-09-05T07:54:17,650][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch {} for read requests. [2022-09-05T07:54:17,650][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing read operation requests from kibanaserver users is disabled. [2022-09-05T07:54:17,650][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for write request is disabled. [2022-09-05T07:54:17,650][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing diffs for write requests is disabled. [2022-09-05T07:54:17,650][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing write operation requests from kibanaserver users is disabled. [2022-09-05T07:54:17,650][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch for write requests. [2022-09-05T07:54:17,650][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] .security is used as internal security index. [2022-09-05T07:54:17,650][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Internal index used for posting audit logs is null [2022-09-05T07:54:17,676][INFO ][c.a.o.s.c.IndexBaseConfigurationRepository] [elasticsearch-cdm-d7ug886s-2] Hot-reloading of audit configuration is disabled. Using configuration with defaults from elasticsearch settings. Populate the configuration in index using audit.yml or securityadmin to enable it. [2022-09-05T07:54:17,676][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on REST API is enabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on Transport API is enabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of request body is enabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Bulk requests resolution is disabled during request auditing. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Index resolution is enabled during request auditing. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Sensitive headers auditing is enabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing requests from kibanaserver users is disabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of external configuration is disabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of internal configuration is disabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for read request is disabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch {} for read requests. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing read operation requests from kibanaserver users is disabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for write request is disabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing diffs for write requests is disabled. [2022-09-05T07:54:17,677][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing write operation requests from kibanaserver users is disabled. [2022-09-05T07:54:17,678][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch for write requests. [2022-09-05T07:54:17,678][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] .security is used as internal security index. [2022-09-05T07:54:17,678][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Internal index used for posting audit logs is null [2022-09-05T07:54:17,678][INFO ][c.a.o.s.c.IndexBaseConfigurationRepository] [elasticsearch-cdm-d7ug886s-2] Node 'elasticsearch-cdm-d7ug886s-2' initialized Picked up _JAVA_OPTIONS: -Dcom.redhat.fips=false Open Distro Security Admin v6 Will connect to localhost:9300 ... done Elasticsearch Version: 6.8.1 Open Distro Security Version: 0.10.1.2 Connected as CN=system.admin,OU=OpenShift,O=Logging Contacting elasticsearch cluster 'elasticsearch' ... Clustername: elasticsearch Clusterstate: GREEN Number of nodes: 3 Number of data nodes: 3 .security index already exists, so we do not need to create one. Populate config from /opt/app-root/src/sgconfig/ Will update 'security/config' with /opt/app-root/src/sgconfig/config.yml SUCC: Configuration for 'config' created or updated Will update 'security/roles' with /opt/app-root/src/sgconfig/roles.yml SUCC: Configuration for 'roles' created or updated Will update 'security/rolesmapping' with /opt/app-root/src/sgconfig/roles_mapping.yml SUCC: Configuration for 'rolesmapping' created or updated Will update 'security/internalusers' with /opt/app-root/src/sgconfig/internal_users.yml SUCC: Configuration for 'internalusers' created or updated Will update 'security/actiongroups' with /opt/app-root/src/sgconfig/action_groups.yml SUCC: Configuration for 'actiongroups' created or updated Will update 'security/nodesdn' with /opt/app-root/src/sgconfig/nodes_dn.yml SUCC: Configuration for 'nodesdn' created or updated [2022-09-05T07:54:23,471][WARN ][c.a.o.s.c.ConfigurationLoader] [elasticsearch-cdm-d7ug886s-2] No data for audit while retrieving configuration for [config, roles, rolesmapping, internalusers, actiongroups, nodesdn, audit] (index=.security) [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on REST API is enabled. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from REST API auditing. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing on Transport API is enabled. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] [AUTHENTICATED, GRANTED_PRIVILEGES] are excluded from Transport API auditing. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of request body is enabled. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Bulk requests resolution is disabled during request auditing. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Index resolution is enabled during request auditing. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Sensitive headers auditing is enabled. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing requests from kibanaserver users is disabled. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of external configuration is disabled. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing of internal configuration is disabled. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for read request is disabled. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch {} for read requests. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing read operation requests from kibanaserver users is disabled. [2022-09-05T07:54:23,476][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing only metadata information for write request is disabled. [2022-09-05T07:54:23,477][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing diffs for write requests is disabled. [2022-09-05T07:54:23,477][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing write operation requests from kibanaserver users is disabled. [2022-09-05T07:54:23,477][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Auditing will watch for write requests. [2022-09-05T07:54:23,477][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] .security is used as internal security index. [2022-09-05T07:54:23,477][INFO ][c.a.o.s.a.i.AuditLogImpl ] [elasticsearch-cdm-d7ug886s-2] Internal index used for posting audit logs is null Done with success [2022-09-05 07:54:23,515][INFO ][container.run ] Seeded the security ACL index [2022-09-05 07:54:23,519][INFO ][container.run ] Adding ingest pipelines [2022-09-05 07:54:23,573][INFO ][container.run ] Create ingest pipeline 'openshift_schema.json' {"acknowledged":true}[2022-09-05 07:54:23,621][INFO ][container.run ] Finished adding ingest pipelines [2022-09-05 07:54:23,625][INFO ][container.run ] Deleting deprecated index templates [2022-09-05 07:54:23,675][INFO ][container.run ] Finished deleting deprecated index templates [2022-09-05 07:54:23,677][INFO ][container.run ] Adding index templates [2022-09-05 07:54:23,725][INFO ][container.run ] Index template 'com.redhat.viaq-openshift-operations.template.json' found in the cluster, overriding it {"acknowledged":true}[2022-09-05 07:54:23,840][INFO ][container.run ] Index template 'com.redhat.viaq-openshift-project.template.json' found in the cluster, overriding it {"acknowledged":true}[2022-09-05 07:54:23,969][INFO ][container.run ] Index template 'common.settings.kibana.template.json' found in the cluster, overriding it {"acknowledged":true}[2022-09-05 07:54:24,094][INFO ][container.run ] Index template 'common.settings.operations.template.json' found in the cluster, overriding it {"acknowledged":true}[2022-09-05 07:54:24,217][INFO ][container.run ] Index template 'common.settings.project.template.json' found in the cluster, overriding it {"acknowledged":true}[2022-09-05 07:54:24,337][INFO ][container.run ] Index template 'jaeger-service.json' found in the cluster, overriding it {"acknowledged":true}[2022-09-05 07:54:24,456][INFO ][container.run ] Index template 'jaeger-span.json' found in the cluster, overriding it {"acknowledged":true}[2022-09-05 07:54:24,541][INFO ][container.run ] Finished adding index templates ~ [2022-09-05T07:55:33,261][INFO ][c.a.o.s.f.OpenDistroSecurityFilter] [elasticsearch-cdm-d7ug886s-2] Request IndexRequest requires new tenant index .kibana_-377444158_kubeadmin_1 with aliases [.kibana_-377444158_kubeadmin] [2022-09-05T07:56:53,993][DEBUG][o.e.a.b.TransportShardBulkAction] [elasticsearch-cdm-d7ug886s-2] [app-nologformat-000001][2] failed to execute bulk item (create) index {[app-nologformat-000001][_doc][MmQ0Y2NiNmYtOWQxNy00ZDBmLWExNWUtN2JkNzI0MTkzNWU4], source[{"kubernetes":{"container_image_id":"registry.fedoraproject.org/fedora@sha256:63fa666e22333d85eaa6684fba51efe4b48dc5f5ff7b194cdc1fa1f1290b5c32","container_name":"fedora","namespace_id":"f9ba845c-65c3-4004-a97b-11c47fe6d050","pod_ip":"10.128.4.15","host":"ip-10-0-205-56.eu-central-1.compute.internal","master_url":"https://kubernetes.default.svc","pod_id":"abbcf9ca-37b5-4d01-a5eb-3b5e6511f849","namespace_labels":{"kubernetes.io/metadata.name":"simon-reproducer"},"container_image":"registry.fedoraproject.org/fedora@sha256:63fa666e22333d85eaa6684fba51efe4b48dc5f5ff7b194cdc1fa1f1290b5c32","namespace_name":"simon-reproducer","pod_name":"fedora-log-test"},"viaq_msg_id":"MmQ0Y2NiNmYtOWQxNy00ZDBmLWExNWUtN2JkNzI0MTkzNWU4","level":"unknown","openshift":{"sequence":392},"message":"{\"test\":1234}","docker":{"container_id":"17c683704791faaf79067ad656d23bbb6a68c7be965383c6c6ac0641a879e686"},"hostname":"ip-10-0-205-56.eu-central-1.compute.internal","log_type":"application","@timestamp":"2022-09-05T07:56:48.303796198+00:00","pipeline_metadata":{"collector":{"received_at":"2022-09-05T07:56:52.793598+00:00","name":"fluentd","inputname":"fluent-plugin-systemd","version":"1.14.6 1.6.0","ipaddr4":"10.0.205.56"}},"structured":{"test":1234}}]} java.lang.IllegalArgumentException: mapper [structured] of different type, current_type [long], merged_type [ObjectMapper] at org.elasticsearch.index.mapper.FieldMapper.doMerge(FieldMapper.java:354) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.index.mapper.NumberFieldMapper.doMerge(NumberFieldMapper.java:1093) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:340) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.index.mapper.FieldMapper.merge(FieldMapper.java:52) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:487) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:278) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:457) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:273) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.index.mapper.Mapping.merge(Mapping.java:91) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.index.mapper.DocumentMapper.merge(DocumentMapper.java:339) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:273) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:231) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:270) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:200) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:135) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-6.8.1.jar:6.8.1.redhat-00020] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?]