-
Bug
-
Resolution: Not a Bug
-
Normal
-
None
-
None
-
False
-
None
-
False
-
NEW
-
NEW
I deployed redhat logging operator using
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: cluster-logging namespace: openshift-logging spec: channel: "stable-5.4" name: cluster-logging source: redhat-operators sourceNamespace: openshift-marketplace And deployed cluster logging instance apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: "openshift-logging" spec: managementState: "Managed" collection: logs: type: "fluentd" fluentd: resources: limits: memory: 2Gi requests: cpu: 100m memory: 256Mi And deployed clusterlogforwarder apiVersion: "logging.openshift.io/v1" kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - name: cloudwatch-1 type: cloudwatch cloudwatch: groupBy: logType groupPrefix: hypershif-jan26 region: us-east-1 secret: name: cw-secret - name: rsyslog-1 type: syslog syslog: appName: lhsaas msgID: lh-dev02 procID: rosa facility: user rfc: RFC5424 severity: informational url: 'tcp://syslog-forwarder:9200' pipelines: - name: audit-logs inputRefs: - audit outputRefs: - cloudwatch-1 - rsyslog-1 - name: infra-logs inputRefs: - infrastructure outputRefs: - cloudwatch-1 - name: app-logs inputRefs: - application outputRefs: - cloudwatch-1 And I don't see any logs in cloud watch and I see error in collector pod oc logs collector-x8qkq -c collector Setting each total_size_limit for 2 buffers to 24118059110 bytes Setting queued_chunks_limit_size for each buffer to 2875 Setting chunk_limit_size for each buffer to 8388608 2023-01-27 19:55:22 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp /usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/elasticsearch_compat.rb:8: warning: already initialized constant TRANSPORT_CLASS /usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/elasticsearch_compat.rb:3: warning: previous definition of TRANSPORT_CLASS was here /usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/elasticsearch_compat.rb:25: warning: already initialized constant SELECTOR_CLASS /usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/elasticsearch_compat.rb:20: warning: previous definition of SELECTOR_CLASS was here 2023-01-27 19:55:24 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified 2023-01-27 19:56:25 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/header.rb:22:in `count' 2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/header.rb:22:in `block in initialize_http_header' 2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/header.rb:16:in `each' 2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/header.rb:16:in `initialize_http_header' 2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/generic_request.rb:46:in `initialize' 2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/request.rb:15:in `initialize' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:153:in `new' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:153:in `build_net_request' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:79:in `block in transmit' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:133:in `block in session' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/connection_pool.rb:104:in `session_for' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:128:in `session' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:76:in `transmit' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:50:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/content_length.rb:24:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/request_callback.rb:85:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/json/error_handler.rb:10:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/signature_v4.rb:58:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:355:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/http_checksum.rb:19:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:136:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/json/handler.rb:13:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:80:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/endpoint.rb:47:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/logging.rb:41:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/request_callback.rb:71:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/response_target.rb:24:in `call' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/request.rb:72:in `send_request' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-cloudwatchlogs-1.52.0/lib/aws-sdk-cloudwatchlogs/client.rb:1033:in `describe_log_groups' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:539:in `check_log_group_existence' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:530:in `log_group_exists?' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:243:in `block in write' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:235:in `each' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:235:in `write' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:1179:in `try_flush' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:1495:in `flush_thread_run' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start' 2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.5/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2023-01-27 19:56:25 +0000 [warn]: disable_chunk_backup is true. 5f3443f1ecddac2f2562167036cb51a9 chunk is thrown away 2023-01-27 19:57:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 19:57:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 19:57:30 +0000 [warn]: disable_chunk_backup is true. 5f34442b8fb70019ab476fab8785c591 chunk is thrown away 2023-01-27 19:58:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 19:58:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 19:58:30 +0000 [warn]: disable_chunk_backup is true. 5f3444691d8d2a72152a5acf76dabf28 chunk is thrown away 2023-01-27 19:59:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 19:59:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 19:59:30 +0000 [warn]: disable_chunk_backup is true. 5f3444a25a9a787cbb4736dfac27b80a chunk is thrown away 2023-01-27 20:00:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 20:00:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:00:30 +0000 [warn]: disable_chunk_backup is true. 5f3444db92f46697c10f82b564ebb9ce chunk is thrown away 2023-01-27 20:01:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 20:01:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:01:30 +0000 [warn]: disable_chunk_backup is true. 5f344514cb73ab157a8e4c87d4df4053 chunk is thrown away 2023-01-27 20:02:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 20:02:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:02:30 +0000 [warn]: disable_chunk_backup is true. 5f34454e03efcb12b9623af13490e727 chunk is thrown away 2023-01-27 20:03:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 20:03:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:03:30 +0000 [warn]: disable_chunk_backup is true. 5f34458738ca6088ede276460b3113f3 chunk is thrown away 2023-01-27 20:04:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 20:04:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:04:30 +0000 [warn]: disable_chunk_backup is true. 5f3445c074ca15e61f08c34aed8ca42f chunk is thrown away 2023-01-27 20:05:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 20:05:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:05:30 +0000 [warn]: disable_chunk_backup is true. 5f3445f9ad0b7393068b76a283ebd3ff chunk is thrown away 2023-01-27 20:06:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8" 2023-01-27 20:06:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:06:30 +0000 [warn]: disable_chunk_backup is true. 5f344632e5a6ae264e9ae6ca5c2a9772 chunk is thrown away