Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3575

In Hypershift clusterlogforwarder is not working.

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • NEW

      I deployed redhat logging operator using 

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: cluster-logging
        namespace: openshift-logging
      spec:
        channel: "stable-5.4"
        name: cluster-logging
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      
      And deployed cluster logging instance 
      
      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance"
        namespace: "openshift-logging"
      spec:
        managementState: "Managed"
        collection:
          logs:
            type: "fluentd"
            fluentd:
              resources:
                limits:
                  memory: 2Gi
                requests:
                  cpu: 100m
                  memory: 256Mi
      
      And deployed clusterlogforwarder
      
      apiVersion: "logging.openshift.io/v1"
      kind: ClusterLogForwarder
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        outputs:
          - name: cloudwatch-1
            type: cloudwatch
            cloudwatch:
              groupBy: logType
              groupPrefix: hypershif-jan26
              region: us-east-1
            secret:
              name: cw-secret
          - name: rsyslog-1
            type: syslog
            syslog:
              appName: lhsaas
              msgID: lh-dev02
              procID: rosa
              facility: user
              rfc: RFC5424
              severity: informational
            url: 'tcp://syslog-forwarder:9200'
        pipelines:
          - name: audit-logs
            inputRefs:
              - audit
            outputRefs:
              - cloudwatch-1
              - rsyslog-1
          - name: infra-logs
            inputRefs:
              - infrastructure
            outputRefs:
              - cloudwatch-1
          - name: app-logs
            inputRefs:
              - application
            outputRefs:
              - cloudwatch-1
      And I don't see any logs in cloud watch and I see error in collector pod
      
      
      oc logs collector-x8qkq  -c collector  Setting each total_size_limit for 2 buffers to 24118059110 bytes Setting queued_chunks_limit_size for each buffer to 2875 Setting chunk_limit_size for each buffer to 8388608 2023-01-27 19:55:22 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp /usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/elasticsearch_compat.rb:8: warning: already initialized constant TRANSPORT_CLASS /usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/elasticsearch_compat.rb:3: warning: previous definition of TRANSPORT_CLASS was here /usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/elasticsearch_compat.rb:25: warning: already initialized constant SELECTOR_CLASS /usr/local/share/gems/gems/fluent-plugin-elasticsearch-5.2.1/lib/fluent/plugin/elasticsearch_compat.rb:20: warning: previous definition of SELECTOR_CLASS was here 2023-01-27 19:55:24 +0000 [warn]: For security reason, setting private_key_passphrase is recommended when cert_path is specified 2023-01-27 19:56:25 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/header.rb:22:in `count'   2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/header.rb:22:in `block in initialize_http_header'   2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/header.rb:16:in `each'   2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/header.rb:16:in `initialize_http_header'   2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/generic_request.rb:46:in `initialize'   2023-01-27 19:56:25 +0000 [warn]: /usr/share/ruby/net/http/request.rb:15:in `initialize'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:153:in `new'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:153:in `build_net_request'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:79:in `block in transmit'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:133:in `block in session'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/connection_pool.rb:104:in `session_for'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:128:in `session'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:76:in `transmit'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/net_http/handler.rb:50:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/content_length.rb:24:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/request_callback.rb:85:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/json/error_handler.rb:10:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/signature_v4.rb:58:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/transfer_encoding.rb:26:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/retry_errors.rb:355:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/http_checksum.rb:19:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/endpoint_pattern.rb:30:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:136:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/json/handler.rb:13:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/recursion_detection.rb:18:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:80:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/endpoint.rb:47:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/param_validator.rb:26:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/logging.rb:41:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/raise_response_errors.rb:16:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:22:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/request_callback.rb:71:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/plugins/response_target.rb:24:in `call'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-core-3.130.0/lib/seahorse/client/request.rb:72:in `send_request'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/aws-sdk-cloudwatchlogs-1.52.0/lib/aws-sdk-cloudwatchlogs/client.rb:1033:in `describe_log_groups'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:539:in `check_log_group_existence'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:530:in `log_group_exists?'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:243:in `block in write'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:235:in `each'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:235:in `write'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:1179:in `try_flush'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:1495:in `flush_thread_run'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.5/lib/fluent/plugin/output.rb:499:in `block (2 levels) in start'   2023-01-27 19:56:25 +0000 [warn]: /usr/local/share/gems/gems/fluentd-1.14.5/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2023-01-27 19:56:25 +0000 [warn]: disable_chunk_backup is true. 5f3443f1ecddac2f2562167036cb51a9 chunk is thrown away 2023-01-27 19:57:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 19:57:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 19:57:30 +0000 [warn]: disable_chunk_backup is true. 5f34442b8fb70019ab476fab8785c591 chunk is thrown away 2023-01-27 19:58:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 19:58:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 19:58:30 +0000 [warn]: disable_chunk_backup is true. 5f3444691d8d2a72152a5acf76dabf28 chunk is thrown away 2023-01-27 19:59:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 19:59:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 19:59:30 +0000 [warn]: disable_chunk_backup is true. 5f3444a25a9a787cbb4736dfac27b80a chunk is thrown away 2023-01-27 20:00:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 20:00:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:00:30 +0000 [warn]: disable_chunk_backup is true. 5f3444db92f46697c10f82b564ebb9ce chunk is thrown away 2023-01-27 20:01:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 20:01:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:01:30 +0000 [warn]: disable_chunk_backup is true. 5f344514cb73ab157a8e4c87d4df4053 chunk is thrown away 2023-01-27 20:02:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 20:02:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:02:30 +0000 [warn]: disable_chunk_backup is true. 5f34454e03efcb12b9623af13490e727 chunk is thrown away 2023-01-27 20:03:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 20:03:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:03:30 +0000 [warn]: disable_chunk_backup is true. 5f34458738ca6088ede276460b3113f3 chunk is thrown away 2023-01-27 20:04:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 20:04:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:04:30 +0000 [warn]: disable_chunk_backup is true. 5f3445c074ca15e61f08c34aed8ca42f chunk is thrown away 2023-01-27 20:05:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 20:05:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:05:30 +0000 [warn]: disable_chunk_backup is true. 5f3445f9ad0b7393068b76a283ebd3ff chunk is thrown away 2023-01-27 20:06:30 +0000 [warn]: got unrecoverable error in primary and no secondary error_class=ArgumentError error="invalid byte sequence in UTF-8"   2023-01-27 20:06:30 +0000 [warn]: suppressed same stacktrace 2023-01-27 20:06:30 +0000 [warn]: disable_chunk_backup is true. 5f344632e5a6ae264e9ae6ca5c2a9772 chunk is thrown away
      

       

              Unassigned Unassigned
              preddy7 Venkata PrasadReddy Annem
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: