Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4866

Fluentd MissingWebIdentityTokenFile when consume Cloudwatch web identity token

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Normal Normal
    • None
    • Logging 5.8.1
    • Log Collection
    • False
    • None
    • False
    • NEW
    • NEW
    • Release Note Not Required
    • Log Collection - Sprint 246

      Description of problem:

      in fluent.conf, web_identity_token_file is using fix path /var/run/secrets/openshift/serviceaccount/token. i should use the token from the secret

      cat fluent.conf

        ...............
        <match **>
          @type cloudwatch_logs
          auto_create_stream true
          region us-east-2
          log_group_name_key cw_group_name
          log_stream_name_key cw_stream_name
          remove_log_stream_name_key true
          remove_log_group_name_key true
          concurrency 2
          <web_identity_credentials>
            role_arn "arn:aws:iam::3xxxxx:role/hypershift-ci-10374-to-cloudwatch"
            web_identity_token_file "/var/run/secrets/openshift/serviceaccount/token"
            role_session_name "cluster-logging"
          </web_identity_credentials>
          include_time_key true
          log_rejected_request true
      
          <buffer>
            disable_chunk_backup true
          </buffer>
        </match>
         ...............
      
       
      

      Version-Release number of selected component (if applicable):
      5.8.1

      How reproducible:

      Steps to Reproduce:

      1. Create Cloudwatch secret using guest role_arn and token.
        https://gitlab.cee.redhat.com/aosqe/aosqe-tools/-/raw/master/logging/log_template/cloudwatch/deploy_aws-sts-for-guest-cluster.sh?ref_type=heads
      2.  Send logs to cloudwatch using fluentd
      cat <<EOF| oc create -f -
      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        name: instance
        namespace: openshift-logging
      spec:
        outputs:
        - name: cloudwatch
          type: cloudwatch
          cloudwatch:
            groupBy: logType
            region: us-east-2
          secret:
            name: cloudwatch-credentials
        pipelines:
        - name: to-cloudwatch
          inputRefs:
          - infrastructure
          - application
          - audit
          outputRefs:
          - cloudwatch
      EOF
      
      cat <<EOF|oc create -f -
      apiVersion: "logging.openshift.io/v1"
      kind: "ClusterLogging"
      metadata:
        name: "instance"
        namespace: openshift-logging
      spec:
        managementState: "Managed"
        collection:
          type: "fluentd"
      EOF
      
      1. ...

      Actual results:

      oc logs collector-jw8d5
      POD_IPS: 10.131.0.51, PROM_BIND_IP: 0.0.0.0
      Setting each total_size_limit for 1 buffers to 19236595507 bytes
      Setting queued_chunks_limit_size for each buffer to 2293
      Setting chunk_limit_size for each buffer to 8388608
      2023-12-08 16:06:59 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
      2023-12-08 16:06:59 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
      2023-12-08 16:07:01 +0000 [error]: unexpected error error_class=Aws::Errors::MissingWebIdentityTokenFile error="Missing :web_identity_token_file parameter or invalid file path provided for Aws::AssumeRoleWebIdentityCredentials provider"
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:89:in `_token_from_file'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:75:in `refresh'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/refreshing_credentials.rb:30:in `initialize'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:65:in `initialize'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:155:in `new'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:155:in `start'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:203:in `block in start'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:182:in `block (2 levels) in lifecycle'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:121:in `block (2 levels) in lifecycle'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:120:in `each'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:120:in `block in lifecycle'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:113:in `each'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:113:in `lifecycle'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:181:in `block in lifecycle'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `each'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `lifecycle'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:202:in `start'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:248:in `start'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in `block in run_worker'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in `main_process'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in `run_worker'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>'
        2023-12-08 16:07:01 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        2023-12-08 16:07:01 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require'
        2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>'
        2023-12-08 16:07:01 +0000 [error]: /usr/bin/fluentd:25:in `load'
        2023-12-08 16:07:01 +0000 [error]: /usr/bin/fluentd:25:in `<main>'
      2023-12-08 16:07:01 +0000 [error]: unexpected error error_class=Aws::Errors::MissingWebIdentityTokenFile error="Missing :web_identity_token_file parameter or invalid file path provided for Aws::AssumeRoleWebIdentityCredentials provider"
      

      Expected results:

      Additional info:

              jcantril@redhat.com Jeffrey Cantrill
              rhn-support-anli Anping Li
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: