-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
Logging 5.8.1
-
False
-
None
-
False
-
NEW
-
NEW
-
Release Note Not Required
-
-
-
Log Collection - Sprint 246
Description of problem:
in fluent.conf, web_identity_token_file is using fix path /var/run/secrets/openshift/serviceaccount/token. i should use the token from the secret
cat fluent.conf
............... <match **> @type cloudwatch_logs auto_create_stream true region us-east-2 log_group_name_key cw_group_name log_stream_name_key cw_stream_name remove_log_stream_name_key true remove_log_group_name_key true concurrency 2 <web_identity_credentials> role_arn "arn:aws:iam::3xxxxx:role/hypershift-ci-10374-to-cloudwatch" web_identity_token_file "/var/run/secrets/openshift/serviceaccount/token" role_session_name "cluster-logging" </web_identity_credentials> include_time_key true log_rejected_request true <buffer> disable_chunk_backup true </buffer> </match> ...............
Version-Release number of selected component (if applicable):
5.8.1
How reproducible:
Steps to Reproduce:
- Create Cloudwatch secret using guest role_arn and token.
https://gitlab.cee.redhat.com/aosqe/aosqe-tools/-/raw/master/logging/log_template/cloudwatch/deploy_aws-sts-for-guest-cluster.sh?ref_type=heads - Send logs to cloudwatch using fluentd
cat <<EOF| oc create -f - apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - name: cloudwatch type: cloudwatch cloudwatch: groupBy: logType region: us-east-2 secret: name: cloudwatch-credentials pipelines: - name: to-cloudwatch inputRefs: - infrastructure - application - audit outputRefs: - cloudwatch EOF
cat <<EOF|oc create -f - apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: openshift-logging spec: managementState: "Managed" collection: type: "fluentd" EOF
- ...
Actual results:
oc logs collector-jw8d5 POD_IPS: 10.131.0.51, PROM_BIND_IP: 0.0.0.0 Setting each total_size_limit for 1 buffers to 19236595507 bytes Setting queued_chunks_limit_size for each buffer to 2293 Setting chunk_limit_size for each buffer to 8388608 2023-12-08 16:06:59 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp 2023-12-08 16:06:59 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp 2023-12-08 16:07:01 +0000 [error]: unexpected error error_class=Aws::Errors::MissingWebIdentityTokenFile error="Missing :web_identity_token_file parameter or invalid file path provided for Aws::AssumeRoleWebIdentityCredentials provider" 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:89:in `_token_from_file' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:75:in `refresh' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/refreshing_credentials.rb:30:in `initialize' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:65:in `initialize' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:155:in `new' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:155:in `start' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:203:in `block in start' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:182:in `block (2 levels) in lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:121:in `block (2 levels) in lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:120:in `each' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:120:in `block in lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:113:in `each' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:113:in `lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:181:in `block in lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `each' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:202:in `start' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:248:in `start' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in `block in run_worker' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in `main_process' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in `run_worker' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>' 2023-12-08 16:07:01 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require' 2023-12-08 16:07:01 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>' 2023-12-08 16:07:01 +0000 [error]: /usr/bin/fluentd:25:in `load' 2023-12-08 16:07:01 +0000 [error]: /usr/bin/fluentd:25:in `<main>' 2023-12-08 16:07:01 +0000 [error]: unexpected error error_class=Aws::Errors::MissingWebIdentityTokenFile error="Missing :web_identity_token_file parameter or invalid file path provided for Aws::AssumeRoleWebIdentityCredentials provider"
Expected results:
Additional info:
- is caused by
-
LOG-4780 Consume Cloudwatch web identity token that is not found at the well known SA path
- Closed
- links to