-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
Logging 5.8.1
-
False
-
-
False
-
NEW
-
NEW
-
Release Note Not Required
-
-
-
Log Collection - Sprint 246
Description of problem:
in fluent.conf, web_identity_token_file is using fix path /var/run/secrets/openshift/serviceaccount/token. i should use the token from the secret
cat fluent.conf
...............
<match **>
@type cloudwatch_logs
auto_create_stream true
region us-east-2
log_group_name_key cw_group_name
log_stream_name_key cw_stream_name
remove_log_stream_name_key true
remove_log_group_name_key true
concurrency 2
<web_identity_credentials>
role_arn "arn:aws:iam::3xxxxx:role/hypershift-ci-10374-to-cloudwatch"
web_identity_token_file "/var/run/secrets/openshift/serviceaccount/token"
role_session_name "cluster-logging"
</web_identity_credentials>
include_time_key true
log_rejected_request true
<buffer>
disable_chunk_backup true
</buffer>
</match>
...............
Version-Release number of selected component (if applicable):
5.8.1
How reproducible:
Steps to Reproduce:
- Create Cloudwatch secret using guest role_arn and token.
https://gitlab.cee.redhat.com/aosqe/aosqe-tools/-/raw/master/logging/log_template/cloudwatch/deploy_aws-sts-for-guest-cluster.sh?ref_type=heads - Send logs to cloudwatch using fluentd
cat <<EOF| oc create -f -
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
spec:
outputs:
- name: cloudwatch
type: cloudwatch
cloudwatch:
groupBy: logType
region: us-east-2
secret:
name: cloudwatch-credentials
pipelines:
- name: to-cloudwatch
inputRefs:
- infrastructure
- application
- audit
outputRefs:
- cloudwatch
EOF
cat <<EOF|oc create -f - apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: openshift-logging spec: managementState: "Managed" collection: type: "fluentd" EOF
- ...
Actual results:
oc logs collector-jw8d5 POD_IPS: 10.131.0.51, PROM_BIND_IP: 0.0.0.0 Setting each total_size_limit for 1 buffers to 19236595507 bytes Setting queued_chunks_limit_size for each buffer to 2293 Setting chunk_limit_size for each buffer to 8388608 2023-12-08 16:06:59 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp 2023-12-08 16:06:59 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp 2023-12-08 16:07:01 +0000 [error]: unexpected error error_class=Aws::Errors::MissingWebIdentityTokenFile error="Missing :web_identity_token_file parameter or invalid file path provided for Aws::AssumeRoleWebIdentityCredentials provider" 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:89:in `_token_from_file' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:75:in `refresh' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/refreshing_credentials.rb:30:in `initialize' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/aws-sdk-core-3.130.2/lib/aws-sdk-core/assume_role_web_identity_credentials.rb:65:in `initialize' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:155:in `new' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluent-plugin-cloudwatch-logs-0.14.2/lib/fluent/plugin/out_cloudwatch_logs.rb:155:in `start' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:203:in `block in start' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:182:in `block (2 levels) in lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:121:in `block (2 levels) in lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:120:in `each' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:120:in `block in lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:113:in `each' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/agent.rb:113:in `lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:181:in `block in lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `each' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:178:in `lifecycle' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/root_agent.rb:202:in `start' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:248:in `start' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/engine.rb:147:in `run' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:617:in `block in run_worker' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:962:in `main_process' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/supervisor.rb:608:in `run_worker' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/lib/fluent/command/fluentd.rb:372:in `<top (required)>' 2023-12-08 16:07:01 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require' 2023-12-08 16:07:01 +0000 [error]: <internal:/usr/share/rubygems/rubygems/core_ext/kernel_require.rb>:85:in `require' 2023-12-08 16:07:01 +0000 [error]: /usr/share/gems/gems/fluentd-1.16.2/bin/fluentd:15:in `<top (required)>' 2023-12-08 16:07:01 +0000 [error]: /usr/bin/fluentd:25:in `load' 2023-12-08 16:07:01 +0000 [error]: /usr/bin/fluentd:25:in `<main>' 2023-12-08 16:07:01 +0000 [error]: unexpected error error_class=Aws::Errors::MissingWebIdentityTokenFile error="Missing :web_identity_token_file parameter or invalid file path provided for Aws::AssumeRoleWebIdentityCredentials provider"
Expected results:
Additional info:
- is caused by
-
LOG-4780 Consume Cloudwatch web identity token that is not found at the well known SA path
-
- Closed
-
- links to