Closing WONTDO.  As stated in prior comments, this is a limitation in the upstream client that has not been addressed there and there exists an open document task so customers are aware.  This may be addressed in future.

As a work around, it is possible to use multi ClusterLogForwarder to push logs using an alternate role ARN

One other thought..... they can also specify to group logs by `namespaceName` rather than `logType` in the forwarder config.   This may make IAM policies easier to manage... in case filtering by log-streams is too granular.    
Additionally, this policy can also be used as a "Deny" policy, rather than "Allow"

Our docs need an update to specifically state that you cannot use multiple "role" secrets, in a single CLF, when authenticating to Cloudwatch via STS. It's confusing because you CAN use multiple "long-lived" secrets for Cloudwatch. As a way to protect the user (especially if it is not documented), we could have also checked for multiple "token" secrets during our config generation and prevented all outputs using the same role.

There are endless ways to prevent users from accessing resources in cloudwatch, depending on whether they are using the CLI, or console... and how they are accessing the logs.   
Users, groups and roles can be restricted from log groups (and log streams) by using IAM policies. They can configure one role to send the logs, while others only have access to view, or filter, for example.

Here's an example policy that can be attached to any role/user/group.   This policy allows only viewing.... and only certain log groups and streams (and one restricted by region):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents",
        "logs:FilterLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:123456789012:log-group:testgroup1:*",
        "arn:aws:logs:us-east-1:123456789012:log-group:testgroup2:log-stream:*my-namespace*",
        "arn:aws:logs:us-west-2:123456789012:*"
      ]
    }
  ]
}

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-identity-based-access-control-cwl.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-access-control-overview-cwl.html#CWL_ARN_Format

rhn-support-ocasalsa unfortunately upstream vector does not support even a single "web_identity" authentication.    The auth functionality we need can not be accomplished using the vector configuration file.    We worked around this by exposing the vector client to env vars "AWS_ROLE_ARN" and "AWS_WEB_IDENTITY_TOKEN".  (again not yet supported by upstream community)  

We should have warned or prevented the user from attempting this, since we have had no resources or time (so far) to patch upstream logic to support this.    It's a bug that we did not document this, and that we did not handle the logic better... but this is also a feature that was never implemented.    Nobody thought this would be necessary until now?     Going forward this needs to be designed very carefully.         

cc jamparke@redhat.com 

Hi,  received an update from the account team trhough an internal escalation:  feature was released in the last version; however, it is not functioning. There is no other business impact, but the open security items have RH and Delta leadership's attention. 

Hello cahartma@redhat.com ,

Creating a second log forwarder is not a solution here. They will create different inputs (50) for each app team and log forwarding using 50 AWS IAM Roles defined. Then, it's not an option to create 50 different clusterLogForwarders, one for each AWS Role.

Casey, do we have a way or workarounding this, even, if it's using Unmanaged until the fix is available?

rhn-support-rkant support doesn't defined the priority, only the severity. I'm removing the priority added by you and allowing to engineering to set it. The current severity is Important as indicated in the support case and visible here.

This was identified as a need for HCP as well, prior to multiple forwarders being the solution. LOG-3650 contains a stub of code that should be in upstream vector by now. Need to verify and implement.

Creating a 2nd forwarder instance should work, since it would allow a unique env var per collector (pod) instance. Please confirm, as I've suggested this in the past.
Multiple STS outputs, as a feature, is not supported in upstream vector, and the existing env variable is our own fix and way to implement token-based auth. We should be able to resolve this eventually by using a shared_credentials env var, then specifying a role/profile to use... but as of last look the upstream had not yet implemented this to work properly. Needs another look.