We need to provide better guidance to help internal teams and customers to troubleshoot STS credentials provided by CCO when deploying a cluster with authentication mode with STS credentials on AWS.

Recently we provided a few contents that helped customers[1], but yesterday [ mimccune@redhat.com asked | https://coreos.slack.com/archives/C011MLLLY4W/p1657827807146079] help with an interesting support case[2] to have something to review the credentials provided by CCO (through CredentialsRequests) and effectively created on IAM Role policies. Marco has provided one script[3] to the customer with steps to make "e2e" tests for MAPI which:

• Discovery of the token provided as ServiceAccount to MAPI
• Compare the IAM Role Policy permissions against CCO CredentialsRequests permissions
• Assume the role with the token provided by the service account, getting STS short-lived credentials, and using it to authenticate to AWS API
• Use the short-lived token to create manually (using AWS CLI) one instance, testing RunInstance EC2 API, based on existing worker machine attributes (tl;dr: adding manually one machine).

This script is useful to validate the flow and component, thus we can detect any step that can be failing. It will be nice to have something ready to be shared with internal teams and customers.

GOAL

Share the information and results collected from customer case and internal research. Some options are:

• KCS describing the tests executed with scenarios with Policies limiting access to ec2:runInstances operation, or equivalent required by MAPI
• Script with better feedback for each step when testing the tokens, comparing the permissions, and trying to use the short-lived credentials
• Spike/Propose some tool to be like a "problem detector" with IAM read permissions comparing and testing the tokens provided by CCO to components that are working correctly and have desired permissions.

ENGINEERING CONTENT

[1] Content created recently with this topic:

• CCO documentation to use private S3 bucket
• Blog post with "Deep dive" in OIDC on AWS using OCP with auth mode with STS
• Blog post with long steps of evaluating CloudFront as an alternative of using private S3 bucket (research that generates the CCO documentation)

[2] Support case: https://access.redhat.com/support/cases/#/case/03226796

[3] Hacking steps/script provided on the support case: https://mtulio.net/playbooks/openshift/ocp-aws-cco-run-instance/