Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-19846

Unable to destroy cluster when AWS Organization SCP prevents use of iam:GetUser


    • Low
    • No
    • Sprint 243
    • 1
    • False
    • Hide


    • N/A
    • Release Note Not Required

      This is a clone of issue OCPBUGS-17724. The following is the description of the original issue:

      Environment: OCP 4.12.24
      Installation Method: IPI: Manual Mode + STS using a customer provider AWS IAM Role

      I am trying to deploy an OCP4 cluster on AWS for my customer. The customer does not permit creation of IAM users so I am performing a Manual Mode with STS IPI installation instead. I have been given an IAM role to assume for the OCP installation, but unfortunately the customer's AWS Organizational Service Control Policy (SCP) does not permit the use of the iam:GetUser{} permission.

      (I have informed my customer that iam:GetUser is an installation requirement - it's clearly documented in our docs, and I have raised a ticket with their internal support team requesting that their SCP is amended to include iam:getUser, however I have been informed that my request is likely to be rejected).

      With this limitation understood, I still attempted to install OCP4. Surprisingly, I was able to deploy an OCP (4.12) cluster without any apparent issues, however when I tried to destroy the cluster I encountered the following error from the installer (note: fields in brackets <> have been redacted):

      DEBUG search for IAM roles
      DEBUG iterating over a page of 74 IAM roles
      DEBUG search for IAM users
      DEBUG iterating over a page of 1 IAM users
      INFO get tags for <ARN of the IAM user>: AccessDenied: User:<ARN of my user> is notauthorized to perform: iam:GetUser on resource: <IAMusername> with an explicit deny in a service control policy
      INFO status code: 403, request id: <request ID>
      DEBUG search for IAM instance profiles
      INFO error while finding resources to delete error=get tags for <ARN of IAM user> AccessDenied: User:<ARN of my user> is not authorized to perform: iam:GetUser on resource: <IAM username> with an explicit deny in a service control policy status code: 403, request id: <request ID>

      Similarly, the error in AWS CloudTrail logs shows the following (note: some fields in brackets have been redacted):
      User: arn:aws:sts::<AWS account no>:assumed-role/<role-name>/<user name> is not authorized to perform: iam:GetUser on resource <IAM User> with an explicit deny in a service control policy

      It appears that the destroy operation is failing when the installer is trying to list tags on the only IAM user in the customer's AWS account. As discussed, the SCP does not permit the use of iam:GetUser and consequently this API call on the IAM user is denied. The installer then enters an endless loop as it continuously retries the operation. We have potentially identified the iamUserSearch function within the installer code at pkg/destroy/aws/iamhelpers.go as the area where this call is failing.

      See: https://github.com/openshift/installer/blob/16f19ea94ecdb056d4955f33ddacc96c57341bb2/pkg/destroy/aws/iamhelpers.go#L95

      There does not appear to be a handler for "AccessDenied" API error in this function. Therefore we request that the access denied event is gracefully handled and skipped over when processing IAM users, allowing the installer to continue with the destroy operation, much in the same way that a similar access denied event is handled within the iamRoleSearch function when processing IAM roles:

      See: https://github.com/openshift/installer/blob/16f19ea94ecdb056d4955f33ddacc96c57341bb2/pkg/destroy/aws/iamhelpers.go#L51

      We therefore request that the following is considered and addressed:

      1. Re-assess if the iam:GetUser permission is actually needed for cluster installation/cluster operations. 
      2. If the permission is required then the installer should provide a warning or halt the installation.
      2. During a "destroy" cluster operation - the installer should gracefully handle AccessDenied errors from the API and "skip over" any IAM Users that the installer does not have permission to list tags for and then continue gracefully with the destroy operation.

            rh-ee-bbarbach Brent Barbachem
            openshift-crt-jira-prow OpenShift Prow Bot
            Yunfei Jiang Yunfei Jiang
            0 Vote for this issue
            6 Start watching this issue