Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-11636

AWS s3 policy changes block all OCP installs on AWS

    • Critical
    • No
    • Approved
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      The ACLs are disabled for all newly created s3 buckets, this causes all OCP installs to fail: the bootstrap ignition can not be uploaded:
      
      level=info msg=Creating infrastructure resources...
      level=error
      level=error msg=Error: error creating S3 bucket ACL for yunjiang-acl413-4dnhx-bootstrap: AccessControlListNotSupported: The bucket does not allow ACLs
      level=error msg=	status code: 400, request id: HTB2HSH6XDG0Q3ZA, host id: V6CrEgbc6eyfJkUbLXLxuK4/0IC5hWCVKEc1RVonSbGpKAP1RWB8gcl5dfyKjbrLctVlY5MG2E4=
      level=error
      level=error msg=  with aws_s3_bucket_acl.ignition,
      level=error msg=  on main.tf line 62, in resource "aws_s3_bucket_acl" "ignition":
      level=error msg=  62: resource "aws_s3_bucket_acl" ignition {
      level=error
      level=error msg=failed to fetch Cluster: failed to generate asset "Cluster": failure applying terraform for "bootstrap" stage: failed to create cluster: failed to apply Terraform: exit status 1
      level=error
      level=error msg=Error: error creating S3 bucket ACL for yunjiang-acl413-4dnhx-bootstrap: AccessControlListNotSupported: The bucket does not allow ACLs
      level=error msg=	status code: 400, request id: HTB2HSH6XDG0Q3ZA, host id: V6CrEgbc6eyfJkUbLXLxuK4/0IC5hWCVKEc1RVonSbGpKAP1RWB8gcl5dfyKjbrLctVlY5MG2E4=
      level=error
      level=error msg=  with aws_s3_bucket_acl.ignition,
      level=error msg=  on main.tf line 62, in resource "aws_s3_bucket_acl" "ignition":
      level=error msg=  62: resource "aws_s3_bucket_acl" ignition {
      
      
      

      Version-Release number of selected component (if applicable):

      4.11+
       

      How reproducible:

      Always
       

      Steps to Reproduce:

      1.Create a cluster via IPI
      
      

      Actual results:

      install fail
       

      Expected results:

      install succeed
       

      Additional info:

      Heads-Up: Amazon S3 Security Changes Are Coming in April of 2023 - https://aws.amazon.com/blogs/aws/heads-up-amazon-s3-security-changes-are-coming-in-april-of-2023/
      
      https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-ownership-error-responses.html - After you apply the bucket owner enforced setting for Object Ownership, ACLs are disabled.
      
       

            [OCPBUGS-11636] AWS s3 policy changes block all OCP installs on AWS

            Per the announcement sent regarding the removal of "Blocker" as an option in the Priority field, this issue (which was already closed at the time of the bulk update) had Priority = "Blocker." It is being updated to Priority = Critical. No additional fields were changed.

            OpenShift Jira Automation Bot added a comment - Per the announcement sent regarding the removal of "Blocker" as an option in the Priority field, this issue (which was already closed at the time of the bulk update) had Priority = "Blocker." It is being updated to Priority = Critical. No additional fields were changed.

            Closing out a few bugs which were fixed prior to 4.13 GA so that it's easier for us to review which 4.14 blockers may be associated with outstanding 4.13 bugs. This seems safe because the bug was fixed on master branch (4.14) but backported to 4.13 prior to 4.13 GA, therefore there's no need to include these bugs in the release notes for 4.14.

            Scott Dodson added a comment - Closing out a few bugs which were fixed prior to 4.13 GA so that it's easier for us to review which 4.14 blockers may be associated with outstanding 4.13 bugs. This seems safe because the bug was fixed on master branch (4.14) but backported to 4.13 prior to 4.13 GA, therefore there's no need to include these bugs in the release notes for 4.14.

            Michael Shen mentioned this issue in a merge request of Service Delivery / clusterimagesets on branch OSD-17134:

            Restrict allowed install versions in prod

            GitLab CEE Bot added a comment - Michael Shen mentioned this issue in a merge request of Service Delivery / clusterimagesets on branch OSD-17134 : Restrict allowed install versions in prod

            Nicholas Schuetz added a comment - - edited

            Confirmed resolution in 4.12.12: https://access.redhat.com/solutions/7007136

            Nicholas Schuetz added a comment - - edited Confirmed resolution in 4.12.12: https://access.redhat.com/solutions/7007136

            Thanks mifiedle@redhat.com! Updated 4.13 bug.

            Sharada Vetsa added a comment - Thanks mifiedle@redhat.com ! Updated 4.13 bug.

            svetsa@redhat.com This is the 4.14 bug.  See https://issues.redhat.com/browse/OCPBUGS-11661 for the 4.13 bug.

            Mike Fiedler added a comment - svetsa@redhat.com This is the 4.14 bug.  See https://issues.redhat.com/browse/OCPBUGS-11661 for the 4.13 bug.

            nmalik mentioned this issue in a merge request of Service Delivery / clusterimagesets on branch OCPBUGS-11636:

            Draft: Changes for OCPBUGS-11636

            GitLab CEE Bot added a comment - nmalik mentioned this issue in a merge request of Service Delivery / clusterimagesets on branch OCPBUGS-11636 : Draft: Changes for OCPBUGS-11636

            Amrita Mahapatra added a comment - Still facing the same issue for 4.13. Build details: https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/job/qe-deploy-ocs-cluster/23068/console

            Neha Berry added a comment -

            Hi,

            Is the issue fixed in OCP 4.13 too ? We tried install yesterday and it is still failing?

            Neha Berry added a comment - Hi, Is the issue fixed in OCP 4.13 too ? We tried install yesterday and it is still failing?

            Verified on 4.14.0-0.ci-2023-04-12-091652, PASS

            Yunfei Jiang added a comment - Verified on 4.14.0-0.ci-2023-04-12-091652, PASS

              padillon Patrick Dillon
              yunjiang-1 Yunfei Jiang
              Yunfei Jiang Yunfei Jiang
              Votes:
              12 Vote for this issue
              Watchers:
              78 Start watching this issue

                Created:
                Updated:
                Resolved: