Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-31525

Bootstrap ignition download error when using s3:// schema on AWS UPI installation

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.14
    • RHCOS
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Access denied error is returned by AWS S3 when trying to retrieve bootstrap ignition using s3:// schema (eg. s3://mybucket/bootstrap.ign).    

      Version-Release number of selected component (if applicable):

      OCP 4.14.17    

      How reproducible:

      Always    

      Steps to Reproduce:

          1. Follow documentation steps up to the section on "Creating the bootstrap node in AWS".
          2. Create s3 bucket and copy bootstrap.ign to it.
          3. Deploy the bootstrap node using the suggested CloudFormation template or deploy it manually. 
            a) If using the template, set value for BootstrapIgnitionLocation such as s3://mybucket/bootrap.ign.
            b) If deploying the instance manually, ensure that userdata is specified as: '{"ignition":{"config":{"replace":{"source":"s3://mybucket/bootstrap.ign"}},"version":"3.1.0"}}'. Also ensure that the role associated to the instance get get objects from S3.
          4. Bootstrap node will boot but it'll be unable to download the ignition file from the s3 bucket.     

      Actual results:

      ignition-fetch.service
      Mar 27 14:42:38 systemd[1]: sysroot.mount: Failed to load environment files: No such file or directory
      Mar 27 14:42:38 systemd[1]: sysroot.mount: Failed to run 'mount' task: No such file or directory
      Mar 27 14:42:38 systemd[1]: sysroot.mount: Failed with result 'resources'.
      Mar 27 14:42:38 systemd[1]: Failed to mount /sysroot.
      Mar 27 14:42:33 ignition[654]: GET result: OK
      Mar 27 14:42:33 ignition[654]: parsing config with SHA512: 2013aebcc2f41e57c232ab01bfc951fa2c27bff1ee63ff66890297384aa7de0124a01601fc498dcfdbbc0202a985345f9c943ae87fdb5bf0ce3f4a49b3acc476
      Mar 27 14:42:34 ignition[654]: failed to fetch config: AccessDenied: Access Denied
                                             status code: 403, request id: SH8QM6VD780G6RAB, host id: 63uqknDCt3SiJbYIRGjlCBOWcQjAXxWQtTP1yyPdduroPBAfJ9YWjrj27Q7zxkRRqZt/RXFRtkU=
      Mar 27 14:42:34 ignition[654]: failed to acquire config: AccessDenied: Access Denied
                                             status code: 403, request id: SH8QM6VD780G6RAB, host id: 63uqknDCt3SiJbYIRGjlCBOWcQjAXxWQtTP1yyPdduroPBAfJ9YWjrj27Q7zxkRRqZt/RXFRtkU=
      Mar 27 14:42:34 ignition[654]: Ignition failed: AccessDenied: Access Denied
                                             status code: 403, request id: SH8QM6VD780G6RAB, host id: 63uqknDCt3SiJbYIRGjlCBOWcQjAXxWQtTP1yyPdduroPBAfJ9YWjrj27Q7zxkRRqZt/RXFRtkU=
      Mar 27 14:42:33 systemd[1]: Starting Ignition (fetch)...
      Mar 27 14:42:35 systemd[1]: ignition-fetch.service: Main process exited, code=exited, status=1/FAILURE
      Mar 27 14:42:35 systemd[1]: ignition-fetch.service: Failed with result 'exit-code'.
      Mar 27 14:42:35 systemd[1]: Failed to start Ignition (fetch).
      Mar 27 14:42:36 systemd[1]: ignition-fetch.service: Triggering OnFailure= dependencies.    

      Expected results:

      Bootstrap node should be able to download ignition using s3:// schema.    

      Additional info:

      1)
      
      In the same bootstrap node, using a small application that uses aws-sdk it is possible to download ignition from the same s3 bucket:
      
      :/# ./sysroot/s3-tests | jq . | head -n10
      {
        "ignition": {
          "version": "3.2.0"
        },
        "passwd": {
          "users": [
            {
              "name": "core",
              "sshAuthorizedKeys": [
                "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCGXygk5XJAP1pddRhAT+xycwI1a33FbYivak1YjpX1ovNdHzPo6MoOONFSx7XXn7LIPR/hRhtsxQ63kKFJQlgE+qjKN7/zBHrYK6Zq0hvJpOil/Rmc+92ZsVxfd/bvx4elRk4SedwijPfGRk0hvYyc2m+fN/zUExMnMCtyz6n6md/pTRIe1MfJFKOneIUnKDIRde0KpSt8QhYpDkKf3ux+5vWcpUZa6GMMCy1QN2NHlp4Ug+NYBh6BxFQmmaycXL2V7hxALfno3/3FR8yYULFjzGyxOtoRcSfodlJoIrt04LYooKVWmpL4G/RjXPQGP3K7x+lMfPWqjg8tf30X6aKh
      
      For reference, the code is here: https://gist.github.com/vagnerfarias/84bd3b93dbf8cd9eade2294c26f024df
      
      2) Downloading from the same bucket, but using a presigned URL instead, works.
      
      3) Downloading from the same bucket, but using an ARN instead, also works (eg. {"ignition":{"config":{"replace":{"source":"arn:aws:s3:::ocp2-mb4d8-infra/bootstrap.ign"}},"version":"3.4.0"}}) - Note ignition version.
      
      4) Trying with ignition version 3.4.0 also didn't work when trying to use s3:// schema.

            Unassigned Unassigned
            vagnerfarias Vagner Farias
            Michael Nguyen Michael Nguyen
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: