Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-25662

ECR Image pull fails in-spite of attaching AmazonEC2ContainerRegistryReadOnly policy to the worker nodes.

XMLWordPrintable

    • +
    • Important
    • Yes
    • CLOUD Sprint 247, CLOUD Sprint 248
    • 2
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the removal of code that provided image credentials from the Kubelet in {product-title} 4.14 caused pulling images from the Amazon Elastic Container Registry (ECR) to fail without a specified pull secret.
      This release includes a separate credential provider that provides ECR credentials for the Kubelet.
      (link:https://issues.redhat.com/browse/OCPBUGS-25662[*OCPBUGS-25662*])
      Show
      * Previously, the removal of code that provided image credentials from the Kubelet in {product-title} 4.14 caused pulling images from the Amazon Elastic Container Registry (ECR) to fail without a specified pull secret. This release includes a separate credential provider that provides ECR credentials for the Kubelet. (link: https://issues.redhat.com/browse/OCPBUGS-25662 [* OCPBUGS-25662 *])
    • Bug Fix
    • Done

      Description of problem:

      In ROSA/OCP 4.14.z, attaching AmazonEC2ContainerRegistryReadOnly policy to the worker nodes (in ROSA's case, this was attached to the ManagedOpenShift-Worker-Role, which is assigned by the installer to all the worker nodes), has no effect on ECR Image pull. User gets an authentication error. Attaching the policy ideally should avoid the need to provide an image-pull-secret. However, the error is resolved only if the user also provides an image-pull-secret.
      This is proven to work correctly in 4.12.z. Seems something has changed in the recent OCP versions.
      

      Version-Release number of selected component (if applicable):

      4.14.2 (ROSA)
      

      How reproducible:

      The issue is reproducible using the below steps.
      

      Steps to Reproduce:

          1. Create a deployment in ROSA or OCP on AWS, pointing at a private ECR repository
          2. The image pulling will fail with Error: ErrImagePull & authentication required errors
          3.
      

      Actual results:

      The image pull fails with "Error: ErrImagePull" & "authentication required" errors. However, the image pull is successful only if the user provides an image-pull-secret to the deployment.
      

      Expected results:

      The image should be pulled successfully by virtue of the ECR-read-only policy attached to the worker node role; without needing an image-pull-secret. 
      

      Additional info:

      
      

      In other words:

      in OCP 4.13 (and below) if a user adds the ECR:* permissions to the worker instance profile, then the user can specify ECR images and authentication of the worker node to ECR is done using the instance profile. In 4.14 this no longer works.

      It is not sufficient as an alternative, to provide a pull secret in a deployment because AWS rotates ECR tokens every 12 hours. That is not a viable solution for customers that until OCP 4.13, did not have to rotate pull secrets constantly.

      The experience in 4.14 should be the same as in 4.13 with ECR.

       

      The current AWS policy that's used is this one: `arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly`

      {
          "Version": "2012-10-17",
          "Statement": [
              {
                  "Effect": "Allow",
                  "Action": [
                      "ecr:GetAuthorizationToken",
                      "ecr:BatchCheckLayerAvailability",
                      "ecr:GetDownloadUrlForLayer",
                      "ecr:GetRepositoryPolicy",
                      "ecr:DescribeRepositories",
                      "ecr:ListImages",
                      "ecr:DescribeImages",
                      "ecr:BatchGetImage",
                      "ecr:GetLifecyclePolicy",
                      "ecr:GetLifecyclePolicyPreview",
                      "ecr:ListTagsForResource",
                      "ecr:DescribeImageScanFindings"
                  ],
                  "Resource": "*"
              }
          ]
      } 

       

       

              joelspeed Joel Speed
              aar@redhat.com Aneesh Abdul Rasheed
              Zhaohua Sun Zhaohua Sun
              Jeana Routh Jeana Routh
              Votes:
              0 Vote for this issue
              Watchers:
              32 Start watching this issue

                Created:
                Updated:
                Resolved: