Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-23245

ImageStreams and Pods fail to pull images when Dev Portal generated secret is added in the namespace

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Normal Normal
    • None
    • 4.14
    • Image Registry
    • No
    • False
    • Hide

      None

      Show
      None

      The Developer Portal contains a service (Registry Service Accounts) for generating account tokens for accessing registry.redhat.io container registry:

      https://access.redhat.com/RegistryAuthentication#registry-service-accounts-for-shared-environments-4

      Creating a token generated by above service in user namespace breaks Developer Sandbox behavior.

      Reproduction steps:

      1. Generate a token in the Registry Service Accounts app, and download it as an OpenShift secret YAML file (7457766-thofman-pull-secret.yml in my case).
      2. Log into Developer Sandbox, and initialize the downloaded secret:

      oc create -f 7457766-thofman-pull-secret.yml
      

      Since now image streams are not able to pull images from registry.redhat.io (I think not even from other external repos):

      # create image stream
      $ oc replace --force -f https://raw.githubusercontent.com/jboss-container-images/jboss-eap-openshift-templates/eap74/eap74-openjdk11-image-stream.json
      # inspect the created image stream
      $ oc describe is jboss-eap74-openjdk11-openshift
      ...
        ! error: Import failed (InternalError): Internal error occurred: registry.redhat.io/jboss-eap-7/eap74-openjdk11-openshift-rhel8:latest: illegal base64 data at input byte 20
            6 seconds ago
      ...
      

      Also, I'm not able to build images anymore:

      $ oc new-build -D $'FROM centos:7\nRUN yum install -y httpd'
      W1114 09:48:18.273102   33679 dockerimagelookup.go:297] container image remote registry lookup failed: docker.io/library/centos:7: illegal base64 data at input byte 20
      error: unable to locate any images in image streams, local docker images with name "centos:7"
      

      3. Link the created secret with the "default" service account:

      oc secrets link default 7457766-thofman-pull-secret --for=pull
      

      Since now pods are not able to pull images from registty.redhat.io:

      # start a rollout on some deployment config (or just try to create some new pod)
      $ oc rollout latest eap7-app-dc
      # inspect the newly created pod
      $ oc describe pod eap8-app-7f4d9f6b59-nwjtw
      ...
      Events:
        Warning  Failed          4m54s (x4 over 6m26s)  kubelet            Failed to pull image "image-registry.openshift-image-registry.svc:5000/thofman-dev/eap8-app@sha256:7944a8c0c12f64b7ccbdb82023b5682736ecee76cba1c8081911d57f8fdfcd23": illegal base64 data at input byte 20
      ...
      

      I reproduced this on the Developer Sandbox (reportedly 4.14), as well as following shared cluster (4.13) from the Resource Hub:

      https://console-openshift-console.apps.shrocp4upi413ovn.lab.upshift.rdu2.redhat.com/k8s/ns/thofman/core~v1~Secret

      I'm aware that it may not be necessary to add pull secrets for accessing registry.redhat.io anymore, but I believe the presence of a secret should not break the behavior as it does. This is even stranger because the secret was generated on Red Hat customer portal (Registry Service Accounts), so it should not be totally invalid. Generating a pull secret on the Registry Service Accounts page used to be required on older OpenShift versions and customer may still be used to do that. If a secret is invalid it should be ignored, but it should not break so many activities...

      Workaround

      To prevent this issue from happening, do not install the pull secret generated on [1] into the namespace. There is no need to create a pull secret for access to the RH Image Registry on OCP 4.x, the access to the registry is enabled by default.

              fmissi Flavian Missi
              thofman Tomas Hofman
              XiuJuan Wang XiuJuan Wang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: