Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-15411

db-backup get stuck on ImagePullBackOf on 3.11

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Blocker
    • Resolution: Done
    • fuse-7.8-GA
    • fuse-7.8-GA
    • Fuse Online
    • Hide

      Add auths for `registry.connect.redhat.com` to syndesis-pull-secret secret before upgrade process

      • Get auths json
        oc get secrets syndesis-pull-secret -o=custom-columns=SECRET:.data.* --no-headers | base64 -d | jq
        

        You get something like

        {
           "auths":{
              "registry.redhat.io":{
                 "username":"<AUTH>",
                 "password":"<AUTH>",
                 "auth":"<AUTH>"
              },
              ...<You can have more auths elements here>
           }
        }
        
      • Copy paste the same information for `registry.connect.redhat.com` ,encode it and save to the variable
        PULL_SECRET=$(base64 -w 0 <<EOF
        {
           "auths":{
              "registry.redhat.io":{
                 "username":"<AUTH>",
                 "password":"<AUTH>",
                 "auth":"<AUTH>"
              },
              ...<You can have more auths elements here>
              },
              "registry.connect.redhat.com":{
                 "username":"<AUTH>",
                 "password":"<AUTH>",
                 "auth":"<AUTH>"
              }
           }
        }
        EOF
        )
        
      • Patch Syndesis-pull-secret
        oc apply -f - <<EOF
        apiVersion: v1
        kind: Secret
        metadata:
          name: syndesis-pull-secret
        data:
          .dockerconfigjson: $PULL_SECRET
        type: kubernetes.io/dockerconfigjson
        EOF
        
      Show
      Add auths for `registry.connect.redhat.com` to syndesis-pull-secret secret before upgrade process Get auths json oc get secrets syndesis-pull-secret -o=custom-columns=SECRET:.data.* --no-headers | base64 -d | jq You get something like { "auths" :{ "registry.redhat.io" :{ "username" : "<AUTH>" , "password" : "<AUTH>" , "auth" : "<AUTH>" }, ...<You can have more auths elements here> } } Copy paste the same information for `registry.connect.redhat.com` ,encode it and save to the variable PULL_SECRET=$(base64 -w 0 <<EOF { "auths" :{ "registry.redhat.io" :{ "username" : "<AUTH>" , "password" : "<AUTH>" , "auth" : "<AUTH>" }, ...<You can have more auths elements here> }, "registry.connect.redhat.com" :{ "username" : "<AUTH>" , "password" : "<AUTH>" , "auth" : "<AUTH>" } } } EOF ) Patch Syndesis-pull-secret oc apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: syndesis-pull-secret data: .dockerconfigjson: $PULL_SECRET type: kubernetes.io/dockerconfigjson EOF

    Description

      During the upgrade process, the db-backup pod get stuck in ImagePullBackOf due to auths to the `registry.connect.redhat.com` repo:

      Events:
        Type     Reason     Age   From               Message
        ----     ------     ----  ----               -------
        Normal   Scheduled  11s   default-scheduler  Successfully assigned mkralik/db-backup-1607343267-4gr6p to node1.fo-311-c.dos.fuse-qe.eng.rdu2.redhat.com
        Normal   Pulling    8s    kubelet            pulling image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0"
        Warning  Failed     6s    kubelet            Failed to pull image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0": rpc error: code = Unknown desc = Get https://registry.connect.redhat.com/v2/crunchydata/crunchy-pgdump/manifests/ubi7-12.4-4.5.0: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
        Warning  Failed     6s    kubelet            Error: ErrImagePull
      

      The install process creates a syndesis-pull-secret with credentials to `registry.redhat.io`
      e.g.:

      {
         "auths":{
            "registry.redhat.io":{
               "username":"<USERNAME>",
               "password":"<PASSWORD>",
               "auth":"<AUTH>"
            }
         }
      }
      

      but the db-upgrade pod uses images from `registry.connect.redhat.com` . In that case, the install process should also create credentials for `registry.connect.redhat.com` as well. (It can use the same values as for `registry.redhat.io`).

      {
         "auths":{
            "registry.redhat.io":{
               "username":"<USERNAME>",
               "password":"<PASSWORD>",
               "auth":"<AUTH>"
            },
            "registry.connect.redhat.com":{
               "username":"<USERNAME>",
               "password":"<PASSWORD>",
               "auth":"<AUTH>"
            }
         }
      }
      

      When I update .dockerconfigjson in syndesis-pull-secret manually (decode it, add registry.connect.redhat.com repo to the json with the same auths as registry.redhat.io repo), the image is pull sucesfully.

      Strange is that I don't have this problem on OCP 4.x even though my syndesis-pull-secret contains credentials only for `registry.redhat.io` and doesn't contain credentials for `registry.connect.redhat.com`

      {
      	"auths": {
      		"quay.io": {
      			"auth": "<AUTH>"
      		},
      		"registry.redhat.io": {
      			"auth": "<AUTH>"
      		}
      	}
      }
      
      Events:
        Type    Reason          Age   From               Message
        ----    ------          ----  ----               -------
        Normal  Scheduled       89s   default-scheduler  Successfully assigned mkraliktestinstall12/db-backup-1607351006-tdhjs to crc-tnpk6-master-0
        Normal  AddedInterface  87s   multus             Add eth0 [10.116.0.82/23]
        Normal  Pulling         87s   kubelet            Pulling image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0"
        Normal  Pulled          56s   kubelet            Successfully pulled image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0" in 31.024212373s
        Normal  Created         55s   kubelet            Created container backup-db-compiler
      

      I see two possibilities how to fix it.

      • - Another problem here is that the customers, which use 7.6, has the syndesis-pull-secret already installed without auths to `registry.connect.redhat.com` so they need to delete it before the upgrade process. The upgrade process creates a new one. This can be documented as known limitation during the upgrade 7.6->7.7 on OCP 3.11

      Attachments

        Issue Links

          Activity

            People

              parichar@redhat.com Paul Richardson
              mkralik@redhat.com Matej Kralik
              Matej Kralik Matej Kralik
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: