Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-15411

db-backup get stuck on ImagePullBackOf on 3.11

XMLWordPrintable

    • False
    • False
    • % %
    • Undefined
    • Hide

      Add auths for `registry.connect.redhat.com` to syndesis-pull-secret secret before upgrade process

      • Get auths json 
        oc get secrets syndesis-pull-secret -o=custom-columns=SECRET:.data.* --no-headers | base64 -d | jq

        You get something like

        {
   "auths":{
      "registry.redhat.io":{
         "username":"<AUTH>",
         "password":"<AUTH>",
         "auth":"<AUTH>"
      },
      ...<You can have more auths elements here>
   }
}
      • Copy paste the same information for `registry.connect.redhat.com` ,encode it and save to the variable 
        PULL_SECRET=$(base64 -w 0 <<EOF
{
   "auths":{
      "registry.redhat.io":{
         "username":"<AUTH>",
         "password":"<AUTH>",
         "auth":"<AUTH>"
      },
      ...<You can have more auths elements here>
      },
      "registry.connect.redhat.com":{
         "username":"<AUTH>",
         "password":"<AUTH>",
         "auth":"<AUTH>"
      }
   }
}
EOF
)
      • Patch Syndesis-pull-secret 
        oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: syndesis-pull-secret
data:
  .dockerconfigjson: $PULL_SECRET
type: kubernetes.io/dockerconfigjson
EOF
      Show
      Add auths for `registry.connect.redhat.com` to syndesis-pull-secret secret before upgrade process Get auths json oc get secrets syndesis-pull-secret -o=custom-columns=SECRET:.data.* --no-headers | base64 -d | jq You get something like { "auths" :{ "registry.redhat.io" :{ "username" : "<AUTH>" , "password" : "<AUTH>" , "auth" : "<AUTH>" }, ...<You can have more auths elements here> } } Copy paste the same information for `registry.connect.redhat.com` ,encode it and save to the variable PULL_SECRET=$(base64 -w 0 <<EOF { "auths" :{ "registry.redhat.io" :{ "username" : "<AUTH>" , "password" : "<AUTH>" , "auth" : "<AUTH>" }, ...<You can have more auths elements here> }, "registry.connect.redhat.com" :{ "username" : "<AUTH>" , "password" : "<AUTH>" , "auth" : "<AUTH>" } } } EOF ) Patch Syndesis-pull-secret oc apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: syndesis-pull-secret data: .dockerconfigjson: $PULL_SECRET type: kubernetes.io/dockerconfigjson EOF

      During the upgrade process, the db-backup pod get stuck in ImagePullBackOf due to auths to the `registry.connect.redhat.com` repo:

      Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  11s   default-scheduler  Successfully assigned mkralik/db-backup-1607343267-4gr6p to node1.fo-311-c.dos.fuse-qe.eng.rdu2.redhat.com
  Normal   Pulling    8s    kubelet            pulling image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0"
  Warning  Failed     6s    kubelet            Failed to pull image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0": rpc error: code = Unknown desc = Get https://registry.connect.redhat.com/v2/crunchydata/crunchy-pgdump/manifests/ubi7-12.4-4.5.0: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
  Warning  Failed     6s    kubelet            Error: ErrImagePull

      The install process creates a syndesis-pull-secret with credentials to `registry.redhat.io`
      e.g.:

      {
   "auths":{
      "registry.redhat.io":{
         "username":"<USERNAME>",
         "password":"<PASSWORD>",
         "auth":"<AUTH>"
      }
   }
}

      but the db-upgrade pod uses images from `registry.connect.redhat.com` . In that case, the install process should also create credentials for `registry.connect.redhat.com` as well. (It can use the same values as for `registry.redhat.io`).

      {
   "auths":{
      "registry.redhat.io":{
         "username":"<USERNAME>",
         "password":"<PASSWORD>",
         "auth":"<AUTH>"
      },
      "registry.connect.redhat.com":{
         "username":"<USERNAME>",
         "password":"<PASSWORD>",
         "auth":"<AUTH>"
      }
   }
}

      When I update .dockerconfigjson in syndesis-pull-secret manually (decode it, add registry.connect.redhat.com repo to the json with the same auths as registry.redhat.io repo), the image is pull sucesfully.

      Strange is that I don't have this problem on OCP 4.x even though my syndesis-pull-secret contains credentials only for `registry.redhat.io` and doesn't contain credentials for `registry.connect.redhat.com`

      {
	"auths": {
		"quay.io": {
			"auth": "<AUTH>"
		},
		"registry.redhat.io": {
			"auth": "<AUTH>"
		}
	}
}

      Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       89s   default-scheduler  Successfully assigned mkraliktestinstall12/db-backup-1607351006-tdhjs to crc-tnpk6-master-0
  Normal  AddedInterface  87s   multus             Add eth0 [10.116.0.82/23]
  Normal  Pulling         87s   kubelet            Pulling image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0"
  Normal  Pulled          56s   kubelet            Successfully pulled image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0" in 31.024212373s
  Normal  Created         55s   kubelet            Created container backup-db-compiler

      I see two possibilities how to fix it.

      • - Another problem here is that the customers, which use 7.6, has the syndesis-pull-secret already installed without auths to `registry.connect.redhat.com` so they need to delete it before the upgrade process. The upgrade process creates a new one. This can be documented as known limitation during the upgrade 7.6->7.7 on OCP 3.11

              parichar@redhat.com Paul Richardson
              mkralik@redhat.com Matej Kralik
              Matej Kralik Matej Kralik
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: