Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-15411

db-backup get stuck on ImagePullBackOf on 3.11

    XMLWordPrintable

Details

    • False
    • False
    • % %
    • Undefined
    • Hide

      Add auths for `registry.connect.redhat.com` to syndesis-pull-secret secret before upgrade process

      • Get auths json 
        oc get secrets syndesis-pull-secret -o=custom-columns=SECRET:.data.* --no-headers | base64 -d | jq

        You get something like

        {
   "auths":{
      "registry.redhat.io":{
         "username":"<AUTH>",
         "password":"<AUTH>",
         "auth":"<AUTH>"
      },
      ...<You can have more auths elements here>
   }
}
      • Copy paste the same information for `registry.connect.redhat.com` ,encode it and save to the variable 
        PULL_SECRET=$(base64 -w 0 <<EOF
{
   "auths":{
      "registry.redhat.io":{
         "username":"<AUTH>",
         "password":"<AUTH>",
         "auth":"<AUTH>"
      },
      ...<You can have more auths elements here>
      },
      "registry.connect.redhat.com":{
         "username":"<AUTH>",
         "password":"<AUTH>",
         "auth":"<AUTH>"
      }
   }
}
EOF
)
      • Patch Syndesis-pull-secret 
        oc apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: syndesis-pull-secret
data:
  .dockerconfigjson: $PULL_SECRET
type: kubernetes.io/dockerconfigjson
EOF
      Show
      Add auths for `registry.connect.redhat.com` to syndesis-pull-secret secret before upgrade process Get auths json oc get secrets syndesis-pull-secret -o=custom-columns=SECRET:.data.* --no-headers | base64 -d | jq You get something like { "auths" :{ "registry.redhat.io" :{ "username" : "<AUTH>" , "password" : "<AUTH>" , "auth" : "<AUTH>" }, ...<You can have more auths elements here> } } Copy paste the same information for `registry.connect.redhat.com` ,encode it and save to the variable PULL_SECRET=$(base64 -w 0 <<EOF { "auths" :{ "registry.redhat.io" :{ "username" : "<AUTH>" , "password" : "<AUTH>" , "auth" : "<AUTH>" }, ...<You can have more auths elements here> }, "registry.connect.redhat.com" :{ "username" : "<AUTH>" , "password" : "<AUTH>" , "auth" : "<AUTH>" } } } EOF ) Patch Syndesis-pull-secret oc apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: syndesis-pull-secret data: .dockerconfigjson: $PULL_SECRET type: kubernetes.io/dockerconfigjson EOF

    Description

      During the upgrade process, the db-backup pod get stuck in ImagePullBackOf due to auths to the `registry.connect.redhat.com` repo:

      Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  11s   default-scheduler  Successfully assigned mkralik/db-backup-1607343267-4gr6p to node1.fo-311-c.dos.fuse-qe.eng.rdu2.redhat.com
  Normal   Pulling    8s    kubelet            pulling image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0"
  Warning  Failed     6s    kubelet            Failed to pull image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0": rpc error: code = Unknown desc = Get https://registry.connect.redhat.com/v2/crunchydata/crunchy-pgdump/manifests/ubi7-12.4-4.5.0: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
  Warning  Failed     6s    kubelet            Error: ErrImagePull

      The install process creates a syndesis-pull-secret with credentials to `registry.redhat.io`
      e.g.:

      {
   "auths":{
      "registry.redhat.io":{
         "username":"<USERNAME>",
         "password":"<PASSWORD>",
         "auth":"<AUTH>"
      }
   }
}

      but the db-upgrade pod uses images from `registry.connect.redhat.com` . In that case, the install process should also create credentials for `registry.connect.redhat.com` as well. (It can use the same values as for `registry.redhat.io`).

      {
   "auths":{
      "registry.redhat.io":{
         "username":"<USERNAME>",
         "password":"<PASSWORD>",
         "auth":"<AUTH>"
      },
      "registry.connect.redhat.com":{
         "username":"<USERNAME>",
         "password":"<PASSWORD>",
         "auth":"<AUTH>"
      }
   }
}

      When I update .dockerconfigjson in syndesis-pull-secret manually (decode it, add registry.connect.redhat.com repo to the json with the same auths as registry.redhat.io repo), the image is pull sucesfully.

      Strange is that I don't have this problem on OCP 4.x even though my syndesis-pull-secret contains credentials only for `registry.redhat.io` and doesn't contain credentials for `registry.connect.redhat.com`

      {
	"auths": {
		"quay.io": {
			"auth": "<AUTH>"
		},
		"registry.redhat.io": {
			"auth": "<AUTH>"
		}
	}
}

      Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       89s   default-scheduler  Successfully assigned mkraliktestinstall12/db-backup-1607351006-tdhjs to crc-tnpk6-master-0
  Normal  AddedInterface  87s   multus             Add eth0 [10.116.0.82/23]
  Normal  Pulling         87s   kubelet            Pulling image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0"
  Normal  Pulled          56s   kubelet            Successfully pulled image "registry.connect.redhat.com/crunchydata/crunchy-pgdump:ubi7-12.4-4.5.0" in 31.024212373s
  Normal  Created         55s   kubelet            Created container backup-db-compiler

      I see two possibilities how to fix it.

      • - Another problem here is that the customers, which use 7.6, has the syndesis-pull-secret already installed without auths to `registry.connect.redhat.com` so they need to delete it before the upgrade process. The upgrade process creates a new one. This can be documented as known limitation during the upgrade 7.6->7.7 on OCP 3.11

      Attachments

        Issue Links

          is caused by

          Bug - A problem that impairs or prevents the functions of the product. ENTESB-15364 Use productized images for backup

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Done

          Activity

            People

              parichar@redhat.com Paul Richardson
              mkralik@redhat.com Matej Kralik
              Matej Kralik Matej Kralik
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: