-
Bug
-
Resolution: Done
-
Critical
-
quay-v3.4.0
-
False
-
False
-
Undefined
-
Description:
This is an issue found when migration quay CR from QuayEcosystem to QuayRegistry, In Quay V3.3. when create QuayEcosystem using loadbalancer as quay external access, after triggered migration, two new loadbalancers are created for quay and quay config editor, that means quay app endpoint was changed, delete old Quay CR, then pull image from existing image repository, it was failed, the reason is with Quay V3.4, the quay hostname is still use previous loadbalancer hostname, also try to create new image repository, then push image was failed, the reason is with Quay V3.4, the quay hostname is still use previous loadbalancer hostname.
Note:
- In Quay V3.3 the quay app hostname is "a9ff79ba67f62488088a5110c62ffaba-494603238.us-east-2.elb.amazonaws.com"
- In Quay V3.4, the quay app hostname is "ac6d09b544fca460499d420dc97a4f7d-1556817724.us-east-2.elb.amazonaws.com"
Before Operator Migration in V3.3:
lizhang@lzha-mac Quay3.3_operator_testing % oc get pod
NAME READY STATUS RESTARTS AGE
mig2-quayecosystem-clair-55bff9bc76-znrxc 1/1 Running 0 19m
mig2-quayecosystem-clair-postgresql-5f7bcd9c55-qg2jj 1/1 Running 0 20m
mig2-quayecosystem-quay-54b647b699-hdl7p 1/1 Running 0 20m
mig2-quayecosystem-quay-config-6db54885d5-9rxf8 1/1 Running 0 21m
mig2-quayecosystem-redis-7d6fd564c9-bxnhr 1/1 Running 0 23m
quay-operator-5cb9dd9974-chsmc 1/1 Running 0 39mlizhang@lzha-mac Quay3.3_operator_testing %oc get svc
mig2-quayecosystem-quay LoadBalancer 172.30.242.159 a9ff79ba67f62488088a5110c62ffaba-494603238.us-east-2.elb.amazonaws.com 443:32664/TCP,9091:30867/TCP 5m56s
mig2-quayecosystem-quay-config LoadBalancer 172.30.116.172 a6604b6a185bd416c8d310b246cffefc-2015600238.us-east-2.elb.amazonaws.com 443:31236/TCP 5m55s
After Migration to V3.40:
lizhang@lzha-mac Quay3.3_operator_testing % oc get pod NAME READY STATUS RESTARTS AGE mig2-quayecosystem-clair-55bff9bc76-znrxc 1/1 Running 0 32m mig2-quayecosystem-clair-app-f9787b4c7-x5t7z 1/1 Running 0 5m26s mig2-quayecosystem-clair-postgres-6c648b6c65-ph5l8 1/1 Running 1 10m mig2-quayecosystem-clair-postgresql-5f7bcd9c55-qg2jj 1/1 Running 0 33m mig2-quayecosystem-quay-54b647b699-hdl7p 1/1 Running 0 33m mig2-quayecosystem-quay-app-6887b88d4f-cb6fx 1/1 Running 0 5m26s mig2-quayecosystem-quay-config-6db54885d5-9rxf8 1/1 Running 0 34m mig2-quayecosystem-quay-config-editor-cf5b8d7b5-htmmg 1/1 Running 0 5m26s mig2-quayecosystem-quay-redis-8ffbd8d79-mjq85 1/1 Running 0 10m mig2-quayecosystem-redis-7d6fd564c9-bxnhr 1/1 Running 0 36mlizhang@lzha-mac Quay3.3_operator_testing % oc get service mig2-quayecosystem-quay LoadBalancer 172.30.242.159 a9ff79ba67f62488088a5110c62ffaba-494603238.us-east-2.elb.amazonaws.com 443:32664/TCP,9091:30867/TCP 39m mig2-quayecosystem-quay-app LoadBalancer 172.30.168.187 ac6d09b544fca460499d420dc97a4f7d-1556817724.us-east-2.elb.amazonaws.com 443:32185/TCP,80:32599/TCP,8081:30920/TCP 14m mig2-quayecosystem-quay-config LoadBalancer 172.30.116.172 a6604b6a185bd416c8d310b246cffefc-2015600238.us-east-2.elb.amazonaws.com 443:31236/TCP 39m mig2-quayecosystem-quay-config-editor LoadBalancer 172.30.94.111 a0eaf53e6a758472096ecdb7cc867b3d-1885073017.us-east-2.elb.amazonaws.com 80:31744/TCP 14m
Quay TNG Operator Image:
lizhang@lzha-mac Quay3.3_operator_testing % oc get pod -n openshift-operators NAME READY STATUS RESTARTS AGE container-security-operator-766c7f8f5b-4tpkv 1/1 Running 0 27d quay-operator-59d4f8b9fd-z54cc 1/1 Running 0 80m lizhang@lzha-mac Quay3.3_operator_testing % oc get pod quay-operator-59d4f8b9fd-z54cc -n openshift-operators -o json | jq '.spec.containers[0].image' "registry.redhat.io/quay/quay-rhel8-operator@sha256:975d9a16750449b98fe6f40077a68dcef6a902e39e90b829c9a12868c8b47280"
Index Image:
brew.registry.redhat.io/rh-osbs/iib:28108
Quay V3.3 QuayEcosystem CR:
apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
name: mig2-quayecosystem
spec:
quay:
imagePullSecretName: redhat-pull-secret
image: quay.io/quay/quay:v3.3.1-3
externalAccess:
type: LoadBalancer
registryBackends:
- name: s3
s3:
accessKey: ****
bucketName: quayperf
secretKey: ****
host: s3.us-east-2.amazonaws.com
database:
credentialsSecretName: quaydb
server: terraform-20201130015301063700000001.cmqwuswughvh.us-east-2.rds.amazonaws.com
envVars:
- name: DEBUGLOG
value: "true"
clair:
enabled: true
image: quay.io/quay/clair-jwt:v3.3.1-2
imagePullSecretName: redhat-pull-secret
updateInterval: "60m"
Pull image was failed after migration:
lizhang@lzha-mac Downloads % docker pull a9ff79ba67f62488088a5110c62ffaba-494603238.us-east-2.elb.amazonaws.com/qateam/demo
Using default tag: latest
Error response from daemon: received unexpected HTTP status: 503 Service Unavailable
Steps:
- Deploy OCP 4.6 on AWS
- Open OCP console
- Deploy Quay 3.3.2 Operator
- Create Quay CR resource under specified namespace using managed postgresql db and external AWS S3 as storage registry, set quay external access as Loadbalancer(refer to the CR file above)
- Login Quay and create new organization, team, image repository, and push images to above new image repository
- Uninstall Quay 3.3.2 Operator
- Deploy Quay V3.4 TNG Operator with all default settings
- Edit Quay CR QuayEcosystem with adding "quay-operator/migrate": "true" to the metadata.labels of the QuayEcosystem, and save modification
- Wait to check the status of CR QuayRegistry
- Get the Quay V3.4 app endpoint from new loadbalancer and login quay console
- Check the existing image repository
- Delete Quay V3.3 CR
- Pull image from existing image repository
- Create new image repository
- Push image to new image repository
Expected Results:
Step 13 and 15 should complete successfully.
Actual Results:
Step 13 was failed because the existing image repository is still using the old loadbalancer hostname.
Step 15 was failed because new image repository is still using the old loadbalancer hostname.
