-
Bug
-
Resolution: Done
-
Blocker
-
quay-v3.6.0
-
False
-
False
-
undefined
-
Description:
This is an issue found when use Quay 3.6.0 Operator to deploy quay, now by using all managed components except monitoring Component(set monitoring as false), create QuayRegistry CR, but the deployment was failed, check Quay Operator Pod logs, get error message "could not update QuayRegistry status with current version", see attached Quay Operator POD logs.
Also if choose to deploy Quay Operator to all OCP Namespaces and use all managed Components, this issue can be reproduced again.
Note: Quay Version is quay-operator-bundle-container-v3.6.0-13
Index image:
Index image v4.8: registry-proxy.engineering.redhat.com/rh-osbs/iib:98138
Quay Operator Pog logs:
oc get pod NAME READY STATUS RESTARTS AGE quay-clair-app-5b957f5cbf-q48km 1/1 Running 0 7m21s quay-clair-app-5b957f5cbf-wtrzl 1/1 Running 0 14m quay-clair-postgres-d5f98f4f4-vjdt5 1/1 Running 1 15m quay-operator.v3.6.0-7b69b6f96c-drkz8 1/1 Running 0 16m quay-quay-app-upgrade-c2954 0/1 Completed 0 15m quay-quay-config-editor-696688c8f4-f9qn7 1/1 Running 0 14m quay-quay-database-7f99c58988-dpcl5 1/1 Running 1 14m quay-quay-mirror-7974b467b-qg9m6 0/1 Init:0/1 5 14m quay-quay-mirror-7974b467b-zq8lw 0/1 Init:0/1 5 14m quay-quay-postgres-init-vnn7g 0/1 Completed 0 14m quay-quay-redis-5f5bbcdcd8-smh7z 1/1 Running 0 15m oc get pod quay-operator.v3.6.0-7b69b6f96c-drkz8 -o json | jq '.spec.containers[0].image' "registry.redhat.io/quay/quay-operator-rhel8@sha256:4f14b042ae9451e361f783bb4721f63150a92b1e1a60e989a179f420f3f15d3c" 2021-08-16T06:32:53.155Z INFO controllers.QuayRegistry Quay upgrade complete, updating `status.currentVersion` {"quayregistry": "quay3600816/quay"} 2021-08-16T06:32:53.155Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"QuayRegistry","namespace":"quay3600816","name":"quay","uid":"f8adf91c-4bdf-4419-a6f5-e84324942cb6","apiVersion":"quay.redhat.com/v1","resourceVersion":"127332"}, "reason": "HealthChecksPassing", "message": "all registry component healthchecks passing"} 2021-08-16T06:32:53.163Z ERROR controllers.QuayRegistry could not update QuayRegistry status with current version {"quayregistry": "quay3600816/quay", "error": "Operation cannot be fulfilled on quayregistries.quay.redhat.com \"quay\": the object has been modified; please apply your changes to the latest version and try again"} github.com/go-logr/zapr.(*zapLogger).Error /workspace/vendor/github.com/go-logr/zapr/zapr.go:132 github.com/quay/quay-operator/controllers/quay.(*QuayRegistryReconciler).Reconcile.func1.1 /workspace/controllers/quay/quayregistry_controller.go:366 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:211 k8s.io/apimachinery/pkg/util/wait.WaitFor /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:541 k8s.io/apimachinery/pkg/util/wait.pollInternal /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:427 k8s.io/apimachinery/pkg/util/wait.Poll /workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:421 github.com/quay/quay-operator/controllers/quay.(*QuayRegistryReconciler).Reconcile.func1 /workspace/controllers/quay/quayregistry_controller.go:337 2021-08-16T06:32:53.163Z ERROR controllers.QuayRegistry Quay upgrade `Job` never completed {"quayregistry": "quay3600816/quay", "error": "Operation cannot be fulfilled on quayregistries.quay.redhat.com \"quay\": the object has been modified; please apply your changes to the latest version and try again"} github.com/go-logr/zapr.(*zapLogger).Error /workspace/vendor/github.com/go-logr/zapr/zapr.go:132 github.com/quay/quay-operator/controllers/quay.(*QuayRegistryReconciler).Reconcile.func1 /workspace/controllers/quay/quayregistry_controller.go:387 oc get quayregistry quay -o yaml apiVersion: quay.redhat.com/v1 kind: QuayRegistry metadata: creationTimestamp: "2021-08-16T06:31:37Z" finalizers: - quay-operator/finalizer generation: 3 managedFields: - apiVersion: quay.redhat.com/v1 fieldsType: FieldsV1 fieldsV1: f:spec: .: {} f:components: {} f:configBundleSecret: {} manager: Mozilla operation: Update time: "2021-08-16T06:31:37Z" - apiVersion: quay.redhat.com/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: {} v:"quay-operator/finalizer": {} f:spec: f:components: {} f:configBundleSecret: {} f:status: .: {} f:conditions: {} f:configEditorCredentialsSecret: {} f:configEditorEndpoint: {} f:lastUpdated: {} f:unhealthyComponents: .: {} f:mirror: {} manager: manager operation: Update time: "2021-08-16T06:32:38Z" name: quay namespace: quay3600816 resourceVersion: "128194" uid: f8adf91c-4bdf-4419-a6f5-e84324942cb6 spec: components: - kind: clair managed: true - kind: postgres managed: true - kind: objectstorage managed: true - kind: redis managed: true - kind: horizontalpodautoscaler managed: true - kind: route managed: true - kind: mirror managed: true - kind: monitoring managed: false - kind: tls managed: true configBundleSecret: quay-config-bundle-hzxwd status: conditions: - lastTransitionTime: "2021-08-16T06:32:21Z" lastUpdateTime: "2021-08-16T06:32:21Z" message: all objects created/updated successfully reason: ComponentsCreationSuccess status: "False" type: RolloutBlocked - lastTransitionTime: "2021-08-16T06:32:22Z" lastUpdateTime: "2021-08-16T06:32:22Z" message: running database migrations reason: MigrationsInProgress status: "False" type: Available configEditorCredentialsSecret: quay-quay-config-editor-credentials-57m9bbc85k configEditorEndpoint: https://quay-quay-config-editor-quay3600816.router-default.apps.quay-perf-738.perfscale.devcluster.openshift.com lastUpdated: 2021-08-16 06:32:22.13910204 +0000 UTC unhealthyComponents: mirror: - lastTransitionTime: "2021-08-16T06:31:48Z" lastUpdateTime: "2021-08-16T06:31:48Z" message: 'Deployment quay-quay-mirror: Deployment does not have minimum availability.' reason: MinimumReplicasUnavailable status: "False" type: Available