Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: quay-v3.4.0
Affects Version/s: quay-v3.4.0
Component/s: quay-operator
Labels:
- quay-enterprise
- triaged

Blocked:
False
Ready:
False
Release Note Text:
Undefined
Git Pull Request:
https://github.com/quay/quay-operator/pull/383
Market:

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description:

This is an issue found when use Quay TNG Operator to migrate QuayEcosystem CR, when Quay V3.3 QuayEcosystem use managed postgresql database, the migration was failed and TNG Operator report error "panic: runtime error: index out of range [0] with length 0 [recovered]", see attached TNG Operator Pod logs.

Docs: https://docs.projectquay.io/deploy_quay_on_openshift_op_tng.html#_upgrading_quay_using_the_quay_operator

Quay POD Status:

lizhang@lzha-mac Quay3.3_operator_testing % oc get pod
NAME                                                              READY   STATUS        RESTARTS   AGE
quay-operator.v3.4.0-6c477dc458-zwbds                             1/1     Running       1          2m35s
quay1201-quayecosystem-clair-78695579bb-pg6d6                     1/1     Running       0          26m
quay1201-quayecosystem-clair-postgresql-85c5c45b6d-wz48k          1/1     Running       0          27m
quay1201-quayecosystem-quay-749fd85cd-jg552                       1/1     Running       0          28m
quay1201-quayecosystem-quay-config-b577b987f-vnpbb                1/1     Running       0          28m
quay1201-quayecosystem-quay-postgres-migration-776c5495b9-ns8ld   0/2     Init:0/1      0          12s
quay1201-quayecosystem-quay-postgresql-7464886ff5-fshvb           1/1     Running       0          17s
quay1201-quayecosystem-quay-postgresql-74795cd6c7-6cz6c           1/1     Terminating   0          29m
quay1201-quayecosystem-quay-repomirror-b6b9c76f5-vb82c            1/1     Running       0          27m
quay1201-quayecosystem-redis-d57dbd4bc-hm8gx                      1/1     Running       0          29m

lizhang@lzha-mac Quay3.3_operator_testing % oc get pod
NAME                                                       READY   STATUS    RESTARTS   AGE
quay-operator.v3.4.0-6c477dc458-zwbds                      1/1     Running   3          4m18s
quay1201-quayecosystem-clair-78695579bb-pg6d6              1/1     Running   0          28m
quay1201-quayecosystem-clair-postgresql-85c5c45b6d-wz48k   1/1     Running   0          29m
quay1201-quayecosystem-quay-749fd85cd-jg552                1/1     Running   0          29m
quay1201-quayecosystem-quay-config-b577b987f-vnpbb         1/1     Running   0          30m
quay1201-quayecosystem-quay-postgresql-7464886ff5-fshvb    1/1     Running   0          2m
quay1201-quayecosystem-quay-repomirror-b6b9c76f5-vb82c     1/1     Running   0          29m
quay1201-quayecosystem-redis-d57dbd4bc-hm8gx               1/1     Running   0          31m

TNG Operator POD Logs:

2020-12-18T07:26:51.144Z	INFO	controllers.QuayEcosystem	checking if database migration `Deployment` completed	{"quayecosystem": "quayv330/quay1201-quayecosystem"}
E1218 07:26:51.244906       1 runtime.go:78] Observed a panic: runtime.boundsError{x:0, y:0, signed:true, code:0x0} (runtime error: index out of range [0] with length 0)
goroutine 407 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x228fec0, 0xc000689ba0)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
panic(0x228fec0, 0xc000689ba0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/runtime/panic.go:969 +0x175
github.com/quay/quay-operator/controllers/redhatcop.(*QuayEcosystemReconciler).Reconcile.func2(0xc000c8e568, 0xc000c8e558, 0x2)
	/workspace/controllers/redhatcop/quayecosystem_controller.go:479 +0x6ac
k8s.io/apimachinery/pkg/util/wait.WaitFor(0xc000689b20, 0xc000c8ed08, 0xc000c52960, 0x0, 0x0)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:434 +0x142
k8s.io/apimachinery/pkg/util/wait.pollInternal(0xc000689b20, 0xc000c8ed08, 0x0, 0x0)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:320 +0x87
k8s.io/apimachinery/pkg/util/wait.Poll(0x12a05f200, 0x8bb2c97000, 0xc000c8ed08, 0xc0006f5d00, 0x2)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x4d
github.com/quay/quay-operator/controllers/redhatcop.(*QuayEcosystemReconciler).Reconcile(0xc000537620, 0xc00080ea60, 0x8, 0xc000712860, 0x16, 0xaf3fa5f7, 0xc0008b2630, 0xc0006c4878, 0xc0006c4870)
	/workspace/controllers/redhatcop/quayecosystem_controller.go:455 +0x4b94
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000228240, 0x21e6500, 0xc00076de20, 0x22d1d00)
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000228240, 0x25fe600)
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000228240)
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000295340)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000295340, 0x3b9aca00, 0x0, 0xc00069e001, 0xc0003ce000)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc000295340, 0x3b9aca00, 0xc0003ce000)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x32d
panic: runtime error: index out of range [0] with length 0 [recovered]
	panic: runtime error: index out of range [0] with length 0goroutine 407 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x228fec0, 0xc000689ba0)
	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/runtime/panic.go:969 +0x175
github.com/quay/quay-operator/controllers/redhatcop.(*QuayEcosystemReconciler).Reconcile.func2(0xc000c8e568, 0xc000c8e558, 0x2)
	/workspace/controllers/redhatcop/quayecosystem_controller.go:479 +0x6ac
k8s.io/apimachinery/pkg/util/wait.WaitFor(0xc000689b20, 0xc000c8ed08, 0xc000c52960, 0x0, 0x0)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:434 +0x142
k8s.io/apimachinery/pkg/util/wait.pollInternal(0xc000689b20, 0xc000c8ed08, 0x0, 0x0)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:320 +0x87
k8s.io/apimachinery/pkg/util/wait.Poll(0x12a05f200, 0x8bb2c97000, 0xc000c8ed08, 0xc0006f5d00, 0x2)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:314 +0x4d
github.com/quay/quay-operator/controllers/redhatcop.(*QuayEcosystemReconciler).Reconcile(0xc000537620, 0xc00080ea60, 0x8, 0xc000712860, 0x16, 0xaf3fa5f7, 0xc0008b2630, 0xc0006c4878, 0xc0006c4870)
	/workspace/controllers/redhatcop/quayecosystem_controller.go:455 +0x4b94
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000228240, 0x21e6500, 0xc00076de20, 0x22d1d00)
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000228240, 0x25fe600)
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc000228240)
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 +0x2b
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000295340)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000295340, 0x3b9aca00, 0x0, 0xc00069e001, 0xc0003ce000)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc000295340, 0x3b9aca00, 0xc0003ce000)
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x32d

Index image:

registry-proxy.engineering.redhat.com/rh-osbs/iib:30546

Quay TNG Operator Image:

lizhang@lzha-mac Quay3.3_operator_testing % oc get pod quay-operator.v3.4.0-6c477dc458-zwbds -o json | jq '.spec.containers[0].image'
"registry.redhat.io/quay/quay-rhel8-operator@sha256:7bd741d0a83dc89a7f9bda53aa017d3109cd9cc6a5a046bc7fe641fe2b14d4a3"

Quay V3.3 QuayEcosystem CR:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: quay1201-quayecosystem
spec:
  quay:
    imagePullSecretName: redhat-pull-secret
    image: quay.io/quay/quay:v3.3.1-3
    enableRepoMirroring: true
    registryBackends:
      - name: s3
        s3:
          accessKey: *******
          bucketName: quayperf
          secretKey: *******
          host: s3.us-east-2.amazonaws.com
    database:
      volumeSize: 30Gi
    envVars:
      - name: DEBUGLOG
        value: "true"
  clair:
    enabled: true
    image: quay.io/quay/clair-jwt:v3.3.1-2
    imagePullSecretName: redhat-pull-secret
    updateInterval: "60m"

Steps:

Deploy Quay with V3.3.2 Operator
Create new CR and use managed postgresql databse, use AWS S3 and quay registry storage, see CR above.
Login Quay, and create new image repository, organization, robot account, push image to some image repository, etc.
Create root password of managed postgresql database
Create new secret with key "database-root-password", and referenced to "spec.quay.database.credentialsSecretName" with command "oc create secret generic quaydb --from-literal=database-root-password=postgresql "
Upgrade Quay Operator to V3.4.0, refer to https://docs.google.com/presentation/d/1o6nv8VqTOo7ffXHw55ifD8RCZP2zhyBcKNvu7Fg_qQ4/edit#slide=id.ga92726e090_0_6
Update QuayEcosystem CR to have label ""quay-operator/migrate": "true""
Check migration status

Expected Results:

Operator migration completed successfully.

Actual Results:

Migration was failed at database migration.

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

quay_tng_operator_pod_20200107.logs
9 kB
2021/01/06 10:13 PM
quay-tng-operator-pod-v34.logs
9 kB
2020/12/18 2:48 AM

Details

Description

Attachments

Attachments

Easy Agile Planning Poker

Activity

People

Dates