Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-2678

Operator crashes during deplyoment of Quay on CRC 1.33

    XMLWordPrintable

Details

    • False
    • False
    • Quay Enterprise
    • 0

    Description

      Logs from the pod:

      ibazulic@cyberdyne:~$ oc logs quay-operator.v3.5.7-bffbcd6cd-p88q8
      I1012 11:04:32.078185       1 request.go:655] Throttling request took 1.039889923s, request: GET:https://10.217.4.1:443/apis/apiserver.openshift.io/v1?timeout=32s
      2021-10-12T11:04:33.834Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
      2021-10-12T11:04:33.834Z	INFO	controller-runtime.injectors-warning	Injectors are deprecated, and will be removed in v0.10.x
      2021-10-12T11:04:33.834Z	INFO	controller-runtime.injectors-warning	Injectors are deprecated, and will be removed in v0.10.x
      2021-10-12T11:04:33.834Z	INFO	controller-runtime.injectors-warning	Injectors are deprecated, and will be removed in v0.10.x
      2021-10-12T11:04:33.834Z	INFO	controller-runtime.injectors-warning	Injectors are deprecated, and will be removed in v0.10.x
      2021-10-12T11:04:33.834Z	INFO	setup	starting server on port 7071
      2021-10-12T11:04:33.834Z	INFO	setup	starting manager
      2021-10-12T11:04:33.835Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
      2021-10-12T11:04:33.835Z	INFO	controller-runtime.manager.controller.quayecosystem	Starting EventSource	{"reconciler group": "redhatcop.redhat.io", "reconciler kind": "QuayEcosystem", "source": "kind source: /, Kind="}
      2021-10-12T11:04:33.835Z	INFO	controller-runtime.manager.controller.quayregistry	Starting EventSource	{"reconciler group": "quay.redhat.com", "reconciler kind": "QuayRegistry", "source": "kind source: /, Kind="}
      2021-10-12T11:04:33.936Z	INFO	controller-runtime.manager.controller.quayecosystem	Starting Controller	{"reconciler group": "redhatcop.redhat.io", "reconciler kind": "QuayEcosystem"}
      2021-10-12T11:04:33.936Z	INFO	controller-runtime.manager.controller.quayecosystem	Starting workers	{"reconciler group": "redhatcop.redhat.io", "reconciler kind": "QuayEcosystem", "worker count": 1}
      2021-10-12T11:04:33.936Z	INFO	controller-runtime.manager.controller.quayregistry	Starting Controller	{"reconciler group": "quay.redhat.com", "reconciler kind": "QuayRegistry"}
      2021-10-12T11:04:33.936Z	INFO	controller-runtime.manager.controller.quayregistry	Starting workers	{"reconciler group": "quay.redhat.com", "reconciler kind": "QuayRegistry", "worker count": 1}
      2021-10-12T11:04:33.936Z	INFO	controllers.QuayRegistry	begin reconcile	{"quayregistry": "quay-enterprise/example-registry"}
      2021-10-12T11:04:34.337Z	INFO	controllers.QuayRegistry	successfully retrieved referenced `configBundleSecret`	{"quayregistry": "quay-enterprise/example-registry", "configBundleSecret": "quay-config-bundle", "resourceVersion": "48316"}
      2021-10-12T11:04:34.350Z	INFO	controllers.QuayRegistry	cluster supports `Routes` API
      2021-10-12T11:04:34.951Z	INFO	controllers.QuayRegistry	Detected cluster hostname apps-crc.testing
      2021-10-12T11:04:34.951Z	INFO	controllers.QuayRegistry	detected router canonical hostname: apps-crc.testing
      2021-10-12T11:04:38.131Z	INFO	controllers.QuayRegistry	cluster does not support `ObjectBucketClaim` API
      2021-10-12T11:04:38.231Z	INFO	controllers.QuayRegistry	cluster supports `ServiceMonitor` API
      2021-10-12T11:04:38.332Z	INFO	controllers.QuayRegistry	cluster supports `PrometheusRules` API
      2021-10-12T11:04:38.433Z	INFO	controllers.QuayRegistry	openshift-config-managed found
      2021-10-12T11:04:39.440Z	INFO	controllers.QuayRegistry	inflating QuayRegistry into Kubernetes objects using Kustomize	{"quayregistry": "quay-enterprise/example-registry"}
      2021-10-12T11:04:39.440Z	INFO	controllers.QuayRegistry	`DATABASE_SECRET_KEY` not found in user-provided config, generating new one	{"quayregistry": "quay-enterprise/example-registry"}
      2021-10-12T11:04:39.440Z	INFO	controllers.QuayRegistry	`SECRET_KEY` not found in user-provided config, generating new one	{"quayregistry": "quay-enterprise/example-registry"}
      E1012 11:04:39.440444       1 runtime.go:78] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
      goroutine 436 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic(0x27f1a80, 0x2d477e0)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x95
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x89
      panic(0x27f1a80, 0x2d477e0)
      	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/runtime/panic.go:969 +0x1b9
      github.com/quay/quay-operator/pkg/kustomize.Inflate(0xc000c3a160, 0xc0067e9dc0, 0xc000810000, 0x2dd17e0, 0xc000654150, 0x0, 0x0, 0xd, 0xc0069780e0, 0x6c)
      	/workspace/pkg/kustomize/kustomize.go:443 +0x171
      github.com/quay/quay-operator/controllers/quay.(*QuayRegistryReconciler).Reconcile(0xc00061c000, 0x2dc4b20, 0xc000566150, 0xc00012d5d0, 0xf, 0xc00012d5b0, 0x10, 0xc000566150, 0xdd63ff, 0xc00003c000, ...)
      	/workspace/controllers/quay/quayregistry_controller.go:269 +0x22e7
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000021180, 0x2dc4a60, 0xc000894000, 0x2878360, 0xc00000eda0)
      	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:297 +0x317
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000021180, 0x2dc4a60, 0xc000894000, 0xc00062be00)
      	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252 +0x205
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x2dc4a60, 0xc000894000)
      	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 +0x4a
      k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185 +0x37
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00062bf50)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc006107f50, 0x2d72d00, 0xc0005660c0, 0xc000894001, 0xc000896120)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00062bf50, 0x3b9aca00, 0x0, 0x1, 0xc000896120)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
      k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x2dc4a60, 0xc000894000, 0xc0006540f0, 0x3b9aca00, 0x0, 0x1)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185 +0xa6
      k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x2dc4a60, 0xc000894000, 0xc0006540f0, 0x3b9aca00)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99 +0x57
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
      	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:212 +0x45e
      panic: assignment to entry in nil map [recovered]
      	panic: assignment to entry in nil map
      ​
      goroutine 436 [running]:
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x10c
      panic(0x27f1a80, 0x2d477e0)
      	/opt/rh/go-toolset-1.15/root/usr/lib/go-toolset-1.15-golang/src/runtime/panic.go:969 +0x1b9
      github.com/quay/quay-operator/pkg/kustomize.Inflate(0xc000c3a160, 0xc0067e9dc0, 0xc000810000, 0x2dd17e0, 0xc000654150, 0x0, 0x0, 0xd, 0xc0069780e0, 0x6c)
      	/workspace/pkg/kustomize/kustomize.go:443 +0x171
      github.com/quay/quay-operator/controllers/quay.(*QuayRegistryReconciler).Reconcile(0xc00061c000, 0x2dc4b20, 0xc000566150, 0xc00012d5d0, 0xf, 0xc00012d5b0, 0x10, 0xc000566150, 0xdd63ff, 0xc00003c000, ...)
      	/workspace/controllers/quay/quayregistry_controller.go:269 +0x22e7
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000021180, 0x2dc4a60, 0xc000894000, 0x2878360, 0xc00000eda0)
      	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:297 +0x317
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000021180, 0x2dc4a60, 0xc000894000, 0xc00062be00)
      	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:252 +0x205
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x2dc4a60, 0xc000894000)
      	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 +0x4a
      k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185 +0x37
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc00062bf50)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
      k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc006107f50, 0x2d72d00, 0xc0005660c0, 0xc000894001, 0xc000896120)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00062bf50, 0x3b9aca00, 0x0, 0x1, 0xc000896120)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
      k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x2dc4a60, 0xc000894000, 0xc0006540f0, 0x3b9aca00, 0x0, 0x1)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:185 +0xa6
      k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x2dc4a60, 0xc000894000, 0xc0006540f0, 0x3b9aca00)
      	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:99 +0x57
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
      	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:212 +0x45e
      

      I've used the most basic config.yaml file that has nothing except my storage definition:

      DISTRIBUTED_STORAGE_CONFIG: 
          default: 
              - RadosGWStorage
              - access_key: minioadmin
                bucket_name: quay-openshift
                hostname: 172.24.0.1
                is_secure: true
                port: "9000"
                secret_key: minioadmin
                storage_path: /datastorage/registry
      DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: 
          - default
      DISTRIBUTED_STORAGE_PREFERENCE: 
          - default
      

      The storage definition works on Quay standalone and I don't think it's the cause of the problem since the logs show that my config was read and failure is somewhere else.

      Attachments

        Activity

          People

            Unassigned Unassigned
            rhn-support-ibazulic Ivan Bazulic
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: