Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-11256

system-searchd-manticore-reindex pod failing with CreateContainerConfigError with Error as secret "zync" not found

XMLWordPrintable

    • RHOAM Sprint 63, RHOAM Sprint 64, RHOAM Sprint 65, RHOAM Sprint 66
    • Critical

      For 3scale v2.15 deployment on OCP for IBM PPC64LE and IBM Z, facing error as below. Causing deployment to fail and most of resources are not created by apimanager itself.

      system-searchd-manticore-reindex-xxxx is in CreateContainerConfigError with Error
      state:
          waiting:
              message: secret "zync" not found
              reason: CreateContainerConfigError

      # oc describe pod system-searchd-manticore-reindex-zl69g
      Events:  Type     Reason                           Age                    From               Message  ----     ------                           ----                   ----               -------  Normal   Scheduled                        33m                    default-scheduler  Successfully assigned 3scale/system-searchd-manticore-reindex-zl69g to ppc64le-qe55m-mgzn2-master-0  Normal   AddedInterface                   33m                    multus             Add eth0 [10.128.1.97/23] from ovn-kubernetes  Normal   Pulling                          33m                    kubelet            Pulling image "registry.redhat.io/3scale-amp2/system-rhel8@sha256:2b6352c76fa8df172a75ec64201717564024c574f50c1fe35b26a2745fc6dcae"  Normal   Pulled                           33m                    kubelet            Successfully pulled image "registry.redhat.io/3scale-amp2/system-rhel8@sha256:2b6352c76fa8df172a75ec64201717564024c574f50c1fe35b26a2745fc6dcae" in 21.695s (21.695s including waiting)  Warning  Failed                           31m (x8 over 33m)      kubelet            Error: secret "zync" not found  Normal   Pulled                           31m (x7 over 33m)      kubelet            Container image "registry.redhat.io/3scale-amp2/system-rhel8@sha256:2b6352c76fa8df172a75ec64201717564024c574f50c1fe35b26a2745fc6dcae" already present on machine  Warning  FailedToRetrieveImagePullSecret  3m20s (x139 over 33m)  kubelet            Unable to retrieve some image pull secrets (threescale-registry-auth); attempting to pull the image may not succeed.

      Tried adding threescale-registry-auth externally, but still facing same issue.

      All pods in 3scale namespace created, most of pods have not created like system-app and zync related.

      # oc get podsNAME                                                     NAME                                                        READY   STATUS                       RESTARTS      AGE
      backend-cron-66d9cc6d54-49jzb                               1/1     Running                      0             28m
      backend-listener-6dc4cbf8b6-4vdn4                           1/1     Running                      0             28m
      backend-redis-686cdd4f8b-j7gnz                              1/1     Running                      0             28m
      backend-worker-7fc5bc9bc6-dfjkj                             1/1     Running                      0             28m
      system-memcache-7b8c5fb9cb-bb6mf                            1/1     Running                      0             28m
      system-mysql-6b84cffc69-zg28q                               1/1     Running                      0             28m
      system-redis-5b9ff7c46-g59gr                                1/1     Running                      0             28m
      system-searchd-76867cf495-kvvlc                             1/1     Running                      0             28m
      system-searchd-manticore-reindex-zl69g                      0/1     CreateContainerConfigError   0             26m
      threescale-operator-controller-manager-v2-7b659dfdb-86h7x   1/1     Running                      1 (27m ago)   28m

      also, same observation for other resources.

      1. oc get jobs

      NAME                               COMPLETIONS   DURATION   AGE

      system-searchd-manticore-reindex   0/1           37m        37m

      Also, threescale-operator-controller-manager showing following error.

      1. oc logs threescale-operator-controller-manager-v2-7b659dfdb-86h7x | grep error
        "level":"error","ts":"2024-08-20T13:14:52Z","msg":"Reconciler error","controller":"apimanager","controllerGroup":"apps.3scale.net","controllerKind":"APIManager","APIManager": {"name":"example-apimanager","namespace":"3scale"},"namespace":"3scale","name":"example-apimanager","reconcileID":"cba65078-fd02-4817-9b30-5d29c9032d65","error":"Job.batch \"system-app-pre\" is invalid: spec.template.spec.containers[0].volumeMounts[0].mountPath: Required value","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"

       

            Unassigned Unassigned
            siprabhu Siddhesh Prabhu
            Matej Dujava Matej Dujava
            Carl Kyrillos Carl Kyrillos
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: