Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-12003

3scale-operator continuously logs errors when InstallPlan-related ConfigMap is deleted after CatalogSource removal

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • 2.15.5 GA
    • 3scale Operator
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started

      The 3scale-operator continuously logs "configmaps not found" errors at a high frequency (e.g., every ~17 minutes, or 80+ times per day).

      This occurs because the operator's controllers.Subscription reconciler attempts to retrieve a ConfigMap that is associated with an 3scale's InstallPlan, but this ConfigMap no longer exists.

      Error Log Example:

      2025-07-30T05:20:31.626564231Z {"level":"info","ts":"2025-07-30T05:20:31Z","logger":"controllers.Subscription","msg":"ReconcileSubscription","subscription":{"name":"3scale-operator","namespace":"examplens"},"Failed to get CSV from the latest install plan":"error retrieving ConfigMap examplens/a686ab39dbfd474d340b0c423b8a062354c97b794d84ae76f332b762559aaaa: configmaps \"a686ab39dbfd474d340b0c423b8a062354c97b794d84ae76f332b762559aaaa\" not found"}
      2025-07-30T05:20:31.626624706Z {"level":"error","ts":"2025-07-30T05:20:31Z","msg":"Reconciler error","controller":"subscription","controllerGroup":"operators.coreos.com","controllerKind":"Subscription","Subscription":{"name":"3scale-operator","namespace":"examplens"},"namespace":"examplens","name":"3scale-operator","reconcileID":"f1cdcaea-4b59-4c97-a56c-b38953b37611","error":"error retrieving ConfigMap examplens/a686ab39dbfd474d340b0c423b8a062354c97b794d84ae76f332b762559aaaa: configmaps \"a686ab39dbfd474d340b0c423b8a062354c97b794d84ae76f332b762559aaaa\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}
      

      Cause

      The root cause is the deletion of the CatalogSource (e.g., mirrored-index).

      The ConfigMap referenced by the InstallPlan is owned by the CatalogSource.

      When the user deletes the CatalogSource, Kubernetes garbage collection performs a cascading deletion of all owned resources, including this ConfigMap.

      The 3scale-operator, however, continues to reconcile its Subscription and attempts to access this now-deleted ConfigMap, leading to the persistent error.

      This behavior is triggered by a standard operational procedure for users in disconnected (air-gapped) environments. To update container images from a mirrored repository, they must often delete and recreate the CatalogSource.

      Analysis & Impact

      This appears to be an issue specific to the 3scale-operator's implementation, not a general OLM issue.

      As noted by the customer who reported this issue, the 3scale-operator seems to be looking for information (like MySQL version requirements) from the CSV/ConfigMap. This information is likely static per operator version and may not need to be fetched dynamically from a resource that can be deleted during a normal workflow.

      The current implementation does not seem to account for the (valid) scenario where the InstallPlan's ConfigMap is deleted.

      Impact:
      Log Spam: The operator logs persistent errors.
      Reduced Visibility: This high volume of this error makes it difficult for administrators to identify other, potentially serious errors in the operator logs.

              Unassigned Unassigned
              rhn-support-tkonishi Takayuki Konishi
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: