Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-12261

Persistent Memory Leak in system-searchd (Manticore/Sphinx) - 3scale 2.15+

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 2.15.3 GA
    • System
    • False
    • Hide

      None

      Show
      None
    • False
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Critical

      We are observing a persistent, linear memory growth in the system-searchd component across multiple customer environments. Standard troubleshooting (reindexing, increasing limits) provides only temporary relief before memory usage hits the limits again.

      Memory grows until ThreescaleContainerMemoryHigh alerts are triggered or the pod is OOMKilled.

      • 3scale 2.15.3 is installed on premise on OpenShift 4.16.
      • pod are beeing killed due to OOMKilled errors
      • Hard Dependency: Disabling system-searchd would be an acceptable workaround for the customer as they do not use the search functionality; however, it is currently not a feasible solution. Internal testing confirms that scaling to 0 replicas breaks the system-sidekiq connection, triggering a ThinkingSphinx::ConnectionError and disrupting core background processing.
      • Memory Growth: Even with limits increased to 800Mi (default is 512Mi), the memory trend is strictly upward.
      • Reindexing: Running the system-searchd-manticore-reindex job only provides a ~10-15% drop in usage before the growth resumes.

      Environment Details:
      Product: Red Hat 3scale API Management
      Version: 2.15.x
      Network Plugin: OVN-Kubernetes
      Scale: Low volume (approx. 27 products, 20 backends) – indicates the issue is likely a leak, not a capacity problem.

              rhn-support-keprice Kevin Price
              rh-ee-dmansour Djamel-Eddine Mansouri
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: