Uploaded image for project: 'Red Hat Process Automation Manager'
  1. Red Hat Process Automation Manager
  2. RHPAM-2477

Indexing in BC HA repeatedly creates threads which cause BC pod failure [CAT1]

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 7.5.0.GA
    • Fix Version/s: 7.6.0.GA
    • Component/s: None
    • Labels:
    • Environment:
    • Target Release:
    • Fix Build:
      CR1
    • Steps to Reproduce:
      1. Create a space.
      2. Open the space by two users.
      3. Import all sample projects by a user. Or import more samples projects one by one
      4. Check that in log /or in Alerts view in BC is indexing starting repeatedly
    • Affects:
      Release Notes

      Description

      Indexing created new threads repeatedly and it cause that the BC pod runs out of resources. During the indexing user can work with the BC (mainly on other pods), but spaces sometimes cannot be loaded. When BC running out of memory in BC on pod witch indexing is visible exception like this:

      | Uncaught exception: java.lang.OutOfMemoryError:unable to create native thread: possibly out of memory or process/resource limits reached
      

      OR this

       | Uncaught exception: Error parsing JSON: SyntaxError: JSON.parse: unexpected character at line 1 column 2 of the JSON data
      

      So I added to the Deployment config properties for debuging so I could connect with VisualVM

      spec: 
        containers: 
          - env: 
              - name: JAVA_TOOL_OPTIONS
                value: >-
                  -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n
                  -Dcom.sun.management.jmxremote=true
                  -Dcom.sun.management.jmxremote.port=3000
                  -Dcom.sun.management.jmxremote.rmi.port=3001
                  -Djava.rmi.server.hostname=127.0.0.1
                  -Dcom.sun.management.jmxremote.authenticate=false
                  -Dcom.sun.management.jmxremote.ssl=false
      

      With this properties I could forward port to my local and connect VisualVM.
      port forwarding: $ oc port-forward myapp-rhpamcentr-2-lcx5q 8000 3000 3001

      When I opened VisualVM there were around 800 Threads created. After few minutes (5-10) there was over 2500 threads. When the pod failed, run out of memory, there was over 4000 threads.
      I also check thread on other BC pods and there was around 200 threads

      $ oc rsh myapp-rhpamcentr-2-c8lcv
      sh-4.4$ cat /proc/<Process-id>(for this pod = 1042)/status | grep Threads
      Threads:	209
      

      In attachemnt I added log from the indexing pod.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  aparedes Adriel Paredes
                  Reporter:
                  davidredhat David Gutierrez
                  Tester:
                  Tomas David
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: