Uploaded image for project: 'Red Hat Process Automation Manager'
  1. Red Hat Process Automation Manager
  2. RHPAM-2477

Indexing in BC HA repeatedly creates threads which cause BC pod failure [CAT1]

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • 7.6.0.GA
    • 7.5.0.GA
    • None
    • Release Notes
    • CR1
      1. Create a space.
      2. Open the space by two users.
      3. Import all sample projects by a user. Or import more samples projects one by one
      4. Check that in log /or in Alerts view in BC is indexing starting repeatedly

    Description

      Indexing created new threads repeatedly and it cause that the BC pod runs out of resources. During the indexing user can work with the BC (mainly on other pods), but spaces sometimes cannot be loaded. When BC running out of memory in BC on pod witch indexing is visible exception like this: 

      | Uncaught exception: java.lang.OutOfMemoryError:unable to create native thread: possibly out of memory or process/resource limits reached

      OR this

       | Uncaught exception: Error parsing JSON: SyntaxError: JSON.parse: unexpected character at line 1 column 2 of the JSON data

      So I added to the Deployment config properties for debuging so I could connect with VisualVM

      spec:   containers:     - env:         - name: JAVA_TOOL_OPTIONS
          value: >-
            -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=n
            -Dcom.sun.management.jmxremote=true
            -Dcom.sun.management.jmxremote.port=3000
            -Dcom.sun.management.jmxremote.rmi.port=3001
            -Djava.rmi.server.hostname=127.0.0.1
            -Dcom.sun.management.jmxremote.authenticate=false
            -Dcom.sun.management.jmxremote.ssl=false

      With this properties I could forward port to my local and connect VisualVM.
      port forwarding: $ oc port-forward myapp-rhpamcentr-2-lcx5q 8000 3000 3001

      When I opened VisualVM there were around 800 Threads created. After few minutes (5-10) there was over 2500 threads. When the pod failed, run out of memory, there was over 4000 threads.
      I also check thread on other BC pods and there was around 200 threads

      $ oc rsh myapp-rhpamcentr-2-c8lcv
sh-4.4$ cat /proc/<Process-id>(for this pod = 1042)/status | grep Threads
Threads:	209

      In attachemnt I added log from the indexing pod.

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. AF-2078 Indexing in BC HA repeatedly creates threads witch cause BC pod failure [CAT1]

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. AF-2269 Workbench is very slow after several projects are created/imported

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Resolved

          Activity

            People

              aparedes@redhat.com Adriel Paredes
              david.magallanes David Gutierrez
              Tomas David Tomas David
              Tomas David Tomas David
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: