Uploaded image for project: 'Kogito'
  1. Kogito
  2. KOGITO-7208

A lot of unworked (stuck) jobs in simple load test

    XMLWordPrintable

Details

    • False
    • None
    • False
    • Job Service throughput improvements when executing jobs removing downstream buffer for emitters
    • ---
    • ---
    • 2022 Week 38-40 (from Sept 19), 2022 Week 41-43 (from Oct 10), 2022 Week 44-46 (from Oct 31), 2022 Week 47-49 (from Nov 21)

    Description

      Runtime: quarkus

      Persistance: postgresql

      Prop: kogito.persistence.optimistic.lock=true

      Process (each parallel task is executing async by jobs service):

      We use a simple load test where 500 processes are run in a loop:

       

      for n in {1..500}; do CURL="curl -d '{ \"username\": \"name_${n}\" }' -H \"Content-Type: application/json\" -X POST http://localhost:8080/parallel?businessKey=key_${n}"; eval $CURL; done 

      After execution script there are a lot of stuck jobs in the persistence storage and jobs service doesn't trying to execute it:

      id correlation_id status last_update retries execution_counter scheduled_id payload type priority recipient trigger
      ASYNC:ed58c35f-2c64-45ee-9b42-8e83f4a452dc ASYNC:ed58c35f-2c64-45ee-9b42-8e83f4a452dc SCHEDULED 2022-05-24 08:42:38.839000 0 0 37 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "b387413b-03ba-44de-b51f-5ab84a8bc90e", "processInstanceId": "2afecbf5-68cc-44ea-ad97-f10fc997565f", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/2afecbf5-68cc-44ea-ad97-f10fc997565f/timers/ASYNC:ed58c35f-2c64-45ee-9b42-8e83f4a452dc", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381758177}
      ASYNC:4454da59-98f7-4433-9d74-160620b4e547 ASYNC:4454da59-98f7-4433-9d74-160620b4e547 SCHEDULED 2022-05-24 08:42:51.012000 0 0 702 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "fe100a04-6e86-4b27-802c-230d67668374", "processInstanceId": "ce0b93fe-e630-4d42-8b64-03722efa4dfa", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/ce0b93fe-e630-4d42-8b64-03722efa4dfa/timers/ASYNC:4454da59-98f7-4433-9d74-160620b4e547", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381762591}
      ASYNC:c7652835-7caa-48a5-afe8-9946c7dbad78 ASYNC:c7652835-7caa-48a5-afe8-9946c7dbad78 SCHEDULED 2022-05-24 08:42:51.818000 0 0 759 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "776c164b-dbea-414e-b212-910aa6e79b86", "processInstanceId": "6398396e-2a6a-4cee-869c-4ba680035c77", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/6398396e-2a6a-4cee-869c-4ba680035c77/timers/ASYNC:c7652835-7caa-48a5-afe8-9946c7dbad78", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381762890}
      ASYNC:5aa5eef2-a5a0-42f5-9df6-4bf0545b0f95 ASYNC:5aa5eef2-a5a0-42f5-9df6-4bf0545b0f95 SCHEDULED 2022-05-24 08:42:48.681000 0 0 555 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "a197e07c-1af2-4504-9b6b-a06c34d8565a", "processInstanceId": "7f97174e-bfa5-4832-b812-cfdd51b89cac", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/7f97174e-bfa5-4832-b812-cfdd51b89cac/timers/ASYNC:5aa5eef2-a5a0-42f5-9df6-4bf0545b0f95", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381761276}
      ASYNC:e2f68d02-ab12-4aa2-bb11-97ac88b8060b ASYNC:e2f68d02-ab12-4aa2-bb11-97ac88b8060b SCHEDULED 2022-05-24 08:42:53.646000 0 0 892 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "d40aa160-8472-4a7e-b3ec-99f0eb9cfd08", "processInstanceId": "9f0e7d07-ea52-4216-bd09-816bb16ad460", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/9f0e7d07-ea52-4216-bd09-816bb16ad460/timers/ASYNC:e2f68d02-ab12-4aa2-bb11-97ac88b8060b", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381763704}
      ASYNC:ad8856af-bad4-4b7c-8d59-18d349e17003 ASYNC:ad8856af-bad4-4b7c-8d59-18d349e17003 SCHEDULED 2022-05-24 08:42:54.940000 0 0 990 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "ceee7848-6a97-4e98-b01b-ea04f4e95872", "processInstanceId": "e058f5ee-4df4-4797-b861-9c7082e92310", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/e058f5ee-4df4-4797-b861-9c7082e92310/timers/ASYNC:ad8856af-bad4-4b7c-8d59-18d349e17003", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381764264}
      ASYNC:98e5ca75-f4d3-4d9e-a86a-9e2b4893090e ASYNC:98e5ca75-f4d3-4d9e-a86a-9e2b4893090e SCHEDULED 2022-05-24 08:42:55.695000 0 0 1045 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "5ae43cab-c152-45ec-bad5-6b59cd1493cc", "processInstanceId": "45561fd9-6f27-47f4-9efb-1e0758f0b7b3", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/45561fd9-6f27-47f4-9efb-1e0758f0b7b3/timers/ASYNC:98e5ca75-f4d3-4d9e-a86a-9e2b4893090e", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381764567}
      ASYNC:48067b64-8d8c-41e8-a7dc-2fd7ee590e18 ASYNC:48067b64-8d8c-41e8-a7dc-2fd7ee590e18 SCHEDULED 2022-05-24 08:42:59.910000 0 0 1369 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "43b697c1-e47f-4ab3-b32c-3fb9f68c7aae", "processInstanceId": "3713bc54-35e3-4d69-b060-814b557958ce", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/3713bc54-35e3-4d69-b060-814b557958ce/timers/ASYNC:48067b64-8d8c-41e8-a7dc-2fd7ee590e18", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381766073}
      ASYNC:483286b2-9206-4bf9-b5f9-bdc9bf0005e8 ASYNC:483286b2-9206-4bf9-b5f9-bdc9bf0005e8 SCHEDULED 2022-05-24 08:43:01.874000 0 0 1499 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "4d1d7d66-43a8-47df-bef8-c8dc2d67e44c", "processInstanceId": "8f6e22ad-3d2f-40c3-8f51-b4e3267a3d86", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/8f6e22ad-3d2f-40c3-8f51-b4e3267a3d86/timers/ASYNC:483286b2-9206-4bf9-b5f9-bdc9bf0005e8", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381766703}
      ASYNC:6ccb0706-a408-4d5f-9cc7-838b9e362fcf ASYNC:6ccb0706-a408-4d5f-9cc7-838b9e362fcf SCHEDULED 2022-05-24 08:43:02.782000 0 0 1559 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "6661c0ea-ff68-47d4-90ff-1904ac76fcfe", "processInstanceId": "67ae1f81-8dc2-4043-a2ef-27662e7ca123", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/67ae1f81-8dc2-4043-a2ef-27662e7ca123/timers/ASYNC:6ccb0706-a408-4d5f-9cc7-838b9e362fcf", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381767020}
      ASYNC:f7576af3-1006-4e85-86d1-49c7537e9424 ASYNC:f7576af3-1006-4e85-86d1-49c7537e9424 SCHEDULED 2022-05-24 08:43:02.801000 0 0 1562 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "6661c0ea-ff68-47d4-90ff-1904ac76fcfe", "processInstanceId": "67ae1f81-8dc2-4043-a2ef-27662e7ca123", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/67ae1f81-8dc2-4043-a2ef-27662e7ca123/timers/ASYNC:f7576af3-1006-4e85-86d1-49c7537e9424", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381767021}
      ASYNC:5b6da3b5-a58c-473d-a5bf-aea3df93b143 ASYNC:5b6da3b5-a58c-473d-a5bf-aea3df93b143 SCHEDULED 2022-05-24 08:43:02.929000 0 0 1575 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "03bcb72a-f93d-4b94-9bc7-9ab141533248", "processInstanceId": "a0ca2f98-07bd-48e1-8348-dde7c0563441", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/a0ca2f98-07bd-48e1-8348-dde7c0563441/timers/ASYNC:5b6da3b5-a58c-473d-a5bf-aea3df93b143", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381767104}
      ASYNC:d43f0a99-1b29-4824-9cd3-7155f582c831 ASYNC:d43f0a99-1b29-4824-9cd3-7155f582c831 SCHEDULED 2022-05-24 08:43:02.895000 0 0 1572 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "03bcb72a-f93d-4b94-9bc7-9ab141533248", "processInstanceId": "a0ca2f98-07bd-48e1-8348-dde7c0563441", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/a0ca2f98-07bd-48e1-8348-dde7c0563441/timers/ASYNC:d43f0a99-1b29-4824-9cd3-7155f582c831", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381767104}
      ASYNC:54b745aa-a6ba-4b35-ac77-9bb02b0b1167 ASYNC:54b745aa-a6ba-4b35-ac77-9bb02b0b1167 SCHEDULED 2022-05-24 08:43:04.710000 0 0 1731 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "661db4e7-6ad6-47d2-b418-e1a4a3789b5b", "processInstanceId": "a048e48a-3884-48c2-9e7b-80d0926bcbd7", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/a048e48a-3884-48c2-9e7b-80d0926bcbd7/timers/ASYNC:54b745aa-a6ba-4b35-ac77-9bb02b0b1167", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381767972}
      ASYNC:7ca88a1c-79e7-46bd-8720-9f12c398732d ASYNC:7ca88a1c-79e7-46bd-8720-9f12c398732d SCHEDULED 2022-05-24 08:43:04.488000 0 0 1712 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "e1b14bdf-0edf-431b-ad68-be95361f8884", "processInstanceId": "8f12d7e3-2f9f-4336-94da-2e08478720a4", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/8f12d7e3-2f9f-4336-94da-2e08478720a4/timers/ASYNC:7ca88a1c-79e7-46bd-8720-9f12c398732d", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381767877}
      ASYNC:56fbb9fe-17b4-403e-91d0-af3a7ed06368 ASYNC:56fbb9fe-17b4-403e-91d0-af3a7ed06368 SCHEDULED 2022-05-24 08:43:04.731000 0 0 1732 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "661db4e7-6ad6-47d2-b418-e1a4a3789b5b", "processInstanceId": "a048e48a-3884-48c2-9e7b-80d0926bcbd7", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/a048e48a-3884-48c2-9e7b-80d0926bcbd7/timers/ASYNC:56fbb9fe-17b4-403e-91d0-af3a7ed06368", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381767972}
      ASYNC:e021dac6-c870-4b73-98d1-dcf22e13cb33 ASYNC:e021dac6-c870-4b73-98d1-dcf22e13cb33 SCHEDULED 2022-05-24 08:43:04.710000 0 0 1729 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "661db4e7-6ad6-47d2-b418-e1a4a3789b5b", "processInstanceId": "a048e48a-3884-48c2-9e7b-80d0926bcbd7", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/a048e48a-3884-48c2-9e7b-80d0926bcbd7/timers/ASYNC:e021dac6-c870-4b73-98d1-dcf22e13cb33", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381767972}
      ASYNC:ad92f255-c6db-4e16-93cd-f8a1bb5ef3fe ASYNC:ad92f255-c6db-4e16-93cd-f8a1bb5ef3fe SCHEDULED 2022-05-24 08:43:04.923000 0 0 1755 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "7dc3b0f3-c073-4dde-961a-3b9fa6e1ed68", "processInstanceId": "c884536f-ac1b-49c6-adbb-872685819063", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/c884536f-ac1b-49c6-adbb-872685819063/timers/ASYNC:ad92f255-c6db-4e16-93cd-f8a1bb5ef3fe", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381768112}
      ASYNC:5d2fb39d-1381-4280-8262-4c741de47bf2 ASYNC:5d2fb39d-1381-4280-8262-4c741de47bf2 SCHEDULED 2022-05-24 08:43:04.922000 0 0 1756 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "7dc3b0f3-c073-4dde-961a-3b9fa6e1ed68", "processInstanceId": "c884536f-ac1b-49c6-adbb-872685819063", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/c884536f-ac1b-49c6-adbb-872685819063/timers/ASYNC:5d2fb39d-1381-4280-8262-4c741de47bf2", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381768112}
      ASYNC:26fcf31d-e166-4bc0-a3a9-a92865e5678e ASYNC:26fcf31d-e166-4bc0-a3a9-a92865e5678e SCHEDULED 2022-05-24 08:43:06.084000 0 0 1888 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "feb734fc-46f1-41b3-9fdf-d53c659cff3d", "processInstanceId": "279616e5-acf0-44dc-a73c-1a77360078e5", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/279616e5-acf0-44dc-a73c-1a77360078e5/timers/ASYNC:26fcf31d-e166-4bc0-a3a9-a92865e5678e", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381768752}
      ASYNC:d6520c3d-345d-4a5b-809d-bd62e9b2c7cd ASYNC:d6520c3d-345d-4a5b-809d-bd62e9b2c7cd SCHEDULED 2022-05-24 08:43:10.509000 0 0 2314 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "7854b245-d778-4d8e-9c29-c3d54ec057c8", "processInstanceId": "e790cbdf-b297-4ee8-aecd-d1db05eeec10", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/e790cbdf-b297-4ee8-aecd-d1db05eeec10/timers/ASYNC:d6520c3d-345d-4a5b-809d-bd62e9b2c7cd", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381771224}
      ASYNC:9692a09b-426d-46f6-8144-6c7f8c1a31d4 ASYNC:9692a09b-426d-46f6-8144-6c7f8c1a31d4 SCHEDULED 2022-05-24 08:43:11.188000 0 0 2383 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "f7e1724b-db23-4cf1-9e4b-53d26990baf2", "processInstanceId": "60f2b59d-c30d-44d4-b331-8b3fa377b903", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/60f2b59d-c30d-44d4-b331-8b3fa377b903/timers/ASYNC:9692a09b-426d-46f6-8144-6c7f8c1a31d4", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381771568}
      ASYNC:a629748d-016d-4915-a483-c2866208b686 ASYNC:a629748d-016d-4915-a483-c2866208b686 SCHEDULED 2022-05-24 08:43:11.737000 0 0 2418 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "2b86363b-3d25-45d5-90fd-8232854a4509", "processInstanceId": "0b6168f2-270f-44d7-8fca-b23a7fe10cc2", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/0b6168f2-270f-44d7-8fca-b23a7fe10cc2/timers/ASYNC:a629748d-016d-4915-a483-c2866208b686", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381771744}
      ASYNC:42244d6c-01f9-4b89-82f2-c5d129634e4f ASYNC:42244d6c-01f9-4b89-82f2-c5d129634e4f SCHEDULED 2022-05-24 08:43:13.269000 0 0 2549 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "2deb2020-8c57-49e2-b72f-3cfb52bf5dff", "processInstanceId": "45baa171-a9f1-4c88-a30c-dc2fc1c8c8f0", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/45baa171-a9f1-4c88-a30c-dc2fc1c8c8f0/timers/ASYNC:42244d6c-01f9-4b89-82f2-c5d129634e4f", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381772428}
      ASYNC:fca9cfe6-eecd-4bd1-b15d-8f87316a009e ASYNC:fca9cfe6-eecd-4bd1-b15d-8f87316a009e SCHEDULED 2022-05-24 08:43:13.330000 0 0 2553 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "aaf4ec4c-0e21-4a52-93aa-b7996ce3c3be", "processInstanceId": "3f5b1270-de0e-4abc-bda0-2de47561e888", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/3f5b1270-de0e-4abc-bda0-2de47561e888/timers/ASYNC:fca9cfe6-eecd-4bd1-b15d-8f87316a009e", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381772478}
      ASYNC:e2aaed06-5fa8-40a0-96df-49614ac37049 ASYNC:e2aaed06-5fa8-40a0-96df-49614ac37049 SCHEDULED 2022-05-24 08:43:14.642000 0 0 2676 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "9fbb8439-be02-4aca-8fbd-410794c014ad", "processInstanceId": "d14227a4-536a-49bc-a87c-58d73adc4183", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/d14227a4-536a-49bc-a87c-58d73adc4183/timers/ASYNC:e2aaed06-5fa8-40a0-96df-49614ac37049", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381773195}
      ASYNC:f4c46bab-6244-4137-8db1-9bf28f5faaf0 ASYNC:f4c46bab-6244-4137-8db1-9bf28f5faaf0 SCHEDULED 2022-05-24 08:43:15.207000 0 0 2725 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "6f6de70a-c271-48ff-bac4-648536353f5e", "processInstanceId": "84992e78-280c-480a-9335-ca0984355ec8", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/84992e78-280c-480a-9335-ca0984355ec8/timers/ASYNC:f4c46bab-6244-4137-8db1-9bf28f5faaf0", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381773444}
      ASYNC:03cc490f-4ea5-45cd-a679-44d744ca1bae ASYNC:03cc490f-4ea5-45cd-a679-44d744ca1bae SCHEDULED 2022-05-24 08:43:15.136000 0 0 2715 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "805365e8-48cb-4e71-91b1-ef31a9bbd601", "processInstanceId": "feee6860-9032-445d-9037-21d856c5f0ab", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/feee6860-9032-445d-9037-21d856c5f0ab/timers/ASYNC:03cc490f-4ea5-45cd-a679-44d744ca1bae", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381773405}
      ASYNC:3e2edf79-a47e-45e1-85ef-30cdd99b75d3 ASYNC:3e2edf79-a47e-45e1-85ef-30cdd99b75d3 SCHEDULED 2022-05-24 08:43:16.651000 0 0 2861 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "9bbb045f-b155-47e0-8c80-9e0cf41a9ca4", "processInstanceId": "e29113a6-b9d8-444c-a05b-0badedcbdb1f", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/e29113a6-b9d8-444c-a05b-0badedcbdb1f/timers/ASYNC:3e2edf79-a47e-45e1-85ef-30cdd99b75d3", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381774204}
      ASYNC:b6b82b20-ae03-4ca7-80e2-62db24761a6a ASYNC:b6b82b20-ae03-4ca7-80e2-62db24761a6a SCHEDULED 2022-05-24 08:43:16.640000 0 0 2859 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "9bbb045f-b155-47e0-8c80-9e0cf41a9ca4", "processInstanceId": "e29113a6-b9d8-444c-a05b-0badedcbdb1f", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/e29113a6-b9d8-444c-a05b-0badedcbdb1f/timers/ASYNC:b6b82b20-ae03-4ca7-80e2-62db24761a6a", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381774204}
      ASYNC:37a31f2a-f6e9-4adb-a90a-0396b7096ac5 ASYNC:37a31f2a-f6e9-4adb-a90a-0396b7096ac5 SCHEDULED 2022-05-24 08:43:16.729000 0 0 2865 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "d54a502e-b7eb-482c-81ac-31cf06bbe1a4", "processInstanceId": "32a37ab0-894d-400f-a55c-70d3a4341fba", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/32a37ab0-894d-400f-a55c-70d3a4341fba/timers/ASYNC:37a31f2a-f6e9-4adb-a90a-0396b7096ac5", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381774261}
      ASYNC:5bc6ae7c-e89c-4d88-b9db-50513e4a8bf5 ASYNC:5bc6ae7c-e89c-4d88-b9db-50513e4a8bf5 SCHEDULED 2022-05-24 08:43:17.223000 0 0 2915 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "e2979178-49ae-4ea8-b2a4-637768599d61", "processInstanceId": "4d4d2b95-2236-411a-828e-109efa6bf890", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/4d4d2b95-2236-411a-828e-109efa6bf890/timers/ASYNC:5bc6ae7c-e89c-4d88-b9db-50513e4a8bf5", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381774504}
      ASYNC:ef41afda-6762-4976-9cbd-0203ad586777 ASYNC:ef41afda-6762-4976-9cbd-0203ad586777 SCHEDULED 2022-05-24 08:43:17.851000 0 0 2976 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "d774d4ec-5e6c-4e43-80e3-487a09c928a6", "processInstanceId": "08e524b3-a97a-416a-801e-79e168fdfa47", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/08e524b3-a97a-416a-801e-79e168fdfa47/timers/ASYNC:ef41afda-6762-4976-9cbd-0203ad586777", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381774864}
      ASYNC:8d746c8d-9ab3-4fe5-9cb0-407f9235420b ASYNC:8d746c8d-9ab3-4fe5-9cb0-407f9235420b SCHEDULED 2022-05-24 08:43:18.978000 0 0 3087 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "22f1950f-b3b0-4012-9221-1e7b77a2b3cb", "processInstanceId": "14849f42-c6bb-4dbc-b37b-b1715b01ec87", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/14849f42-c6bb-4dbc-b37b-b1715b01ec87/timers/ASYNC:8d746c8d-9ab3-4fe5-9cb0-407f9235420b", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381775392}
      ASYNC:1e738399-39d4-43ba-bf0e-ab1cf8415fec ASYNC:1e738399-39d4-43ba-bf0e-ab1cf8415fec SCHEDULED 2022-05-24 08:43:18.838000 0 0 3073 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "0923f02b-c449-4e05-b6c6-748a827c045b", "processInstanceId": "f1a0df13-4633-4778-93d1-97c0ceb76022", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/f1a0df13-4633-4778-93d1-97c0ceb76022/timers/ASYNC:1e738399-39d4-43ba-bf0e-ab1cf8415fec", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381775356}
      ASYNC:1752f4e7-a1cb-4573-9ce8-db2c7e690c4a ASYNC:1752f4e7-a1cb-4573-9ce8-db2c7e690c4a SCHEDULED 2022-05-24 08:43:19.769000 0 0 3170 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "b42606d9-23f0-41cc-869a-4ee03aad99b4", "processInstanceId": "fc8a495d-ceff-43a3-ae8f-89d58184b6c4", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/fc8a495d-ceff-43a3-ae8f-89d58184b6c4/timers/ASYNC:1752f4e7-a1cb-4573-9ce8-db2c7e690c4a", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381775875}
      ASYNC:0ec2d3c8-22b1-4c12-ba82-394f4152c2ef ASYNC:0ec2d3c8-22b1-4c12-ba82-394f4152c2ef SCHEDULED 2022-05-24 08:43:21.164000 0 0 3307 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "b936fe70-94fb-48c5-9c36-34b073f26c3c", "processInstanceId": "91a74ffc-8182-4919-a7f5-f2fa0176dee0", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/91a74ffc-8182-4919-a7f5-f2fa0176dee0/timers/ASYNC:0ec2d3c8-22b1-4c12-ba82-394f4152c2ef", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381776619}
      ASYNC:ba64882f-d79e-41e9-8bfb-8c020a44c79b ASYNC:ba64882f-d79e-41e9-8bfb-8c020a44c79b SCHEDULED 2022-05-24 08:43:22.773000 0 0 3456 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "893698af-0ec6-4a9b-ba98-29f60f5b3bf6", "processInstanceId": "5f98c5ea-2ce1-4d1e-911a-a97f8d9eaca7", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/5f98c5ea-2ce1-4d1e-911a-a97f8d9eaca7/timers/ASYNC:ba64882f-d79e-41e9-8bfb-8c020a44c79b", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381778345}
      ASYNC:0b5b7c08-38eb-4fa2-9976-acdd1606f8be ASYNC:0b5b7c08-38eb-4fa2-9976-acdd1606f8be SCHEDULED 2022-05-24 08:43:22.875000 0 0 3465 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "3285bc98-d73c-4a2f-ad1f-07df768a3559", "processInstanceId": "c82cf41d-16f7-4f31-8b56-8dc27dced03d", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/c82cf41d-16f7-4f31-8b56-8dc27dced03d/timers/ASYNC:0b5b7c08-38eb-4fa2-9976-acdd1606f8be", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381778392}
      ASYNC:76c0f056-d9c8-4796-a131-c8739809fd20 ASYNC:76c0f056-d9c8-4796-a131-c8739809fd20 SCHEDULED 2022-05-24 08:43:25.884000 0 0 3764 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "4c306e1c-ab29-4586-8ddf-10c0e23c1868", "processInstanceId": "70d27e88-ac89-484f-a42a-b04c6429ab41", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/70d27e88-ac89-484f-a42a-b04c6429ab41/timers/ASYNC:76c0f056-d9c8-4796-a131-c8739809fd20", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381780281}
      ASYNC:1c9df48b-8695-427a-9e1a-60a55c536b6b ASYNC:1c9df48b-8695-427a-9e1a-60a55c536b6b SCHEDULED 2022-05-24 08:43:26.932000 0 0 3882 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "c1f47ce1-24be-48de-b452-608380f9f1a7", "processInstanceId": "bec4a5b8-40a5-48f7-a847-3fdcad1fbd41", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/bec4a5b8-40a5-48f7-a847-3fdcad1fbd41/timers/ASYNC:1c9df48b-8695-427a-9e1a-60a55c536b6b", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381781005}
      ASYNC:682065af-2fb1-41f8-b9a2-1c51b0c28190 ASYNC:682065af-2fb1-41f8-b9a2-1c51b0c28190 SCHEDULED 2022-05-24 08:43:27.625000 0 0 3940 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "72fd922e-8f2e-4806-8167-f0ba631e3c19", "processInstanceId": "406fe0eb-6efe-485a-bc9c-b9c80b442c32", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/406fe0eb-6efe-485a-bc9c-b9c80b442c32/timers/ASYNC:682065af-2fb1-41f8-b9a2-1c51b0c28190", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381781363}
      ASYNC:6bf3796f-59f9-489b-a46f-8c6ade912a53 ASYNC:6bf3796f-59f9-489b-a46f-8c6ade912a53 SCHEDULED 2022-05-24 08:43:28.420000 0 0 4007 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "e567f2c1-1048-4005-9eaa-d5f63a144ee0", "processInstanceId": "5a0d3f31-4faa-4204-b03f-db4a34c55f06", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/5a0d3f31-4faa-4204-b03f-db4a34c55f06/timers/ASYNC:6bf3796f-59f9-489b-a46f-8c6ade912a53", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381781751}
      ASYNC:be84f980-f7ec-40ea-8a44-f98cfca1bab6 ASYNC:be84f980-f7ec-40ea-8a44-f98cfca1bab6 SCHEDULED 2022-05-24 08:43:32.216000 0 0 4406 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "06a18737-87aa-4150-8fbb-6ab28c95ea6c", "processInstanceId": "414ba788-85cb-4bfb-89ba-cd11c80ba1fe", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/414ba788-85cb-4bfb-89ba-cd11c80ba1fe/timers/ASYNC:be84f980-f7ec-40ea-8a44-f98cfca1bab6", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381784086}
      ASYNC:ef5af85e-684c-4ec7-9f71-d85c91d4e194 ASYNC:ef5af85e-684c-4ec7-9f71-d85c91d4e194 SCHEDULED 2022-05-24 08:43:32.301000 0 0 4414 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "df5b887e-58dd-4db9-9ab7-18a77bfff95c", "processInstanceId": "b762227f-823c-4819-a18a-e078d349db25", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/b762227f-823c-4819-a18a-e078d349db25/timers/ASYNC:ef5af85e-684c-4ec7-9f71-d85c91d4e194", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381784153}
      ASYNC:34486743-518d-4249-b1d3-27feb67142dd ASYNC:34486743-518d-4249-b1d3-27feb67142dd SCHEDULED 2022-05-24 08:43:32.352000 0 0 4422 {"processId": "parallel", "rootProcessId": null, "nodeInstanceId": "e50f058b-e449-473c-ae61-2f68aa8d6e46", "processInstanceId": "8335514a-9537-498e-9651-eaa4b01c4611", "rootProcessInstanceId": null} HTTP 0 {"endpoint": "http://host.docker.internal:8080/management/jobs/parallel/instances/8335514a-9537-498e-9651-eaa4b01c4611/timers/ASYNC:34486743-518d-4249-b1d3-27feb67142dd", "classType": "org.kie.kogito.jobs.service.model.job.Recipient$HTTPRecipient"} {"classType": "org.kie.kogito.timer.impl.PointInTimeTrigger", "nextFireTime": 1653381784209}

       

      As a result, there is a large number of stuck processes.

      Logs of the jobs-service in attachment (job-service.zip)

       

      Attachments

        Issue Links

          Activity

            People

              tdolphin-1 Tiago Dolphine
              iurybakov iuriy rybakov (Inactive)
              Marian Macik Marian Macik
              Marian Macik Marian Macik
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: