Uploaded image for project: 'Kogito'
  1. Kogito
  2. KOGITO-943

Jobs Service: Random failure on startup for Infinispan and load of scheduled jobs

    XMLWordPrintable

Details

    • Hide

      Repeat as long as you don't have the issuer, as it is random ...

      Start infinispan-server (don't forget to restart each time to clean the cache)

      docker run -p 11222:11222 -e USER=dev -e PASS=dev jboss/infinispan-server:10.0.0.Final-1

      In kogito-runtimes/addons/jobs/jobs-service:

      1. Set application properties 
        kogito.jobs-service.persistence=infinispan
quarkus.infinispan-client.server-list=172.17.0.2:11222
quarkus.infinispan-client.auth-username=dev
quarkus.infinispan-client.auth-password=dev
quarkus.infinispan-client.use-auth=true
quarkus.infinispan-client.auth-realm=default
quarkus.infinispan-client.sasl-mechanism=PLAIN
      1. Launch command 
        mvn clean compile quarkus:dev
      Show
      Repeat as long as you don't have the issuer, as it is random ... Start infinispan-server (don't forget to restart each time to clean the cache) docker run -p 11222:11222 -e USER=dev -e PASS=dev jboss/infinispan-server:10.0.0.Final-1 In kogito-runtimes/addons/jobs/jobs-service: Set application properties kogito.jobs-service.persistence=infinispan quarkus.infinispan-client.server-list=172.17.0.2:11222 quarkus.infinispan-client.auth-username=dev quarkus.infinispan-client.auth-password=dev quarkus.infinispan-client.use-auth=true quarkus.infinispan-client.auth-realm=default quarkus.infinispan-client.sasl-mechanism=PLAIN Launch command mvn clean compile quarkus:dev
    • 2020 Week 04-06 (from Jan 20)

    Description

      When starting the Job Service, we get random failures on startup.
      Error logs:

      2020-01-22 13:31:42,231 jobs-service-6849b858d7-wc8qw INFO  [io.smallrye.reactive.messaging.extension.MediatorManager:205] (main) Attempt to resolve org.kie.kogito.jobs.service.scheduler.impl.VertxJobScheduler#jobSuccessProcessor
2020-01-22 13:31:42,232 jobs-service-6849b858d7-wc8qw INFO  [io.smallrye.reactive.messaging.extension.MediatorManager:211] (main) Connecting org.kie.kogito.jobs.service.scheduler.impl.VertxJobScheduler#jobSuccessProcessor to `job-success-events` (org.eclipse.microprofile.reactive.streams.operators.core.PublisherBuilderImpl@36b0fcd5)
2020-01-23 10:05:19,555 tradisso INFO  [org.kie.kogito.jobs.service.repository.impl.JobRepositoryDelegate:54] (main) JobRepository selected class org.kie.kogito.jobs.service.repository.infinispan.InfinispanJobRepository_ClientProxy
2020-01-23 10:05:20,055 tradisso INFO  [org.infinispan.HOTROD:359] (main) ISPN004021: Infinispan version: Infinispan 'Chupacabra' 10.0.0.Final
2020-01-23 10:05:20,460 tradisso WARN  [org.infinispan.HOTROD:335] (HotRod-client-async-pool-1-1) ISPN004005: Error received from the server: org.infinispan.server.hotrod.CacheNotFoundException: Cache with name 'SCHEDULED_JOBS' not found amongst the configured caches
2020-01-23 10:05:20,462 tradisso ERROR [io.quarkus.application:295] (main) Failed to start application: java.lang.IllegalArgumentException: cache parameter cannot be null

      Correct logs:

      2020-01-23 10:12:51,576 tradisso INFO  [io.smallrye.reactive.messaging.extension.MediatorManager:205] (main) Attempt to resolve org.kie.kogito.jobs.service.scheduler.impl.VertxJobScheduler#jobSuccessProcessor
2020-01-23 10:12:51,576 tradisso INFO  [io.smallrye.reactive.messaging.extension.MediatorManager:211] (main) Connecting org.kie.kogito.jobs.service.scheduler.impl.VertxJobScheduler#jobSuccessProcessor to `job-success-events` (org.eclipse.microprofile.reactive.streams.operators.core.PublisherBuilderImpl@7c75db8b)
2020-01-23 10:12:51,601 tradisso INFO  [org.kie.kogito.jobs.service.repository.infinispan.InfinispanConfiguration:66] (main) Persistence config Optional[infinispan]
2020-01-23 10:12:51,963 tradisso INFO  [org.infinispan.HOTROD:359] (main) ISPN004021: Infinispan version: Infinispan 'Chupacabra' 10.0.0.Final
2020-01-23 10:12:52,272 tradisso INFO  [org.kie.kogito.jobs.service.repository.infinispan.InfinispanConfiguration:78] (main) Executed Infinispan configuration
2020-01-23 10:12:52,273 tradisso INFO  [org.kie.kogito.jobs.service.scheduler.JobSchedulerManager:45] (main) Loading scheduled jobs
2020-01-23 10:12:52,274 tradisso INFO  [org.kie.kogito.jobs.service.repository.impl.JobRepositoryDelegate:54] (main) JobRepository selected class org.kie.kogito.jobs.service.repository.infinispan.InfinispanJobRepository_ClientProxy
2020-01-23 10:12:52,558 tradisso INFO  [org.kie.kogito.jobs.service.scheduler.JobSchedulerManager:53] (main) Loading scheduled jobs completed !

      It seems that, sometimes, the 'onStart' of InfinispanConfiguration is not done before the InfinispanJobRepository initialization. And so we get the error of cache is null (full logs in attachments)

      This might be caused by the update of Quarkus to 1.1.0 => https://github.com/kiegroup/kogito-bom/commit/deee85f77ce68de19844806203d58b36b8c9a96f

      Attachments

        1. correct.log
          6 kB
        2. error.log
          11 kB

        Issue Links

          blocks

          Task - Represents a small unit of work that is not end-user facing. KOGITO-960 Smoke tests: Enable jobs service with persistence test

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. KOGITO-1291 Creating a default cache for Jobs Service if none is found

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              tdolphin-1 Tiago Dolphine
              tradisso@redhat.com Tristan Radisson
              Tristan Radisson Tristan Radisson
              Tristan Radisson Tristan Radisson
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: