Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-19425

Camel-K: In a specific scenario a new integration pod is created every ~0.5 seconds

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • camel-k-1.8
    • camel-k-1.8
    • Camel-K
    • None

      I stumbled upon this issue when running the quickstarts test, namely with https://github.com/openshift-integration/camel-k-example-event-streaming, I didn't see it in any other test/quickstart (therefore I didn't set it as a blocker, as I don't know how likely the issue is, but definitely something strange is going on with 1.8 and it didn't happen with 1.6).

      For some reason, after creating UserReportSystem integration from the mentioned quickstart:

      • two integration pods are created simultaneously, where at least one of them crashes with:
      2022-07-01 13:24:26,294 ERROR [io.qua.run.Application] (main) Failed to start application (with profile prod): java.lang.IllegalArgumentException: Unable to find a resource definition for channel/sink/audit
      	at org.apache.camel.component.knative.KnativeEndpoint.lambda$lookupServiceDefinition$2(KnativeEndpoint.java:160)
      	at java.base/java.util.Optional.orElseThrow(Optional.java:408)
      	at org.apache.camel.component.knative.KnativeEndpoint.lookupServiceDefinition(KnativeEndpoint.java:159)
      	at org.apache.camel.component.knative.KnativeEndpoint.createProducer(KnativeEndpoint.java:81)
      	at org.apache.camel.support.DefaultEndpoint.createAsyncProducer(DefaultEndpoint.java:197)
      	at org.apache.camel.processor.SendProcessor.doStart(SendProcessor.java:245)
      	at org.apache.camel.support.service.BaseService.start(BaseService.java:119)
      	at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:113)
      	at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:130)
      	at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1655)
      	at org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:60)
      	at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:113)
      	at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:130)
      	at org.apache.camel.impl.engine.DefaultChannel.doStart(DefaultChannel.java:126)
      	at org.apache.camel.support.service.BaseService.start(BaseService.java:119)
      	at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:113)
      	at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:116)
      	at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:130)
      	at org.apache.camel.processor.Pipeline.doStart(Pipeline.java:221)
      	at org.apache.camel.support.service.BaseService.start(BaseService.java:119)
      	at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:113)
      	at org.apache.camel.support.processor.DelegateAsyncProcessor.doStart(DelegateAsyncProcessor.java:89)
      	at org.apache.camel.support.service.BaseService.start(BaseService.java:119)
      	at org.apache.camel.support.service.ServiceHelper.startService(ServiceHelper.java:113)
      	at org.apache.camel.impl.engine.RouteService.startChildServices(RouteService.java:396)
      	at org.apache.camel.impl.engine.RouteService.doWarmUp(RouteService.java:193)
      	at org.apache.camel.impl.engine.RouteService.warmUp(RouteService.java:121)
      	at org.apache.camel.impl.engine.InternalRouteStartupManager.doWarmUpRoutes(InternalRouteStartupManager.java:306)
      	at org.apache.camel.impl.engine.InternalRouteStartupManager.safelyStartRouteServices(InternalRouteStartupManager.java:189)
      	at org.apache.camel.impl.engine.InternalRouteStartupManager.doStartOrResumeRoutes(InternalRouteStartupManager.java:147)
      	at org.apache.camel.impl.engine.AbstractCamelContext.doStartCamel(AbstractCamelContext.java:3300)
      	at org.apache.camel.impl.engine.AbstractCamelContext.doStartContext(AbstractCamelContext.java:2952)
      	at org.apache.camel.impl.engine.AbstractCamelContext.doStart(AbstractCamelContext.java:2903)
      	at org.apache.camel.support.service.BaseService.start(BaseService.java:119)
      	at org.apache.camel.impl.engine.AbstractCamelContext.start(AbstractCamelContext.java:2587)
      	at org.apache.camel.impl.DefaultCamelContext.start(DefaultCamelContext.java:253)
      	at org.apache.camel.quarkus.main.CamelMain.doStart(CamelMain.java:94)
      	at org.apache.camel.support.service.BaseService.start(BaseService.java:119)
      	at org.apache.camel.quarkus.main.CamelMain.startEngine(CamelMain.java:140)
      	at org.apache.camel.quarkus.main.CamelMainRuntime.start(CamelMainRuntime.java:49)
      	at org.apache.camel.quarkus.core.CamelBootstrapRecorder.start(CamelBootstrapRecorder.java:45)
      	at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot173480958.deploy_0(Unknown Source)
      	at io.quarkus.deployment.steps.CamelBootstrapProcessor$boot173480958.deploy(Unknown Source)
      	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
      	at io.quarkus.runtime.Application.start(Application.java:101)
      	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:103)
      	at io.quarkus.runtime.Quarkus.run(Quarkus.java:67)
      	at io.quarkus.runtime.Quarkus.run(Quarkus.java:41)
      	at io.quarkus.runtime.Quarkus.run(Quarkus.java:120)
      	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.doRun(QuarkusEntryPoint.java:60)
      	at io.quarkus.bootstrap.runner.QuarkusEntryPoint.main(QuarkusEntryPoint.java:31)
      
      • after that, approximately every 0.5 seconds a new integration pod is started, for example, after ~4 minutes I have:
      oc get integration
      NAME                 PHASE   KIT                        REPLICAS
      user-report-system   Error   kit-cavfa1jmo6valktkhosg   662
      
      oc get pods
      NAME                                                   READY   STATUS              RESTARTS        AGE
      camel-k-kit-cavfa1jmo6valktkhosg-1-build               0/1     Completed           0               5m14s
      camel-k-operator-845496f957-tw29g                      1/1     Running             0               6m22s
      user-report-system-00001-deployment-cdb854f65-pnnn4    0/2     CrashLoopBackOff    4 (76s ago)     4m45s
      user-report-system-00002-deployment-6cb85f8b6c-ss9x9   0/2     CrashLoopBackOff    4 (87s ago)     4m45s
      user-report-system-00003-deployment-7bd9d98c85-2rqg8   0/2     CrashLoopBackOff    4 (55s ago)     4m39s
      user-report-system-00004-deployment-5c7786dfcf-vw274   0/2     CrashLoopBackOff    4 (87s ago)     4m39s
      user-report-system-00005-deployment-6564c54c54-72nc6   0/2     Error               4 (2m46s ago)   4m38s
      user-report-system-00006-deployment-8647677869-vnr89   0/2     CrashLoopBackOff    4 (88s ago)     4m38s
      user-report-system-00007-deployment-68c87755-tq9qv     0/2     Error               4 (2m51s ago)   4m38s
      user-report-system-00008-deployment-5fc48d47bd-cwbvp   0/2     Error               4 (2m29s ago)   4m38s
      user-report-system-00009-deployment-594cff9989-nrt6p   0/2     CrashLoopBackOff    4 (59s ago)     4m38s
      user-report-system-00010-deployment-7c9f97dfcb-xpw9l   0/2     Error               4 (2m45s ago)   4m38s
      user-report-system-00011-deployment-685fcfb86b-qhvj4   0/2     CrashLoopBackOff    4 (89s ago)     4m36s
      user-report-system-00012-deployment-667d947b9d-h4skx   0/2     CrashLoopBackOff    4 (59s ago)     4m36s
      user-report-system-00013-deployment-76cbbb5f75-fwjtx   0/2     CrashLoopBackOff    4 (47s ago)     4m35s
      user-report-system-00014-deployment-fb694b97b-6q2b2    0/2     CrashLoopBackOff    4 (28s ago)     4m35s
      user-report-system-00015-deployment-7dc897bc79-j4cq7   0/2     Error               4 (2m50s ago)   4m35s
      user-report-system-00016-deployment-5f45d8699f-lk9rx   0/2     CrashLoopBackOff    4 (58s ago)     4m35s
      user-report-system-00017-deployment-85c8488b8f-q59rt   0/2     CrashLoopBackOff    4 (56s ago)     4m34s
      user-report-system-00018-deployment-64c457554b-8mvs7   0/2     CrashLoopBackOff    3 (2m11s ago)   4m34s
      user-report-system-00019-deployment-658954fb4d-z885d   0/2     CrashLoopBackOff    4 (59s ago)     4m34s
      user-report-system-00020-deployment-57bf8d4747-xq2gq   0/2     CrashLoopBackOff    3 (2m8s ago)    4m34s
      user-report-system-00021-deployment-65b65bc885-457jh   0/2     CrashLoopBackOff    4 (59s ago)     4m34s
      user-report-system-00022-deployment-b56b478bb-tzm6z    0/2     CrashLoopBackOff    4 (28s ago)     4m33s
      user-report-system-00023-deployment-666454586f-pq4sb   0/2     CrashLoopBackOff    3 (2m10s ago)   4m32s
      user-report-system-00024-deployment-555cff94cb-ls6ml   0/2     Error               4 (2m11s ago)   4m32s
      user-report-system-00025-deployment-864d4f69f8-2zcpp   0/2     Error               4 (2m32s ago)   4m32s
      user-report-system-00026-deployment-997586b76-svvm9    0/2     CrashLoopBackOff    3 (2m10s ago)   4m32s
      user-report-system-00027-deployment-866b7c5958-95wzr   0/2     Error               4 (2m12s ago)   4m32
      ... a lot more
      

      The minimal working example I was able to create is:

      0. deploy serverless operator + knative eventing + knative serving
      1. create a fresh namespace
      2. install camel-k 1.8 operator, for example kamel install --maven-repository="<some repository with necessary artifacts>" --olm=false --operator-image registry-proxy.engineering.redhat.com/rh-osbs/integration-camel-k-rhel8-operator:1.8.0-10
      3. as mentioned in the quickstart, create the secret oc create secret generic example-event-streaming-user-reporting --from-file=application.properties
      4. create the channel with kn channel create audit --type imc
      5. create the user-report-system integration kamel run UserReportSystem.java

              cmiranda@redhat.com Claudio Miranda
              avano@redhat.com Andrej Vano
              Andrej Vano Andrej Vano
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: