Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-10820

Concurrent Modification Exception thrown by camel cxf

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • fuse-7.4-GA
    • fuse-7.0
    • Camel, CXF
    • None

    • All

    • Hide

      Work in progress

      Show
      Work in progress
    • Fuse 7.4 Sprint 46 - Dev #2

      Fuse 7.0 on Spring Boot on Openshift Container Platform 3.9 makes external http connections via cxf to third party soap services. Inbound requests are REST/json in this use case.

      Running a single user test works fine but when load is applied we start seeing exceptions occasionally in the logs indicating concurrentModificationException in the apache cxf component which results in a 503 response code from the service. This also leads to High CPU.

      java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
        at java.util.HashMap$EntryIterator.next(HashMap.java:1476)
        at java.util.HashMap$EntryIterator.next(HashMap.java:1474)
        at java.util.AbstractMap.hashCode(AbstractMap.java:530)
        at java.util.Objects.hashCode(Objects.java:98)
        at java.util.HashMap$Node.hashCode(HashMap.java:297)
        at java.util.HashMap.hash(HashMap.java:339)
        at java.util.HashMap.put(HashMap.java:612)
        at java.util.HashSet.add(HashSet.java:220)
        at org.apache.cxf.jaxws.context.WrappedMessageContext.entrySet(WrappedMessageContext.java:397)
        at org.apache.cxf.jaxws.context.WrappedMessageContext.putAll(WrappedMessageContext.java:476)
        at org.apache.camel.component.cxf.DefaultCxfBinding.extractInvocationContextFromCamel(DefaultCxfBinding.java:516)
        at org.apache.camel.component.cxf.DefaultCxfBinding.populateCxfRequestFromExchange(DefaultCxfBinding.java:113)
        at org.apache.camel.component.cxf.CxfProducer.prepareRequest(CxfProducer.java:223)
        at org.apache.camel.component.cxf.CxfProducer.process(CxfProducer.java:129)
        at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
        at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:181)
        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110)
        at org.apache.camel.builder.NoErrorHandlerBuilder$1.process(NoErrorHandlerBuilder.java:40)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.component.direct.DirectBlockingProducer.process(DirectBlockingProducer.java:53)
        at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
        at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:181)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.component.direct.DirectBlockingProducer.process(DirectBlockingProducer.java:53)
        at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
        at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:181)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
        at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:117)
        at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:181)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715)
        at org.apache.camel.processor.MulticastProcessor.access$900(MulticastProcessor.java:86)
        at org.apache.camel.processor.MulticastProcessor$2.done(MulticastProcessor.java:777)
        at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
        at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
        at org.apache.camel.processor.interceptor.TraceInterceptor$1.done(TraceInterceptor.java:198)
        at org.apache.camel.processor.SendProcessor$1.done(SendProcessor.java:160)
        at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
        at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:166)
        at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
        at org.apache.camel.processor.RedeliveryErrorHandler$2.done(RedeliveryErrorHandler.java:560)
        at org.apache.camel.processor.interceptor.TraceInterceptor$1.done(TraceInterceptor.java:198)
        at org.apache.camel.processor.SendProcessor$1.done(SendProcessor.java:160)
        at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
        at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
        at org.apache.camel.processor.interceptor.TraceInterceptor$1.done(TraceInterceptor.java:198)
        at org.apache.camel.processor.TryProcessor$1.done(TryProcessor.java:141)
        at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:166)
        at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
        at org.apache.camel.processor.interceptor.TraceInterceptor$1.done(TraceInterceptor.java:198)
        at org.apache.camel.processor.SendProcessor$1.done(SendProcessor.java:160)
        at org.apache.camel.component.cxf.CxfClientCallback.handleResponse(CxfClientCallback.java:74)
        at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:831)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1680)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream$1.run(HTTPConduit.java:1176)
        at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$3.run(AutomaticWorkQueueImpl.java:428)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$AWQThreadFactory$1.run(AutomaticWorkQueueImpl.java:353)
        at java.lang.Thread.run(Thread.java:748)"

        1. cxf-http-server-client-payload-format.tar
          243 kB
        2. cxf-core-3.1.11.redhat-SNAPSHOT.jar
          1.32 MB
        3. cxf-cme-reproducer-vgohel.zip
          108 kB
        4. camel-cxf-2.21.0-SNAPSHOT-new.jar
          257 kB
        5. camel-cxf-2.21.0-SNAPSHOT-20190418.jar
          256 kB
        6. camel-cxf-2.21.0-SNAPSHOT.jar
          257 kB

              yfang@redhat.com Freeman(Yue) Fang
              ppecka Petr Pecka
              Viliam Kasala Viliam Kasala
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: