Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-10489

Concurrent Modification Exception thrown by camel cxf

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: fuse-7.0
    • Fix Version/s: fuse-7.3.1
    • Component/s: Camel, CXF
    • Labels:
      None
    • Sprint:
      Fuse 7.4 Sprint 47 - Bug Fix
    • Steps to Reproduce:
      Hide

      Work in progress

      Show
      Work in progress
    • QE Test Coverage:
      ?

      Description

      Fuse 7.0 on Spring Boot on Openshift Container Platform 3.9 makes external http connections via cxf to third party soap services. Inbound requests are REST/json in this use case.

      Running a single user test works fine but when load is applied we start seeing exceptions occasionally in the logs indicating concurrentModificationException in the apache cxf component which results in a 503 response code from the service. This also leads to High CPU.

      java.util.ConcurrentModificationException
              at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
              at java.util.HashMap$EntryIterator.next(HashMap.java:1476)
              at java.util.HashMap$EntryIterator.next(HashMap.java:1474)
              at java.util.AbstractMap.hashCode(AbstractMap.java:530)
              at java.util.Objects.hashCode(Objects.java:98)
              at java.util.HashMap$Node.hashCode(HashMap.java:297)
              at java.util.HashMap.hash(HashMap.java:339)
              at java.util.HashMap.put(HashMap.java:612)
              at java.util.HashSet.add(HashSet.java:220)
              at org.apache.cxf.jaxws.context.WrappedMessageContext.entrySet(WrappedMessageContext.java:397)
              at org.apache.cxf.jaxws.context.WrappedMessageContext.putAll(WrappedMessageContext.java:476)
              at org.apache.camel.component.cxf.DefaultCxfBinding.extractInvocationContextFromCamel(DefaultCxfBinding.java:516)
              at org.apache.camel.component.cxf.DefaultCxfBinding.populateCxfRequestFromExchange(DefaultCxfBinding.java:113)
              at org.apache.camel.component.cxf.CxfProducer.prepareRequest(CxfProducer.java:223)
              at org.apache.camel.component.cxf.CxfProducer.process(CxfProducer.java:129)
              at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
              at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:181)
              at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:110)
              at org.apache.camel.builder.NoErrorHandlerBuilder$1.process(NoErrorHandlerBuilder.java:40)
              at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
              at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
              at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
              at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
              at org.apache.camel.component.direct.DirectBlockingProducer.process(DirectBlockingProducer.java:53)
              at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
              at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:181)
              at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
              at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
              at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
              at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
              at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
              at org.apache.camel.component.direct.DirectBlockingProducer.process(DirectBlockingProducer.java:53)
              at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
              at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:181)
              at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
              at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
              at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
              at org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:117)
              at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:181)
              at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
              at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
              at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:715)
              at org.apache.camel.processor.MulticastProcessor.access$900(MulticastProcessor.java:86)
              at org.apache.camel.processor.MulticastProcessor$2.done(MulticastProcessor.java:777)
              at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
              at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
              at org.apache.camel.processor.interceptor.TraceInterceptor$1.done(TraceInterceptor.java:198)
              at org.apache.camel.processor.SendProcessor$1.done(SendProcessor.java:160)
              at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
              at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:166)
              at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
              at org.apache.camel.processor.RedeliveryErrorHandler$2.done(RedeliveryErrorHandler.java:560)
              at org.apache.camel.processor.interceptor.TraceInterceptor$1.done(TraceInterceptor.java:198)
              at org.apache.camel.processor.SendProcessor$1.done(SendProcessor.java:160)
              at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
              at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
              at org.apache.camel.processor.interceptor.TraceInterceptor$1.done(TraceInterceptor.java:198)
              at org.apache.camel.processor.TryProcessor$1.done(TryProcessor.java:141)
              at org.apache.camel.processor.Pipeline$1.done(Pipeline.java:166)
              at org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:262)
              at org.apache.camel.processor.interceptor.TraceInterceptor$1.done(TraceInterceptor.java:198)
              at org.apache.camel.processor.SendProcessor$1.done(SendProcessor.java:160)
              at org.apache.camel.component.cxf.CxfClientCallback.handleResponse(CxfClientCallback.java:74)
              at org.apache.cxf.endpoint.ClientImpl.onMessage(ClientImpl.java:831)
              at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1680)
              at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream$1.run(HTTPConduit.java:1176)
              at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$3.run(AutomaticWorkQueueImpl.java:428)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$AWQThreadFactory$1.run(AutomaticWorkQueueImpl.java:353)
              at java.lang.Thread.run(Thread.java:748)"
      

        Gliffy Diagrams

          Attachments

          1. cxf-http-server-client-payload-format.tar
            243 kB
            Freeman(Yue) Fang

            Issue Links

              Activity

                People

                • Assignee:
                  ffang Freeman(Yue) Fang
                  Reporter:
                  herambhn Herambh Naram
                  Tester:
                  Viliam Kasala
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  18 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: