Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-31691

[GSS](8.1.z) WFLY-20914 - ConcurrentModificationException in org.jboss.as.webservices.service.EndpointDeployService.start

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      With the attached reproducer app and byteman rule:

      1. Have two EAP 8.1 instances, one or port 8080 with helloworld-ws.war and the other on 8180 with helloworld-ws-client.war deployed
      2. configure node 2 with byteman using the included JBEAP-31681.btm root like below in its JVM options:

      -javaagent:<mypath>/lib/byteman.jar=script:<mypath>/JBEAP-31691.btm,boot:<mypath>/lib/byteman.jar -Dorg.jboss.byteman.transform.all

      3. start node 1 and node 2:

      $NODE1_HOME/bin/standalone.sh
      $NODE2_HOME/bin/standalone.sh -Djboss.socket.binding.port-offset=100

      4. Send following request to node2 to initiate its concurrent ws client creation attempts and note the thrown ConcurrentModificationExceptions:

      curl localhost:8180/HelloWorld/foo 
      Show
      With the attached reproducer app and byteman rule: 1. Have two EAP 8.1 instances, one or port 8080 with helloworld-ws.war and the other on 8180 with helloworld-ws-client.war deployed 2. configure node 2 with byteman using the included JBEAP-31681 .btm root like below in its JVM options: -javaagent:<mypath>/lib/byteman.jar=script:<mypath>/JBEAP-31691.btm,boot:<mypath>/lib/byteman.jar -Dorg.jboss.byteman.transform.all 3. start node 1 and node 2: $NODE1_HOME/bin/standalone.sh $NODE2_HOME/bin/standalone.sh -Djboss.socket.binding.port-offset=100 4. Send following request to node2 to initiate its concurrent ws client creation attempts and note the thrown ConcurrentModificationExceptions: curl localhost:8180/HelloWorld/foo

      Randomly stumbled on this failure in CI:

      https://ci.wildfly.org/viewLog.html?buildId=517983&buildTypeId=WF_PullRequest_LinuxJdk17&fromSakuraUI=true

      BridgeFromJTATestCase.testRollback failed.

      The interesting bit was in the server.log:

      31m23:53:22,778 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.ws.endpoint-deploy.ws-t11-client: org.jboss.msc.service.StartException in service jboss.ws.endpoint-deploy.ws-t11-client: jakarta.xml.ws.WebServiceException: java.util.ConcurrentModificationException
      	at org.jboss.as.webservices//org.jboss.as.webservices.service.EndpointDeployService.start(EndpointDeployService.java:55)
      	at org.jboss.msc@1.5.6.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1553)
      	at org.jboss.msc@1.5.6.Final//org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1516)
      	at org.jboss.msc@1.5.6.Final//org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1374)
      	at org.jboss.threads@3.9.1//org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
      	at org.jboss.threads@3.9.1//org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
      	at org.jboss.threads@3.9.1//org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
      	at org.jboss.threads@3.9.1//org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
      	at org.jboss.threads@3.9.1//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
      	at java.base/java.lang.Thread.run(Thread.java:840)
      Caused by: jakarta.xml.ws.WebServiceException: java.util.ConcurrentModificationException
      	at org.apache.cxf.impl//org.apache.cxf.jaxws.EndpointImpl.doPublish(EndpointImpl.java:373)
      	at org.jboss.ws.cxf.jbossws-cxf-server@7.3.3.Final//org.jboss.wsf.stack.cxf.deployment.EndpointImpl.doPublish(EndpointImpl.java:77)
      	at org.apache.cxf.impl//org.apache.cxf.jaxws.EndpointImpl.publish(EndpointImpl.java:255)
      	at org.apache.cxf.impl//org.apache.cxf.jaxws.EndpointImpl.publish(EndpointImpl.java:541)
      	at org.jboss.ws.cxf.jbossws-cxf-server@7.3.3.Final//org.jboss.wsf.stack.cxf.configuration.BusHolder.configure(BusHolder.java:233)
      	at org.jboss.ws.cxf.jbossws-cxf-server@7.3.3.Final//org.jboss.wsf.stack.cxf.deployment.aspect.BusDeploymentAspect.startDeploymentBus(BusDeploymentAspect.java:96)
      	at org.jboss.ws.cxf.jbossws-cxf-server@7.3.3.Final//org.jboss.wsf.stack.cxf.deployment.aspect.BusDeploymentAspect.start(BusDeploymentAspect.java:57)
      	at org.jboss.ws.common@5.1.0.Final//org.jboss.ws.common.deployment.DeploymentAspectManagerImpl.deploy(DeploymentAspectManagerImpl.java:115)
      	at org.jboss.as.webservices//org.jboss.as.webservices.publish.EndpointPublisherImpl.doDeploy(EndpointPublisherImpl.java:150)
      	at org.jboss.as.webservices//org.jboss.as.webservices.publish.EndpointPublisherHelper.doDeployStep(EndpointPublisherHelper.java:44)
      	at org.jboss.as.webservices//org.jboss.as.webservices.service.EndpointDeployService.start(EndpointDeployService.java:53)
      	... 9 more
      Caused by: java.util.ConcurrentModificationException
      	at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1221)
      	at org.glassfish.jaxb//org.glassfish.jaxb.runtime.v2.model.impl.TypeInfoSetImpl.getXmlNs(TypeInfoSetImpl.java:298)
      	at org.glassfish.jaxb//org.glassfish.jaxb.runtime.v2.schemagen.XmlSchemaGenerator$Namespace.writeTo(XmlSchemaGenerator.java:551)
      	at org.glassfish.jaxb//org.glassfish.jaxb.runtime.v2.schemagen.XmlSchemaGenerator.write(XmlSchemaGenerator.java:413)
      	at org.glassfish.jaxb//org.glassfish.jaxb.runtime.v2.runtime.JAXBContextImpl.generateSchema(JAXBContextImpl.java:770)
      	at org.apache.cxf@4.0.9//org.apache.cxf.common.jaxb.JAXBUtils.generateJaxbSchemas(JAXBUtils.java:810)
      	at org.apache.cxf.impl//org.apache.cxf.jaxb.JAXBDataBinding.generateJaxbSchemas(JAXBDataBinding.java:474)
      	at org.apache.cxf.impl//org.apache.cxf.jaxb.JAXBDataBinding.initialize(JAXBDataBinding.java:391)
      	at org.apache.cxf@4.0.9//org.apache.cxf.service.factory.AbstractServiceFactoryBean.initializeDataBindings(AbstractServiceFactoryBean.java:87)
      	at org.apache.cxf.impl//org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromClass(ReflectionServiceFactoryBean.java:469)
      	at org.apache.cxf.impl//org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.buildServiceFromClass(JaxWsServiceFactoryBean.java:693)
      	at org.apache.cxf.impl//org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:529)
      	at org.apache.cxf.impl//org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:262)
      	at org.apache.cxf.impl//org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:199)
      	at org.apache.cxf.impl//org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(AbstractWSDLBasedEndpointFactory.java:103)
      	at org.apache.cxf.impl//org.apache.cxf.frontend.ServerFactoryBean.create(ServerFactoryBean.java:168)
      	at org.apache.cxf.impl//org.apache.cxf.jaxws.JaxWsServerFactoryBean.create(JaxWsServerFactoryBean.java:210)
      	at org.apache.cxf.impl//org.apache.cxf.jaxws.EndpointImpl.getServer(EndpointImpl.java:458)
      	at org.apache.cxf.impl//org.apache.cxf.jaxws.EndpointImpl.doPublish(EndpointImpl.java:336)
      	... 19 more
      

      Followed later by:

      31m23:53:23,043 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "xts")]) - Stage: (VERIFY) - failure description: {
          "WFLYCTL0080: Failed services" => {"jboss.ws.endpoint-deploy.ws-t11-client" => "jakarta.xml.ws.WebServiceException: java.util.ConcurrentModificationException
          Caused by: jakarta.xml.ws.WebServiceException: java.util.ConcurrentModificationException
          Caused by: java.util.ConcurrentModificationException"},
          "WFLYCTL0412: Required services that are not installed:" => [
              "jboss.ws.endpoint.\"ws-t11-client.deployment\".TerminationParticipantService",
              "jboss.ws.endpoint.\"ws-t11-client.deployment\".CompletionInitiatorService"
          ],
          "WFLYCTL0180: Services with missing/unavailable dependencies" => ["service jboss.ws.endpoint-publish.ws-t11-client is missing [jboss.ws.endpoint.\"ws-t11-client.deployment\".TerminationParticipantService, jboss.ws.endpoint.\"ws-t11-client.deployment\".CompletionInitiatorService]"]
      }
      &amp#27;[0m&amp#27;[0m23:53:23,079 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
      WFLYCTL0184:    New missing/unsatisfied dependencies:
            service jboss.ws.endpoint."ws-t11-client.deployment".CompletionInitiatorService (missing) dependents: [service jboss.ws.endpoint-publish.ws-t11-client] 
            service jboss.ws.endpoint."ws-t11-client.deployment".TerminationParticipantService (missing) dependents: [service jboss.ws.endpoint-publish.ws-t11-client] 
      WFLYCTL0186:   Services which failed to start:      service jboss.ws.endpoint-deploy.ws-t11-client: jakarta.xml.ws.WebServiceException: java.util.ConcurrentModificationException
      WFLYCTL0448: 2 additional services are down due to their dependencies being missing or failed
      

      I assume the server.log stuff is why the test failed.

      In any case a ConcurrentModificationException like that is not good.

        1. JBEAP-31691.btm
          0.7 kB
          Aaron Ogburn

              ropalka Richard Opalka
              rhn-support-aogburn Aaron Ogburn
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: