Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-10749

There is ConcurrentModificationException when we write new aliases to one credential store resource through multiple threads (management clients)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • 7.1.0.DR19
    • 7.1.0.DR17
    • Security
    • None
    • Regression
    • Not Required
    • Hide
      1. git clone git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/tests-security.git
        
      2. cd tests-security/elytron/
      3. unignore test com.redhat.eap.qe.elytron.credential.store.CredentialStoreTestCase#testParallelAccessToCS
      4. set correct path to EAP dist zip and to maven repository
        ./build-elytron.sh -Djboss.dist.zip=/home/hsvabek/static_build_env/eap/7.1.0.DR17/jboss-eap-7.1.0.DR17.zip -Dmaven.repo.local=/home/hsvabek/.m2/jboss-eap-7.1.0.DR17-maven-repository/maven-repository -Dversion.jboss.bom=7.1.0.Beta1 -Dtest=com.redhat.eap.qe.elytron.credential.store.CredentialStoreTestCase#testParallelAccessToCS
        

      You can run it again without building, please set correct paths.

      mvn test -Djboss.init.skip -Dversion.jboss.bom=7.1.0.Beta1 -Dmaven.repo.local=/home/hsvabek/.m2/jboss-eap-7.1.0.DR17-maven-repository/maven-repository -Dtest=com.redhat.eap.qe.elytron.credential.store.CredentialStoreTestCase#testParallelAccessToCS
      

      For me fails 1 run from 5.

      Output is:

      Unable to find source-code formatter for language: collapse. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: "WFLYCTL0158: Operation handler failed: java.util.ConcurrentModificationException"
      	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
      	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
      	at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase.testAccessFromMultipleCredentialStores(AbstractCredentialStoreTestCase.java:645)
      	at com.redhat.eap.qe.elytron.credential.store.CredentialStoreTestCase.testParallelAccessToCS(CredentialStoreTestCase.java:447)
      Caused by: java.lang.AssertionError: "WFLYCTL0158: Operation handler failed: java.util.ConcurrentModificationException"
      	at org.junit.Assert.fail(Assert.java:88)
      	at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase.checkOperationResult(AbstractCredentialStoreTestCase.java:352)
      	at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase.executeAndCheck(AbstractCredentialStoreTestCase.java:346)
      	at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase.writeAliasToCredentialStoreAndCheck(AbstractCredentialStoreTestCase.java:211)
      	at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase$1.call(AbstractCredentialStoreTestCase.java:627)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Server log output:

      Unable to find source-code formatter for language: collapse. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      [31m14:43:24,475 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 2) WFLYCTL0013: Operation ("add") failed - address: ([
          ("subsystem" => "elytron"),
          ("credential-store" => "storeName_unmF326q3f"),
          ("alias" => "alias_gkexssohxy")
      ]): java.util.ConcurrentModificationException
      	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1545)
      	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
      	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
      	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
      	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
      	at org.wildfly.extension.elytron.CredentialStoreResource.getChildren(CredentialStoreResource.java:128)
      	at org.jboss.as.controller.registry.Resource.shallowCopy(Resource.java:200)
      	at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:930)
      	at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:895)
      	at org.wildfly.extension.elytron.CredentialStoreAliasDefinition$AddHandler.execute(CredentialStoreAliasDefinition.java:172)
      	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:971)
      	at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:717)
      	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:441)
      	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1392)
      	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:421)
      	at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
      	at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:263)
      	at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:229)
      	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
      	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:217)
      	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$400(ModelControllerClientOperationHandler.java:137)
      	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:161)
      	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:157)
      	at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:287)
      	at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:244)
      	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:254)
      	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:225)
      	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:157)
      	at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70)
      	at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      	at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      
      Show
      git clone git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/tests-security.git cd tests-security/elytron/ unignore test com.redhat.eap.qe.elytron.credential.store.CredentialStoreTestCase#testParallelAccessToCS set correct path to EAP dist zip and to maven repository ./build-elytron.sh -Djboss.dist.zip=/home/hsvabek/static_build_env/eap/7.1.0.DR17/jboss-eap-7.1.0.DR17.zip -Dmaven.repo.local=/home/hsvabek/.m2/jboss-eap-7.1.0.DR17-maven-repository/maven-repository -Dversion.jboss.bom=7.1.0.Beta1 -Dtest=com.redhat.eap.qe.elytron.credential.store.CredentialStoreTestCase#testParallelAccessToCS You can run it again without building, please set correct paths. mvn test -Djboss.init.skip -Dversion.jboss.bom=7.1.0.Beta1 -Dmaven.repo.local=/home/hsvabek/.m2/jboss-eap-7.1.0.DR17-maven-repository/maven-repository -Dtest=com.redhat.eap.qe.elytron.credential.store.CredentialStoreTestCase#testParallelAccessToCS For me fails 1 run from 5. Output is: Unable to find source-code formatter for language: collapse. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: "WFLYCTL0158: Operation handler failed: java.util.ConcurrentModificationException" at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase.testAccessFromMultipleCredentialStores(AbstractCredentialStoreTestCase.java:645) at com.redhat.eap.qe.elytron.credential.store.CredentialStoreTestCase.testParallelAccessToCS(CredentialStoreTestCase.java:447) Caused by: java.lang.AssertionError: "WFLYCTL0158: Operation handler failed: java.util.ConcurrentModificationException" at org.junit.Assert.fail(Assert.java:88) at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase.checkOperationResult(AbstractCredentialStoreTestCase.java:352) at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase.executeAndCheck(AbstractCredentialStoreTestCase.java:346) at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase.writeAliasToCredentialStoreAndCheck(AbstractCredentialStoreTestCase.java:211) at com.redhat.eap.qe.elytron.credential.store.AbstractCredentialStoreTestCase$1.call(AbstractCredentialStoreTestCase.java:627) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Server log output: Unable to find source-code formatter for language: collapse. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml [31m14:43:24,475 ERROR [org.jboss.as.controller.management-operation] (management-handler-thread - 2) WFLYCTL0013: Operation ( "add" ) failed - address: ([ ( "subsystem" => "elytron" ), ( "credential-store" => "storeName_unmF326q3f" ), ( "alias" => "alias_gkexssohxy" ) ]): java.util.ConcurrentModificationException at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1545) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at org.wildfly.extension.elytron.CredentialStoreResource.getChildren(CredentialStoreResource.java:128) at org.jboss.as.controller.registry.Resource.shallowCopy(Resource.java:200) at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:930) at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:895) at org.wildfly.extension.elytron.CredentialStoreAliasDefinition$AddHandler.execute(CredentialStoreAliasDefinition.java:172) at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:971) at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:717) at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:441) at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1392) at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:421) at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243) at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:263) at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:229) at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:217) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$400(ModelControllerClientOperationHandler.java:137) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:161) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:157) at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:287) at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:244) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:254) at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:225) at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:157) at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70) at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320)

      It is regression because this problem wasn't in EAP7.1.0.DR16.

      There is ConcurrentModificationException when
      we write new aliases to one credential store resource through multiple threads (management clients)

      Use case:
      More users are connected through CLI to same application server and work parallel with same credential store resource.

      You can see this error on jenkins too https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/view/EAP7/view/EAP7-Security/view/Wildfly-Elytron/job/eap-7x-security-elytron-testsuite-rhel/25/jdk=java18_default,label_exp=eap-sustaining&&RHEL6&&x86/testReport/junit/com.redhat.eap.qe.elytron.credential.store/CredentialStoreTestCase/testParallelAccessToCS/

              yborgess1@redhat.com Yeray Borges Santana
              hsvabek_jira Hynek Švábek (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: