Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-2046

KeyManager synchronization issue when using IBM JDK

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Blocker Blocker
    • None
    • None
    • Security
    • None
    • Hide

      We've prepared a byteman script which shows what happens when we hit the problem. It's not a reproducer, it just adds debug messages into the server log.

      1. Install byteman and IBM JDK
      2. store attached script into /tmp/ibmKeyManagerFactory.btm
      3. generate keystore /tmp/test.keystore 
        keytool -genkey -alias test -keystore /tmp/test.keystore -dname "CN=test" -storepass wildfly -keypass wildfly
      4. start EAP
      5. use JBoss CLI to add a new security realm with the generated keystore 
        /core-service=management/security-realm=JBossTest:add
/core-service=management/security-realm=JBossTest/server-identity=ssl:add(keystore-path=/tmp/test.keystore, keystore-password=wildfly, alias=test)
      6. stop EAP
      7. configure byteman for EAP - add following (with correct byteman path) at the end of bin/standalone.conf: 
        BYTEMAN_HOME="$HOME/tools/byteman"
JAVA_OPTS="$JAVA_OPTS -Dorg.jboss.byteman.transform.all -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:/tmp/ibmKeyManagerFactory.btm,boot:${BYTEMAN_HOME}/lib/byteman.jar,listener:true"
      8. start EAP (until you see the exception in server log repeat stop and start again) - don't forget to use IBM JDK

      When we hit the issue, the server.log contains something like:

      2016-11-25 14:44:52,877 INFO  [stdout] (MSC service thread 1-5) Incoming KeyStore password password
2016-11-25 14:44:52,879 INFO  [stdout] (MSC service thread 1-4) Incoming KeyStore password changeit
2016-11-25 14:44:52,882 INFO  [stdout] (MSC service thread 1-4) Used KeyStore password changeit
2016-11-25 14:44:52,883 INFO  [stdout] (MSC service thread 1-5) Used KeyStore password changeit
2016-11-25 14:44:52,891 INFO  [stdout] (MSC service thread 1-7) Incoming KeyStore password changeit
2016-11-25 14:44:52,892 INFO  [stdout] (MSC service thread 1-7) Used KeyStore password changeit
2016-11-25 14:44:52,891 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.server.controller.management.security_realm.ApplicationRealm.key-manager: org.jboss.msc.service.StartException in service jboss.server.controller.management.security_realm.ApplicationRealm.key-manager: WFLYDM0018: Unable to start service
        at org.jboss.as.domain.management.security.AbstractKeyManagerService.start(AbstractKeyManagerService.java:85)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1963)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1896)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.lang.Thread.run(Thread.java:785)
Caused by: java.security.UnrecoverableKeyException: Cannot recover key
        at com.ibm.crypto.provider.x.recover(Unknown Source)
        at com.ibm.crypto.provider.JavaKeyStore.engineGetKey(Unknown Source)
        at java.security.KeyStore.getKey(KeyStore.java:1034)
        at com.ibm.jsse2.aw.<init>(aw.java:146)
        at com.ibm.jsse2.ae$a.engineInit(ae$a.java:4)
        at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:19)
        at org.jboss.as.domain.management.security.AbstractKeyManagerService.createKeyManagers(AbstractKeyManagerService.java:130)
        at org.jboss.as.domain.management.security.AbstractKeyManagerService.start(AbstractKeyManagerService.java:83)
        ... 5 more
      Show
      We've prepared a byteman script which shows what happens when we hit the problem. It's not a reproducer, it just adds debug messages into the server log. Install byteman and IBM JDK store attached script into /tmp/ibmKeyManagerFactory.btm generate keystore /tmp/test.keystore keytool -genkey -alias test -keystore /tmp/test.keystore -dname "CN=test" -storepass wildfly -keypass wildfly start EAP use JBoss CLI to add a new security realm with the generated keystore /core-service=management/security-realm=JBossTest:add /core-service=management/security-realm=JBossTest/server-identity=ssl:add(keystore-path=/tmp/test.keystore, keystore-password=wildfly, alias=test) stop EAP configure byteman for EAP - add following (with correct byteman path) at the end of bin/standalone.conf : BYTEMAN_HOME= "$HOME/tools/byteman" JAVA_OPTS= "$JAVA_OPTS -Dorg.jboss.byteman.transform.all -javaagent:${BYTEMAN_HOME}/lib/byteman.jar=script:/tmp/ibmKeyManagerFactory.btm,boot:${BYTEMAN_HOME}/lib/byteman.jar,listener: true " start EAP (until you see the exception in server log repeat stop and start again) - don't forget to use IBM JDK When we hit the issue, the server.log contains something like: 2016-11-25 14:44:52,877 INFO [stdout] (MSC service thread 1-5) Incoming KeyStore password password 2016-11-25 14:44:52,879 INFO [stdout] (MSC service thread 1-4) Incoming KeyStore password changeit 2016-11-25 14:44:52,882 INFO [stdout] (MSC service thread 1-4) Used KeyStore password changeit 2016-11-25 14:44:52,883 INFO [stdout] (MSC service thread 1-5) Used KeyStore password changeit 2016-11-25 14:44:52,891 INFO [stdout] (MSC service thread 1-7) Incoming KeyStore password changeit 2016-11-25 14:44:52,892 INFO [stdout] (MSC service thread 1-7) Used KeyStore password changeit 2016-11-25 14:44:52,891 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.server.controller.management.security_realm.ApplicationRealm.key-manager: org.jboss.msc.service.StartException in service jboss.server.controller.management.security_realm.ApplicationRealm.key-manager: WFLYDM0018: Unable to start service at org.jboss.as.domain.management.security.AbstractKeyManagerService.start(AbstractKeyManagerService.java:85) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1963) at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1896) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang. Thread .run( Thread .java:785) Caused by: java.security.UnrecoverableKeyException: Cannot recover key at com.ibm.crypto.provider.x.recover(Unknown Source) at com.ibm.crypto.provider.JavaKeyStore.engineGetKey(Unknown Source) at java.security.KeyStore.getKey(KeyStore.java:1034) at com.ibm.jsse2.aw.<init>(aw.java:146) at com.ibm.jsse2.ae$a.engineInit(ae$a.java:4) at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:19) at org.jboss.as.domain.management.security.AbstractKeyManagerService.createKeyManagers(AbstractKeyManagerService.java:130) at org.jboss.as.domain.management.security.AbstractKeyManagerService.start(AbstractKeyManagerService.java:83) ... 5 more

      We hit a KeyManagerFactory related synchronization issue in org.jboss.as.domain.management.security.AbstractKeyManagerService.createKeyManagers(boolean) method on IBM JDK. The issue occurs if there are more security realms with SSL identities in EAP and they have keystores with different passwords.

      As the ApplicationRealm (in EAP 7.1) has preconfigured ssl identity configuration, the risk customers will hit this when they add their own security realm with a ssl identity is big. The frequency we hit this issue is more than 10% cases on our machines.

      Our debugging suggests the problem is located in IBM JDK implementation of javax.net.ssl.KeyManagerFactorySpi (class com.ibm.jsse2.ae$a).
      The workflow:

      1. user calls keyManagerFactory.init(keyStore, keystorePassword) which invokes com.ibm.jsse2.ae$a.engineInit(Keystore keyStore, char[] password)
      2. the password (from the second method parameter) is stored into static field com.ibm.jsse2.ae.d and in the next step the field is used as parameter for creating new object new com.ibm.jsse2.aw(keyStore, d)
      3. the previous step is not synchronized and when more threads call keyManagerFactory.init() with different passwords, wrong password may be used for retrieving a key from keystore.

      Possible workaround
      We could workaround this issue on EAP side (until it's fixed in the JDK) by synchronizing keyManagerFactory.init() call in AbstractKeyManagerService.createKeyManagers(boolean) when IBM JDK is used.

        1. test-app-ibm-jdk-keymanager-sync.zip
          6 kB

              darran.lofthouse@redhat.com Darran Lofthouse
              josef.cacek@gmail.com Josef Cacek (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: