Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-3408

Error on startup when multiple FIPS Credential Stores are configured

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 4.0.0.Alpha3
    • 4.0.0.Alpha2
    • Security
    • None
    • Hide
      • Start EAP with fips java
      • Add multiple (e.g. 5) PKCS11 external credential stores : 
        /subsystem=elytron/credential-store=MyStore:add(modifiable=true, implementation-properties={"keyStoreType"=>"PKCS11", "external"=>"true", "keyAlias"=>"my-key"}, credential-reference={clear-text="pass123+"}, create=true)
/subsystem=elytron/credential-store=MyStore2:add(modifiable=true, implementation-properties={"keyStoreType"=>"PKCS11", "external"=>"true", "keyAlias"=>"my-key"}, credential-reference={clear-text="pass123+"}, create=true)
/subsystem=elytron/credential-store=MyStore3:add(modifiable=true, implementation-properties={"keyStoreType"=>"PKCS11", "external"=>"true", "keyAlias"=>"my-key"}, credential-reference={clear-text="pass123+"}, create=true)
/subsystem=elytron/credential-store=MyStore4:add(modifiable=true, implementation-properties={"keyStoreType"=>"PKCS11", "external"=>"true", "keyAlias"=>"my-key"}, credential-reference={clear-text="pass123+"}, create=true)
/subsystem=elytron/credential-store=MyStore5:add(modifiable=true, implementation-properties={"keyStoreType"=>"PKCS11", "external"=>"true", "keyAlias"=>"my-key"}, credential-reference={clear-text="pass123+"}, create=true)
      • Restart server couple of times until exception occures during server startup 
        shutdown --restart
      Show
      Start EAP with fips java Add multiple (e.g. 5) PKCS11 external credential stores : /subsystem=elytron/credential-store=MyStore:add(modifiable= true , implementation-properties={ "keyStoreType" => "PKCS11" , "external" => " true " , "keyAlias" => "my-key" }, credential-reference={clear-text= "pass123+" }, create= true ) /subsystem=elytron/credential-store=MyStore2:add(modifiable= true , implementation-properties={ "keyStoreType" => "PKCS11" , "external" => " true " , "keyAlias" => "my-key" }, credential-reference={clear-text= "pass123+" }, create= true ) /subsystem=elytron/credential-store=MyStore3:add(modifiable= true , implementation-properties={ "keyStoreType" => "PKCS11" , "external" => " true " , "keyAlias" => "my-key" }, credential-reference={clear-text= "pass123+" }, create= true ) /subsystem=elytron/credential-store=MyStore4:add(modifiable= true , implementation-properties={ "keyStoreType" => "PKCS11" , "external" => " true " , "keyAlias" => "my-key" }, credential-reference={clear-text= "pass123+" }, create= true ) /subsystem=elytron/credential-store=MyStore5:add(modifiable= true , implementation-properties={ "keyStoreType" => "PKCS11" , "external" => " true " , "keyAlias" => "my-key" }, credential-reference={clear-text= "pass123+" }, create= true ) Restart server couple of times until exception occures during server startup shutdown --restart
    • Release Notes
    • Workaround Exists
    • Hide

      Don't use multiple CS - use just one CS.

      Show
      Don't use multiple CS - use just one CS.

      In case there is multiple external PKCS11 credential stores configured, intermittently it happens on startup exception occurs. Seems more CS are configured, it is bigger chance to hit error. If only one CS is configured error does not occur.

      We have automatic tests with 3 CS and so far we have not hit this issue. With 5 CS from reproducer it happens nearly with each restart. I hit this during discussing analysis of advanced keystore features [1] with Farah.

      Multiple CS can be expected by users, when they want to logically separate CS files for some reason. However technically everything can be stored in one CS.

      That is analogy of https://issues.jboss.org/browse/JBEAP-11693 for PKCS11 credential store. Fix for that issue was proper synchronization of PKCS11 keystore loading. I assume something similar will be needed here.

      If there is missing synchronization of PKCS11 keystore in Credential store implementation, that can in theory occur in combination of 1 CS and multiple PKCS11 keystores as well. However this scenario is tested for and such problem haven't occurred yet.

      bmaxwell Is GSS is ok with this being critical since a workaround exists?

      External Credential Store is new feature of Elytron tracked by EAP7-277.

      KeyStoreException: invalid KeyStore state: found N secret keys sharing CKA_LABEL [my-key]
      09:56:15,574 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "elytron"),
    ("credential-store" => "MyStore")
]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.security.credential-store.MyStore" => "WFLYELY00004: Unable to start the service.
    Caused by: org.wildfly.security.credential.store.CredentialStoreException: ELY09514: Unable to initialize credential store
    Caused by: org.wildfly.security.credential.store.CredentialStoreException: java.security.KeyStoreException: expected but could not find secret key
    Caused by: java.security.KeyStoreException: expected but could not find secret key"}}
09:56:15,575 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "elytron"),
    ("credential-store" => "MyStore4")
]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.security.credential-store.MyStore4" => "WFLYELY00004: Unable to start the service.
    Caused by: org.wildfly.security.credential.store.CredentialStoreException: ELY09514: Unable to initialize credential store
    Caused by: org.wildfly.security.credential.store.CredentialStoreException: java.security.KeyStoreException: invalid KeyStore state: found 2 secret keys sharing CKA_LABEL [my-key]
    Caused by: java.security.KeyStoreException: invalid KeyStore state: found 2 secret keys sharing CKA_LABEL [my-key]"}}
09:56:15,576 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
    ("subsystem" => "elytron"),
    ("credential-store" => "MyStore5")
]) - failure description: {"WFLYCTL0080: Failed services" => {"org.wildfly.security.credential-store.MyStore5" => "WFLYELY00004: Unable to start the service.
    Caused by: org.wildfly.security.credential.store.CredentialStoreException: ELY09514: Unable to initialize credential store
    Caused by: org.wildfly.security.credential.store.CredentialStoreException: java.security.KeyStoreException: invalid KeyStore state: found 3 secret keys sharing CKA_LABEL [my-key]
    Caused by: java.security.KeyStoreException: invalid KeyStore state: found 3 secret keys sharing CKA_LABEL [my-key]"}}

      [1] https://developer.jboss.org/wiki/AnalysisDesign-AdvancedElytronKey-storeManipulationOperations

              thofman Tomas Hofman
              thofman Tomas Hofman
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: