We've got applications which use the IAIK JCE provider (http://jce.iaik.tugraz.at/sic/Products/Core-Crypto-Toolkits/JCA-JCE) whose library is included directly in our application.
On JBoss AS 5 and later, we can't use this provider as usually.
The new VFS mechanism seems to break the path determination to the IAIK provider library (needed to determine provider code base) - it still points into the ear/war structure inside the server deploy directory but as all this is packed into one ear file, the provider verification.fails because of unsigned ear/war and thus the following exception is thrown:

2010-04-01 15:04:20,034 INFO [STDOUT] (http-0.0.0.0-8180-1) Provider code base: jar:file:/opt/jboss-5.1.0.GA/server/test1/deploy/IAIKTest.ear!/IAIKTestWeb.war
...
2010-04-01 15:04:20,890 ERROR [STDERR] (http-0.0.0.0-8180-1) java.lang.SecurityException: JCE cannot authenticate the provider IAIK
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at javax.crypto.Cipher.getInstance(DashoA13*..)
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at javax.crypto.Cipher.getInstance(DashoA13*..)
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at iaik.test.ControllerServlet.service(ControllerServlet.java:123)
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
2010-04-01 15:04:20,893 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:905)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:592)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2036)
2010-04-01 15:04:20,894 ERROR [STDERR] (http-0.0.0.0-8180-1) at java.lang.Thread.run(Thread.java:619)
2010-04-01 15:04:20,895 ERROR [STDERR] (http-0.0.0.0-8180-1) Caused by: java.util.jar.JarException: Cannot parse jar:file:/opt/jboss-5.1.0.GA/server/test1/deploy/IAIKTest.ear!/IAIKTestWeb.war
2010-04-01 15:04:20,895 ERROR [STDERR] (http-0.0.0.0-8180-1) at javax.crypto.SunJCE_c.a(DashoA13*..)
2010-04-01 15:04:20,895 ERROR [STDERR] (http-0.0.0.0-8180-1) at javax.crypto.SunJCE_b.b(DashoA13*..)
2010-04-01 15:04:20,895 ERROR [STDERR] (http-0.0.0.0-8180-1) at javax.crypto.SunJCE_b.a(DashoA13*..)
2010-04-01 15:04:20,895 ERROR [STDERR] (http-0.0.0.0-8180-1) ... 24 more

This works when one of tho three mentioned workarounds is used.
The most interesting of course is the third workaround for which the code base corectly points to the temporary IAIK library instance in the java.io.tmpdir:
2010-04-01 15:09:51,291 INFO [STDOUT] (http-0.0.0.0-8180-1) Provider code base: file:/tmp/nestedjar2481829661547119990.tmp

I'll attach a simple testcase ear (IAIKTest.ear) which contains an evaluation version of IAIK provider library (thanks for permission to the IAIK authors) and a simple servlet (including source) which demonstrates the problem.
All you need to do is to have the unrestricted JCE policy files installed in your JRE and then just put the IAIKTest.ear into the deploy directory, start the JBoss instance and access the test servlet URL in browser, for example: http://localhost:8080/IAIKTestWeb/ControllerServlet
On init (which happens on start) it will register the IAIK provider
On service (after you access it in browser) it will attempt to get some cipher instance and here it fails.

I'll also attach two log files covering the problem:

• server-default.log with default server setup where the issue occurs
• server-forceVfsJar.log with jboss.vfs.forceVfsJar=true where the issue doesn't occur

Also I think this is more generic problem because we had similar issues, where some jar or resource location was pointing to the packed ear/war structure inside the deploy directory instead of those VFS temporary locations, with our own code (when attemting to obtain full path) and had to perform some modifications in order to make it working. The same set of workarounds could resolve these issues as well.