Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-27331

XP5 CR2 (Micrometer + OTel collector integration) - JMX metrics are not exported

XMLWordPrintable

      We've backported tests from https://github.com/wildfly/wildfly/pull/17898 to the EAP MicroProfile TS [1] in order to have Micrometer + OTel integration tested in EAP XP 5 CR2.

      One test is failing regularly in MicrometerOtelIntegrationTestCase [2], on an assertion that verifies the value for some JMX metrics is not 0:

      [ERROR] Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.292 s <<< FAILURE! - in org.jboss.eap.qe.micrometer.MicrometerOtelIntegrationTestCase
      [ERROR] org.jboss.eap.qe.micrometer.MicrometerOtelIntegrationTestCase.testJmxMetrics  Time elapsed: 1.021 s  <<< FAILURE!
      java.lang.AssertionError: Metric value should be non-zero: thread_max_count. Actual: 0
      	at org.junit.Assert.fail(Assert.java:89)
      	at org.junit.Assert.failEquals(Assert.java:187)
      	at org.junit.Assert.assertNotEquals(Assert.java:163)
      	at org.jboss.eap.qe.micrometer.MicrometerOtelIntegrationTestCase.lambda$testJmxMetrics$3(MicrometerOtelIntegrationTestCase.java:143)
      	at java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4204)
      	at org.jboss.eap.qe.micrometer.MicrometerOtelIntegrationTestCase.testJmxMetrics(MicrometerOtelIntegrationTestCase.java:142)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
      	...
      	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
      	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
      	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
      
      22:14:53,775 INFO  [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0272: Suspending server
      22:14:53,778 INFO  [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
      22:14:53,779 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
      22:14:53,780 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
      22:14:53,780 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0008: Undertow HTTP listener default suspending
      22:14:53,780 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0008: Undertow HTTPS listener https suspending
      22:14:53,780 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = h2
      22:14:53,780 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 127.0.0.1:8443
      22:14:53,780 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 127.0.0.1:8080
      22:14:53,781 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0004: Undertow 2.3.11.SP1-redhat-00001 stopping
      22:14:53,783 INFO  [org.jboss.as] (MSC service thread 1-2) WFLYSRV0050: JBoss EAP 8.0 Update 2.0 (WildFly Core 21.0.9.Final-redhat-00001) stopped in 3ms
      [INFO] 
      [INFO] Results:
      [INFO] 
      [ERROR] Failures: 
      [ERROR]   MicrometerOtelIntegrationTestCase.testJmxMetrics:142->lambda$testJmxMetrics$3:143 Metric value should be non-zero: thread_max_count. Actual: 0
      [INFO] 
      [ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0
      

      From the investigation we could quickly perform, the issue seems to be with the selected JMX metrics, which all have a value of 0.
      Also, we verified that the server is properly using a WildFlyOtlpRegistry instance, so the issue doesn't depend on a broken Docker configuration that would instead make the server fallback to using NoOpRegistry.
      By looking at the code at runtime, we noticed that the metric metadata expose attribute names with different case, e.g.: "Thread_count" vs. the expected "thread_count", that is the root cause for not exporting the metric [3][4].

      Some configuration and tooling implementation details (e.g.: Docker APIs) in the backported code are different from the original PR, so feel free to let us know whether we're doing something wrong, or missing anything here, CC jaslee@redhat.com, bstansbe@redhat.com

      [1]
      https://github.com/jboss-eap-qe/eap-microprofile-test-suite/pull/289/files#top

      [2]
      https://github.com/jboss-eap-qe/eap-microprofile-test-suite/pull/289/files#diff-1e6a1ab0475047532fd416e240a456c8e8d51ebdf73b4d004b671906f57165e9R130

      [3]
      https://github.com/jbossas/jboss-eap8/blob/xp-5.0.x/observability/micrometer/src/main/java/org/wildfly/extension/micrometer/jmx/JmxMicrometerCollector.java#L212

      [4]
      https://github.com/jbossas/jboss-eap8/blob/xp-5.0.x/observability/micrometer/src/main/java/org/wildfly/extension/micrometer/jmx/JmxMicrometerCollector.java#L224

            jaslee@redhat.com Jason Lee
            fburzigo Fabio Burzigotti
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: