Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-14525

Intermittent error due to ConcurrentModificationException while exporting metrics during server start / stop with standalone.xml

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 23.0.0.Final
    • 23.0.0.Beta1
    • MP Metrics
    • None
    • Hide

      run a long loop making 10000 requests to /metrics endpoint.

      for i in \{1..10000}
      do
       curl http://localhost:9990/metrics/
      done
      

      As long as the loop doesn't end, start and stop server with standalone.xml for a few (3-4) times, I can see that error with WFLY master.

      Show
      run a long loop making 10000 requests to /metrics endpoint. for i in \{1..10000} do curl http://localhost:9990/metrics/ done As long as the loop doesn't end, start and stop server with standalone.xml for a few (3-4) times, I can see that error with WFLY master.
    • Undefined
    • ---
    • ---

      Intermittent error due to ConcurrentModificationException by hitting http://localhost:9990/metrics/ in server start / stop process (started with standalone.xml, with no deployment).

      07:02:49,557 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 23.0.0.Final-SNAPSHOT (WildFly Core 15.0.0.Beta1) started in 3173ms - Started 319 of 558 services (344 services are lazy, passive or on-demand)
      07:02:49,560 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
      07:02:49,561 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
      ^C07:02:50,769 INFO [org.jboss.as.server] (Thread-1) WFLYSRV0272: Suspending server
      07:02:50,773 INFO [org.jboss.as.ejb3] (Thread-1) WFLYEJB0493: EJB subsystem suspension complete
      07:02:50,776 INFO [org.jboss.as.server] (Thread-1) WFLYSRV0220: Server shutdown has been requested via an OS signal
      07:02:50,774 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange\{ GET /metrics/}: java.util.ConcurrentModificationException
       at java.base/java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1208)
       at java.base/java.util.TreeMap$EntryIterator.next(TreeMap.java:1244)
       at java.base/java.util.TreeMap$EntryIterator.next(TreeMap.java:1239)
       at org.wildfly.extension.metrics@23.0.0.Final-SNAPSHOT//org.wildfly.extension.metrics.PrometheusExporter.export(PrometheusExporter.java:41)
       at org.wildfly.extension.metrics@23.0.0.Final-SNAPSHOT//org.wildfly.extension.metrics.MetricsContextService$2.handleRequest(MetricsContextService.java:96)
       at org.jboss.as.domain-http-interface@15.0.0.Beta1//org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
       at org.jboss.as.domain-http-interface@15.0.0.Beta1//org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
       at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:104)
       at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
       at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
       at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
       at io.undertow.core@2.2.4.Final//io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
       at org.jboss.as.domain-http-interface@15.0.0.Beta1//org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
       at org.jboss.as.domain-http-interface@15.0.0.Beta1//org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
       at org.jboss.as.domain-http-interface@15.0.0.Beta1//org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:717)
       at io.undertow.core@2.2.4.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
       at io.undertow.core@2.2.4.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
       at io.undertow.core@2.2.4.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
       at io.undertow.core@2.2.4.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
       at io.undertow.core@2.2.4.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
       at io.undertow.core@2.2.4.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
       at org.jboss.xnio@3.8.4.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
       at org.jboss.xnio@3.8.4.Final//org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
       at org.jboss.xnio@3.8.4.Final//org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
       at org.jboss.xnio@3.8.4.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
       at org.jboss.xnio.nio@3.8.4.Final//org.xnio.nio.QueuedNioTcpServer2.acceptTask(QueuedNioTcpServer2.java:178)
       at org.jboss.xnio.nio@3.8.4.Final//org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
       at org.jboss.xnio.nio@3.8.4.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
      
      07:02:50,779 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
      07:02:50,800 INFO [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0019: Host default-host stopping
      07:02:50,802 INFO [org.jboss.as.mail.extension] (MSC service thread 1-6) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
      07:02:50,802 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-7) WFLYJCA0019: Stopped Driver service with driver-name = h2
      07:02:50,805 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0008: Undertow HTTPS listener https suspending
      07:02:50,806 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
      07:02:50,807 INFO [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 127.0.0.1:8443
      07:02:50,807 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 127.0.0.1:8080
      07:02:50,808 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0004: Undertow 2.2.4.Final stopping
      07:02:50,829 INFO [org.jboss.as] (MSC service thread 1-3) WFLYSRV0050: WildFly Full 23.0.0.Final-SNAPSHOT (WildFly Core 15.0.0.Beta1) stopped in 45ms
      

      I run a long loop making 10000 requests to /metrics endpoint.

      for i in \{1..10000}
      do
       curl http://localhost:9990/metrics/
      done
      

      As long as the loop doesn't end, start and stop server with standalone.xml for a few (3-4) times, I can see that error with WFLY master.

              bstansbe@redhat.com Brian Stansberry
              chaowan@redhat.com Chao Wang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: