Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-12504

Memoryleak in Metrics-Endpoint

    XMLWordPrintable

Details

    • Hide 
      docker run -it --rm -p 8080:8080 -p 9990:9990 jboss/keycloak:6.0.1 -Djboss.bind.address.management=0.0.0.0 -b 0.0.0.0

      Make a lot requests to :9990/metrics

      Show
      docker run -it --rm -p 8080:8080 -p 9990:9990 jboss/keycloak:6.0.1 -Djboss.bind.address.management=0.0.0.0 -b 0.0.0.0 Make a lot requests to :9990/metrics

    Description

      I noticed there seems to be a memory leak when calling the /metrics endpoint (with Prometheus). I don't know if this relates to the Keycloak metrics or the general Wildfly base metrics.

      Every request to /metrics seems to consume some more heap memory until a OOME occurs. No other requests are done.

      If the metrics endpoint isn't used at all the server works fine. 

      15:14:50,951 ERROR [org.jboss.as.controller.management-operation] (DeploymentScanner-threads - 1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([("deployment" => "keycloak-server.war")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.HashMap.resize(HashMap.java:704)
        at java.util.HashMap.putVal(HashMap.java:629)
        at java.util.HashMap.put(HashMap.java:612)
        at java.util.HashSet.add(HashSet.java:220)
        at org.jboss.as.controller.PathAddress.pathAddress(PathAddress.java:95)
        at org.jboss.as.controller.AbstractOperationContext$OperationId.<init>(AbstractOperationContext.java:1571)
        at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1326)
        at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1321)
        at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.doExecuteInternal(ReadAttributeHandler.java:244)
        at org.jboss.as.controller.operations.global.ReadAttributeHandler$AuthorizeAttributeReadHandler.execute(ReadAttributeHandler.java:229)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
        at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
        at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
        at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:255)
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
        at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient$$Lambda$552/516471669.apply(Unknown Source)
        at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeInModelControllerCl(ModelControllerClientFactoryImpl.java:259)
        at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:157)
        at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$546/98454422.apply(Unknown Source)
        at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:351)
        at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$550/226565561.run(Unknown Source)
        at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
        at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:351)
        at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:59)
        at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:82)
        at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)

15:14:50,955 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0012: Scan of /opt/jboss/keycloak/standalone/deployments threw Exception: java.lang.RuntimeException: WFLYDS0036: Deployment model operation failed. undefined
        at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getUnrelatedDeployments(DefaultDeploymentOperations.java:109)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1692)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1640)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
        at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at org.jboss.threads.JBossThread.run(JBossThread.java:485)

15:14:52,319 ERROR [org.jboss.as.controller.management-operation] (management I/O-1) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
    ("subsystem" => "jgroups"),
    ("channel" => "ee"),
    ("protocol" => "pbcast.NAKACK2")
]): java.lang.OutOfMemoryError: GC overhead limit exceeded
        at sun.reflect.generics.parser.SignatureParser.parsePackageNameAndSimpleClassTypeSignature(SignatureParser.java:331)
        at sun.reflect.generics.parser.SignatureParser.parseClassTypeSignature(SignatureParser.java:310)
        at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:289)
        at sun.reflect.generics.parser.SignatureParser.parseFieldTypeSignature(SignatureParser.java:283)
        at sun.reflect.generics.parser.SignatureParser.parseTypeSignature(SignatureParser.java:485)
        at sun.reflect.generics.parser.SignatureParser.parseTypeSig(SignatureParser.java:188)
        at sun.reflect.annotation.AnnotationParser.parseSig(AnnotationParser.java:436)
        at sun.reflect.annotation.AnnotationParser.parseAnnotation2(AnnotationParser.java:241)
        at sun.reflect.annotation.AnnotationParser.parseAnnotations2(AnnotationParser.java:120)
        at sun.reflect.annotation.AnnotationParser.parseAnnotations(AnnotationParser.java:72)
        at java.lang.reflect.Field.declaredAnnotations(Field.java:1150)
        at java.lang.reflect.Field.declaredAnnotations(Field.java:1148)
        at java.lang.reflect.Field.getAnnotation(Field.java:1120)
        at org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:926)
        at org.jgroups.stack.Configurator.resolveAndAssignFields(Configurator.java:921)
        at org.jgroups.stack.Protocol.setProperties(Protocol.java:133)
        at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.lambda$createProtocol$0(AbstractProtocolConfigurationServiceConfigurator.java:119)
        at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator$$Lambda$590/1559509553.run(Unknown Source)
        at org.wildfly.security.manager.WildFlySecurityManager.doUnchecked(WildFlySecurityManager.java:849)
        at org.jboss.as.clustering.jgroups.subsystem.AbstractProtocolConfigurationServiceConfigurator.createProtocol(AbstractProtocolConfigurationServiceConfigurator.java:125)
        at org.jboss.as.clustering.jgroups.subsystem.ChannelRuntimeResourceRegistration.findProtocol(ChannelRuntimeResourceRegistration.java:87)
        at org.jboss.as.clustering.jgroups.subsystem.ProtocolMetricsHandler.executeRuntimeStep(ProtocolMetricsHandler.java:244)
        at org.jboss.as.controller.AbstractRuntimeOnlyHandler$1.execute(AbstractRuntimeOnlyHandler.java:59)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:999)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:743)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:467)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1412)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:423)
        at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
        at org.jboss.as.controller.ModelControllerImpl$$Lambda$553/974545249.run(Unknown Source)
        at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$554/2003274744.run(Unknown Source)
        at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:289)

15:14:52,322 ERROR [io.undertow.request] (management I/O-1) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute xmit_table_num_purges on [
    ("subsystem" => "jgroups"),
    ("channel" => "ee"),
    ("protocol" => "pbcast.NAKACK2")
]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
        at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
        at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
        at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
        at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
        at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
        at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
        at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
        at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
        at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
        at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
        at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
        at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
        at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
        at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
        at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

15:14:58,673 ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
    ("subsystem" => "jgroups"),
    ("channel" => "ee"),
    ("protocol" => "UDP")
]): java.lang.OutOfMemoryError: GC overhead limit exceeded

15:14:58,675 ERROR [io.undertow.request] (management I/O-2) UT005071: Undertow request failed HttpServerExchange{ GET /metrics request {Accept=[text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8], Connection=[keep-alive], Accept-Language=[de,en-US;q=0.7,en;q=0.3], Cache-Control=[max-age=0], Accept-Encoding=[gzip, deflate], User-Agent=[Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0], Upgrade-Insecure-Requests=[1], Host=[192.168.99.100:9990]} response {Access-Control-Allow-Origin=[*], Access-Control-Allow-Headers=[origin, content-type, accept, authorization], Access-Control-Allow-Credentials=[true], Content-Type=[text/plain], Access-Control-Allow-Methods=[GET, POST, PUT, DELETE, OPTIONS, HEAD], Access-Control-Max-Age=[1209600]}}: java.lang.IllegalStateException: WFLYMETRICS0003: Unable to read attribute max_bundle_size on [
    ("subsystem" => "jgroups"),
    ("channel" => "ee"),
    ("protocol" => "UDP")
]: "WFLYCTL0158: Operation handler failed: java.lang.OutOfMemoryError: GC overhead limit exceeded".
        at org.wildfly.extension.microprofile.metrics.MetricCollector.lambda$collectResourceMetrics0$0(MetricCollector.java:143)
        at org.wildfly.extension.microprofile.metrics.PrometheusCollector.collect(PrometheusCollector.java:51)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:168)
        at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:130)
        at org.wildfly.extension.microprofile.metrics.MetricsContextService$1.handleRequest(MetricsContextService.java:100)
        at org.jboss.as.domain.http.server.security.RealmReadinessHandler.handleRequest(RealmReadinessHandler.java:51)
        at org.jboss.as.domain.http.server.security.ServerErrorReadinessHandler.handleRequest(ServerErrorReadinessHandler.java:35)
        at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:91)
        at io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:211)
        at io.undertow.server.handlers.cache.CacheHandler.handleRequest(CacheHandler.java:92)
        at io.undertow.server.handlers.error.SimpleErrorPageHandler.handleRequest(SimpleErrorPageHandler.java:78)
        at io.undertow.server.handlers.CanonicalPathHandler.handleRequest(CanonicalPathHandler.java:49)
        at org.jboss.as.domain.http.server.ManagementHttpRequestHandler.handleRequest(ManagementHttpRequestHandler.java:57)
        at org.jboss.as.domain.http.server.cors.CorsHttpHandler.handleRequest(CorsHttpHandler.java:75)
        at org.jboss.as.domain.http.server.ManagementHttpServer$UpgradeFixHandler.handleRequest(ManagementHttpServer.java:662)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364)
        at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
        at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
        at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
        at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
        at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
        at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)

      Attachments

        Issue Links

          is caused by

          Bug - A problem that impairs or prevents the functions of the product. MSC-245 ServiceContainerImpl.registry is leaking memory resources

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          relates to

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. WFCORE-4663 Upgrade JBoss MSC to 1.4.11.Final

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. WFCORE-4660 Upgrade JBoss MSC to 1.4.10.Final

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              ropalka Richard Opalka
              mwerlitz Mathias Werlitz (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: