Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-32303

[GSS](7.4.z) WFLYCTL0013, WFLYCTL0216 ERROR logs are repeatedly generated when accessing the /metrics endpoint after a deployment is disabled or undeployed

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 7.4.23.GA
    • Clustering, MP Metrics
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      Steps to Reproduce
      1.  Put a web application (e.g., test.war) that having <distributable/> utilizes session replication caching, in "depoyments" folder.
      2.  Start JBoss EAP 7.4 with the default standalone.xml configuration that includes the metrics and infinispan (web cache) subsystems.
      3.  Verify that metrics are collected correctly: curl http://localhost:9990/metrics
      4.  Disable or undeploy the application.

      • Delete test.war from deployments directory.
      • OR execute "undeploy test.war" or "deployment disable test.war" in JBoss-CLI

      5.  Access the metrics endpoint again: `curl http://localhost:9990/metrics`.
      6.  Check `server.log`.

      Actual Results
      The log is flooded with WFLYCTL0013 / WFLYCTL0216 ERROR messages for every attribute of the missing Infinispan cache resource.

      Expected Results
      Metrics associated with the disabled/undeployed application should be automatically unregistered from the WildFlyMetricRegistry. No ERROR logs should be produced during metric collection.
       

      Show
      Steps to Reproduce 1.  Put a web application (e.g., test.war) that having <distributable/> utilizes session replication caching, in "depoyments" folder. 2.  Start JBoss EAP 7.4 with the default standalone.xml configuration that includes the metrics and infinispan (web cache) subsystems. 3.  Verify that metrics are collected correctly: curl http://localhost:9990/metrics 4.  Disable or undeploy the application. Delete test.war from deployments directory. OR execute "undeploy test.war" or "deployment disable test.war" in JBoss-CLI 5.  Access the metrics endpoint again: `curl http://localhost:9990/metrics `. 6.  Check `server.log`. Actual Results The log is flooded with WFLYCTL0013 / WFLYCTL0216 ERROR messages for every attribute of the missing Infinispan cache resource. Expected Results Metrics associated with the disabled/undeployed application should be automatically unregistered from the WildFlyMetricRegistry. No ERROR logs should be produced during metric collection.  

      Environment

      • Product/Version: JBoss EAP 7.4
      • Subsystem: Metrics, Infinispan

      If a deployment (e.g., test.war) that uses Infinispan for web session clustering is disabled or undeployed, subsequent requests to the MicroProfile Metrics endpoint (/metrics) trigger multiple ERROR messages in server.log.

      This occurs because the Infinispan cache resources associated with the deployment are removed from the management model, but the Metrics subsystem fails to unregister the corresponding metrics. Consequently, the Prometheus exporter attempts to read attributes from non-existent management resources.

      ERROR [org.jboss.as.controller.management-operation] (management I/O-2) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
    ("subsystem" => "infinispan"),
    ("cache-container" => "web"),
    ("cache" => "test.war")
]) - failure description: "WFLYCTL0216: Management resource '[
    (\"subsystem\" => \"infinispan\"),
    (\"cache-container\" => \"web\"),
    (\"cache\" => \"test.war\")
]' not found"

      Additional Notes:

      • This issue is partially mitigated in JBoss EAP 8.x by WFCORE-6476 and JBEAP-25654, which reduced the logging level of failed internal read-only operations to DEBUG.
      • However, the root cause (the unregistered metrics remain as "ghost" entries in the registry) is still there. So, you can still see the same log message when enabling DEBUG logging for "org.jboss.as.controller.management-operation" category in JBoss EAP 8.x.

        1. byteman-script.btm
          1 kB
        2. test.war
          0.7 kB

              jboss-set_jira JBoss SET
              rhn-support-mmiura Masafumi Miura
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: