[JBEAP-25489] [GSS](8.0.z) Artemis is logging warnings during clean shutdown of server in cluster

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: 8.0 Update 2
Affects Version/s: 8.0.0.Beta, 8.0.0.GA-CR1
Component/s: ActiveMQ
Labels:
- eap
- test_included

Blocked:
False
Blocked Reason:
None
Ready:
False
CDW devel_ack:
CDW docs_ack:
CDW pm_ack:
CDW qa_ack:
CDW release:
QE Test Coverage:
+
Release Note Type:
Known Issue
Target Release:

8.0.z.GA
Workaround Description:

Hide

None

Show
None
Git Pull Request:
https://github.com/rh-messaging/activemq-artemis/commit/fe70ef7fc9420810dbba29c53f2d8eaaf5b8fd3f
Steps to Reproduce:
Hide

Steps to reproduce:

git clone -b master git@gitlab.mw.lab.eng.bos.redhat.com:mnovak/messaging-testsuite.git messaging-testsuite cd messaging-testsuite/scripts/ groovy -DEAP_ZIP_URL=https://jenkins.eapqe.psi.redhat.com/job/eap-8.x-messaging-testing-prepare/1906/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean install -B -Dartemis.version=2.26.0 -Deap7.org.jboss.qa.hornetq.apps.clients.version=8.1692999002-SNAPSHOT -DreuseForks=false -Dmaven.test.failure.ignore=true -Deap7.clients.version=8.1692999002-SNAPSHOT -Dsurefire.failIfNoSpecifiedTests=false -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -Dtest=ClusterTestCase#testNoWarningErrorsDuringRestartingNodesInCluster | tee log
Show
Steps to reproduce: git clone -b master git@gitlab.mw.lab.eng.bos.redhat.com:mnovak/messaging-testsuite.git messaging-testsuite cd messaging-testsuite/scripts/ groovy -DEAP_ZIP_URL=https: //jenkins.eapqe.psi.redhat.com/job/eap-8.x-messaging-testing-prepare/1906/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean install -B -Dartemis.version=2.26.0 -Deap7.org.jboss.qa.hornetq.apps.clients.version=8.1692999002-SNAPSHOT -DreuseForks= false -Dmaven.test.failure.ignore= true -Deap7.clients.version=8.1692999002-SNAPSHOT -Dsurefire.failIfNoSpecifiedTests= false -Dmaven.wagon.http.ssl.insecure= true -Dmaven.wagon.http.ssl.allowall= true -Dtest=ClusterTestCase#testNoWarningErrorsDuringRestartingNodesInCluster | tee log
Intelligence Requested:
Market:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

When shutting down the server in cluster, Artemis is sometimes logging warnings:

21:54:05,122 WARN  [org.apache.activemq.artemis.core.server] (Thread-0 (ActiveMQ-client-netty-threads)) AMQ222295: There is a possible split brain on nodeID db71c866-4391-11ee-863c-fa163e50fa38. Topology update ignored

22:38:32,250 WARN  [org.apache.activemq.artemis.core.client] (Thread-3 (ActiveMQ-scheduled-threads)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
        at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:497)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory.run(ActiveMQThreadFactory.java:118)

22:38:06,362 WARN  [org.apache.activemq.artemis.core.client] (Thread-20 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@120a57f2)) AMQ212004: Failed to connect to server.

22:38:16,533 WARN  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 39) AMQ222002: Timed out waiting for pool to terminate org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@69043f7c[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 204]. Interrupting all its threads!

22:38:16,533 WARN  [org.apache.activemq.artemis.core.client] (Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@120a57f2)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
        at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.run(ServerLocatorImpl.java:491)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory.run(ActiveMQThreadFactory.java:118)

Expectation is that during clean shutdown no warnings/errors will be logged.

Customer Impact: All warnings are harmless but restarting EAP server is normal admin operation and server should not log any warning/error. Log monitoring might cause false alarm on customer side.

is cloned by

ENTMQBR-8407 Artemis is logging warnings during clean shutdown of server in cluster

Closed

is incorporated by

JBEAP-25251 (8.0.z) Upgrade Artemis from 2.21.0.redhat-00045 to 2.21.0.redhat-00048

Closed

Michaela Osmerova made changes - 2024/08/19 1:56 PM

Status

Original: Verified [ 10015 ]

New: Closed [ 6 ]

Michaela Osmerova added a comment - 2024/08/19 1:56 PM

As all verified issues should change to closed status, closing them with the bulk update.

Michaela Osmerova added a comment - 2024/08/19 1:56 PM As all verified issues should change to closed status, closing them with the bulk update.

Daniel Cihak made changes - 2024/05/28 11:33 AM

Status

Original: QA In Progress [ 10000 ]

New: Verified [ 10015 ]

Daniel Cihak made changes - 2024/05/28 11:02 AM

Status

Original: Ready for QA [ 10007 ]

New: QA In Progress [ 10000 ]

Daniel Kreling made changes - 2024/05/20 3:49 PM

Status

Original: Resolved [ 5 ]

New: Ready for QA [ 10007 ]

Ivo Studensky made changes - 2024/04/04 8:42 AM

Resolution		New: Done [ 1 ]
Status	Original: Open [ 1 ]	New: Resolved [ 5 ]

Ivo Studensky made changes - 2024/04/04 8:41 AM

Status

Original: New [ 10016 ]

New: Open [ 1 ]

Ivo Studensky made changes - 2024/04/04 8:41 AM

Fix Version/s		New: 8.0 Update 2 [ 12399052 ]
Fix Version/s	Original: 8.0 Update 3 [ 12424460 ]

Ivo Studensky made changes - 2024/04/03 10:39 AM

Fix Version/s		New: 8.0 Update 3 [ 12424460 ]
Fix Version/s	Original: 8.0 Update 2 [ 12399052 ]

Daniel Cihak made changes - 2024/03/19 7:48 AM

QE Test Coverage		New: + [ 11656 ]
Description	Original: When shutting down the server in cluster, Artemis is sometimes logging warnings: {code:java} 21:54:05,122 WARN [org.apache.activemq.artemis.core.server] (Thread-0 (ActiveMQ-client-netty-threads)) AMQ222295: There is a possible split brain on nodeID db71c866-4391-11ee-863c-fa163e50fa38. Topology update ignored {code} {code:java} 22:38:32,250 WARN [org.apache.activemq.artemis.core.client] (Thread-3 (ActiveMQ-scheduled-threads)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081) at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369) at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:497) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) {code} {code:java} 22:38:06,362 WARN [org.apache.activemq.artemis.core.client] (Thread-20 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@120a57f2)) AMQ212004: Failed to connect to server. {code} {code:java} 22:38:16,533 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 39) AMQ222002: Timed out waiting for pool to terminate org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@69043f7c[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 204]. Interrupting all its threads! {code} {code:java} 22:38:16,533 WARN [org.apache.activemq.artemis.core.client] (Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@120a57f2)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081) at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369) at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:491) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) {code} Expectation is that during clean shutdown no warnings/errors will be logged. Customer Impact: All warnings are harmless but restarting EAP server is normal admin operation and server should not log any warning/error. Log monitoring might cause false alarm on customer side.	New: When shutting down the server in cluster, Artemis is sometimes logging warnings: {code:java} 21:54:05,122 WARN [org.apache.activemq.artemis.core.server] (Thread-0 (ActiveMQ-client-netty-threads)) AMQ222295: There is a possible split brain on nodeID db71c866-4391-11ee-863c-fa163e50fa38. Topology update ignored {code} {code:java} 22:38:32,250 WARN [org.apache.activemq.artemis.core.client] (Thread-3 (ActiveMQ-scheduled-threads)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081) at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369) at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:497) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory.run(ActiveMQThreadFactory.java:118) {code} {code:java} 22:38:06,362 WARN [org.apache.activemq.artemis.core.client] (Thread-20 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@120a57f2)) AMQ212004: Failed to connect to server. {code} {code:java} 22:38:16,533 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 39) AMQ222002: Timed out waiting for pool to terminate org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@69043f7c[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 204]. Interrupting all its threads! {code} {code:java} 22:38:16,533 WARN [org.apache.activemq.artemis.core.client] (Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@120a57f2)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081) at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369) at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528) at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.run(ServerLocatorImpl.java:491) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory.run(ActiveMQThreadFactory.java:118) {code} Expectation is that during clean shutdown no warnings/errors will be logged. Customer Impact: All warnings are harmless but restarting EAP server is normal admin operation and server should not log any warning/error. Log monitoring might cause false alarm on customer side.
Labels	Original: eap	New: eap test_included

Ivo Studensky made changes - 2024/03/18 8:08 PM

Git Pull Request

New: https://github.com/rh-messaging/activemq-artemis/commit/fe70ef7fc9420810dbba29c53f2d8eaaf5b8fd3f

Brad Maxwell made changes - 2024/03/18 6:52 PM

Fix Version/s

New: 8.0 Update 2 [ 12399052 ]

Brad Maxwell made changes - 2024/03/18 6:52 PM

Link

New: This issue is incorporated by ~~JBEAP-25251~~ [ ~~JBEAP-25251~~ ]

Brad Maxwell made changes - 2024/03/18 6:43 PM

Summary

Original: (8.0.z) Artemis is logging warnings during clean shutdown of server in cluster

New: [GSS](8.0.z) Artemis is logging warnings during clean shutdown of server in cluster

Darran Lofthouse made changes - 2023/12/06 1:38 PM

Release Note Type

New: Known Issue [ 27783 ]

Ingo Weiss made changes - 2023/11/20 4:28 PM

Priority

Original: Blocker [ 1 ]

New: Critical [ 2 ]

Darran Lofthouse made changes - 2023/10/23 3:12 PM

Target Release

Original: 8.0.0.GA [ 12353898 ]

New: 8.0.z.GA [ 12402757 ]

Darran Lofthouse made changes - 2023/10/23 2:53 PM

Labels

New: eap

Miroslav Novak made changes - 2023/10/03 1:52 PM

Affects Version/s

New: 8.0.0.GA-CR1 [ 12400124 ]

Emmanuel Hugonnet made changes - 2023/09/22 9:41 AM

Link

New: This issue is cloned by ~~JBEAP-25689~~ [ ~~JBEAP-25689~~ ]

Miroslav Novak made changes - 2023/08/29 8:22 AM

Description

Original: When shutting down the server in cluster, Artemis is sometimes logging warnings:
{code:java}
21:54:05,122 WARN [org.apache.activemq.artemis.core.server] (Thread-0 (ActiveMQ-client-netty-threads)) AMQ222295: There is a possible split brain on nodeID db71c866-4391-11ee-863c-fa163e50fa38. Topology update ignored
{code}
{code:java}
22:38:32,250 WARN [org.apache.activemq.artemis.core.client] (Thread-3 (ActiveMQ-scheduled-threads)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
        at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:497)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
{code}
{code:java}
22:38:06,362 WARN [org.apache.activemq.artemis.core.client] (Thread-20 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@120a57f2)) AMQ212004: Failed to connect to server.
{code}
{code:java}
22:38:16,533 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 39) AMQ222002: Timed out waiting for pool to terminate org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@69043f7c[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 204]. Interrupting all its threads!
{code}
{code:java}
22:38:16,533 WARN [org.apache.activemq.artemis.core.client] (Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@120a57f2)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
        at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:491)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
{code}

Expectation is that during clean shutdown no warnings/errors will be logged.

*Customer Impact:* Restarting EAP server is normal admin operation and server should not log any warning/error.

New: When shutting down the server in cluster, Artemis is sometimes logging warnings:
{code:java}
21:54:05,122 WARN [org.apache.activemq.artemis.core.server] (Thread-0 (ActiveMQ-client-netty-threads)) AMQ222295: There is a possible split brain on nodeID db71c866-4391-11ee-863c-fa163e50fa38. Topology update ignored
{code}
{code:java}
22:38:32,250 WARN [org.apache.activemq.artemis.core.client] (Thread-3 (ActiveMQ-scheduled-threads)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
        at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:497)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
{code}
{code:java}
22:38:06,362 WARN [org.apache.activemq.artemis.core.client] (Thread-20 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@120a57f2)) AMQ212004: Failed to connect to server.
{code}
{code:java}
22:38:16,533 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 39) AMQ222002: Timed out waiting for pool to terminate org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@69043f7c[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 204]. Interrupting all its threads!
{code}
{code:java}
22:38:16,533 WARN [org.apache.activemq.artemis.core.client] (Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@120a57f2)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
        at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:491)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
{code}

Expectation is that during clean shutdown no warnings/errors will be logged.

*Customer Impact:* All warnings are harmless but restarting EAP server is normal admin operation and server should not log any warning/error. Log monitoring might cause false alarm on customer side.

Miroslav Novak made changes - 2023/08/29 8:19 AM

Description

New: When shutting down the server in cluster, Artemis is sometimes logging warnings:
{code:java}
21:54:05,122 WARN [org.apache.activemq.artemis.core.server] (Thread-0 (ActiveMQ-client-netty-threads)) AMQ222295: There is a possible split brain on nodeID db71c866-4391-11ee-863c-fa163e50fa38. Topology update ignored
{code}
{code:java}
22:38:32,250 WARN [org.apache.activemq.artemis.core.client] (Thread-3 (ActiveMQ-scheduled-threads)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
        at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.server.cluster.ClusterController$ConnectRunnable.run(ClusterController.java:497)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
{code}
{code:java}
22:38:06,362 WARN [org.apache.activemq.artemis.core.client] (Thread-20 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@120a57f2)) AMQ212004: Failed to connect to server.
{code}
{code:java}
22:38:16,533 WARN [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 39) AMQ222002: Timed out waiting for pool to terminate org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@69043f7c[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 204]. Interrupting all its threads!
{code}
{code:java}
22:38:16,533 WARN [org.apache.activemq.artemis.core.client] (Thread-4 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@120a57f2)) AMQ212064: Unable to receive cluster topology : java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1081)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
        at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:278)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.waitForTopology(ClientSessionFactoryImpl.java:525)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:741)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:549)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:528)
        at org.apache.activemq.artemis@2.21.0.redhat-00044//org.apache.activemq.artemis.core.client.impl.ServerLocatorImpl$4.run(ServerLocatorImpl.java:491)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:56)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:67)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.journal//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
{code}

Expectation is that during clean shutdown no warnings/errors will be logged.

*Customer Impact:* Restarting EAP server is normal admin operation and server should not log any warning/error.

Miroslav Novak made changes - 2023/08/29 8:06 AM

Priority

Original: Major [ 3 ]

New: Blocker [ 1 ]

Miroslav Novak made changes - 2023/08/28 8:55 AM

Summary

Original: (8.0.z) Artemis is logging errors/warnings during clean shutdown of server in cluster

New: (8.0.z) Artemis is logging warnings during clean shutdown of server in cluster

Miroslav Novak made changes - 2023/08/28 8:54 AM

Assignee

New: Emmanuel Hugonnet [ ehugonne1@redhat.com ]

Miroslav Novak made changes - 2023/08/28 8:54 AM

Component/s

New: ActiveMQ [ 12322543 ]

Miroslav Novak made changes - 2023/08/28 8:54 AM

Affects Version/s

New: 8.0.0.Beta [ 12375711 ]

Miroslav Novak created issue - 2023/08/28 8:54 AM

Assignee:: Emmanuel Hugonnet

Reporter:: Miroslav Novak

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Created:: 2023/08/28 8:54 AM

Updated:: 2024/08/19 1:56 PM

Resolved:: 2024/04/04 8:42 AM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

Collapse comment: Michaela Osmerova added a comment - 2024/08/19 1:56 PM

Expand comment: Michaela Osmerova added a comment - 2024/08/19 1:56 PM

People

Dates