Uploaded image for project: 'HornetQ'
  1. HornetQ
  2. HORNETQ-1047

Fail-over after fail-back doesn't work in JBoss AS

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Done
    • 2.2.20.EAP5.GA/2.2.21.AS7.Final
    • 2.2.EAP5.Next
    • None
    • None
    • Hide

      I set up a simple Live/backup server pair with allow-failback=true using EAP 5.1.2 + HornetQ.

      I created a simple, stand-alone client accessing a topic:

      import java.util.Hashtable;
      
      import javax.jms.Connection;
      import javax.jms.ConnectionFactory;
      import javax.jms.Message;
      import javax.jms.MessageConsumer;
      import javax.jms.Session;
      import javax.jms.Topic;
      import javax.jms.TopicSession;
      import javax.naming.Context;
      import javax.naming.InitialContext;
      
      import org.jboss.logging.Logger;
      
      public class TopicClient {
          private static final Logger logger = Logger.getLogger(TopicClient.class);
      
          public static void main(String args[]) throws Exception {
              Connection conn = null;
              try {
                  Hashtable<Object, Object> env = new Hashtable<Object, Object>();
                  env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
                  env.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces");
                  env.put(Context.PROVIDER_URL, "jnp://127.0.0.1:1099,jnp://127.0.0.2:1099");
      
                  InitialContext iniCtx = new InitialContext(env);
      
                  ConnectionFactory tcf = (ConnectionFactory) iniCtx.lookup("ConnectionFactory");
                  conn = tcf.createConnection();
                  Topic topic = (Topic) iniCtx.lookup("/topic/testTopic");
                  Session session = conn.createSession(false, TopicSession.AUTO_ACKNOWLEDGE);
                  conn.start();
      
                  MessageConsumer recv = session.createConsumer(topic);
      
                  logger.info("Subscriber started");
                  Message msg = recv.receive(10);
                  if (msg != null) {
                      logger.info(msg.toString());
                  }
              } catch (Exception e) {
                  e.printStackTrace();
              } finally {
                  if (conn != null) {
                      conn.close();
                  }
              }
              logger.info("Subscriber stopped");
          }
      }
      
      1. Start the live.
      2. Start the backup.
      3. Run the client - success.
      4. Kill-9 the live. Backup becomes live.
      5. Run the client - success.
      6. Start the live. Live becomes live, backup becomes backup.
      7. Run the client - success.
      8. Kill-9 the live. Backup becomes live again.
      9. Run the client - fail:
      javax.jms.InvalidDestinationException: Topic testTopic does not exist
      	at org.hornetq.jms.client.HornetQSession.createConsumer(HornetQSession.java:545)
      	at org.hornetq.jms.client.HornetQSession.createConsumer(HornetQSession.java:383)
      	at org.hornetq.jms.client.HornetQSession.createConsumer(HornetQSession.java:353)
      	at TopicClient.main(TopicClient.java:35)
      
      Show
      I set up a simple Live/backup server pair with allow-failback=true using EAP 5.1.2 + HornetQ. I created a simple, stand-alone client accessing a topic: import java.util.Hashtable; import javax.jms.Connection; import javax.jms.ConnectionFactory; import javax.jms.Message; import javax.jms.MessageConsumer; import javax.jms.Session; import javax.jms.Topic; import javax.jms.TopicSession; import javax.naming.Context; import javax.naming.InitialContext; import org.jboss.logging.Logger; public class TopicClient { private static final Logger logger = Logger.getLogger(TopicClient.class); public static void main( String args[]) throws Exception { Connection conn = null ; try { Hashtable< Object , Object > env = new Hashtable< Object , Object >(); env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory" ); env.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces" ); env.put(Context.PROVIDER_URL, "jnp: //127.0.0.1:1099,jnp://127.0.0.2:1099" ); InitialContext iniCtx = new InitialContext(env); ConnectionFactory tcf = (ConnectionFactory) iniCtx.lookup( "ConnectionFactory" ); conn = tcf.createConnection(); Topic topic = (Topic) iniCtx.lookup( "/topic/testTopic" ); Session session = conn.createSession( false , TopicSession.AUTO_ACKNOWLEDGE); conn.start(); MessageConsumer recv = session.createConsumer(topic); logger.info( "Subscriber started" ); Message msg = recv.receive(10); if (msg != null ) { logger.info(msg.toString()); } } catch (Exception e) { e.printStackTrace(); } finally { if (conn != null ) { conn.close(); } } logger.info( "Subscriber stopped" ); } } Start the live. Start the backup. Run the client - success. Kill-9 the live. Backup becomes live. Run the client - success. Start the live. Live becomes live, backup becomes backup. Run the client - success. Kill-9 the live. Backup becomes live again. Run the client - fail: javax.jms.InvalidDestinationException: Topic testTopic does not exist at org.hornetq.jms.client.HornetQSession.createConsumer(HornetQSession.java:545) at org.hornetq.jms.client.HornetQSession.createConsumer(HornetQSession.java:383) at org.hornetq.jms.client.HornetQSession.createConsumer(HornetQSession.java:353) at TopicClient.main(TopicClient.java:35)

    Description

      When the backup server first starts, the application server scans all the HornetQ configuration files and deploys all the related HornetQ artifacts. However, since HornetQ is in backup mode it does not actually deploy all the destinations, connection factories, etc. so HornetQ catches the commands from the application server and holds on to them in a "command cache." Then when the live server fails and the backup becomes live it replays those cached commands so that all the necessary artifacts are deployed. When the real live server comes back and fail-back occurs the backup server because the actual backup again and destroys all the previously deployed artifacts. If the live server fails again and the backup becomes live (again) it won't deploy those HornetQ artifacts again because the command cache I mentioned previously is now empty. This is a weakness in the integration between the application server and HornetQ which (for now) requires the back-up server to be restarted after fail-back occurs. Short of restarting the server, one could redeploy (i.e. 'touch') the hornetq-jms.xml file. However, that is just a work-around.

      To fix the issue I believe we can simply not clear the command cache after the back-up is activated so that the commands are still present when the back-up becomes live again.

      Attachments

        Issue Links

          Activity

            People

              rhn-support-jbertram Justin Bertram
              rhn-support-jbertram Justin Bertram
              Archiver:
              samahaja@redhat.com Sagar Mahajan

              Dates

                Created:
                Updated:
                Resolved:
                Archived: