WildFly HornetQ Message Replication-Based HA Configuration: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
No edit summary
 
(34 intermediate revisions by the same user not shown)
Line 6: Line 6:


* [[WildFly HornetQ-Based Messaging Subsystem Configuration|HornetQ-Based Messaging Configuration]]
* [[WildFly HornetQ-Based Messaging Subsystem Configuration|HornetQ-Based Messaging Configuration]]
* [[WildFly HornetQ-Based Messaging Subsystem Concepts#Shared_Filesystem-based_Replication|Concepts: Shared Filesystem-based Replication]]
* [[WildFly HornetQ-Based Messaging Subsystem Concepts#Message_Replication|Concepts: Message Replication]]


=Overview=
=Overview=


For high availability purposes, the live server and the backup server must be installed on two separated physical (or virtual) hosts, provisioned in such a way to minimize the probability of both host failing at the same time. Highly available HornetQ requires access to reliable shared file system storage, so a file system such as GFS2 or a SAN must be made available to both hosts. HornetQ instances will store on the shared directory, among other things, their bindings and journal files. NFS v4 appropriately configured is also an option.
For high availability purposes, the live server and the backup server must be installed on two separated physical (or virtual) hosts, provisioned in such a way to minimize the probability of both host failing at the same time.


=WildFly Clustering and HornetQ High Availability=
=Configuration=


This document contains instructions for setting up a configuration where HornetQ HA is independently configured from WildFly clustering.
* <tt>[[WildFly HornetQ-Based Messaging Subsystem HA Configuration Parameters#shared-store|shared-store]]</tt>
 
* <tt>[[WildFly HornetQ-Based Messaging Subsystem HA Configuration Parameters#backup-group-name|backup-group-name]]</tt>
For more details see:
* <tt>[[WildFly HornetQ-Based Messaging Subsystem HA Configuration Parameters#check-for-live-server|check-for-live-server]]</tt>
 
* <tt>[[WildFly HornetQ-Based Messaging Subsystem HA Configuration Parameters#failover-on-shutdown|failover-on-shutdown]]</tt>
<blockquote style="background-color: #f9f9f9; border: solid thin lightgrey;">
* <tt>[[WildFly HornetQ-Based Messaging Subsystem HA Configuration Parameters#allow-failback|allow-failback]]</tt>
:[[WildFly HornetQ-Based Messaging Subsystem Concepts#WildFly_Clustering_and_HornetQ_High_Availability|Concepts: WildFly Clustering and HornetQ High Availability]]
* <tt>[[WildFly HornetQ-Based Messaging Subsystem HA Configuration Parameters#max-saved-replicated-journal-siz|max-saved-replicated-journal-siz]]</tt>
</blockquote>


=Procedure=
=Procedure=


==Common Configuration==
==Active Node Configuration==
 
Common configuration specification using system properties for per-server instance externalization makes sense for WildFly instance running in domain mode, because it permits the use of the same configuration for both the live server and the backup server. The differences in behavior are specified via system properties. In standalone mode, the sequences below can be copied and pasted in their respective <tt>standalone*.xml</tt> files.
 
===Shared-Storage Based High Availability===
 
Use the following "messaging" subsystem configuration on both live and stand-by servers. This is convenient because the servers can be made part of the same server group.


<pre>
<pre>
Line 36: Line 29:
<subsystem xmlns="urn:jboss:domain:messaging:1.4">  
<subsystem xmlns="urn:jboss:domain:messaging:1.4">  
   <hornetq-server>  
   <hornetq-server>  
      <backup>false</backup>
      <shared-store>false</shared-store>
      <backup-group-name>pair-A</backup-group-name>
      <check-for-live-server>true</check-for-live-server>
      <failover-on-shutdown>true</failover-on-shutdown>


       <persistence-enabled>true</persistence-enabled>
       <persistence-enabled>true</persistence-enabled>
      <cluster-user>hq</cluster-user>
      <cluster-password>hq123</cluster-password>
       ...
       ...
       <shared-store>true</shared-store>
        
       <backup>${jboss.messaging.hornetq.backup:false}</backup>
       <connectors>
      <create-bindings-dir>true</create-bindings-dir>
            ...
      <create-journal-dir>true</create-journal-dir>
            <netty-connector name="backup-node-connector" socket-binding="backup-node-hornetq-binding"/>
      <failover-on-shutdown>true</failover-on-shutdown>
      </connectors>


       <paging-directory path="paging" relative-to="hornetq.shared.dir"/>
       ...
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir"/>
      <journal-directory path="journal" relative-to="hornetq.shared.dir"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir"/>
        
        
      <cluster-connections>
          <cluster-connection name="pair-A-high-availability-cluster">
            <address>jms</address>
            <connector-ref>netty</connector-ref>
            <retry-interval>500</retry-interval>
            <use-duplicate-detection>true</use-duplicate-detection>
            <forward-when-no-consumers>true</forward-when-no-consumers>
            <max-hops>1</max-hops>
            <static-connectors>
                <connector-ref>backup-node-connector</connector-ref>
              </static-connectors>
          </cluster-connection>
      </cluster-connections>
       ...
       ...


Line 70: Line 84:
   </hornetq-server>
   </hornetq-server>
</subsystem>
</subsystem>
...
...
<socket-binding-group name="standard-sockets"...>
        ...
        <outbound-socket-binding name="backup-node-hornetq-binding">
            <remote-destination host="1.2.3.5" port="5445"/>
        </outbound-socket-binding>
</socket-binding-group>
</pre>
</pre>


===Shared Path Declaration===
==Backup Node Configuration==


This is usually common for the entire domain, so it can be specified in the domain top level section.
<pre>
...
<subsystem xmlns="urn:jboss:domain:messaging:1.4">
  <hornetq-server>


<pre>
      <backup>true</backup>
  ...
      <shared-store>false</shared-store>
  <paths>
       <backup-group-name>pair-A</backup-group-name>
       <path name="hornetq.shared.dir" path="/nfs/hornetq-shared-storage"/>
      <check-for-live-server>true</check-for-live-server>
  </paths>
      <failover-on-shutdown>true</failover-on-shutdown>
  ...
</pre>


==Live Server Configuration==
      <persistence-enabled>true</persistence-enabled>


<tt>jboss.messaging.hornetq.backup</tt> is by default false, but it's actually a good idea to make the configuration obvious. Add the following in the active node's <tt>host.xml</tt>:
      <cluster-user>hq</cluster-user>
      <cluster-password>hq123</cluster-password>


<pre>
      ...
<host ...>
     
  <system-properties>
      <connectors>
       <property name="jboss.messaging.hornetq.backup" value="false"/>
            ...
  </system-properties>
            <netty-connector name="active-node-connector" socket-binding="active-node-hornetq-binding"/>
  ...
      </connectors>
</host>
 
</pre>
       ...
     
      <cluster-connections>
          <cluster-connection name="pair-A-high-availability-cluster">
            <address>jms</address>
            <connector-ref>netty</connector-ref>
            <retry-interval>500</retry-interval>
            <use-duplicate-detection>true</use-duplicate-detection>
            <forward-when-no-consumers>true</forward-when-no-consumers>
            <max-hops>1</max-hops>
            <static-connectors>
                <connector-ref>active-node-connector</connector-ref>
              </static-connectors>
          </cluster-connection>
      </cluster-connections>


==Stand-By Server Configuration==
      ...


<tt>jboss.messaging.hornetq.backup</tt> should be set to "<tt>true</tt>" in the stand-by node's <tt>host.xml</tt>:
      <!--
      <jms-connection-factories>
        ...
      </jms-connection-factories>
      -->
  </hornetq-server>
</subsystem>


<pre>
...
<host ...>
 
  <system-properties>
<socket-binding-group name="standard-sockets"...>
      <property name="jboss.messaging.hornetq.backup" value="true"/>
        ...
  </system-properties>
        <outbound-socket-binding name="active-node-hornetq-binding">
  ...
            <remote-destination host="1.2.3.4" port="5445"/>
</host>
        </outbound-socket-binding>
</socket-binding-group>
</pre>
</pre>


===JMS Connection Factories===
===JMS Connection Factories and JMS Destinations on the Backup Server===


A backup HornetQ instance does not need the <tt><jms-connection-factories></tt> and <tt><jms-destinations></tt> sections as any JMS components are created from the shared journal when the backup server becomes live.
A backup HornetQ instance does not need the <tt><jms-connection-factories></tt> and <tt><jms-destinations></tt> sections as any JMS components are created from the shared journal when the backup server becomes live.
Line 120: Line 165:


<pre>
<pre>
13:14:00,312 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221000: live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/nfs/hornetq-shared-storage/journal,bindingsDirectory=/nfs/hornetq-shared-storage/bindings,largeMessagesDirectory=/nfs/hornetq-shared-storage/large-messages,pagingDirectory=/nfs/hornetq-shared-storage/paging)
00:54:10,313 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221000: live server is starting with configuration HornetQ Configuration (
13:14:00,313 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221006: Waiting to obtain live lock
clustered=true,
[...]
backup=false,
13:14:00,614 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221035: Live Server Obtained live lock
sharedStore=false,
[...]
journalDirectory=/opt/jboss/standalone/data/messagingjournal,
13:14:01,800 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221007: Server is now live
bindingsDirectory=/opt/jboss/standalone/data/messagingbindings,
13:14:01,801 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221001: HornetQ Server version 2.3.25.Final (2.3.x, 123) [db446058-de41-11e5-aea0-174ba3e38330]  
largeMessagesDirectory=/opt/jboss/standalone/data/messaginglargemessages,
pagingDirectory=/opt/jboss/standalone/data/messagingpaging)
00:54:17,461 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221013: Using NIO Journal
00:54:19,925 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221007: Server is now live
00:54:19,925 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221001: HornetQ Server version 2.3.25.Final (2.3.x, 123) [5802038e-e5bb-11e5-89d2-3d869d769af8]
...
</pre>
</pre>


Line 132: Line 182:


<pre>
<pre>
13:18:19,380 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221000: backup server is starting with configuration HornetQ Configuration (clustered=false,backup=true,sharedStore=true,journalDirectory=/nfs/hornetq-shared-storage/journal,bindingsDirectory=/nfs/hornetq-shared-storage/bindings,largeMessagesDirectory=/nfs/hornetq-shared-storage/large-messages,pagingDirectory=/nfs/hornetq-shared-storage/paging)
01:05:23,765 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221000: backup server is starting with configuration HornetQ Configuration (
13:18:19,402 INFO [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=db446058-de41-11e5-aea0-174ba3e38330) HQ221032: Waiting to become backup node
clustered=true,
13:18:19,449 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=db446058-de41-11e5-aea0-174ba3e38330) HQ221033: ** got backup lock
backup=true,
[...]
sharedStore=false,
13:18:19,680 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=db446058-de41-11e5-aea0-174ba3e38330) HQ221109: HornetQ Backup Server version 2.3.25.Final (2.3.x, 123) [db446058-de41-11e5-aea0-174ba3e38330] started, waiting live to fail before it gets active
journalDirectory=/opt/jboss/standalone/data/messagingjournal,
root@h2#
bindingsDirectory=/opt/jboss/standalone/data/messagingbindings,
largeMessagesDirectory=/opt/jboss/standalone/data/messaginglargemessages,
pagingDirectory=/opt/jboss/standalone/data/messagingpaging)
01:05:23,784 WARN [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messagingjournal to /opt/jboss/standalone/data/messagingjournal1
01:05:23,841 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221013: Using NIO Journal
01:05:24,068 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221109: HornetQ Backup Server version 2.3.25.Final (2.3.x, 123) [null] started, waiting live to fail before it gets active
01:05:28,335 INFO  [org.hornetq.core.server] (Old I/O client worker ([id: 0x241e2d9c, /172.31.22.24:43128 => /172.31.24.172:5445])) HQ221024: Backup server HornetQServerImpl::serverUUID=5802038e-e5bb-11e5-89d2-3d869d769af8 is synchronized with live-server.
01:05:28,339 INFO  [org.hornetq.core.server] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=null-213578370)) HQ221031: backup announced
</pre>
 
Upon backup server startup, the live server log shows this:
 
<pre>
01:05:25,144 INFO  [org.hornetq.core.server] (Thread-81) HQ221025: Replication: sending JournalFileImpl: (hornetq-data-2.hq id = 2, recordID = 2) (size=10,485,760) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingjournal/hornetq-data-2.hq
01:05:27,573 INFO  [org.hornetq.core.server] (Thread-81) HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-3.bindings id = 1, recordID = 1) (size=1,048,576) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingbindings/hornetq-bindings-3.bindings
01:05:27,604 INFO  [org.hornetq.core.server] (Thread-81) HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-2.bindings id = 2, recordID = 2) (size=1,048,576) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingbindings/hornetq-bindings-2.bindings
</pre>
</pre>


Line 143: Line 208:


<pre>
<pre>
[...]
01:08:16,869 WARN  [org.hornetq.core.client] (Thread-6 (HornetQ-client-global-threads-926703232)) HQ212004: Failed to connect to server.
13:20:21,911 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=db446058-de41-11e5-aea0-174ba3e38330) HQ221010: Backup Server is now live
01:08:16,884 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221037: HornetQServerImpl::serverUUID=5802038e-e5bb-11e5-89d2-3d869d769af8 to become live
01:08:17,334 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221020: Started Netty Acceptor version 3.6.10.Final-266dbdf 172.31.22.24:5445 for CORE protocol
01:08:17,346 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221020: Started Netty Acceptor version 3.6.10.Final-266dbdf 172.31.22.24:5455 for CORE protocol
</pre>
 
Failback on the active server:
 
<pre>
01:09:12,140 INFO  [org.hornetq.core.server] (Thread-78) HQ221025: Replication: sending JournalFileImpl: (hornetq-data-3.hq id = 9, recordID = 9) (size=10,485,760) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingjournal/hornetq-data-3.hq
01:09:14,168 INFO  [org.hornetq.core.server] (Thread-78) HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-3.bindings id = 1, recordID = 1) (size=1,048,576) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingbindings/hornetq-bindings-3.bindings
01:09:14,192 INFO  [org.hornetq.core.server] (Thread-78) HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-2.bindings id = 8, recordID = 8) (size=1,048,576) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingbindings/hornetq-bindings-2.bindings
01:09:19,221 WARN  [org.hornetq.core.server] (Thread-78) HQ222015: LIVE IS STOPPING?!? message=STOP_CALLED enabled=true
01:09:19,221 WARN  [org.hornetq.core.server] (Thread-78) HQ222015: LIVE IS STOPPING?!? message=STOP_CALLED true
01:09:19,249 WARN  [org.hornetq.core.server] (Thread-78) HQ222015: LIVE IS STOPPING?!? message=FAIL_OVER enabled=true
01:09:19,250 WARN  [org.hornetq.core.server] (Thread-78) HQ222015: LIVE IS STOPPING?!? message=FAIL_OVER true
01:09:19,276 INFO  [org.hornetq.core.server] (Thread-78) HQ221002: HornetQ Server version 2.3.25.Final (2.3.x, 123) [5802038e-e5bb-11e5-89d2-3d869d769af8] stopped
01:09:19,277 INFO  [org.hornetq.core.server] (Thread-78) HQ221039: Restarting as Replicating backup server after live restart
01:09:19,277 INFO  [org.hornetq.core.server] (Thread-78) HQ221000: backup server is starting with configuration HornetQ Configuration (
clustered=true,
backup=true,
sharedStore=false,
journalDirectory=/opt/jboss/standalone/data/messagingjournal,
bindingsDirectory=/opt/jboss/standalone/data/messagingbindings,
largeMessagesDirectory=/opt/jboss/standalone/data/messaginglargemessages,
pagingDirectory=/opt/jboss/standalone/data/messagingpaging)
01:09:19,278 WARN  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messagingbindings to /opt/jboss/standalone/data/messagingbindings2
01:09:19,279 WARN  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messagingjournal to /opt/jboss/standalone/data/messagingjournal2
01:09:19,279 WARN  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messagingpaging to /opt/jboss/standalone/data/messagingpaging2
01:09:19,279 WARN  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messaginglargemessages to /opt/jboss/standalone/data/messaginglargemessages2
01:09:19,279 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221013: Using NIO Journal
01:09:21,293 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221109: HornetQ Backup Server version 2.3.25.Final (2.3.x, 123) [null] started, waiting live to fail before it gets active
01:09:24,890 INFO  [org.hornetq.core.server] (Old I/O client worker ([id: 0x45afcbfa, /172.31.22.24:43263 => /172.31.24.172:5445])) HQ221024: Backup server HornetQServerImpl::serverUUID=5802038e-e5bb-11e5-89d2-3d869d769af8 is synchronized with live-server.
01:09:24,895 INFO  [org.hornetq.core.server] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=null-1956723314)) HQ221031: backup announced
</pre>
</pre>

Latest revision as of 19:31, 13 May 2016

External

Internal

Overview

For high availability purposes, the live server and the backup server must be installed on two separated physical (or virtual) hosts, provisioned in such a way to minimize the probability of both host failing at the same time.

Configuration

Procedure

Active Node Configuration

...
<subsystem xmlns="urn:jboss:domain:messaging:1.4"> 
   <hornetq-server> 

      <backup>false</backup>
      <shared-store>false</shared-store>
      <backup-group-name>pair-A</backup-group-name>
      <check-for-live-server>true</check-for-live-server>
      <failover-on-shutdown>true</failover-on-shutdown>

      <persistence-enabled>true</persistence-enabled>

      <cluster-user>hq</cluster-user>
      <cluster-password>hq123</cluster-password>
 
      ...
      
      <connectors>
            ...
            <netty-connector name="backup-node-connector" socket-binding="backup-node-hornetq-binding"/>
       </connectors>

      ...
      
      <cluster-connections>
          <cluster-connection name="pair-A-high-availability-cluster">
             <address>jms</address>
             <connector-ref>netty</connector-ref>
             <retry-interval>500</retry-interval>
             <use-duplicate-detection>true</use-duplicate-detection>
             <forward-when-no-consumers>true</forward-when-no-consumers>
             <max-hops>1</max-hops>
             <static-connectors>
                <connector-ref>backup-node-connector</connector-ref>
              </static-connectors>
          </cluster-connection>
      </cluster-connections>

      ...

      <jms-connection-factories>
         ...
         <connection-factory name="RemoteConnectionFactory">
            <ha>true</ha>
            <retry-interval>1000</retry-interval>
            <retry-interval-multiplier>1.0</retry-interval-multiplier>
            <reconnect-attempts>-1</reconnect-attempts> 
            <connectors> 
               <connector-ref connector-name="netty"/>
            </connectors> 
            <entries> 
               <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> 
            </entries> 
         </connection-factory>
         ...
      </jms-connection-factories>
   </hornetq-server>
</subsystem>

...

<socket-binding-group name="standard-sockets"...>
        ...
        <outbound-socket-binding name="backup-node-hornetq-binding">
            <remote-destination host="1.2.3.5" port="5445"/>
        </outbound-socket-binding>
</socket-binding-group>

Backup Node Configuration

...
<subsystem xmlns="urn:jboss:domain:messaging:1.4"> 
   <hornetq-server> 

      <backup>true</backup>
      <shared-store>false</shared-store>
      <backup-group-name>pair-A</backup-group-name>
      <check-for-live-server>true</check-for-live-server>
      <failover-on-shutdown>true</failover-on-shutdown>

      <persistence-enabled>true</persistence-enabled>

      <cluster-user>hq</cluster-user>
      <cluster-password>hq123</cluster-password>

      ...
      
      <connectors>
            ...
            <netty-connector name="active-node-connector" socket-binding="active-node-hornetq-binding"/>
       </connectors>

      ...
      
      <cluster-connections>
          <cluster-connection name="pair-A-high-availability-cluster">
             <address>jms</address>
             <connector-ref>netty</connector-ref>
             <retry-interval>500</retry-interval>
             <use-duplicate-detection>true</use-duplicate-detection>
             <forward-when-no-consumers>true</forward-when-no-consumers>
             <max-hops>1</max-hops>
             <static-connectors>
                <connector-ref>active-node-connector</connector-ref>
              </static-connectors>
          </cluster-connection>
      </cluster-connections>

      ...

      <!--
      <jms-connection-factories>
         ...
      </jms-connection-factories>
      -->
   </hornetq-server>
</subsystem>

...

<socket-binding-group name="standard-sockets"...>
        ...
        <outbound-socket-binding name="active-node-hornetq-binding">
            <remote-destination host="1.2.3.4" port="5445"/>
        </outbound-socket-binding>
</socket-binding-group>

JMS Connection Factories and JMS Destinations on the Backup Server

A backup HornetQ instance does not need the <jms-connection-factories> and <jms-destinations> sections as any JMS components are created from the shared journal when the backup server becomes live.

Log Output

Active server starting: 

00:54:10,313 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221000: live server is starting with configuration HornetQ Configuration (
clustered=true,
backup=false,
sharedStore=false,
journalDirectory=/opt/jboss/standalone/data/messagingjournal,
bindingsDirectory=/opt/jboss/standalone/data/messagingbindings,
largeMessagesDirectory=/opt/jboss/standalone/data/messaginglargemessages,
pagingDirectory=/opt/jboss/standalone/data/messagingpaging)
00:54:17,461 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221013: Using NIO Journal
00:54:19,925 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221007: Server is now live
00:54:19,925 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221001: HornetQ Server version 2.3.25.Final (2.3.x, 123) [5802038e-e5bb-11e5-89d2-3d869d769af8]
...

Stand-by server starting: 

01:05:23,765 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221000: backup server is starting with configuration HornetQ Configuration (
clustered=true,
backup=true,
sharedStore=false,
journalDirectory=/opt/jboss/standalone/data/messagingjournal,
bindingsDirectory=/opt/jboss/standalone/data/messagingbindings,
largeMessagesDirectory=/opt/jboss/standalone/data/messaginglargemessages,
pagingDirectory=/opt/jboss/standalone/data/messagingpaging)
01:05:23,784 WARN  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messagingjournal to /opt/jboss/standalone/data/messagingjournal1
01:05:23,841 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221013: Using NIO Journal
01:05:24,068 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221109: HornetQ Backup Server version 2.3.25.Final (2.3.x, 123) [null] started, waiting live to fail before it gets active
01:05:28,335 INFO  [org.hornetq.core.server] (Old I/O client worker ([id: 0x241e2d9c, /172.31.22.24:43128 => /172.31.24.172:5445])) HQ221024: Backup server HornetQServerImpl::serverUUID=5802038e-e5bb-11e5-89d2-3d869d769af8 is synchronized with live-server.
01:05:28,339 INFO  [org.hornetq.core.server] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=null-213578370)) HQ221031: backup announced

Upon backup server startup, the live server log shows this: 

01:05:25,144 INFO  [org.hornetq.core.server] (Thread-81) HQ221025: Replication: sending JournalFileImpl: (hornetq-data-2.hq id = 2, recordID = 2) (size=10,485,760) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingjournal/hornetq-data-2.hq
01:05:27,573 INFO  [org.hornetq.core.server] (Thread-81) HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-3.bindings id = 1, recordID = 1) (size=1,048,576) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingbindings/hornetq-bindings-3.bindings
01:05:27,604 INFO  [org.hornetq.core.server] (Thread-81) HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-2.bindings id = 2, recordID = 2) (size=1,048,576) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingbindings/hornetq-bindings-2.bindings

Failover to stand-by server: 

01:08:16,869 WARN  [org.hornetq.core.client] (Thread-6 (HornetQ-client-global-threads-926703232)) HQ212004: Failed to connect to server.
01:08:16,884 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221037: HornetQServerImpl::serverUUID=5802038e-e5bb-11e5-89d2-3d869d769af8 to become live
01:08:17,334 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221020: Started Netty Acceptor version 3.6.10.Final-266dbdf 172.31.22.24:5445 for CORE protocol
01:08:17,346 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221020: Started Netty Acceptor version 3.6.10.Final-266dbdf 172.31.22.24:5455 for CORE protocol

Failback on the active server: 

01:09:12,140 INFO  [org.hornetq.core.server] (Thread-78) HQ221025: Replication: sending JournalFileImpl: (hornetq-data-3.hq id = 9, recordID = 9) (size=10,485,760) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingjournal/hornetq-data-3.hq
01:09:14,168 INFO  [org.hornetq.core.server] (Thread-78) HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-3.bindings id = 1, recordID = 1) (size=1,048,576) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingbindings/hornetq-bindings-3.bindings
01:09:14,192 INFO  [org.hornetq.core.server] (Thread-78) HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-2.bindings id = 8, recordID = 8) (size=1,048,576) to backup. NIOSequentialFile /opt/jboss/standalone/data/messagingbindings/hornetq-bindings-2.bindings
01:09:19,221 WARN  [org.hornetq.core.server] (Thread-78) HQ222015: LIVE IS STOPPING?!? message=STOP_CALLED enabled=true
01:09:19,221 WARN  [org.hornetq.core.server] (Thread-78) HQ222015: LIVE IS STOPPING?!? message=STOP_CALLED true
01:09:19,249 WARN  [org.hornetq.core.server] (Thread-78) HQ222015: LIVE IS STOPPING?!? message=FAIL_OVER enabled=true
01:09:19,250 WARN  [org.hornetq.core.server] (Thread-78) HQ222015: LIVE IS STOPPING?!? message=FAIL_OVER true
01:09:19,276 INFO  [org.hornetq.core.server] (Thread-78) HQ221002: HornetQ Server version 2.3.25.Final (2.3.x, 123) [5802038e-e5bb-11e5-89d2-3d869d769af8] stopped
01:09:19,277 INFO  [org.hornetq.core.server] (Thread-78) HQ221039: Restarting as Replicating backup server after live restart
01:09:19,277 INFO  [org.hornetq.core.server] (Thread-78) HQ221000: backup server is starting with configuration HornetQ Configuration (
clustered=true,
backup=true,
sharedStore=false,
journalDirectory=/opt/jboss/standalone/data/messagingjournal,
bindingsDirectory=/opt/jboss/standalone/data/messagingbindings,
largeMessagesDirectory=/opt/jboss/standalone/data/messaginglargemessages,
pagingDirectory=/opt/jboss/standalone/data/messagingpaging)
01:09:19,278 WARN  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messagingbindings to /opt/jboss/standalone/data/messagingbindings2
01:09:19,279 WARN  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messagingjournal to /opt/jboss/standalone/data/messagingjournal2
01:09:19,279 WARN  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messagingpaging to /opt/jboss/standalone/data/messagingpaging2
01:09:19,279 WARN  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ222162: Moving data directory /opt/jboss/standalone/data/messaginglargemessages to /opt/jboss/standalone/data/messaginglargemessages2
01:09:19,279 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221013: Using NIO Journal
01:09:21,293 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=null) HQ221109: HornetQ Backup Server version 2.3.25.Final (2.3.x, 123) [null] started, waiting live to fail before it gets active
01:09:24,890 INFO  [org.hornetq.core.server] (Old I/O client worker ([id: 0x45afcbfa, /172.31.22.24:43263 => /172.31.24.172:5445])) HQ221024: Backup server HornetQServerImpl::serverUUID=5802038e-e5bb-11e5-89d2-3d869d769af8 is synchronized with live-server.
01:09:24,895 INFO  [org.hornetq.core.server] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=null-1956723314)) HQ221031: backup announced
Retrieved from "https://kb.novaordis.com/index.php?title=WildFly_HornetQ_Message_Replication-Based_HA_Configuration&oldid=9016"