WildFly HornetQ Shared Filesystem-Based Collocated HA Configuration: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
No edit summary
 
(25 intermediate revisions by the same user not shown)
Line 5: Line 5:
=Internal=
=Internal=


* [[WildFly HornetQ-Based Messaging Subsystem - HA Configuration#Subjects|WildFly HornetQ HA Configuration]]
* [[WildFly HornetQ-Based Messaging Subsystem Configuration|HornetQ-Based Messaging Configuration]]
* [[WildFly_HornetQ-Based_Messaging_Subsystem_Concepts#Collocated_Topology_with_Shared_Filesystem.2C_no_Load_Balancing|HornetQ-Based Messaging Subsystem Concepts]]


=Overview=
=Overview=


For high availability purposes, the live server and the backup server must be installed on two separated physical (or virtual) hosts, provisioned in such a way to minimize the probability of both host failing at the same time. Highly available HornetQ requires access to reliable shared file system storage, so a file system such as GFS2 or a SAN must be made available to both hosts. HornetQ instances will store on the shared directory, among other things, their bindings and journal files. NFS v4 appropriately configured is also an option.
This article describes the steps required to implement a highly available collocated HornetQ topology with EAP 6 and higher, using a shared filesystem. The concepts behind such a topology are presented here:


=WildFly Clustering and HornetQ High Availability=
<blockquote style="background-color: #f9f9f9; border: solid thin lightgrey;">
:[[WildFly HornetQ-Based Messaging Subsystem Concepts#Collocated_Topology|HornetQ Collocated HA Topology Concepts]]
</blockquote>
 
For high availability purposes, the live server and the backup server instances must be installed on two separated physical (or virtual) hosts, provisioned in such a way to minimize the probability of both host failing at the same time. Highly available HornetQ requires access to reliable shared file system storage, so a file system such as GFS2 or a SAN must be made available to both hosts. HornetQ instances will store on the shared directory, among other things, their bindings and journal files. NFS v4 appropriately configured is also an option.


This document contains instructions for setting up a configuration where HornetQ HA is independently configured from WildFly clustering.
==Collocation Considerations==


For more details see:
In a collocated configuration, we might end in a situation where there are two active HornetQ nodes running within the same JVM. This happens if one of the hosts HornetQ is deployed on fails, and a stand-by HornetQ node becomes active. That is why we need to make sure that acceptor ID and ports do not overlap for instances running on the same host. The configuration examples below comply with this requirement.


<blockquote style="background-color: #f9f9f9; border: solid thin lightgrey;">
===In-VM Acceptor ID===
:[[WildFly HornetQ-Based Messaging Subsystem Concepts#WildFly_Clustering_and_HornetQ_High_Availability|HornetQ Concepts - WildFly Clustering and HornetQ High Availability]]
 
</blockquote>
Note that the collocated nodes must have different in-vm acceptor IDs, otherwise on activating the stand-by node we'll get:
 
<pre>
15:42:46,651 ERROR [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=2c99027a-e569-11e5-987a-0981dfa56d61) HQ224000: Failure in initialisation: java.lang.IllegalArgumentException: HQ119062: Acceptor with id 0 already registered
at org.hornetq.core.remoting.impl.invm.InVMRegistry.registerAcceptor(InVMRegistry.java:36) [hornetq-server-2.3.25.Final-redhat-1.jar:2.3.25.Final-redhat-1]
        ...
</pre>
 
===Netty Acceptor Port===
 
Note that the collocated nodes must have different netty acceptor ports, otherwise on activating the stand-by node we'll get:
 
<pre>
15:55:44,035 ERROR [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=f50df054-e56f-11e5-b2e2-ab1fe58dc3e0) HQ224000: Failure in initialisation: org.jboss.netty.channel.ChannelException: Failed to bind to: /172.31.19.27:5445
at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        [...]
Caused by: java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method) [rt.jar:1.8.0_74]
        [...]
</pre>


=Procedure=
=Procedure=


==Common Configuration==
==Declare the Shared Filesystem Paths on Both Nodes==


Common configuration specification using system properties for per-server instance externalization makes sense for WildFly instance running in domain mode, because it permits the use of the same configuration for both the live server and the backup server. The differences in behavior are specified via system properties. In standalone mode, the sequences below can be copied and pasted in their respective <tt>standalone*.xml</tt> files.
<pre>
 
<paths>
===Shared-Storage Based High Availability===
  <path name="hornetq.shared.dir.pair.A" path="/nfs/hornetq-shared-storage-pair-A"/>     
  <path name="hornetq.shared.dir.pair.B" path="/nfs/hornetq-shared-storage-pair-B"/>       
</paths>
</pre>


Use the following "messaging" subsystem configuration on both live and stand-by servers. This is convenient because the servers can be made part of the same server group.
==Node 1 Configuration==


<pre>
<pre>
...
<subsystem xmlns="urn:jboss:domain:messaging:1.4">  
<subsystem xmlns="urn:jboss:domain:messaging:1.4">  
  <hornetq-server>


  <hornetq-server name="active-node-pair-A">
      ...
      <backup>false</backup>
       <persistence-enabled>true</persistence-enabled>
       <persistence-enabled>true</persistence-enabled>
      ...
       <shared-store>true</shared-store>
       <shared-store>true</shared-store>
      <backup>${jboss.messaging.hornetq.backup:false}</backup>
       <create-bindings-dir>true</create-bindings-dir>
       <create-bindings-dir>true</create-bindings-dir>
       <create-journal-dir>true</create-journal-dir>
       <create-journal-dir>true</create-journal-dir>
       <failover-on-shutdown>true</failover-on-shutdown>
       <failover-on-shutdown>true</failover-on-shutdown>
 
       <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.A"/>
       <paging-directory path="paging" relative-to="hornetq.shared.dir"/>
       <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.A"/>  
       <bindings-directory path="bindings" relative-to="hornetq.shared.dir"/>  
       <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.A"/>
       <journal-directory path="journal" relative-to="hornetq.shared.dir"/>
       <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.A"/>
       <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir"/>
      ...
        
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-A"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="0"/>
       </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-A"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="0"/>
      </acceptors>
       ...
       ...
       <jms-connection-factories>
       <jms-connection-factories>
         ...
         ...
Line 67: Line 105:
         ...
         ...
       </jms-connection-factories>
       </jms-connection-factories>
  </hornetq-server>
  <hornetq-server name="standby-node-pair-B">
      ...
      <backup>true</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.B"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.B"/>
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.B"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.B"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-B"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="50"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-B"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="50"/>
      </acceptors>     
      ...
      <!--
          The <jms-connection-factories> section must be completely commented out.
          <jms-connection-factories>
              ...
          </jms-connection-factories>
      -->
   </hornetq-server>
   </hornetq-server>
</subsystem>
</subsystem>
...
...
</pre>


===Shared Path Declaration===
<socket-binding-group name="standard-sockets" ...>
    ...
    <socket-binding name="messaging-pair-A" port="5445"/>
    <socket-binding name="messaging-throughput-pair-A" port="5455"/>
    <socket-binding name="messaging-pair-B" port="5465"/>
    <socket-binding name="messaging-throughput-pair-B" port="5475"/>
    ...
</socket-binding-group>


This is usually common for the entire domain, so it can be specified in the domain top level section.
...
 
<pre>
  ...
  <paths>
      <path name="hornetq.shared.dir" path="/nfs/hornetq-shared-storage"/>
  </paths>
  ...
</pre>
</pre>


==Live Server Configuration==
==Node 2 Configuration==


<tt>jboss.messaging.hornetq.backup</tt> is by default false, but it's actually a good idea to make the configuration obvious. Add the following in the active node's <tt>host.xml</tt>:
<pre>
<subsystem xmlns="urn:jboss:domain:messaging:1.4">  


<pre>
  <hornetq-server name="active-node-pair-B">
<host ...>
      ...
  <system-properties>
      <backup>false</backup>
       <property name="jboss.messaging.hornetq.backup" value="false"/>
      <persistence-enabled>true</persistence-enabled>
  </system-properties>
      <shared-store>true</shared-store>
  ...
      <create-bindings-dir>true</create-bindings-dir>
</host>
      <create-journal-dir>true</create-journal-dir>
</pre>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.B"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.B"/>  
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.B"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.B"/>
       ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-B"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="50"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-B"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="50"/>
      </acceptors>
      ...
      <jms-connection-factories>
        ...
        <connection-factory name="RemoteConnectionFactory">
            <ha>true</ha>
            <retry-interval>1000</retry-interval>
            <retry-interval-multiplier>1.0</retry-interval-multiplier>
            <reconnect-attempts>-1</reconnect-attempts>
            <connectors>
              <connector-ref connector-name="netty"/>
            </connectors>
            <entries>
              <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>  
            </entries>
        </connection-factory>
        ...
      </jms-connection-factories>
  </hornetq-server>


==Stand-By Server Configuration==
  <hornetq-server name="standby-node-pair-A">
      ...
      <backup>true</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.A"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.A"/>
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.A"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.A"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-A"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="0"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-A"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="0"/>
      </acceptors>
      ...
      <!--
          The <jms-connection-factories> section must be completely commented out.
          <jms-connection-factories>
              ...
          </jms-connection-factories>
      -->
  </hornetq-server>
</subsystem>


<tt>jboss.messaging.hornetq.backup</tt> should be set to "<tt>true</tt>" in the stand-by node's <tt>host.xml</tt>:
...


<pre>
<socket-binding-group name="standard-sockets" ...>
<host ...>
    ...
  <system-properties>
    <socket-binding name="messaging-pair-A" port="5445"/>
      <property name="jboss.messaging.hornetq.backup" value="true"/>
    <socket-binding name="messaging-throughput-pair-A" port="5455"/>
  </system-properties>
    <socket-binding name="messaging-pair-B" port="5465"/>
  ...
    <socket-binding name="messaging-throughput-pair-B" port="5475"/>
</host>
    ...
</socket-binding-group>
</pre>
</pre>


Line 114: Line 262:
A backup HornetQ instance does not need the <tt><jms-connection-factories></tt> and <tt><jms-destinations></tt> sections as any JMS components are created from the shared journal when the backup server becomes live.
A backup HornetQ instance does not need the <tt><jms-connection-factories></tt> and <tt><jms-destinations></tt> sections as any JMS components are created from the shared journal when the backup server becomes live.


=Log Output=
=CLI Procedure=
 
Active server starting:
 
<pre>
13:14:00,312 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221000: live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/nfs/hornetq-shared-storage/journal,bindingsDirectory=/nfs/hornetq-shared-storage/bindings,largeMessagesDirectory=/nfs/hornetq-shared-storage/large-messages,pagingDirectory=/nfs/hornetq-shared-storage/paging)
13:14:00,313 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221006: Waiting to obtain live lock
[...]
13:14:00,614 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221035: Live Server Obtained live lock
[...]
13:14:01,800 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221007: Server is now live
13:14:01,801 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221001: HornetQ Server version 2.3.25.Final (2.3.x, 123) [db446058-de41-11e5-aea0-174ba3e38330]
</pre>
 
Stand-by server starting:
 
<pre>
13:18:19,380 INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 60) HQ221000: backup server is starting with configuration HornetQ Configuration (clustered=false,backup=true,sharedStore=true,journalDirectory=/nfs/hornetq-shared-storage/journal,bindingsDirectory=/nfs/hornetq-shared-storage/bindings,largeMessagesDirectory=/nfs/hornetq-shared-storage/large-messages,pagingDirectory=/nfs/hornetq-shared-storage/paging)
13:18:19,402 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=db446058-de41-11e5-aea0-174ba3e38330) HQ221032: Waiting to become backup node
13:18:19,449 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=db446058-de41-11e5-aea0-174ba3e38330) HQ221033: ** got backup lock
[...]
13:18:19,680 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=db446058-de41-11e5-aea0-174ba3e38330) HQ221109: HornetQ Backup Server version 2.3.25.Final (2.3.x, 123) [db446058-de41-11e5-aea0-174ba3e38330] started, waiting live to fail before it gets active
root@h2#
</pre>
 
Failover to stand-by server:
 
<pre>
[...]
13:20:21,911 INFO  [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=db446058-de41-11e5-aea0-174ba3e38330) HQ221010: Backup Server is now live
</pre>
 
=Failover Limitations=
 
Due to the way HornetQ was designed, the failover is not fully transparent and it requires application’s cooperation.
 
There are two notable situations when the application will be notified of live server failure:
 
# The application performs a blocking operations (for example a message <tt>send()</tt>). In this situation, if a live server failure occurs, the client side messaging runtime will interrupt the send operation and make it throw a <tt>JMSExcepiton</tt>.
# The live server failure occurs during a transaction. In this case, the client-side messaging runtime rolls back the transaction.
 
=Failover in Case of Administrative Shutdown of the Live Server=
 
HornetQ allows the possibility to specify the client-side failover behavior in case of administrative shutdown of the live server. There are two options:
# Client does not fail over to the backup server on administrative shutdown of the live server. If the connection factory is configured to contain other live server connectors, the client will reconnect to those; if not, it will issue a warning log entry and close the connection.
# Client does fail over to the backup server on administrative shutdown of the live server. If there are no other live servers available, this is probably a sensible option.
 
=HornetQ Data Directories=


The configuration allows the possibility of creating the HornetQ bindings and journal data directories at startup, if they do not already exist. This configuration could be useful in "experimental" mode when one deletes and recreates HornetQ data files for whatever reason, and probably not that useful in production. If the directories exist, they are not re-created, so the "create" options can be left in place, even in a production configuration. However, there is another set of directories (large messages and paging) that will be created automatically if they don’t exist, in absence of any explicit configuration option. For production, it’s probably best if the directories are created manually as part of the installation procedure, and "create-*" options are removed from configuration.
<font color=red>'''TODO''': https://access.redhat.com/solutions/400873</font>

Latest revision as of 01:24, 7 April 2016

External

Internal

Overview

This article describes the steps required to implement a highly available collocated HornetQ topology with EAP 6 and higher, using a shared filesystem. The concepts behind such a topology are presented here:

HornetQ Collocated HA Topology Concepts

For high availability purposes, the live server and the backup server instances must be installed on two separated physical (or virtual) hosts, provisioned in such a way to minimize the probability of both host failing at the same time. Highly available HornetQ requires access to reliable shared file system storage, so a file system such as GFS2 or a SAN must be made available to both hosts. HornetQ instances will store on the shared directory, among other things, their bindings and journal files. NFS v4 appropriately configured is also an option.

Collocation Considerations

In a collocated configuration, we might end in a situation where there are two active HornetQ nodes running within the same JVM. This happens if one of the hosts HornetQ is deployed on fails, and a stand-by HornetQ node becomes active. That is why we need to make sure that acceptor ID and ports do not overlap for instances running on the same host. The configuration examples below comply with this requirement.

In-VM Acceptor ID

Note that the collocated nodes must have different in-vm acceptor IDs, otherwise on activating the stand-by node we'll get:

15:42:46,651 ERROR [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=2c99027a-e569-11e5-987a-0981dfa56d61) HQ224000: Failure in initialisation: java.lang.IllegalArgumentException: HQ119062: Acceptor with id 0 already registered
	at org.hornetq.core.remoting.impl.invm.InVMRegistry.registerAcceptor(InVMRegistry.java:36) [hornetq-server-2.3.25.Final-redhat-1.jar:2.3.25.Final-redhat-1]
        ...

Netty Acceptor Port

Note that the collocated nodes must have different netty acceptor ports, otherwise on activating the stand-by node we'll get:

15:55:44,035 ERROR [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=f50df054-e56f-11e5-b2e2-ab1fe58dc3e0) HQ224000: Failure in initialisation: org.jboss.netty.channel.ChannelException: Failed to bind to: /172.31.19.27:5445
	at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        [...]
Caused by: java.net.BindException: Address already in use
	at java.net.PlainSocketImpl.socketBind(Native Method) [rt.jar:1.8.0_74]
        [...]

Procedure

Declare the Shared Filesystem Paths on Both Nodes

<paths>
   <path name="hornetq.shared.dir.pair.A" path="/nfs/hornetq-shared-storage-pair-A"/>       
   <path name="hornetq.shared.dir.pair.B" path="/nfs/hornetq-shared-storage-pair-B"/>        
</paths>

Node 1 Configuration

<subsystem xmlns="urn:jboss:domain:messaging:1.4"> 

   <hornetq-server name="active-node-pair-A"> 
      ...
      <backup>false</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.A"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.A"/> 
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.A"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.A"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-A"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="0"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-A"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="0"/>
      </acceptors>
      ...
      <jms-connection-factories>
         ...
         <connection-factory name="RemoteConnectionFactory">
            <ha>true</ha>
            <retry-interval>1000</retry-interval>
            <retry-interval-multiplier>1.0</retry-interval-multiplier>
            <reconnect-attempts>-1</reconnect-attempts> 
            <connectors> 
               <connector-ref connector-name="netty"/>
            </connectors> 
            <entries> 
               <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> 
            </entries> 
         </connection-factory>
         ...
      </jms-connection-factories>
   </hornetq-server>

   <hornetq-server name="standby-node-pair-B"> 
      ...
      <backup>true</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.B"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.B"/> 
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.B"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.B"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-B"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="50"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-B"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="50"/>
      </acceptors>      
      ...
      <!--
          The <jms-connection-factories> section must be completely commented out.
          <jms-connection-factories>
              ...
          </jms-connection-factories>
      -->
   </hornetq-server>
</subsystem>

...

<socket-binding-group name="standard-sockets" ...>
    ...
    <socket-binding name="messaging-pair-A" port="5445"/>
    <socket-binding name="messaging-throughput-pair-A" port="5455"/>
    <socket-binding name="messaging-pair-B" port="5465"/>
    <socket-binding name="messaging-throughput-pair-B" port="5475"/>
    ...
</socket-binding-group>

...

Node 2 Configuration

<subsystem xmlns="urn:jboss:domain:messaging:1.4"> 

   <hornetq-server name="active-node-pair-B"> 
      ...
      <backup>false</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.B"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.B"/> 
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.B"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.B"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-B"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="50"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-B"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="50"/>
      </acceptors>
      ...
      <jms-connection-factories>
         ...
         <connection-factory name="RemoteConnectionFactory">
            <ha>true</ha>
            <retry-interval>1000</retry-interval>
            <retry-interval-multiplier>1.0</retry-interval-multiplier>
            <reconnect-attempts>-1</reconnect-attempts> 
            <connectors> 
               <connector-ref connector-name="netty"/>
            </connectors> 
            <entries> 
               <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> 
            </entries> 
         </connection-factory>
         ...
      </jms-connection-factories>
   </hornetq-server>

   <hornetq-server name="standby-node-pair-A"> 
      ...
      <backup>true</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.A"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.A"/> 
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.A"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.A"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-A"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-A">
               ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="0"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-A"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="0"/>
      </acceptors>
      ...
      <!--
          The <jms-connection-factories> section must be completely commented out.
          <jms-connection-factories>
              ...
          </jms-connection-factories>
      -->
   </hornetq-server>
</subsystem>

...

<socket-binding-group name="standard-sockets" ...>
    ...
    <socket-binding name="messaging-pair-A" port="5445"/>
    <socket-binding name="messaging-throughput-pair-A" port="5455"/>
    <socket-binding name="messaging-pair-B" port="5465"/>
    <socket-binding name="messaging-throughput-pair-B" port="5475"/>
    ...
</socket-binding-group>

JMS Connection Factories

A backup HornetQ instance does not need the <jms-connection-factories> and <jms-destinations> sections as any JMS components are created from the shared journal when the backup server becomes live.

CLI Procedure

TODO: https://access.redhat.com/solutions/400873