WildFly HornetQ Shared Filesystem-Based Collocated HA Configuration

External

• Collocated Live-Backup HornetQ setup in EAP 6 https://access.redhat.com/solutions/400873

Internal

• HornetQ-Based Messaging Configuration
• HornetQ-Based Messaging Subsystem Concepts

Overview

This article describes the steps required to implement a highly available collocated HornetQ topology with EAP 6 and higher, using a shared filesystem. The concepts behind such a topology are presented here:

HornetQ Collocated HA Topology Concepts

For high availability purposes, the live server and the backup server instances must be installed on two separated physical (or virtual) hosts, provisioned in such a way to minimize the probability of both host failing at the same time. Highly available HornetQ requires access to reliable shared file system storage, so a file system such as GFS2 or a SAN must be made available to both hosts. HornetQ instances will store on the shared directory, among other things, their bindings and journal files. NFS v4 appropriately configured is also an option.

Collocation Considerations

In a collocated configuration, we might end in a situation where there are two active HornetQ nodes running within the same JVM. This happens if one of the hosts HornetQ is deployed on fails, and a stand-by HornetQ node becomes active. That is why we need to make sure that acceptor ID and ports do not overlap for instances running on the same host. The configuration examples below comply with this requirement.

In-VM Acceptor ID

Note that the collocated nodes must have different in-vm acceptor IDs, otherwise on activating the stand-by node we'll get:

15:42:46,651 ERROR [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=2c99027a-e569-11e5-987a-0981dfa56d61) HQ224000: Failure in initialisation: java.lang.IllegalArgumentException: HQ119062: Acceptor with id 0 already registered
	at org.hornetq.core.remoting.impl.invm.InVMRegistry.registerAcceptor(InVMRegistry.java:36) [hornetq-server-2.3.25.Final-redhat-1.jar:2.3.25.Final-redhat-1]
        ...

Netty Acceptor Port

Note that the collocated nodes must have different netty acceptor ports, otherwise on activating the stand-by node we'll get:

15:55:44,035 ERROR [org.hornetq.core.server] (HQ119000: Activation for server HornetQServerImpl::serverUUID=f50df054-e56f-11e5-b2e2-ab1fe58dc3e0) HQ224000: Failure in initialisation: org.jboss.netty.channel.ChannelException: Failed to bind to: /172.31.19.27:5445
	at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) [netty-3.6.10.Final-redhat-1.jar:3.6.10.Final-redhat-1]
        [...]
Caused by: java.net.BindException: Address already in use
	at java.net.PlainSocketImpl.socketBind(Native Method) [rt.jar:1.8.0_74]
        [...]

Procedure

Declare the Shared Filesystem Paths on Both Nodes

<paths>
   <path name="hornetq.shared.dir.pair.A" path="/nfs/hornetq-shared-storage-pair-A"/>       
   <path name="hornetq.shared.dir.pair.B" path="/nfs/hornetq-shared-storage-pair-B"/>        
</paths>

Node 1 Configuration

<subsystem xmlns="urn:jboss:domain:messaging:1.4"> 

   <hornetq-server name="active-node-pair-A"> 
      ...
      <backup>false</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.A"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.A"/> 
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.A"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.A"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-A"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="0"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-A"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="0"/>
      </acceptors>
      ...
      <jms-connection-factories>
         ...
         <connection-factory name="RemoteConnectionFactory">
            <ha>true</ha>
            <retry-interval>1000</retry-interval>
            <retry-interval-multiplier>1.0</retry-interval-multiplier>
            <reconnect-attempts>-1</reconnect-attempts> 
            <connectors> 
               <connector-ref connector-name="netty"/>
            </connectors> 
            <entries> 
               <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> 
            </entries> 
         </connection-factory>
         ...
      </jms-connection-factories>
   </hornetq-server>

   <hornetq-server name="standby-node-pair-B"> 
      ...
      <backup>true</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.B"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.B"/> 
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.B"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.B"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-B"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="50"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-B"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="50"/>
      </acceptors>      
      ...
      <!--
          The <jms-connection-factories> section must be completely commented out.
          <jms-connection-factories>
              ...
          </jms-connection-factories>
      -->
   </hornetq-server>
</subsystem>

...

<socket-binding-group name="standard-sockets" ...>
    ...
    <socket-binding name="messaging-pair-A" port="5445"/>
    <socket-binding name="messaging-throughput-pair-A" port="5455"/>
    <socket-binding name="messaging-pair-B" port="5465"/>
    <socket-binding name="messaging-throughput-pair-B" port="5475"/>
    ...
</socket-binding-group>

...

Node 2 Configuration

<subsystem xmlns="urn:jboss:domain:messaging:1.4"> 

   <hornetq-server name="active-node-pair-B"> 
      ...
      <backup>false</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.B"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.B"/> 
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.B"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.B"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-B"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="50"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-B"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-B">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="50"/>
      </acceptors>
      ...
      <jms-connection-factories>
         ...
         <connection-factory name="RemoteConnectionFactory">
            <ha>true</ha>
            <retry-interval>1000</retry-interval>
            <retry-interval-multiplier>1.0</retry-interval-multiplier>
            <reconnect-attempts>-1</reconnect-attempts> 
            <connectors> 
               <connector-ref connector-name="netty"/>
            </connectors> 
            <entries> 
               <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> 
            </entries> 
         </connection-factory>
         ...
      </jms-connection-factories>
   </hornetq-server>

   <hornetq-server name="standby-node-pair-A"> 
      ...
      <backup>true</backup>
      <persistence-enabled>true</persistence-enabled>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <paging-directory path="paging" relative-to="hornetq.shared.dir.pair.A"/>
      <bindings-directory path="bindings" relative-to="hornetq.shared.dir.pair.A"/> 
      <journal-directory path="journal" relative-to="hornetq.shared.dir.pair.A"/>
      <large-messages-directory path="large-messages" relative-to="hornetq.shared.dir.pair.A"/>
      ...
      <connectors>
          <netty-connector name="netty" socket-binding="messaging-pair-A"/>
          <netty-connector name="netty-throughput" socket-binding="messaging-throughput-pair-A">
               ...
          </netty-connector>
          <in-vm-connector name="in-vm" server-id="0"/>
      </connectors>
      <acceptors>
          <netty-acceptor name="netty" socket-binding="messaging-pair-A"/>
          <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput-pair-A">
              ...
          </netty-acceptor>
          <in-vm-acceptor name="in-vm" server-id="0"/>
      </acceptors>
      ...
      <!--
          The <jms-connection-factories> section must be completely commented out.
          <jms-connection-factories>
              ...
          </jms-connection-factories>
      -->
   </hornetq-server>
</subsystem>

...

<socket-binding-group name="standard-sockets" ...>
    ...
    <socket-binding name="messaging-pair-A" port="5445"/>
    <socket-binding name="messaging-throughput-pair-A" port="5455"/>
    <socket-binding name="messaging-pair-B" port="5465"/>
    <socket-binding name="messaging-throughput-pair-B" port="5475"/>
    ...
</socket-binding-group>

JMS Connection Factories

A backup HornetQ instance does not need the <jms-connection-factories> and <jms-destinations> sections as any JMS components are created from the shared journal when the backup server becomes live.

CLI Procedure

TODO: https://access.redhat.com/solutions/400873