WildFly HornetQ Shared Filesystem-Based Dedicated HA Configuration

External

• EAP 6 Administration and Configuration Manual - Configuring HornetQ for Dedicated Topology with Shared Store https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.4/html/Administration_and_Configuration_Guide/sect-High_Availability.html#Configuring_HornetQ_Shared_Stores
• HornetQ live-backup configuration in domain mode in EAP 6 https://access.redhat.com/solutions/239013
• EAP 5 HornetQ User Guide https://access.redhat.com/knowledge/docs/en-US/JBoss_Enterprise_Application_Platform/5/html/HornetQ_User_Guide/High_Availability_and_Fail_Over.html
• Indicated by Red Hat Support as "official" documentation http://docs.jboss.org/hornetq/2.2.14.Final/user-manual/en/html/ha.html

Internal

• WildFly HornetQ HA Configuration

Overview

For high availability purposes, the live server and the backup server must be installed on two separated physical (or virtual) hosts, provisioned in such a way to minimize the probability of both host failing at the same time. Highly available HornetQ requires access to reliable shared file system storage, so a file system such as GFS2 or a SAN must be made available to both hosts. HornetQ instances will store on the shared directory, among other things, their bindings and journal files. NFS v4 appropriately configured is also an option.

WildFly Clustering and HornetQ High Availability

This document contains instructions for setting up a configuration where HornetQ HA is independently configured from WildFly clustering.

For more details see:

HornetQ Concepts - WildFly Clustering and HornetQ High Availability

Procedure

Common Configuration

Common configuration specification and system properties makes sense for WildFly instance running in domain mode, because it permits the use of the same configuration for both the live server and the backup server and specify the differences via system properties. In standalone mode, the sequences below can be copied and pasted in the respective standalone*.xml files.

Both the live and the standup server

Live Server Configuration

...
<subsystem xmlns="urn:jboss:domain:messaging:1.4"> 
   <hornetq-server> 
      <persistence-enabled>true</persistence-enabled>
      ...
      <backup>${jboss.messaging.hornetq.backup:false}</backup>
      <shared-store>true</shared-store>
      <create-bindings-dir>true</create-bindings-dir>
      <create-journal-dir>true</create-journal-dir>
      <failover-on-shutdown>true</failover-on-shutdown>
      <bindings-directory path="../shared-hornetq-storage/bindings" relative-to="jboss.server.base.dir"/> 
      <journal-directory path="../shared-hornetq-storage/journal" relative-to="jboss.server.base.dir"/>
      <large-messages-directory path="../shared-hornetq-storage/large-messages" relative-to="jboss.server.base.dir"/>
      <paging-directory path="../shared-hornetq-storage/paging" relative-to="jboss.server.base.dir"/>
      ...
      <jms-connection-factories>
         ...
         <connection-factory name="RemoteConnectionFactory">
            <ha>true</ha>
            <retry-interval>1000</retry-interval>
            <retry-interval-multiplier>1.0</retry-interval-multiplier>
            <reconnect-attempts>-1</reconnect-attempts> 
            <connectors> 
               <connector-ref connector-name="netty"/>
            </connectors> 
            <entries> 
               <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> 
            </entries> 
         </connection-factory>
         ...
      </jms-connection-factories>
   </hornetq-server>
</subsystem>
...

Stand-By Server Configuration

...
<subsystem xmlns="urn:jboss:domain:messaging:1.4"> 
   <hornetq-server> 
      <persistence-enabled>true</persistence-enabled>
      ...
      <backup>true</backup>
      <shared-store>true</shared-store>
      <allow-failback>true</allow-failback>
      <bindings-directory path="../shared-hornetq-storage/bindings" relative-to="jboss.server.base.dir"/>
      <journal-directory path="../shared-hornetq-storage/journal" relative-to="jboss.server.base.dir"/>
      <large-messages-directory path="../shared-hornetq-storage/large-messages" relative-to="jboss.server.base.dir"/>
      <paging-directory path="../shared-hornetq-storage/paging" relative-to="jboss.server.base.dir"/>
      ...
      <!-- Remove all <jms-connection-factories> declarations -->
   </hornetq-server>
</subsystem>
...

A backup HornetQ instance does not need the <jms-connection-factories> and <jms-destinations> sections as any JMS components are created from the shared journal when the backup server becomes live.

Failover Limitations

Due to the way HornetQ was designed, the failover is not fully transparent and it requires application’s cooperation.

There are two notable situations when the application will be notified of live server failure:

1. The application performs a blocking operations (for example a message send()). In this situation, if a live server failure occurs, the client side messaging runtime will interrupt the send operation and make it throw a JMSExcepiton.
2. The live server failure occurs during a transaction. In this case, the client-side messaging runtime rolls back the transaction.

Failover in Case of Administrative Shutdown of the Live Server

HornetQ allows the possibility to specify the client-side failover behavior in case of administrative shutdown of the live server. There are two options:

1. Client does not fail over to the backup server on administrative shutdown of the live server. If the connection factory is configured to contain other live server connectors, the client will reconnect to those; if not, it will issue a warning log entry and close the connection.
2. Client does fail over to the backup server on administrative shutdown of the live server. If there are no other live servers available, this is probably a sensible option.

HornetQ Data Directories

The configuration allows the possibility of creating the HornetQ bindings and journal data directories at startup, if they do not already exist. This configuration could be useful in "experimental" mode when one deletes and recreates HornetQ data files for whatever reason, and probably not that useful in production. If the directories exist, they are not re-created, so the "create" options can be left in place, even in a production configuration. However, there is another set of directories (large messages and paging) that will be created automatically if they don’t exist, in absence of any explicit configuration option. For production, it’s probably best if the directories are created manually as part of the installation procedure, and "create-*" options are removed from configuration.