WildFly HornetQ-Based Messaging Subsystem - Clustering with TCP

External

• Switch from UDP to TCP for HornetQ clustering in JBoss EAP 6 https://access.redhat.com/solutions/293823
• Administration and Configuration Manual - Switch UDP to TCP for HornetQ Clustering https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.4/html/Administration_and_Configuration_Guide/sect-Clustering.html#Switch_UDP_to_TCP_for_HornetQ_Clustering

Internal

• HornetQ-Based Messaging Subsystem Configuration

Relevance

• EAP 6.4

Overview

This article shows how to configure a network of HornetQ brokers, assuming that the network has no multicast capabilities. For more details on HornetQ Clustering see WildFly HornetQ Clustering Concepts.

Procedure

Make Sure <clustering> is Enabled

The "messaging" subsystem must be configured with <clustered>true</clustered> on all nodes of the cluster:

<subsystem xmlns="urn:jboss:domain:messaging:1.4">
   <hornetq-server>
      ... 
      <clustered>true</clustered>
      ...
</subsystem>

On Each Cluster Node Configure Connectors To All Other Nodes

Assuming the cluster has three nodes ("node1", "node2" and "node3"), the configuration for node1 should declare <connector>s and <outbound-socket-binding>s for node2 and node3:

   ...
   <subsystem xmlns="urn:jboss:domain:messaging:1.4">
      <hornetq-server>
         ...
         <connectors>
            ...
            <netty-connector name="node2-connector" socket-binding="node2-hornetq-binding"/>
            <netty-connector name="node3-connector" socket-binding="node3-hornetq-binding"/>
         </connectors>
         ...
   </subsystem>
   ...
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        ...
        <outbound-socket-binding name="node2-hornetq-binding">
            <remote-destination host="1.2.3.5" port="5445"/>
        </outbound-socket-binding>
        <outbound-socket-binding name="node3-hornetq-binding">
            <remote-destination host="1.2.3.6" port="5445"/>
        </outbound-socket-binding>
    </socket-binding-group>
   ...

Note: make sure the nodes at the other end of the connection actually listen on the specified interfaces/ports.

Comment out <broadcast-groups>, <discovery-groups>

They are useless without multicast.

Add a <cluster-connection>

   ...
   </address-settings>
   <cluster-connections>
      <cluster-connection name="tcp-based-cluster">
         <address>jms</address>
         <connector-ref>netty</connector-ref>
         <retry-interval>500</retry-interval>
         <use-duplicate-detection>true</use-duplicate-detection>
         <forward-when-no-consumers>true</forward-when-no-consumers>
         <max-hops>1</max-hops>
         <static-connectors>
           <connector-ref>node2-connector</connector-ref>
           <connector-ref>node3-connector</connector-ref>
          </static-connectors>
       </cluster-connection>
   </cluster-connections>
   ...

Configure Security

If security is enabled (the <securit-enabled> configuration element is "true", and this is the default value even if it is not explicitly declared), then connections authenticate themselves as "HORNETQ.CLUSTER.ADMIN.USER" - the "cluster user". Unless a "cluster password" is specified in the configuration, a random password is used, and the connection establishing will fail. In order to provide known credentials for the cluster connections, configure the following (<cluster-user> is optional and if not specified will default to "HORNETQ.CLUSTER.ADMIN.USER"):

...
<subsystem xmlns="urn:jboss:domain:messaging:1.4">
   <hornetq-server>
      ...
      <cluster-user>hq</cluster-user>
      <cluster-password>hq123</cluster-password>
      ...
   </hornetq-server>
</subsystem>
...      

Verify this: Note that the user declared as such does not need to be declared in any security repository, the configuration file declaration is sufficient.

Confirmation

You should see this in the logs:

EAP 6.4

20:37:47,719 INFO  [org.hornetq.core.server] (Thread-16 (HornetQ-server-HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16-46422202)) HQ221027: Bridge ClusterConnectionBridge@20600d51 [name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, queue=QueueImpl[name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16]]@298aa2ce targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@20600d51 [name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, queue=QueueImpl[name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16]]@298aa2ce targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-17-79], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1116810330[nodeUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-28-2, address=jms, server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-17-79], discoveryGroupConfiguration=null]] is connected

Other

15:11:19,234 INFO  [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-7 (HornetQ-server-HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c-544291442)) Bridge ClusterConnectionBridge@26250f55 [name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, queue=QueueImpl[name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c]]@4b11a903 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@26250f55 [name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, queue=QueueImpl[name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c]]@4b11a903 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-155], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1267429957[nodeUUID=388611e3-810b-11e2-85c9-694bd06e7c3c, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-154, address=jms, server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-155], discoveryGroupConfiguration=null]] is connected

Configure ConnectionFactory for Load Balancing

Add the extra connectors to each ConnectionFactories that are exposed and supposed to be used for load balancing/failover. The connectors are the same that have been added to the cluster connection <static-connectors> list:

...
<subsystem xmlns="urn:jboss:domain:messaging:1.4">
   <hornetq-server>
      ...
      <jms-connection-factories>
         ...
         <connection-factory name="RemoteConnectionFactory">
            <connectors>
               <connector-ref connector-name="netty"/>
               <connector-ref connector-name="node2-connector"/>
               <connector-ref connector-name="node3-connector"/>
            </connectors>
            <entries>
               <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
            </entries>
         </connection-factory>
         ...
      </jms-connection-factories>
   </hornetq-server>
 </subsystem>
...

What Happens if a Cluster Node is Down when Other Starts Up

The node that is starting up attempts to connect ? times.

If it can't, it logs as DEBUG:

2013-02-20 11:39:45,131 DEBUG [org.hornetq.core.client.impl.ServerLocatorImpl] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=afe53124-6e53-11e2-bb90-005056ac34a6-1045936331)) Connector [initialConnector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-153-198-31]::Exception on establish connector initial connection
HornetQException[errorCode=2 message=Unable to connect to server using configuration org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-153-198-31]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:233)
        at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector$Connector.tryConnect(ServerLocatorImpl.java:1787)
        at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1624)
        at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:597)
        at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:564)
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Client-Side

Messaging Concepts - Client-Side Load Balancing