External

Switch from UDP to TCP for HornetQ clustering in JBoss EAP 6 https://access.redhat.com/solutions/293823
Administration and Configuration Manual - Switch UDP to TCP for HornetQ Clustering https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.4/html/Administration_and_Configuration_Guide/sect-Clustering.html#Switch_UDP_to_TCP_for_HornetQ_Clustering

Internal

HornetQ-Based Messaging Subsystem Configuration

Relevance

EAP 6.4

Overview

This article shows how to configure a network of HornetQ brokers, assuming that the network has no multicast capabilities. For more details on HornetQ Clustering see WildFly HornetQ Clustering Concepts.

Procedure

Make Sure <clustering> is Enabled

The "messaging" subsystem must be configured with <clustered>true</clustered> on all nodes of the cluster:

<subsystem xmlns="urn:jboss:domain:messaging:1.4">
   <hornetq-server>
      ... 
      <clustered>true</clustered>
      ...
</subsystem>

<clustered> post EAP 6.1

Starting with EAP 6.1, the <clustered> configuration attribute was deprecated. If specified, the following message shows up in the logs:

11:52:05,484 WARN  [org.jboss.as.messaging] (Controller Boot Thread) JBAS011608: Element CLUSTERED is deprecated and will not be taken into account

Messaging clustering is enabled if one or more <cluster-connection> elements are defined. For more details about <cluster-connection> see Add a <cluster-connection> section.

On Each Cluster Node Configure Connectors To All Other Nodes

Assuming the cluster has three nodes ("node1", "node2" and "node3"), the configuration for node1 should declare <connector>s and <outbound-socket-binding>s for node2 and node3:

   ...
   <subsystem xmlns="urn:jboss:domain:messaging:1.4">
      <hornetq-server>
         ...
         <connectors>
            ...
            <netty-connector name="node2-connector" socket-binding="node2-hornetq-binding"/>
            <netty-connector name="node3-connector" socket-binding="node3-hornetq-binding"/>
         </connectors>
         ...
   </subsystem>
   ...
    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        ...
        <outbound-socket-binding name="node2-hornetq-binding">
            <remote-destination host="1.2.3.5" port="5445"/>
        </outbound-socket-binding>
        <outbound-socket-binding name="node3-hornetq-binding">
            <remote-destination host="1.2.3.6" port="5445"/>
        </outbound-socket-binding>
    </socket-binding-group>
   ...

Note: make sure the nodes at the other end of the connection actually listen on the specified interfaces/ports.

Comment out <broadcast-groups>, <discovery-groups>

They are useless without multicast.

Add a <cluster-connection>

Cluster connections are unidirectiona and point-to-point. Only one initiator node and one target node can participate to a cluster connection, so for this reason, only one connector-ref can be added as a static-connector for a cluster connection. If connections to more than one nodes are needed, add more <cluster-connection>s.

   ...
   </address-settings>

   <cluster-connections>

      <cluster-connection name="tcp-based-cluster-node1-to-node2">
         <address>jms</address>
         <connector-ref>netty</connector-ref>
         <retry-interval>500</retry-interval>
         <use-duplicate-detection>true</use-duplicate-detection>
         <forward-when-no-consumers>true</forward-when-no-consumers>
         <max-hops>1</max-hops>
         <static-connectors>
           <!-- just one connector-ref here -->
           <connector-ref>node2-connector</connector-ref>
          </static-connectors>
       </cluster-connection>

      <cluster-connection name="tcp-based-cluster-node1-to-node3">
         <address>jms</address>
         <connector-ref>netty</connector-ref>
         <retry-interval>500</retry-interval>
         <use-duplicate-detection>true</use-duplicate-detection>
         <forward-when-no-consumers>true</forward-when-no-consumers>
         <max-hops>1</max-hops>
         <static-connectors>
           <!-- just one connector-ref here -->
           <connector-ref>node3-connector</connector-ref>
          </static-connectors>
       </cluster-connection>

   </cluster-connections>
   ...

In EAP 6.1 or newer, adding a <cluster-connection> makes the <clustered> configuration element redundant.

Configure Security

If security is enabled (the <securit-enabled> configuration element is "true", and this is the default value even if it is not explicitly declared), then connections authenticate themselves as "HORNETQ.CLUSTER.ADMIN.USER" - the "cluster user". Unless a "cluster password" is specified in the configuration, a random password is used, and the connection establishing will fail. In order to provide known credentials for the cluster connections, configure the following (<cluster-user> is optional and if not specified will default to "HORNETQ.CLUSTER.ADMIN.USER"):

...
<subsystem xmlns="urn:jboss:domain:messaging:1.4">
   <hornetq-server>
      ...
      <cluster-user>hq</cluster-user>
      <cluster-password>hq123</cluster-password>
      ...
   </hornetq-server>
</subsystem>
...

The user declared as such does not need to be declared in any security repository, the configuration file declaration is sufficient.

Confirmation

You should see this in the logs:

EAP 6.4

20:37:47,719 INFO  [org.hornetq.core.server] (Thread-16 (HornetQ-server-HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16-46422202)) HQ221027: Bridge ClusterConnectionBridge@20600d51 [name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, queue=QueueImpl[name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16]]@298aa2ce targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@20600d51 [name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, queue=QueueImpl[name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16]]@298aa2ce targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-17-79], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1116810330[nodeUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-28-2, address=jms, server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-17-79], discoveryGroupConfiguration=null]] is connected

Other

15:11:19,234 INFO  [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-7 (HornetQ-server-HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c-544291442)) Bridge ClusterConnectionBridge@26250f55 [name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, queue=QueueImpl[name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c]]@4b11a903 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@26250f55 [name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, queue=QueueImpl[name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c]]@4b11a903 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-155], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1267429957[nodeUUID=388611e3-810b-11e2-85c9-694bd06e7c3c, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-154, address=jms, server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-155], discoveryGroupConfiguration=null]] is connected

Configure `ConnectionFactory` for Load Balancing

Add the extra connectors to each ConnectionFactories that are exposed and supposed to be used for load balancing/failover. The connectors are the same that have been added to the cluster connection <static-connectors> list:

...
<subsystem xmlns="urn:jboss:domain:messaging:1.4">
   <hornetq-server>
      ...
      <jms-connection-factories>
         ...
         <connection-factory name="RemoteConnectionFactory">
            <connectors>
               <connector-ref connector-name="netty"/>
               <connector-ref connector-name="node2-connector"/>
               <connector-ref connector-name="node3-connector"/>
            </connectors>
            <entries>
               <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
            </entries>

             <!--
                 Optionally:
             -->
             <connection-load-balancing-policy-class-name>
                 org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy
             </connection-load-balancing-policy-class-name>
         </connection-factory>
         ...
      </jms-connection-factories>
   </hornetq-server>
 </subsystem>
...

For more details on load balancing, see:

Client-Side Load Balancing

What Happens if a Cluster Node is Down when Other Starts Up

The node that is starting up attempts to connect ? times.

If it can't, it logs as DEBUG:

2013-02-20 11:39:45,131 DEBUG [org.hornetq.core.client.impl.ServerLocatorImpl] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=afe53124-6e53-11e2-bb90-005056ac34a6-1045936331)) Connector [initialConnector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-153-198-31]::Exception on establish connector initial connection
HornetQException[errorCode=2 message=Unable to connect to server using configuration org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-153-198-31]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:233)
        at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector$Connector.tryConnect(ServerLocatorImpl.java:1787)
        at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1624)
        at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:597)
        at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:564)
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Client-Side

Messaging Concepts - Client-Side Load Balancing

WildFly HornetQ-Based Messaging Subsystem - Clustering with TCP

Contents

External

Internal

Relevance

Overview

Procedure

Make Sure <clustering> is Enabled

<clustered> post EAP 6.1

On Each Cluster Node Configure Connectors To All Other Nodes

Comment out <broadcast-groups>, <discovery-groups>

Add a <cluster-connection>

Configure Security

Confirmation

EAP 6.4

Other

Configure `ConnectionFactory` for Load Balancing

What Happens if a Cluster Node is Down when Other Starts Up

Client-Side

Navigation menu

WildFly HornetQ-Based Messaging Subsystem - Clustering with TCP

External

Internal

Relevance

Overview

Procedure

Make Sure <clustering> is Enabled

<clustered> post EAP 6.1

On Each Cluster Node Configure Connectors To All Other Nodes

Comment out <broadcast-groups>, <discovery-groups>

Add a <cluster-connection>

Configure Security

Confirmation

EAP 6.4

Other

Configure ConnectionFactory for Load Balancing

What Happens if a Cluster Node is Down when Other Starts Up

Client-Side

Navigation menu

Search

Configure `ConnectionFactory` for Load Balancing