WildFly HornetQ-Based Messaging Subsystem - Clustering with TCP
External
- Switch from UDP to TCP for HornetQ clustering in JBoss EAP 6 https://access.redhat.com/solutions/293823
- Administration and Configuration Manual - Switch UDP to TCP for HornetQ Clustering https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.4/html/Administration_and_Configuration_Guide/sect-Clustering.html#Switch_UDP_to_TCP_for_HornetQ_Clustering
Internal
Overview
This article shows how to configure a network of HornetQ brokers, assuming that the network has no multicast capabilities. For more details on HornetQ Clustering see WildFly HornetQ Clustering Concepts.
Procedure
Make Sure <clustering> is Enabled
The "messaging" subsystem must be configured with <clustered>true</clustered> on all nodes of the cluster:
<subsystem xmlns="urn:jboss:domain:messaging:1.4"> <hornetq-server> ... <clustered>true</clustered> ... </subsystem>
On Each Cluster Node Configure Connectors To All Other Nodes
Assuming the cluster has three nodes ("node1", "node2" and "node3"), the configuration for node1 should declare <connector>s and <outbound-socket-binding>s for node2 and node3:
... <subsystem xmlns="urn:jboss:domain:messaging:1.4"> <hornetq-server> ... <connectors> ... <netty-connector name="node2-connector" socket-binding="node2-hornetq-binding"/> <netty-connector name="node3-connector" socket-binding="node3-hornetq-binding"/> </connectors> ... </subsystem> ... <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}"> ... <outbound-socket-binding name="node2-hornetq-binding"> <remote-destination host="1.2.3.5" port="5445"/> </outbound-socket-binding> <outbound-socket-binding name="node3-hornetq-binding"> <remote-destination host="1.2.3.6" port="5445"/> </outbound-socket-binding> </socket-binding-group> ...
Note: make sure the nodes at the other end of the connection actually listen on the specified interfaces/ports.
Comment out <broadcast-groups>, <discovery-groups>
They are useless without multicast.
Add a <cluster-connection>
... </address-settings> <cluster-connections> <cluster-connection name="tcp-based-cluster"> <address>jms</address> <connector-ref>netty</connector-ref> <retry-interval>500</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <forward-when-no-consumers>true</forward-when-no-consumers> <max-hops>1</max-hops> <static-connectors> <connector-ref>node2-connector</connector-ref> <connector-ref>node3-connector</connector-ref> </static-connectors> </cluster-connection> </cluster-connections> ...
Configure Security
If security is enabled (the <securit-enabled> configuration element is "true", and this is the default value even if it is not explicitly declared), then connections authenticate themselves as "HORNETQ.CLUSTER.ADMIN.USER" - the "cluster user". Unless a "cluster password" is specified in the configuration, a random password is used, and the connection establishing will fail. In order to provide known credentials for the cluster connections, configure the following (<cluster-user> is optional and if not specified will default to "HORNETQ.CLUSTER.ADMIN.USER"):
... <subsystem xmlns="urn:jboss:domain:messaging:1.4"> <hornetq-server> ... <cluster-user>hq</cluster-user> <cluster-password>hq123</cluster-password> ... </hornetq-server> </subsystem> ...
Verify this: Note that the user declared as such does not need to be declared in any security repository, the configuration file declaration is sufficient.
Confirmation
You should see this in the logs:
15:11:19,234 INFO [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-7 (HornetQ-server-HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c-544291442)) Bridge ClusterConnectionBridge@26250f55 [name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, queue=QueueImpl[name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c]]@4b11a903 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@26250f55 [name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, queue=QueueImpl[name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c]]@4b11a903 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-155], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1267429957[nodeUUID=388611e3-810b-11e2-85c9-694bd06e7c3c, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-154, address=jms, server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-155], discoveryGroupConfiguration=null]] is connected
Configure ConnectionFactory for Load Balancing/Failover
Add the extra connectors to each ConnectionFactories that are exposed and supposed to be used for load balancing/failover. The connectors are the same that have been added to the cluster connection <static-connectors> list:
... <subsystem xmlns="urn:jboss:domain:messaging:1.4"> <hornetq-server> ... <jms-connection-factories> ... <connection-factory name="RemoteConnectionFactory"> <connectors> <connector-ref connector-name="netty"/> <connector-ref connector-name="node2-connector"/> <connector-ref connector-name="node3-connector"/> </connectors> <entries> <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> </entries> </connection-factory> ... </jms-connection-factories> </hornetq-server> </subsystem> ...
What Happens if a Cluster Node is Down when Other Starts Up
The node that is starting up attempts to connect ? times.
If it can't, it logs as DEBUG:
2013-02-20 11:39:45,131 DEBUG [org.hornetq.core.client.impl.ServerLocatorImpl] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=afe53124-6e53-11e2-bb90-005056ac34a6-1045936331)) Connector [initialConnector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-153-198-31]::Exception on establish connector initial connection HornetQException[errorCode=2 message=Unable to connect to server using configuration org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-153-198-31] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:233) at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector$Connector.tryConnect(ServerLocatorImpl.java:1787) at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1624) at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:597) at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:564) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)
Client-Side Failover
final Properties env = new Properties(); env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory"); env.put("jboss.naming.client.connect.timeout", "10000"); env.put(Context.PROVIDER_URL, "remote://<host1>:4447,remote://<host2>:4447"); env.put(Context.SECURITY_PRINCIPAL, "username"); env.put(Context.SECURITY_CREDENTIALS, "password"); Context context = new InitialContext(env); ConnectionFactory cf = (ConnectionFactory) context.lookup("jms/RemoteConnectionFactory");
If <host1> (i.e. your "live" server) is down it will automatically try <host2> (i.e. your "backup" server).
If you want the JMS connections to move back to the live when it comes back then you should set <allow-failback> to "true" on both servers.