WildFly HornetQ-Based Messaging Subsystem - Clustering with TCP: Difference between revisions
(→Add a) |
|||
(21 intermediate revisions by the same user not shown) | |||
Line 7: | Line 7: | ||
* [[WildFly HornetQ-Based Messaging Subsystem Configuration|HornetQ-Based Messaging Subsystem Configuration]] | * [[WildFly HornetQ-Based Messaging Subsystem Configuration|HornetQ-Based Messaging Subsystem Configuration]] | ||
=Relevance= | |||
* EAP 6.4 | |||
=Overview= | =Overview= | ||
Line 26: | Line 30: | ||
</subsystem> | </subsystem> | ||
</pre> | </pre> | ||
===<clustered> post EAP 6.1=== | |||
Starting with EAP 6.1, the <tt><clustered></tt> configuration attribute was deprecated. If specified, the following message shows up in the logs: | |||
<pre> | |||
11:52:05,484 WARN [org.jboss.as.messaging] (Controller Boot Thread) JBAS011608: Element CLUSTERED is deprecated and will not be taken into account | |||
</pre> | |||
Messaging clustering is enabled if one or more <tt><cluster-connection></tt> elements are defined. For more details about <tt><cluster-connection></tt> see [[WildFly_HornetQ-Based_Messaging_Subsystem_-_Clustering_with_TCP#Add_a_.3Ccluster-connection.3E|Add a <tt><cluster-connection></tt>]] section. | |||
==On Each Cluster Node Configure Connectors To All Other Nodes== | ==On Each Cluster Node Configure Connectors To All Other Nodes== | ||
Line 63: | Line 77: | ||
==Add a <cluster-connection>== | ==Add a <cluster-connection>== | ||
<blockquote style="background-color: Gold; border: solid thin Goldenrod;"> | |||
:Cluster connections are unidirectiona and point-to-point. Only one initiator node and one target node can participate to a cluster connection, so for this reason, only one <tt>connector-ref</tt> can be added as a <tt>static-connector</tt> for a cluster connection. If connections to more than one nodes are needed, add more <tt><cluster-connection></tt>s.<br> | |||
</blockquote> | |||
<pre> | <pre> | ||
... | ... | ||
</address-settings> | </address-settings> | ||
<cluster-connections> | <cluster-connections> | ||
<cluster-connection name="tcp-based-cluster"> | |||
<cluster-connection name="tcp-based-cluster-node1-to-node2"> | |||
<address>jms</address> | <address>jms</address> | ||
<connector-ref>netty</connector-ref> | <connector-ref>netty</connector-ref> | ||
Line 76: | Line 96: | ||
<max-hops>1</max-hops> | <max-hops>1</max-hops> | ||
<static-connectors> | <static-connectors> | ||
<!-- just one connector-ref here --> | |||
<connector-ref>node2-connector</connector-ref> | <connector-ref>node2-connector</connector-ref> | ||
</static-connectors> | |||
</cluster-connection> | |||
<cluster-connection name="tcp-based-cluster-node1-to-node3"> | |||
<address>jms</address> | |||
<connector-ref>netty</connector-ref> | |||
<retry-interval>500</retry-interval> | |||
<use-duplicate-detection>true</use-duplicate-detection> | |||
<forward-when-no-consumers>true</forward-when-no-consumers> | |||
<max-hops>1</max-hops> | |||
<static-connectors> | |||
<!-- just one connector-ref here --> | |||
<connector-ref>node3-connector</connector-ref> | <connector-ref>node3-connector</connector-ref> | ||
</static-connectors> | </static-connectors> | ||
</cluster-connection> | </cluster-connection> | ||
</cluster-connections> | </cluster-connections> | ||
... | ... | ||
</pre> | </pre> | ||
In EAP 6.1 or newer, adding a <tt><cluster-connection></tt> makes the [[WildFly_HornetQ-Based_Messaging_Subsystem_-_Clustering_with_TCP#.3Cclustered.3E_post_EAP_6.1|<tt><clustered></tt> configuration element redundant]]. | |||
==Configure Security== | ==Configure Security== | ||
If security is enabled (the <securit-enabled> configuration element is "<tt>true</tt>"), then connections authenticate themselves as "HORNETQ.CLUSTER.ADMIN.USER" - the "cluster user". Unless a "cluster password" is specified in the configuration, a random password is used, and the connection establishing will fail. In order to provide known credentials for the cluster connections, configure the following (<cluster-user> is optional and if not specified will default to "HORNETQ.CLUSTER.ADMIN.USER"): | If security is enabled (the <securit-enabled> configuration element is "<tt>true</tt>", and this is the default value even if it is not explicitly declared), then connections authenticate themselves as "HORNETQ.CLUSTER.ADMIN.USER" - the "cluster user". Unless a "cluster password" is specified in the configuration, a random password is used, and the connection establishing will fail. In order to provide known credentials for the cluster connections, configure the following (<cluster-user> is optional and if not specified will default to "HORNETQ.CLUSTER.ADMIN.USER"): | ||
<pre> | <pre> | ||
Line 101: | Line 137: | ||
</pre> | </pre> | ||
The user declared as such does not need to be declared in any security repository, the configuration file declaration is sufficient. | |||
==Confirmation== | ==Confirmation== | ||
You should see this in the logs: | You should see this in the logs: | ||
===EAP 6.4=== | |||
<pre> | |||
20:37:47,719 INFO [org.hornetq.core.server] (Thread-16 (HornetQ-server-HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16-46422202)) HQ221027: Bridge ClusterConnectionBridge@20600d51 [name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, queue=QueueImpl[name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16]]@298aa2ce targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@20600d51 [name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, queue=QueueImpl[name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16]]@298aa2ce targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-17-79], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1116810330[nodeUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-28-2, address=jms, server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-17-79], discoveryGroupConfiguration=null]] is connected | |||
</pre> | |||
===Other=== | |||
<pre> | <pre> | ||
Line 111: | Line 155: | ||
</pre> | </pre> | ||
==Configure <tt>ConnectionFactory</tt> for Load Balancing | ==Configure <tt>ConnectionFactory</tt> for Load Balancing== | ||
Add the extra connectors to each <tt>ConnectionFactories</tt> that are exposed and supposed to be used for load balancing/failover. The connectors are the same that have been added to the cluster connection <tt><static-connectors></tt> list: | Add the extra connectors to each <tt>ConnectionFactories</tt> that are exposed and supposed to be used for load balancing/failover. The connectors are the same that have been added to [[#Add_a_.3Ccluster-connection.3E|the cluster connection <tt><static-connectors></tt> list]]: | ||
<pre> | <pre> | ||
Line 131: | Line 175: | ||
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> | <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> | ||
</entries> | </entries> | ||
<!-- | |||
Optionally: | |||
--> | |||
<connection-load-balancing-policy-class-name> | |||
org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy | |||
</connection-load-balancing-policy-class-name> | |||
</connection-factory> | </connection-factory> | ||
... | ... | ||
Line 138: | Line 189: | ||
... | ... | ||
</pre> | </pre> | ||
For more details on load balancing, see: | |||
<blockquote style="background-color: #f9f9f9; border: solid thin lightgrey;"> | |||
:[[WildFly HornetQ-Based Messaging Subsystem Concepts#Client-Side_Load_Balancing|Client-Side Load Balancing]] | |||
</blockquote> | |||
=What Happens if a Cluster Node is Down when Other Starts Up= | =What Happens if a Cluster Node is Down when Other Starts Up= | ||
Line 159: | Line 216: | ||
</pre> | </pre> | ||
=Client-Side | =Client-Side= | ||
<blockquote style="background-color: #f9f9f9; border: solid thin lightgrey;"> | <blockquote style="background-color: #f9f9f9; border: solid thin lightgrey;"> | ||
:[[WildFly HornetQ-Based Messaging Subsystem - | :[[WildFly HornetQ-Based Messaging Subsystem Concepts#Client-Side_Load_Balancing|Messaging Concepts - Client-Side Load Balancing]] | ||
</blockquote> | </blockquote> |
Latest revision as of 21:25, 1 June 2017
External
- Switch from UDP to TCP for HornetQ clustering in JBoss EAP 6 https://access.redhat.com/solutions/293823
- Administration and Configuration Manual - Switch UDP to TCP for HornetQ Clustering https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Application_Platform/6.4/html/Administration_and_Configuration_Guide/sect-Clustering.html#Switch_UDP_to_TCP_for_HornetQ_Clustering
Internal
Relevance
- EAP 6.4
Overview
This article shows how to configure a network of HornetQ brokers, assuming that the network has no multicast capabilities. For more details on HornetQ Clustering see WildFly HornetQ Clustering Concepts.
Procedure
Make Sure <clustering> is Enabled
The "messaging" subsystem must be configured with <clustered>true</clustered> on all nodes of the cluster:
<subsystem xmlns="urn:jboss:domain:messaging:1.4"> <hornetq-server> ... <clustered>true</clustered> ... </subsystem>
<clustered> post EAP 6.1
Starting with EAP 6.1, the <clustered> configuration attribute was deprecated. If specified, the following message shows up in the logs:
11:52:05,484 WARN [org.jboss.as.messaging] (Controller Boot Thread) JBAS011608: Element CLUSTERED is deprecated and will not be taken into account
Messaging clustering is enabled if one or more <cluster-connection> elements are defined. For more details about <cluster-connection> see Add a <cluster-connection> section.
On Each Cluster Node Configure Connectors To All Other Nodes
Assuming the cluster has three nodes ("node1", "node2" and "node3"), the configuration for node1 should declare <connector>s and <outbound-socket-binding>s for node2 and node3:
... <subsystem xmlns="urn:jboss:domain:messaging:1.4"> <hornetq-server> ... <connectors> ... <netty-connector name="node2-connector" socket-binding="node2-hornetq-binding"/> <netty-connector name="node3-connector" socket-binding="node3-hornetq-binding"/> </connectors> ... </subsystem> ... <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}"> ... <outbound-socket-binding name="node2-hornetq-binding"> <remote-destination host="1.2.3.5" port="5445"/> </outbound-socket-binding> <outbound-socket-binding name="node3-hornetq-binding"> <remote-destination host="1.2.3.6" port="5445"/> </outbound-socket-binding> </socket-binding-group> ...
Note: make sure the nodes at the other end of the connection actually listen on the specified interfaces/ports.
Comment out <broadcast-groups>, <discovery-groups>
They are useless without multicast.
Add a <cluster-connection>
- Cluster connections are unidirectiona and point-to-point. Only one initiator node and one target node can participate to a cluster connection, so for this reason, only one connector-ref can be added as a static-connector for a cluster connection. If connections to more than one nodes are needed, add more <cluster-connection>s.
... </address-settings> <cluster-connections> <cluster-connection name="tcp-based-cluster-node1-to-node2"> <address>jms</address> <connector-ref>netty</connector-ref> <retry-interval>500</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <forward-when-no-consumers>true</forward-when-no-consumers> <max-hops>1</max-hops> <static-connectors> <!-- just one connector-ref here --> <connector-ref>node2-connector</connector-ref> </static-connectors> </cluster-connection> <cluster-connection name="tcp-based-cluster-node1-to-node3"> <address>jms</address> <connector-ref>netty</connector-ref> <retry-interval>500</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <forward-when-no-consumers>true</forward-when-no-consumers> <max-hops>1</max-hops> <static-connectors> <!-- just one connector-ref here --> <connector-ref>node3-connector</connector-ref> </static-connectors> </cluster-connection> </cluster-connections> ...
In EAP 6.1 or newer, adding a <cluster-connection> makes the <clustered> configuration element redundant.
Configure Security
If security is enabled (the <securit-enabled> configuration element is "true", and this is the default value even if it is not explicitly declared), then connections authenticate themselves as "HORNETQ.CLUSTER.ADMIN.USER" - the "cluster user". Unless a "cluster password" is specified in the configuration, a random password is used, and the connection establishing will fail. In order to provide known credentials for the cluster connections, configure the following (<cluster-user> is optional and if not specified will default to "HORNETQ.CLUSTER.ADMIN.USER"):
... <subsystem xmlns="urn:jboss:domain:messaging:1.4"> <hornetq-server> ... <cluster-user>hq</cluster-user> <cluster-password>hq123</cluster-password> ... </hornetq-server> </subsystem> ...
The user declared as such does not need to be declared in any security repository, the configuration file declaration is sufficient.
Confirmation
You should see this in the logs:
EAP 6.4
20:37:47,719 INFO [org.hornetq.core.server] (Thread-16 (HornetQ-server-HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16-46422202)) HQ221027: Bridge ClusterConnectionBridge@20600d51 [name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, queue=QueueImpl[name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16]]@298aa2ce targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@20600d51 [name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, queue=QueueImpl[name=sf.tcp-based-cluster.c9e7d432-e3fe-11e5-915f-d3dc1740ef3d, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16]]@298aa2ce targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-17-79], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1116810330[nodeUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-28-2, address=jms, server=HornetQServerImpl::serverUUID=5032e902-e3fe-11e5-926c-a50d5e1d9a16])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=172-31-17-79], discoveryGroupConfiguration=null]] is connected
Other
15:11:19,234 INFO [org.hornetq.core.server.cluster.impl.BridgeImpl] (Thread-7 (HornetQ-server-HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c-544291442)) Bridge ClusterConnectionBridge@26250f55 [name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, queue=QueueImpl[name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c]]@4b11a903 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@26250f55 [name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, queue=QueueImpl[name=sf.o01-o02-tcp-cluster.c07654e0-810b-11e2-a372-f9f6868e86f7, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c]]@4b11a903 targetConnector=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-155], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1267429957[nodeUUID=388611e3-810b-11e2-85c9-694bd06e7c3c, connector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-154, address=jms, server=HornetQServerImpl::serverUUID=388611e3-810b-11e2-85c9-694bd06e7c3c])) [initialConnectors=[org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-76-128-155], discoveryGroupConfiguration=null]] is connected
Configure ConnectionFactory for Load Balancing
Add the extra connectors to each ConnectionFactories that are exposed and supposed to be used for load balancing/failover. The connectors are the same that have been added to the cluster connection <static-connectors> list:
... <subsystem xmlns="urn:jboss:domain:messaging:1.4"> <hornetq-server> ... <jms-connection-factories> ... <connection-factory name="RemoteConnectionFactory"> <connectors> <connector-ref connector-name="netty"/> <connector-ref connector-name="node2-connector"/> <connector-ref connector-name="node3-connector"/> </connectors> <entries> <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> </entries> <!-- Optionally: --> <connection-load-balancing-policy-class-name> org.hornetq.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy </connection-load-balancing-policy-class-name> </connection-factory> ... </jms-connection-factories> </hornetq-server> </subsystem> ...
For more details on load balancing, see:
What Happens if a Cluster Node is Down when Other Starts Up
The node that is starting up attempts to connect ? times.
If it can't, it logs as DEBUG:
2013-02-20 11:39:45,131 DEBUG [org.hornetq.core.client.impl.ServerLocatorImpl] (Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=afe53124-6e53-11e2-bb90-005056ac34a6-1045936331)) Connector [initialConnector=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-153-198-31]::Exception on establish connector initial connection HornetQException[errorCode=2 message=Unable to connect to server using configuration org-hornetq-core-remoting-impl-netty-NettyConnectorFactory?port=5445&host=10-153-198-31] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:233) at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector$Connector.tryConnect(ServerLocatorImpl.java:1787) at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1624) at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:597) at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:564) at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)