JGroups Protocol FD: Difference between revisions
| (8 intermediate revisions by the same user not shown) | |||
| Line 10: | Line 10: | ||
=Overview= | =Overview= | ||
FD is a failure detection protocol based on sending regular messages to a neighbor and waiting for replies. | |||
=Failure Detection= | |||
FD implement a failure detection mechanism based on heartbeat messages. A member sends 'are-you-alive' messages with a periodicity of 'timeout' milliseconds. These are "FD" messages containing a HEARTBEAT header. The messages are sent to the neighbor to the right, identified by the position in the view. For an [A, B, C] view, A will send messages to B, B will send heartbeat messages to C and C will send heartbeat messages to A. | FD implement a failure detection mechanism based on heartbeat messages. A member sends 'are-you-alive' messages with a periodicity of 'timeout' milliseconds. These are "FD" messages containing a HEARTBEAT header. The messages are sent to the neighbor to the right, identified by the position in the view. For an [A, B, C] view, A will send messages to B, B will send heartbeat messages to C and C will send heartbeat messages to A. | ||
When the neighbor receives the HEARTBEAT, it replies with a message containing a "FD" - HEARTBEAT_ACK header. When HEARTBEAT_ACK is received the timestamp is set to current time and the counter is set it to 0. | When the neighbor receives the HEARTBEAT, it replies with a message containing a "FD" - HEARTBEAT_ACK header. When HEARTBEAT_ACK is received the timestamp is set to current time and the counter is set it to 0. | ||
After the ''first'' missing heartbeat response, the initiating member send ''more'' 'max_tries' heartbeat messages and the target member is declared suspect only after all heartbeat messages go unanswered. The SUSPECT message is sent down the stack, is addressed to all members. | After the ''first'' missing heartbeat response, the initiating member send ''more'' 'max_tries' heartbeat messages and the target member is declared suspect only after all heartbeat messages go unanswered. The SUSPECT message is sent down the stack, is addressed to all members. | ||
| Line 33: | Line 35: | ||
2015-06-17 15:31:06,727 DEBUG [org.jgroups.protocols.FD] (54er-2,shared=udp) server26/sr: broadcasting SUSPECT message (suspects=[server24/sr]) | 2015-06-17 15:31:06,727 DEBUG [org.jgroups.protocols.FD] (54er-2,shared=udp) server26/sr: broadcasting SUSPECT message (suspects=[server24/sr]) | ||
</pre> | </pre> | ||
Note that the "<tt>received no heartbeat from [...] for [...] times ([...] milliseconds), suspecting it"</tt> log entry reports the exact amount of time the node waited to declare suspicion. | |||
In the worst case, when the target member dies immediately after answering a heartbeat, the failure takes <tt>timeout + timeout + max_tries * timeout = (max_tries + 2) * timeout</tt> milliseconds to detect. | In the worst case, when the target member dies immediately after answering a heartbeat, the failure takes <tt>timeout + timeout + max_tries * timeout = (max_tries + 2) * timeout</tt> milliseconds to detect. | ||
| Line 38: | Line 42: | ||
Once a member is declared suspected, it will be verified by the [[JGroups Protocol VERIFY_SUSPECT|VERIFY_SUSPECT]] protocol, and if it indeed is unavailable, it will be excluded by [[JGroups Protocol GMS|GMS]]. | Once a member is declared suspected, it will be verified by the [[JGroups Protocol VERIFY_SUSPECT|VERIFY_SUSPECT]] protocol, and if it indeed is unavailable, it will be excluded by [[JGroups Protocol GMS|GMS]]. | ||
If we use FD_SOCK instead, then we don't send heartbeats, but establish TCP sockets and declare a member dead only when a socket is closed. | If we use [[JGroups Protocol FD_SOCK|FD_SOCK]] instead, then we don't send heartbeats, but establish TCP sockets and declare a member dead only when a socket is closed. | ||
==Also See== | |||
<blockquote style="background-color: #f9f9f9; border: solid thin lightgrey;"> | |||
:[[JGroups Failure Detection|JGroups Events - Failure Detection]] | |||
</blockquote> | |||
=Recommendations= | |||
<blockquote style="background-color: #f9f9f9; border: solid thin lightgrey;"> | |||
:[[JGroups Protocol FD_ALL#Recommendations|FD_ALL Recommendations]] | |||
</blockquote> | |||
=Configuration= | |||
JGroups standalone: | |||
<pre> | |||
<FD timeout="6000" max_tries="5"/> | |||
</pre> | |||
WildFly: | |||
<pre> | |||
<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp"> | |||
<stack name="udp"> | |||
... | |||
<protocol type="FD"> | |||
<property name="timeout">60000</property> | |||
</protocol> | |||
... | |||
</stack> | |||
</subsystem> | |||
</pre> | |||
==max_tries== | |||
The default max_tries is 5 on EAP. | The default max_tries is 5 on EAP. | ||
==timeout== | |||
<font color=red>The timeout is 6000 ms on EAP.</font> | <font color=red>The timeout is 6000 ms on EAP.</font> | ||
Latest revision as of 03:08, 3 March 2016
External
- User Manual FD http://www.jgroups.org/manual/index.html#FD
- JGroups Wiki http://community.jboss.org/wiki/JGroupsFD
- FD versus FD_SOCK http://community.jboss.org/wiki/FDVersusFDSOCK
Internal
Overview
FD is a failure detection protocol based on sending regular messages to a neighbor and waiting for replies.
Failure Detection
FD implement a failure detection mechanism based on heartbeat messages. A member sends 'are-you-alive' messages with a periodicity of 'timeout' milliseconds. These are "FD" messages containing a HEARTBEAT header. The messages are sent to the neighbor to the right, identified by the position in the view. For an [A, B, C] view, A will send messages to B, B will send heartbeat messages to C and C will send heartbeat messages to A.
When the neighbor receives the HEARTBEAT, it replies with a message containing a "FD" - HEARTBEAT_ACK header. When HEARTBEAT_ACK is received the timestamp is set to current time and the counter is set it to 0.
After the first missing heartbeat response, the initiating member send more 'max_tries' heartbeat messages and the target member is declared suspect only after all heartbeat messages go unanswered. The SUSPECT message is sent down the stack, is addressed to all members.
This is how the log reflects a heartbeat missing:
2015-06-17 15:26:21,588 DEBUG [org.jgroups.protocols.FD] (54er-4,shared=udp) server09/SchemaRepository: heartbeat missing from server26/SchemaRepository (number=1)
The number of attempts is logged as: (number=1), (number=2), ...
This is how the log reflects "suspicion":
2015-06-17 15:31:06,727 DEBUG [org.jgroups.protocols.FD] (54er-3,shared=udp) server26/sr: received no heartbeat from server24/sr for 5 times (300000 milliseconds), suspecting it 2015-06-17 15:31:06,727 DEBUG [org.jgroups.protocols.FD] (54er-2,shared=udp) server26/sr: broadcasting SUSPECT message (suspects=[server24/sr])
Note that the "received no heartbeat from [...] for [...] times ([...] milliseconds), suspecting it" log entry reports the exact amount of time the node waited to declare suspicion.
In the worst case, when the target member dies immediately after answering a heartbeat, the failure takes timeout + timeout + max_tries * timeout = (max_tries + 2) * timeout milliseconds to detect.
Once a member is declared suspected, it will be verified by the VERIFY_SUSPECT protocol, and if it indeed is unavailable, it will be excluded by GMS.
If we use FD_SOCK instead, then we don't send heartbeats, but establish TCP sockets and declare a member dead only when a socket is closed.
Also See
Recommendations
Configuration
JGroups standalone:
<FD timeout="6000" max_tries="5"/>
WildFly:
<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">
<stack name="udp">
...
<protocol type="FD">
<property name="timeout">60000</property>
</protocol>
...
</stack>
</subsystem>
max_tries
The default max_tries is 5 on EAP.
timeout
The timeout is 6000 ms on EAP.