JGroups Protocol FD

External

• User Manual FD http://www.jgroups.org/manual/index.html#FD
• JGroups Wiki http://community.jboss.org/wiki/JGroupsFD
• FD versus FD_SOCK http://community.jboss.org/wiki/FDVersusFDSOCK

Internal

• JGroups

Overview

FD is a failure detection protocol based on sending regular messages to a neighbor and waiting for replies.

Failure Detection

FD implement a failure detection mechanism based on heartbeat messages. A member sends 'are-you-alive' messages with a periodicity of 'timeout' milliseconds. These are "FD" messages containing a HEARTBEAT header. The messages are sent to the neighbor to the right, identified by the position in the view. For an [A, B, C] view, A will send messages to B, B will send heartbeat messages to C and C will send heartbeat messages to A.

When the neighbor receives the HEARTBEAT, it replies with a message containing a "FD" - HEARTBEAT_ACK header. When HEARTBEAT_ACK is received the timestamp is set to current time and the counter is set it to 0.

After the first missing heartbeat response, the initiating member send more 'max_tries' heartbeat messages and the target member is declared suspect only after all heartbeat messages go unanswered. The SUSPECT message is sent down the stack, is addressed to all members.

This is how the log reflects a heartbeat missing:

2015-06-17 15:26:21,588 DEBUG [org.jgroups.protocols.FD] (54er-4,shared=udp) server09/SchemaRepository: heartbeat missing from server26/SchemaRepository (number=1)

The number of attempts is logged as: (number=1), (number=2), ...

This is how the log reflects "suspicion":

2015-06-17 15:31:06,727 DEBUG [org.jgroups.protocols.FD] (54er-3,shared=udp) server26/sr: received no heartbeat from server24/sr for 5 times (300000 milliseconds), suspecting it
2015-06-17 15:31:06,727 DEBUG [org.jgroups.protocols.FD] (54er-2,shared=udp) server26/sr: broadcasting SUSPECT message (suspects=[server24/sr])

Note that the "received no heartbeat from [...] for [...] times ([...] milliseconds), suspecting it" log entry reports the exact amount of time the node waited to declare suspicion.

In the worst case, when the target member dies immediately after answering a heartbeat, the failure takes timeout + timeout + max_tries * timeout = (max_tries + 2) * timeout milliseconds to detect.

Once a member is declared suspected, it will be verified by the VERIFY_SUSPECT protocol, and if it indeed is unavailable, it will be excluded by GMS.

If we use FD_SOCK instead, then we don't send heartbeats, but establish TCP sockets and declare a member dead only when a socket is closed.

Also See

JGroups Events - Failure Detection

Recommendations

FD_ALL Recommendations

Configuration

JGroups standalone:

    <FD timeout="6000" max_tries="5"/>

WildFly:

<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">
   <stack name="udp">
      ...
      <protocol type="FD">
         <property name="timeout">60000</property>
      </protocol>
      ...
   </stack>
</subsystem>

max_tries

The default max_tries is 5 on EAP.

timeout

The timeout is 6000 ms on EAP.