JGroups Protocol TCP

Internal

• JGroups

Overview

TCP creates a server socket, which gives this group member its local address.

For each accept() on the server socket, a new thread is created to handle the message. For each outgoing message, if the message's destination is in the outgoing hash table, the associated socket will be reused to send message, otherwise a new socket is created and put in the table. When a socket connection breaks or a member is removed from the group, the corresponding items in the incoming and outgoing hash tables are removed as well. This functionality is in TCPConnectionMap, which is used by TCP. TCP sends messages using ct.send() and registers with the connection table to receive all incoming messages.

Configuration

TP Configuration

TP Configuration

Sending and Receiving

A TCP-based stack may receive the messages either via the main server socket (when a direct connection to another member is not established yet) or via a dedicated socket - each peer-to-peer connection is maintained over a dedicated socket. The TCP protocol maintains an internal structure named "connection map". TCPConnectionMap is the class that implements it. Each "connection" in the map essentially consists in a TCP socket and data structures and functionality aimed at servicing it. If your group has 100 members, each member maintains 99 connections in its "connection map", one for everybody else.

Upon creation, each connection starts a receiving thread, in charge with reading socket incoming data. The naming pattern for the receiving threads is "Connection.Receiver [<bind_add>:<bind_port> - <remote-address>:<remote-port>]”. You can recognize them in a thread dump, they will always be present there - they cannot be configured out.

However, as long as there is no network activity from peers, these threads will be blocked awaiting the results of a network read, so they will not consume CPU.

If the stack is configured with TCP.use_send_queue="true", each connection also starts a separated sender thread, in charge with picking up the contents of a per-connection sending queue and writing it on the socket's DataOutputStream. The naming pattern for the sender threads is “Connection.Sender [<bind_add>:<bind_port> - <remote-address>:<remote-port>],test,<logical-address>,5,JGroups]” This configuration option will increase the number of TCP threads per group member to 99 + 99 = 198, assuming a 100 member group. Same as in receiver threads' case, as long as the sender threads are not busy writhing data on the output stream, they're blocked waiting on the sending queue's monitor so they won't consume CPU.

Aside for the mandatory receiver threads and optional sender threads, there is an extra "acceptor" thread that accept()s on the TCP protocol ServerSocket. This threads accepts TCP connections from group members that are in process of starting up and did not establish a "connection" yet.

So, for a 100 member group, each group member can maintain up to 99 + 99 + 1 = 199 TCP threads.