Infinispan HotRod Java Client
Internal
Overview
HotRod defines three level of intelligence for the clients:
- basic client, interested in neither cluster nor hash information
- topology-aware client, interested in cluster information
- hash-distribution-aware client, that is interested in both cluster and hash information
The java client implementation supports all three levels of intelligence. It is transparently notified whenever a new server is added/removed from the HotRod cluster. At startup it only needs to know the address of one HotRod server On connection to the server the cluster topology is piggybacked to the client, and all further requests are being dispatched to all available servers. Any further topology change is also piggybacked.
The client is also hash-distribution-aware. This means that, for each operation, the client chooses the most appropriate remote server to go to: the data owner. As an example, for a put(k,v) operation, the client calculates k’s hash value and knows exactly on which server the data resides on. Then it picks up a tcp connection to that particular server and dispatches the operation to it. This means less burden on the server side which would otherwise need to lookup the value based on the key’s hash. It also results in a quicker response from the server, as an additional network roundtrip is skipped. This hash-distribution-aware aspect is only relevant to the distributed HotRod clusters and makes no difference for replicated server deployments.
For a replicated cluster, the client can load balance requests. This aspect is irrelevant with distributed clusters, where, as shown above, the client sends/receives data from the data owner node. The default strategy is round-robin: requests are being dispatched to all existing servers in a circular manner. Custom types of balancing policies can defined by implementing the RequestBalancingStrategy and by specifying it through the infinispan.client.hotrod.request-balancing-strategy configuration property.
In order to avoid creating a TCP connection on each request (which is a costly operation), the client keeps a pool of persistent connections to all the available servers and it reuses these connections whenever it is possible. The validity of the connections is checked using an async thread that iterates over the connections in the pool and sends a HotRod ping command to the server. By using this connection validation process the client is being proactive: there’s a hight chance for broken connections to be found while being idle in the pool and no on actual request from the application. The number of connections per server, total number of connections, how long should a connection be kept idle in the pool before being closed - all these (and more) can be configured. RemoteCacheManager documentation describes configuration options.
Various server usage statistics can be obtained through the RemoteCache.stats() method. This returns a ServerStatistics object.
API
The basic API to interact with an Infinispan server over the HotRod protocol:
import org.infinispan.client.hotrod.RemoteCache;
import org.infinispan.client.hotrod.RemoteCacheManager;
import org.infinispan.client.hotrod.configuration.Configuration;
import org.infinispan.client.hotrod.configuration.ConfigurationBuilder;
...
String host = "localhost";
int port = 11222;
Configuration c = new ConfigurationBuilder().addServer().host(host).port(port).build();
RemoteCacheManager remoteCacheManager = new RemoteCacheManager(c);
RemoteCache defaultCache = emoteCacheManager.getCache();
RemoteCache namedCache = remoteCacheManager.getCache("some-cache");
RemoteCacheManager
In order to be able to use an RemoteCache instance, the RemoteCacheManager must be started first: beside other things, this instantiates connections to Hot Rod server(s). A constructor that has being passed a Configuration instance starts automatically. Otherwise, the start() method can be used. The RemoteCacheManager is an "expensive" object, as it manages a set of persistent TCP connections to the Hot Rod servers. It is recommended to only have one instance of this per JVM, and to cache it between calls to the server. stop() needs to be called explicitly in order to release all the resources (e.g. threads, TCP connections).
Relationship between RemoteCacheManager and a Server-Side Cache Container
The RemoteCacheManager is configured with a host name and port. The host:port pair is associated on the server-side with a specific socket binding definition ("hotrod" in this case):
<socket-binding-group name="standard-sockets" ...>
...
<socket-binding name="hotrod" port="11222"/>
...
</socket-binding-group>
The socket binding definition is associated with the HotRod connector configuration declared in the "infinispan:server:endpoint" subsystem (more details about the HotRod connector are available here: The HotRod Connector):
<subsystem xmlns="urn:infinispan:server:endpoint:6.1">
<hotrod-connector cache-container="clustered" socket-binding="hotrod">
<topology-state-transfer lazy-retrieval="false" lock-timeout="1000" replication-timeout="5000"/>
</hotrod-connector>
</subsystem>
The HotRod connector thus declared is bound to a specific cache container, and this is what the client will get access to:
<subsystem xmlns="urn:infinispan:server:core:6.4">
<cache-container name="clustered" default-cache="default" statistics="true">
...
<distributed-cache name="default" .../>
<distributed-cache name="something" .../>
<cache-container name="security"/>
</subsystem>
RemoteCacheManager Configuration
The cache manager is configured through a Configuration object passed to the constructor.
Supported configuration elements:
infinispan.client.hotrod.request_balancing_strategy
The default value is org.infinispan.client.hotrod.impl.transport.tcp.RoundRobinBalancingStrategy. For replicated (vs distributed) Hot Rod server clusters, the client balances requests to the servers according to this strategy. Distributed clusters do not require this. For more details see load balancing above.
infinispan.client.hotrod.server_list
The default is 127.0.0.1:11222. This is the initial list of Hot Rod servers to connect to, specified in the following format: host1:port1;host2:port2... At least one host:port must be specified.
infinispan.client.hotrod.force_return_values
The default value is false. Whether or not to implicitly org.infinispan.client.hotrod.Flag.FORCE_RETURN_VALUE for all calls.
infinispan.client.hotrod.tcp_no_delay
The default is true. Affects TCP NODELAY on the TCP stack.
infinispan.client.hotrod.tcp_keep_alive
The default is false. Affects TCP KEEPALIVE on the TCP stack.
infinispan.client.hotrod.transport_factory
The default is org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory. It controls which transport to use. Currently only the TcpTransport is supported.
infinispan.client.hotrod.marshaller
The default is org.infinispan.marshall.jboss.GenericJBossMarshaller. Allows to specify a custom org.infinispan.marshall.Marshaller implementation to serialize and deserialize user objects. For portable serialization payloads, you should configure the marshaller to be org.infinispan.client.hotrod.marshall.ApacheAvroMarshaller.
infinispan.client.hotrod.async_executor_factory
Default is org.infinispan.client.hotrod.impl.async.DefaultAsyncExecutorFactory. Allows to specify a custom asynchroous executor for async calls.
infinispan.client.hotrod.default_executor_factory.pool_size
Default is 99. If the default executor is used, this configures the number of threads to initialize the executor with.
infinispan.client.hotrod.default_executor_factory.queue_size
Default value 100000. If the default executor is used, this configures the queue size to initialize the executor with.
infinispan.client.hotrod.hash_function_impl.1
Default = It uses the hash function specified by the server in the responses as indicated in org.infinispan.client.hotrod.impl.consistenthash.ConsistentHashFactory. This specifies the version of the hash function and consistent hash algorithm in use, and is closely tied with the HotRod server version used.
infinispan.client.hotrod.key_size_estimate
Default 64. This hint allows sizing of byte buffers when serializing and deserializing keys, to minimize array resizing.
*
*
*
*
*
*
*
* The following properties are related to connection pooling:
*
*
-
*
- maxActive - controls the maximum number of connections per server that are allocated (checked out to client threads, or idle in * the pool) at one time. When non-positive, there is no limit to the number of connections per server. When maxActive * is reached, the connection pool for that server is said to be exhausted. The default setting for this parameter is * -1, i.e. there is no limit. *
- maxTotal - sets a global limit on the number persistent connections that can be in circulation within the combined set of * servers. When non-positive, there is no limit to the total number of persistent connections in circulation. When * maxTotal is exceeded, all connections pools are exhausted. The default setting for this parameter is -1 (no limit). * *
- maxIdle - controls the maximum number of idle persistent connections, per server, at any time. When negative, there is no limit * to the number of connections that may be idle per server. The default setting for this parameter is -1. *
-
* whenExhaustedAction - specifies what happens when asking for a connection from a server's pool, and that pool is exhausted. Possible values:
*
-
*
- 0 - an exception will be thrown to the calling user *
- 1 - the caller will block (invoke waits until a new or idle connections is available. *
- 2 - a new persistent connection will be created and returned (essentially making maxActive meaningless.) *
* The default whenExhaustedAction setting is 1.
*
* -
* Optionally, one may configure the pool to examine and possibly evict connections as they sit idle in the pool and to
* ensure that a minimum number of idle connections is maintained for each server. This is performed by an "idle connection
* eviction" thread, which runs asynchronously. The idle object evictor does not lock the pool
* throughout its execution. The idle connection eviction thread may be configured using the following attributes:
*
-
*
- timeBetweenEvictionRunsMillis - indicates how long the eviction thread should sleep before "runs" of examining idle * connections. When non-positive, no eviction thread will be launched. The default setting for this parameter is * 2 minutes *
- minEvictableIdleTimeMillis - specifies the minimum amount of time that an connection may sit idle in the pool before it * is eligible for eviction due to idle time. When non-positive, no connection will be dropped from the pool due to * idle time alone. This setting has no effect unless timeBetweenEvictionRunsMillis > 0. The default setting for this * parameter is 1800000(30 minutes). *
- testWhileIdle - indicates whether or not idle connections should be validated by sending an TCP packet to the server, * during idle connection eviction runs. Connections that fail to validate will be dropped from the pool. This setting * has no effect unless timeBetweenEvictionRunsMillis > 0. The default setting for this parameter is true. * *
- minIdle - sets a target value for the minimum number of idle connections (per server) that should always be available. * If this parameter is set to a positive number and timeBetweenEvictionRunsMillis > 0, each time the idle connection * eviction thread runs, it will try to create enough idle instances so that there will be minIdle idle instances * available for each server. The default setting for this parameter is 1. *
* - *
*
*
*
Interacting with a HotRod Server from within the Same JVM
TODO: