HTTP Session Replication

Internal

• Servlet Session Tracking API
• web.xml
• HTTP Session

Overview

The web application will be clustered if its web.xml contains the <distributable> tag:

<web-app  xmlns="http://java.sun.com/xml/ns/j2ee"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
          xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee 
                              http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd" 
          version="2.4">
    <distributable/>
    <!-- ... -->
</web-app>

From a server's perspective, <distributable> tells it to use a clustered session manager, which replicates (or distributes) sessions across the cluster, so all EAP nodes can have access to it. This way if a node fails, requests can be directed to other node, which still has access to the session state.

The sever also must to be configured to support HTTP session clustering, see WildFly Server Configuration below.

WildFly-Specific Implementation

Synchronous and Asynchronous Replication

Sessions can replicate either synchronously or asynchronously.

In case of synchronous replication, the implementation of the session management makes sure a session chance is replicated before returning the HTTP response.

In case of asynchronous replication, the session management initiates the replication of the change in a separate thread and returns the HTTP response without waiting for a successful replication confirmation.

Synchronous vs. asynchronous replication is a trade-off between performance and high-availability. Synchronous replication provides full HA, but requires a per-request performance cost. Asynchronous replication provides almost full HA, but with low per-request performance cost. Synchronous replication can technically be used with or without session stickiness. Asynchronous replication needs session stickiness to work correctly.

From Support: Synchronous replication mode means that replication calls block until they receive acknowledgments from all nodes in a cluster that they received and applied the modification successfully. Asynchronous does not wait for those acknowledgements, but the replication call still occurs within the calling request thread by default.

For more details on how session replication mode can be configured, see below: jboss-web.xml replication-mode.

Asynchronous Replication Implementation

Asynchronous Replication Implementation Details

Layers of Synchronicity

The web container session manager can be configured to perform HTTP session replication synchronously or asynchronously.

Then, in JBoss' case, the underlying cache that actually performs the replication can be configured in a synchronous or asynchronous mode.

If the underlying Infinispan container is configured to delegate to a remote JDG store, the underlying JDG cache can be configured in a synchronous or asynchronous mode.

In order for replication to happen synchronously, all these layers must be configured to perform synchronous replication.

If the HTTP session is configured with a SYNCHRONOUS replication mode, but the cache is ASYNC, the replication will look like synchronous from the HTTP session perspective, but the actual propagation in the cache will be asynchronous, so the replication will be effectively asynchronous.

Session Stickiness and Replication

Session stickiness configuration and the replication mechanism are related. Synchronous replication can technically be used with or without session stickiness. Asynchronous replication needs session stickiness to work correctly.

If sticky sessions are disabled and the replication is asynchronous, it is possible that a subsequent request associated with the session arrives on a different node than the previous request, but before the change introduced by the previous request is replicated to the node. This is an invalid state, that must be avoided, because the change introduced by the subsequent request will be overwritten by the replication event, when it finally arrives. This is why non-sticky sessions and asynchronous replication must not be used together.

If sticky sessions are enabled and a node fails, the load balancer will detect the node failure and send the request to another node. The session should be available to that node because the session is replicated.

However, if asynchronous replication was used, it is possible that the last change has not been replicated to the target node at the time the subsequent request for that session arrives from the load balancer, and this is another invalid state we want to avoid.

Session Stickiness and Session Ownership

Before a node tries to use a session, it first tries to obtain the session ownership via locking. For more implementation details related to session ownership and locking see Session Ownership and Locking - Implementation Details. If sticky sessions are disabled, everytime a request for a given session bounces between nodes, it will require an RPC to obtain session ownership. This can be expensive since session ownership is acquired before the request is processed.This is why sticky session is in most cases a good idea.

WildFly-Specific Configuration

WildFly-specific HTTP session replication configuration can be specified in jboss-web.xml in the <replication-config> section. All the following configuration elements are optional:

<jboss-web>
    ...
    <replication-config>
        <!--
              <cache-name>web.repl</cache-name>
        -->
        <cache-name>${app.cache.container.name}.${app.cache.name}</cache-name>
        <replication-trigger>SET</replication-trigger>
        <replication-granularity>SESSION</replication-granularity>
        <replication-mode>ASYNCHRONOUS</replication-mode>
    </replication-config>
    ...
</jboss-web>

Configuration Elements

cache-name

The name of the cache that should be used for storing distributable sessions and clustering them around the cluster.

If not explicitly set in the deployment descriptor, the cache to use is the web container default set in the infinispan subsystem:

Infinispan Subsystem Default Web Container Cache

replication-trigger

Determines when the container should consider that a session must be replicated across the cluster. Possible values are:

• ACCESS: merely accessing the session marks the session as dirty.
• SET_AND_GET: is conservative but not optimal (performance-wise): it will always replicate the session even if its content has not been modified but simply accessed.
• SET_AND_NON_PRIMITIVE_GET (default value) is conservative but will only replicate if a non-primitive Object has been accessed (i.e. the object is not of a well-known immutable JDK type such as Integer, Long, String, etc.) This is the default value.
• SET assumes that the developer will explicitly call setAttribute on the session if it needs to be replicated. This setting prevents unnecessary replication, but requires very good coding practices to ensure setAttribute is always called whenever an attribute value is modified.

The default, if not set, is SET_AND_NON_PRIMITIVE_GET.

The rationale for this setting is that after a mutable object stored as a session attribute is accessed from the session, in the absence of a setAttribute() call the container has no clear way to know if the object (and hence the session state) has been modified.

In all cases, calling setAttribute() marks the session as needing replication.

replication-granularity

Determines the session replication granularity level. Possible values are:

• SESSION (default) - indicates that replication is done per session instance, i.e. when the session is considered modified, the whole session object will be serialized and replicated. This is the preferred policy when the sessions are generally small.
• ATTRIBUTE - indicates that replication is performed only for the the dirty attributes in the session, plus some session data, like lastAccessTime. For sessions carrying large amounts of data, parts of which are infrequently accessed, this option can increase replication performance.
• FIELD was available in AS 4.x and AS 5.x, but it is not supported by AS 6.

replication-mode

Determines the session replication mode. Possible values are:

• SYNCHRONOUS - session replication occurs in the same thread that processes a given request. A request will not complete until the associated session is successfully replicated.
• ASYNCHRONOUS - session replication occurs in a separate thread from the one that processes a given request. The request thread only initiates replication, but does not wait for it to complete.

If not defined, the replication mode defined by the underlying cache configuration will be used. Infinispan configuration details are available here. In the default distributed session cache configuration, sessions replicate asynchronously.

For more general considerations on synchronous vs. asynchronous replication see above: Synchronous and Asynchronous Replication. That section also explains when the HTTP layer and the underlying caching layer are configured differently from a synchronicity point of view, see Layers of Synchronicity.

backups

Defines the number of backup nodes on which to replicate a given session. Possible values are:

• Positive integral value - session will be replicated only to the specified number of nodes (i.e. distribution mode).
• 0 - Session will not be replicated (i.e. local mode).
• Negative integral value - session will be replicated to *all* nodes (i.e. total replication).

if not explicitly set, is taken from the distributed cache configuration. By default, this is -1, i.e. total replication.

use-jk

Whether the container should assume mod_jk is used for load balancing for this webapp. If set to 'true', the container will examine the session id associated with every request and replace the JvmRoute portion of the session id if it detects a failover. In addition, for each host you will need to set a unique JvmRoute inside the server.xml file:

...
<Engine name="jboss.web" jvmRoute="Node1" defaultHost="localhost">
    ...
</Engine>
...

Default value if not explicitly set is the overall web container default.

max-unreplicated-interval

Determines the maximum interval between requests, in seconds, after which a request will trigger replication of the session's timestamp and other metadata regardless of whether the request has otherwise made the session dirty. Such replication ensures that other nodes in the cluster are aware of the most recent value for the session's timestamp and won't incorrectly expire an unreplicated session upon failover. It also results in correct values for HttpSession.getLastAccessedTime() calls following failover.

The cost of this metadata replication depends on the configured replication-granularity. With SESSION, the session's attribute map is replicated along with the metadata, so it can be fairly costly. With other granularities, the metadata object is replicated separately from the attributes and only contains a String, and a few longs, ints and booleans. A value of 0 means the metadata will be replicated whenever the session is accessed. A value of -1 means the metadata will be replicated only if some other activity during the request (e.g. modifying an attribute) has resulted in other replication work involving the session. A positive value greater than the HttpSession.getMaxInactiveInterval() value will be treated as a likely misconfiguration and converted to 0; i.e. replicate the metadata on every request.

snapshot-mode

Defines when the sessions are replicated to the other nodes. Default value if not explicitly set is the overall web container default; by default that is set to "INSTANT".

The typical value, "INSTANT", replicates changes to the other nodes at the end of requests, using the request processing thread to perform the replication. In this case, the "snapshot-interval" property is ignored.

With "INTERVAL" mode, a background process is created that runs every "snapshot-interval" milliseconds, checking for modified sessions and replicating them.

This property has no effect if replication-granularity is set to FIELD. If it is FIELD, "instant" mode will be used.

snapshot-interval

Defines how often (in milliseconds) the background process that replicates modified sessions should be started for this web app. Only meaningful if snapshot-mode is set to "interval". Default value if not explicitly set is the overall web container default. By default that is set to "1000".

session-notification-policy

Fully qualified class name of the implementation of the org.jboss.web.tomcat.service.session.notification.ClusteredSessionNotificationPolicy interface that should be used to govern whether servlet specification notifications should be emitted to any registered HttpSessionListener, HttpSessionAttributeListener and/or HttpSessionBindingListener. Event notifications that may make sense in a non-clustered environment may or may not make sense in a clustered environment; configuring an appropriate ClusteredSessionNotificationPolicy gives the application author fine-grained control over what notifications are issued.Default value if not explicitly set is the org.jboss.web.tomcat.service.session.notification.IgnoreUndeployLegacyClusteredSessionNotificationPolicy.

System Properties Usage

jboss-web.xml supports the usage of system properties, whose references will be replaced at deployment time with values defined in the application server's JVM. As an example, the name of the cache container and the cache name can be specified as system properties as follows:

...
<replication-config>
    <cache-name>${app.cache.container.name}.${app.cache.name}</cache-name>
    ...
</replication-config>
...

For more details on system properties usage in jboss-web.xml, see:

System Properties Usage in jboss-web.xml

WildFly Server Configuration

WildFly Infinispan HTTP Session Replication Configuration

JBossWeb/Tomcat HTTP Session Implementation Details

JBossWeb/Tomcat HTTP Session Implementation Details