# Tomcat 5 Clustering In JBoss Release 3.2.6

### Ben Wang

August, 2004

## 1. Introduction

Nowadays http web clustering (or failover) typically employs either a hardware or software front-end load balancer to distribute the load (with capability of session affinity). And the session state replication is then delegated to the web container. Naturally, JBoss also has a session replication layer for Tomcat since release 3.x.

In JBoss release 3.2.6, we have re-factored the Tomcat 5 http session replication framework using JBossCache (http://www.jboss.org/products/jbosscache) as the underlying in-memory replication store. JBossCache is an open-source replicated, transactional, and fine-grained cache system. Using JBossCache as the framework for session replication offers separation of replication layer concerns (e.g., replication mode, lock behavior, partitioning, persistence, and transaction) from the Tomcat session handling logics (e.g., sticky session, replication granularity, and trigger). The end result is a reduced development effort in Tomcat application layer and more robust behavior in the overall system.

Note that current implementation will be ported to JBoss4.1 (instead of 4.0) release in the near future.

## 2. Software Design

This section describes briefly the high level design and the web application deployment process pertaining to the session replication.

### 2.1. Class Responsibility

The following is a class diagram focusing on the http session replication using JBossCache as the distributed store service under Tomcat5.

Figure 1. Tomcat5 http session replication class diagram

In addition to the JBossCacheService class that provides replication service, there are three main interfaces:

- `org.jboss.web.tomcat.tc5.session.SnapshotManger.` Interface that defines the API to check for necessary session replication. Note that it has a reference to JBossManager to perform session replication. Two concrete implementation classes are:

  - `org.jboss.web.tomcat.tc5.session.InstantSnapshotManager` . Use of this snapshot manager will check for dirty session after every single http request (through `org.jboss.web.tomcat.tc5.session.ClusteredSessionValve` ). If a session is dirty, it will then be replicated.
  - `org.jboss.web.tomcat.tc5.session.IntervalSnapshotManager` . Use of this snapshot manager will check for dirty session(s) only periodically (with a configurable timer interval). The dirty session(s) will be replicated when the timer kicks in.

- `org.apache.catalina.Manager.` This is a Tomcat catalina interface that handles the session management. The JBossCacheManager is the implemented class that also manages both in-memory session and the sessions located in the distributed store.

  The reason that we need to use the in-memory session instance is because of the web class loader can be different from the system one. As a result, we will need to keep a "serialized" version of the session data in the distributed store (done via `org.jboss.invocation.MarshalledValue`) so that the class loading scope will be correct. Otherwise, it will be simply too expensive to retrieve and de-serialize every time from the underneath store.

  Note that because of the dual use of in-memory and distributed store sessions and because JBossCache does yet not support the notion of modification event only from remote node(s), the current usage of session replication must use sticky session. That is, only one dedicated clustering node will handle the session request unless there is a failover. This should not pose as a severe restriction since it will be rare indeed that sticky session is not used in production today. We will relax this restriction in the future release.

  This manager is also called periodically by catalina to process expired session(s) (or session timeout). When a session is timed-out, it will be deleted from the store. However, the operation is local only. That is, the delete operation will not replicate to other cluster nodes since the other nodes will have similar expiring process in place already. Therefore, the session will be expired simultaneously from the whole cluster view.

- `org.apache.catalina.Session.` This is a catalina interface that manages the session attributes and track the session life time itself. We have two different concrete implementation classes in this release:

  - `org.jboss.web.tomcat.tc5.session.SessionBasedClusteredSession.` This corresponds to the setting in `jboss-web.xml` of `replication-granulairty` set to `session.` That is, whenever a session attribute is changed, it will replicate the whole session, regardless.
  - `org.jboss.web.tomcat.tc5.session.AttributeBasedClusteredSession.` This corresponds to the setting in `jboss-web.xml` of `replication-granulairty` set to `attribute.` That is, whenever a session attribute is changed, it will replicate only the modified attribute. Note that this option which is finer grain in nature has the potential for session replication performance boost, especially when the session is carrying a large amount of data.

### 2.2. Deployment Process

When a user deploys (or hot-deploys) a web application, the JBoss `org.jboss.web.tomcat.tc5.TomcatDeployer` will first determine if the web application is declared `distributable.` If it is, JBoss will instantiate `JBossCacheManager`, create `SnapshotManager` and then add a `ClusteredSessionValve` to the Tomcat interceptor chain. As a result, every single http request will be intercepted by this valve to process the necessary session replication (through `SnapshotManager`).

## 3. Features

Here are the features provided with this release. For clarity, we will divide them into Tomcat and JBossCache level aspects, meaning the configuration will be done either from JBossCache or the TC5 layer.

### 3.1. Tomcat level aspect

In `jbossweb-tomcat5.0/META-INF/jboss-service.xml` , you can configure the followings:

- `SnapshotMode`
  - `instant` Session replication is performed during each HTTP request. But whether it is instantaneously or not also depends on the replication mode (synchronous or asynchronous) in JBossCache setting.
  - `Interval` Replicated during a specified time interval. There is an accompanying flag, `SnapshotInterval`, that indicates the snapshot interval in milliseconds. The `IntervalSnapshotManager` will perform session dirty check every `SnapshotInterval.` This manager keeps track of the dirty session(s) internally.

- `UseJK` A flag indicating whether to use the Apache mod_jk(2) for the front end software load balancing with sticky session combined with JvmRoute. If set to true, it will insert a JvmRouteFilter to intercept every request during deployment and replace the JvmRoute if it detects a failover in runtime. In addition, you will need to set the JvmRoute inside Tomcat, e.g., `<Engine name="jboss.web" jmvRoute="Node1" defaultHost="localhost">` in `jbossweb-tomcat4.0/server.xml`.

Note that the above configuration parameters are applied to per Tomcat instance. Change of the parameter will cause Tomcat to re-deploy.

In each web application, you can also specify in `jboss-web.xml` to customize the clustering behavior. Specifically, there is an element called `replication-config` that contains two sub-elements so far:

- `replication-trigger` It determines what triggers a session replication (or when is a session is considered dirty). It has 4 options:

  - `ACCESS` Indicating if accessing the session is considered dirty. If set, it is considered dirty with pure access and thus will cause replication. Note that a session is "accessed" during each http request regardless. It will update the access time stamp in the session instance as well. Since the time stamp may not be updated in other clustering nodes (because of no replication), this may cause session in other nodes to expire before the active node if http request does not retrieve or modify any session attributes.

    When this option is set, the session timestamps will be synchronized throughout the cluster nodes. Note that use of this option can have a significant performance impact so you will need to use with caution.

  - `SET_AND_GET` Every session get and set attributes are considered dirty. This option also can have significant performance impact as well.
  - `SET_AND_NON_PRIMITIVE_GET` Session get and non-primitive get are considered dirty. For example, the http session request may retrieve a non-primitive object instance from the attribute and then modify the instance. If we don't specify that non-primitive get is considered dirty, then the modification will not be replication properly. This is the default value.
  - `SET` Only set attribute operation is considered dirty. If you are certain that only set attribute operation is necessary for replication, this option will be most optimized in term of performance.
- `replication-granularity`
  - `session` Replication is per session instance. As long as it is considered modified when the snapshot manager is called, the whole session object will be serialized.
  - `attribute` Replication is only for the dirty attributes plus some session data, like, lastAccessTime. For session that carries large amount of data, this option can increase replication performance.
  - `field` (Not yet supported) Will need aop (aspect-oriented programming) and be available in 4.1. User can have control of even finer grain replication through its own object graph. For example, if one stores another map object into the session as an attribute, we can track the dirty field down to the value object inside the map.

### 3.2. JBossCache level aspect

The main configuration file is `tc5-cluster-service.xml` of which is basically a minimum version of JBossCache configuration xml file (more details later). In other wor

- `CacheMode`
  - `REPL_SYNC` Replication is synchronous and call is blocking until replication succeeds or fails. That is, replication will be completed before the http request returns.
  - `REPLY_ASYNC` Replication is asynchronous and call is non-blocking. Does not wait for results for the http request to return. When session state replication is allowed to have some latency, this is useful and will have a much better performance throughput.

In the future release, we will also add the following features from JBossCache.

- (sub)-Partitioning. This feature will partition the cluster into different sub-partitions such that replication only happens in the sub-partition nodes. With this feature, the session replication cluster can then be easily scaled.
- Persistence. Persist the session data to the backed store. Currently, clustered session does not support any persistency.
- Eviction policy and cache data overflow memory size or time-based eviction policy with overflow to back end persistent store. This is useful for users who have long running sessions that needs to support passivation.

### 3.3. Deprecated

The following features have been deprecated in the new release:

- `replication-type` in `jboss-web.xml.` User should directly configure the replication type from `tc5-cluster-service.xml` with the element ``CacheMode``. Please also note that the specification of ``CacheMode`` is now per Tomcat instance instead of per web application.

- `UseLocalCache` in `jbossweb-tomcat5.0/META-INF/jboss-service.xml.` Local in-vm session is used by default now.

## 4. What's New?

JBossCache can be configured through a xml configuration and deployed under JBoss3.2.6 as a MBean service. Since JBossCache is used not only for clustering but also for `SSO` (single sign on) replication as well, we have introduced a JBossCache tc5 clustering MBean service for everyone to share (such as `SSO` replication). The xml configuration file will be called `tc5-cluster-service.xml`. Once the file is deployed, a MBean called `TomcatClusteringCache` will be created.

In the file, there are JBossCache configuration parameters but customized for the tc5 clustering. Here is a snippet of the xml:

    <server>
    <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TomcatClusteringCache">
      <depends>jboss:service=Naming</depends>
      <depends>jboss:service=TransactionManager</depends>

    <!--
                Configure the TransactionManager. Default should be JBossTransactionManager.
    -->
      <attribute name="TransactionManagerLookupClass">org.jboss.cache.JBossTransactionManagerLookup</attribute>

    <!--
                 Valid modes are LOCAL
                                 REPL_ASYNC
                                 REPL_SYNC
    -->
      <attribute name="CacheMode">REPL_ASYNC</attribute>

    <!--
     Name of cluster. Needs to be the same for all clusters, in order
                 to find each other
    -->
      <attribute name="ClusterName">TOMCAT-Cluster</attribute>

    <!--
     JGroups protocol stack properties. Can also be a URL,
                 e.g. file:/home/bela/default.xml
               <attribute name="ClusterProperties"></attribute>
    -->
    <attribute name="ClusterConfig">
      <config>
        <UDP mcast_addr="228.1.2.5" mcast_port="45566" ip_ttl="64" ip_mcast="true"
          mcast_send_buf_size="150000" mcast_recv_buf_size="80000" ucast_send_buf_size="150000"
          ucast_recv_buf_size="80000" loopback="false" />
         ... more config params here ...
      </config>
    </attribute>

    </mbean>
    </server>

## 5. Test Case

All http session clustering test cases are located under `JBoss-3.2/testsuite/src/main/org/jboss/test/cluster/apache-tomcat` directory.

## 6. Sample Usage

In this section, we describe how to setup a sample web application for http session replication. User should refer to other documentation on how to setup a front end load balancer (e.g., Apache mod_jk(2)).

To run a web application session replication under the new tc5 clustering, you will need to set the application to be distributable as usual. For example, in `web.xml` (under the web app `WEB-INF` directory),

    <web-app>
       <distributable/>
       ...
    </web-app>

Next, you can create an optional `jboss-web.xml` (also under `WEB-INF` directory) that has the following options:

    <jboss-web>
       <replication-config>
          <replication-trigger>SET_AND_NON_PRIMITIVE_GET</replication-trigger>
          <replication-granularity>SESSION</replication-granularity>
       </replication-config>
    </jboss-web>

In addition, you will need to start jboss from `all` configuration directory. Under the new `all` configuration, it includes the JBossCache library and also the `tc5-cluster-service.xml`file to deploy the cache as a specific tc5 clustering MBean service.

Finally, before you start JBoss, you can edit `jboss-service.xml` under `jbossweb-tomcat5.0/META-INF` directory for the following:

    <jboss-service>
        <attribute name="SnapshotMode">instant</attribute>
        <attribute name="UseJK">false</attribute>
    </jboss-service>

You are now ready to use http session replication!