How to setup terracotta session clustering and replication for apache tomcat 6

Terracotta is an amazing piece of software and it comes with some really cool tools and features. To enable Tomcat 6 session replication via terracotta you need to do a few things but its relatively simple lets do it.

How does Terracotta session work with tomact?

You will have one or more terracotta servers to spread the load and allow for high reliability. Each server will run in its own processes in separate JVM (and probably on separate machine). Then you will have clients using API and bytecode instrumentation to allow existing applications and web containers to connect to terracotta and use its features.

In case of tomcat you will have your regular tomcat 6 with a set of special runtime parameters that will include terracotta at startup. During startup tomcat will connect to server and sessions clustering will be taken care of without and application change. You can have as many tomcat instances connecting to the terracotta servers cluster.

Setting up terracotta servers

First you need to download the binaries of terracotta

The current version of terracotta (at the moment of writing) is 3.3. Download the installer and run

java -jar terracotta-3.3.0-installer.jar

Once that is done you need to create a configuration file for the server and clients to use. Create tc-config.xml and put it in the terracotta folder or in /etc/ or whatever.

<?xml version="1.0" encoding="UTF-8"?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.terracotta.org/schema/terracotta-4.xsd">
 <servers>
   <server name="nodea" host="10.0.0.1">
     <data>/whatever/mydatafolder</data>
     <logs>/whatever/mylogsfolder</logs>
     <l2-group-port>9530</l2-group-port>
   </server>
   <server name="nodea" host="10.0.0.2">
     <data>/whatever/mydatafolder</data>
     <logs>/whatever/mylogsfolder</logs>
     <l2-group-port>9530</l2-group-port>
   </server>
 </servers>
 <clients>
   <logs>/var/log/myclientlogsfolder</logs>
       <modules>
       <module name="tim-tomcat-6.0" version="2.2.0"/>
   </modules>
 </clients>
 <application>
   <dso>
     <instrumented-classes>
       <include>
         <class-expression>*..*</class-expression>
       </include>
       <exclude>org.apache.coyote..*</exclude>
       <exclude>org.apache.catalina..*</exclude>
       <exclude>org.apache.jasper..*</exclude>
       <exclude>org.apache.tomcat..*</exclude>
     </instrumented-classes>
     <web-applications>
       <web-application>sessionapp</web-application>
     </web-applications>
    </dso>
 </application>
</tc:tc-config>

This will create 2 nodes, it will instruct that clients should load tomcat integration module and it will also tell terracotta which applications and classes should be enhanced.

Its quite a lot of magic behind the scenes as terracotta replaces some of the libraries at the start time and uses byte code modification to enrich classes and allow transparent clustering! I know ... it is cool!

Creation of boot jar for tomcat

As i mentioned before apache tomcat will need a special jar with classes to be replaced during jvm startup. You can generate that jar running following command:

cd /opt/springsource/terracotta/platform/bin/
./make-boot-jar.sh -f /opt/springsource/terracotta/tc-config.xml

Installation of Tomcat TIM (Terracotta Integration Module)

To make sessions clustering work you will need to include special module in your installations. They take care of plugging into frameworks and making sure things work.

To install missing modules based on config file just run the following:

./tim-get.sh list
./tim-get.sh install-for /opt/springsource/terracotta/tc-config.xml

Starting up Terracotta server and using the tools

Now you can start up your servers that make up the cluster:

cd /opt/springsource/terracotta/bin/
./start-tc-server.sh -f /opt/springsource/terracotta/tc-config.xml

If everything is ok you can now startup a terracotta console tools by running

cd /opt/springsource/terracotta/bin/
./dev-console.sh

This amazing tool will let you inspect session, clustered objects graphs, monitor requests velocity and much more! it will also show you live view of clients connected to the cluster and nodes operating in the cluster. It is a quite impressive tool. Really worth playing with, you can inspect objects in your HTTP session live in your tomcat web servers (tomcats are not clustered only sessions are!)

You can monitor status of the cluster with tons of great features!

Setting up apache tomcat 6 to use terracotta session clustering

You have to modify the startup.sh script and add something like this:

TC_INSTALL_DIR="/opt/springsource/terracotta";
export TC_INSTALL_DIR;
TC_CONFIG_PATH="/opt/springsource/terracotta/tc-config.xml";
export TC_CONFIG_PATH;
TC_BOOT_JAR="/opt/springsource/terracotta/lib/dso-boot/dso-boot-hotspot_osx_160_20.jar";
export TC_BOOT_JAR;

export JAVA_OPTS="-Xbootclasspath/p:$TC_BOOT_JAR -Dtc.install-root=$TC_INSTALL_DIR -Dtc.config=$TC_CONFIG_PATH -Xmx512M -XX:MaxPermSize=512m"

You have to replace the boot jar name (make boot jar script shows you a location and name of your jar).

Testing if it works

I just create an empty grails app with one controller and one view showing something like this:

       session.counterValue = (session.counterValue ?: 0) + 1
       int ct   = session.counterValue
       def time = new Date()
       def name = ManagementFactory.getRuntimeMXBean().getName()
       render( view:"index", model : ["counter":ct, "time":time, "name":name] )

Then i deploy it to separate tomcat instances (on one or more machines). I access the app to see that counters are incremented independently.

It also lets me see that my tomcats are actually separate JVMs and what is the value in the session.

Now if i switch terracotta session replication and clustering on for apache tomcat servers using the modified startup script that included bootjar and configuration file i can see that as long as my session id is the same i can access any of the servers and my session is valid and counter increments as expected.

Its easiest to have 2 tomcats on one box as then browser uses same host name just different port. If host name is the same you dont even have to override session id (browser will send the same session id to both tomcat instances as they are on the same host so cookies are shared).

Comments

i have installed terracotta

i have installed terracotta 3.6.2,i want to know , how to connect a client with that, i am using glassfish server ???

2012-05-30 23:00
priya

No, unfortunately i cant help

No, unfortunately i cant help you on this one.

Hope you find the solution, all the best.

Art

2012-05-07 02:40
admin

We have this exact setup,

We have this exact setup, with 2 webapps running under the same tomcat instance ROOT, and estore. ROOT behaves perfectly, however for every hit to estore, I can see 3 sessions being created in terracotta. Have you ever seen anything like this?

2012-05-02 13:27
Sarah Holt

Post new comment

Image CAPTCHA

About the author

Artur Ejsmont

Hi, my name is Artur Ejsmont,
welcome to my blog. I am a passionate software engineer living in Sydney and working for Yahoo!

Web Scalability for Startup Engineers

If you are into technology, you can order my book Web Scalability for Startup Engineers on Amazon. I would love to hear what are your thoughts so please feel free to drop me a line or leave a comment.

Follow my RSS