Performance tuning WebSphere Application Server 7 on AIX 6.1

This week I've been working with AIX 6.1.0.0 and WAS 7.0.7 Network Deployment. The cluster topology is as follows:

• 1 Power6 blade running the Deployment Manager and a managed Web Server (IHS 7.0.7 + Plugin)
• 4 LPARs with 8 x 3.5Ghz and 16 Gb RAM (on a Power 750) as nodes with 1 Application Server instance per node

Before I get into the detail, I'd like to point out that I hadn't used AIX for a while, and found this developerWorks article really useful to remind me of all the commands that I'd forgotten.

The aim of the game this week has been throughput, the test involves a 25 step HTTP interaction delivered by jMeter with no pauses and very little ramp up. It's not a "real world" test at all and the goal was to identify the optimum
throughput point for the application beyond which latency becomes unacceptable.

Before we started tuning, nmon was showing the CPUs as busy, but with a higer than expected proportion of system time and noticable context switching suggesting that threads were spinning waiting for resources.  I must stress that the tuning below is not a replacement for performance analysis of your application to understand where programmatic improvements can be introduced to minimise locking (see page 16 of this for some app dev guidance). Further, as with any performance tuning, application of this tuning may deliver a throughput enhancement, but equally it may move the point of contention to elsewhere in the stack. In short, I'm not claiming that I have a one-size-fits-all set of magic setting that will act as the silver bullet to all your performance problems, but I wanted to share what's worked for me this week along with why I used the settings that I did. I'll also collate the information that is documented in a variety of places into a single resource for you and cover:

• AIX environment variables
  
• IBM HTTP Server configuration
• WebSphere web server plug-in configuration

AIX environment variables
The WAS Info Center has plenty of information on AIX OS setup, but not much that seems to be pertinent to our issue. There is good information however, in both the AIX 6.1 Info Center and chapter 4.5 of this Redbook - you should bookmark both. The AIX Info Center has a set of recommended environment variables, and the Redbook builds on that with network settings which I confess I didn't use this week as we've not been network bound. Below is the list of environment variables from the AIX Info Center plus a few extras from some expert colleagues:

• AIXTHREAD_COND_DEBUG = OFF
• AIXTHREAD_COND_RWLOCK = OFF
• AIXTHREAD_MUTEX_DEBUG = OFF
• AIXTHREAD_MUTEX_FAST = ON
• AIXTHREAD_SCOPE = S
• SPINLOOPTIME = 500
• YEILDLOOPTIME = 32

Chapter 4.7 of the Redbook details creating a shared rc.was files so that you can apply these settings to all of your servers, but I simply applied them in Servers -> Server Types -> Application Servers -> <server name> -> Java and Process Management -> Process definition -> Environment Entries as shown below:


IBM HTTP Server configuration
After reading a number of Technotes and support articles, I finally decided that this Technote was the most useful resource for improving the out of the box configuration of the multi-processing module in IBM HTTP Server. As each IHS server thread has it's own copy of the WebSphere plugin, under the relentless load we were generating this seemed like a potential area of weakness wherby different plugins could have different views of the availability of the available Application Servers, so we settled on the configuration used in example 1 of the Technote which has a single server thread:

httpd.conf
ThreadLimit 2000
ServerLimit 1
StartServers 1
MaxClients 2000
MinSpareThreads 2000
MaxSpareThreads 2000
ThreadsPerChild  2000
MaxRequestsPerChild  0

If you're running on a UNIX-based platform, you may need to have ulimit -s 512 in the session which starts IHS.

Note that using a single server in this way means that if that process exits unexpectedly you potentially lose 2000 in-flight connections.

Web server plug-in configuration
Combining the wisdom from both of these two Technotes (Understanding plugin load balancing and Recommended configuration values) I ended up making the following changes to the default plugin configuration:

• Set all but one of your servers LoadBalanceWeight to 20 and the remaining server to 19. This prevents the plugin from reducing the weights by finding a common denominator and results in the the weights getting reset less frequently.
• If you're using session affinity (who isn't?) the ensure that you set IgnoreAffinityRequests=false on your ServerCluster entry. This works around a known limitation of the plugin which can result in skewed weighting when using round robin distribution and session affinity.

Prior to making these changes we were seeing servers marked down by the plugin, which consequentially resulted in the other servers bearing more load. These changes prevented servers from being marked down and gave a smoother workload distribution amongst the cluster.

plugin-cfg.xml
<ServerCluster CloneSeparatorChange="false" GetDWLMTable="true" IgnoreAffinityRequests="false" LoadBalance="Round Robin" Name="myCluster" PostBufferSize="64" PostSizeLimit="-1" RemoveSpecialHeaders="true" RetryInterval="60">
<Server CloneID="14uddvvpt" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="20" MaxConnections="-1" Name="node01Server01" ServerIOTimeout="60" WaitForContinue="false">
...
<Server CloneID="14udeeloa" ConnectTimeout="5" ExtendedHandshake="false" LoadBalanceWeight="19" MaxConnections="-1" Name="node01Server02" ServerIOTimeout="60" WaitForContinue="false">