JBoss Community

What is the Cache Benchmark Framework?

It is a framework that allows running multiple performance tests against various cache products. It was primarily conceived as a tool to benchmark different releases of JBoss Cache.

Here are some useful features offered by the framework:

• Works on any caching product - simply write a CacheWrapper implementation, which delegates calls like put(), get() and remove().

• pluggable tests - JUnit-like tests can be written to mimic your data access patterns and cache usage scenarios, making benchmarks relevant and useful.  We ship with a few typical tests, such as SessionSimulatorTest which mimics storing and replicating of HTTP sessions.

• pluggable reports - prefer excel spreadsheets to CSV?  Or SVG images to PNGs?  Write your own report generators.  Ships with a basic CSV generator.

• cluster barriers - guarantee that all running nodes synchronize during each stage of the test, such as warm-up, test and report generation phases, guaranteeing that each of these phases happens at the same time on each cluster node.

• centralized reports - each node of the cluster produces its own performance report.  The primary node (first node in the cluster) then collates these reports into a single report file.

• cache warmup - triggers a sequence of operations on the cache before running any tests, to allow for hotspot compilers to optimize.

• replication assertion - asserts that any values intended to be replicated have in fact been replicated.

Usage

Download and install

Prerequisites:

• Apache ANT version 1.7.x or higher is needed for building the binaries.

• A Java 5 JDK is needed to build and run the binaries.

• The framework ships with a number of shell scripts to enable proper classpath setting and command execution.

• Cygwin can be used for running the test on windows; if so add the scripts from :pserver:anoncvs@cygwin.com:/cvs/cygwin-apps/wrappers/java to $cygwin_home/usr/local/bin. Those scripts make an automatic conversion from unix CLASSPATH to win CLASSPATH, needed when executing java -cp

• Running cluster-wide tests require SSH and passphrase-less public keys set up on all servers, to enable cluster-wide execution.

Download

There is no public release of the framework at the moment.  However, it can be built from sources.  The Subversion repository is http://anonsvn.jboss.org/repos/jbosscache/benchmarks/benchmark-fwk.

After getting the sources from Subversion,

1. Download the JARs for specific cache products; SRC_ROOT/cache-products/<cache-product-name>/lib/README.txt for details pertaining to specific cache products.

2. Build the sources using ant all in the source root directory.

Configure and run a test

Configuring

1. Before running a test configure conf/cachebench.xml. The configuration settings are documented within the file. IMPORTANT please read this file carefully as it contains important information, like where reports will be generated, etc.

2. Configure logging by editing conf/log4j.xml. Recommended verbosity levels are WARN or ERROR, so that benchmark measurements won't be affected by unnecessary logging. In order to have individual log files for each node in the cluster, use org.cachebench.utils.PerNodeRollingFileAppender rather than log4j's org.apache.log4j.RollingFileAppender. This extends org.apache.log4j.RollingFileAppender (i.e. support its configurations) and prepend the node index on the node's log file name.

3. Edit allJBossCacheTests.sh and ensure that the configs, scaling and products variables accurately describe the test configs you intend to run. The framework will execute each config against each existing product, on all specified cluster sizes.

4. Edit bindAddress.sh so it accurately points to the network interface to bind clustered caches to.  It is a good idea for this to point to an environment variable that is set individually by each host.

Running tests

• some cache products, such as Terracotta, might need additional setup, such as starting of a master servers. All this information is described in an Readme.txt file in the directory which coresponds to the benchmark product, e.g. cache-products/terracotta-2.5.0/Readme.txt. Make sure you follow any steps described in that file before running the tests.
  

• allJBossCacheTests.sh will kick a test run, according to the configuration you set when editing it.

• cluster.sh stop will kill all instances running all all nodes. If you are not sure whether the benchmark exit gracefully at the previous run, better run this first to make sure that there are no daemon nodes interfering with new ones.

Results

Results will appear a format and in the location specified in SRC_ROOT/conf/cachebench.xml.  By default this is a CSV file, in the project root directory.

Local mode

The framework can now deal with non-clustered tests as well.  Use the runLocalNode.sh script instead of the cluster.sh script which remotely runs runNode.sh.  Note that you should use an appropriately tuned configuration for your cache, so that it does not create unnecessary network resources.  Also, you should NOT use the ClusteredReportGenerator when running tests in local mode - you won't get any results!!

This is very useful for benchmarking non-clustered performance.

Write your own tests

• Every newly created test should implement org.cachebench.tests.CacheTest. For more details please refer to CacheTest's javadoc.  The framework ships with some tests already, see the org.cachebench.tests package for details and examples.

• Archive your new test as a JAR file and drop it in the lib directory so that it is picked up by the framework.

• Specify your new test to be run in SRC_ROOT/conf/cachebench.xml, within a <test /> element.  Refer to how existing tests are specified for examples.

Benchmark a new cache product

To add Cache-Product-X.Y.Z to the cache benchmark framework, follow these steps:

1. Create a directory SRC_ROOT/cache-products/Cache-Product-X.Y.Z

2. Create SRC_ROOT/cache-products/Cache-Product-X.Y.Z/lib for the cache product distribution jars.

3. Create SRC_ROOT/cache-products/Cache-Product-X.Y.Z/src and write a cache wrapper for the product.  See SRC_ROOT/cache-products/jbosscache-2.0.0/src/org/cachebench/cachewrappers/JBossCache200Wrapper.java for example.

4. Copy SRC_ROOT/cache-products/config.sh to SRC_ROOT/cache-products/Cache-Product-X.Y.Z/, and modify the file as per the comments to suit your product.  This script builds necessary classpaths.

5. Your cache product is now ready to be benchmarked, as per the instructions above.

TODOs

See here

Feedback

Please provide all feedback on this on the JBoss Cache User Forums for now, until a separate mailing list can be set up.