Chapter 8. Performance Tests

8.1. How to execute our performance tests

JBoss Messaging 2.0 ships with a set of basic performance measurements comparing our performance against a selection of other open source messaging systems. We would like to provide performance figures against proprietary systems too, but unfortunately, this is usually prohibited by the other messaging system's licence terms

Note

It should be stressed that these performance figures are far from exhaustive and provide a basic view for how JBoss Messaging 2.0 performs against the other systems in a selection of simple and classic JMS use cases. All the tests use the standard JMS 1.1 API. Please remember JBoss Messaging 2.0 is only an alpha release so don't expect this release to be perfect in all scenarios, although it is interesting to observe that even at this early stage JBoss Messaging 2.0 appears to provide better performance than the other messaging systems. JBoss Messaging 2.0 is still largely un-optimised - we have more performance to squeeze out still. More in depth performance figures will be provided for the final GA release.

In the spirit of open-ness and not wanting to make performance claims we cannot substantiate, all these performance figures can easily be reproduced using just the tools available in this distribution, assuming you have installed the other messaging systems. We have used the JMS Examples to produce these numbers, and we used perfSender and perfListener with different scenarios. You can replicate those scenarios using these instructions provided below:

All messages used in the tests are BytesMessage instances with 1K bodies. Measurement time is taken from the time of the first message sent after the warmup period to the time of the last message consumed. Message throughput rates are measured in messages / sec. In most tests 200000 messages were sent, although this was reduced with some providers due to the provider running out of memory. In all tests we first start a consumer listening on the queue, then start a producer sending to the queue. Broker was running on the server machine, and both the producer and the consumer were running on the client machine.

The tests were performed on very basic commodity hardware. In the near future we will be obtaining the use of a large performance lab with serious hardware, on which we look forward to obtaining more results

  • Test 1. Send non persistent, non transactional messages. Consume messages non transactional with ack mode DUPS_OK_ACKNOWLEDGE

               

    ant perfListener

    ant perfSender

  • Test 2. Send non persistent, non transactional, messages. Consume mesages non transactional with ack mode AUTO_ACKNOWLEDGE

    	   

    ant perfListener -Dsess.ackmode=AUTO_ACK

    ant perfSender

  • Test 3. Send persistent (blocking *) non transactional, messages. Consume with ack mode DUPS_OK_ACKNOWLEDGE

               

    ant perfListener

    ant perfSender -Ddelivery.mode=PERSISTENT

  • Test 4. Send persistent (non blocking *), non transactional, messages. Consume with ack mode DUPS_OK_ACKNOWLEDGE

               

    ant perfListener

    ant perfSender -Ddelivery.mode=PERSISTENT

  • Test 5. Send persistent (blocking *), non transactional, messages. Consume with ack mode AUTO_ACKNOWLEDGE

    	   

    ant perfListener -Dsess.ackmode=AUTO_ACK

    ant perfSender -Ddelivery.mode=PERSISTENT

  • Test 6. Send persistent (non blocking *), non transactional, messages. Consume with ack mode AUTO_ACKNOWLEDGE

    	   

    ant perfListener -Dsess.ackmode=AUTO_ACK

    ant perfSender -Ddelivery.mode=PERSISTENT

  • Test 7. Large transaction test. Send persistent, transactional, messages with transaction size = 1000. Consume persistent transactional, transaction size = 1000

    	   

    ant perfListener -Dsess.trans=true -Dsess.trans.size=1000

    ant perfSender -Dsess.trans=true -Dsess.trans.size=1000 -Ddelivery.mode=PERSISTENT

  • Test 8. Small transaction test. Send persistent, transactional, messages with transaction size = 2. Consume persistent transactional, transaction size = 2

               

    ant perfListener -Dsess.trans=true -Dsess.trans.size=2

    ant perfSender -Dsess.trans=true -Dsess.trans.size=2 -Ddelivery.mode=PERSISTENT

* Some messaging systems send persistent messages by default synchronously, and others send them by default asynchronously. Some supports both modes and others only support one. To avoid confusion we consider sending persistent messages synchronously or asynchronously separately. Configuring a particular system to send synchronously or asynchronously is specific to the system. For instance in Apache QPID we can specify use blocking mode by providing the system property "sync_persistence" with the value "true" to the client.

8.2. Systems Used

All the tests were executed on the same hardware, operating system and JDK configuration. All messaging system configuration was with out of the box defaults unless otherwise stated.

  • Client:

    Hardware: Dell Latitude D820, dual core 2 x 2GHz Intel CoreDuo, 2GB RAM, 1GB ethernet card.

    Operating system: Linux, 32 bit, kernel version 2.6.22-14-generic

    JDK: Sun JDK 1.6.0_03-b05

  • Server:

    Dell Precision 470 workstation, dual cpu 2 X 2.8 GHz Intel Xeon 64 bit, 2GB RAM, 1GB ethernet card.

    Operating system: Linux, 64 bit, kernel version 2.6.22-14-generic

    JDK: Sun JDK 1.6.0_03-b05 64bit

  • Network:

    1 GB ethernet

    Netgear GS105 Gigabit switch

8.3. Performance Results

All the tests were executed against these following messaging systems:

  • 1. JBM 2.0 alpha. Out of the box config used.

  • 2. JBM 1.4.0.SP3_CP02 - this is the production version used in JBoss Enterprise Application Platform 4.3 GA CP01. Out of the box config used. Used with MySQL 5.0.45 colocated with server and InnoDB tables with innodb_flush_log_at_trx_commit=1 (enable sync at tx commit)

  • 3. ActiveMQ 5.1. This is the latest production version of ActiveMQ. Out of the box config was used with two changes: 1) syncOnWrite was set to true in the persistence config (otherwise ActiveMQ won't sync to disc on tx boundaries) 2) The upper queue memory limit was extended to 100MB otherwise the tests would block as queues became full.

  • 4. JBoss MQ in JBoss AS 4.2.2.GA. This is JBoss' legacy JMS provider - now superceded by JBoss Messaging. Used with MySQL 5.0.45 colocated with server and Innodb tables with innodb_flush_log_at_trx_commit=1 (enable sync at tx commit)

Table 8.1. Performance Results (all results in messages/sec)

TestJBM 2.0_AlphaJBM 1.4.0.SP3_CP02Apache ActiveMQ 5.1JBoss MQ
Test 1. NP/Dups18,83613,40112,963881
Test 2. NP/AutoAck14,1433,8899,813672
Test 3. Persist/Blocking/NonTX/DupsOk1,372442312622
Test 4. Persist/Non Blocking/NonTX/DupsOk14,977JBM 1.x does not support non blocking persistent message sendsActiveMQ 5.1 does not support non blocking persistent message sendsJBoss MQ does not support non blocking persistent message sends
Test 5. Persist/Blocking/NonTX/AutoAck1,265421524637
Test 6. Persist/Non Blocking/NonTX/AutoAck12,056JBM 1.x does not support non blocking persistent message sendsActiveMQ 5.1 does not support non blocking persistent message sendsJBoss MQ does not support non blocking persistent message sends
Test 7. Large transactions. Persistent/Transacted size=10009,6071,3551,576805
Test 8. Small transactions. Persistent/Transacted size=21,818546396523

8.4. Performance conclusions.

JBoss Messaging 2.0 provides the highest throughput in all tests, and in particular shows ground breaking persistent performance for both transactional and non transactional operations.

JBoss Messaging 1.4.0.SP3_CP02 and ActiveMQ 5.1 are good all-round performers and show similar and respectable performance.

JBoss MQ has reasonable persistent performance, but is let down by poor non persistent results.