I have looked at the state of different messaging backends recently and i ran a little benchmark to see what is the rough comparison of message publishing throughput. Results that i got are quite surprising.
What i wanted to achieve was some sort of reassurance before choosing a messaging bus for my PHP project. PHP is usually a bit special as it's runtime environment is different than java / .net. I wanted to use rabbitmq because of it's routing flexibility and implementation of AMQP. After the simple benchmark i am not convinced any more if that is the best way to go for me right now.
This is not a comprehensive benchmark
Please do not get stressed out too much if my results are poor. I spent just one day playing with the backends so i do not claim expertise in optimising their performance at all. It is not a real benchmark as PHP code and message bus were both ran on the same machine so network protocol overhead was neglected.
I have used local deployment of each backend engine with pretty much default settings. As a client i had a very simple set of scripts trying to insert 1000 messages per request. Then i ran requests from jmeter in 1, 5, 10 and 15 concurrent threads.
Why persistent messages?
I wanted to see what is the throughput on heavy insert loads. I also wanted to see what would happen if your publishers outperform consumers or if you had to disconnect consumers for a while (workers upgrade or crash). I can not afford to loose messages in such scenario so I need them to be persistent.
Why 1000 messages per execution?
I wanted to see the rough throughput and cost of posting a message, posting just a few messages would not work as most time would be spent processing php and bootstraping the engine not executing publishing code. I did not run them within one transaction though (which makes huge difference for postgres and innodb).
- PHP 5.3.6
- Mysql Server 5.1.61. Client: PDO mysql used directly. Connecting once and pushing inserts into a simple 2 column table.
- Variant 1: inserts into MyISAM table which was fast
- Variant 2: inserts into InnoDB table with innodb_flush_log_at_trx_commit=2 to disable ACID properties of innodb, this way inserts were much faster but not as safe as default.
- Variant 3: inserts InnoDB table while using a default transaction log fsyncing where ACID is enforced.
- ActiveMq 5.5.1 on ubuntu's openjdk. Client: pecl stopm extension 1.0.3. Publishing as persistent messages.
- RabbitMq 2.7.1. compiled.
- Client1: pecl amq extension 1.0.1
- Client2: php-amqplib taken from https://github.com/videlalvaro/php-amqplib
- Mongodb 1.8.2 ubuntu binaries. Client pecl mongo 1.2.9. Publishing as a fire and forget insert into a collection
- Redis 2.2.11 ubuntu's binaries. Client pecl extension 2.1.3
- APC 3.1.9 ubuntu's binaries. Not really considered as a message queue engine but just to have a comparison
- Memcached 1.4.7 ubuntu's binaries. Client pecl memcached extension 1.0.2. Not really considered as a message queue engine but just to have a comparison
- CouchDB 1.0.1. Not really considered as a message queue engine but just to have a comparison
Update, i have used the binary extension for RabbitMQ this time but i still kept the amqp-php as well just to get comparison. I still think accessing RabbitMQ from PHP is not perfect, lets hope VM Ware supports the PHP driver and makes it super awesome!
Hardware and system
I have used my Intel Core 2 Duo laptop with 4GB of RAM, Ubuntu 11.10, Kernel 3.0, ext4 file system.
I used databases as they are common for application-level queue backends. I have used mongodb and Redis as I have seen PHP based queue implementations using them and because their performance is just amazing.
I used memcached and APC only for comparison as in general you can not go much faster than them. They are non-persistent so there is no way to compare apples and oranges, its just to get more reference points.
I have extended the test by creating a better jmeter test suite and running long tests on each backend. Each thread group was running for 2 minutes non stop so total test time was 8 minutes per backend with increasing concurrency.
My initial results were incorrect as my tests ran for short time so they did not actually force all the backends to write to the disk. As you will see below ActiveMQ was extremely fast up to the point when it started writing to the drive and throttling clients. ActiveMQ is fast as a monster as long as you write to memory and you can consume the messages more or less as fast as they come. In this scenario ActiveMQ was as fast as memcached and for the first 300 000 messages it went like a thunder. Once it ran out of memory it's performance dropped extremely though.
First chart shows throughput of each PHP client + backend. It is throughput of persistent messages per second where my messages were really small (about 250 characters each).
Second chart shows throughput measured in the same way but using APC and memcached (the really fast backends).
The last chart shows throughput measured in the same way but using the slowest backends.
More detailed numbers
Here is my google doc with all the numbers.
First of all ActiveMQ is a very solid message bus and PHP connectivity is really fast. It does not handle persistent messages as fast as i hoped though and MongoDB is still 5 times faster at accepting messages and writing them to disk!
RabbitMQ seemed a decent while using the pecl extension but very inefficient with the PHP based AMQP client. I did not like the fact that it took some manual compilation of dependencies to get c library and pecl extension built.
Finally Redis and MongoDB have shown just amazing performance on long-time extreme write workload.
If it was not for the PHP client issues, the fact that the AMQP protocol is stateful and so complex/verbose i think i would go for RabitMQ now. Considering my initial experiences i think i will go with ActiveMQ and i give another star to MongoDB in my book :)
Download the test files
Finally you can download all the sources of all the tests i have used. Please feel free to run them yourself and provide feedback on what did i do wrong if you spot anything :)
To run the PHP scripts you will need to have all the PECL extensions loaded and PHP 5.3 for some of them.
To run the jmeter suite you will have to use the latest version of jmeter 2.6. All you have to do is to change config element and use your host name and put the php prefix name in the backend variable.
Enjoy it !