We recently had serious issues with the ActiveMQ redelivery plugin. Under high load messages would not be redelivered at all (according to the application logs), they went straight to the DLQ. In isolated tests redelivery worked, but not as intended. We would get six redeliveries in a few milliseconds and then nothing. Why?
It turned out to be a conflict between client-side redelivery and broker redelivery. The client side redelivery kicked in and quickly failed six times. Then when the broker got a chance the maximum redelivery count had already been reached, so the message was moved to the DLQ. See this post.
Following the advice in the post we added
jms.redeliveryPolicy.maximumRedeliveries=0 to the broker url, and voila! It worked.
ActiveMQ (and Red Hat’s AMQ derivative) is gaining ground. Many companies that need high availability already have highly available databases, so it makes sense to use ActiveMQ in a master/slave configuration with a JDBC message store. A bit slower than KahaDB or LevelDB, but simple to configure and very safe. Or is it? Well, it depends.
ActiveMQ really has two different message stores: one for normal messages and one for scheduled messages. For normal persistent messages it is possible to use KahaDB, LevelDB or JDBC. For scheduled messages there are only two alternatives: KahaDB and in-memory. So, if you have a highly available master/slave configuration with a JDBC backend for normal messages, all scheduled messages will be lost when the master dies, as they are stored in the local file system or in memory.
Scheduled messages are seldom used, so perhaps this is trivial? Unfortunately not. Most applications want to redeliver failed messages a few times before relegating them to a DLQ. ActiveMQ supports redelivery, but under the hoods it is implemented using scheduled messages. In other words all messages waiting for redelivery are lost at failover unless the job scheduler store is highly available, which means a highly available KahaDB store. Installations that use JDBC don’t do HA KahaDB.
In other words, take care when using ActiveMQ or AMQ with a JDBC backend! Normal messages are safe, but some are likely to be lost at failover. Is that acceptable?
A documentation bug has been filed for this.