Archive

Posts Tagged ‘ActiveMQ’

Negative queue depths in ActiveMQ

For quite some time we battled with negative and sometimes false positive queue depths in ActiveMQ and the commercial AMQ equivalent. Apparently there is a race condition somewhere in the broker. It is usually easy to see when it has happened in the graphs, as the number of messages on a queue suddenly jumps up or down dramatically in an instant. It does play havoc with monitoring, though. How can you set alarms if you can’t trust the reported queue depth? Restarting the broker fixes the counter, but that is not a viable solution.

Fortunately there is a configuration option that solves this. Define a policyEntry for the affected queues/topics and set useCache="false". That is a fairly large change, but it works.

Advertisements
Categories: Java

ActiveMQ redelivery plugin fails when client side redelivery is active

We recently had serious issues with the ActiveMQ redelivery plugin. Under high load messages would not be redelivered at all (according to the application logs), they went straight to the DLQ. In isolated tests redelivery worked, but not as intended. We would get six redeliveries in a few milliseconds and then nothing. Why?

It turned out to be a conflict between client-side redelivery and broker redelivery. The client side redelivery kicked in and quickly failed six times. Then when the broker got a chance the maximum redelivery count had already been reached, so the message was moved to the DLQ. See this post.

Following the advice in the post we added jms.redeliveryPolicy.maximumRedeliveries=0 to the broker url, and voila! It worked.

Categories: Java

Beware of ActiveMQ with JDBC persistence

ActiveMQ (and Red Hat’s AMQ derivative) is gaining ground. Many companies that need high availability already have highly available databases, so it makes sense to use ActiveMQ in a master/slave configuration with a JDBC message store. A bit slower than KahaDB or LevelDB, but simple to configure and very safe. Or is it? Well, it depends.

ActiveMQ really has two different message stores: one for normal messages and one for scheduled messages. For normal persistent messages it is possible to use KahaDB, LevelDB or JDBC. For scheduled messages there are only two alternatives: KahaDB and in-memory. So, if you have a highly available master/slave configuration with a JDBC backend for normal messages, all scheduled messages will be lost when the master dies, as they are stored in the local file system or in memory.

Scheduled messages are seldom used, so perhaps this is trivial? Unfortunately not. Most applications want to redeliver failed messages a few times before relegating them to a DLQ. ActiveMQ supports redelivery, but under the hoods it is implemented using scheduled messages. In other words all messages waiting for redelivery are lost at failover unless the job scheduler store is highly available, which means a highly available KahaDB store. Installations that use JDBC don’t do HA KahaDB.

In other words, take care when using ActiveMQ or AMQ with a JDBC backend! Normal messages are safe, but some are likely to be lost at failover. Is that acceptable?

A documentation bug has been filed for this.

Categories: Java