Enable JCA resource adapter statistics in JBoss EAP 6

Performance tuning is guesswork without good instrumentation. Knowing the bottleneck is at least half the battle. Thus it is always nice to graph connection pool statistics for databases and resource adapters. With EAP 6, statistics can be enabled in the connection-definition element, for example:

<connection-definition
      class-name="org.apache.activemq.ra.ActiveMQManagedConnectionFactory"
      jndi-name="java:/QCF" enabled="true" pool-name="QCF"
      statistics-enabled="true"
      >

Unfortunately it doesn’t work, most metrics (such as InUseCount) are reported as 0. Why? Apparently this is a bug that has been fixed in EAP 7. However, for 6.4.8+ statistics can be enabled by setting the system property -Dorg.jboss.as.connector.deployers.ra.enablePoolStatistics=true. Do it now…

JBoss EAP 6 and CACHE_CONSUMER with Spring DMLC

The documentation for Spring’s DefaultMessageListenerContainer states:

Some Java EE servers only register their JMS resources with an ongoing XA transaction in case of a freshly obtained JMS Connection and Session.

Rumour has it that JBoss EAP is one of the guilty servers, but is it true? We couldn’t find any definitive answers for the recent versions. But yes, caching consumers with JBoss EAP 6 using XA transactions will fail. The application server only enlists with the active transaction when a connection is checked out from the pool. If the consumer is reused for other transactions that will seem to work (no error messages), but in fact the messages are consumed without transactions. If a rollback occurs messages are lost.

In short, be sure to use CACHE_NONE with JBoss EAP 6.

Prevent hawtio from phoning home

Hawtio is bundled in several applications as a management console. In one project we are using it with JBoss EAP 6. However, when the application server starts hawtio attempts to update itself from Github:

Performing a pull in git repository .hawtio/config on remote URL:
https://github.com/hawtio/hawtio-config.git.
Subsequent pull attempts will use debug logging
Failed to pull from the remote git repo with credentials null due:
https://github.com/hawtio/hawtio-config.git:
407 Proxy Authentication Required. This exception is ignored.

I certainly don’t want applications in production to update themselves dynamically with unforseen effects and neither do I want them to phone home. What to do? Fortunately it is possible to control this, see the documentation. Simply add the following Java options:

-Dhawtio.offline=true
-Dhawtio.config.cloneOnStartup=false
-Dhawtio.config.pullOnStartup=false

Problem solved.

Debug XA transactions in JBoss EAP 6

How do you find out if XA transactions are actually working and if there are problems, how do you find them? Apart from good tests the answer is to read the logs. So, how do you enable detailed logs for XA transactions in JBoss EAP 6? Fire up the cli (jbossctl.sh cli) and run:

/subsystem=logging/logger=com.arjuna.ats.jta:add(level=TRACE)
/subsystem=logging/file-handler=ARJUNA:add(file=
 {"path"=>"arjuna.log",
  "relative-to"=>"jboss.server.log.dir"})
/subsystem=logging/file-handler=ARJUNA:read-resource
/subsystem=logging/logger=com.arjuna.ats.jta:assign-handler(name="ARJUNA")

This will log most of the relevant XA-related events to a new log file, arjuna.log. The noise ratio is very high, so the file will grow quickly. Be sure to disable logging after a while:

/subsystem=logging/logger=com.arjuna.ats.jta:write-attribute(
  name="level", value="WARN")

Sometimes it may be necessary to log even more. These are recommended by Red Hat:

/subsystem=logging/logger=org.jboss.jca:add(level=TRACE)
/subsystem=logging/logger=org.jboss.as.connector:add(level=TRACE)

Obviously that will make the log files grow even faster, so take care! Don’t try this on a busy system in production.

JBoss EAP6 JGroups MPING fails with invalid argument

What to do if JBoss EAP6 fails to discover other cluster members with the following error from JGroups?

[org.jgroups.protocols.MPING] failed sending discovery request:
  java.io.IOException: Invalid argument
  at java.net.PlainDatagramSocketImpl.send(Native Method)
  at java.net.DatagramSocket.send(DatagramSocket.java:693)
  at org.jgroups.protocols.MPING.sendMcastDiscoveryRequest(MPING.java:300)

In my case the solution was simple. Add -Djava.net.preferIPv4Stack=true to jbossctl.sh. Apparently the multicast code doesn’t work with IPv6 on my Linux version.

Timeouts for Oracle XA datasources in JBoss EAP 6

The documentation for configuring datasources in JBoss EAP 6 is somewhat lacking when it comes to timeouts. Normally this is fine, but what if there are network issues? With the wrong timeout settings the application can hang until it is killed and restarted. With proper timeouts it can handle an outage and recover.

Here is an example:

<xa-datasource jndi-name="java:/AppDS" pool-name="AppDS">
  <xa-datasource-property name="URL">
    ${appdb.url}
  </xa-datasource-property>
  <xa-datasource-property name="nativeXA">true</xa-datasource-property>
  <xa-datasource-property name="ConnectionProperties">
    oracle.jdbc.ReadTimeout=330000
  </xa-datasource-property>
  <xa-datasource-class>
    oracle.jdbc.xa.client.OracleXADataSource
  </xa-datasource-class>
  <driver>oracle</driver>
  <security>
    <user-name>${appdb.user}</user-name>
    <password>${appdb.password}</password>
  </security>
  <xa-pool>
    <min-pool-size>${appdb.min.pool.size}</min-pool-size>
    <max-pool-size>${appdb.max.pool.size}</max-pool-size>
    <prefill>false</prefill>
    <use-strict-min>false</use-strict-min>
    <flush-strategy>FailingConnectionOnly</flush-strategy>
    <is-same-rm-override>false</is-same-rm-override>
    <no-tx-separate-pools/>
    <pad-xid>true</pad-xid>
    <wrap-xa-resource>true</wrap-xa-resource>
  </xa-pool>
  <validation>
    <valid-connection-checker class-name=
      "org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker"/>
    <stale-connection-checker class-name=
      "org.jboss.jca.adapters.jdbc.extensions.oracle.OracleStaleConnectionChecker"/>
    <exception-sorter class-name=
      "org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter"/>
  </validation>
  <timeout>
    <blocking-timeout-millis>60000</blocking-timeout-millis>
    <xa-resource-timeout>310</xa-resource-timeout>
    <query-timeout>300</query-timeout>
    <set-tx-query-timeout/>
  </timeout>
  <statement>
    <track-statements>false</track-statements>
  </statement>
  <recovery no-recovery="false">
    <recover-credential>
      <user-name>${appdb.user}</user-name>
      <password>${appdb.password}</password>
    </recover-credential>
  </recovery>
</xa-datasource>

The oracle.jdbc.ReadTimeout is essential. It sets the network timeout on the socket, making reads time out eventually in the face of a broken connection. The default TCP timeout for established connections is very long, so it is important to set a value. It should be larger than the transaction timeouts. The query timeout and tx-query-timeout both limit the time that individual statements can take. The query timeout is used when there is no transaction, otherwise the remaining time until transaction timeout is used.

Note that read timeout is in milliseconds, query timeout is in seconds.