Archive

Archive for the ‘Networking’ Category

VirtualBox intermittent network outages

For about a year I have had terrible problems with short but very frequent network outages between Linux guests running in VirtualBox and the world at large at a customer site. At home it works, but there the connection is lost and then restored every few minutes. Very frustrating. Today I finally found a solution. It appears that it is a known bug that goes way back, see ticket 13839. Sure enough, changing the virtual network card to PCnet-PCI II solved the issue! No more outages.

Advertisements
Categories: Networking

JBoss EAP6 JGroups MPING fails with invalid argument

What to do if JBoss EAP6 fails to discover other cluster members with the following error from JGroups?


[org.jgroups.protocols.MPING] failed sending discovery request:
  java.io.IOException: Invalid argument
  at java.net.PlainDatagramSocketImpl.send(Native Method)
  at java.net.DatagramSocket.send(DatagramSocket.java:693)
  at org.jgroups.protocols.MPING.sendMcastDiscoveryRequest(MPING.java:300)

In my case the solution was simple. Add -Djava.net.preferIPv4Stack=true to jbossctl.sh. Apparently the multicast code doesn’t work with IPv6 on my Linux version.

Categories: Java, Networking

Timeouts for Oracle XA datasources in JBoss EAP 6

The documentation for configuring datasources in JBoss EAP 6 is somewhat lacking when it comes to timeouts. Normally this is fine, but what if there are network issues? With the wrong timeout settings the application can hang until it is killed and restarted. With proper timeouts it can handle an outage and recover.

Here is an example:


<xa-datasource jndi-name="java:/AppDS" pool-name="AppDS">
  <xa-datasource-property name="URL">
    ${appdb.url}
  </xa-datasource-property>
  <xa-datasource-property name="nativeXA">true</xa-datasource-property>
  <xa-datasource-property name="ConnectionProperties">
    oracle.jdbc.ReadTimeout=330000
  </xa-datasource-property>
  <xa-datasource-class>
    oracle.jdbc.xa.client.OracleXADataSource
  </xa-datasource-class>
  <driver>oracle</driver>
  <security>
    <user-name>${appdb.user}</user-name>
    <password>${appdb.password}</password>
  </security>
  <xa-pool>
    <min-pool-size>${appdb.min.pool.size}</min-pool-size>
    <max-pool-size>${appdb.max.pool.size}</max-pool-size>
    <prefill>false</prefill>
    <use-strict-min>false</use-strict-min>
    <flush-strategy>FailingConnectionOnly</flush-strategy>
    <is-same-rm-override>false</is-same-rm-override>
    <no-tx-separate-pools/>
    <pad-xid>true</pad-xid>
    <wrap-xa-resource>true</wrap-xa-resource>
  </xa-pool>
  <validation>
    <valid-connection-checker class-name=
      "org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker"/>
    <stale-connection-checker class-name=
      "org.jboss.jca.adapters.jdbc.extensions.oracle.OracleStaleConnectionChecker"/>
    <exception-sorter class-name=
      "org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter"/>
  </validation>
  <timeout>
    <blocking-timeout-millis>60000</blocking-timeout-millis>
    <xa-resource-timeout>310</xa-resource-timeout>
    <query-timeout>300</query-timeout>
    <set-tx-query-timeout/>
  </timeout>
  <statement>
    <track-statements>false</track-statements>
  </statement>
  <recovery no-recovery="false">
    <recover-credential>
      <user-name>${appdb.user}</user-name>
      <password>${appdb.password}</password>
    </recover-credential>
  </recovery>
</xa-datasource>

The oracle.jdbc.ReadTimeout is essential. It sets the network timeout on the socket, making reads time out eventually in the face of a broken connection. The default TCP timeout for established connections is very long, so it is important to set a value. It should be larger than the transaction timeouts. The query timeout and tx-query-timeout both limit the time that individual statements can take. The query timeout is used when there is no transaction, otherwise the remaining time until transaction timeout is used.

Note that read timeout is in milliseconds, query timeout is in seconds.

Categories: Java, Networking, Oracle

Stupid network issues

Why do all network cards work with Windows, but not with Linux or BSD? Well, whatever. I spent a lot of time this summer trying to get two Internet connections from the same Linux box for high availability. No luck, worked for a while and then failed. Just found out that the built-in network interface from Sundance shuts down under moderate load on Linux. That explains why it didn’t work.

Now I want to use the same box for a DMZ. We have a smart TV and I don’t trust it at all. It needs Internet access, but I won’t allow it anywhere near anything important. I don’t trust my phone (unfortunately) and I certainly won’t trust a TV!

Fortunately the Linux box has one free PCI-e slot, so this should be easy. Just install a new network card and go! Well, not quite. I hate VIA on Linux. I bought two network cards as the first failed to deliver, but the final solution was to add some boot options to grub: pci=nomsi,noaer.

There are still warnings about interrupts that nobody cares about (irqpoll might help, but increases power consumption), but for now it seems to work.

Categories: Linux, Networking

Docker pitfalls for Internet-facing hosts

Planning to use Docker on an unprotected Internet-facing host? If so, don’t rush it. It works, but the default installation is probably not what you want.

By default Docker sets up iptables firewall rules for connections between the host and the containers. This is how it works on Ubuntu 14.04 and CentOS 7 and it is probably true for most distributions. The last thing I want on an Internet-facing host is something messing with the firewall!

What to do? A friend at Red Hat recommends overlay networking, for example with flannel as described here for Kubernetes with Fedora. It certainly seems like a much better (safer) option.

In summary, take care and make sure to test the firewall configuration not only when things are stable, but as containers are started and stoppped!

Categories: Linux, Networking

Routing for multiple uplinks with CentOS 7

How do you configure a Linux host to connect a local network to the Internet through multiple providers? Linux Advanced Routing and Traffic Control HOWTO is a good starting point. However, it only describes the commands, not where to put them.

With Red Hat and CentOS 7 the commands are split between several files. Following the HOWTO, assume that eth0 is connected to the LAN, eth1 to provider 1 and eth2 to provider 2.

Edit /etc/iproute2/rt_tables and add one line for T1 and one for T2. This creates two custom routing tables.

Optionally edit /etc/sysconfig/network and add NOZEROCONF=yes to get rid of the zero-config routing rules that are created by default.

In /etc/sysconfig/network-scripts/ there should be configuration files for all network cards: ifcfg-eth0, ifcfg-eth1 and ifcfg-eth2. Edit the files and remove GATEWAY, as we will add our gateways manually.

Create three new files named route-eth0, route-eth1 and route-eth2. They should contain the routing commands from the HOWTO. For example the route-eth1 could contain:


$P1_NET dev eth1 src $IP1 table T1
default via $P1 table T1
$P1_NET dev eth1 src $IP1

Optionally add a default as well (see the HOWTO for more advanced setups):


default via $P1

This covers the routes. Create rule-eth1 and rule-eth2 with the rules, for example rule-eth1:


from $IP1 table T1

Try it out and see how the routing tables change when a network interface is started or stopped.

This is just a starting point. My goal is to point out where the configuration in the HOWTO should go in Red Hat/CentOS 7, not to create a full-blown configuration. Good luck!

Categories: Linux, Networking

Dynamic ports in Windows

Recently we had some issues with a Windows 2008 server. The dynamic port range used by the server did not match the firewall rules. The dynamic port range is used when an application listens on port 0 in order to get an arbitrary free port. I turns out that it is quite easy to find and set the port range:

netsh int ipv4 show dynamicport tcp


The same syntax applies for IPv6 and UDP as well. To set the port range, use a similar command:

netsh int ipv4 set dynamicport tcp start=40000 num=1000


This sets the port range to 40000-41000. The smallest range of ports possible is 255 and the highest port number can’t exceed 65535.

Categories: Networking, Windows