Archive

Archive for the ‘Linux’ Category

Compression ratio and time for common tools

One application I’m working with produces horrendous amounts of log files. They are so large that the process of compressing them and moving them to another server actually has a performance impact on the system in spite of nice and ionice. I decided to compare the most common compression tools. At home I’m using bzip2, as I want the best possible compression ratio. Would that be a good idea?

Compression ratio and time

As the table shows bzip2 is frightfully expensive, but it does produce small files. In addition it makes a big difference if gzip is told to optimize for speed (1) or compression (9), or if it runs at the default level (6). Compress is basically an outdated tool and was included as it is sometimes quoted as being faster than gzip. Here that was not the case.

Categories: Linux

Stupid network issues

Why do all network cards work with Windows, but not with Linux or BSD? Well, whatever. I spent a lot of time this summer trying to get two Internet connections from the same Linux box for high availability. No luck, worked for a while and then failed. Just found out that the built-in network interface from Sundance shuts down under moderate load on Linux. That explains why it didn’t work.

Now I want to use the same box for a DMZ. We have a smart TV and I don’t trust it at all. It needs Internet access, but I won’t allow it anywhere near anything important. I don’t trust my phone (unfortunately) and I certainly won’t trust a TV!

Fortunately the Linux box has one free PCI-e slot, so this should be easy. Just install a new network card and go! Well, not quite. I hate VIA on Linux. I bought two network cards as the first failed to deliver, but the final solution was to add some boot options to grub: pci=nomsi,noaer.

There are still warnings about interrupts that nobody cares about (irqpoll might help, but increases power consumption), but for now it seems to work.

Categories: Linux, Networking

Take care with BTRFS

I have long been an avid fan of ZFS, but though I have been running it on Linux for quite some time I’m not ready to trust it there. The upgrades can be painfull. No lost data so far, but several temporary outages with down time. Perhaps BTRFS is ready for prime time? In short no. It seems very promising, but Red Hat still has reservations. I have tested it with non-essential data. The basics are there and there are many cool features, but when I started to use some of them the file system suddenly turned read only. Why? Tried again, several times. Finally found it – old and fixed bug, but the fix is in the 4.x kernels. Way ahead of Red Hat/CentOS 7.
Of course, a read-only file system is much better than a corrupted one, so that was good. Even so I’ll wait a few more years before I trust anything vital to BTRFS, just in case. Having said that the long-term promise is there, so in five years I bet my servers will be running on BTRFS (unless the ZFS on Linux story changes)!

Categories: Linux

Sorting and rotating photos

I have long used my own Java program for sorting and adjusting photos, but now I have found a better solution: jhead for Linux. Combining it with Dropbox and a cron job I can get my uploaded images, move them out of Dropbox (to save space online, I don’t pay and I like to keep my data close), rotate if needed and sort into folders by year, month and day. The jhead command is simple:


jhead -n%Y/%m/%d/%f -autorot -exonly *.*

This sorts photos by the date taken and rotates them as needed. Images without creation date are left in the main folder for manual sorting.

Categories: Linux

Windows-like keyboard shortcuts in Linux Eclipse

I have worked with Windows as my main development platform since Windows 3.1 and the keyboard shortcuts are hardwired by now. Unfortunately Microsoft has failed utterly. In my opinion Windows 7 is the pinnacle from a usability standpoint, Windows 8 was a disaster and Windows 10 is not that much of an improvement. It is time to move on, in particular as Microsoft seems determined to spy on customers.

Java is cross-platform, so I can use Linux. It works well and bash is great, but the keyboard shortcuts are plain wrong. Fortunately there is a solution. For example, to expand a treeview in Eclipse StackOverflow recommends this for GTK 2:


binding "gtk-binding-tree-view" {
    bind "j"        { "move-cursor" (display-lines, 1) }
    bind "k"        { "move-cursor" (display-lines, -1) }
    bind "h"        { "expand-collapse-cursor-row" (1,0,0) }
    bind "l"        { "expand-collapse-cursor-row" (1,1,0) }
    bind "o"        { "move-cursor" (pages, 1) }
    bind "u"        { "move-cursor" (pages, -1) }
    bind "g"        { "move-cursor" (buffer-ends, -1) }
    bind "y"        { "move-cursor" (buffer-ends, 1) }
    bind "p"        { "select-cursor-parent" () }
    bind "Left"     { "expand-collapse-cursor-row" (0,0,0) }
    bind "Right"    { "expand-collapse-cursor-row" (0,1,0) }
    bind "semicolon" { "expand-collapse-cursor-row" (0,1,1) }
    bind "slash"    { "start-interactive-search" () }
}
class "GtkTreeView" binding "gtk-binding-tree-view"

And this for GTK 3:


@binding-set MyTreeViewBinding {
    bind "Left"     { "select-cursor-parent" ()
                      "expand-collapse-cursor-row" (0,0,0) };
    bind "Right"    { "expand-collapse-cursor-row" (0,1,0) };
}
GtkTreeView {
    gtk-key-bindings: MyTreeViewBinding;
}

With Ubuntu the files to edit are found below /usr/share/themes.

Categories: Java, Linux, Windows

Docker pitfalls for Internet-facing hosts

Planning to use Docker on an unprotected Internet-facing host? If so, don’t rush it. It works, but the default installation is probably not what you want.

By default Docker sets up iptables firewall rules for connections between the host and the containers. This is how it works on Ubuntu 14.04 and CentOS 7 and it is probably true for most distributions. The last thing I want on an Internet-facing host is something messing with the firewall!

What to do? A friend at Red Hat recommends overlay networking, for example with flannel as described here for Kubernetes with Fedora. It certainly seems like a much better (safer) option.

In summary, take care and make sure to test the firewall configuration not only when things are stable, but as containers are started and stoppped!

Categories: Linux, Networking

Routing for multiple uplinks with CentOS 7

How do you configure a Linux host to connect a local network to the Internet through multiple providers? Linux Advanced Routing and Traffic Control HOWTO is a good starting point. However, it only describes the commands, not where to put them.

With Red Hat and CentOS 7 the commands are split between several files. Following the HOWTO, assume that eth0 is connected to the LAN, eth1 to provider 1 and eth2 to provider 2.

Edit /etc/iproute2/rt_tables and add one line for T1 and one for T2. This creates two custom routing tables.

Optionally edit /etc/sysconfig/network and add NOZEROCONF=yes to get rid of the zero-config routing rules that are created by default.

In /etc/sysconfig/network-scripts/ there should be configuration files for all network cards: ifcfg-eth0, ifcfg-eth1 and ifcfg-eth2. Edit the files and remove GATEWAY, as we will add our gateways manually.

Create three new files named route-eth0, route-eth1 and route-eth2. They should contain the routing commands from the HOWTO. For example the route-eth1 could contain:


$P1_NET dev eth1 src $IP1 table T1
default via $P1 table T1
$P1_NET dev eth1 src $IP1

Optionally add a default as well (see the HOWTO for more advanced setups):


default via $P1

This covers the routes. Create rule-eth1 and rule-eth2 with the rules, for example rule-eth1:


from $IP1 table T1

Try it out and see how the routing tables change when a network interface is started or stopped.

This is just a starting point. My goal is to point out where the configuration in the HOWTO should go in Red Hat/CentOS 7, not to create a full-blown configuration. Good luck!

Categories: Linux, Networking