Игорь Олемской — практические заметки по системному администрированию Linux CentOS

As many of you might have noticed from my previous GlusterFS blog post and my various tweets, I've been working with GlusterFS in production for my personal hosting needs for just over a month. I've also been learning quite a bit from some of the folks in the #gluster channel on Freenode. On a few occasions I've even been able to help out with some configuration problems from other users.

There has been quite a bit of interest in GlusterFS as of late and I've been inundated with questions from coworkers, other system administrators and developers. Most folks want to know about its reliability and performance in demanding production environments. I'll try to do my best to cover the big points in this post.

First off, here's now I'm using it in production: I have two web nodes that keep content in sync for various web sites. They each run a GlusterFS server instance and they also mount their GlusterFS share. I'm using the replicate translator to keep both web nodes in sync with client side replication.

Here are my impressions after a month:

I/O speed is often tied heavily to network throughput
This one may seem obvious, but it's not always true in all environments. If you deal with a lot of small files like I do, a 40mbit/sec link between the Xen guests is plenty. Adding extra throughput didn't add any performance to my servers. However, if you wrangle large files on your servers regularly, you may want to consider higher throughput links between your servers. I was able to push just under 900mbit/sec by using dd to create a large file within a GlusterFS mount.

Network and I/O latency are big factors for small file performance
If you have a busy network and the latency creeps up from time to time, you'll find that your small file performance will drop significantly (especially with the replicate translator). Without getting too nerdy (you're welcome to read the technical document on replication), replication is an intensive process. When a file is accessed, the client goes around to each server node to ensure that it not only has a copy of the file being read, but that it has the correct copy. If a server didn't save a copy of a file (due to disk failure or the server being offline when the file was written), it has to be synced across the network from one of the good nodes.

When you write files on replicated servers, the client has to roll through the same process first. Once that's done, it has to lock the file, write to the change log, then do the write operation, drop the change log entries, and then unlock the file. All of those operations must be done on all of the servers. High latency networks will wreak havoc on this process and cause it to take longer than it should.

It's quite obvious that if you have a fast, low-latency network between your servers, slow disks can still be a problem. If the client is waiting on the server nodes' disks to write data, the read and write performance will suffer. I've tested this in environments with fast networks and very busy RAID arrays. Even if the network was very underutilized, slow disks could cut performance drastically.

Monitoring GlusterFS isn't easy
When the client has communication problems with the server nodes, some weird things can happen. I've seen situations where the client loses connections to the servers (see the next section on reliability) and the client mount simply hangs. In other situations, the client has been knocked offline entirely and the process is missing from the process tree by the time I logged in. Your monitoring will need to ensure that the mount is active and is responding in a timely fashion.

There's a handy script which allows you to monitor GlusterFS mounts via nagios that Ian Rogers put together. Also, you can get some historical data with acrollet's munin-glusterfs plugin.

GlusterFS 3.x is pretty reliable
When I first started working with GlusterFS, I was using a version from the 2.x tree. The Fedora package maintainer hadn't updated the package in quite some time, but I figured it should work well enough for my needs. I found that the small file performance was lacking and the nodes often had communication issues when many files were being accessed or written simultaneously. This improved when I built my own RPMs of 3.0.4 (and later 3.0.5) and began using those instead.

I did some failure testing by hard cycling the server and client nodes and found some interesting results. First off, abruptly pulling clients had no effects on the other clients or the server nodes. The connection eventually timed out and the servers logged the timeout as expected.

Abruptly pulling servers led to some mixed results. In the 2.x branch, I saw client hangs and timeouts when I abruptly removed a server. This appears to be mostly corrected in the 3.x branch. If you're using replicate, it's important to keep in mind that the first server volume listed in your client's volume file is the one that will be coordinating the file and directory locking. Should that one fall offline quickly, you'll see a hiccup in performance for a brief moment and the next server will be used for coordinating the locking. When your original server comes back up, the locking coordination will shift back.

Conclusion
I'm really impressed with how much GlusterFS can do with the simplicity of how it operates. Sure, you can get better performance and more features (sometimes) from something like Lustre or GFS2, but the amount of work required to stand up that kind of cluster isn't trivial. GlusterFS really only requires that your kernel have FUSE support (it's been in mainline kernels since 2.6.14).

There are some things that GlusterFS really needs in order to succeed:

• Documentation — The current documentation is often out of date and confusing. I've even found instances where the documentation contradicts itself. While there are some good technical documents about the design of some translators, they really ought to do some more work there.
• Statistics gathering — It's very difficult to find out what GlusterFS is doing and where it can be optimized. Profiling your environment to find your bottlenecks is nearly impossible with the 2.x and 3.x branches. It doesn't make it easier when some of the performance translators actually decrease performance.
• Community involvement — This ties back into the documentation part a little, but it would be nice to see more participation from Gluster employees on IRC and via the mailing lists. They're a little better with mailing list responses than other companies I've seen, but there is still room for improvement.

If you're considering GlusterFS for your servers but you still have more questions, feel free to leave a comment or find me on Freenode (I'm 'rackerhacker').

One month with GlusterFS in production is a post from: Major Hayden's Racker Hacker blog.

c0b6ad7e-f251-11df-b20b-4040336e00ef

After I wrote a recent post on best practices for iptables, I noticed that I forgot to mention comments for iptables rules. They can be extremely handy if you have some obscure rules for odd situations.

To make an iptables rule with a comment, simply add on the following arguments to the rule:

-m comment --comment "limit ssh access"

Depending on your distribution, you may need to load the ipt_comment or xt_comment modules into your running kernel first.

A full iptables rule to limit ssh access would look something like this:

iptables -A INPUT -j DROP -p tcp --dport 22 -m comment --comment "limit ssh access"

Adding comments to iptables rules is a post from: Major Hayden's Racker Hacker blog.

c0b6ad7e-f251-11df-b20b-4040336e00ef

When you need to measure network throughput and capacity, I haven't found a simpler solution than iperf. There isn't much to say about the operation of iperf — it's a very simple application.

In short, iperf can be installed on two machines within your network. You'll run one as a server, and one as a client. On the server side, simply run:

iperf -s

On the client side, run:

iperf -c [server_ip]

The client side will try to shove TCP packets through the network interface as quickly as possible for a period of 10 seconds by default. Once that's complete, you'll see a report on the server and client that will look like this:

$ iperf -c 192.168.10.10
------------------------------------------------------------
Client connecting to 192.168.10.10, TCP port 5001
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.30 port 53345 connected with 192.168.10.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  37.9 MBytes  31.8 Mbits/sec

The previous test was run over an 802.11n network between a wired and wireless device. The typical downlink for an 802.11n network is about 40Mbit/s, so it's obvious that my home network could use an adjustment.

You can also run bidirectional tests from the client either at the same time (-d flag) or one after the other (-r flag). The server side will keep running until you stop it, so you can leave it running and run tests from multiple locations over time. You can daemonize the server end if that makes things easier.

For the full list of options, refer to iperf's man page.

©2010 Racker Hacker. All Rights Reserved.

.

Regardless of the type of hosting you're using — dedicated or cloud — it's important to take network interface security seriously. Most often, threats from the internet are the only ones mentioned. However, if you share a private network with other customers, you have just as much risk on that interface.

Many cloud providers allow you access to a private network environment where you can exchange data with other instances or other services offered by the provider. The convenience of this access comes with a price: other instances can access your instance on the private network just as easily as they could on the public interface.

Here are some security tips for your private interfaces:

Disable the private interface
This one is pretty simple. If you have only one instance or server, and you don't need to communicate privately with any other instances, just disable the interface. Remember to configure your networking scripts to leave the interface disabled after reboots.

Use packet filtering
The actual mechanism will vary based on your operating system, but filtering packets is the one of the simplest ways to secure your private interface. You can take some different approaches with them, but I find the easiest method is to allow access from your other instances and reject all other traffic.

For additional security, you can limit access based on ports as well as source IP addresses. This could prevent an attacker from having easy access to your other instances if they're able to break into one of them.

Configure your daemons to listen on the appropriate interfaces
If there are services that don't need to be listening on the private network, don't allow them to listen on your private interface. For example, MySQL might need to listen on the private interface so the web server can talk to it, but apache won't need to listen on the private interface. This reduces the profile of your instance on the private network and makes it a less likely target for attack.

Use hosts.allow and hosts.deny
Many new systems administrators forget about how handy tcpwrappers can be for limiting access. If your firewall is down in error, host.allow and hosts.deny could be an extra layer of protection. It's important to ensure that the daemons you are attempting to control are build with tcpwrappers support. Daemons like sshd support it, but apache and MySQL do not.

Encrypt all traffic on the private network
Just because it's called a «private» network doesn't mean that your traffic can traverse the network privately. You should always err on the side of caution and encrypt all traffic traversing the private network. You can use ssh tunnels, stunnel, or the built-in SSL features found in most daemons.

This also brings up an important point: you should know how your provider's private network works. Are there safeguards to prevent sniffing? Could someone else possibly ARP spoof your instance's private IP addresses? Is your private network's subnet shared among many customers?

With all of that said, it's also very important to have proper change control policies so that administrators working after you are fully aware of the security measures in place and why they are important. This will ensure that all of the administrators on your instances will understand the security of the system and they should be able to make sensible adjustments later for future functionality.

©2010 Racker Hacker. All Rights Reserved.

.