The discussions about the , or «pvops», support in upstream kernels at last month really piqued my interest.
Quite a few distribution maintainers have gone to great lengths to keep Xen domU support in their kernels and it's been an uphill battle. Some kernels, such as Ubuntu's kernels, have patches from 2.6.18 dragged forward into 2.6.32 and even 2.6.33. It certainly can't be enjoyable to keep dragging those patches forward into new kernel trees.
The paravirt_ops support for Xen guests was added in 2.6.23 and continues to be included and improved in the latest kernel trees. However, there are two significant problems with these new kernels if you're trying to work with legacy environments:
the console is on hvc0, not tty1
block devices are now /dev/xvdX rather than /dev/sdX
If you only have a few guests, these changes are generally pretty easy. Switching the console just requires some changes to your inittab or upstart configurations. Changing the block device names requires changes to the guest's Xen configuration file and /etc/fstab within the guest itself.
Considering the I work with daily at Rackspace, changing the guest configuration is definitely not an option. I needed a way to keep the console and block devices unchanged so that our customers could have a consistent experience on our infrastructure.
Luckily, offered to pitch in and a solution became apparent. Through some , the legacy console and block device support was available in the latest 2.6.32 version (2.6.32.12 as of this post's writing).
So far, I've tested x86_64 and i386 versions of 2.6.32.12 with the console and block device patches. It's gone through its paces on Xen 3.0.3, 3.1.2, 3.3.0 and 3.4.2. All revisions of Fedora, CentOS, Ubuntu, Debian, Gentoo and Arch made within the last two years are working well with the new kernels.
I flew out to San Jose, California with two for the Xen Summit at AMD's headquarters. The first day of the two-day conference was very informative. Lots of people asked for some highlights, so I'll provide those here.
and started off the day with an overview of Xen's current roadmap. They talked about their largest products and where they're going.
Open source Xen development is going strong with the recent release of Xen 4.0 and there are some great features in the works for Xen 4.1. There should be a new credit scheduler called «» in that release. Also, the project is improving and the feature set is growing. The was recently integrated with XCP.
The caught my attention quickly as it allows a user to run a very thin hypervisor on a client machine, such as a laptop or desktop computer, and then run multiple operating systems on top of that hypervisor. This would reduce the need for products like VMWare Fusion, VirtualBox or Parallels Desktop.
Still not impressed? Watch Ian Pratt do a quick demonstration of XCI:
Tom Woller from AMD talked about some of the new hardware enhancements that allow Xen to virtualize devices more efficiently. While much of the low-level hardware discussion was a little over my head (I'm terrible with hardware), the improvements make sense and should improve the relationship with Xen.
from the University of California San Diego talked about SleepServers. He's done some impressive work with Xen to reduce the energy usage of PC's left on when their users are no longer in the office.
Jonathan Ludlum covered the roadmap for Xen Cloud Platform in greater detail. This is definitely going to be a product to watch as it has a tight integration between various products, including XAPI, Open vSwitch, and the Xen hypervisor itself. It currently runs a CentOS-based userland and Jonathan said there are no plans to change it.
Later in the afternoon, from Oracle talked about the current status of the Xen kernels. The PVOps kernels are used with Xen 4.0 for the dom0 (and soon for the 3.4.x branch as well). The domU support has been present in PVOps code in the upstream kernels for some time. Many distribution vendors are making one kernel for bare metal and virtualized instances by simply adding PVOps support when they build their kernels. This eliminates the need for the traditional «kernel-xen» kernels that needed to be loaded for virtualized instances to work properly. helped Konrad answer some questions from the audience and their presentation was one of the most informative ones of the day.
talked about the new credit2 scheduler that is due to be released with Xen 4.1 later this year. He found that the current scheduler negatively affects VM's running applications that have low CPU requirements but are affected greatly by higher latency. He tested this with a VM that was playing back an audio stream. When the rest of the VM's on the server used a lot of CPU time, the audio skipped many times. The credit2 scheduler allows for these low-CPU, latency-sensitive applications to keep running as expected without interruptions. I'm looking forward to testing this out later this year.
The last presentation of the day was from . He covered the work being done to simplify how applications interact with Xen. The new library, libxenlight, strives to be a common layer between client applications and the Xen hypervisor.
The day wrapped up with a great party at Dave and Buster's. We had the opportunity to meet many people from Citrix as well as other people working in the virtualization space. Overall, the first day was very informative and I'm eager to hear some of the presentations scheduled for the second day.
Anyone who has used before has locked themselves out of a remote server at least once. It's easily avoided, but often forgotten. Lots of people have asked me for a list of best practices for iptables firewalls and I certainly hope this post helps.
Understand how iptables operates Before you can begin using iptables, you need to fully understand how it matches packets with chains and rules. There is a that will make it easier to understand. It's imperative to remember that iptables rules are read top-down until a matching rule is found. If no matching rule is found, the default policy of the chain will be applied (more on that in a moment).
Don't set the default policy to DROP All iptables chains have a default policy setting. If a packet doesn't match any of the rules in a relevant chain, it will match the default policy and will be handled accordingly. I've seen quite a few users set their default policy to DROP, and this can bring about some unintended consequences.
Consider a situation where your INPUT chain contains quite a few rules allowing traffic, and you've set the default policy to DROP. Later on, another administrator logs into the server and flushes the rules (which isn't a good practice, either). I've met quite a few good systems administrators who are unaware of the default policy for iptables chains. Your server will be completely inaccessible immediately. All of the packets will be dropped since they match the default policy in the chain.
Instead of using the default policy, I normally recommend making an explicit DROP/REJECT rule at the bottom of your chain that matches everything. You can leave your default policy set to ACCEPT and this should reduce the chance of blocking all access to the server.
Don't blindly flush iptables rules Before running iptables -F, always check each chain's default policy. If the INPUT chain is set to DROP, you'll need to set it to ACCEPT if you want to access the server after the rules are flushed. Also, consider the security implications of your network when you clear the rules. Your services will be completely exposed and any masquerading or NAT rules will be removed.
Remember localhost Lots of applications require access to the lo interface. Ensure that you set up your rules carefully so that the lo interface is not disturbed.
Split complicated rule groups into separate chains Even if you're the only systems administrator for your particular network, it's important to keep your iptables rules manageable. If you have a certain subset of rules that may be a little complicated, consider breaking them out into their own chain. You can just add in a jump to that chain from your default set of chains.
Use REJECT until you know your rules are working properly When you're writing iptables rules, you'll probably be testing them pretty often. One way to speed up that process is to use the REJECT target rather than DROP. You'll get an immediate rejection of your traffic (a TCP reset) instead of wondering if your packet is being dropped or if it's making it to your server at all. Once you're done with your testing, you can flip the rules from REJECT to DROP if you prefer.
For those folks working towards their RHCE, this is a huge help during the test. When you're nervous and in a hurry, the immediate packet rejection is a welcomed sight.
Be stringent with your rules Try to make your rules as specific as possible for your needs. For example, I like to allow ICMP pings on my servers so that I can run network tests against them. I could easily toss a rule into my INPUT chain that looks like this:
iptables -A INPUT -p icmp -m icmp -j ACCEPT
However, I don't want to simply allow all ICMP traffic. There have been some ICMP flaws from time to time and I'd rather keep as low of a profile as possible. There are , but I only want to allow echo requests:
This will allow echo requests (standard ICMP pings), but it won't explicitly allow any other ICMP traffic to pass through the firewall.
Use comments for obscure rules If you have rules to cover edge cases that other administrators might not understand, consider using iptables comments by adding the following arguments to your rules:
-m comment --comment "limit ssh access"
The comments will appear in the iptables output if you list the current rules. They will also appear in your saved iptables rules.
Always save your rules Most distributions offer some way to save your iptables rules so that they persist through reboots. Red Hat-based distributions offer /etc/init.d/iptables save, but Debian and Ubuntu require some . An errant reboot would easily take out your unsaved rules, so save them often.
Fedora 13 has quite a few changes related to , and one of the biggest ones is how terminals are configured. Most distributions tuck the tty configuration away in /etc/inittab, /etc/event.d/ or /etc/init/. If you want to adjust the number of tty's in Fedora 13, you'll need to look in /etc/sysconfig/init:
# color => new RH6.0 bootup# verbose => old-style bootup# anything else => new style bootup without ANSI colors or positioningBOOTUP=color
# column to start "[ OK ]" label in RES_COL=60# terminal sequence to move to that column. You could change this# to something like "tput hpa ${RES_COL}" if your terminal supports itMOVE_TO_COL="echo -en \\033[${RES_COL}G"# terminal sequence to set color to a 'success' color (currently: green)SETCOLOR_SUCCESS="echo -en \\033[0;32m"# terminal sequence to set color to a 'failure' color (currently: red)SETCOLOR_FAILURE="echo -en \\033[0;31m"# terminal sequence to set color to a 'warning' color (currently: yellow)SETCOLOR_WARNING="echo -en \\033[0;33m"# terminal sequence to reset to the default color.SETCOLOR_NORMAL="echo -en \\033[0;39m"# default kernel loglevel on boot (syslog will reset this)LOGLEVEL=3# Set to anything other than 'no' to allow hotkey interactive startup...PROMPT=yes# Set to 'yes' to allow probing for devices with swap signaturesAUTOSWAP=no
# What ttys should gettys be started on?ACTIVE_CONSOLES=/dev/tty[1-6]
The very last line controls the number of tty's that are kept alive on your system. If you need more tty's, simply increase the 6 to a higher number. If you only want one terminal (which is usually what I want in Xen), just make this adjustment:
# What ttys should gettys be started on?ACTIVE_CONSOLES=/dev/tty1
A normal telinit q doesn't seem to adjust the terminals on the fly as it did before upstart was involved. I'm not sure if this is a bug or an intended feature. Either way, a reboot solves the problem and you should see the changes afterwards.
When you need to measure network throughput and capacity, I haven't found a simpler solution than . There isn't about the operation of iperf — it's a very simple application.
In short, iperf can be installed on two machines within your network. You'll run one as a server, and one as a client. On the server side, simply run:
iperf -s
On the client side, run:
iperf -c [server_ip]
The client side will try to shove TCP packets through the network interface as quickly as possible for a period of 10 seconds by default. Once that's complete, you'll see a report on the server and client that will look like this:
$ iperf -c 192.168.10.10
------------------------------------------------------------
Client connecting to 192.168.10.10, TCP port 5001
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.10.30 port 53345 connected with 192.168.10.10 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 37.9 MBytes 31.8 Mbits/sec
The previous test was run over an 802.11n network between a wired and wireless device. The for an 802.11n network is about 40Mbit/s, so it's obvious that my home network could use an adjustment.
You can also run bidirectional tests from the client either at the same time (-d flag) or one after the other (-r flag). The server side will keep running until you stop it, so you can leave it running and run tests from multiple locations over time. You can daemonize the server end if that makes things easier.
Sending signals to processes using <a href="http://en.wikipedia.org/wiki/Kill_(command)">kill</a> on a Unix system is not a new topic for most systems administrators, but I've been asked many times about the difference between kill and kill -9.
Anytime you use kill on a process, you're actually sending the process a signal (in almost all situations — I'll get into that soon). Standard C applications have a that contains the steps that the process should follow if it receives a particular signal. You can get an entire list of the available signals on your system by checking the man page for kill.
Consider a command like this:
kill 2563
This would send a signal called to the process. Once the process receives the notice, a few different things can happen:
the process may stop immediately
the process may stop after a short delay after cleaning up resources
the process may keep running indefinitely
The application can determine what it wants to do once a SIGTERM is received. While most applications will clean up their resources and stop, some may not. An application may be configured to do something completely different when a SIGTERM is received. Also, if the application is in a bad state, such as waiting for disk I/O, it may not be able to act on the signal that was sent.
Most system administrators will usually resort to the more abrupt signal when an application doesn't respond to a SIGTERM:
kill -9 2563
The -9 tells the kill command that you want to send signal #9, which is called . With a name like that, it's obvious that this signal carries a little more weight.
Although SIGKILL is defined in the same signal header file as SIGTERM, it cannot be ignored by the process. In fact, the process isn't even made aware of the SIGKILL signal since the signal goes straight to the kernel init. At that point, init will stop the process. The process never gets the opportunity to catch the signal and act on it.
However, the kernel may not be able to successfully kill the process in some situations. If the process is waiting for network or disk I/O, the kernel won't be able to stop it. and processes caught in an cannot be stopped by the kernel, either. A reboot is required to clear those processes from the system.
My synergy setup at work is relatively simple. I have a MacBook Pro running Snow Leopard that acts as a synergy server and a desktop running Fedora 12 as a synergy client. On the Mac, I use SynergyKM to manage the synergy server. The Fedora box uses for starting synergy at the login screen and in GNOME.
I kept having an issue where the shift key would become stuck regardless of the settings I set for the client or server. The halfDuplexCapsLock had no effect. After installing xkeycaps, I found that both shift keys were getting stuck if I brought the mouse back and forth between Mac and Fedora twice.
I decided to run a test. I started the client with the debug argument and moved the mouse to my Fedora box. At that point, I pressed the letter 'a' and saw:
DEBUG1: CXWindowsKeyState.cpp,195: 032 (00000000) up
DEBUG1: CXWindowsKeyState.cpp,195: 03e (00000000) up
DEBUG1: CXWindowsKeyState.cpp,195: 026 (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195: 032 (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195: 03e (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195: 026 (00000000) up
I brought the mouse back to the Mac and then back to Fedora. I pressed 'a' again and saw:
DEBUG1: CXWindowsKeyState.cpp,195: 026 (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195: 026 (00000000) up
DEBUG1: CXWindowsKeyState.cpp,195: 026 (00000000) down
DEBUG1: CXWindowsKeyState.cpp,195: 026 (00000000) up
After dumping the keyboard layout with xmodmap I found the keys that corresponded with the key numbers:
032 — Left shift
03e — Right shift
026 — a
If I tapped the left shift, I could clear the key press, but I couldn't clear the right shift key (it was stuck down according to Fedora's X server). When I hooked up a physical keyboard and mouse, I was able to use them normally without any keybinding problems.
The root cause: When synergy started in /etc/gdm/PreSession/Default after the gdm login, the keyboard layout wasn't set up properly. The X server was setting up the keyboard layout later in the startup process and this confusion caused the shift keys to get stuck. Fedora 12 uses evdev to probe for keyboards during X's startup and eventually settles on a default layout if none are explicitly defined.
The fix: I added the synergy startup to the GNOME startup items and it works flawlessly.
Regardless of the type of hosting you're using — dedicated or cloud — it's important to take network interface security seriously. Most often, threats from the internet are the only ones mentioned. However, if you share a private network with other customers, you have just as much risk on that interface.
Many cloud providers allow you access to a private network environment where you can exchange data with other instances or other services offered by the provider. The convenience of this access comes with a price: other instances can access your instance on the private network just as easily as they could on the public interface.
Here are some security tips for your private interfaces:
Disable the private interface
This one is pretty simple. If you have only one instance or server, and you don't need to communicate privately with any other instances, just disable the interface. Remember to configure your networking scripts to leave the interface disabled after reboots.
Use packet filtering
The actual mechanism will vary based on your operating system, but filtering packets is the one of the simplest ways to secure your private interface. You can take some different approaches with them, but I find the easiest method is to allow access from your other instances and reject all other traffic.
For additional security, you can limit access based on ports as well as source IP addresses. This could prevent an attacker from having easy access to your other instances if they're able to break into one of them.
Configure your daemons to listen on the appropriate interfaces
If there are services that don't need to be listening on the private network, don't allow them to listen on your private interface. For example, MySQL might need to listen on the private interface so the web server can talk to it, but apache won't need to listen on the private interface. This reduces the profile of your instance on the private network and makes it a less likely target for attack.
Use hosts.allow and hosts.deny
Many new systems administrators forget about how handy tcpwrappers can be for limiting access. If your firewall is down in error, host.allow and hosts.deny could be an extra layer of protection. It's important to ensure that the daemons you are attempting to control are build with tcpwrappers support. Daemons like sshd support it, but apache and MySQL do not.
Encrypt all traffic on the private network
Just because it's called a «private» network doesn't mean that your traffic can traverse the network privately. You should always err on the side of caution and encrypt all traffic traversing the private network. You can use ssh tunnels, stunnel, or the built-in SSL features found in most daemons.
This also brings up an important point: you should know how your provider's private network works. Are there safeguards to prevent sniffing? Could someone else possibly ARP spoof your instance's private IP addresses? Is your private network's subnet shared among many customers?
With all of that said, it's also very important to have proper change control policies so that administrators working after you are fully aware of the security measures in place and why they are important. This will ensure that all of the administrators on your instances will understand the security of the system and they should be able to make sensible adjustments later for future functionality.
Earlier this year, to encourage systems administrators to refine their troubleshooting abilities. This is the second post in that series.
Almost every system administrator has found themselves in a situation where they're confronted with a server which has a problem. However, if you're not the primary administrator for the server, you may not always know what has changed recently or you may not be aware of changes in the server's environment. In these situations, if the fix isn't obvious, try going through these steps:
Localize the problem to a specific daemon or service
In the case of a problem where a website isn't loading properly, is it a problem with the web server itself? Could something other than the actual web server daemon be having an issue?
As an example, consider a ruby on rails application which runs through apache's mod_proxy_balancer and queries data from MySQL. If any of those individual puzzle pieces were not functioning correctly, you'd get a different result. A downed MySQL instance could make the application throw errors or appear to be unresponsive. If the mongrel cluster had failed, apache might be returning internal server errors. Your browser might return a connection refused if apache was down. These are all relatively easy to determine.
What if you are unable to determine which daemon is causing the problem?
If it's broken, break it a little more
Let's say that you've reviewed the process list and all of the appropriate daemons appear to be running. However, the website is still not loading properly. What do you do? Bring down a service and try again. Did something change? Did a new error appear? If not, bring that daemon back up and try taking down one of the other ones.
I've also had some good results by making small adjustments in the web server's configuration file. If you have a virtual host that isn't returning the correct data, try commenting it out temporarily. For rewrite rules, try removing them temporarily or strip them down to a more basic form. Test again, and then begin adding lines back incrementally. As much as a single period or quotation mark can derail a perfectly good set of rewrite rules.
In short — try to think outside the box when you're troubleshooting a difficult issue on an unfamiliar system. Always remember to back up your configurations before making changes and ensure your daemons will start properly if you bring them down.