Today, on my 28th birthday, I'm finally delivering on a promise to my readers which I made about two months ago. I've on how to host a web application redundantly in a cloud environment. While it's still a bit of a rough draft, it should be a good starting point for those who haven't worked in virtualized environments before. Also, it may show some of the more experienced systems administrators a new way to do things.
The guide:
As always, if you find anything in the guide that needs improvement, I'm all ears.
I am still at the OpenVZ booth at LinuxTag 2010 in Berlin. At least two people asked me about the status of OpenVZ kernel for the upcoming Debian Squeeze. Specifically, they said, there is no openvz kernel in «testing» repository (i.e. what will become Squeeze when it will be released). My guess is some more people interesting in that, so here's the public answer.
We are working pretty close with the Debian kernel team, you can see some traces of that on either or mailing lists. Specifically, we work together to bring good quality OpenVZ kernel to Squeeze, and this was one of the main reasons for us to port to 2.6.32.
But yesterday we tried to on packages.debian.org and it gave us no results for testing. I then emailed Max Attems (who maintains our kernels in Debian) and this is his response:
it should be there now, the switch to libata did uphold testing transition of linux-2.6 for quite some time, so testing had an outdated linux-2.6 for quite some while
Indeed, the kernel is now there. So yes, Squeeze will have OpenVZ kernel, and I guess it can also be used by people who switched to Ubuntu 10.4.
The discussions about the , or «pvops», support in upstream kernels at last month really piqued my interest.
Quite a few distribution maintainers have gone to great lengths to keep Xen domU support in their kernels and it's been an uphill battle. Some kernels, such as Ubuntu's kernels, have patches from 2.6.18 dragged forward into 2.6.32 and even 2.6.33. It certainly can't be enjoyable to keep dragging those patches forward into new kernel trees.
The paravirt_ops support for Xen guests was added in 2.6.23 and continues to be included and improved in the latest kernel trees. However, there are two significant problems with these new kernels if you're trying to work with legacy environments:
the console is on hvc0, not tty1
block devices are now /dev/xvdX rather than /dev/sdX
If you only have a few guests, these changes are generally pretty easy. Switching the console just requires some changes to your inittab or upstart configurations. Changing the block device names requires changes to the guest's Xen configuration file and /etc/fstab within the guest itself.
Considering the I work with daily at Rackspace, changing the guest configuration is definitely not an option. I needed a way to keep the console and block devices unchanged so that our customers could have a consistent experience on our infrastructure.
Luckily, offered to pitch in and a solution became apparent. Through some , the legacy console and block device support was available in the latest 2.6.32 version (2.6.32.12 as of this post's writing).
So far, I've tested x86_64 and i386 versions of 2.6.32.12 with the console and block device patches. It's gone through its paces on Xen 3.0.3, 3.1.2, 3.3.0 and 3.4.2. All revisions of Fedora, CentOS, Ubuntu, Debian, Gentoo and Arch made within the last two years are working well with the new kernels.
We have just announced that we stop making new releases for OpenVZ kernel branches 2.6.24, 2.6.26, and 2.6.18. So, from now on we only have 2.6.27, 2.6.32, RHEL4-2.6.9 and RHEL5-2.6.18. Removing the number of parallel kernel branches we have to maintain really helps to concentrate on supporting the remaining ones and moving to mainline. I hope that doesn't affect anyone too much — from where I stand most users run either stable (i.e. RHEL5-2.6.18) or bleeding edge (2.6.32, before it used to be 2.6.27). In any case, we are not dropping support for vendor kernels, such as OpenVZ kernels in Debian and Ubuntu — those are still supported from us for the lifetime of the distributions that carry it, we will help with OpenVZ bugs in those kernels through the .
On the remaining branches. Last Thursday we did an update to 2.6.32 kernel fixing some nasty bugs found in the first public version, and today we updated 2.6.27 kernel as well. Speaking of 2.6.27, it will eventually be dropped as well, but we will keep maintaining it for at least a few more months.
Stable kernel update (RHEL5.5 based, 028stab069...) is currently in testing, but don't expect it to be released real soon now — previous experience tells us that .y updates are not that easy. We also anticipate to open RHEL6-2.6.32 branch soon, since Red Hat already shooted a beta of their upcoming release.
Happy New Year! I certainly hope it's a great one for you, your family, and your business. As the new year begins, I figured it would be a good time to sit down and answer a question that I hear very often:
How do I become a better systems administrator?
The best way to become a better systems administrator is to fully understand the theory of what's happening in your server's environment.
What do I mean by that? Learn why things aren't happening as you expected and think about all of the factors that could possibly be involved. Instead of thinking purely about cause and effect, you'll find it much easier and rewarding to consider everything inside and outside your environment before you make any changes.
This still may be a little difficult to fully understand, so he's an example. Let's say you're handling an issue where a customer can't reach a website hosted on their server. When you ask them for more details, they might give you the dreaded reply: «It's not coming up.» Start by making a mental list of the problems that are easiest to check:
Is the web server daemon running?
If a database server is being used, is it running and accessible?
Is there a software/hardware firewall blocking port 80?
Is a script stuck on the server tying up resources?
Could there be a DNS resolution problem?
Is the server up?
Did a switch fail?
Is the server's hard disk out of space?
Can the customer reach other websites like Google or Yahoo?
If SELinux is involved, have the appropriate contexts been set?
Could the site be a target of a denial of service attack?
Has the server reached its connection tracking limit?
Of course, this is a relatively short list, but these are all easy to check. If you're thinking about cause and effect, you might only consider the web server daemon and some basic network issues. By considering all of the other factors that may be related, you've ensured that all of the basics are covered before you consider more complex problems.
Most systems administrators have taken an error message and tossed it in en masse into Google before. Occasionally, no results will appear for the search. If you find yourself in this situation, try to understand the individual parts of the error message. Work outward from what you know already. You should know which daemon said it, and you may have an idea of what the application was doing when the error occurred. Take time to consider what the daemon is trying to tell you within the context of what it was doing at the time.
One of the easiest ways to force yourself to be immersed into this way of thinking is to host applications for non-technical people. You'll find that many customers want things done differently, and they're all at different levels of technical aptitude. Some may find it a frustrating experience at first, but you'll think yourself later. It will force you to consider all aspects of how a server operates since you might not always know what's happening within a customer's application.
As always, if you find yourself stumbling, remember to ask your peers and colleagues. Even if they haven't seen the particular issue, they will probably be able to guide you closer to the solution you seek.
Если ваш сервер под ОС CentOS 5 начинает тормозить, еле откликаться и т.п., при этом загрузка процессора и Load Average в норме — наиболее вероятно, что вам нужно обновить сетевой драйвер.
Данная проблема неоднократно замечена при аренде немецких серверов (hetzner.de, тариф EQ).
Проверить, в этом ли проблема, просто. Запустите:
lspci -v
И найдите ваше Ethernet-устройство. Если в описании указано «RTL8111/8168B» и «rev2», значит это ваш случай.
Подробно о решении задачи написано по ссылке:
Если вы пользуетесь OpenVZ, предлагаем универсальное решение — собранный пакет dkms-r8168-openvz (приложение, которое автоматически после переустановки ядра, при загрузке системы компилирует и устанавливает нужный сетевой драйвер). Установить его можно из :
Если на экране появятся ссылки на «r8168», значит вы уже работаете на новом драйвере. Если — «r8169», значит загрузился старый драйвер и необходимо еще раз перезагрузиться (после второй перезагрузки будет установлен драйвер «r8168»).
В дальнейшем, при обновлении ядра, dkms будет автоматически компилировать и устанавливать нужный драйвер для сетевой карты.
I am preparing an updated set of precreated templates; those should be ready tonight or tomorrow, available from .
In addition to a bunch of updated templates, this time we add a few new ones: — Fedora 10 (aka Cambridge) — openSUSE 11.1 — Ubuntu 9.04 (aka The Jaunty Jackalope)
OpenSUSE is interesting — apparently they dropped yum (which was available in 10.3 and 11.0 but not in 11.1) and now they have something called . Also note that openSUSE lacks the code name. the SUSE guys are already aware of the issue and have a plan to fix it — the next release (openSUSE 11.2) will be codenamed . Subsequent openSUSE releases will also be named after famous philosophers — , , Lessing (although I'm not sure which Lessing do they have in mind, probably ). Interesting... maybe they got the naming idea from OpenVZ kernels.
Also, during the next update (i.e. in about a month, not now) we are going to remove a few templates that are old and unsupported: — Debian 3.1 «Sarge» () — Fedora 7 () — openSUSE 10.3 () — Fedora 8 () — Ubuntu 7.10 () Anybody who's using those distros inside containers should updated to something more (r|d) ecent and supported. You have been warned.
PS For people who use our stable kernels (i.e. RHEL5 branch) — please note that you have to update to the latest kernel (028stab062.3 at the moment) in order to use Fedora 10 in containers. This is due to a few new system calls recently added to the Linux kernel which Fedora 10 userland expect to have in the kernel. Those syscalls were just backported to our RHEL5 branch by the OpenVZ team.
From time to time, somebody critisizes OpenVZ kernel patch for its intrusiveness and size. Right, it is big and intrusive — it adds a whole lot of new features into the kernel. But how big is it?
Our engineer prepared some stats on three different kernels: 1. OpenVZ stable kernel (based on 2.6.18-RHEL5); 2. OpenVZ development kernel (based on 2.6.27); 3. RHEL5.3 kernel (based on 2.6.18). You can see the results by clicking the image at the right.
Some notes for the graph. For OpenVZ kernels, we distinguish between core kernel changes and the stuff that is built as modules. For RHEL kernel, we break the patchset down into a few categories, such as drivers, Xen, GFS, ext4 and so on; «other» means everything not covered by any other category. The numbers are thousands lines of code added and deleted, combined. A table below the graph has some more details, like how many files were changed, how many lines added and deleted.
Now to the conclusions. Two major points can be made: 1. Even without drivers, RHEL5 kernel patches add/delete 434 KLOCs*, which is 8.5x times bigger then OpenVZ kernel modifications (51 KLOC). So, yes, OpenVZ patch set is big, but not that big. 2. OpenVZ based on mainstream 2.6.27 kernel requires 40% less** modifications to the kernel due to on-going effort to integrate the functionality into mainstream.
* KLOC is a thousand . ** we only count the core changes, omitting the modules.
A new 2.6.27-based OpenVZ branch is opened, and the first 2.6.27-based OpenVZ is released.
The idea of using names instead of numbers for kernel releases is working for 2.6.26, and we decided to have some fun with 2.6.27 kernels, too. These kernels are [to be] named after famous Russian painters, of course in the alphabetical order.
First 2.6.27 OpenVZ kernel is named after . Aivazovsky is a great painter and I'd love to add a link to some of his masterpieces here, but while trying to find a good reproduction of «The Ninth Wave» I realized that a typical notebook/PC is incapable of displaying such art. Then you try to fit a 2×3 meters painting into a 22" computer screen, nothing good is to be expected. Even in case of high resolution copy stored in a lossless format you either see a full picture but details are lost, or you see some part of it with all the details but then you don't see the full picture. So be aware that a painting that you see online is a pathetic shadow of what you can enjoy in a real (i.e. offline) museum or art gallery.
The 2.6.27-aivazovsky kernel, on the other side, is perfect for you PC, so enjoy.
When my colleague Pavel Emelyanov returned from the 2008 Linux kernel summit back in September he brought a small present for me — a (every LKS participant got one for free; yet another reason to become a high-profile kernel developer!). Overo is a computer (well, actually a set of boards and cables) with a CPU board the size of a gum stick, featuring TI OMAP3 CPU, 128 megs of RAM and a microSD slot. It also has 802.11g Wi-Fi and Bluetooth but those happens to be completely dead as this the first beta release of hardware.
For the last few days I was digging into a project to make OpenVZ running on this Overo thing. That involved patching OpenVZ kernel to support ARM architecture, building vzctl package (.ipk) for ARM using bitbake, and creating a template.
It was amazingly easy to port the OpenVZ kernel to ARM; you can see that besides a big-all-in-one-openvz-for-2.6.27 patch I only had to add 4 tiny ARM-specific patches (, , , ). For vzctl, it was even easier — all I had to do is to for ARM which were added, and create a .
Creating a template for ARM architecture was tougher but I managed to win that fight, too — you can find a Debian Lenny template .
Here is an except from a terminal session showing OpenVZ on Overo: <br />root@overo:~# uname -a<br />Linux overo 2.6.27-omap1 #1 Tue Oct 21 21:19:40 MSD 2008 armv7l unknown unknown GNU/Linux<br /><br />root@overo:~# cat /proc/vz/version<br />037test001<br /><br />root@overo:~# vzlist<br /> CTID NPROC STATUS IP_ADDR HOSTNAME<br /> 777 5 running - -<br /><br />root@overo:~# vzctl enter 777<br />entered into CT 777<br />-bash-3.2# ps axf<br />Unknown HZ value! (0) Assume 100.<br /> PID TTY STAT TIME COMMAND<br /> 310 ? Ss 0:00 vzctl: pts/0 <br /> 311 pts/0 Ss 0:00 \_ -bash<br /> 313 pts/0 R+ 0:00 \_ ps axf<br /> 1 ? Ss 0:00 init [2] <br /> 208 ? Sl 0:00 /usr/sbin/rsyslogd -c3<br /> 227 ? Ss 0:00 /usr/sbin/cron<br />
Please note that all this is still in very alpha stage — there are errors, bugs, ugly warnings, you have to modify some things in place etc. But it's working. If someone is interested in running OpenVZ on ARM hardware, please let me know — leave a comment here or email kir (A) openvz (.) org.
When my colleague Pavel Emelyanov returned from the 2008 Linux kernel summit back in September he brought a small present for me — a (every LKS participant got one for free; yet another reason to become a high-profile kernel developer!). Overo is a computer (well, actually a set of boards and cables) with a CPU board the size of a gum stick, featuring TI OMAP3 CPU, 128 megs of RAM and a microSD slot. It also has 802.11g Wi-Fi and Bluetooth but those happens to be completely dead as this the first beta release of hardware.