Игорь Олемской — практические заметки по системному администрированию Linux CentOS

VSwap is an excellent feature, simplifying container resource management a lot. Now it's time to also simplify the command line interface, i.e. vzctl. Here is what we did recently (take a look at vzctl git repo if you want to review the actual changes):

1. We no longer require to set kmemsize, dcachesize and lockedpages parameters to non-unlimited values (this is one of the enhancements in the recent kernels we have talked about recently). Therefore, setting these parameters to fractions of CT RAM (physpages) are now removed from configuration files and vzsplit output.

2. There is no longer a need to specify all the UBC parameters in per-container configuration file. If you leave some parameters unset, the kernel will use default values (usually unlimited). So, VSwap example configs are now much smaller, with as much as 19 parameters removed from those.

3. vzctl set now supports two new options: --ram and --swap. These are just convenient short aliases for --physpages and --swappages, the differences being that you only need to specify one value (the limit) and the argument is in bytes rather than pages.

So, to configure a container named MyCT to have 1.5GB of RAM and 3GB of swap space available, all you need to do is just run this command:
vzctl set MyCT --ram 1.5G --swap 3G --save

4. This is not VSwap-related, but nevertheless worth sharing. Let's illustrate it by a copy-paste from a terminal:
# vzctl create 123 --ostemplate centos-4-x86_64<br />Creating container private area (centos-4-x86_64)<br />Found centos-4-x86_64.tar.gz at <a href='http://download.openvz.org/template/precreated//centos-4-x86_64.tar.gz'>http://download.openvz.org/template/precreated//centos-4-x86_64.tar.gz</a><br />Downloading...<br />-- 2011-11-29 18:54:08 -- <a href='http://download.openvz.org/template/precreated//centos-4-x86_64.tar.gz'>http://download.openvz.org/template/precreated//centos-4-x86_64.tar.gz</a><br />Resolving download.openvz.org... 64.131.90.11<br />Connecting to download.openvz.org|64.131.90.11|:80... connected.<br />HTTP request sent, awaiting response... 200 OK<br />Length: 171979832 (164M) [application/x-gzip]<br />Saving to: `/vz/template/cache/centos-4-x86_64.tar.gz'<br /><br />100%[======================================>] 171,979,832 588K/s in 4m 27s <br /><br />2011-11-29 18:58:35 (630 KB/s) - `/vz/template/cache/centos-4-x86_64.tar.gz' saved [171979832/171979832]<br /><br />Success!<br />Performing postcreate actions<br />Saved parameters for CT 123<br />Container private area was created

All this will be available in vzctl-3.0.30, which is to be released soon (next week? who knows). If you can't wait and want to test this stuff, here's a nightly build of vzctl available (version 3.0.29.3-27.git.0535fe1) from http://wiki.openvz.org/Download/vzctl/nightly. Please give it a try and tell me what you think.

Hard CPU limit (ability to specify that you don't want this container to use more than X per cent of CPU no matter what) is back in latest RHEL6-based kernel, 042test006.1, which has just been released.

The feature was only available for the stable (i.e RHEL4 and RHEL5-based) kernels, and was missing from all of our development kernels from 2.6.20 to 2.6.32. So while it was always there in stable branches, the feeling is like it's back.

In order to use CPU limit feature, set the limit using vzctl set $CTID --cpulimit X, where X is in per cent of one single CPU. For example, if you have single 2 GHz CPU and want container 123 to use no more than 1 GHz, use vzctl set 123 --cpulimit 50. If you have 2 GHz quad-core system and want to use no more than 4 GHz, use vzctl set 123 --cpulimit 200. Well, in the second case it might be better to just use --cpus 2. Anyways, see vzctl man page.

Hard CPU limit (ability to specify that you don't want this container to use more than X per cent of CPU no matter what) is back in latest RHEL6-based kernel, 042test006.1, which has just been released.

The feature was only available for the stable (i.e RHEL4 and RHEL5-based) kernels, and was missing from all of our development kernels from 2.6.20 to 2.6.32. So while it was always there in stable branches, the feeling is like it's back.

In order to use CPU limit feature, set the limit using vzctl set $CTID --cpulimit X, where X is in per cent of one single CPU. For example, if you have single 2 GHz CPU and want container 123 to use no more than 1 GHz, use vzctl set 123 --cpulimit 50. If you have 2 GHz quad-core system and want to use no more than 4 GHz, use vzctl set 123 --cpulimit 200. Well, in the second case it might be better to just use --cpus 2. Anyways, see vzctl man page.

I thought that Monday is a good day to release a new version of vzctl which was worked on for the last six months. There are a few important changes in this release, let me go through it.

First, we have finally removed all the cronscripts trickery required for CT reboot. The thing is, if the container owner issues 'reboot' command from inside, the container just stops. Now, something needs to be done on the host system to start it again. Until this release, this was achieved by a hackish combination of vzctl (which adds an initscript inside container to add a reboot mark) and a cron script (which checks for the stopped containers having that reboot mark and starts those). Yet another cron script takes care about a situation when a CT is stopped from the inside — in this case some cleanup needs to be done from the host system, namely we need to unmount the CT private area, and remove the routing and ARP records for the CT IP.

There are a few problems with this cron-based approach. First, initscript handling can be different in different distributions, and it's really hard to support all of the distros. Second, cron script is run every 5 minutes, which means a mean time to reboot (or clean up network rules) is 2.5 minutes. To say it simple, it's all hackish and unreliable.

Now, this hairy trickery is removed and replaced by a simple and clean daemon called vzeventd, which listens to CT stop and reboot events, and runs clean and simple scripts. No more trickery, no more waiting for reboot. The only catch is this requires support from the kernel (which comes in a form of vzevent kernel module).

Second, new vzctl is able to start Fedora 14 containers on our stable (i.e. RHEL5-2.6.18) kernels. The thing is, Fedora 14 have glibc patched to check for specific kernel version (>=2.6.32 in this case) and refuse to work otherwise. This is done to prevent glibc from using the old kernels with some required features missing. We patch our kernels to have those features, but glibc just checks the version. So, our recent kernels is able to set osrelease field of uname structure to any given value for a given container. Now, vzctl 3.0.25 comes with a file (/etc/vz/osrelease.conf) which lists different distros and their required kernel version, which it sets during start and exec.

I want to briefly mention yet another feature of recent vzctl (which, again, needs kernel support) — an ability to delegate a PCI device into a container. It is only supported on RHEL6 kernel at the moment, and the only devices that we have tried are NVidia GPUs.

Besides these three big things, there are a lot of improvements, fixes, and documentation updates all over the tree. I don't know of any known regressions in this release but I guess it's not entirely Bug Free. Fortunately there's a way to handle it — if anything really bad appears in this version, it will be fixed by a quick 3.0.25.1 update. This worked pretty well for vzctl-3.0.24, should work fine this time, too.

As always, please report all the bugs found to http://bugzilla.openvz.org/

I thought that Monday is a good day to release a new version of vzctl which was worked on for the last six months. There are a few important changes in this release, let me go through it.

First, we have finally removed all the cronscripts trickery required for CT reboot. The thing is, if the container owner issues 'reboot' command from inside, the container just stops. Now, something needs to be done on the host system to start it again. Until this release, this was achieved by a hackish combination of vzctl (which adds an initscript inside container to add a reboot mark) and a cron script (which checks for the stopped containers having that reboot mark and starts those). Yet another cron script takes care about a situation when a CT is stopped from the inside — in this case some cleanup needs to be done from the host system, namely we need to unmount the CT private area, and remove the routing and ARP records for the CT IP.

There are a few problems with this cron-based approach. First, initscript handling can be different in different distributions, and it's really hard to support all of the distros. Second, cron script is run every 5 minutes, which means a mean time to reboot (or clean up network rules) is 2.5 minutes. To say it simple, it's all hackish and unreliable.

Now, this hairy trickery is removed and replaced by a simple and clean daemon called vzeventd, which listens to CT stop and reboot events, and runs clean and simple scripts. No more trickery, no more waiting for reboot. The only catch is this requires support from the kernel (which comes in a form of vzevent kernel module).

Second, new vzctl is able to start Fedora 14 containers on our stable (i.e. RHEL5-2.6.18) kernels. The thing is, Fedora 14 have glibc patched to check for specific kernel version (>=2.6.32 in this case) and refuse to work otherwise. This is done to prevent glibc from using the old kernels with some required features missing. We patch our kernels to have those features, but glibc just checks the version. So, our recent kernels is able to set osrelease field of uname structure to any given value for a given container. Now, vzctl 3.0.25 comes with a file (/etc/vz/osrelease.conf) which lists different distros and their required kernel version, which it sets during start and exec.

I want to briefly mention yet another feature of recent vzctl (which, again, needs kernel support) — an ability to delegate a PCI device into a container. It is only supported on RHEL6 kernel at the moment, and the only devices that we have tried are NVidia GPUs.

Besides these three big things, there are a lot of improvements, fixes, and documentation updates all over the tree. I don't know of any known regressions in this release but I guess it's not entirely Bug Free. Fortunately there's a way to handle it — if anything really bad appears in this version, it will be fixed by a quick 3.0.25.1 update. This worked pretty well for vzctl-3.0.24, should work fine this time, too.

As always, please report all the bugs found to http://bugzilla.openvz.org/

Just wanted to mention a few news items from the OpenVZ Project.

Updated vzctl — vzctl 3.0.24 has been released. Even though the version number only changed from 3.0.23 to 3.0.24 there are a ton of changes, fixes and some feature additions. Of special interest is the --swappages option as well as being able to refer to a container by its name rather than requiring the CTID with vzmigrate. Overall it was a long overdue, much appreciated update.

Updated Official OS Templates — The last wiki notice is dated April 27th but looking today at the dates on the OS Templates they appear to have been updated May 27th. One thing to note is that there are now OS Templates for Ubuntu 10.04 which I'm sure Ubuntu folks will be happy about.

Beta Fedora 13 OS Templates — And speaking of OS Templates, Kir just released Beta OS Templates for Fedora 13. The day Fedora 13 was released I tried creating my own OS Templates by taking Fedora 12 containers and upgrading them but ran into a snag. With Fedora 13 a lot of new stuff has been added to the init setup and some of it causes a container to just hang during init. I was glad to see the beta OS Templates released. I created containers from them, made my own changes, and then uploaded those to the contrib section.

As luck would have it, later in the afternoon the Fedora Project release a whole bunch of updates and among them was a new initscripts package. I suspected that when I upgraded my container whatever changes the OpenVZ folks had made to the init setup that made it work in a container would be wiped out and I was correct as upgrading the initscripts package did indeed make the container get stuck in the init process upon container reboot. I ended up filing two bugs: 1566 and 1567 and await their joyful resolution.

*** Please note*** Any URLs mentioned (and the information they contain) in this posting are time sensitive and will surely be outdated not long after posting.

Just wanted to mention a few news items from the OpenVZ Project.

Updated vzctl — vzctl 3.0.24 has been released. Even though the version number only changed from 3.0.23 to 3.0.24 there are a ton of changes, fixes and some feature additions. Of special interest is the --swappages option as well as being able to refer to a container by its name rather than requiring the CTID with vzmigrate. Overall it was a long overdue, much appreciated update.

Updated Official OS Templates — The last wiki notice is dated April 27th but looking today at the dates on the OS Templates they appear to have been updated May 27th. One thing to note is that there are now OS Templates for Ubuntu 10.04 which I'm sure Ubuntu folks will be happy about.

Beta Fedora 13 OS Templates — And speaking of OS Templates, Kir just released Beta OS Templates for Fedora 13. The day Fedora 13 was released I tried creating my own OS Templates by taking Fedora 12 containers and upgrading them but ran into a snag. With Fedora 13 a lot of new stuff has been added to the init setup and some of it causes a container to just hang during init. I was glad to see the beta OS Templates released. I created containers from them, made my own changes, and then uploaded those to the contrib section.

As luck would have it, later in the afternoon the Fedora Project release a whole bunch of updates and among them was a new initscripts package. I suspected that when I upgraded my container whatever changes the OpenVZ folks had made to the init setup that made it work in a container would be wiped out and I was correct as upgrading the initscripts package did indeed make the container get stuck in the init process upon container reboot. I ended up filing two bugs: 1566 and 1567 and await their joyful resolution.

*** Please note*** Any URLs mentioned (and the information they contain) in this posting are time sensitive and will surely be outdated not long after posting.

Long time no see. It's 11pm here now and I'm still in the office. Just though about why not to post to the blog right before I drive home. Really, why not?

I'm mostly working on new vzctl release lately (you can see the progress in vzctl's git web interface here). The ultimate task is to fix most of bugs opened for vzctl in OpenVZ bugzilla (some are dated back to 2007 — no, they are not critical or even major, but anyway). So far the list of vzctl bugs yet opened are down to one singe page (i.e. scroll bar has disappeared from the browser window) which is a big improvement.

The process is somewhat slow because of my attitute to fix even minor things thenever I notice them — known as code refactoring. The simplest example is when you open a C file in vim (or Emacs, whatever) and see that there are spaces instead of tabs. When you notice it, there are three ways to go: (A) leave it as is; (B) fix it in place, keep working on what you've planned, commit; © fix it, make a separate commit, then continue working on what you've planned.

Way A is not that bad, actually, it's definitely better than B, because the result of B is a spaghetti of a functional (e.g. a new feature or a bugfix) and non-functional (e.g. whitespace cleanup) changes. Mixing apples and oranges is a bad thing to do: it makes patch review harder, it makes porting between branches harder, and in case you'll want to revert the patch (say because of a bug in new code) you will also revert those cleanups (which are not buggy). So, if you are not a perfectionist, better follow way A but please not B.

Unfortunately I just can't ignore the bad things I see, so I follow the way C. Sometimes this is very annoying, because instead of implementing a new feature you keep refactoring your code for hours, sometimes even days. No, it's not only whitespace cleanups or fixing typos in comments. Sometimes you see that there are a few identical snippets of code and you put that code into a new function. Sometimes you notice that some function arguments are unused (or always the same) and you remove those. Sometimes you just read the code and see there is a bug in it, and you fix it. Oh, the text of error message is not clear enough — you fix it. The typesetting of a man page is wrong (e.g. bold is used instead of italic for variable argument) — you fix it. You see that a function is not used outside of the source file — you make it static and remove its prototype from a header file. You notice a typo in the function name (and of course also in every place that calls it, since otherwise that code won't compile) — you fix it. A typo in a comment — fix it! And so on, and so forth...

Every single thing from the above paragraph (and some more) happened to me when I was working on the boot order priority feature (the subject of OpenVZ bug #1300). So I ended up with 22 (twenty two) code commits, from which 3 implement the actual feature, 1 documents it, and the other 18 are all code refactoring, preparation and minor bugfixing.

The way to perfection is never easy. That doesn't mean we shouldn't try.

There is a nice feature in vzctl (well, technically not in vzctl binary itself; it just comes in vzctl package) that many people don't know about — completion. This basically makes it able to save a few keystrokes when typing.

Say you want to create a container. You type vzct and press <TAB> — it completes that to vzctl and a space after. This is usual feature of bash — it looks all the binaries available in $PATH and tries to complete their names.

Now let's see the vzctl completion:
# vzctl cr<TAB>
completes to
# vzctl create
and then after yet another <TAB> it suggests a CT ID which is the MAX+1 (i.e. if you have containers 101, 102 and 105 it will suggest 106):
# vzctl create 106
Now we want to specify an OS template:
# vzctl create 106 --os<TAB>
will get you to
# vzctl create 106 --ostemplate
and then you press <TAB> again twice to see the list of available OS templates:
# vzctl create 106 --ostemplate <TAB><TAB>
centos-5-x86 centos-5-x86-devel fedora-9-x86 suse-11.1-x86
Now you type in the first few characters of the OS template you want to use:
# vzctl create 106 --ostemplate f<TAB>
and it will complete that to
# vzctl create 106 --ostemplate fedora-9-x86
Now, unless you want to specify --config or some other parameters, just press Enter.

This completion is smart — say, if you want to start a container, type in
# vzctl start <TAB><TAB>
and it will give you the list of container IDs that can be started (i.e. all the stopped containers).

And so on and so forth. Well, you say, it doesn't work! In that case you have to enable it, here's how.

On a RHEL, CentOS or Fedora system run yum install bash-completion and then relogin (i.e. log out and log in again). If your host system is Gentoo, run emerge bash-completion and then eselect bashcomp enable vzctl. I hope someone will comment on how to enable this for Debian/Ubuntu/SUSE or whatever your favorite distro is.

Today is definitely a day of releases.

OpenVZ project has released both new vzctl and vzquota tools today.

New vzctl has a handful of new small features and a bunch of bugfixes, including compatibility with recent glibc, bash, and kernel headers.

New vzquota has only one (but quite useful) new feature — an ability to explain what's wrong when it can not turn container's disk quota on or off. Recent OpenVZ kernels have a feature to report open files in container's private area, and now with the new vzquota the feature is finally available for mere mortals.

In the meantime Parallels has released Parallels Desktop for Mac 4.0 — and that's just a coincidence, I'm sure they do not sync their release cycles with OpenVZ. Or maybe it's not a coincidence... We're sitting in the same office and for the last few weeks they've been providing free late dinners because of their release, that maybe made me leave the office later and thus maybe gave more time to work on OpenVZ tools.