Игорь Олемской — практические заметки по системному администрированию Linux CentOS

Getting XenServer installed on some unusual platforms takes a bit of work and the AOpen MP57 is a challenging platform for a XenServer 6.0.2 installation.

My MP57 box came with the i57QMx-vP motherboard. If yours came with something else, this post may or may not work for you.

You'll need the XenServer 6 installation ISO burned to a CD to get started. Boot the CD in your MP57 and wait for the initial boot screen to appear. Type safe at the prompt and press enter. Go through the normal installation steps and reboot.

After the reboot, you'll notice that there's no video output for dom0. Hop on another nearby computer and ssh to your XenServer installation using the root user and the password that you set during the installation process. Open up /boot/extlinux.conf in your favorite text editor and make sure the label xe section looks like this:

label xe
  # XenServer
  kernel mboot.c32
  append /boot/xen.gz mem=1024G dom0_max_vcpus=4 dom0_mem=752M lowmem_emergency_pool=1M crashkernel=64M@32M acpi=off console=vga --- /boot/vmlinuz-2.6-xen root=LABEL=root-aouozuoo ro xencons=hvc console=hvc0 console=tty0 vga=785 --- /boot/initrd-2.6-xen.img

The console=vga adjustment ensures that the dom0 console is piped to the vga output and acpi=off fixes the lockup that will occur when the vga output is sent to your display. I also removed splash and quiet from the kernel line so that I could see all of the boot messages in detail.

Installing XenServer 6.0.2 on an AOpen MP57 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Getting Fedora 16 working in XenServer isn't the easiest thing to do, but I've put together a repository on GitHub that should help. The repository contains a kickstart file along with some brief instructions to help with the installation. If you're ready to get started right now, just clone the repository:

git clone git://github.com/rackerhacker/kickstarts.git kickstarts

There are some big issues with Fedora 16 which cause problems for installations within XenServer:

• the installer sets up a console on something other than hvc0
• anaconda won't start without being in serial mode
• anaconda tries to use GPT partitions by default
• grub2 is now standard, but it causes problems for older XenServer versions

My kickstart works around the grub2 problem by throwing down an old-style grub configuration file and creating the proper symlinks. This config will still be updated when you upgrade kernels (at least in Fedora 16). It also sets up a very simple partitioning schema with one root and one swap partition. A DOS partition table is used in lieu of a GPT partition table.

When you start the installation, be sure to review the README.md in the git repository. It has some special instructions for boot options to meet the requirements of Fedora 16 and the kickstart file.

Installing Fedora 16 in XenServer is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Although Citrix recommends against using software RAID with XenServer due to performance issues, I've had some pretty awful experiences with hardware RAID cards over the last few years. In addition, the price of software RAID makes it a very desirable solution.

Before you get started, go through the steps to disable GPT. That post also explains an optional adjustment to get a larger root partition (which I would recommend). You cannot complete the steps in this post if your XenServer installation uses GPT.

You should have three partitions on your first disk after the installation:

# fdisk -l /dev/sda
-- SNIP --
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2611    20971520   83  Linux
/dev/sda2            2611        5222    20971520   83  Linux
/dev/sda3            5222       19457   114345281   8e  Linux LVM

Here's a quick explanation of your partitions:

• /dev/sda1: the XenServer root partition
• /dev/sda2: XenServer uses this partition for temporary space during upgrades
• /dev/sda3: your storage repository should be in this logical volume

We need to replicate the same partition structure across each of your drives and the software RAID volume will span the across the third partition on each disk. Copying the partition structure from disk to disk is done easily with sfdisk:

WHOA THERE! NO TURNING BACK! This step is destructive! If your other disks have any data on them, this step will make it (relatively) impossible to retrieve data on those disks again. Back up any data on the other disks in your XenServer machine before running these next commands.

sfdisk -d /dev/sda | sfdisk --force /dev/sdb
sfdisk -d /dev/sda | sfdisk --force /dev/sdc
sfdisk -d /dev/sda | sfdisk --force /dev/sdd

If you have only two disks, stop with /dev/sdb and you'll be making a RAID 1 array. My machine has four disks and I'll be making a RAID 10 array.

We need to destroy the main storage repository, but we need to unplug the physical block device first. Get the storage repository uuid first, then use it to find the corresponding physical block device. Once the physical block device is unplugged, the storage repository can be destroyed:

# xe sr-list name-label=Local\ storage | head -1
uuid ( RO)                : 75264965-f981-749e-0f9a-e32856c46361
# xe pbd-list sr-uuid=75264965-f981-749e-0f9a-e32856c46361 | head -1
uuid ( RO)                  : ff7e9656-c27c-1889-7a6d-687a561f0ad0
# xe pbd-unplug uuid=ff7e9656-c27c-1889-7a6d-687a561f0ad0
# xe sr-destroy uuid=75264965-f981-749e-0f9a-e32856c46361

All of the LVM data from /dev/sda3 should now be gone:

# lvdisplay && vgdisplay && pvdisplay
#

Change the third partition on each physical disk to be a software RAID partition type:

echo -e "t\n3\nfd\nw\n" | fdisk /dev/sda
echo -e "t\n3\nfd\nw\n" | fdisk /dev/sdb
echo -e "t\n3\nfd\nw\n" | fdisk /dev/sdc
echo -e "t\n3\nfd\nw\n" | fdisk /dev/sdd

Stop here and reboot your XenServer box to pick up the new partition changes. Once the server comes back from the reboot, start up a software RAID volume with mdadm:

// RAID 1 for two drives
mdadm --create /dev/md0 -l 1 -n 2 /dev/sda3 /dev/sdb3
// RAID 10 for four drives
mdadm --create /dev/md0 -l 10 -n 4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

Check to see that your RAID array is building:

# cat /proc/mdstat
Personalities : [raid10]
md0 : active raid10 sdd3[3] sdc3[2] sdb3[1] sda3[0]
      228690432 blocks 64K chunks 2 near-copies [4/4] [UUUU]
      [>....................]  resync =  0.3% (694272/228690432) finish=16.4min speed=231424K/sec

Although you don't have to wait for the resync to complete, just be aware that XenServer doesn't do well with a lot of disk I/O within dom0. You may notice unusually slow performance in dom0 until it finishes. Save the array's configuration for reboots:

mdadm --detail --scan > /etc/mdadm.conf

Edit the /etc/mdadm.conf file and append auto=yes to the end of the line (but leave everything on one line):

ARRAY /dev/md0 level=raid10 num-devices=4 metadata=0.90 \
  UUID=2876748c:5117eed5:ce4d62d3:9592bd84 auto=yes

Create a new storage repository on the RAID volume with thin provisioning (thanks to Spherical Chicken for the command):

xe sr-create content-type=user type=ext device-config:device=/dev/md0 shared=false name-label="Local storage"

This command takes some time to complete since it makes logical volumes and then makes an ext3 filesystem for the new storage repository. Bigger RAID arrays will take more time and it's guaranteed to take longer than you'd expect if your RAID array is still building. As soon as it completes, you'll be given the uuid of your new storage repository and it should appear within the XenCenter interface.

TIP: If you run into any problems during reboots, open /boot/extlinux.conf and remove splash and quiet from the label xe boot section. This removes the framebuffer during boot-up and it causes a lot more output to be printed to the console. It won't affect the display once your XenServer box has fully booted.

XenServer 6: Storage repository on software RAID is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

XenServer 6 is a solid virtualization platform, but the installer doesn't give you many options for customized configurations. By default, it installs with a 4GB root partition and uses GUID Partition Tables (GPT). GPT is new in XenServer 6.

I'd rather use MBR partition tables and get a larger root partition. If you want to make these adjustments in your XenServer 6 installation, follow these steps after booting into the XenServer 6 install disc:

When the installer initially boots, press F2 to access the advanced installation options.

Type shell and press enter. The installer should begin booting into a pre-installation shell where you can make your adjustments.

Once you've booted into the pre-installation shell, type vi /opt/xensource/installer/constants.py and press enter.

Change GPT_SUPPORT = True to GPT_SUPPORT = False to disable GPT and use MBR partition tables. Adjust the value of root_size from 4096 (the default) to a larger number to get a bigger root partition. The size is specified in MB, so 4096 is 4GB. Save the file and exit vim.

Type exit and the installer should start.

Once the installation is complete, you should have a bigger root partition on a MBT partition table:

# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              20G  1.8G   17G  10% /
# fdisk -l /dev/sda
 
Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2611    20971520   83  Linux
/dev/sda2            2611        5222    20971520   83  Linux
/dev/sda3            5222       19457   114345281   8e  Linux LVM

XenServer 6: Disable GPT and get a larger root partition is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Fedora 15 was released with some updates to allow for consistent network device names. Once it's installed, you'll end up with network devices that are named something other than eth0, eth1, and so on.

For example, all onboard ethernet adapters are labeled as emX (em1, em2...) and all PCI ethernet adapters are labeled as pXpX (p[slot]p[port], like p7p1 for port 1 on slot 7). Ethernet devices within Xen virtual machines aren't adjusted.

This may make sense to people who swap out the chassis on servers regularly and they don't want to mess with hard-coding MAC addresses in network configuration files. Also, it should give users predictable names even if a running system's drives are inserted into a newer hardware revision of the same server.

However, I don't like this on my personal dedicated servers and I prefer to revert back to the old way of doing things. Getting back to eth0 is pretty simple and it only requires a few configuration files to be edited followed by a reboot.

First, add biosdevname=0 to your grub.conf on the kernel line:

title Fedora (2.6.40.4-5.fc15.x86_64)
	root (hd0,0)
	kernel /boot/vmlinuz-2.6.40.4-5.fc15.x86_64 ro root=/dev/md0 SYSFONT=latarcyrheb-sun16 KEYTABLE=us biosdevname=0 quiet LANG=en_US.UTF-8
	initrd /boot/initramfs-2.6.40.4-5.fc15.x86_64.img

Open /etc/udev/rules.d/70-persistent-net.rules in your favorite text editor (create it if it doesn't exist) and add in the following:

# Be sure to put your MAC addresses in the fields below
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:10", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:11", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Be sure to rename your ifcfg-* files in /etc/sysconfig/network-scripts/ to match the device names you've assigned. Just for good measure, I add in the MAC address in /etc/sysconfig/network-scripts/ifcfg-ethX:

...
HWADDR=00:11:22:33:44:10
...

Reboot the server and you should be back to eth0 and eth1 after a reboot.

Getting back to using eth0 in Fedora 15 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

root@d17:~# xm create pan0.vm1.h2.d0.dddddddd.ru.cfg -c
Using config file "/etc/xen/pan0.vm1.h2.d0.ddddddd.ru.cfg".
Error: 'NoneType' object has no attribute 'rfind'

В дебиан лечится установкой xen-qemu-dm-4.0

apt-get install xen-qemu-dm-4.0

При попытке создать новый DomU на Dom0 получил ошибку:

root@d17:~# Error: Device 51718 (vbd) could not be connected. Failed to find an unused loop device

Лечится увеличением loop устройств в системе.

В случае CentOS/RHEL/FedoraCore в файл /etc/modprobe.conf добавляем строку:

options loop max_loop=128

В случае Debian/Ubuntu в файл /etc/modules приводим строку с loop к виду:

loop max_loop=128

В некоторых случаях это не срабатывает, тогда можно передать это как параметр инициализации ядра.
Вот кусок из /boot/grub/menu.lst

title Xen
root (hd0,0)
  kernel /xen.gz
  module /vmlinuz-2.6-xen root=/dev/sda1 ro max_loop=128
  module /initrd.img-2.6.16.29-xen

Самый простой способ «войти» в гостевую систему на Xen это

xm console guest-vm

где guest-vm это нужная гостевая ОС, но тут нам понадобится рутовый или иной административный пароль. Если таким богатством не обладаем то есть вариант такой:
1. останавливаем гостевую машину:

xm shutdown -H guest-vm

2. проверяем остановлен ли гость через несколько минут:

xm list

3. Монтируем образ гостевой ОС и чрутимся в него:

mkdir /root/temp/
mount -o loop /mnt/vm_stor/domains/guest-vm/disk.img /root/temp/
mount -o bind /dev /root/temp/dev/
mount -t proc none /root/temp/proc/
chroot /root/temp/

После выполнения работы, например смены рутового пароля на Xen guest, нужно выйти из чрута, размонтировать и стартануть гостя:

umount /root/temp/dev
umount /root/temp/proc
umount /root/temp
xm create guest-vm

Ну и обязательно проконтролировать загрузку:

xm console guest-vm

Quite a few people who couldn't make it to Xen Summit 2011 this year asked me to write a post summarizing my takeaways from the event. I'm not generally one to back down from peer pressure, so read on if you're interested about the discussions at this year's Summit.

The feeling I had at last year's summit is that Xen was on the verge of losing traction in the market. Very few distributions still had Xen support going forward and much of the discussion was around the lack of dom0 support in upstream Linux kernels. Distribution vendors were hesitant to drag patches forward into modern kernels and this made it much more difficult to get Xen working for many people.

This year was quite different. The number of attendees was up, the venue was much better, and there was an obvious buzz of energy in the room. As many of the presenters noted, this excitement stemmed from the upstream dom0 support in Linux 3.0. This inclusion is a huge win and it helps to drive Xen forward since the developers don't have to worry about dragging patches forward. They can focus on improving performance, adding features, and tightening security.

Many of the discussions this year focused on security and performance. Ian Pratt discussed Xen's ability to view memory pages of virtual machines via an API to detect malware running inside the instance. Memory pages could be identified and marked as not executable or applications could be triggered when a VM attempts to touch a particular memory page. Also, the whole VM could be frozen if needed.

There's also a big push to bring code out of the dom0 and push it into utility VM's. Driver domains could manage the network or I/O infrastructure and this would further reduce the amount of privileged code actively running in dom0. There is already very little code required for the Xen hypervisor itself (much much less than the Linux kernel — I'm looking at you, KVM) and this reduces the attack surface for potential compromises of the hypervisor. Some projects even aim to restart driver domains multiple times per minute to ensure that any malicious code injected into those virtual machines can't exist for long periods.

Pradeep Vincent from Amazon talked about how Amazon uses Xen and the pain points they have with its current architecture. Much of his discussion was around scaling problems (and we see many of the same issues at Rackspace). Higher performance could easily be gained by multi-threaded operations in dom0 when attaching block devices and creating virtual network interfaces. He also saw some areas for performance gains in the pvops I/O code.

Quite a few of the talks centered on the ARM architecture and what Xen is able to do on those systems after Samsung published their port in 2008. HVM is on the way for ARM and it might even show up in Xen 4.2. Some demos of Xen on mobile phones from Samsung were amazing. They showed how an attacker could compromise the web browser on the phone with a keylogger, but that application was running in a VM. Once the user switched back to the phone's main menu, the keylogger couldn't access the keystrokes any longer. After that, a simple close of the browser killed the VM and destroyed the malicious code.

Xen 4.2 should be available in early 2012 and the feature list is staggering. Improvements to libxenlight, pvops performance (even in HVM), and guest memory sharing should be available with the new release. Nested virtualization (run a hypervisor inside a hypervisor) is also coming in Xen 4.2 and I'm sure Xzibit will be a huge fan. This should streamline hypervisor testing, allow for embedded hypervisor options and extend the capabilities of client hypervisors. Remus should be available in 4.2 as well, but it might be marked as experimental. OVMF will be added as a BIOS option for UEFI (along with the standard SeaBIOS) and this should allow for Mac OS X guests. UEFI allows Windows to boot faster since it switches to PV mode sooner and it allows for simpler platform certification for software vendors.

Mike McClurg's presentation on XCP was pretty important to me since Rackspace is a big consumer of XenServer. If you're not familiar with XCP, it's basically open-source XenServer which runs on bleeding edge (and sometimes unstable) components. XCP 1.5 and XenServer 6 should be available in November with Xen 4.1 and Linux 2.6.32. GPU passthrough, up to 1TB RAM, and disaster recovery will be available. Another goal for the XCP team is to work closely with OpenStack via Project Olympus. Mike's vision is to have XCP become the configuration of choice for open source clouds. Project Kronos was also extremely interesting. It's essentially XCP's XenAPI stack running on Debian and Ubuntu. You'd be able to install either OS on a physical server and run XCP's services on it for a fully OSS hypervisor.

Konrad Wilk gave an update on Linux pvops and it appears there is a shift to get Xen working well on a desktop. This includes 3D graphics support, S3/hibernate capabilities and various bug fixes. There's also a push to get PV functionality into HVM and get HVM functionality into PV. Driver/device domains were discussed again in Patrick Kolp's talk and he had plenty of graphs showing performance changes when regularly restarting device domains. The performance dips were almost negligible with 10 second restarts and the security gains were significant.

There were several other great presentations on other topics like GlusterFS, OpenStack Nova, and Linpicker (from the NSA!). If these types of things interests you, keep your eyes peeled for Xen Summit 2012 next year. The weather in the bay area is well worth the trip.

Xen Summit 2011: My Takeways is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you haven't noticed already, full Xen dom0 support was added in the Linux 3.0 kernel. This means there's no longer a need to drag patches forward from old kernels and work from special branches and git repositories when building a kernel for dom0.

Something else you might not have noticed is that the Fedora kernel team has quietly slipped Linux 3.0 into Fedora 15's update channels in disguise. Click that link, scroll down, and you'll see «Rebase to 3.0. Version reports as 2.6.40 for compatibility with older userspace.» Although I'm not a fan of calling something what it isn't (2.6.40 doesn't exist on kernel.org), I can understand some of the reasoning behind the choice.

This change makes the Xen installation on Fedora 15 pretty trivial. To get started, update your kernel to the latest if you're not already on Fedora's 2.6.40 kernels:

yum -y upgrade kernel

We need three more packages (quite a few dependencies will roll in with them):

yum -y install xen libvirt python-virtinst

The xen package reels in the hypervisor itself along with libraries and command line tools (like xl and xm). Libvirt gives us easy access to VM management with the virsh command and python-virtinst gives us the handy virt-install command to make OS installations easy.

Once those packages are installed, we need to make some adjustments in your grub configuration. Open /boot/grub/menu.lst in your text editor of choice and add something like this at the bottom:

title Fedora + Xen (2.6.40-4.fc15.x86_64)
        root (hd0,1)
	kernel /boot/xen.gz
        module /boot/vmlinuz-2.6.40-4.fc15.x86_64 ro root=/dev/sda1
        module /boot/initramfs-2.6.40-4.fc15.x86_64.img

Ensure that the root (hd0,1) is applicable to your system (adjust it if it isn't). Also, check the kernel version to ensure it matches your installed kernel and adjust the root= portion to match your root volume. Flip the default line to a value which will boot your new grub entry and ensure the timeout is set to a reasonable number if you need to temporarily switch back to your original grub entry at boot time. (Hey, we all make mistakes.)

I take one extra precaution and change the UPDATEDEFAULT=yes line to no in /etc/sysconfig/kernel. This ensures that future kernel updates don't trample the entry you've just made. Keep in mind that you'll need to manually update your grub configuration when you do kernel upgrades later.

Cross your fingers and reboot. If your system doesn't reboot properly, reboot it again and choose your old kernel from the grub menu. Double-check your configuration for fat-fingering and give it another try. If your system boots and pings but you have no output via a monitor, don't fret. There's a patch for the problem which should appear soon in Linux 3.0. The impatient can snag a kernel source RPM, add the patch file, and build a local kernel (or you can download my local build from when I did it).

Log in and verify that you booted into the dom0:

[root@xenbox ~]# xm dmesg | head -n 5
 __  __            _  _    _   _   ____     __      _ ____
 \ \/ /___ _ __   | || |  / | / | |___ \   / _| ___/ | ___|
  \  // _ \ '_ \  | || |_ | | | |__ __) | | |_ / __| |___ \
  /  \  __/ | | | |__   _|| |_| |__/ __/ _|  _| (__| |___) |
 /_/\_\___|_| |_|    |_|(_)_(_)_| |_____(_)_|  \___|_|____/

Once you're done with that, make sure libvirtd is running:

/etc/init.d/libvirtd start; chkconfig libvirtd on

Try installing a VM:

virt-install \
  --paravirt \
  --name=testvm \
  --ram=512 \
  --vcpus=4 \
  --file /dev/vmstorage/testvm \
  --graphics vnc,port=5905 --noautoconsole \
  --autostart --noreboot \
  --location=http://mirrors.kernel.org/debian/dists/squeeze/main/installer-amd64/

You should have a VM installation underway pretty quickly and it will be visible via port 5905 on the local host. Enjoy the power and freedom of your brand new type 1 hypervisor.

Xen 4.1 on Fedora 15 with Linux 3.0 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.