Игорь Олемской — практические заметки по системному администрированию Linux CentOS

I'm using SELinux more often now on my Fedora 15 installations and I came up against a peculiar issue today on a new server. My PHP installation is configured to store its sessions in memcached and I brought over some working configurations from another server. However, each time I accessed a page which tried to initiate a session, the page load would hang for about a minute and I'd find this in my apache error logs:

[Thu Sep 08 03:23:40 2011] [error] [client 11.22.33.44] PHP Warning:
Unknown: Failed to write session data (memcached). Please verify that
the current setting of session.save_path is correct (127.0.0.1:11211)
in Unknown on line 0

I ran through my usual list of checks:

• netstat showed memcached bound to the correct ports/interfaces
• memcached was running and I could reach it via telnet
• memcached-tool could connect and pull stats from memcached
• double-checked my php.ini
• tested memcached connectivity via a PHP and ruby script — they worked

Even after all that, I still couldn't figure out what was wrong. I ran strace on memcached while I ran a curl against the page which creates a session and I found something significant — memcached wasn't seeing any connections whatsoever at that time. A quick check of the lo interface with tcpdump showed the same result. Just before I threw a chair, I remembered one thing:

SELinux.

A quick check for AVC denials showed the problem:

# aureport --avc | tail -n 1
4021. 09/08/2011 03:23:38 httpd system_u:system_r:httpd_t:s0 42 tcp_socket name_connect system_u:object_r:memcache_port_t:s0 denied 31536

I'm far from being a guru on SELinux, so I leaned on audit2allow for help:

# grep memcache /var/log/audit/audit.log | audit2allow
 
#============= httpd_t ==============
#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_relay, httpd_can_network_memcache, httpd_can_network_connect
 
allow httpd_t memcache_port_t:tcp_socket name_connect;

The boolean we're looking for is httpd_can_network_memcache. Flipping the boolean can be done in a snap:

# setsebool -P httpd_can_network_memcache 1
# getsebool httpd_can_network_memcache
httpd_can_network_memcache --> on

After adjusting the boolean, apache was able to make connections to memcached without a hitch. My page which created sessions loaded quickly and I could see data being stored in memcached. If you want to check the status of all of the apache-related SELinux booleans, just use getsebool:

# getsebool -a | grep httpd | grep off$
allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
httpd_can_check_spam --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_read_user_content --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off

If you're interested in SELinux, a good way to get your feet wet is to head over to the CentOS Wiki and review their SELinux Howtos

Getting apache, PHP, and memcached working with SELinux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

My daily work involves working with a large number of servers and one of my frustrations with Firefox is that it's not possible to select an entire IP address with a double click with the default settings. Although it works right out of the box with Safari, you have to make a configuration adjustment in Firefox to get the same behavior.

To change the setting in Firefox, open up a new Firefox tab and go to about:config in the browser. Paste word_select.stop in the search bar that appears below your tab bar and double click the layout.word_select.stop_at_punctuation line. It should become bold and the value on the end will flip from true to false.

Go back to another tab and open a web page which displays an IP address. Double click on any portion of the IP address and Firefox should highlight the entire address.

Highlight IP addresses with a double click in Firefox is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Мунин идеально подойдет для тех администраторов, которым надо мониторить нагрузку (именно нагрузку, а не доступность!) одного-двух-трех серверов и не хочется заморачиваться с такими монстрами как Cacti.

Мунин состоит из двух частей: Munin-node – демон, который ставится на хост, который надо мониторить – она собирает всю информацию о системе. Вторая часть – пакет munin – он уже рисует графики по данным с одной или нескольких машин и генерит из них статические странички и картинки.
Итак, сначала установим пакет munin, к слову, это перловый скрипт и ставит с собой несколько перловых библиотек:

Он пропишется в /etc/cron.d для запуска раз в 5 минут, создаст /var/www/html/munin – туда будет генериться статика, которую можно отдавать любым веб-сервером. Конфиги мунина лежат в /etc/munin. Прописываем мунин в Апаче, можно просто алиасом к каком-нибудь существующему виртуалхосту:

Релоадим апач и устанавливаем munin-node (эту часть проделываем на всех серверах, которые надо мониторить):

Если мы ставим оба пакета на одну машину, то нижесказанное проделывать не надо – все работает по дефолту.
Munin-node работает как демон, к которому на порт 4949 периодически подключается munin, поэтому в /etc/munin/munin-node.conf добавляем дериктиву allow ^127\.0\.0\.1$ с адресом munin.

На машине, на которой установлен Munin в /etc/munin/munin.conf добавляем новый хост по типу:

Quite a few people who couldn't make it to Xen Summit 2011 this year asked me to write a post summarizing my takeaways from the event. I'm not generally one to back down from peer pressure, so read on if you're interested about the discussions at this year's Summit.

The feeling I had at last year's summit is that Xen was on the verge of losing traction in the market. Very few distributions still had Xen support going forward and much of the discussion was around the lack of dom0 support in upstream Linux kernels. Distribution vendors were hesitant to drag patches forward into modern kernels and this made it much more difficult to get Xen working for many people.

This year was quite different. The number of attendees was up, the venue was much better, and there was an obvious buzz of energy in the room. As many of the presenters noted, this excitement stemmed from the upstream dom0 support in Linux 3.0. This inclusion is a huge win and it helps to drive Xen forward since the developers don't have to worry about dragging patches forward. They can focus on improving performance, adding features, and tightening security.

Many of the discussions this year focused on security and performance. Ian Pratt discussed Xen's ability to view memory pages of virtual machines via an API to detect malware running inside the instance. Memory pages could be identified and marked as not executable or applications could be triggered when a VM attempts to touch a particular memory page. Also, the whole VM could be frozen if needed.

There's also a big push to bring code out of the dom0 and push it into utility VM's. Driver domains could manage the network or I/O infrastructure and this would further reduce the amount of privileged code actively running in dom0. There is already very little code required for the Xen hypervisor itself (much much less than the Linux kernel — I'm looking at you, KVM) and this reduces the attack surface for potential compromises of the hypervisor. Some projects even aim to restart driver domains multiple times per minute to ensure that any malicious code injected into those virtual machines can't exist for long periods.

Pradeep Vincent from Amazon talked about how Amazon uses Xen and the pain points they have with its current architecture. Much of his discussion was around scaling problems (and we see many of the same issues at Rackspace). Higher performance could easily be gained by multi-threaded operations in dom0 when attaching block devices and creating virtual network interfaces. He also saw some areas for performance gains in the pvops I/O code.

Quite a few of the talks centered on the ARM architecture and what Xen is able to do on those systems after Samsung published their port in 2008. HVM is on the way for ARM and it might even show up in Xen 4.2. Some demos of Xen on mobile phones from Samsung were amazing. They showed how an attacker could compromise the web browser on the phone with a keylogger, but that application was running in a VM. Once the user switched back to the phone's main menu, the keylogger couldn't access the keystrokes any longer. After that, a simple close of the browser killed the VM and destroyed the malicious code.

Xen 4.2 should be available in early 2012 and the feature list is staggering. Improvements to libxenlight, pvops performance (even in HVM), and guest memory sharing should be available with the new release. Nested virtualization (run a hypervisor inside a hypervisor) is also coming in Xen 4.2 and I'm sure Xzibit will be a huge fan. This should streamline hypervisor testing, allow for embedded hypervisor options and extend the capabilities of client hypervisors. Remus should be available in 4.2 as well, but it might be marked as experimental. OVMF will be added as a BIOS option for UEFI (along with the standard SeaBIOS) and this should allow for Mac OS X guests. UEFI allows Windows to boot faster since it switches to PV mode sooner and it allows for simpler platform certification for software vendors.

Mike McClurg's presentation on XCP was pretty important to me since Rackspace is a big consumer of XenServer. If you're not familiar with XCP, it's basically open-source XenServer which runs on bleeding edge (and sometimes unstable) components. XCP 1.5 and XenServer 6 should be available in November with Xen 4.1 and Linux 2.6.32. GPU passthrough, up to 1TB RAM, and disaster recovery will be available. Another goal for the XCP team is to work closely with OpenStack via Project Olympus. Mike's vision is to have XCP become the configuration of choice for open source clouds. Project Kronos was also extremely interesting. It's essentially XCP's XenAPI stack running on Debian and Ubuntu. You'd be able to install either OS on a physical server and run XCP's services on it for a fully OSS hypervisor.

Konrad Wilk gave an update on Linux pvops and it appears there is a shift to get Xen working well on a desktop. This includes 3D graphics support, S3/hibernate capabilities and various bug fixes. There's also a push to get PV functionality into HVM and get HVM functionality into PV. Driver/device domains were discussed again in Patrick Kolp's talk and he had plenty of graphs showing performance changes when regularly restarting device domains. The performance dips were almost negligible with 10 second restarts and the security gains were significant.

There were several other great presentations on other topics like GlusterFS, OpenStack Nova, and Linpicker (from the NSA!). If these types of things interests you, keep your eyes peeled for Xen Summit 2012 next year. The weather in the bay area is well worth the trip.

Xen Summit 2011: My Takeways is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you haven't noticed already, full Xen dom0 support was added in the Linux 3.0 kernel. This means there's no longer a need to drag patches forward from old kernels and work from special branches and git repositories when building a kernel for dom0.

Something else you might not have noticed is that the Fedora kernel team has quietly slipped Linux 3.0 into Fedora 15's update channels in disguise. Click that link, scroll down, and you'll see «Rebase to 3.0. Version reports as 2.6.40 for compatibility with older userspace.» Although I'm not a fan of calling something what it isn't (2.6.40 doesn't exist on kernel.org), I can understand some of the reasoning behind the choice.

This change makes the Xen installation on Fedora 15 pretty trivial. To get started, update your kernel to the latest if you're not already on Fedora's 2.6.40 kernels:

yum -y upgrade kernel

We need three more packages (quite a few dependencies will roll in with them):

yum -y install xen libvirt python-virtinst

The xen package reels in the hypervisor itself along with libraries and command line tools (like xl and xm). Libvirt gives us easy access to VM management with the virsh command and python-virtinst gives us the handy virt-install command to make OS installations easy.

Once those packages are installed, we need to make some adjustments in your grub configuration. Open /boot/grub/menu.lst in your text editor of choice and add something like this at the bottom:

title Fedora + Xen (2.6.40-4.fc15.x86_64)
        root (hd0,1)
	kernel /boot/xen.gz
        module /boot/vmlinuz-2.6.40-4.fc15.x86_64 ro root=/dev/sda1
        module /boot/initramfs-2.6.40-4.fc15.x86_64.img

Ensure that the root (hd0,1) is applicable to your system (adjust it if it isn't). Also, check the kernel version to ensure it matches your installed kernel and adjust the root= portion to match your root volume. Flip the default line to a value which will boot your new grub entry and ensure the timeout is set to a reasonable number if you need to temporarily switch back to your original grub entry at boot time. (Hey, we all make mistakes.)

I take one extra precaution and change the UPDATEDEFAULT=yes line to no in /etc/sysconfig/kernel. This ensures that future kernel updates don't trample the entry you've just made. Keep in mind that you'll need to manually update your grub configuration when you do kernel upgrades later.

Cross your fingers and reboot. If your system doesn't reboot properly, reboot it again and choose your old kernel from the grub menu. Double-check your configuration for fat-fingering and give it another try. If your system boots and pings but you have no output via a monitor, don't fret. There's a patch for the problem which should appear soon in Linux 3.0. The impatient can snag a kernel source RPM, add the patch file, and build a local kernel (or you can download my local build from when I did it).

Log in and verify that you booted into the dom0:

[root@xenbox ~]# xm dmesg | head -n 5
 __  __            _  _    _   _   ____     __      _ ____
 \ \/ /___ _ __   | || |  / | / | |___ \   / _| ___/ | ___|
  \  // _ \ '_ \  | || |_ | | | |__ __) | | |_ / __| |___ \
  /  \  __/ | | | |__   _|| |_| |__/ __/ _|  _| (__| |___) |
 /_/\_\___|_| |_|    |_|(_)_(_)_| |_____(_)_|  \___|_|____/

Once you're done with that, make sure libvirtd is running:

/etc/init.d/libvirtd start; chkconfig libvirtd on

Try installing a VM:

virt-install \
  --paravirt \
  --name=testvm \
  --ram=512 \
  --vcpus=4 \
  --file /dev/vmstorage/testvm \
  --graphics vnc,port=5905 --noautoconsole \
  --autostart --noreboot \
  --location=http://mirrors.kernel.org/debian/dists/squeeze/main/installer-amd64/

You should have a VM installation underway pretty quickly and it will be visible via port 5905 on the local host. Enjoy the power and freedom of your brand new type 1 hypervisor.

Xen 4.1 on Fedora 15 with Linux 3.0 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Тут надо было в два часа ночи продержаться не засыпая еще два часика до четырех. После этого следовало обработать кое какое событие и дальше спать. Но спички в глазах уже ломались и я стал нервно шарить вокруг в поисках мобильника, дабы включить будильник. Мобильника я не нашел. Задумался… «Есть ли будильник в линуксе?» – спросило само себя тухнущее сознание сисадмина. Уже автоматически стал открывать вкладку хрома, как вдруг нашло озарение. Открываю терминал, пишу…

#!/bin/bash
while /bin/true ; do
   /usr/bin/mplayer "/home/user/music/cf/Cradle Of Filth - Nymphetamine.mp3"
   sleep 5
done

Сохраняю, ставлю в крон на нужное время и спаааааать. mplayer в списке процессов я смогу покилять точно когда проснусь.

Да, за это я люблю Linux.

Some might call me paranoid, but I get nervous when my package manager automatically removes a kernel. I logged into my Fedora 15 VM this morning and found this:

================================================================================
 Package        Arch           Version                   Repository        Size
================================================================================
Installing:
 kernel         x86_64         2.6.35.13-92.fc14         updates           22 M
Removing:
 kernel         x86_64         2.6.35.11-83.fc14         @updates         104 M
 
Transaction Summary
================================================================================
Install       1 Package(s)
Remove        1 Package(s)

Fedora 15's default behavior is to keep three kernels: the latest one and the two previous versions. However, this behavior may be counter-productive if you compile your own modules, or if you have compatibility issues with subsequent kernel versions.

You can change how yum handles kernel packages with some simple changes to your /etc/yum.conf. The installonly_limit option controls how many old packages are kept:

installonly_limit Number of packages listed in installonlypkgs to keep installed at the same time. Setting to 0 disables this feature. Default is '0'.

I disabled the functionality altogether by setting installonly_limit to 0:

#installonly_limit=3
installonly_limit=0

It's important to keep in mind that you will need to purge these packages from your system yourself now. Kernel packages can occupy a fair amount of disk space, so make a note to go back and clean them up when you no longer need them.

Keep all old kernels when upgrading via yum is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

It's no secret that I'm a big fan of the RouterBoard network devices paired with Mikrotik's RouterOS. I discovered today that these devices offer Cisco NetFlow-compatible statistics gathering which can be directed to a Linux box running ntop. Mikrotik calls it «traffic flow» and it's much more efficient than setting up a mirrored or spanned port and then using ntop to dump traffic on that interface.

These instructions are for Fedora 15, but they should be pretty similar on most other Linux distributions. Install ntop first:

yum -y install ntop

Adjust /etc/ntop.conf so that ntop listens on something other than localhost:

# limit ntop to listening on a specific interface and port
--http-server 0.0.0.0:3000 --https-server 0.0.0.0:3001

I had to comment out the sched_yield() option to get ntop to start:

# Under certain circumstances, the sched_yield() function causes the ntop web
# server to lock up.  It shouldn't happen, but it does.  This option causes
# ntop to skip those calls, at a tiny performance penalty.
# --disable-schedyield

Set an admin password for ntop:

ntop --set-admin-password

Once you set the password, you may need to press CTRL-C to get back to a prompt in some ntop versions.

Start ntop:

/etc/init.d/ntop start

Open a web browser and open http://example.com:3000 to access the ntop interface. Roll your mouse over the Plugins menu, then NetFlow, and then click Activate. Roll your mouse over the Plugins menu again, then NetFlow, and then click Configure. Click Add NetFlow Device and fill in the following:

• Type «Mikrotik» in the NetFlow Device section and click Set Interface Name.
• Type 2055 in the Local Collector UDP Port section and click Set Port.
• Type in your router's IP/netmask in the Virtual NetFlow Interface Network Address section and click Set Interface Address.

Enabling traffic flow on the Mikrotik can be done with just two configuration lines:

/ip traffic-flow
set enabled=yes interfaces=all
/ip traffic-flow target
add address=192.168.10.65:2055 disabled=no version=5

Wait about a minute and then try reviewing some of the data in the ntop interface. Depending on the amount of traffic on your network, you might see data in as little as 10-15 seconds.

Measure traffic flows with Mikrotik's RouterOS and ntop on Fedora 15 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Вы видели когда нибудь папочку весом 1Gb? Имеется в виду вес не содержимого низлежащих файлов, а вес самой папочки, т.е. листинг имен файлов весом 1Gb… около 13млн инод внутри…

hosted-by:/> stat /var/www/admin/data/mod-tmp
  File: `/var/www/admin/data/mod-tmp'
  Size: 1022976000      Blocks: 1999960    IO Block: 4096   directory
Device: 802h/2050d      Inode: 11403505    Links: 2
Access: (2700/drwx--S---)  Uid: (   33/www-data)   Gid: (  502/   admin)
Access: 2011-05-30 23:44:44.000000000 +0300
Modify: 2011-05-30 08:36:27.000000000 +0300
Change: 2011-05-30 08:36:27.000000000 +0300

При попытке просмотра папочки, ls задумывается минут на 20, а потом выдает:

hosted-by:/> ls -1 /var/www/admin/data/mod-tmp > /tmp/ls
ls: memory exhausted

Поступили жалобы на «некорректную работу сервера». Да я думаю некорректная…

hosted-by:/> df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda2            14884864 14884864       0  100% /

Найти потенциальных пожирателей инод смог по команде

find / -type d -size +100k -print

На этой папочке find заткнулся навечно, что навело на мысль.

Диагноз: рак. Лечение: ампутация.

screen
rm -rf /var/www/admin/data/mod-tmp && mkdir /var/www/admin/data/mod-tmp && chmod 777 /var/www/admin/data/mod-tmp

При скорости удаления около 100 файлов в секунду, а файлов более 10 млн, можно идти спать….

Дальнейшее вскрытие показало что в папочке были файлы сессий php. На Debian Lenny почему то выпиливатель старых файлов сессий выключен по дефолту:

hosted-by:~> grep -B 5 session.gc_probability /etc/php5/apache2/php.ini
 
; This is disabled in the Debian packages, due to the strict permissions
; on /var/lib/php5.  Instead of setting this here, see the cronjob at
; /etc/cron.d/php5, which uses the session.gc_maxlifetime setting below.
; php scripts using their own session.save_path should make sure garbage
; collection is enabled by setting session.gc_probability
;session.gc_probability = 0

Вопиющее распиздяйство. Хотя скорей всего дело в нестандартном расположении session.save_path, который ставит ISP Manager.

The Guide to IP Layer Network Administration with Linux was conceived as a project to document and provide examples for the use of Linux-specific IP networking tools.

The book includes coverage of the following topics: network address translation (NAT), address resolution protocol (ARP), routing, and masquerading. There is a nascent section on bridging, although the focus of the documentation is the use of layer 3 Linux technologies.

Original news source