Игорь Олемской — практические заметки по системному администрированию Linux CentOS

I'll be the first one to admit that Kerberos drives me a little insane. It's a requirement for two of the exams in Red Hat's RHCA certification track and I've been forced to learn it. It provides some pretty nice security features for large server environments. You get central single sign ons, encrypted authentication, and bidirectional validation. However, getting it configured can be a real pain due to some rather archaic commands and shells.

Here's Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:

• sshd calls to pam to authenticate the user
• pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
• the KDC replies to the client with a TGT encrypted with the user's password
• pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
• if pam can decrypt the TGT, it knows the user is providing the right password

Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?

• client calls the KDC and asks for a ticket to grant access to the other service
• KDC replies with two copies of the ticket:
  • one copy is encrypted with the user's current TGT
  • a second copy is encrypted with the password of the network service the user wants to access
• the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
• client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
• client passes the authenticator and the second copy of the ticket it received from the KDC
• the other network service decrypts the second copy of the ticket and verifies the password
• the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
• if the timestamp looks good, the other network service allows the user access

Okay, that's confusing. Let's take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user's password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user's password back to the KDC before they can ask for a KDC. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they'll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.

Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they're also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client's own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn't have any clue what that password is and the attack will be thwarted.

By this point, you're either saying «Oh man, I don't ever want to do this.» or «How do I set up Kerberos?». Stay tuned if you're in the second group. I'll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.

In the meantime, here are a few links for extra Kerberos bedtime reading:

• Kerberos on Wikipedia
• MIT's «Why Kerberos» [PDF]
• How Kerberos Authentication Works

Kerberos for haters is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Fedora 15 was released with some updates to allow for consistent network device names. Once it's installed, you'll end up with network devices that are named something other than eth0, eth1, and so on.

For example, all onboard ethernet adapters are labeled as emX (em1, em2...) and all PCI ethernet adapters are labeled as pXpX (p[slot]p[port], like p7p1 for port 1 on slot 7). Ethernet devices within Xen virtual machines aren't adjusted.

This may make sense to people who swap out the chassis on servers regularly and they don't want to mess with hard-coding MAC addresses in network configuration files. Also, it should give users predictable names even if a running system's drives are inserted into a newer hardware revision of the same server.

However, I don't like this on my personal dedicated servers and I prefer to revert back to the old way of doing things. Getting back to eth0 is pretty simple and it only requires a few configuration files to be edited followed by a reboot.

First, add biosdevname=0 to your grub.conf on the kernel line:

title Fedora (2.6.40.4-5.fc15.x86_64)
	root (hd0,0)
	kernel /boot/vmlinuz-2.6.40.4-5.fc15.x86_64 ro root=/dev/md0 SYSFONT=latarcyrheb-sun16 KEYTABLE=us biosdevname=0 quiet LANG=en_US.UTF-8
	initrd /boot/initramfs-2.6.40.4-5.fc15.x86_64.img

Open /etc/udev/rules.d/70-persistent-net.rules in your favorite text editor (create it if it doesn't exist) and add in the following:

# Be sure to put your MAC addresses in the fields below
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:10", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:11", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Be sure to rename your ifcfg-* files in /etc/sysconfig/network-scripts/ to match the device names you've assigned. Just for good measure, I add in the MAC address in /etc/sysconfig/network-scripts/ifcfg-ethX:

...
HWADDR=00:11:22:33:44:10
...

Reboot the server and you should be back to eth0 and eth1 after a reboot.

Getting back to using eth0 in Fedora 15 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

It's no secret that I'm a big fan of the RouterBoard network devices paired with Mikrotik's RouterOS. I discovered today that these devices offer Cisco NetFlow-compatible statistics gathering which can be directed to a Linux box running ntop. Mikrotik calls it «traffic flow» and it's much more efficient than setting up a mirrored or spanned port and then using ntop to dump traffic on that interface.

These instructions are for Fedora 15, but they should be pretty similar on most other Linux distributions. Install ntop first:

yum -y install ntop

Adjust /etc/ntop.conf so that ntop listens on something other than localhost:

# limit ntop to listening on a specific interface and port
--http-server 0.0.0.0:3000 --https-server 0.0.0.0:3001

I had to comment out the sched_yield() option to get ntop to start:

# Under certain circumstances, the sched_yield() function causes the ntop web
# server to lock up.  It shouldn't happen, but it does.  This option causes
# ntop to skip those calls, at a tiny performance penalty.
# --disable-schedyield

Set an admin password for ntop:

ntop --set-admin-password

Once you set the password, you may need to press CTRL-C to get back to a prompt in some ntop versions.

Start ntop:

/etc/init.d/ntop start

Open a web browser and open http://example.com:3000 to access the ntop interface. Roll your mouse over the Plugins menu, then NetFlow, and then click Activate. Roll your mouse over the Plugins menu again, then NetFlow, and then click Configure. Click Add NetFlow Device and fill in the following:

• Type «Mikrotik» in the NetFlow Device section and click Set Interface Name.
• Type 2055 in the Local Collector UDP Port section and click Set Port.
• Type in your router's IP/netmask in the Virtual NetFlow Interface Network Address section and click Set Interface Address.

Enabling traffic flow on the Mikrotik can be done with just two configuration lines:

/ip traffic-flow
set enabled=yes interfaces=all
/ip traffic-flow target
add address=192.168.10.65:2055 disabled=no version=5

Wait about a minute and then try reviewing some of the data in the ntop interface. Depending on the amount of traffic on your network, you might see data in as little as 10-15 seconds.

Measure traffic flows with Mikrotik's RouterOS and ntop on Fedora 15 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you find yourself forgetting bits and pieces about network topics, Packet Life's cheat sheets should be a handy resource for you. Lots of topics are included, such as VOIP, NAT, MPLS, BGP, and IOS basics. They're all in PDF form and free to download.

Cheat Sheets — Packet Life

Thanks to @speude for mentioning the site on Twitter.

Handy networking cheat sheets from Packet Life is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

There are few things which will rattle systems administrators more than a compromised server. It gives you the same feeling that you would have if someone broke into your house or car, except that it's much more difficult (with a server) to determine how to clean up the compromise and found out how the attacker gained access. In addition, leaving a compromise in place for an extended period can lead to other problems:

• your server could be used to gain access other servers
• data could be stolen from your server's databases or storage devices
• an attacker could capture data from your server's local network
• denial of service attacks could be launched using your server as an active participant

The best ways to limit your server's attack surface are pretty obvious: limit network access, keep your OS packages up to date, and regularly audit any code which is accessible externally or internally. As we all know, your server can still become compromised even with all of these preventative measures in place.

Here are some tips which will allow you to rapidly detect a compromise on your servers:

Abnormal network usage patterns and atypical bandwidth consumption
Most sites will have a fairly normal traffic pattern which repeats itself daily. If your traffic graph suddenly has a plateau or spikes drastically during different parts of the day, that could signify that there is something worth reviewing. Also, if your site normally consumes about 2TB of traffic per month and you're at the 1.5TB mark on the fifth day of the month, you might want to examine the server more closely.

On the flip side, look for dips in network traffic as well. This may mean that a compromise is interfering with the operation of a particular daemon, or there may be a rogue daemon listening on a trusted port during certain periods.

Many compromises consist of simple scripts which scan for other servers to infect or participate in large denial of service attacks. The scans may show up as a large amount of packets, but the denial of service attacks will usually consume a large amount of bandwidth. Keeping tabs on network traffic is easily done with open source software like munin, cacti, or MRTG.

Unusual open ports
If you run a web server on port 80, but netstat -ntlp shows something listening on various ports over 1024, those processes are worth reviewing. Use commands like lsof to probe the system for the files and network ports held open by the processes. You can also check within /proc/[pid] to find the directory where the processes were originally launched.

Watch out for processes started within directories like /dev/shm, /tmp or any directories in which your daemons have write access. You might see that some processes were started in a user's home directory. If that's the case, it might be a good time to reset that user's password or clear out their ssh key. Review the output from last authentication logs to see if there are account logins from peculiar locations. If you know the user lives in the US, but there are logins from various other countries over a short period, you've got a serious problem.

I've used applications like chkrootkit and rkhunter in the past, but I still prefer a keen eye and netstat on most occasions.

Command output is unusual
I've seen compromises in the past where the attacker actually took the time to replace integral applications like ps, top and lsof to hide the evidence of the ongoing compromise. However, a quick peek in /proc revealed that there was a lot more going on.

If you suspect a compromise like this one, you may want to use the functionality provided by rpm to verify the integrity of the packages currently installed. You can quickly hunt for changed files by running rpm -Va | grep ^..5.

Keeping tabs on changing files can be a challenge, but applications like tripwire and good ol' logwatch can save you in a pinch.

Summary
We can all agree that the best way to prevent a compromise is to take precautions before putting anything into production. In real life, something will always be forgotten, so detection is a must. It's critical to keep in mind that monitoring a server means more than keeping track on uptime. Keeping tabs on performance anomalies will allow you to find the compromise sooner and that keeps the damage done to a minimum.

Strategies for detecting a compromised Linux server is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you're a fan of icanhazip.com, you'll be excited to know that it now supports IPv4 and IPv6 addresses. The icanhazip.com domain now has A and AAAA records, so you should now be able to reach it on connections that are IPv6 only.

Should you find yourself in a situation in which you need to force it to display your IPv4 or IPv6 address (which is handy if you are running a dual stack), simply access ipv4.icanhazip.com or ipv6.icanhazip.com. Those sites will always return your IPv4 and IPv6 addresses, respectively.

As always, if you find any bugs or other problems, be sure to let me know!

icanhazip.com now supports both IPv4 and IPv6 addresses is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Diagram: OpenVPN to Rackspace Cloud Servers and Slicehost

A recent blog post from Mixpanel inspired me to write a quick how-to for Fedora users on using OpenVPN to talk to instances privately in the Rackspace Cloud.

The diagram at the right gives an idea of what this guide will allow you to accomplish. Consider a situation where you want to talk to the MySQL installation on db1 directly without requiring extra ssh tunnels or MySQL over SSL via the public network. If you tunnel into one of your instances, you can utilize the private network to talk between your instances very easily.

There's one important thing to keep in mind here: even though you'll be utilizing the private network between your tunnel endpoint and your other instances, your traffic will still traverse the public network. That means that the instance with your tunnel endpoint will still get billed for the traffic flowing through your tunnel.

You'll only need the openvpn package on the server side:

yum -y install openvpn

Throw down this simple configuration file into /etc/openvpn/server.conf:

port 1194
proto tcp
dev tun
persist-key
persist-tun
 
server 10.66.66.0 255.255.255.0
ifconfig-pool-persist ipp.txt
 
#push "route 10.0.0.0 255.0.0.0"
push "route 10.176.0.0 255.248.0.0"
keepalive 10 120
 
ca      /etc/openvpn/my_certificate_authority.pem
cert    /home/major/vpn_server_cert.pem
key     /home/major/vpn_server_key.pem
dh      /etc/openvpn/easy-rsa/2.0/keys/dh1024.pem
 
status log/openvpn-status.log
verb 3

Here's a bit of explanation for some things you may want to configure:

• push — These are the routes that will be sent over the VPN that are pushed to the clients. If you don't use any IP addresses in the 10.0.0.0/8 network block in your office, you can probably use the commented out line above. However, you may want to be more specific with the routes if you happen to use any 10.0.0.0/8 space in your office.
• server — These are the IP addresses that the VPN server will assign and NAT out through the private interface. I've used a /24 above, but you may want to adjust the netmask if you have a lot of users making tunnels to your VPN endpoint.
• ca, cert, key — You will need to create a certificate authority as well as a certificate/key pair for your VPN endpoint. I already use SimpleAuthority on my Mac to manage some other CA's and certificates, but you can use openvpn's easy-rsa scripts if you wish. They are already included with the openvpn installation.

Build your Diffie-Hellman parameters file:

cd /etc/openvpn/easy-rsa/2.0/ && ./build-dh

Tell iptables that you want to NAT your VPN endpoint traffic out to all 10.x.x.x IP addresses on the private network:

iptables -t nat -A POSTROUTING -s 10.0.0.0/8 -o eth1 -j MASQUERADE

The last step on the server side is to ensure that the kernel will forward packets from the VPN endpoint out through the private interface. Ensure that your /etc/sysctl.conf looks like this:

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

Adjusting your sysctl.conf ensures that forwarding is enabled at boot time, but you'll need to enable it on your VPN endpoint right now:

echo 1 > /proc/sys/net/ipv4/ip_forward

Start the openvpn server:

/etc/init.d/openvpn start

If all is well, you should see openvpn listening on port 1194:

[root@lb2 ~]# netstat -ntlp | grep openvpn
tcp        0      0 0.0.0.0:1194      0.0.0.0:*         LISTEN      2020/openvpn

You'll need to configure a client to talk to your VPN now. This involves three steps: creating a new certificate/key pair for the client (same procedure as making your server certificates), signing the client's certificate with your CA certificate (same one that you used above to sign your server certificates), and then configuring your client application to access the VPN.

There are many openvpn clients out there to choose from.

If you're using a Linux desktop, you may want to consider using the built-in VPN functionality in NetworkManager. For Mac users, I'd highly recommend using Viscosity ($9), but there's also tunnelblick (free).

Accessing Rackspace Cloud Servers and Slicehost slices privately via OpenVPN is a post from: Major Hayden's Racker Hacker blog.

c0b6ad7e-f251-11df-b20b-4040336e00ef

Появилась задача, удаленному офису нужно подключаться к астериску, и тут и там статические «белые» IP адреса. Для решения задачи были добавлены правила:

iptables -A RH-Firewall-1-INPUT -i eth0 -s 91.211.000.000 -j ACCEPT
iptables -A RH-Firewall-1-INPUT -i eth0 -d 91.211.000.000 -j ACCEPT

если нужно открыть доступ для всех внешних подключений по SIP, то правила следующие:

iptables -A INPUT -p udp -i eth0 --dport 5060 -j ACCEPT
iptables -A INPUT -p tcp -i eth0 --dport 5060 -j ACCEPT
iptables -A INPUT -p udp -i eth0 --dport 10000:20000 -j ACCEPT

где eth0 интерфейс с «белым» внешним IP адресом, а 91.211.000.000 это IP адрес удаленного офиса.

Моя домашняя сеть находится в 192.168.1.0/24, а голос вынесен в подсеть 192.168.0.0/24.
доступ «в мир» роутится через 192.168.1.1 а в голос через 192.168.1.111 (да вот захотелось мне так).
Домашние машины получают ip адреса из подсети 192.168.1.0/24, а айпифоны из подсети 192.168.0.0/24

соответственно по дефолту машины не могут получить доступа к фойсовым аппаратам (ATA и IP фоны), прописывать на каждой машине статик роуты, я считаю моветоном. К счастью, не я один так считаю и есть rfc3442 позволяющий с DHCP сервера передавать клиентам статичные маршруты.

итак настроим dhcp сервер давать статик роут на сеть 192.168.0.0/24 через 192.168.1.111

Так как Microsoft компания новатор, срать ей хотелось на RFC. По этому в приведенном выше конфиге присутствуют ms-classless строки. Если у вас в сети нет windows машин, то можно их не уазывать.

На сервере делаем service dhcpd restart, а на клиенте /etc/init.d/networking restart и на клиенте получим:

справедливо это для Ubuntu на клиенте, говорят что в других дистрибутивах нужно в /etc/dhcp3/dhclient.conf добавить:

И в том же файле в поле request добавить параметр rfc3442-classless-static-routes; выглядеть в итоге должно так:

И еще, если нет, то надо создать файл /etc/dhcp3/dhclient-exit-hooks.d/rfc3442-classless-routes

Ну и конечно сделать его исполняемым.

кстати, если нужно передать несколько статичных маршрутов клиентам, то перечислять их надо через запятую, примерно вот так:

И на последок, если используете OpenWRT и сервер dnsmasq, то вам приголится мануал http://www.debian-administration.org/articles/471

Today, on my 28th birthday, I'm finally delivering on a promise to my readers which I made about two months ago. I've written a guide on how to host a web application redundantly in a cloud environment. While it's still a bit of a rough draft, it should be a good starting point for those who haven't worked in virtualized environments before. Also, it may show some of the more experienced systems administrators a new way to do things.

The guide: Redundant Cloud Hosting Guide

As always, if you find anything in the guide that needs improvement, I'm all ears.

A simple guide to redundant cloud hosting is a post from: Major Hayden's Racker Hacker blog.

c0b6ad7e-f251-11df-b20b-4040336e00ef