Игорь Олемской — практические заметки по системному администрированию Linux CentOS

It's been a few years since I started a little project to operate a service to return your IPv4 and IPv6 address. Although there are a bunch of other sites that offer this service as well, I've been amazed by the gradually increasing traffic to icanhazip.com.

Here's a sample of the latest statistics:

• Hits per day: 1.8 million (about 21 hits/second)
• Unique IP addresses per day: 25,555
• Hits per day from IPv6 addresses: 1,069 (a little sad)
• Bandwidth used per day: ~ 400MB

The site is now running on multiple Cloud Servers at Rackspace behind a load balancer cluster. In addition, the DNS records are hosted with Rackspace's Cloud DNS service.

This should allow the site to reply more quickly and reliably. If you have suggestions for other improvements, let me know!

Performance and redundancy boost for icanhazip.com is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

One of the handiest tools in the OpenSSL toolbox is s_client. You can quickly view lots of details about the SSL certificates installed on a particular server and diagnose problems. For example, use this command to look at Google's SSL certificates:

openssl s_client -connect encrypted.google.com:443

You'll see the chain of certificates back to the original certificate authority where Google bought its certificate at the top, a copy of their SSL certificate in plain text in the middle, and a bunch of session-related information at the bottom.

This works really well when a site has one SSL certificate installed per IP address (this used to be a hard requirement). With Server Name Indication (SNI), a web server can have multiple SSL certificates installed on the same IP address. SNI-capable browsers will specify the hostname of the server they're trying to reach during the initial handshake process. This allows the web server to determine the correct SSL certificate to use for the connection.

If you try to connect to rackerhacker.com with s_client, you'll find that you receive the default SSL certificate installed on my server and not the one for this site:

$ openssl s_client -connect rackerhacker.com:443
Certificate chain
 0 s:/C=US/ST=Texas/L=San Antonio/O=MHTX Enterprises/CN=*.mhtx.net
   i:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
 1 s:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
   i:/C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority

Add on the -servername argument and s_client will do the additional SNI negotiation step for you:

$ openssl s_client -connect rackerhacker.com:443 -servername rackerhacker.com
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=rackerhacker.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
 2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

You may be asking yourself this question:

Why doesn't the web server just use the Host: header that my browser sends already to figure out which SSL certificate to use?

Keep in mind that the SSL negotiation must occur prior to sending the HTTP request through to the remote server. That means that the browser and the server have to do the certificate exchange earlier in the process and the browser wouldn't get the opportunity to specify which site it's trying to reach. SNI fixes that by allowing a Host: header type of exchange during the SSL negotiation process.

Using OpenSSL's s_client command with web servers using Server Name Indication (SNI) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

As promised in my earlier post entitled Kerberos for haters, I've assembled the simplest possible guide to get Kerberos up an running on two CentOS 5 servers.

Also, I don't really hate Kerberos. It's a bit of an inside joke with my coworkers who are studying for some of the RHCA exams at Rackspace. The additional security provided by Kerberos is quite good but the setup involves a lot of small steps. If you miss one of the steps or if you get something done out of order, you may have to scrap the whole setup and start over unless you can make sense of the errors in the log files. A lot of my dislikes for Kerberos comes from the number of steps required in the setup process and the difficulty in tracking down issues when they crop up.

To complete this guide, you'll need the following:

• two CentOS, Red Hat Enterprise Linux or Scientific Linux 5 servers or VM's
• some patience

Here's how I plan to name my servers:

• kdc.example.com — the Kerberos KDC server at 192.168.250.2
• client.example.com — the Kerberos client at 192.168.250.3

CRITICAL STEP: Before getting started, ensure that both systems have their hostnames properly set and both systems have the hostnames and IP addresses of both systems in /etc/hosts. Your server and client must be able to know the IP and hostname of the other system as well as themselves.

First off, we will need NIS working to serve up the user information for our client. Install the NIS server components on the KDC server:

[root@kdc ~]# yum install ypserv

Set the NIS domain and set a static port for ypserv to make it easier to firewall off. Edit /etc/sysconfig/network on the KDC server:

NISDOMAINNAME=EXAMPLE.COM
YPSERV_ARGS="-p 808"

Manually set the NIS domain on the KDC server and add it to /etc/yp.conf:

[root@kdc ~]# nisdomain EXAMPLE.COM
[root@kdc ~]# echo "domain EXAMPLE.COM server kdc.example.com" >> /etc/yp.conf

Adjust /var/yp/securenets on the KDC server for additional security:

[root@kdc ~]# echo "255.0.0.0 127.0.0.0" >> /var/yp/securenets
[root@kdc ~]# echo "255.255.255.0 192.168.250.0" >> /var/yp/securenets

Start the NIS server and generate the NIS maps:

[root@kdc ~]# /etc/init.d/ypserv start; chkconfig ypserv on
[root@kdc ~]# make -C /var/yp

I usually like to prepare my iptables rules ahead of time so I ensure that it doesn't derail me later on. Paste this into the KDC's terminal:

iptables -N SERVICES
iptables -I INPUT -j SERVICES
iptables -A SERVICES -p tcp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p udp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p tcp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p udp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p tcp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 464 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p tcp --dport 749 -j ACCEPT -m comment --comment "kerberos"
/etc/init.d/iptables save

We need our time in sync for Kerberos to work properly. Install NTP on both nodes, start it, and ensure it comes up at boot time:

[root@kdc ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start
[root@client ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start

Now we're ready to set up Kerberos. Start by installing some packages on the KDC:

[root@kdc ~]# yum install krb5-server krb5-workstation

We will need to make some edits to /etc/krb5.conf on the KDC to set up our KDC realm. Ensure that the default_realm is set:

default_realm = EXAMPLE.COM

The [realms] section should look like this:

[realms]
EXAMPLE.COM = {
	kdc = 192.168.250.2:88
	admin_server = 192.168.250.2:749
}

The [domain_realm] section should look like this:

[domain_realm]
kdc.example.com = EXAMPLE.COM
client.example.com = EXAMPLE.COM

Add validate = true within the pam { } block of the [appdefaults] section:

[appdefaults]
 pam = {
   validate = true

Adjust /var/kerberos/krb5kdc/kdc.conf on the KDC:

[realms]
EXAMPLE.COM = {
	master_key_type = des-hmac-sha1
	default_principal_flags = +preauth
}

There's one last configuration file to edit on the KDC! Ensure that /var/kerberos/krb5kdc/kadm5.acl looks like this:

*/admin@EXAMPLE.COM	    *

We're now ready to make a KDC database to hold our sensitive Kerberos data. Create the database and set a good password which you can remember. This command also stashes your password on the KDC so you don't have to enter it each time you start the KDC:

kdb5_util create -r EXAMPLE.COM -s

On the KDC, create a principal for the admin user as well as user1 (which we'll create shortly). Also, export the admin details to the kadmind key tab. You'll get some extra output after each one of these commands but I've snipped it to reduce the length of the post.

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc root/admin
kadmin.local:  addprinc user1
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/admin
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/changepw
kadmin.local:  exit

Let's start the Kerberos KDC and kadmin daemons:

[root@kdc ~]# /etc/init.d/krb5kdc start; /etc/init.d/kadmin start
[root@kdc ~]# chkconfig krb5kdc on; chkconfig kadmin on

Now that the administration work is done, let's create a principal for our KDC server and stick it in it's keytab:

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc -randkey host/kdc.example.com
kadmin.local:  ktadd host/kdc.example.com

Transfer your /etc/krb5.conf from the KDC server to the client. Hop onto the client server, install the Kerberos client package and add some host principals:

[root@client ~]# yum install krb5-workstation
[root@client ~]# kadmin.local
kadmin.local:  addpinc --randkey host/client.example.com
kadmin.local:  ktadd host/kdc.example.com

There aren't any daemons on the client side, so the configuration is pretty much wrapped up there for Kerberos. However, we now need to tell both servers to use Kerberos for auth and your client servers needs to use NIS to get user data.

• On the KDC:
  • run authconfig-tui
  • choose Use Kerberos from the second column
  • press Next
  • don't edit the configuration (authconfig got the data from /etc/krb.conf)
  • press OK
• On the client:
  • run authconfig-tui
  • choose Use NIS and Use Kerberos
  • press Next
  • enter your NIS domain (EXAMPLE.COM) and NIS server (kdc.example.com or 192.168.250.2)
  • press Next
  • don't edit the Kerberos configuration (authconfig got the data from /etc/krb.conf)
  • press OK

Got NIS problems? If the NIS connection stalls on the client, ensure that you have the iptables rules present on the KDC that we added near the beginning of this guide. Also, if you forgot to add both hosts to both servers' /etc/hosts, go do that now.

Let's make our test user on the KDC. Don't add this user to the client — we'll get the user information via NIS and authenticate via Kerberos shortly. We'll also rebuild our NIS maps after adding the user:

[root@kdc ~]# useradd user1
[root@kdc ~]# passwd user1
[root@kdc ~]# make -C /var/yp/

On the client, see if you can get the password hash for the user1 account via NIS:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:$1$sUlSTlCv$riK5El3z8N4y.mi5Fe3Q60:500:500::/home/user1:/bin/bash

You can see why NIS isn't a good way to authenticate users. Someone could easily pull the hash for any account and brute force the hash on their own server. Go back to the KDC and lock out the user account:

[root@kdc ~]# usermod -p '!!' user1

Go back to the client and try to pull the password hash now:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:!!:500:500::/home/user1:/bin/bash

On the plus side, the user's password hash is now gone. On the negative side, you've just prevented this user from logging in locally or via NIS. Don't worry, the user can log in via Kerberos now. Let's prepare a home directory on the client for the user:

[root@client ~]# mkdir /home/user1
[root@client ~]# cp -av /etc/skel/.bash* /home/user1/
[root@client ~]# chown -R user1:user1 /home/user1/

Note: In a real-world scenario, you'd probably want to export this user's home directory via NFS so they didn't get a different home directory on every server.

While you're still on the client, try to log into the client via the user. Use the password that you used when you created the user1 principal on the KDC.

[root@client ~]# ssh user1@localhost
user1@localhost's password:
[user1@client ~]$ whoami
user1

List your Kerberos tickets and you should see one for your user principal:

[user1@client ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500_fCKPnZ
Default principal: user1@EXAMPLE.COM
 
Valid starting     Expires            Service principal
02/05/12 14:18:53  02/06/12 00:18:53  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 02/05/12 14:18:53

Your KDC should have a couple of lines in its /var/log/krb5kdc.log showing the authentication:

Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): AS_REQ (12 etypes {18 17 16 23 1 3 2 11 10 15 12 13}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=16 ses=16}, user1@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=18 ses=18}, user1@EXAMPLE.COM for host/client.example.com@EXAMPLE.COM

The first line shows that the client asked for a Authentication Server Request (AS_REQ) and the second line shows that the client then asked for a Ticket Granting Server Request (TGS_REQ). In layman's terms, the client first asked for a ticket-granting ticket (TGT) so it could authenticate to other services. When it actually tried to log in via ssh it asked for a ticket (and received it).

YOU JUST CONFIGURED KERBEROS!

From here, the sky's the limit. Another popular implementation of Kerberos is encrypted NFSv4. You can even go crazy and use Kerberos with apache.

Let me know if you have any questions about this post or if you spot any errors. With this many steps, there's bound to be a typo or two in this guide. Keep in mind that there are some obvious spots for network-level and service-level security improvements. This guide was intended to give you the basics and it doesn't cover all of the security implications involved with a Kerberos implementation.

The Kerberos-hater's guide to installing Kerberos is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I'll be the first one to admit that Kerberos drives me a little insane. It's a requirement for two of the exams in Red Hat's RHCA certification track and I've been forced to learn it. It provides some pretty nice security features for large server environments. You get central single sign ons, encrypted authentication, and bidirectional validation. However, getting it configured can be a real pain due to some rather archaic commands and shells.

Here's Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:

• sshd calls to pam to authenticate the user
• pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
• the KDC replies to the client with a TGT encrypted with the user's password
• pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
• if pam can decrypt the TGT, it knows the user is providing the right password

Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?

• client calls the KDC and asks for a ticket to grant access to the other service
• KDC replies with two copies of the ticket:
  • one copy is encrypted with the user's current TGT
  • a second copy is encrypted with the password of the network service the user wants to access
• the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
• client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
• client passes the authenticator and the second copy of the ticket it received from the KDC
• the other network service decrypts the second copy of the ticket and verifies the password
• the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
• if the timestamp looks good, the other network service allows the user access

Okay, that's confusing. Let's take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user's password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user's password back to the KDC before they can ask for a TGT. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they'll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.

Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they're also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client's own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn't have any clue what that password is and the attack will be thwarted.

By this point, you're either saying «Oh man, I don't ever want to do this.» or «How do I set up Kerberos?». Stay tuned if you're in the second group. I'll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.

In the meantime, here are a few links for extra Kerberos bedtime reading:

• Kerberos on Wikipedia
• MIT's «Why Kerberos» [PDF]
• How Kerberos Authentication Works

Kerberos for haters is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Fedora 15 was released with some updates to allow for consistent network device names. Once it's installed, you'll end up with network devices that are named something other than eth0, eth1, and so on.

For example, all onboard ethernet adapters are labeled as emX (em1, em2...) and all PCI ethernet adapters are labeled as pXpX (p[slot]p[port], like p7p1 for port 1 on slot 7). Ethernet devices within Xen virtual machines aren't adjusted.

This may make sense to people who swap out the chassis on servers regularly and they don't want to mess with hard-coding MAC addresses in network configuration files. Also, it should give users predictable names even if a running system's drives are inserted into a newer hardware revision of the same server.

However, I don't like this on my personal dedicated servers and I prefer to revert back to the old way of doing things. Getting back to eth0 is pretty simple and it only requires a few configuration files to be edited followed by a reboot.

First, add biosdevname=0 to your grub.conf on the kernel line:

title Fedora (2.6.40.4-5.fc15.x86_64)
	root (hd0,0)
	kernel /boot/vmlinuz-2.6.40.4-5.fc15.x86_64 ro root=/dev/md0 SYSFONT=latarcyrheb-sun16 KEYTABLE=us biosdevname=0 quiet LANG=en_US.UTF-8
	initrd /boot/initramfs-2.6.40.4-5.fc15.x86_64.img

Open /etc/udev/rules.d/70-persistent-net.rules in your favorite text editor (create it if it doesn't exist) and add in the following:

# Be sure to put your MAC addresses in the fields below
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:10", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:11:22:33:44:11", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Be sure to rename your ifcfg-* files in /etc/sysconfig/network-scripts/ to match the device names you've assigned. Just for good measure, I add in the MAC address in /etc/sysconfig/network-scripts/ifcfg-ethX:

...
HWADDR=00:11:22:33:44:10
...

Reboot the server and you should be back to eth0 and eth1 after a reboot.

Getting back to using eth0 in Fedora 15 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

It's no secret that I'm a big fan of the RouterBoard network devices paired with Mikrotik's RouterOS. I discovered today that these devices offer Cisco NetFlow-compatible statistics gathering which can be directed to a Linux box running ntop. Mikrotik calls it «traffic flow» and it's much more efficient than setting up a mirrored or spanned port and then using ntop to dump traffic on that interface.

These instructions are for Fedora 15, but they should be pretty similar on most other Linux distributions. Install ntop first:

yum -y install ntop

Adjust /etc/ntop.conf so that ntop listens on something other than localhost:

# limit ntop to listening on a specific interface and port
--http-server 0.0.0.0:3000 --https-server 0.0.0.0:3001

I had to comment out the sched_yield() option to get ntop to start:

# Under certain circumstances, the sched_yield() function causes the ntop web
# server to lock up.  It shouldn't happen, but it does.  This option causes
# ntop to skip those calls, at a tiny performance penalty.
# --disable-schedyield

Set an admin password for ntop:

ntop --set-admin-password

Once you set the password, you may need to press CTRL-C to get back to a prompt in some ntop versions.

Start ntop:

/etc/init.d/ntop start

Open a web browser and open http://example.com:3000 to access the ntop interface. Roll your mouse over the Plugins menu, then NetFlow, and then click Activate. Roll your mouse over the Plugins menu again, then NetFlow, and then click Configure. Click Add NetFlow Device and fill in the following:

• Type «Mikrotik» in the NetFlow Device section and click Set Interface Name.
• Type 2055 in the Local Collector UDP Port section and click Set Port.
• Type in your router's IP/netmask in the Virtual NetFlow Interface Network Address section and click Set Interface Address.

Enabling traffic flow on the Mikrotik can be done with just two configuration lines:

/ip traffic-flow
set enabled=yes interfaces=all
/ip traffic-flow target
add address=192.168.10.65:2055 disabled=no version=5

Wait about a minute and then try reviewing some of the data in the ntop interface. Depending on the amount of traffic on your network, you might see data in as little as 10-15 seconds.

Measure traffic flows with Mikrotik's RouterOS and ntop on Fedora 15 is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you find yourself forgetting bits and pieces about network topics, Packet Life's cheat sheets should be a handy resource for you. Lots of topics are included, such as VOIP, NAT, MPLS, BGP, and IOS basics. They're all in PDF form and free to download.

Cheat Sheets — Packet Life

Thanks to @speude for mentioning the site on Twitter.

Handy networking cheat sheets from Packet Life is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

There are few things which will rattle systems administrators more than a compromised server. It gives you the same feeling that you would have if someone broke into your house or car, except that it's much more difficult (with a server) to determine how to clean up the compromise and found out how the attacker gained access. In addition, leaving a compromise in place for an extended period can lead to other problems:

• your server could be used to gain access other servers
• data could be stolen from your server's databases or storage devices
• an attacker could capture data from your server's local network
• denial of service attacks could be launched using your server as an active participant

The best ways to limit your server's attack surface are pretty obvious: limit network access, keep your OS packages up to date, and regularly audit any code which is accessible externally or internally. As we all know, your server can still become compromised even with all of these preventative measures in place.

Here are some tips which will allow you to rapidly detect a compromise on your servers:

Abnormal network usage patterns and atypical bandwidth consumption
Most sites will have a fairly normal traffic pattern which repeats itself daily. If your traffic graph suddenly has a plateau or spikes drastically during different parts of the day, that could signify that there is something worth reviewing. Also, if your site normally consumes about 2TB of traffic per month and you're at the 1.5TB mark on the fifth day of the month, you might want to examine the server more closely.

On the flip side, look for dips in network traffic as well. This may mean that a compromise is interfering with the operation of a particular daemon, or there may be a rogue daemon listening on a trusted port during certain periods.

Many compromises consist of simple scripts which scan for other servers to infect or participate in large denial of service attacks. The scans may show up as a large amount of packets, but the denial of service attacks will usually consume a large amount of bandwidth. Keeping tabs on network traffic is easily done with open source software like munin, cacti, or MRTG.

Unusual open ports
If you run a web server on port 80, but netstat -ntlp shows something listening on various ports over 1024, those processes are worth reviewing. Use commands like lsof to probe the system for the files and network ports held open by the processes. You can also check within /proc/[pid] to find the directory where the processes were originally launched.

Watch out for processes started within directories like /dev/shm, /tmp or any directories in which your daemons have write access. You might see that some processes were started in a user's home directory. If that's the case, it might be a good time to reset that user's password or clear out their ssh key. Review the output from last authentication logs to see if there are account logins from peculiar locations. If you know the user lives in the US, but there are logins from various other countries over a short period, you've got a serious problem.

I've used applications like chkrootkit and rkhunter in the past, but I still prefer a keen eye and netstat on most occasions.

Command output is unusual
I've seen compromises in the past where the attacker actually took the time to replace integral applications like ps, top and lsof to hide the evidence of the ongoing compromise. However, a quick peek in /proc revealed that there was a lot more going on.

If you suspect a compromise like this one, you may want to use the functionality provided by rpm to verify the integrity of the packages currently installed. You can quickly hunt for changed files by running rpm -Va | grep ^..5.

Keeping tabs on changing files can be a challenge, but applications like tripwire and good ol' logwatch can save you in a pinch.

Summary
We can all agree that the best way to prevent a compromise is to take precautions before putting anything into production. In real life, something will always be forgotten, so detection is a must. It's critical to keep in mind that monitoring a server means more than keeping track on uptime. Keeping tabs on performance anomalies will allow you to find the compromise sooner and that keeps the damage done to a minimum.

Strategies for detecting a compromised Linux server is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you're a fan of icanhazip.com, you'll be excited to know that it now supports IPv4 and IPv6 addresses. The icanhazip.com domain now has A and AAAA records, so you should now be able to reach it on connections that are IPv6 only.

Should you find yourself in a situation in which you need to force it to display your IPv4 or IPv6 address (which is handy if you are running a dual stack), simply access ipv4.icanhazip.com or ipv6.icanhazip.com. Those sites will always return your IPv4 and IPv6 addresses, respectively.

As always, if you find any bugs or other problems, be sure to let me know!

icanhazip.com now supports both IPv4 and IPv6 addresses is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Diagram: OpenVPN to Rackspace Cloud Servers and Slicehost

A recent blog post from Mixpanel inspired me to write a quick how-to for Fedora users on using OpenVPN to talk to instances privately in the Rackspace Cloud.

The diagram at the right gives an idea of what this guide will allow you to accomplish. Consider a situation where you want to talk to the MySQL installation on db1 directly without requiring extra ssh tunnels or MySQL over SSL via the public network. If you tunnel into one of your instances, you can utilize the private network to talk between your instances very easily.

There's one important thing to keep in mind here: even though you'll be utilizing the private network between your tunnel endpoint and your other instances, your traffic will still traverse the public network. That means that the instance with your tunnel endpoint will still get billed for the traffic flowing through your tunnel.

You'll only need the openvpn package on the server side:

yum -y install openvpn

Throw down this simple configuration file into /etc/openvpn/server.conf:

port 1194
proto tcp
dev tun
persist-key
persist-tun
 
server 10.66.66.0 255.255.255.0
ifconfig-pool-persist ipp.txt
 
#push "route 10.0.0.0 255.0.0.0"
push "route 10.176.0.0 255.248.0.0"
keepalive 10 120
 
ca      /etc/openvpn/my_certificate_authority.pem
cert    /home/major/vpn_server_cert.pem
key     /home/major/vpn_server_key.pem
dh      /etc/openvpn/easy-rsa/2.0/keys/dh1024.pem
 
status log/openvpn-status.log
verb 3

Here's a bit of explanation for some things you may want to configure:

• push — These are the routes that will be sent over the VPN that are pushed to the clients. If you don't use any IP addresses in the 10.0.0.0/8 network block in your office, you can probably use the commented out line above. However, you may want to be more specific with the routes if you happen to use any 10.0.0.0/8 space in your office.
• server — These are the IP addresses that the VPN server will assign and NAT out through the private interface. I've used a /24 above, but you may want to adjust the netmask if you have a lot of users making tunnels to your VPN endpoint.
• ca, cert, key — You will need to create a certificate authority as well as a certificate/key pair for your VPN endpoint. I already use SimpleAuthority on my Mac to manage some other CA's and certificates, but you can use openvpn's easy-rsa scripts if you wish. They are already included with the openvpn installation.

Build your Diffie-Hellman parameters file:

cd /etc/openvpn/easy-rsa/2.0/ && ./build-dh

Tell iptables that you want to NAT your VPN endpoint traffic out to all 10.x.x.x IP addresses on the private network:

iptables -t nat -A POSTROUTING -s 10.0.0.0/8 -o eth1 -j MASQUERADE

The last step on the server side is to ensure that the kernel will forward packets from the VPN endpoint out through the private interface. Ensure that your /etc/sysctl.conf looks like this:

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

Adjusting your sysctl.conf ensures that forwarding is enabled at boot time, but you'll need to enable it on your VPN endpoint right now:

echo 1 > /proc/sys/net/ipv4/ip_forward

Start the openvpn server:

/etc/init.d/openvpn start

If all is well, you should see openvpn listening on port 1194:

[root@lb2 ~]# netstat -ntlp | grep openvpn
tcp        0      0 0.0.0.0:1194      0.0.0.0:*         LISTEN      2020/openvpn

You'll need to configure a client to talk to your VPN now. This involves three steps: creating a new certificate/key pair for the client (same procedure as making your server certificates), signing the client's certificate with your CA certificate (same one that you used above to sign your server certificates), and then configuring your client application to access the VPN.

There are many openvpn clients out there to choose from.

If you're using a Linux desktop, you may want to consider using the built-in VPN functionality in NetworkManager. For Mac users, I'd highly recommend using Viscosity ($9), but there's also tunnelblick (free).

Accessing Rackspace Cloud Servers and Slicehost slices privately via OpenVPN is a post from: Major Hayden's Racker Hacker blog.

c0b6ad7e-f251-11df-b20b-4040336e00ef