Kristóf Kovács about some lesser-known Linux tools that can really come in handy in different situations.
If you haven't tried dstat (I hadn't until I saw Kristóf's post), this is a great one to try. You can keep a running tally on various server metrics including load average, network transfer, and disk operations.
I found myself stuck in a particularly nasty situation a few weeks ago where I had two git branches with some commits that were mixed up. Some commits destined for a branch called development ended up in master. To make matters worse, development was rebased on top of master and the history was obviously mangled.
My goal was to find out which commits existed in development but didn't exist anywhere in master. From there, I needed to find out which commits existed in master that didn't exist in development. That would give me all of the commits that needed to be in the development branch.
I constructed this awful looking bash mess to figure out which commits were in development but not in master:
I had a list of commits that existed in development but not in master:
965cf71 Trollface
acda854 Some patch 2
bf1f3e2 Some patch 1
db1980c Packaging
From there, I could swap MASTER and DEV to figure out which commits existed in master but not in development. Only a couple of commits showed up and these were the ones which were committed and pushed to master inadvertently. After a couple of careful cherry picks and reversions, my branches were back to normal.
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
A fellow Racker showed me about five years ago and I've had in my toolbox as a handy way to watch HTTP traffic. I'd used some crazy tcpdump arguments and some bash one-liners to pull out the information I needed but I never could get the live look that I really wanted.
Here's an example of what httpry's output looks like on a busy site like icanhazip.com:
You can watch the requests come in and the responses go out in real time. It even allows for BPF-style packet filters which allow you to narrow down the source and/or destination IP addresses and ports you want to watch. You can run it as a foreground process or as a daemon depending on your needs.
It's now available as a for Fedora 15, 16, 17 (and rawhide) as well as EPEL 6 (for RHEL/CentOS/SL 6).
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
One of the handiest tools in the OpenSSL toolbox is s_client. You can quickly view lots of details about the SSL certificates installed on a particular server and diagnose problems. For example, use this command to look at Google's SSL certificates:
You'll see the chain of certificates back to the original certificate authority where Google bought its certificate at the top, a copy of their SSL certificate in plain text in the middle, and a bunch of session-related information at the bottom.
This works really well when a site has one SSL certificate installed per IP address (this used to be a hard requirement). With (SNI), a web server can have multiple SSL certificates installed on the same IP address. SNI-capable browsers will specify the hostname of the server they're trying to reach during the initial handshake process. This allows the web server to determine the correct SSL certificate to use for the connection.
If you try to connect to rackerhacker.com with s_client, you'll find that you receive the default SSL certificate installed on my server and not the one for this site:
$ openssl s_client -connect rackerhacker.com:443
Certificate chain
0 s:/C=US/ST=Texas/L=San Antonio/O=MHTX Enterprises/CN=*.mhtx.net
i:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
1 s:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
i:/C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority
Add on the -servername argument and s_client will do the additional SNI negotiation step for you:
$ openssl s_client -connect rackerhacker.com:443 -servername rackerhacker.com
Certificate chain
0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=rackerhacker.com
i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
You may be asking yourself this question:
Why doesn't the web server just use the Host: header that my browser sends already to figure out which SSL certificate to use?
Keep in mind that the SSL negotiation must occur prior to sending the HTTP request through to the remote server. That means that the browser and the server have to do the certificate exchange earlier in the process and the browser wouldn't get the opportunity to specify which site it's trying to reach. SNI fixes that by allowing a Host: header type of exchange during the SSL negotiation process.
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
As promised in my earlier post entitled , I've assembled the simplest possible guide to get Kerberos up an running on two CentOS 5 servers.
Also, I don't really hate Kerberos. It's a bit of an inside joke with my coworkers who are studying for some of the exams at Rackspace. The additional security provided by Kerberos is quite good but the setup involves a lot of small steps. If you miss one of the steps or if you get something done out of order, you may have to scrap the whole setup and start over unless you can make sense of the errors in the log files. A lot of my dislikes for Kerberos comes from the number of steps required in the setup process and the difficulty in tracking down issues when they crop up.
To complete this guide, you'll need the following:
two CentOS, Red Hat Enterprise Linux or Scientific Linux 5 servers or VM's
some patience
Here's how I plan to name my servers:
kdc.example.com — the Kerberos KDC server at 192.168.250.2
client.example.com — the Kerberos client at 192.168.250.3
CRITICAL STEP: Before getting started, ensure that both systems have their hostnames properly set and both systems have the hostnames and IP addresses of both systems in /etc/hosts. Your server and client must be able to know the IP and hostname of the other system as well as themselves.
First off, we will need working to serve up the user information for our client. Install the NIS server components on the KDC server:
[root@kdc ~]# yum install ypserv
Set the NIS domain and set a static port for ypserv to make it easier to firewall off. Edit /etc/sysconfig/network on the KDC server:
NISDOMAINNAME=EXAMPLE.COM
YPSERV_ARGS="-p 808"
Manually set the NIS domain on the KDC server and add it to /etc/yp.conf:
There's one last configuration file to edit on the KDC! Ensure that /var/kerberos/krb5kdc/kadm5.acl looks like this:
*/admin@EXAMPLE.COM *
We're now ready to make a KDC database to hold our sensitive Kerberos data. Create the database and set a good password which you can remember. This command also stashes your password on the KDC so you don't have to enter it each time you start the KDC:
kdb5_util create -r EXAMPLE.COM -s
On the KDC, create a principal for the admin user as well as user1 (which we'll create shortly). Also, export the admin details to the kadmind key tab. You'll get some extra output after each one of these commands but I've snipped it to reduce the length of the post.
Transfer your /etc/krb5.conf from the KDC server to the client. Hop onto the client server, install the Kerberos client package and add some host principals:
There aren't any daemons on the client side, so the configuration is pretty much wrapped up there for Kerberos. However, we now need to tell both servers to use Kerberos for auth and your client servers needs to use NIS to get user data.
On the KDC:
run authconfig-tui
choose Use Kerberos from the second column
press Next
don't edit the configuration (authconfig got the data from /etc/krb.conf)
press OK
On the client:
run authconfig-tui
choose Use NIS and Use Kerberos
press Next
enter your NIS domain (EXAMPLE.COM) and NIS server (kdc.example.com or 192.168.250.2)
press Next
don't edit the Kerberos configuration (authconfig got the data from /etc/krb.conf)
press OK
Got NIS problems? If the NIS connection stalls on the client, ensure that you have the iptables rules present on the KDC that we added near the beginning of this guide. Also, if you forgot to add both hosts to both servers' /etc/hosts, go do that now.
Let's make our test user on the KDC. Don't add this user to the client — we'll get the user information via NIS and authenticate via Kerberos shortly. We'll also rebuild our NIS maps after adding the user:
You can see why NIS isn't a good way to authenticate users. Someone could easily pull the hash for any account and brute force the hash on their own server. Go back to the KDC and lock out the user account:
[root@kdc ~]# usermod -p '!!' user1
Go back to the client and try to pull the password hash now:
On the plus side, the user's password hash is now gone. On the negative side, you've just prevented this user from logging in locally or via NIS. Don't worry, the user can log in via Kerberos now. Let's prepare a home directory on the client for the user:
Note: In a real-world scenario, you'd probably want to export this user's home directory via NFS so they didn't get a different home directory on every server.
While you're still on the client, try to log into the client via the user. Use the password that you used when you created the user1 principal on the KDC.
The first line shows that the client asked for a Authentication Server Request (AS_REQ) and the second line shows that the client then asked for a Ticket Granting Server Request (TGS_REQ). In layman's terms, the client first asked for a ticket-granting ticket (TGT) so it could authenticate to other services. When it actually tried to log in via ssh it asked for a ticket (and received it).
YOU JUST CONFIGURED KERBEROS!
From here, the sky's the limit. Another popular implementation of Kerberos is encrypted NFSv4. You can even go crazy and use .
Let me know if you have any questions about this post or if you spot any errors. With this many steps, there's bound to be a typo or two in this guide. Keep in mind that there are some obvious spots for network-level and service-level security improvements. This guide was intended to give you the basics and it doesn't cover all of the security implications involved with a Kerberos implementation.
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
I'll be the first one to admit that Kerberos drives me a little insane. It's a requirement for two of the exams in and I've been forced to learn it. It provides some pretty nice security features for large server environments. You get central single sign ons, encrypted authentication, and bidirectional validation. However, getting it configured can be a real pain due to some rather archaic commands and shells.
Here's Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:
sshd calls to pam to authenticate the user
pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
the KDC replies to the client with a TGT encrypted with the user's password
pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
if pam can decrypt the TGT, it knows the user is providing the right password
Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?
client calls the KDC and asks for a ticket to grant access to the other service
KDC replies with two copies of the ticket:
one copy is encrypted with the user's current TGT
a second copy is encrypted with the password of the network service the user wants to access
the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
client passes the authenticator and the second copy of the ticket it received from the KDC
the other network service decrypts the second copy of the ticket and verifies the password
the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
if the timestamp looks good, the other network service allows the user access
Okay, that's confusing. Let's take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user's password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user's password back to the KDC before they can ask for a TGT. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they'll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.
Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they're also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client's own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn't have any clue what that password is and the attack will be thwarted.
By this point, you're either saying «Oh man, I don't ever want to do this.» or «How do I set up Kerberos?». Stay tuned if you're in the second group. I'll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.
In the meantime, here are a few links for extra Kerberos bedtime reading:
[PDF]
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
I sometimes enjoy living on the edge occasionally and that sometimes means I keep up with OpenStack changes commit by commit. If you're in the same boat as I am, you may save some time by using my repository of bleeding-edge Python packages from the OpenStack projects:
Python packages are updated moments after the commit is merged into the repositories under .
Although the packages will contain the latest code available, rest assured that the code has passed an initial code review (by humans), unit tests, and varying levels of functional or integrated testing. There may still be a bug or two cropping up after that, so be aware of that as you utilize these packages.
Regular users of Python's package tools like or are probably familiar with the repository. It's a one-stop-shop to learn more about available Python packages and get them installed on your server.
However, certain folks may find the need to host a local PyPi repository for their own packages. You may need it to store Python code which you don't plan to release publicly or you may need to add proprietary patches to upstream Python packages. Regardless of the reason to have it, a local PyPi repository is relatively easy to configure.
You'll need to start with a base directory for your PyPi repository. For this example, I chose /var/pypi. The directory structure should look something like this:
/var/pypi/simple/[package_name]/[package_tarball]
For a package like pip, you'd make a structure like this:
/var/pypi/simple/pip/pip-1.0.2.tar.gz
Once you have at least one package stored locally, it's time to configure apache. Here's a snippet from the virtual host I configured:
The last set of rewrite directives check to see if the request refers to an existing file or directory under your document root. If it does, your server will reply with a directory listing or with the actual file to download. If the directory or file doesn't exist, apache will send the client a redirection to the main PyPi site.
Reload your apache configuration to bring in your new changes. Let's try to download the pip tarball from our local server in the example I mentioned above:
$ curl -I http://pypi.example.com/simple/pip/
HTTP/1.1 200 OK
$ curl -I http://pypi.example.com/simple/pip/pip-1.0.2.tar.gz
HTTP/1.1 200 OK
I've obviously snipped a bit of the response above, but you can see that apache is responding with 200's since it has the directories and files that I was trying to retrieve via curl. Let's try to get something we don't have locally, like kombu:
$ curl -I http://pypi.example.com/simple/kombu/
HTTP/1.1 302 Found
Location: http://pypi.python.org/simple/kombu/
Our local PyPi repository doesn't have kombu so it will refer our Python tools over to the official PyPi repository to get the listing of available package versions for kombu.
Now we need to tell pip to use our local repository. Edit ~/.pip/pip.conf and add:
Once your tools are configured, try installing a package you have locally and try to install one that you know you won't have locally. You can add -v to pip install to watch it retrieve different URL's to get the packages it needs. If you spot any peculiar behavior or unexpected redirections, double-check your mod_rewrite rules in your apache configuration and check the spelling of your directories under your document root.
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
I used to be one of those folks who would install Fedora, CentOS, Scientific Linux, or Red Hat and disable SELinux during the installation. It always seemed like SELinux would get in my way and keep me from getting work done.
Later on, I found that one of my servers (which I'd previously secured quite thoroughly) had some rogue processes running that were spawned through httpd. Had I actually been using SELinux in enforcing mode, those processes would have probably never even started.
If you're trying to get started with SELinux but you're not sure how to do it without completely disrupting your server's workflow, these tips should help:
Get some good reporting and monitoring
Two of the most handy SELinux tools are . If you're running a server without X, you can use . You will receive email alerts within seconds of an AVC denial and the emails should contain tips on how to resolve the denial if the original action should be allowed. If the AVC denial caught something you didn't expect, you'll know about the potential security breach almost immediately.
Start out with SELinux in permissive mode
If you're overly concerned about SELinux getting in your way, or if you're enabling SELinux on a server that has been running without SELinux since it was installed, start out with SELinux in permissive mode. To make the change effective immediately, just run:
# setenforce 0
# getenforce
Permissive
Edit /etc/sysconfig/selinux to make it persistent across reboots:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=permissive
Adjust booleans before adding your own custom modules
There are a lot of booleans you can toggle to get the functionality you need without adding your own custom SELinux modules with audit2allow. If you wanted to see all of the applicable booleans for httpd, just use getsebool:
# getsebool -a | grep httpd
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> on
httpd_can_sendmail --> on
... and so on ...
Toggling booleans is easy with togglesebool:
# togglesebool httpd_can_network_memcache
httpd_can_network_memcache: active
Now httpd can talk to memcache. You can also use setsebool if you want to be specific about your setting (this is good for scripts):
# setsebool httpd_can_network_memcache on
Tracking your history of AVC denials
All of your AVC denals are logged by auditd in /var/log/audit/audit.log but it's not the easiest file to read and parse. That's where aureport comes in:
Summary
There's no need to be scared of or be annoyed by SELinux in your server environment. While it takes some getting used to (and what new software doesn't?), you'll have an extra layer of security and access restrictions which should let you sleep a little better at night.
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
Although Citrix recommends against using software RAID with XenServer due to performance issues, I've had some pretty awful experiences with hardware RAID cards over the last few years. In addition, the price of software RAID makes it a very desirable solution.
Before you get started, . That post also explains an optional adjustment to get a larger root partition (which I would recommend). You cannot complete the steps in this post if your XenServer installation uses GPT.
You should have three partitions on your first disk after the installation:
# fdisk -l /dev/sda
-- SNIP --
Device Boot Start End Blocks Id System
/dev/sda1 * 1 2611 20971520 83 Linux
/dev/sda2 2611 5222 20971520 83 Linux
/dev/sda3 5222 19457 114345281 8e Linux LVM
Here's a quick explanation of your partitions:
/dev/sda1: the XenServer root partition
/dev/sda2: XenServer uses this partition for temporary space during upgrades
/dev/sda3: your storage repository should be in this logical volume
We need to replicate the same partition structure across each of your drives and the software RAID volume will span the across the third partition on each disk. Copying the partition structure from disk to disk is done easily with sfdisk:
WHOA THERE! NO TURNING BACK! This step is destructive! If your other disks have any data on them, this step will make it (relatively) impossible to retrieve data on those disks again. Back up any data on the other disks in your XenServer machine before running these next commands.
If you have only two disks, stop with /dev/sdb and you'll be making a RAID 1 array. My machine has four disks and I'll be making a RAID 10 array.
We need to destroy the main storage repository, but we need to unplug the physical block device first. Get the storage repository uuid first, then use it to find the corresponding physical block device. Once the physical block device is unplugged, the storage repository can be destroyed:
Stop here and reboot your XenServer box to pick up the new partition changes. Once the server comes back from the reboot, start up a software RAID volume with mdadm:
// RAID 1 for two drives
mdadm --create /dev/md0 -l 1 -n 2 /dev/sda3 /dev/sdb3
// RAID 10 for four drives
mdadm --create /dev/md0 -l 10 -n 4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3
Although you don't have to wait for the resync to complete, just be aware that XenServer doesn't do well with a lot of disk I/O within dom0. You may notice unusually slow performance in dom0 until it finishes. Save the array's configuration for reboots:
mdadm --detail --scan > /etc/mdadm.conf
Edit the /etc/mdadm.conf file and append auto=yes to the end of the line (but leave everything on one line):
This command takes some time to complete since it makes logical volumes and then makes an ext3 filesystem for the new storage repository. Bigger RAID arrays will take more time and it's guaranteed to take longer than you'd expect if your RAID array is still building. As soon as it completes, you'll be given the uuid of your new storage repository and it should appear within the XenCenter interface.
TIP: If you run into any problems during reboots, open /boot/extlinux.conf and remove splash and quiet from the label xe boot section. This removes the framebuffer during boot-up and it causes a lot more output to be printed to the console. It won't affect the display once your XenServer box has fully booted.
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.