Игорь Олемской — практические заметки по системному администрированию Linux CentOS

You've probably noticed that the blog has slowed down a bit recently. Part of the slowdown is due to an uptick in work required to get OpenStack Nova and its related software up and running at Rackspace for Cloud Servers and another part of it is a severe case of writer's block. I threw out some questions on Twitter about the topics people would like to see covered in some new posts and a commonly requested topic was employment at Rackspace.

First things first, getting a job at Rackspace isn't easy. We don't intentionally make the process difficult. It's just that the work we do is unique and demanding.

We work in a fast-paced, extremely dynamic team-centric environment. While some people in the company work in extremely small teams or sometimes all by themselves, that's pretty few and far between. We look for people who can survive and flourish in this atmosphere and we look for people who can do it all while working as a team. Even with all of this hustle and bustle, we still remember why we're doing it: the pursuit of Fanatical Support for our customers.

Another thing to keep in mind is that there's no true secret for making it through the application process. There's no magic combination of skills or «silver bullet» that will scoot you through. Every candidate is reviewed individually for each position. There have been several times at the end of an interview where we've gotten together and said: «Wow, this candidate is solid, but they're just not right for this position. Let's find the right spot and see if there's a spot open.» We look for the right candidate for the right position at the right time.

One of the best ways to get ahead in the screening or interview process is to do a little homework about Rackspace and the products we offer. Much of this is covered in a post I wrote in 2011. You'll go into the interviews with more confidence and it will be much more obvious that you're really interested in the position.

Don't be discouraged if the process takes a little longer than you expected. When I was hired in 2006, I went through two phone pre-screens and then three back-to-back interviews in person. Things have changed a little since then and I've heard of some candidates receiving two to three pre-screens via telephone and then one or two interviews in person. The additional screening and interviews may be due to Rackers trying to find the right fit for a particular applicant. As I said previously, we look for the right fit for each applicant. We may consider you for a different position than you applied for if we feel like your skill set or personality fits that role better.

A very common question is what to wear to a Rackspace interview. It's confusing to know exactly what's expected since we have Rackers in the building wearing everything from suits to flip-flops. This is where you really have to go with your gut. Interviewing for a customer-facing sales position while wearing a hoodie and shorts is probably going to bring a suboptimal result. Keep in mind that there's really nothing negative about overdressing (but keep your tuxedo in the closet, seriously). I wore a shirt and tie for my interviews in 2006 but my tie got caught in the car door and was shredded. After a lot of cursing, I took off the tie and decided to wing it with my dress shirt. Nobody ever said a word about it.

Remember to be flexible during the interviews. You might be asked to draw a solution on a whiteboard or think through a really complicated situation. Roll with it and keep your confidence up. When you don't know something, admit it, but then talk about how you'd research an answer.

There's one last thing to keep in mind and it's really critical. If you're ever asked about how you would solve a problem or how you solved a problem in the past, don't divulge any information which is confidential or proprietary to your current company. Just tell the interviewers that you've solved the solution in the past but you'll need to keep things vague to maintain confidentiality. We will definitely understand and we will encourage you to maintain that confidentiality.

Leave your comments if you have any! I'll be glad to answer any questions you have.

Getting a Technical Job at Rackspace is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

My quest to get better at Python led me to create a new project on GitHub. It's called mysql-json-bridge and it's ready for you to use.

Why do we need a JSON API for MySQL?
The real need sprang from a situation I was facing daily at Rackspace. We have a lot of production and pre-production environments which are in flux but we need a way to query data from various MySQL servers for multiple purposes. Some folks need data in ruby or python scripts while others need to drag in data with .NET and Java. Wrestling with the various adapters and all of the user privileges on disparate database servers behind different firewalls on different networks was less than enjoyable.

That's where this bridge comes in.

The bridge essentially gives anyone the ability to talk to multiple database servers across different environments by talking to a single endpoint with easily configurable security and encryption. As long as the remote user can make an HTTP POST and parse some JSON, they can query data from multiple MySQL endpoints.

How does it work?
It all starts with a simple HTTP POST. I've become a big fan of the Python requests module. If you're using it, this is all you need to submit a query:

import requests
payload = {'sql': 'SELECT * FROM some_tables WHERE some_column=some_value'}
url = "http://localhost:5000/my_environment/my_database"
r = requests.post(url, data=payload)
print r.text

The bridge takes your query and feeds it into the corresponding MySQL server. When the results come back, they're converted to JSON and returned via the same HTTP connection.

What technology does it use?
Flask does the heavy lifting for the HTTP requests and Facebook's Tornado database class wraps the MySQLdb module in something a little more user friendly. Other than those modules, PyYAML and requests are the only other modules not provided by the standard Python libraries.

Is it fast?
Yes. I haven't done any detailed benchmarks on it yet, but the overhead is quite low even with a lot of concurrency. The biggest slowdowns come from network latency between you and the bridge or between the bridge and the database server. Keep in mind that gigantic result sets will take a longer time to transfer across the network and get transformed into JSON.

I found a bug. I have an idea for an improvement. You're terrible at Python.
All feedback (and every pull request) is welcome. I'm still getting the hang of Python (hey, I've only been writing in it seriously for a few weeks!) and I'm always eager to learn a new or better way to accomplish something. Feel free to create an issue in GitHub or submit a pull request with a patch.

mysql-json-bridge: a simple JSON API for MySQL is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Today marks the fifth year that this blog has existed on the internet. I bought the domain on February 14th, 2007 and tossed together a quick WordPress installation (I can't even remember the version now!) to hold my notes that I was gathering at work.

Photo credit: Will Clayton

At the time, I had recently parted ways with a very small internet startup and joined the ranks at Rackspace as an entry-level Linux system administrator. The abrupt change from «top dog at the startup» to «wow, I don't know anything about Linux» caught me by surprise and I was trying to stuff as much knowledge into my brain as quickly as I could. My teammates at Rackspace were eager to show me the ropes of wrangling servers and supporting customers.

As I mentioned already, the blog started out just as a place to stuff my notes from the things I learned at work. I figured that it would be nice to store it in a searchable format but it would also be great if I could link other people to certain posts if they needed more information to fix a problem. It was a way to retain knowledge but yet give it back to the people around me who needed it.

The blog has hit 456 posts (this one is #457) and it's gone from a few page views per day to just over 20,000 per day. Here are the top five most accessed posts (since I've been keeping stats):

1. Syncing an iPhone with a new Mac without hassles
2. ip_conntrack: table full, dropping packet
3. Delete a single iptables rule
4. Increase MySQL connection limit
5. MySQL Error 1040: too many connections

I'd like to send out a big thanks to the people who read this blog, add comments (or complaints!), and suggest new topics. You are the reason why I take the time to keep this blog going.

Five years of rackerhacker.com is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The grades came back last Friday and I've passed the last exam in the requirements to become a Red Hat Certified Architect (RHCA). I was fortunate enough to be part of Rackspace's RHCA pilot program and we took our first exam back at the end of 2010. It's definitely a good feeling to be finished and I'm definitely ready to give back some knowledge to the readers of this blog.

First things first: there are going to be many part of this post which probably aren't as specific as you'd like. A lot of that is due to the NDA that all Red Hat examinees agree to when they take an exam. We aren't allowed to talk about what was on the exam or our experiences during the exam. If we do, penalties range from smaller things like losing certifications all the way up to serious stuff like legal action. It goes without saying that I want to protect the security of the exams, I don't want to lose my certifications, and I don't want to hire a lawyer. Please try to keep this in mind if you yearn for more specifics than I'm able to give.

Red Hat Certified Engineer
The RHCSA and RHCE exams are the first step on the path to the RHCA. You can't take any of the RHCA prerequisite exams without it. These exams cover a really broad spectrum of material including apache configuration, NFS, iptables and mail services. The two links above will take you to the exam objectives for each exam.

I've always recommended the RHCE exam for Linux administrators who are trying to sharpen their skills and get to the next level whether they use Red Hat or not. The exam covers a lot of good material that makes a solid foundation for any Linux user without throwing in too many Red Hat-specific knowledge.

The exam (like all Red Hat exams) is fully practical. There are no multiple choice questions or essays. You'll have to meet all of the objectives by logging into a local Red Hat system and making the system do what it needs to do.

Quick tips for the RHCSA/RHCE exams:

• Keep your eye on the clock. Time can really get away from you if you get stuck in the weeds on a problem that should be relatively straightforward.
• Leave time at the end to check your work. When you set up a lot of services, it's inevitable that you might configure a service for one problem that breaks the functionality required by a problem you completed already.
• Always reboot before you leave. We all forget to use chkconfig when we're in a hurry.
• Practice, practice, practice. There's not one objective on this exam that you can't test in a VM on your own.

Red Hat Enterprise System Monitoring and Performance Tuning
Our group at Rackspace started off with EX442 and it was a very difficult way to start off the RHCA track. Take a look at the objectives and you'll see that much of the exam is related to tweaking system performance and then monitoring that performance with graphs and raw data. You'll have to turn a lot of knobs on the kernel and you'll need to know where to store these configurations so they'll be persistent.

In addition, the objective regarding TCP buffers and related settings is a real challenge. You'll have to wrestle with some math that appears to be relatively simple, but can get confusing quickly. Some of the settings can't really be checked to know if your setting is correct. The objectives mention tuning disk scheduling — you don't really have the time or tools to know if your setting is ideal.

Quick tips for EX442:

• Use the documentation available to you. Install the kernel-doc package while you practice and during the exam.
• Be careful with your math. You have a Linux machine in front of you! Don't forget about bc.
• Watch your units. Know the difference between a kilobyte (KB) and a kibibyte (KiB).
• Make comments in files where you adjust kernel configurations. It will help you keep track of which question the kernel adjustment is meant to satisfy.

Red Hat Enterprise Storage Management
I'm surprised to say this now, but I actually enjoyed EX436. I've always used other clustering tools like heartbeat and pacemaker, but I've never had the need to use the Red Hat Cluster Suite. Although RHCS definitely has a lot of quirks and rough edges, it's pretty solid once you get familiar with the GUI and command line tools.

You get the opportunity to mess around with some pretty useful technology like iSCSI, GFS, and clustered LVM. These are things that you're probably already using or will be using soon in a large server environment. The web interface for RHCS is quite peculiar and you may find yourself wanting to put your fist through the screen when you're staring down the endless animated GIFs when the cluster is syncing its configuration. Do your best to be patient because you certainly don't want to short circuit the cluster sync.

Quick tips for EX436:

• Be patient. You'll feel like the RHCS web interface is mocking you when you're pressed for time.
• Watch the clock. It's extremely easy to burn a lot of time on this exam if you get stuck on a particular problem.
• Double check your entries in the web interface. Make sure you're doing things in the right order and that you've set up the prerequisites before adding services to the cluster. If you get it wrong, you could put your cluster into a weird state.
• Use man pages. If you don't mess with GFS a lot, the man pages will save you in a pinch.

Red Hat Enterprise Deployment and Systems Management
If there's one exam where time management is critical, it's EX401. Importing data into the Satellite Server takes quite a bit of time and there's almost nothing you can do to speed it up. It probably goes without saying, but as with most long-running tasks, you'll want to run it in screen. The last thing you'd ever want is to abort the import due to an errant click or CTRL-C (I did it while practicing — it's aggravating).

There are other test objectives which you can either complete or partially complete while you wait for the import to finish.

Also, take the time to really dig into the Satellite Server web interface while your practicing for the exam. Knowing where to find the most common configuration items will really save some time when you're in the exam. You can sometimes get pretty bogged down in the interface so don't forget to use multiple tabs to keep your work organized.

I felt like this exam was the easiest out of the bunch since you could go back and test every single question with good time management. Did I mention how important time management was on this exam already? If I forgot to mention it earlier, be sure to focus on time management for this test.

Quick tips for EX401:

• Time management will make or break you on this test. Keep an eye on the clock and make sure you've done absolutely every piece of the exam that you can while you wait for the server to do its work.
• Scour the web interface. Keep a mental map in your mind where the big chunks of configuration items are.
• Go back and test everything. If you manage your time well, you should have enough time to verify each and every objective on this exam.

Red Hat Enterprise Directory Services and Authentication
At first, EX423 looks pretty straightforward. Red Hat's authentication configuration tools make LDAP authentication setup pretty easy. However, this exam comes with a lot of curveballs.

The GUI interface for the Directory Services component is a little frustrating to use. I found that the GUI stopped responding to keyboard input occasionally unless I clicked on another window and came back. If you misconfigure the SSL certificates in the interface, your LDAP server is down for the count. If you don't input the correct data into the setup scripts at the beginning, you might not notice it until much later when it's either too difficult to dig yourself out of the hole or it's too late to start over with a clean configuration.

I didn't feel pressed for time on this exam too much and that was pretty refreshing after taking the EX401 test. It's extremely critical to watch what you type and click on this exam. Some mistakes can be quickly corrected while others may require you to blow away the LDAP server configuration and re-provision the whole thing.

Quick tips for EX423:

• Always watch what you're typing. A simple mistake can lead to confusion or bigger issues down the road.
• Don't ignore the LDIF objectives. As you practice, you'll find that manipulating LDIF files is a little more involved than you expected.
• Practice starting over. Throw out your Directory Services configuration and get the experience of what it's like to start over and get back in the game.

Red Hat Enterprise Security: Network Services
There's no sugar coating it — EX333 is a beast. It's a six hour exam broken into two three-hour chunks. It covers a ton of material and I refer to it as «the RHCE on steroids.» You might argue that I thought it was hard since it was the last test and I was ready to be finished, but I really think this exam is a tough one.

Practicing for the Kerberos and DNS objectives was the hardest for me. I just couldn't understand Kerberos, no matter how hard I tried. The realization that I would really have to learn it soon set in. I dug into the Kerberos design documentation on MIT's site, read the summaries on Wikipedia, and scoured the documentation available in the Kerberos RPM packages. Once I understood why Kerberos is set up the way it is and why the security measures are present, everything began to come together. I was able to remember the steps not because I was memorizing them, but because I understood how Kerberos worked.

When you're working through the DNS objectives, keep an eye out for punctuation. I blew through a good 20 minutes in what seemed like the blink of an eye when I forgot a period in my TSIG key configuration while studying. Make sure you use the resources available to you, like system-config-bind and sample configs in /usr/share/doc/bind*/examples/. Get to know commands like dig really well.

If you're overwhelmed by OpenSSL's command line syntax, check out the /etc/pki/tls/misc/CA script. There are some handy comments at the top of the script that explain how to use it. You can also pluck OpenSSL commands right out of the script if you need to run them yourself.

• Don't just memorize. Do some research to understand how everything fits together.
• Manage your time. DNS and Kerberos have lots of small nuances that can become time sinks when done incorrectly.
• Use the available documentation and tools. Try practicing without study materials so that you're forced to use the docs and tools available within the server.

Ranking the exams
A couple of folks on Twitter asked me to rank the exams from most difficult to least difficult. Keep in mind that these are a little subjective since I was more familiar with some objectives than others for certain tests.

• EX333 — Enterprise Security: Network Services: a tubload of material and a very long exam
• EX442 — System Monitoring and Performance Tuning: very difficult to check your work, lots of calculations
• EX423 — Directory Services and Authentication: not a lot of material to cover, but tons of curveballs
• EX436 — Storage Management: the web interface made things much easier, lots of documentation available
• EX401 — Deployment and Systems Management: every objective can be tested, I build RPM's already

Looking back at the long road to becoming a Red Hat Certified Architect is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

As promised in my earlier post entitled Kerberos for haters, I've assembled the simplest possible guide to get Kerberos up an running on two CentOS 5 servers.

Also, I don't really hate Kerberos. It's a bit of an inside joke with my coworkers who are studying for some of the RHCA exams at Rackspace. The additional security provided by Kerberos is quite good but the setup involves a lot of small steps. If you miss one of the steps or if you get something done out of order, you may have to scrap the whole setup and start over unless you can make sense of the errors in the log files. A lot of my dislikes for Kerberos comes from the number of steps required in the setup process and the difficulty in tracking down issues when they crop up.

To complete this guide, you'll need the following:

• two CentOS, Red Hat Enterprise Linux or Scientific Linux 5 servers or VM's
• some patience

Here's how I plan to name my servers:

• kdc.example.com — the Kerberos KDC server at 192.168.250.2
• client.example.com — the Kerberos client at 192.168.250.3

CRITICAL STEP: Before getting started, ensure that both systems have their hostnames properly set and both systems have the hostnames and IP addresses of both systems in /etc/hosts. Your server and client must be able to know the IP and hostname of the other system as well as themselves.

First off, we will need NIS working to serve up the user information for our client. Install the NIS server components on the KDC server:

[root@kdc ~]# yum install ypserv

Set the NIS domain and set a static port for ypserv to make it easier to firewall off. Edit /etc/sysconfig/network on the KDC server:

NISDOMAINNAME=EXAMPLE.COM
YPSERV_ARGS="-p 808"

Manually set the NIS domain on the KDC server and add it to /etc/yp.conf:

[root@kdc ~]# nisdomain EXAMPLE.COM
[root@kdc ~]# echo "domain EXAMPLE.COM server kdc.example.com" >> /etc/yp.conf

Adjust /var/yp/securenets on the KDC server for additional security:

[root@kdc ~]# echo "255.0.0.0 127.0.0.0" >> /var/yp/securenets
[root@kdc ~]# echo "255.255.255.0 192.168.250.0" >> /var/yp/securenets

Start the NIS server and generate the NIS maps:

[root@kdc ~]# /etc/init.d/ypserv start; chkconfig ypserv on
[root@kdc ~]# make -C /var/yp

I usually like to prepare my iptables rules ahead of time so I ensure that it doesn't derail me later on. Paste this into the KDC's terminal:

iptables -N SERVICES
iptables -I INPUT -j SERVICES
iptables -A SERVICES -p tcp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p udp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p tcp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p udp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p tcp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 464 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p tcp --dport 749 -j ACCEPT -m comment --comment "kerberos"
/etc/init.d/iptables save

We need our time in sync for Kerberos to work properly. Install NTP on both nodes, start it, and ensure it comes up at boot time:

[root@kdc ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start
[root@client ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start

Now we're ready to set up Kerberos. Start by installing some packages on the KDC:

[root@kdc ~]# yum install krb5-server krb5-workstation

We will need to make some edits to /etc/krb5.conf on the KDC to set up our KDC realm. Ensure that the default_realm is set:

default_realm = EXAMPLE.COM

The [realms] section should look like this:

[realms]
EXAMPLE.COM = {
	kdc = 192.168.250.2:88
	admin_server = 192.168.250.2:749
}

The [domain_realm] section should look like this:

[domain_realm]
kdc.example.com = EXAMPLE.COM
client.example.com = EXAMPLE.COM

Add validate = true within the pam { } block of the [appdefaults] section:

[appdefaults]
 pam = {
   validate = true

Adjust /var/kerberos/krb5kdc/kdc.conf on the KDC:

[realms]
EXAMPLE.COM = {
	master_key_type = des-hmac-sha1
	default_principal_flags = +preauth
}

There's one last configuration file to edit on the KDC! Ensure that /var/kerberos/krb5kdc/kadm5.acl looks like this:

*/admin@EXAMPLE.COM	    *

We're now ready to make a KDC database to hold our sensitive Kerberos data. Create the database and set a good password which you can remember. This command also stashes your password on the KDC so you don't have to enter it each time you start the KDC:

kdb5_util create -r EXAMPLE.COM -s

On the KDC, create a principal for the admin user as well as user1 (which we'll create shortly). Also, export the admin details to the kadmind key tab. You'll get some extra output after each one of these commands but I've snipped it to reduce the length of the post.

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc root/admin
kadmin.local:  addprinc user1
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/admin
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/changepw
kadmin.local:  exit

Let's start the Kerberos KDC and kadmin daemons:

[root@kdc ~]# /etc/init.d/krb5kdc start; /etc/init.d/kadmin start
[root@kdc ~]# chkconfig krb5kdc on; chkconfig kadmin on

Now that the administration work is done, let's create a principal for our KDC server and stick it in it's keytab:

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc -randkey host/kdc.example.com
kadmin.local:  ktadd host/kdc.example.com

Transfer your /etc/krb5.conf from the KDC server to the client. Hop onto the client server, install the Kerberos client package and add some host principals:

[root@client ~]# yum install krb5-workstation
[root@client ~]# kadmin.local
kadmin.local:  addpinc --randkey host/client.example.com
kadmin.local:  ktadd host/kdc.example.com

There aren't any daemons on the client side, so the configuration is pretty much wrapped up there for Kerberos. However, we now need to tell both servers to use Kerberos for auth and your client servers needs to use NIS to get user data.

• On the KDC:
  • run authconfig-tui
  • choose Use Kerberos from the second column
  • press Next
  • don't edit the configuration (authconfig got the data from /etc/krb.conf)
  • press OK
• On the client:
  • run authconfig-tui
  • choose Use NIS and Use Kerberos
  • press Next
  • enter your NIS domain (EXAMPLE.COM) and NIS server (kdc.example.com or 192.168.250.2)
  • press Next
  • don't edit the Kerberos configuration (authconfig got the data from /etc/krb.conf)
  • press OK

Got NIS problems? If the NIS connection stalls on the client, ensure that you have the iptables rules present on the KDC that we added near the beginning of this guide. Also, if you forgot to add both hosts to both servers' /etc/hosts, go do that now.

Let's make our test user on the KDC. Don't add this user to the client — we'll get the user information via NIS and authenticate via Kerberos shortly. We'll also rebuild our NIS maps after adding the user:

[root@kdc ~]# useradd user1
[root@kdc ~]# passwd user1
[root@kdc ~]# make -C /var/yp/

On the client, see if you can get the password hash for the user1 account via NIS:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:$1$sUlSTlCv$riK5El3z8N4y.mi5Fe3Q60:500:500::/home/user1:/bin/bash

You can see why NIS isn't a good way to authenticate users. Someone could easily pull the hash for any account and brute force the hash on their own server. Go back to the KDC and lock out the user account:

[root@kdc ~]# usermod -p '!!' user1

Go back to the client and try to pull the password hash now:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:!!:500:500::/home/user1:/bin/bash

On the plus side, the user's password hash is now gone. On the negative side, you've just prevented this user from logging in locally or via NIS. Don't worry, the user can log in via Kerberos now. Let's prepare a home directory on the client for the user:

[root@client ~]# mkdir /home/user1
[root@client ~]# cp -av /etc/skel/.bash* /home/user1/
[root@client ~]# chown -R user1:user1 /home/user1/

Note: In a real-world scenario, you'd probably want to export this user's home directory via NFS so they didn't get a different home directory on every server.

While you're still on the client, try to log into the client via the user. Use the password that you used when you created the user1 principal on the KDC.

[root@client ~]# ssh user1@localhost
user1@localhost's password:
[user1@client ~]$ whoami
user1

List your Kerberos tickets and you should see one for your user principal:

[user1@client ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500_fCKPnZ
Default principal: user1@EXAMPLE.COM
 
Valid starting     Expires            Service principal
02/05/12 14:18:53  02/06/12 00:18:53  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 02/05/12 14:18:53

Your KDC should have a couple of lines in its /var/log/krb5kdc.log showing the authentication:

Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): AS_REQ (12 etypes {18 17 16 23 1 3 2 11 10 15 12 13}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=16 ses=16}, user1@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=18 ses=18}, user1@EXAMPLE.COM for host/client.example.com@EXAMPLE.COM

The first line shows that the client asked for a Authentication Server Request (AS_REQ) and the second line shows that the client then asked for a Ticket Granting Server Request (TGS_REQ). In layman's terms, the client first asked for a ticket-granting ticket (TGT) so it could authenticate to other services. When it actually tried to log in via ssh it asked for a ticket (and received it).

YOU JUST CONFIGURED KERBEROS!

From here, the sky's the limit. Another popular implementation of Kerberos is encrypted NFSv4. You can even go crazy and use Kerberos with apache.

Let me know if you have any questions about this post or if you spot any errors. With this many steps, there's bound to be a typo or two in this guide. Keep in mind that there are some obvious spots for network-level and service-level security improvements. This guide was intended to give you the basics and it doesn't cover all of the security implications involved with a Kerberos implementation.

The Kerberos-hater's guide to installing Kerberos is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

A videographer from CNN Money stopped by the office today to ask about what makes Rackspace a unique place to work. As soon as we got started, everyone started to make as many distractions as they could to crack me up. Very few succeeded.

Thanks to @pinojo for snapping the photo.

You might see me on CNN Money soon is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Quite a few people who couldn't make it to Xen Summit 2011 this year asked me to write a post summarizing my takeaways from the event. I'm not generally one to back down from peer pressure, so read on if you're interested about the discussions at this year's Summit.

The feeling I had at last year's summit is that Xen was on the verge of losing traction in the market. Very few distributions still had Xen support going forward and much of the discussion was around the lack of dom0 support in upstream Linux kernels. Distribution vendors were hesitant to drag patches forward into modern kernels and this made it much more difficult to get Xen working for many people.

This year was quite different. The number of attendees was up, the venue was much better, and there was an obvious buzz of energy in the room. As many of the presenters noted, this excitement stemmed from the upstream dom0 support in Linux 3.0. This inclusion is a huge win and it helps to drive Xen forward since the developers don't have to worry about dragging patches forward. They can focus on improving performance, adding features, and tightening security.

Many of the discussions this year focused on security and performance. Ian Pratt discussed Xen's ability to view memory pages of virtual machines via an API to detect malware running inside the instance. Memory pages could be identified and marked as not executable or applications could be triggered when a VM attempts to touch a particular memory page. Also, the whole VM could be frozen if needed.

There's also a big push to bring code out of the dom0 and push it into utility VM's. Driver domains could manage the network or I/O infrastructure and this would further reduce the amount of privileged code actively running in dom0. There is already very little code required for the Xen hypervisor itself (much much less than the Linux kernel — I'm looking at you, KVM) and this reduces the attack surface for potential compromises of the hypervisor. Some projects even aim to restart driver domains multiple times per minute to ensure that any malicious code injected into those virtual machines can't exist for long periods.

Pradeep Vincent from Amazon talked about how Amazon uses Xen and the pain points they have with its current architecture. Much of his discussion was around scaling problems (and we see many of the same issues at Rackspace). Higher performance could easily be gained by multi-threaded operations in dom0 when attaching block devices and creating virtual network interfaces. He also saw some areas for performance gains in the pvops I/O code.

Quite a few of the talks centered on the ARM architecture and what Xen is able to do on those systems after Samsung published their port in 2008. HVM is on the way for ARM and it might even show up in Xen 4.2. Some demos of Xen on mobile phones from Samsung were amazing. They showed how an attacker could compromise the web browser on the phone with a keylogger, but that application was running in a VM. Once the user switched back to the phone's main menu, the keylogger couldn't access the keystrokes any longer. After that, a simple close of the browser killed the VM and destroyed the malicious code.

Xen 4.2 should be available in early 2012 and the feature list is staggering. Improvements to libxenlight, pvops performance (even in HVM), and guest memory sharing should be available with the new release. Nested virtualization (run a hypervisor inside a hypervisor) is also coming in Xen 4.2 and I'm sure Xzibit will be a huge fan. This should streamline hypervisor testing, allow for embedded hypervisor options and extend the capabilities of client hypervisors. Remus should be available in 4.2 as well, but it might be marked as experimental. OVMF will be added as a BIOS option for UEFI (along with the standard SeaBIOS) and this should allow for Mac OS X guests. UEFI allows Windows to boot faster since it switches to PV mode sooner and it allows for simpler platform certification for software vendors.

Mike McClurg's presentation on XCP was pretty important to me since Rackspace is a big consumer of XenServer. If you're not familiar with XCP, it's basically open-source XenServer which runs on bleeding edge (and sometimes unstable) components. XCP 1.5 and XenServer 6 should be available in November with Xen 4.1 and Linux 2.6.32. GPU passthrough, up to 1TB RAM, and disaster recovery will be available. Another goal for the XCP team is to work closely with OpenStack via Project Olympus. Mike's vision is to have XCP become the configuration of choice for open source clouds. Project Kronos was also extremely interesting. It's essentially XCP's XenAPI stack running on Debian and Ubuntu. You'd be able to install either OS on a physical server and run XCP's services on it for a fully OSS hypervisor.

Konrad Wilk gave an update on Linux pvops and it appears there is a shift to get Xen working well on a desktop. This includes 3D graphics support, S3/hibernate capabilities and various bug fixes. There's also a push to get PV functionality into HVM and get HVM functionality into PV. Driver/device domains were discussed again in Patrick Kolp's talk and he had plenty of graphs showing performance changes when regularly restarting device domains. The performance dips were almost negligible with 10 second restarts and the security gains were significant.

There were several other great presentations on other topics like GlusterFS, OpenStack Nova, and Linpicker (from the NSA!). If these types of things interests you, keep your eyes peeled for Xen Summit 2012 next year. The weather in the bay area is well worth the trip.

Xen Summit 2011: My Takeways is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you work for a growing company like I do, it's inevitable that you'll have to do your fair share of interviewing. I love it when I leave an interview with a good feeling about the candidate. That «wow, they really nailed it» feeling is always great to have when you need to fill a critical role. Most often, the successful candidates are the ones who do their homework before they ever walk in our office doors.

What do I mean by «do your homework?» Here are some bullet points to get you on your way:

Know what the company does.
This one is critical and it should be easy. However, make sure to do thorough research first. For example, if you interviewed at a company like Apple, becoming familiar with their hardware lineup should be a no-brainer. That's their bread and butter. On the other hand, remember that Apple isn't solely a hardware company; they write lots of software, provide online productivity services, and they distribute music, movies, and other entertainment media.

While you're doing this research, try to discover what makes the company unique. Sure, Apple sells laptops and desktops (just like a lot of other companies), but what makes their particular products unique? Is there something unique about the way they provide their services? Have they cornered a certain market segment by providing a combination of products and services to that group of consumers? Answering these simple questions may help you tip the scales in the interview process.

Try one or more of the company's products.
The feasibility of trying a company's product before an interview could be debatable. For example, if you wanted to interview at Cray, you probably don't need to drop $2M USD on your own XE6 before walking in the door. For companies where the barrier to entry for purchasing a product is much lower, such as cloud computing companies, there's no excuse to not try things out first. Amazon has a free tier and a Rackspace Cloud Server could cost you as little as $2.50 per week.

It's concerning when I talk to an applicant about a job working with Rackspace's Cloud Servers and they haven't tried out any cloud products from any provider. How can I take a candidate's interest seriously when they haven't shown interest in any portion of my group's market segment?

Know what the company's competitors do.
It's often more impressive to an interviewer to know what a company's competitors are doing and how it compares to what that company is doing in the market. For example, if you can walk into an interview and say «I like the way your company makes these widgets, but Company X is able to make them more lightweight, and I value that more than the added customer service your company offers.» This shows the interviewer that you're familiar with various products in the segment and you've used them enough to understand what makes them different.

Some of you might be thinking: «Why would I say something like that to the interviewer? They'll think I'm being too negative about their product.» That's always possible, but you can guard against it by wording everything carefully. Make sure you have a solid reason for the way you feel that is based on something substantial (usability, price, features, etc). I've had candidates talk for five to ten minutes about why one of our product is inferior to one of our competitors' products and I was very impressed.

One quick gotcha: your interviewer might turn your comments back on you and ask you how you would improve one of the inferior products (I do this regularly). Make sure that you're prepared for that question and consider offering up a suggestion before the question is presented to you.

Can't get the information you need? Ask!
When you reach the end of the interview and the interviewer asks if you have questions, be sure to ask any questions about topics you had trouble researching. Going back to the Cray example, compare what you know about an XE6 to servers you've used before. You could mention a problem you had with the density of your previous configurations and ask how they overcame that hurdle at Cray. If it's a proprietary trade secret, you might not get an answer, but they'll know that you did some serious research ahead of time. If they can share the answer, they might still be impressed, and you might end up learning something you didn't know prior to the interview.

Conclusion
In summary, doing your homework and learning about the company shows the interviewers that you not only have what it takes to do the work, but that the work interests you as well. I've interviewed folks in the past who lacked on technical ability but had plenty of desire and drive. More often than not, those people are now Rackers.

Do your homework before a technical interview is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Anyone who says management is easy obviously hasn't done it for very long or they're not doing their job very well. Coordinating the activities and personal development of each person on the team is always a challenge and it introduces an unbelievable number of variables into an already difficult job. However, watching members of the team grow and succeed in their work is tremendously rewarding.

Taking on the job of a technical manager presents its own unique challenges. It's easy for a technical manager to lose focus and get down in the weeds of daily work. It's also very difficult to let go of the reins and resign to the fact that the direct involvement in technical work will have to be reduced.

These problems resonate with me as I've recently taken on another technical management role at Rackspace. My previous experience involved managing a team of technicians at various skill levels who were working on customer environments made up of dedicated servers and network equipment. The current position has quite a few differences. I'm now managing a small group of highly technical and extremely dedicated Linux engineers and we're responsible for maintaining the systems and networks which run the Cloud Servers product.

One of my goals of this blog is to help others learn things much more easily than I have. Here are some things I've had to learn the hard way while working as a technical manager:

Get out of the mindset of an individual contributor
When you're a system administrator on a team (or by yourself), you're often judged on your personal job performance. Team interaction is important for some companies (especially at Rackspace), but not for others. Breaking the mindset of being an individual contributor was extremely difficult for me to do.

Managers are judged on the success of the team as a whole. Encouraging your team members to succeed and playing an active role in their personal and professional development is key. Each time you find yourself buried in the weeds of a problem rather than facilitating your team's work on the problem is when your performance as a manager will drop. If you do it more often, you may find that your team members aren't getting the support they need.

Don't be afraid of your team becoming smarter than you
One of the biggest things I've heard from my team is: «Aren't you worried about losing your technical skills when you're a manager?» My answer: «Of course.» Anyone who has technical abilities will always be afraid of watching those abilities wane over time. However, as your team becomes stronger, you should be able to continue learning not through your own work, but through theirs. When your team members see that you're still interested in learning and you're now able to learn from them, they'll become more energized about their own work.

If you find yourself thinking negatively about a potential job candidate because they're smarter than you, step back and think for a moment. Put your own ego aside and consider what's best for you, your team, and your company. Your goal is to build a strong and successful team, not to pad your own ego. If your managers are judging you (as a technical manager) on your technical ability, then you need to solve that problem first.

Inspire instead of direct
Every manager faces the challenge of working with team members who disagree with a particular company policy or with the direction of their particular infrastructure. Keep in mind that your team members are probably not intending to be insubordinate and they might have something useful to contribute.

When you find yourself locking horns with your team members, inspire a discussion about the problem. Break out the disagreement onto a whiteboard and let the team make suggestions for improvements. Even if the entire discussion leads back to the fact that the original problem is inevitable, fostering that feedback loop is critical. You'll learn more about your team while they find ways to express their opinions and feel empowered.

The really tough part is when your team comes up with an alternative plan and you find yourself presenting to your leadership team. Always remember to take it seriously and know that you may need to refine the plan many times over before you find something acceptable for your team and the business.

De-stress by staying on task
If you're anything like me, you need some way to keep tabs on action items coming from meetings, e-mails, phone calls, and walk-ups. I've heard great things about applications like OmniFocus and Things, but I settled on 2Do. I really enjoy a strong to-do list which allows me to set priorities, due dates, and write extended notes about a particular task.

The best way to tackle a wall of tasks is to keep them organized into a concise list. Even if it's a small task, get it into your list so it's on your radar and you won't forget it. Work through the simple tasks and the high priority ones first but watch out for tasks with due dates.

Conclusion
All of these processes get easier over time and although your job will surely have challenges and pitfalls, the enjoyment will greatly increase. I feel privileged to lead a team of talented people who work on a complex and ever-expanding product.

Also, I'd like to mention that I'm not an expert on management! There are probably much better ways to do much of this than I've outlined in this post. Be sure to share your ideas in the comments section below.

How to survive as a technical manager is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The first day of FUDCon 2011 in Tempe is coming to a close tonight and I'm completely exhausted. As promised, I'll try to summarize the day and cover the talks which I attended.

The day started out with Jared Smith's «State of Fedora» address. The audio has already been posted on the wiki, but the speech was very positive overall. He talked about some of the struggles that have happened in the past and how they'll probably happen again in some form or another. It was pretty inspirational and you could obviously tell that people in the room were energized by it.

After the address, all of the talks were pitched in BarCamp format. It was a very efficient and entertaining way to create a schedule for the conference. Everyone had 15-20 seconds to present their talk and then they had to rush outside to post their topic on the wall. We all had the opportunity to go outside and vote for the talks that sounded interesting. Once the votes were tallied, the schedule was set and the conference was fully underway.

The first talk for me was about Marek Goldmann's BoxGrinder. (Note: If you Google for BoxGrinder, make sure that you enter it as a single word. You'll get some wild unrelated results if you use two words.) In short, BoxGrinder gives you the ability to have a kickstart-ish method for automatically building images for virtual machine environments. It's completely plugin-based, so you can have different platform and delivery plugins depending on where your VM needs to be deployed. For example, you could deploy a VM with BoxGrinder that is in a format for VMWare (platform) and is delivered to the target server via SFTP (delivery). The public cloud plugins are only compatible with Amazon's products, but I'm eager to change that during one of the upcoming hackfests.

The Sheepdog talk started up right after lunch and although it was interesting, I think it left most people with quite a few questions when it was over. However, I think people are generally apprehensive when anyone tries to do anything innovative with storage. Losing data due to a bug is a big concern and many of the questions went deeper into data safety than performance and functionality.

Next up was Dave Malcolm's talk about the different implementations of python. This was definitely an eye-opening talk for my coworker and I. Dave covered CPython, Jython, PyPy and various other implementations and compared their advantages and disadvantages. I'm still pretty new to Python (I'm clutching on to ruby, PHP and perl still), but this talk really had me thinking about which implementations are best for a particular environment or task. It was quite a bit of fun to learn about some of the deep underpinnings of Python and how they differ depending on the specific implementation.

Jeff Darcy's talk about CloudFS was very intriguing. I've been a fan of GlusterFS recently, but I eventually moved away due to a lack of enterprise features and degrading performance. Jeff is working to add in encryption and authentication without rewriting the filesystem itself. There are quite a few tricky problems involved in the encryption portion due to partial writes and general security during the handshake process. CloudFS could potentially be a network filesystem which could be shared by multiple tenants with their own individual namespaces and segregated UID's. This could be a big win for providers as they could offer up large amounts of storage in an organized fashion without too many management headaches.

We wrapped up the day of talks with Chris Lalancette's presentation about Deltacloud. In short, it's a bag of daemons that allow you to manage multiple public or private clouds. Everything from image management to provisioning are included in the project. Questions were raised about whether another application was needed since vendor-specific libraries are abundant and libcloud offers many of the same features in a simpler package.

Tonight's social event was FUDPub at ASU's Memoral Union building. The food and drinks were excellent (thanks to Rackspace!) and it was a great opportunity to relax and talk with other Fedora users and developers. We had the opportunity to meet people from around the world while playing round after round of bowling and billiards. The discussions were extremely valuable, but as I said before, it was quite tiring.

I've compiled the FUDCon photos I've taken into a Flickr photo set.

That's the end of today's summary. I'll try to keep this going tomorrow as well. Thanks for reading!

FUDCon 2011: Day One is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.