Игорь Олемской — практические заметки по системному администрированию Linux CentOS

Kristóf Kovács has a fantastic post about some lesser-known Linux tools that can really come in handy in different situations.

If you haven't tried dstat (I hadn't until I saw Kristóf's post), this is a great one to try. You can keep a running tally on various server metrics including load average, network transfer, and disk operations.

Here is some sample output:

----total-cpu-usage---- ---paging-- ---load-avg--- ------memory-usage----- -net/total- ---procs--- --io/total- ---system-- ----tcp-sockets----
usr sys idl wai hiq siq|  in   out | 1m   5m  15m | used  buff  cach  free| recv  send|run blk new| read  writ| int   csw |lis act syn tim clo
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1314B  180B|  0   0   0|   0     0 |  70    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1779B 1004B|  0   0   0|   0     0 |  84    78 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M| 904B  362B|1.0   0 1.0|   0     0 |  75    86 | 13   9   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M|2203B 1559B|  0   0   0|   0     0 | 180   127 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M| 260B  130B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  114B|  0   0   0|   0     0 |  54    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|2271B  872B|  0   0   0|   0     0 |  94    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  54    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1126B 1254B|  0   0   0|   0  24.0 |  80    87 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1030B  130B|  0   0   0|   0     0 |  88    82 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 578B  114B|  0   0   0|   0     0 |  53    64 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1597B  890B|  0   0   0|   0     0 |  85    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 552B  114B|  0   0   0|   0     0 |  63    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1624B 1254B|  0   0   0|   0     0 |  81    75 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 478B  114B|  0   0   0|   0     0 |  67    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 418B  114B|  0   0   0|   0     0 |  59    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1265B  874B|  0   0   0|   0     0 |  82    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 758B  114B|  0   0   0|   0     0 |  60    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1236B 1255B|  0   0   0|   0  4.00 |  93    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  71    70 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 214B  114B|  0   0   0|   0     0 |  55    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1201B  890B|  0   0   0|   0     0 |  80    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 108B  114B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1344B 1254B|  0   0   0|   0  10.0 | 119    85 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 172B  130B|  0   0   0|   0  8.00 |  80    82 | 13   7   0   0   5

Learn more about dstat on Dag Wieërs' site.

Lesser-known but extremely handy Linux tools is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

A fellow Racker showed me httpry about five years ago and I've had in my toolbox as a handy way to watch HTTP traffic. I'd used some crazy tcpdump arguments and some bash one-liners to pull out the information I needed but I never could get the live look that I really wanted.

Here's an example of what httpry's output looks like on a busy site like icanhazip.com:

2012-03-13 23:29:39 186.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	186.x.x.x < -	-	-	HTTP/1.1	200	OK
2012-03-13 23:29:39 187.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.0	-	-
2012-03-13 23:29:39 192.x.x.x	187.x.x.x < -	-	-	HTTP/1.0	200	OK
2012-03-13 23:29:39 188.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	188.x.x.x < -	-	-	HTTP/1.1	200	OK
2012-03-13 23:29:39 189.x.x.x	192.x.x.x > GET	icanhazip.com	/	HTTP/1.1	-	-
2012-03-13 23:29:39 192.x.x.x	189.x.x.x < -	-	-	HTTP/1.1	200	OK

You can watch the requests come in and the responses go out in real time. It even allows for BPF-style packet filters which allow you to narrow down the source and/or destination IP addresses and ports you want to watch. You can run it as a foreground process or as a daemon depending on your needs.

It's now available as a RPM package for Fedora 15, 16, 17 (and rawhide) as well as EPEL 6 (for RHEL/CentOS/SL 6).

New Fedora and EPEL package: httpry is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I originally wrote this post for the Rackspace Blog but I've posted it here just in case anyone following my blog's feed finds it useful. Feel free to share your feedback!

Getting yourself ready for any type of examination is usually a stressful experience that involves procrastination and some late nights leading up to the test. Every time I take one, I always say to myself, “I’m really going to get ahead of this next time and study early. This last minute stuff is terrible.” But I always forget all of this as the next exam rolls around.

Quick note: As you read through the remainder of the post, you may wonder why some of it is a bit vague. Every Red Hat test taker is under a NDA to prevent disclosure of test information that may reduce the security of the exam itself. Penalties start with losing credit for the exams previously taken and they can escalate up to legal action. I hope you’ll understand why I’m not able to go into details about certain portions of the Red Hat examinations.

I’ve taken seven Red Hat exams already: two for the RHCE and five for the RHCA. These tests certainly aren’t easy, but there are some good guidelines and tips you can use to make your studying efforts less stressful and more productive. Without further ado, here are my recommendations for prospective Red Hat examinees:

Build a flexible study environment

This is critical. You’ll need some spare servers or some available virtual machines to practice the objectives on each exam. However, don’t feel like you need to spend the money on a Red Hat subscription to get your studying done. Most of the test objectives on the majority of exams can be completed with very similar Linux distributions, like Scientific Linux or CentOS. Look for a version of the distribution that is closest to what you’ll be tested on at exam time. Your study environment should meet some basic criteria:

• You should be able to quickly build and tear down servers or virtual machines
• Keep the latency to your environment low to avoid getting frustrated
• Use applications like VirtualBox, VMWare Fusion/Workstation to practice on your own computer
• Consider using VMs from cloud providers if you’re under a time crunch

Some exams may require some bare-metal access to the server itself (especially EX442), so keep that in mind when you’re looking for a good practice environment. You may need some specific network or storage setups for some exams (as with EX436). If you’re not sure what you need, be sure to ask your instructor or someone else you know who has taken the exam already.

Prioritize doing over reading

The Red Hat exams are all hands-on, practical exams. You won’t find any essays or multiple-choice questions in these exams. Although the materials from Red Hat are full of good information, reading this information can only get you so far. You need to practice setting up the services on your own to be fully prepared for the test. If you’re not pressed for time, reading through the book can give you some details about the lab sequences, which you might miss by solely reading through labs themselves.

Research the why, not the what, to remember

This is especially important for the RHCA exam track. You may find that there is a ton of material to cover for the exam and that it’s difficult to remember each command to bring a certain service online or to repair a problem. Instead of thinking through the problem as “first, I do this, then I do this”, try to understand why each step is important in the first place.

Here’s a good example. I’ll be the first one to admit that Kerberos drives me crazy. I’ve even written posts about it. The commands seemed really archaic, the daemons didn’t make sense, and the lack of readline support in the Kerberos tools made me want to throw my computer out the window (come on, MIT!). I put my class materials aside, went to Google in a browser, and started researching Kerberos.

I read some of MIT’s documentation, ventured over to Wikipedia, and poked at some of the documentation within the Kerberos RPM packages. After a while, I began to realize how it all fit together. “Okay,” I thought to myself, “I need principals in a keytab to do these things, but I need to have a database for the admin stuff first.” Suddenly, the order of things in my head wasn’t just memorized any longer. The process of operations seemed to make logical sense because I fully understood how the pieces of a Kerberos infrastructure fit together.

If you start to get discouraged, take a break and learn more about why you’re doing what you’re doing. Once it becomes second nature, working through the problems on the exam becomes much easier.

Lean on your available resources

Don’t forget that there are other knowledgeable folks available to talk to when you get bogged down. Lean on other RHCE’s, RHCA’s, or experienced Linux users to get the answers or explanations you need. If you already have a Red Hat certification, head over to the Red Hat Certification Forums and meet up with other examinees that are discussing test preparation.

Also, you’ll find some knowledgeable (but sometimes snarky or quirky) people on IRC who are eager to point you in the right direction. Try the #rhel, #centos, or #fedora channels if you’re struggling through the configuration of a certain service. Many Linux users may roll their eyes about it, but Twitter is also a pretty good way to reach out to people who have a lot of Linux experience.

Summary

Remember to lean on the knowledge of others, get hands-on with the test objectives and do your research when you’re frustrated. The exams from Red Hat are generally difficult and cover a lot of material, but with the right amount of preparation and determination you can pass the exams and get the certifications you want.

Preparing for Red Hat Exams is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The grades came back last Friday and I've passed the last exam in the requirements to become a Red Hat Certified Architect (RHCA). I was fortunate enough to be part of Rackspace's RHCA pilot program and we took our first exam back at the end of 2010. It's definitely a good feeling to be finished and I'm definitely ready to give back some knowledge to the readers of this blog.

First things first: there are going to be many part of this post which probably aren't as specific as you'd like. A lot of that is due to the NDA that all Red Hat examinees agree to when they take an exam. We aren't allowed to talk about what was on the exam or our experiences during the exam. If we do, penalties range from smaller things like losing certifications all the way up to serious stuff like legal action. It goes without saying that I want to protect the security of the exams, I don't want to lose my certifications, and I don't want to hire a lawyer. Please try to keep this in mind if you yearn for more specifics than I'm able to give.

Red Hat Certified Engineer
The RHCSA and RHCE exams are the first step on the path to the RHCA. You can't take any of the RHCA prerequisite exams without it. These exams cover a really broad spectrum of material including apache configuration, NFS, iptables and mail services. The two links above will take you to the exam objectives for each exam.

I've always recommended the RHCE exam for Linux administrators who are trying to sharpen their skills and get to the next level whether they use Red Hat or not. The exam covers a lot of good material that makes a solid foundation for any Linux user without throwing in too many Red Hat-specific knowledge.

The exam (like all Red Hat exams) is fully practical. There are no multiple choice questions or essays. You'll have to meet all of the objectives by logging into a local Red Hat system and making the system do what it needs to do.

Quick tips for the RHCSA/RHCE exams:

• Keep your eye on the clock. Time can really get away from you if you get stuck in the weeds on a problem that should be relatively straightforward.
• Leave time at the end to check your work. When you set up a lot of services, it's inevitable that you might configure a service for one problem that breaks the functionality required by a problem you completed already.
• Always reboot before you leave. We all forget to use chkconfig when we're in a hurry.
• Practice, practice, practice. There's not one objective on this exam that you can't test in a VM on your own.

Red Hat Enterprise System Monitoring and Performance Tuning
Our group at Rackspace started off with EX442 and it was a very difficult way to start off the RHCA track. Take a look at the objectives and you'll see that much of the exam is related to tweaking system performance and then monitoring that performance with graphs and raw data. You'll have to turn a lot of knobs on the kernel and you'll need to know where to store these configurations so they'll be persistent.

In addition, the objective regarding TCP buffers and related settings is a real challenge. You'll have to wrestle with some math that appears to be relatively simple, but can get confusing quickly. Some of the settings can't really be checked to know if your setting is correct. The objectives mention tuning disk scheduling — you don't really have the time or tools to know if your setting is ideal.

Quick tips for EX442:

• Use the documentation available to you. Install the kernel-doc package while you practice and during the exam.
• Be careful with your math. You have a Linux machine in front of you! Don't forget about bc.
• Watch your units. Know the difference between a kilobyte (KB) and a kibibyte (KiB).
• Make comments in files where you adjust kernel configurations. It will help you keep track of which question the kernel adjustment is meant to satisfy.

Red Hat Enterprise Storage Management
I'm surprised to say this now, but I actually enjoyed EX436. I've always used other clustering tools like heartbeat and pacemaker, but I've never had the need to use the Red Hat Cluster Suite. Although RHCS definitely has a lot of quirks and rough edges, it's pretty solid once you get familiar with the GUI and command line tools.

You get the opportunity to mess around with some pretty useful technology like iSCSI, GFS, and clustered LVM. These are things that you're probably already using or will be using soon in a large server environment. The web interface for RHCS is quite peculiar and you may find yourself wanting to put your fist through the screen when you're staring down the endless animated GIFs when the cluster is syncing its configuration. Do your best to be patient because you certainly don't want to short circuit the cluster sync.

Quick tips for EX436:

• Be patient. You'll feel like the RHCS web interface is mocking you when you're pressed for time.
• Watch the clock. It's extremely easy to burn a lot of time on this exam if you get stuck on a particular problem.
• Double check your entries in the web interface. Make sure you're doing things in the right order and that you've set up the prerequisites before adding services to the cluster. If you get it wrong, you could put your cluster into a weird state.
• Use man pages. If you don't mess with GFS a lot, the man pages will save you in a pinch.

Red Hat Enterprise Deployment and Systems Management
If there's one exam where time management is critical, it's EX401. Importing data into the Satellite Server takes quite a bit of time and there's almost nothing you can do to speed it up. It probably goes without saying, but as with most long-running tasks, you'll want to run it in screen. The last thing you'd ever want is to abort the import due to an errant click or CTRL-C (I did it while practicing — it's aggravating).

There are other test objectives which you can either complete or partially complete while you wait for the import to finish.

Also, take the time to really dig into the Satellite Server web interface while your practicing for the exam. Knowing where to find the most common configuration items will really save some time when you're in the exam. You can sometimes get pretty bogged down in the interface so don't forget to use multiple tabs to keep your work organized.

I felt like this exam was the easiest out of the bunch since you could go back and test every single question with good time management. Did I mention how important time management was on this exam already? If I forgot to mention it earlier, be sure to focus on time management for this test.

Quick tips for EX401:

• Time management will make or break you on this test. Keep an eye on the clock and make sure you've done absolutely every piece of the exam that you can while you wait for the server to do its work.
• Scour the web interface. Keep a mental map in your mind where the big chunks of configuration items are.
• Go back and test everything. If you manage your time well, you should have enough time to verify each and every objective on this exam.

Red Hat Enterprise Directory Services and Authentication
At first, EX423 looks pretty straightforward. Red Hat's authentication configuration tools make LDAP authentication setup pretty easy. However, this exam comes with a lot of curveballs.

The GUI interface for the Directory Services component is a little frustrating to use. I found that the GUI stopped responding to keyboard input occasionally unless I clicked on another window and came back. If you misconfigure the SSL certificates in the interface, your LDAP server is down for the count. If you don't input the correct data into the setup scripts at the beginning, you might not notice it until much later when it's either too difficult to dig yourself out of the hole or it's too late to start over with a clean configuration.

I didn't feel pressed for time on this exam too much and that was pretty refreshing after taking the EX401 test. It's extremely critical to watch what you type and click on this exam. Some mistakes can be quickly corrected while others may require you to blow away the LDAP server configuration and re-provision the whole thing.

Quick tips for EX423:

• Always watch what you're typing. A simple mistake can lead to confusion or bigger issues down the road.
• Don't ignore the LDIF objectives. As you practice, you'll find that manipulating LDIF files is a little more involved than you expected.
• Practice starting over. Throw out your Directory Services configuration and get the experience of what it's like to start over and get back in the game.

Red Hat Enterprise Security: Network Services
There's no sugar coating it — EX333 is a beast. It's a six hour exam broken into two three-hour chunks. It covers a ton of material and I refer to it as «the RHCE on steroids.» You might argue that I thought it was hard since it was the last test and I was ready to be finished, but I really think this exam is a tough one.

Practicing for the Kerberos and DNS objectives was the hardest for me. I just couldn't understand Kerberos, no matter how hard I tried. The realization that I would really have to learn it soon set in. I dug into the Kerberos design documentation on MIT's site, read the summaries on Wikipedia, and scoured the documentation available in the Kerberos RPM packages. Once I understood why Kerberos is set up the way it is and why the security measures are present, everything began to come together. I was able to remember the steps not because I was memorizing them, but because I understood how Kerberos worked.

When you're working through the DNS objectives, keep an eye out for punctuation. I blew through a good 20 minutes in what seemed like the blink of an eye when I forgot a period in my TSIG key configuration while studying. Make sure you use the resources available to you, like system-config-bind and sample configs in /usr/share/doc/bind*/examples/. Get to know commands like dig really well.

If you're overwhelmed by OpenSSL's command line syntax, check out the /etc/pki/tls/misc/CA script. There are some handy comments at the top of the script that explain how to use it. You can also pluck OpenSSL commands right out of the script if you need to run them yourself.

• Don't just memorize. Do some research to understand how everything fits together.
• Manage your time. DNS and Kerberos have lots of small nuances that can become time sinks when done incorrectly.
• Use the available documentation and tools. Try practicing without study materials so that you're forced to use the docs and tools available within the server.

Ranking the exams
A couple of folks on Twitter asked me to rank the exams from most difficult to least difficult. Keep in mind that these are a little subjective since I was more familiar with some objectives than others for certain tests.

• EX333 — Enterprise Security: Network Services: a tubload of material and a very long exam
• EX442 — System Monitoring and Performance Tuning: very difficult to check your work, lots of calculations
• EX423 — Directory Services and Authentication: not a lot of material to cover, but tons of curveballs
• EX436 — Storage Management: the web interface made things much easier, lots of documentation available
• EX401 — Deployment and Systems Management: every objective can be tested, I build RPM's already

Looking back at the long road to becoming a Red Hat Certified Architect is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I'll be the first one to admit that Kerberos drives me a little insane. It's a requirement for two of the exams in Red Hat's RHCA certification track and I've been forced to learn it. It provides some pretty nice security features for large server environments. You get central single sign ons, encrypted authentication, and bidirectional validation. However, getting it configured can be a real pain due to some rather archaic commands and shells.

Here's Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:

• sshd calls to pam to authenticate the user
• pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
• the KDC replies to the client with a TGT encrypted with the user's password
• pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
• if pam can decrypt the TGT, it knows the user is providing the right password

Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?

• client calls the KDC and asks for a ticket to grant access to the other service
• KDC replies with two copies of the ticket:
  • one copy is encrypted with the user's current TGT
  • a second copy is encrypted with the password of the network service the user wants to access
• the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
• client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
• client passes the authenticator and the second copy of the ticket it received from the KDC
• the other network service decrypts the second copy of the ticket and verifies the password
• the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
• if the timestamp looks good, the other network service allows the user access

Okay, that's confusing. Let's take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user's password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user's password back to the KDC before they can ask for a TGT. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they'll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.

Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they're also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client's own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn't have any clue what that password is and the attack will be thwarted.

By this point, you're either saying «Oh man, I don't ever want to do this.» or «How do I set up Kerberos?». Stay tuned if you're in the second group. I'll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.

In the meantime, here are a few links for extra Kerberos bedtime reading:

• Kerberos on Wikipedia
• MIT's «Why Kerberos» [PDF]
• How Kerberos Authentication Works

Kerberos for haters is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Although Citrix recommends against using software RAID with XenServer due to performance issues, I've had some pretty awful experiences with hardware RAID cards over the last few years. In addition, the price of software RAID makes it a very desirable solution.

Before you get started, go through the steps to disable GPT. That post also explains an optional adjustment to get a larger root partition (which I would recommend). You cannot complete the steps in this post if your XenServer installation uses GPT.

You should have three partitions on your first disk after the installation:

# fdisk -l /dev/sda
-- SNIP --
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2611    20971520   83  Linux
/dev/sda2            2611        5222    20971520   83  Linux
/dev/sda3            5222       19457   114345281   8e  Linux LVM

Here's a quick explanation of your partitions:

• /dev/sda1: the XenServer root partition
• /dev/sda2: XenServer uses this partition for temporary space during upgrades
• /dev/sda3: your storage repository should be in this logical volume

We need to replicate the same partition structure across each of your drives and the software RAID volume will span the across the third partition on each disk. Copying the partition structure from disk to disk is done easily with sfdisk:

WHOA THERE! NO TURNING BACK! This step is destructive! If your other disks have any data on them, this step will make it (relatively) impossible to retrieve data on those disks again. Back up any data on the other disks in your XenServer machine before running these next commands.

sfdisk -d /dev/sda | sfdisk --force /dev/sdb
sfdisk -d /dev/sda | sfdisk --force /dev/sdc
sfdisk -d /dev/sda | sfdisk --force /dev/sdd

If you have only two disks, stop with /dev/sdb and you'll be making a RAID 1 array. My machine has four disks and I'll be making a RAID 10 array.

We need to destroy the main storage repository, but we need to unplug the physical block device first. Get the storage repository uuid first, then use it to find the corresponding physical block device. Once the physical block device is unplugged, the storage repository can be destroyed:

# xe sr-list name-label=Local\ storage | head -1
uuid ( RO)                : 75264965-f981-749e-0f9a-e32856c46361
# xe pbd-list sr-uuid=75264965-f981-749e-0f9a-e32856c46361 | head -1
uuid ( RO)                  : ff7e9656-c27c-1889-7a6d-687a561f0ad0
# xe pbd-unplug uuid=ff7e9656-c27c-1889-7a6d-687a561f0ad0
# xe sr-destroy uuid=75264965-f981-749e-0f9a-e32856c46361

All of the LVM data from /dev/sda3 should now be gone:

# lvdisplay && vgdisplay && pvdisplay
#

Change the third partition on each physical disk to be a software RAID partition type:

echo -e "t\n3\nfd\nw\n" | fdisk /dev/sda
echo -e "t\n3\nfd\nw\n" | fdisk /dev/sdb
echo -e "t\n3\nfd\nw\n" | fdisk /dev/sdc
echo -e "t\n3\nfd\nw\n" | fdisk /dev/sdd

Stop here and reboot your XenServer box to pick up the new partition changes. Once the server comes back from the reboot, start up a software RAID volume with mdadm:

// RAID 1 for two drives
mdadm --create /dev/md0 -l 1 -n 2 /dev/sda3 /dev/sdb3
// RAID 10 for four drives
mdadm --create /dev/md0 -l 10 -n 4 /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

Check to see that your RAID array is building:

# cat /proc/mdstat
Personalities : [raid10]
md0 : active raid10 sdd3[3] sdc3[2] sdb3[1] sda3[0]
      228690432 blocks 64K chunks 2 near-copies [4/4] [UUUU]
      [>....................]  resync =  0.3% (694272/228690432) finish=16.4min speed=231424K/sec

Although you don't have to wait for the resync to complete, just be aware that XenServer doesn't do well with a lot of disk I/O within dom0. You may notice unusually slow performance in dom0 until it finishes. Save the array's configuration for reboots:

mdadm --detail --scan > /etc/mdadm.conf

Edit the /etc/mdadm.conf file and append auto=yes to the end of the line (but leave everything on one line):

ARRAY /dev/md0 level=raid10 num-devices=4 metadata=0.90 \
  UUID=2876748c:5117eed5:ce4d62d3:9592bd84 auto=yes

Create a new storage repository on the RAID volume with thin provisioning (thanks to Spherical Chicken for the command):

xe sr-create content-type=user type=ext device-config:device=/dev/md0 shared=false name-label="Local storage"

This command takes some time to complete since it makes logical volumes and then makes an ext3 filesystem for the new storage repository. Bigger RAID arrays will take more time and it's guaranteed to take longer than you'd expect if your RAID array is still building. As soon as it completes, you'll be given the uuid of your new storage repository and it should appear within the XenCenter interface.

TIP: If you run into any problems during reboots, open /boot/extlinux.conf and remove splash and quiet from the label xe boot section. This removes the framebuffer during boot-up and it causes a lot more output to be printed to the console. It won't affect the display once your XenServer box has fully booted.

XenServer 6: Storage repository on software RAID is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

XenServer 6 is a solid virtualization platform, but the installer doesn't give you many options for customized configurations. By default, it installs with a 4GB root partition and uses GUID Partition Tables (GPT). GPT is new in XenServer 6.

I'd rather use MBR partition tables and get a larger root partition. If you want to make these adjustments in your XenServer 6 installation, follow these steps after booting into the XenServer 6 install disc:

When the installer initially boots, press F2 to access the advanced installation options.

Type shell and press enter. The installer should begin booting into a pre-installation shell where you can make your adjustments.

Once you've booted into the pre-installation shell, type vi /opt/xensource/installer/constants.py and press enter.

Change GPT_SUPPORT = True to GPT_SUPPORT = False to disable GPT and use MBR partition tables. Adjust the value of root_size from 4096 (the default) to a larger number to get a bigger root partition. The size is specified in MB, so 4096 is 4GB. Save the file and exit vim.

Type exit and the installer should start.

Once the installation is complete, you should have a bigger root partition on a MBT partition table:

# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              20G  1.8G   17G  10% /
# fdisk -l /dev/sda
 
Disk /dev/sda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        2611    20971520   83  Linux
/dev/sda2            2611        5222    20971520   83  Linux
/dev/sda3            5222       19457   114345281   8e  Linux LVM

XenServer 6: Disable GPT and get a larger root partition is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

It's no secret that I'm a big fan of the Routerboard devices and the RouterOS software from Mikrotik that runs on them. The hardware is solid, the software is stable and feature-rich, and I found a great vendor that ships quickly.

I recently added a RB493G (~ $230 USD) to sit in front of a pair of colocated servers. The majority of the setup routine was the same as with my previous devices except for the IPv6 configuration.

In the past, I've set up IPv6 tunnels with Hurricane Electric and it's been mostly a cut-and-paste operation from the sample configuration in their IPv6 tunnel portal. Setting up native IPv6 involved a little more legwork.

If your provider will give you two /64's or an entire /48, getting IPv6 connectivity for your WAN/LAN interfaces is simple. However, if you can only get one /64, you'll have to see if your provider can route it to you via your Mikrotik's link local interface (I wouldn't recommend this for many reasons).

I split my Mikrotik into two interfaces: wan and lanbridge. The lanbridge bridge joins all of the LAN ethernet ports (ether2-9 on the RB493G) and the wan interface connects to the upstream switch.

My configuration:

/ipv6 address
add address=2001:DB8:0:1::2/64 advertise=yes disabled=no eui-64=no interface=wan
add address=2001:DB8:0:2::1/64 advertise=yes disabled=no eui-64=no interface=lanbridge
/ipv6 route
add disabled=no distance=1 dst-address=::/0 gateway=2001:DB8:0:1::1 scope=30 \
  target-scope=10
/ipv6 nd
add advertise-dns=no advertise-mac-address=yes disabled=no hop-limit=64 \
  interface=all managed-address-configuration=no mtu=unspecified \
  other-configuration=no ra-delay=3s ra-interval=3m20s-10m ra-lifetime=30m \
  reachable-time=unspecified retransmit-interval=unspecified
/ipv6 nd prefix default
set autonomous=yes preferred-lifetime=1w valid-lifetime=4w2d

Explanation:

/ipv6 address
add address=2001:DB8:0:1::2/64 advertise=yes disabled=no eui-64=no interface=wan
add address=2001:DB8:0:2::1/64 advertise=yes disabled=no eui-64=no interface=lanbridge

These two lines configure the IPv6 addresses for the firewall's interfaces. My provider's router holds the 2001:DB8:0:1::1/64 address and routes the remainder of that /64 to me via 2001:DB8:0:1::2/64. The second /64 is on the lanbridge interface and my LAN devices take their IP addresses from that block. My provider routes that second /64 to me via the 2001:DB8:0:1::2/64 IP on my wan interface.

/ipv6 route
add disabled=no distance=1 dst-address=::/0 gateway=2001:DB8:0:1::1 scope=30 \
  target-scope=10

I've set a gateway for IPv6 traffic so that the Mikrotik knows where to send internet-bound IPv6 traffic (in this case, to my ISP's core router).

/ipv6 nd
add advertise-dns=no advertise-mac-address=yes disabled=no hop-limit=64 \
  interface=lanbridge managed-address-configuration=no mtu=unspecified \
  other-configuration=no ra-delay=3s ra-interval=3m20s-10m ra-lifetime=30m \
  reachable-time=unspecified retransmit-interval=unspecified
/ipv6 nd prefix default
set autonomous=yes preferred-lifetime=1w valid-lifetime=4w2d

These last two lines configure the neighbor discovery on my lanbridge interface. This allows my LAN devices to do stateless autoconfiguration (which gives them an IPv6 address as well as the gateway).

Want to read up on IPv6?

• Linux IPv6 HOWTO
• IPv6 on Wikipedia
• IPv6 Cheat Sheet [PDF]
• IPv6 Subnetting Card

Native IPv6 connectivity in Mikrotik's RouterOS is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Anyone who has used a 3G ExpressCard or USB stick knows how handy they can be when you need internet access away from home (and away from Wi-Fi). I've run into some situations recently where I needed to share my 3G connection with more than one device without using internet sharing on my MacBook Pro.

That led me to pick up a CradlePoint PHS-300 (discontinued by the manufacturer, but available from Amazon for about $35). It's compatible with my AT&T USBConnect Mercury (a.k.a. Sierra Wireless Compass 885/885U) USB stick.

Configuring the PHS-300 was extremely easy since I could just associate with the wireless network and enter the password printed on the bottom of the unit. However, getting the 3G stick to work was an immense pain. If you're trying to pair up these products, these steps should help:

• Access the PHS-300's web interface
• Click the Modem tab
• Click Settings on the left
• Click Always on under Reconnect Mode
• Uncheck Aggressive Modem Reset
• Put the following into the AT Dial Script text box:

  ATE0V1&F&D2&C1S0=0
  ATDT*99***1#

• Add ISP.CINGULAR to the Access Point Name (APN) box
• Flip the Connect Mode under Dual WiMAX/3G Settings to 3G Only
• Scroll up and push Save Settings and then Reboot Now

Once the PHS-300 reboots, the USB stick may light up, then turn off, and the display on the PHS-300 might show a red light for the 3G card. Wait about 10-15 seconds for the light to turn green. The lights on the 3G stick should be glowing and blinking as well.

So how did I figure this out?

After scouring Google search results, Sierra Wireless FAQ's, CradlePoint's support pages, and trolling through minicom (yes, minicom), I thought I'd try connecting with my MacBook Pro using the 3G Watcher application provided by Sierra Wireless. Before connecting, I opened up Console.app and watched the ppp.log file. Sure enough, two lines popped up that were quite relevant to my interests:

Fri Dec 16 00:37:51 2011 : Initializing phone: ATE0V1&F&D2&C1S0=0
Fri Dec 16 00:37:51 2011 : Dialing: ATDT*99***1#

I didn't have the exact initialization string in the PHS-300 and that was the cause of the failure the entire time.

If you'd like to talk to your USBConnect Mercury stick with minicom, just install minicom from macports (sudo port -v install minicom) and start it up like so:

sudo minicom -D /dev/cu.sierra04

For other Sierra Wireless cards and adapters, there's a helpful page on Sierra Wireless' site for Eee PC users.

Getting online with a CradlePoint PHS-300 and an AT&T USBConnect Mercury is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Before we get started, I really ought to drop this here:

Upgrading Fedora via yum is not the recommended method. Your first choice for upgrading Fedora should be to use preupgrade. Seriously.

This begs the question: When should you use another method to upgrade Fedora? What other methods are there?

You have a few other methods to get the upgrade done:

• Toss in a CD or DVD: You can upgrade via the anaconda installer provided on the CD, DVD or netinstall media. My experiences with this method for Fedora (as well as CentOS, Scientific Linux, and Red Hat) haven't been too positive, but your results may vary.
• Download the newer release's fedora-release RPM, install it with rpm, and yum upgrade: This is the really old way of doing things. Don't try this (read the next bullet).
• Use yum's distro-sync functionality: If you can't go the preupgrade route, I'd recommend giving this a try. However, leave plenty of time to fix small glitches after it's done (and after your first reboot).

Personal anecdote time (Keep scrolling for the meat and potatoes)
I have a dedicated server at Joe's Datacenter (love those folks) with IPMI and KVM-over-LAN access. The preupgrade method won't work for me because my /boot partition is on a software RAID volume. There's a rat's nest of a Bugzilla ticket over on Red Hat's site about this problem. I'm really only left with a live upgrade using yum.

Live yum upgrade process
Before even beginning the upgrade, I double-checked that I'd applied all of the available updates for my server. Once that was done, I realized I was one kernel revision behind and I rebooted to ensure I was in the latest Fedora 15 kernel.

A good practice here is to run package-cleanup --orphans (it's in the yum-utils package) to find any packages which don't exist on any Fedora mirrors. In my case, I had two old kernels and a JungleDisk package. I removed the two old kernels (probably wasn't necessary) and left JungleDisk alone (it worked fine after the upgrade). If you have any external repositories, such as Livna or RPMForge, you may want to disable those until the upgrade is done. Should the initial upgrade checks bomb out, try adding as few repositories back in as possible to see if it clears up the problem.

Once you make it this far, just follow the instructions available in Fedora's documentation: Upgrading Fedora using yum. I set SELinux to permissive mode during the upgrade just in case it caused problems.

I'd recommend skipping the grub2-install portion since your original grub installation will still be present after the upgrade. If your server has EFI (not BIOS), don't use grub2 yet. Keep an eye on the previously mentioned documentation page to see if the problems get ironed out between grub2 and EFI.

Before you reboot, be sure to get a list of your active processes and daemons. After your reboot, some old SysVinit scripts will be converted into Systemd service scripts. They might not start automatically and you might need to enable and/or start some services.

New to Systemd? This will be an extremely handy resource: SysVinit to Systemd Cheatsheet.

I haven't seen too many issues after cleaning up some daemons that didn't start properly. There is a problem between asterisk and SELinux that I haven't nailed down yet but it's not a showstopper.

Good luck during your upgrades. Keep in mind that Fedora 15 could be EOL'd as early as May or June 20102 when Fedora 17 is released.

Live upgrade Fedora 15 to Fedora 16 using yum is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.