Игорь Олемской — практические заметки по системному администрированию Linux CentOS

I'll be the first one to admit that Kerberos drives me a little insane. It's a requirement for two of the exams in Red Hat's RHCA certification track and I've been forced to learn it. It provides some pretty nice security features for large server environments. You get central single sign ons, encrypted authentication, and bidirectional validation. However, getting it configured can be a real pain due to some rather archaic commands and shells.

Here's Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:

• sshd calls to pam to authenticate the user
• pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
• the KDC replies to the client with a TGT encrypted with the user's password
• pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
• if pam can decrypt the TGT, it knows the user is providing the right password

Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?

• client calls the KDC and asks for a ticket to grant access to the other service
• KDC replies with two copies of the ticket:
  • one copy is encrypted with the user's current TGT
  • a second copy is encrypted with the password of the network service the user wants to access
• the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
• client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
• client passes the authenticator and the second copy of the ticket it received from the KDC
• the other network service decrypts the second copy of the ticket and verifies the password
• the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
• if the timestamp looks good, the other network service allows the user access

Okay, that's confusing. Let's take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user's password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user's password back to the KDC before they can ask for a KDC. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they'll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.

Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they're also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client's own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn't have any clue what that password is and the attack will be thwarted.

By this point, you're either saying «Oh man, I don't ever want to do this.» or «How do I set up Kerberos?». Stay tuned if you're in the second group. I'll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.

In the meantime, here are a few links for extra Kerberos bedtime reading:

• Kerberos on Wikipedia
• MIT's «Why Kerberos» [PDF]
• How Kerberos Authentication Works

Kerberos for haters is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

It's no secret that I'm a big fan of the Routerboard devices and the RouterOS software from Mikrotik that runs on them. The hardware is solid, the software is stable and feature-rich, and I found a great vendor that ships quickly.

I recently added a RB493G (~ $230 USD) to sit in front of a pair of colocated servers. The majority of the setup routine was the same as with my previous devices except for the IPv6 configuration.

In the past, I've set up IPv6 tunnels with Hurricane Electric and it's been mostly a cut-and-paste operation from the sample configuration in their IPv6 tunnel portal. Setting up native IPv6 involved a little more legwork.

If your provider will give you two /64's or an entire /48, getting IPv6 connectivity for your WAN/LAN interfaces is simple. However, if you can only get one /64, you'll have to see if your provider can route it to you via your Mikrotik's link local interface (I wouldn't recommend this for many reasons).

I split my Mikrotik into two interfaces: wan and lanbridge. The lanbridge bridge joins all of the LAN ethernet ports (ether2-9 on the RB493G) and the wan interface connects to the upstream switch.

My configuration:

/ipv6 address
add address=2001:DB8:0:1::2/64 advertise=yes disabled=no eui-64=no interface=wan
add address=2001:DB8:0:2::1/64 advertise=yes disabled=no eui-64=no interface=lanbridge
/ipv6 route
add disabled=no distance=1 dst-address=::/0 gateway=2001:DB8:0:1::1 scope=30 \
  target-scope=10
/ipv6 nd
add advertise-dns=no advertise-mac-address=yes disabled=no hop-limit=64 \
  interface=all managed-address-configuration=no mtu=unspecified \
  other-configuration=no ra-delay=3s ra-interval=3m20s-10m ra-lifetime=30m \
  reachable-time=unspecified retransmit-interval=unspecified
/ipv6 nd prefix default
set autonomous=yes preferred-lifetime=1w valid-lifetime=4w2d

Explanation:

/ipv6 address
add address=2001:DB8:0:1::2/64 advertise=yes disabled=no eui-64=no interface=wan
add address=2001:DB8:0:2::1/64 advertise=yes disabled=no eui-64=no interface=lanbridge

These two lines configure the IPv6 addresses for the firewall's interfaces. My provider's router holds the 2001:DB8:0:1::1/64 address and routes the remainder of that /64 to me via 2001:DB8:0:1::2/64. The second /64 is on the lanbridge interface and my LAN devices take their IP addresses from that block. My provider routes that second /64 to me via the 2001:DB8:0:1::2/64 IP on my wan interface.

/ipv6 route
add disabled=no distance=1 dst-address=::/0 gateway=2001:DB8:0:1::1 scope=30 \
  target-scope=10

I've set a gateway for IPv6 traffic so that the Mikrotik knows where to send internet-bound IPv6 traffic (in this case, to my ISP's core router).

/ipv6 nd
add advertise-dns=no advertise-mac-address=yes disabled=no hop-limit=64 \
  interface=lanbridge managed-address-configuration=no mtu=unspecified \
  other-configuration=no ra-delay=3s ra-interval=3m20s-10m ra-lifetime=30m \
  reachable-time=unspecified retransmit-interval=unspecified
/ipv6 nd prefix default
set autonomous=yes preferred-lifetime=1w valid-lifetime=4w2d

These last two lines configure the neighbor discovery on my lanbridge interface. This allows my LAN devices to do stateless autoconfiguration (which gives them an IPv6 address as well as the gateway).

Want to read up on IPv6?

• Linux IPv6 HOWTO
• IPv6 on Wikipedia
• IPv6 Cheat Sheet [PDF]
• IPv6 Subnetting Card

Native IPv6 connectivity in Mikrotik's RouterOS is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Anyone who has used a 3G ExpressCard or USB stick knows how handy they can be when you need internet access away from home (and away from Wi-Fi). I've run into some situations recently where I needed to share my 3G connection with more than one device without using internet sharing on my MacBook Pro.

That led me to pick up a CradlePoint PHS-300 (discontinued by the manufacturer, but available from Amazon for about $35). It's compatible with my AT&T USBConnect Mercury (a.k.a. Sierra Wireless Compass 885/885U) USB stick.

Configuring the PHS-300 was extremely easy since I could just associate with the wireless network and enter the password printed on the bottom of the unit. However, getting the 3G stick to work was an immense pain. If you're trying to pair up these products, these steps should help:

• Access the PHS-300's web interface
• Click the Modem tab
• Click Settings on the left
• Click Always on under Reconnect Mode
• Uncheck Aggressive Modem Reset
• Put the following into the AT Dial Script text box:

  ATE0V1&F&D2&C1S0=0
  ATDT*99***1#

• Add ISP.CINGULAR to the Access Point Name (APN) box
• Flip the Connect Mode under Dual WiMAX/3G Settings to 3G Only
• Scroll up and push Save Settings and then Reboot Now

Once the PHS-300 reboots, the USB stick may light up, then turn off, and the display on the PHS-300 might show a red light for the 3G card. Wait about 10-15 seconds for the light to turn green. The lights on the 3G stick should be glowing and blinking as well.

So how did I figure this out?

After scouring Google search results, Sierra Wireless FAQ's, CradlePoint's support pages, and trolling through minicom (yes, minicom), I thought I'd try connecting with my MacBook Pro using the 3G Watcher application provided by Sierra Wireless. Before connecting, I opened up Console.app and watched the ppp.log file. Sure enough, two lines popped up that were quite relevant to my interests:

Fri Dec 16 00:37:51 2011 : Initializing phone: ATE0V1&F&D2&C1S0=0
Fri Dec 16 00:37:51 2011 : Dialing: ATDT*99***1#

I didn't have the exact initialization string in the PHS-300 and that was the cause of the failure the entire time.

If you'd like to talk to your USBConnect Mercury stick with minicom, just install minicom from macports (sudo port -v install minicom) and start it up like so:

sudo minicom -D /dev/cu.sierra04

For other Sierra Wireless cards and adapters, there's a helpful page on Sierra Wireless' site for Eee PC users.

Getting online with a CradlePoint PHS-300 and an AT&T USBConnect Mercury is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

The Guide to IP Layer Network Administration with Linux was conceived as a project to document and provide examples for the use of Linux-specific IP networking tools.

The book includes coverage of the following topics: network address translation (NAT), address resolution protocol (ARP), routing, and masquerading. There is a nascent section on bridging, although the focus of the documentation is the use of layer 3 Linux technologies.

Original news source

I've had to do this on a couple of servers to check on the network speeds and sysctl tuning.

On the receiving end (192.168.10.1) bring up netcat listening to a port:

Use the below one liner to send over some data via netcat from 192.168.10.2 which prints out the transfer speed.

Increase the default maximum TCP buffer size and rerun the above test for proper tuning. Make sure to restart netstat listening socket upon sysctl changes.

The following are recommended:

read more

I was recently stumped by an odd incoming network connection hang problem on a linux server. This is a writeup of how I troubleshot the issue and eventually fixed it.

Original news source

The guide to redundant cloud hosting that I wrote recently will need some adjustments as I've fallen hard for the performance and reliability of DRBD and OCFS2. As a few of my sites were gaining in popularity, I noticed that GlusterFS simply couldn't keep up. High I/O latency and broken replication threw a wrench into my love affair with GlusterFS and I knew there had to be a better option.

Diagram of two web nodes with a replicated filesystem using DRBD & OCFS2

I've shared my configuration with my coworkers and I've received many good questions about it. Let's get to the Q&A:

How does the performance compare to GlusterFS?
On Gluster's best days, the data throughput speeds were quite good, but the latency to retrieve the data was often much too high. Page loads on this site were taking upwards of 3-4 seconds with GlusterFS latency accounting for well over 75% of the delays. For small files, GlusterFS's performance was about 20-25x slower than accessing the disk natively. The performance hit for DRBD and OCFS2 is usually between 1.5-3x for small files and difficult to notice for large file transfers.

Couldn't you keep the data separate and then sync it with rsync?
Everyone knows that rsync can be a resource consuming monster and it seems wasteful to call rsync via a cron job to keep my data in sync. There are some periods of the day where the actual data on the web root rarely changes. There are other times where it changes rapidly and I'd end up with nodes out of sync for a few minutes.

To get the just-in-time synchronization that I want, I'd have to run rsync at least once a minute. If the data isn't changing over a long period, rsync would end up crushing the disk and consuming CPU for no reason. DRBD only syncs data when data changes. Also, all reads with DRBD are done locally. This makes is a highly efficient and effective choice for instant synchronization.

Why OCFS2? Isn't that overkill?
When you use DRBD in dual-primary mode, it's functionally equivalent to having a raw storage device (like a SAN) mounted in two places. If you threw an ext4 filesystem onto a LUN on your SAN and then mounted it on two different servers, you'd be in bad shape very quickly. Non-clustered filesystems like ext3 or ext4 can't handle being mounted in more than one environment.

OCFS2 is built primarily to be mounted in more than one place and it comes with its own distributed locking manager (DLM). The configuration files for OCFS2 are extremely simple and you mount it like any other filesystem. It's been part of the mainline Linux kernel since 2.6.19.

What happens when you lose one of the nodes?
The configuration shown above can operate with just one node in an emergency. When the failed node comes back online, DRBD will resync the block device and you can mount the OCFS2 filesystem as you normally would.

You're using an Oracle product? Really?
You've got me there. I'm not a fan of how they treat the open source community with regards to some of their projects, but the OCFS2 filesystem is robust, free, and it meets my needs.

Where's the how-to?
It's coming soon! Stay tuned.

Keep web servers in sync with DRBD and OCFS2 is a post from: Major Hayden's Racker Hacker blog.

c0b6ad7e-f251-11df-b20b-4040336e00ef

One of the most common questions that I see in href="irc://irc.freenode.net/slicehost">my favorite IRC channel is: «How can I secure sshd on my server?» There's no single right answer, but most systems administrators combine multiple techniques to provide as much security as possible with the least inconvenience to the end user.

Here are my favorite techniques listed from most effective to least effective:

SSH key pairs /> By disabling password-based authentication and requiring ssh key pairs, you reduce the chances of compromise via a brute force attack. This can also help you protect against weak account passwords since a valid private key is required to gain access to the server. However, a weak account password is still a big problem if you allow your users to use sudo.

If you're new to using ssh keys, there are href="http://sial.org/howto/openssh/publickey-auth/">many href="http://www.debian-administration.org/articles/530">great href="http://www.linuxquestions.org/linux/answers/Networking/Public_key_authentication_with_ssh">guides that can walk you through the process.

Firewall /> Limiting the source IP addresses that can access your server on port 22 is simple and effective. However, if you travel on vacation often or your home IP address changes frequently, this may not be a convenient way to limit access. Acquiring a server with trusted access through your firewall would make this method easier to use, but you'd need to href="http://en.wikipedia.org/wiki/Recursion">consider the security of that server as well.

The iptables rules would look something like this:

iptables -A INPUT -j ACCEPT -p tcp --dport 22 -s 10.0.0.20
iptables -A INPUT -j ACCEPT -p tcp --dport 22 -s 10.0.0.25
iptables -A INPUT -j DROP -p tcp --dport 22