Игорь Олемской — практические заметки по системному администрированию Linux CentOS

Scientific Linux installations have a package called yum-autoupdate by default and the package contains two files:

# rpm -ql yum-autoupdate
/etc/cron.daily/yum-autoupdate
/etc/sysconfig/yum-autoupdate

The cron job contains the entire script to run automatic updates once a day and the configuration file controls its behavior. However, you can't get the same functionality as Fedora's yum-updatesd package where you can receive notifications for updates rather than automatically updating the packages.

To get those notifications in Scientific Linux, just make two small edits to this portion of /etc/cron.daily/yum-autoupdate:

173           echo "    Starting Yum with command"
174           echo "     /usr/bin/yum -c $TEMPCONFIGFILE -e 0 -d 1 -y update"
175   fi
176   /usr/bin/yum -c $TEMPCONFIGFILE -e 0 -d 1 -y update > $TEMPFILE 2>&1
177   if [ -s $TEMPFILE ] ; then

Adjust the update commands to look like this:

173           echo "    Starting Yum with command"
174           echo "     /usr/bin/yum -c $TEMPCONFIGFILE -e 0 -d 1 -y check-update"
175   fi
176   /usr/bin/yum -c $TEMPCONFIGFILE -e 0 -d 1 -y check-update > $TEMPFILE 2>&1
177   if [ -s $TEMPFILE ] ; then

Since you won't be auto-updating with this script any longer, you may want to comment out the EXCLUDE= line in /etc/sysconfig/yum-autoupdate so that you'll receive notifications for all packages with updates. Also, to avoid having your changes updated with a newer yum-autoupdate package later, add the package to your list of excluded packages in /etc/yum.conf.

Get notifications instead of automatic updates in Scientific Linux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I'll be the first one to admit that Kerberos drives me a little insane. It's a requirement for two of the exams in Red Hat's RHCA certification track and I've been forced to learn it. It provides some pretty nice security features for large server environments. You get central single sign ons, encrypted authentication, and bidirectional validation. However, getting it configured can be a real pain due to some rather archaic commands and shells.

Here's Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:

• sshd calls to pam to authenticate the user
• pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
• the KDC replies to the client with a TGT encrypted with the user's password
• pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
• if pam can decrypt the TGT, it knows the user is providing the right password

Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?

• client calls the KDC and asks for a ticket to grant access to the other service
• KDC replies with two copies of the ticket:
  • one copy is encrypted with the user's current TGT
  • a second copy is encrypted with the password of the network service the user wants to access
• the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
• client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
• client passes the authenticator and the second copy of the ticket it received from the KDC
• the other network service decrypts the second copy of the ticket and verifies the password
• the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
• if the timestamp looks good, the other network service allows the user access

Okay, that's confusing. Let's take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user's password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user's password back to the KDC before they can ask for a KDC. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they'll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.

Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they're also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client's own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn't have any clue what that password is and the attack will be thwarted.

By this point, you're either saying «Oh man, I don't ever want to do this.» or «How do I set up Kerberos?». Stay tuned if you're in the second group. I'll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.

In the meantime, here are a few links for extra Kerberos bedtime reading:

• Kerberos on Wikipedia
• MIT's «Why Kerberos» [PDF]
• How Kerberos Authentication Works

Kerberos for haters is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I used to be one of those folks who would install Fedora, CentOS, Scientific Linux, or Red Hat and disable SELinux during the installation. It always seemed like SELinux would get in my way and keep me from getting work done.

Later on, I found that one of my servers (which I'd previously secured quite thoroughly) had some rogue processes running that were spawned through httpd. Had I actually been using SELinux in enforcing mode, those processes would have probably never even started.

If you're trying to get started with SELinux but you're not sure how to do it without completely disrupting your server's workflow, these tips should help:

Get some good reporting and monitoring
Two of the most handy SELinux tools are setroubleshoot and setroubleshoot-server. If you're running a server without X, you can use my guide for configuring setroubleshoot-server. You will receive email alerts within seconds of an AVC denial and the emails should contain tips on how to resolve the denial if the original action should be allowed. If the AVC denial caught something you didn't expect, you'll know about the potential security breach almost immediately.

Start out with SELinux in permissive mode
If you're overly concerned about SELinux getting in your way, or if you're enabling SELinux on a server that has been running without SELinux since it was installed, start out with SELinux in permissive mode. To make the change effective immediately, just run:

# setenforce 0
# getenforce
Permissive

Edit /etc/sysconfig/selinux to make it persistent across reboots:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive

Adjust booleans before adding your own custom modules
There are a lot of booleans you can toggle to get the functionality you need without adding your own custom SELinux modules with audit2allow. If you wanted to see all of the applicable booleans for httpd, just use getsebool:

# getsebool -a | grep httpd
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> on
httpd_can_sendmail --> on
... and so on ...

Toggling booleans is easy with togglesebool:

# togglesebool httpd_can_network_memcache
httpd_can_network_memcache: active

Now httpd can talk to memcache. You can also use setsebool if you want to be specific about your setting (this is good for scripts):

# setsebool httpd_can_network_memcache on

Tracking your history of AVC denials
All of your AVC denals are logged by auditd in /var/log/audit/audit.log but it's not the easiest file to read and parse. That's where aureport comes in:

# aureport --avc | tail -n 5
45. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 4 fifo_file getattr system_u:object_r:postfix_public_t:s0 denied 1061
46. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 2 fifo_file write system_u:object_r:postfix_public_t:s0 denied 1062
47. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 2 fifo_file open system_u:object_r:postfix_public_t:s0 denied 1062
48. 01/24/2012 14:01:58 sendmail unconfined_u:system_r:httpd_t:s0 160 process setrlimit unconfined_u:system_r:httpd_t:s0 denied 1123
49. 01/24/2012 14:01:58 postdrop unconfined_u:system_r:httpd_t:s0 4 dir search system_u:object_r:postfix_public_t:s0 denied 1124

Summary
There's no need to be scared of or be annoyed by SELinux in your server environment. While it takes some getting used to (and what new software doesn't?), you'll have an extra layer of security and access restrictions which should let you sleep a little better at night.

Getting started with SELinux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

If you want to forward e-mail from root to another user, you can usually place a .forward file in root's home directory and your mail server will take care of the rest:

echo "user@example.com" > /root/.forward

With SELinux, you'll end up getting an AVC denial each time your mail server tries to read the contents of the .forward file:

type=AVC msg=audit(1325543823.787:7416): avc:  denied  { open } for  pid=9850
  comm="local" name=".forward" dev=md0 ino=17694734
  scontext=system_u:system_r:postfix_local_t:s0
  tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file

The reason is that your .forward file doesn't have the right SELinux contexts. You can set the correct contest quickly with restorecon:

# ls -Z /root/.forward
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 /root/.forward
# restorecon -v /root/.forward
restorecon reset /root/.forward context unconfined_u:object_r:admin_home_t:s0->system_u:object_r:mail_forward_t:s0
# ls -Z /root/.forward
-rw-r--r--. root root system_u:object_r:mail_home_t:s0 /root/.forward

Try to send another e-mail to root and you should see the mail server forward the e-mail properly without any additional AVC denials.

SELinux and .forward files is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

When you install Scientific Linux, it will keep you on the same point release that you installed. For example, if you install it from a 6.0 DVD, you'll stay on 6.0 and get security releases for that point release only.

Getting it to behave like Red Hat Enterprise Linux and CentOS is a painless process. Just install the sl6x repository with yum:

yum install yum-conf-sl6x

Check to ensure that you're getting updates from the new repository:

# yum repolist
repo id            repo name                                              status
sl                 Scientific Linux 6.1 - x86_64                          6,251
sl-security        Scientific Linux 6.1 - x86_64 - security updates         548
sl6x               Scientific Linux 6x - x86_64                           6,251
sl6x-security      Scientific Linux 6x - x86_64 - security updates          548
repolist: 13,598

Automatically upgrading to new point releases of Scientific Linux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

SELinux isn't a technology that's easy to tackle for newcomers. However, there's been a lot of work to smooth out the rough edges while still keeping a tight grip on what applications and users are allowed to do on a Linux system. One of the biggest efforts has been around setroubleshoot.

The purpose behind setroubleshoot is to let users know when access has been denied, help them resolve it if necessary, and to reduce overall frustration while working through tight security restrictions in the default SELinux policies. The GUI frontend for setroubleshoot is great for users who run Linux desktops or those who run servers with a display attached. Don't worry, you can configure setroubleshoot on remote servers to send alerts elsewhere when a GUI alert isn't an option.

Install a few packages to get started:

yum install setroubleshoot{-server,-plugins,-doc}

Open /etc/setroubleshoot/setroubleshoot.conf in your favorite text editor and adjust the [email] section to fit your server:

recipients_filepath = /var/lib/setroubleshoot/email_alert_recipients
smtp_port = 25
smtp_host = localhost
from_address = selinux@myserver.com
subject = [MyServer] SELinux AVC Alert

You could probably see it coming, but you need to put the e-mail addresses for your recipients into /var/lib/setroubleshoot/email_alert_recipients:

echo "selinux@mycompany.com" >> /var/lib/setroubleshoot/email_alert_recipients

You'll notice that setroubleshoot doesn't have an init script and it doesn't exist in systemd in Fedora 15. It runs through the dbus-daemon and a quick bounce of the messagebus via its init script brings in the necessary components to run setroubleshoot:

service messagebus restart

A really easy (and safe) test is to ask sshd to bind to a non-standard port. Simply define an additional port on in your /etc/ssh/sshd_config like this:

Port 22
Port 222

When you restart sshd, it will bind to port 22 with success, but it won't be allowed to bind to port 222 (since that's blocked by SELinux as a non-standard port for the ssh_port_t port type). DON'T WORRY! Your sshd server will still be listening on port 22. If you wait a moment, you'll get an e-mail (perhaps two) that not only notify you of the denial, but they make suggestions for how to fix it:

SELinux is preventing /usr/sbin/sshd from name_bind access on the tcp_socket port 222.
 
*****  Plugin bind_ports (99.5 confidence) suggests  *************************
 
If you want to allow /usr/sbin/sshd to bind to network port 222
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 222
   where PORT_TYPE is one of the following: ...

For this particular example, the quick fix would be to run:

semanage port -a -t ssh_port_t -p tcp 222

Much of this post's information was gathered from the detailed documentation on Fedora's setroubleshoot User's FAQ as well as Dan Walsh's setroubleshoot blog post.

Receive e-mail reports for SELinux AVC denials is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I'm using SELinux more often now on my Fedora 15 installations and I came up against a peculiar issue today on a new server. My PHP installation is configured to store its sessions in memcached and I brought over some working configurations from another server. However, each time I accessed a page which tried to initiate a session, the page load would hang for about a minute and I'd find this in my apache error logs:

[Thu Sep 08 03:23:40 2011] [error] [client 11.22.33.44] PHP Warning:
Unknown: Failed to write session data (memcached). Please verify that
the current setting of session.save_path is correct (127.0.0.1:11211)
in Unknown on line 0

I ran through my usual list of checks:

• netstat showed memcached bound to the correct ports/interfaces
• memcached was running and I could reach it via telnet
• memcached-tool could connect and pull stats from memcached
• double-checked my php.ini
• tested memcached connectivity via a PHP and ruby script — they worked

Even after all that, I still couldn't figure out what was wrong. I ran strace on memcached while I ran a curl against the page which creates a session and I found something significant — memcached wasn't seeing any connections whatsoever at that time. A quick check of the lo interface with tcpdump showed the same result. Just before I threw a chair, I remembered one thing:

SELinux.

A quick check for AVC denials showed the problem:

# aureport --avc | tail -n 1
4021. 09/08/2011 03:23:38 httpd system_u:system_r:httpd_t:s0 42 tcp_socket name_connect system_u:object_r:memcache_port_t:s0 denied 31536

I'm far from being a guru on SELinux, so I leaned on audit2allow for help:

# grep memcache /var/log/audit/audit.log | audit2allow
 
#============= httpd_t ==============
#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_relay, httpd_can_network_memcache, httpd_can_network_connect
 
allow httpd_t memcache_port_t:tcp_socket name_connect;

The boolean we're looking for is httpd_can_network_memcache. Flipping the boolean can be done in a snap:

# setsebool -P httpd_can_network_memcache 1
# getsebool httpd_can_network_memcache
httpd_can_network_memcache --> on

After adjusting the boolean, apache was able to make connections to memcached without a hitch. My page which created sessions loaded quickly and I could see data being stored in memcached. If you want to check the status of all of the apache-related SELinux booleans, just use getsebool:

# getsebool -a | grep httpd | grep off$
allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
httpd_can_check_spam --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_read_user_content --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off

If you're interested in SELinux, a good way to get your feet wet is to head over to the CentOS Wiki and review their SELinux Howtos

Getting apache, PHP, and memcached working with SELinux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

In a CentOS 5.5 host, we were requested to upgrade openssh to its latest version. Here are steps I took to quickly do the upgrade. You may like to compile it from source or can take my way of installing it from some repository. Checking existing verison shows 4.3p2: $ ssh -v OpenSSH_4.3p2, OpenSSL 0.9.8e-fips-rhel5 [...]

There are few things which will rattle systems administrators more than a compromised server. It gives you the same feeling that you would have if someone broke into your house or car, except that it's much more difficult (with a server) to determine how to clean up the compromise and found out how the attacker gained access. In addition, leaving a compromise in place for an extended period can lead to other problems:

• your server could be used to gain access other servers
• data could be stolen from your server's databases or storage devices
• an attacker could capture data from your server's local network
• denial of service attacks could be launched using your server as an active participant

The best ways to limit your server's attack surface are pretty obvious: limit network access, keep your OS packages up to date, and regularly audit any code which is accessible externally or internally. As we all know, your server can still become compromised even with all of these preventative measures in place.

Here are some tips which will allow you to rapidly detect a compromise on your servers:

Abnormal network usage patterns and atypical bandwidth consumption
Most sites will have a fairly normal traffic pattern which repeats itself daily. If your traffic graph suddenly has a plateau or spikes drastically during different parts of the day, that could signify that there is something worth reviewing. Also, if your site normally consumes about 2TB of traffic per month and you're at the 1.5TB mark on the fifth day of the month, you might want to examine the server more closely.

On the flip side, look for dips in network traffic as well. This may mean that a compromise is interfering with the operation of a particular daemon, or there may be a rogue daemon listening on a trusted port during certain periods.

Many compromises consist of simple scripts which scan for other servers to infect or participate in large denial of service attacks. The scans may show up as a large amount of packets, but the denial of service attacks will usually consume a large amount of bandwidth. Keeping tabs on network traffic is easily done with open source software like munin, cacti, or MRTG.

Unusual open ports
If you run a web server on port 80, but netstat -ntlp shows something listening on various ports over 1024, those processes are worth reviewing. Use commands like lsof to probe the system for the files and network ports held open by the processes. You can also check within /proc/[pid] to find the directory where the processes were originally launched.

Watch out for processes started within directories like /dev/shm, /tmp or any directories in which your daemons have write access. You might see that some processes were started in a user's home directory. If that's the case, it might be a good time to reset that user's password or clear out their ssh key. Review the output from last authentication logs to see if there are account logins from peculiar locations. If you know the user lives in the US, but there are logins from various other countries over a short period, you've got a serious problem.

I've used applications like chkrootkit and rkhunter in the past, but I still prefer a keen eye and netstat on most occasions.

Command output is unusual
I've seen compromises in the past where the attacker actually took the time to replace integral applications like ps, top and lsof to hide the evidence of the ongoing compromise. However, a quick peek in /proc revealed that there was a lot more going on.

If you suspect a compromise like this one, you may want to use the functionality provided by rpm to verify the integrity of the packages currently installed. You can quickly hunt for changed files by running rpm -Va | grep ^..5.

Keeping tabs on changing files can be a challenge, but applications like tripwire and good ol' logwatch can save you in a pinch.

Summary
We can all agree that the best way to prevent a compromise is to take precautions before putting anything into production. In real life, something will always be forgotten, so detection is a must. It's critical to keep in mind that monitoring a server means more than keeping track on uptime. Keeping tabs on performance anomalies will allow you to find the compromise sooner and that keeps the damage done to a minimum.

Strategies for detecting a compromised Linux server is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

This is going to be very short post SSH v1 is not very safe and if you are looking to pass your site/server for PCI compliance then you must disable it. Don’t worry it is too easy to do. Open /etc/ssh/sshd_config file and disable version 1: $ vi /etc/ssh/sshd_config find line: #Protocol 2,1 and remove [...]