Игорь Олемской — практические заметки по системному администрированию Linux CentOS

I sometimes enjoy living on the edge occasionally and that sometimes means I keep up with OpenStack changes commit by commit. If you're in the same boat as I am, you may save some time by using my repository of bleeding-edge Python packages from the OpenStack projects:

• pypi.mhtx.net

Python packages are updated moments after the commit is merged into the repositories under OpenStack's github account.

Although the packages will contain the latest code available, rest assured that the code has passed an initial code review (by humans), unit tests, and varying levels of functional or integrated testing. There may still be a bug or two cropping up after that, so be aware of that as you utilize these packages.

The package versions utilize a standard format:

[package]-[version]-[git commit count]-[short commit hash]

If you need to check the git log up to that particular commit, just run git log:

git log [short commit hash]

Instructions for configuring pip or easy_install are provided within the repository.

In addition, the repository is accessible via IPv4 and IPv6.

OpenStack bleeding-edge Python packages are now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Regular users of Python's package tools like pip or easy_install are probably familiar with the PyPi repository. It's a one-stop-shop to learn more about available Python packages and get them installed on your server.

However, certain folks may find the need to host a local PyPi repository for their own packages. You may need it to store Python code which you don't plan to release publicly or you may need to add proprietary patches to upstream Python packages. Regardless of the reason to have it, a local PyPi repository is relatively easy to configure.

You'll need to start with a base directory for your PyPi repository. For this example, I chose /var/pypi. The directory structure should look something like this:

/var/pypi/simple/[package_name]/[package_tarball]

For a package like pip, you'd make a structure like this:

/var/pypi/simple/pip/pip-1.0.2.tar.gz

Once you have at least one package stored locally, it's time to configure apache. Here's a snippet from the virtual host I configured:

DocumentRoot /var/pypi/
ServerName pypi.example.com
 
Options +Indexes
 
RewriteEngine On
RewriteRule ^/robots.txt - [L]
RewriteRule ^/icons/.* - [L]
RewriteRule ^/index\..* - [L]
 
RewriteCond /var/pypi/$1 !-f
RewriteCond /var/pypi/$1 !-d
RewriteRule ^/(.*)/?$ http://pypi.python.org/$1 [R,L]

The last set of rewrite directives check to see if the request refers to an existing file or directory under your document root. If it does, your server will reply with a directory listing or with the actual file to download. If the directory or file doesn't exist, apache will send the client a redirection to the main PyPi site.

Reload your apache configuration to bring in your new changes. Let's try to download the pip tarball from our local server in the example I mentioned above:

$ curl -I http://pypi.example.com/simple/pip/
HTTP/1.1 200 OK
 
$ curl -I http://pypi.example.com/simple/pip/pip-1.0.2.tar.gz
HTTP/1.1 200 OK

I've obviously snipped a bit of the response above, but you can see that apache is responding with 200's since it has the directories and files that I was trying to retrieve via curl. Let's try to get something we don't have locally, like kombu:

$ curl -I http://pypi.example.com/simple/kombu/
HTTP/1.1 302 Found
Location: http://pypi.python.org/simple/kombu/

Our local PyPi repository doesn't have kombu so it will refer our Python tools over to the official PyPi repository to get the listing of available package versions for kombu.

Now we need to tell pip to use our local repository. Edit ~/.pip/pip.conf and add:

[global]
index-url = http://pypi.example.com/simple/

If you'd rather use easy_install, edit ~/.pydistutils.cfg and add:

[easy_install]
index_url = http://pypi.example.com/simple/

Once your tools are configured, try installing a package you have locally and try to install one that you know you won't have locally. You can add -v to pip install to watch it retrieve different URL's to get the packages it needs. If you spot any peculiar behavior or unexpected redirections, double-check your mod_rewrite rules in your apache configuration and check the spelling of your directories under your document root.

Create a local PyPi repository using only mod_rewrite is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

I was able to get htauth againt ldap and restricting against groups using:

Here is what the ldap search entry looks like:

It is essential to enter «AuthLDAPGroupAttributeIsDN off» and «AuthLDAPGroupAttribute memberUid» for it to get to the member attribute.

Reference: mod_authnz_ldap

read more

I'm using SELinux more often now on my Fedora 15 installations and I came up against a peculiar issue today on a new server. My PHP installation is configured to store its sessions in memcached and I brought over some working configurations from another server. However, each time I accessed a page which tried to initiate a session, the page load would hang for about a minute and I'd find this in my apache error logs:

[Thu Sep 08 03:23:40 2011] [error] [client 11.22.33.44] PHP Warning:
Unknown: Failed to write session data (memcached). Please verify that
the current setting of session.save_path is correct (127.0.0.1:11211)
in Unknown on line 0

I ran through my usual list of checks:

• netstat showed memcached bound to the correct ports/interfaces
• memcached was running and I could reach it via telnet
• memcached-tool could connect and pull stats from memcached
• double-checked my php.ini
• tested memcached connectivity via a PHP and ruby script — they worked

Even after all that, I still couldn't figure out what was wrong. I ran strace on memcached while I ran a curl against the page which creates a session and I found something significant — memcached wasn't seeing any connections whatsoever at that time. A quick check of the lo interface with tcpdump showed the same result. Just before I threw a chair, I remembered one thing:

SELinux.

A quick check for AVC denials showed the problem:

# aureport --avc | tail -n 1
4021. 09/08/2011 03:23:38 httpd system_u:system_r:httpd_t:s0 42 tcp_socket name_connect system_u:object_r:memcache_port_t:s0 denied 31536

I'm far from being a guru on SELinux, so I leaned on audit2allow for help:

# grep memcache /var/log/audit/audit.log | audit2allow
 
#============= httpd_t ==============
#!!!! This avc can be allowed using one of the these booleans:
#     httpd_can_network_relay, httpd_can_network_memcache, httpd_can_network_connect
 
allow httpd_t memcache_port_t:tcp_socket name_connect;

The boolean we're looking for is httpd_can_network_memcache. Flipping the boolean can be done in a snap:

# setsebool -P httpd_can_network_memcache 1
# getsebool httpd_can_network_memcache
httpd_can_network_memcache --> on

After adjusting the boolean, apache was able to make connections to memcached without a hitch. My page which created sessions loaded quickly and I could see data being stored in memcached. If you want to check the status of all of the apache-related SELinux booleans, just use getsebool:

# getsebool -a | grep httpd | grep off$
allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
httpd_can_check_spam --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_read_user_content --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off

If you're interested in SELinux, a good way to get your feet wet is to head over to the CentOS Wiki and review their SELinux Howtos

Getting apache, PHP, and memcached working with SELinux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Звонит техподдержка, говорит что на одном из вебсерверов лежит апач. Ну дело не хитрое, рестарт и смотрим error_log. Тишина... всмысле вообще тишина ничего нет в логе. Ну делаем Loglevel debug и снова рестарт и снова в логе тишина и пара warn-ов не относящихся к делу.

Опытным путем выяснил что апач мрет на загрузке конфига с виртуалхостами, виртуалхостов на сервере чуть больше 3000, какой косячит? Как его найти если в логах ничего нет? strace ни на какие мысли не на талкивает.

Но специфика сервера такова что каждый вхост в отдельном конфиге, потом эти конфиги парсятся, правятся и сливаются в один файл. Вобщем переношу все конфиги вхостов в /root/tmp и по 100 штук начинаю возвращать на место и рестартить апач. Таким образом нахожу ОДНУ паршивую овцу. Лезу в конфиг и падаю со стула.

Для одного домена прописано 530 алиасов, у того же клиента смотрю другие домены, на втором прописано еще 370 алиасов. Судя по всему лимит у апача 512 алиасов, потом смерть.

И блять молчаливая смерть, ни строчки в логи!!

Вобщем так можно хостерам гадить, регаешься на самый дешевый тариф и набиваешь 600 алиасов. Всё апачу пездос.

Here is a one liner to check for IPs of bots that are misusing your site.

This will give you the top 10 IPs and URIs with a hit count.

Consider blocking those rogue IPs with a high hit count via iptables.

read more

Анализ DDoS можно производить конечно своими скриптами, парсить логи. Но лучше предоставить это апачевскому mod_evasive.

Ставим mod_evasive, в конфигурации пишем

DOSHashTableSize    3097
DOSPageCount        15
DOSSiteCount        15
DOSPageInterval     3
DOSSiteInterval     3
DOSBlockingPeriod   300
DOSSystemCommand    "/usr/bin/sudo /usr/bin/fwban %s"

• DOSPageInterval – интервал для хитов определенной страницы
• DOSSiteInterval – интервал для хитов определенного vhost
• DOSPageCount – после этого количества хитов по определенному URI в течении интервала DOSPageInterval, айпи будет забанен
• DOSSiteCount – после этого количества хитов по определенному vhost в течении интервала DOSSiteInterval, айпи будет забанен

Нам понадобиться скрипт для бана на уровне файрвола «/usr/bin/fwban» (вариант для Linux):

#!/bin/bash
if [ "x$1" = "x" ] ; then
    echo "USAGE: $0 IPADDR"
    exit
fi
/sbin/iptables -A BAN -s $1 -j DROP

Ему надо поставить права 755.

Так же нам понадобиться утилита sudo. Она стоит практически везде. В «visudo» закомментируем опцию:

#Defaults    requiretty

И добавим строку

apache  ALL = NOPASSWD: /usr/bin/fwban

где apache – юзер от которого работает апач.

Так же нам понадобиться цепочка BAN в iptables:

iptables -N BAN
iptables -I INPUT -j BAN

Сохраним правила файрвола

/etc/init.d/iptables save

Рестартанем апач. Теперь попробуйте уложить ваш сайт (только не со своего айпи!!!):

ab -n 1000  -c 20 http://yoursite.info/

В логах «жертвы» можно увидеть:

May  6 15:18:25 Server1 mod_evasive[26514]: Blacklisting address 1.2.3.4: possible DoS attack.

А в файрволе:

# iptables-save
---многа букав---
-A BAN -s 1.2.3.4 -j DROP
---многа букав---

Ура! No pasaran.

Да. И конечно, апач лучше бы прикрыть извне nginx’ом.

Да. И данный метод банит айпишнеги перманентно, пока не рестартанет сервер, или не будет сброшена цепочка BAN. Вот такой брутальный метод )

This works for both HTTP and HTTPS where any front end web server such as nginx which handles the actual request sets a header when request comes via HTTPS. In Apache configuration you then use mod_setenvif to set the HTTPS variable, which Django then picks up to use for redirection.

With front end nginx server which handles SSL, set header «X-Forwarded-Proto=https» via:

On Apache, add directive:

The HTTPS variable is picked up as being special by mod_wsgi and it will fix the wsgi.url_scheme in WSGI environment which Django then uses for redirection.

This way you don't need to customize Django stack.

read more

This is again short post for people lazy enough to not compile and always looking for some quick way to upgrade/install software. The machine is having CentOS 5.2 and httpd 2.2.8. We are looking to upgrade httpd to 2.2.17 to succeed in PCI compliance. While I assured that current Apache is having all security upgrades [...]

Иногда бывает нужно сделать так, чтобы апач и скрипты запускаемые кроном имели разные php.ini. К примеру, для того чтобы ограничить скрипты апача разными disable_functions.
Просто создаем копию php.ini в любом месте, допустим, это будет версия для Apache. Добавляем в нее:

Это серьезно усложнит жизнь злоумышленнику на пути повышения привилегий.
И прописываем путь к этому php.ini в httpd.conf:

Перезапускаем апач и простеньким скриптом проверяем что получилось: