Today, on my 28th birthday, I'm finally delivering on a promise to my readers which I made about two months ago. I've on how to host a web application redundantly in a cloud environment. While it's still a bit of a rough draft, it should be a good starting point for those who haven't worked in virtualized environments before. Also, it may show some of the more experienced systems administrators a new way to do things.
The guide:
As always, if you find anything in the guide that needs improvement, I'm all ears.
When Twitter was still in its early stages, you could track certain search terms in near-realtime . It was quite popular and its performance degraded over time as more users signed up and began posting updates. Eventually, Twitter killed the jabber bot altogether. .
Well, it hasn't returned, but you can build your own replacement with ruby, a jabber account, and a few gems. While it won't do everything that the original jabber bot did, you can still track tweets mentioning certain terms very quickly.
Here's how to get started:
First, install the tweetstream and xmpp4r-simple gems:
gem install tweetstream xmpp4r-simple
Next, you'll need a jabber account. You'll probably want to make one for the exclusive use of your jabber bot. I chose to make up a quick account at for mine.
The last step is to drop a copy of this script on your server:
the username and password for the twitter account that will monitor the stream
the search terms you want to track
the destination jabber account where the messages should be sent
Ensure that your jabber account has authorized the jabber bot's account so that you'll actually receive the messages. Also, Twitter is . It's a good idea to review their to ensure that you're not going to end up having your account temporarily or permanently blacklisted.
Once everything is ready to go, you can just run the script within GNU screen or via nohup. There's still a bit more error checking to do around jabber reconnections, but the script has run non-stop for well over two weeks at a time without a failure.
My curiosity is always piqued when I find new ways to manipulate command line output in simple ways. While working on a solution to parse /proc/mdstat output, I stumbled upon the utility.
The offers a very simple description of its features:
Write lines consisting of the sequentially corresponding lines from each FILE, separated by TABs, to standard output.
Here's an example of how it works. Let's say you want to parse some software raid output that looks like this:
It would be handy if we had both on one line as that would make it easier to parse with a script. Of course, you can do this with utilities like awk and tr, but paste makes it so much easier:
High availability is certainly not a new concept, but if there's one thing that frustrates me with high availability VM setups, it's storage. If you don't mind going active-passive, you can set up , toss your favorite filesystem on it, and you're all set.
If you want to go active-active, or if you want multiple nodes active at the same time, you need to use a clustered filesystem like , or . These are certainly good options to consider but they're not trivial to implement. They usually rely on additional systems and scripts to provide reliable and capabilities.
What about the rest of us who want multiple active VM's with simple replicated storage that doesn't require any additional elaborate systems? This is where really shines. GlusterFS can ride on top of whichever filesystem you prefer, and that's a huge win for those who want a simple solution. However, that means that it has to use , and that will limit your performance.
Let's get this thing started!
Consider a situation where you want to run a WordPress blog on two VM's with load balancers out front. You'll probably want to use GlusterFS's replicated volume mode (RAID 1-ish) so that the same files are on both nodes all of the time. To get started, build two small Slicehost slices or Rackspace Cloud Servers. I'll be using Fedora 13 in this example, but the instructions for other distributions should be very similar.
First things first — be sure to set a new root password and update all of the packages on the system. This should go without saying, but it's important to remember. We can clear out the default iptables ruleset since we will make a customized set later:
# iptables -F
# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]
GlusterFS communicates over the network, so we will want to ensure that traffic only moves over the private network between the instances. We will need to add the private IP's and a special hostname for each instance to /etc/hosts on both instances. I'll call mine gluster1 and gluster2:
10.xx.xx.xx gluster1
10.xx.xx.xx gluster2
You're now ready to install the required packages on both instances:
Make the directories for the GlusterFS volumes on each instance:
mkdir -p /export/store1
We're ready to make the configuration files for our storage volumes. Since we want the same files on each instance, we will use the --raid 1 option. This only needs to be run on the first node:
# glusterfs-volgen --name store1 --raid 1 gluster1:/export/store1 gluster2:/export/store1
Generating server volfiles.. for server 'gluster2'
Generating server volfiles.. for server 'gluster1'
Generating client volfiles.. for transport 'tcp'
Once that's done, you'll have four new files:
booster.fstab — you won't need this file
gluster1-store1-export.vol — server-side configuration file for the first instance
gluster2-store1-export.vol — server-side configuration file for the second instance
store1-tcp.vol — client side configuration file for GlusterFS clients
Copy the gluster1-store1-export.vol file to /etc/glusterfs/glusterfsd.vol on your first instance. Then, copy gluster2-store1-export.vol to /etc/glusterfs/glusterfsd.vol on your second instance. The store1-tcp.vol should be copied to /etc/glusterfs/glusterfs.vol on both instances.
At this point, you're ready to start the GlusterFS servers on each instance:
/etc/init.d/glusterfsd start
You can now mount the GlusterFS volume on both instances:
mkdir -p /mnt/glusterfs
glusterfs /mnt/glusterfs/
You should now be able to see the new GlusterFS volume in both instances:
# df -h /mnt/glusterfs
Filesystem Size Used Avail Use% Mounted on
/etc/glusterfs/glusterfs.vol
9.4G 831M 8.1G 10% /mnt/glusterfs
As a test, you can create a file on your first instance and verify that your second instance can read the data:
If you remove that file on your second instance, it should disappear from your first instance as well.
Obviously, this is a very simple and basic implementation of GlusterFS. You can increase performance by making dedicated VM's just for serving data and you can adjust the default performance options when you mount a GlusterFS volume. Limiting access to the GlusterFS servers is also a good idea.
If you want to read more, I'd recommend reading the and the .
Thank you for your e-mails! I'll be expanding on this post later with some sample benchmarks and additional tips/tricks, so please stay tuned.
It certainly shouldn't be difficult, but I always have a tough time with . Twitter is on . I have some automated Twitter bots that need an upgrade, so I've been working on a quick solution to generate tokens for my scripts.
I formulated a pretty simple script using that will get it done manually for any scripts you have that read from or update Twitter:
#!/usr/bin/rubyrequire'rubygems'require'twitter'# These credentials are specific to your *application* and not your *user*# Get these credentials from Twitter directly: http://twitter.com/apps
application_token = '[this should be the shorter one]'
application_secret = '[this should be the longer one]'
oauth = Twitter::OAuth.new(application_token,application_secret)
request_token = oauth.request_token.token
request_secret = oauth.request_token.secretputs"Request token => #{request_token}"puts"Request secret => #{request_secret}"puts"Authentication URL => #{oauth.request_token.authorize_url}"print"Provide the PIN that Twitter gave you here: "
pin = gets.chomp
oauth.authorize_from_request(request_token,request_secret,pin)
access_token = oauth.access_token.token
access_secret = oauth.access_token.secretputs"Access token => #{oauth.access_token.token}"puts"Access secret => #{oauth.access_token.secret}"
oauth.authorize_from_access(access_token, access_secret)
twitter = Twitter::Base.new(oauth)puts twitter.friends_timeline(:count=>1)
When you run the script, it will give you a request token, request secret and a URL to visit. When you access the URL, you'll be given a PIN. Type the PIN into the prompt and you'll get your access token and secret. This is what you can use to continue authenticating with Twitter, so be sure to save the access token and secret.
From then on, you should be able to login with a script like this:
#!/usr/bin/rubyrequire'rubygems'require'twitter'
application_token = '[this should be the shorter one]'
application_secret = '[this should be the longer one]'
oauth = Twitter::OAuth.new(application_token,application_secret)
oauth.authorize_from_access(access_token, access_secret)
twitter = Twitter::Base.new(oauth)puts twitter.friends_timeline(:count=>1)
Fedora 13 has quite a few changes related to , and one of the biggest ones is how terminals are configured. Most distributions tuck the tty configuration away in /etc/inittab, /etc/event.d/ or /etc/init/. If you want to adjust the number of tty's in Fedora 13, you'll need to look in /etc/sysconfig/init:
# color => new RH6.0 bootup# verbose => old-style bootup# anything else => new style bootup without ANSI colors or positioningBOOTUP=color
# column to start "[ OK ]" label in RES_COL=60# terminal sequence to move to that column. You could change this# to something like "tput hpa ${RES_COL}" if your terminal supports itMOVE_TO_COL="echo -en \\033[${RES_COL}G"# terminal sequence to set color to a 'success' color (currently: green)SETCOLOR_SUCCESS="echo -en \\033[0;32m"# terminal sequence to set color to a 'failure' color (currently: red)SETCOLOR_FAILURE="echo -en \\033[0;31m"# terminal sequence to set color to a 'warning' color (currently: yellow)SETCOLOR_WARNING="echo -en \\033[0;33m"# terminal sequence to reset to the default color.SETCOLOR_NORMAL="echo -en \\033[0;39m"# default kernel loglevel on boot (syslog will reset this)LOGLEVEL=3# Set to anything other than 'no' to allow hotkey interactive startup...PROMPT=yes# Set to 'yes' to allow probing for devices with swap signaturesAUTOSWAP=no
# What ttys should gettys be started on?ACTIVE_CONSOLES=/dev/tty[1-6]
The very last line controls the number of tty's that are kept alive on your system. If you need more tty's, simply increase the 6 to a higher number. If you only want one terminal (which is usually what I want in Xen), just make this adjustment:
# What ttys should gettys be started on?ACTIVE_CONSOLES=/dev/tty1
A normal telinit q doesn't seem to adjust the terminals on the fly as it did before upstart was involved. I'm not sure if this is a bug or an intended feature. Either way, a reboot solves the problem and you should see the changes afterwards.
When you need to measure network throughput and capacity, I haven't found a simpler solution than . There isn't about the operation of iperf — it's a very simple application.
In short, iperf can be installed on two machines within your network. You'll run one as a server, and one as a client. On the server side, simply run:
iperf -s
On the client side, run:
iperf -c [server_ip]
The client side will try to shove TCP packets through the network interface as quickly as possible for a period of 10 seconds by default. Once that's complete, you'll see a report on the server and client that will look like this:
$ iperf -c 192.168.10.10
------------------------------------------------------------
Client connecting to 192.168.10.10, TCP port 5001
TCP window size: 65.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.10.30 port 53345 connected with 192.168.10.10 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 37.9 MBytes 31.8 Mbits/sec
The previous test was run over an 802.11n network between a wired and wireless device. The for an 802.11n network is about 40Mbit/s, so it's obvious that my home network could use an adjustment.
You can also run bidirectional tests from the client either at the same time (-d flag) or one after the other (-r flag). The server side will keep running until you stop it, so you can leave it running and run tests from multiple locations over time. You can daemonize the server end if that makes things easier.
Regardless of the type of hosting you're using — dedicated or cloud — it's important to take network interface security seriously. Most often, threats from the internet are the only ones mentioned. However, if you share a private network with other customers, you have just as much risk on that interface.
Many cloud providers allow you access to a private network environment where you can exchange data with other instances or other services offered by the provider. The convenience of this access comes with a price: other instances can access your instance on the private network just as easily as they could on the public interface.
Here are some security tips for your private interfaces:
Disable the private interface This one is pretty simple. If you have only one instance or server, and you don't need to communicate privately with any other instances, just disable the interface. Remember to configure your networking scripts to leave the interface disabled after reboots.
Use packet filtering The actual mechanism will vary based on your operating system, but filtering packets is the one of the simplest ways to secure your private interface. You can take some different approaches with them, but I find the easiest method is to allow access from your other instances and reject all other traffic.
For additional security, you can limit access based on ports as well as source IP addresses. This could prevent an attacker from having easy access to your other instances if they're able to break into one of them.
Configure your daemons to listen on the appropriate interfaces If there are services that don't need to be listening on the private network, don't allow them to listen on your private interface. For example, MySQL might need to listen on the private interface so the web server can talk to it, but apache won't need to listen on the private interface. This reduces the profile of your instance on the private network and makes it a less likely target for attack.
Use hosts.allow and hosts.deny Many new systems administrators forget about how handy tcpwrappers can be for limiting access. If your firewall is down in error, host.allow and hosts.deny could be an extra layer of protection. It's important to ensure that the daemons you are attempting to control are build with tcpwrappers support. Daemons like sshd support it, but apache and MySQL do not.
Encrypt all traffic on the private network Just because it's called a «private» network doesn't mean that your traffic can traverse the network privately. You should always err on the side of caution and encrypt all traffic traversing the private network. You can use ssh tunnels, stunnel, or the built-in SSL features found in most daemons.
This also brings up an important point: you should know how your provider's private network works. Are there safeguards to prevent sniffing? Could someone else possibly ARP spoof your instance's private IP addresses? Is your private network's subnet shared among many customers?
With all of that said, it's also very important to have proper change control policies so that administrators working after you are fully aware of the security measures in place and why they are important. This will ensure that all of the administrators on your instances will understand the security of the system and they should be able to make sensible adjustments later for future functionality.
This problem has cropped up for me a few times, but I've always forgotten to make a post about it. If you're working with a large InnoDB table and you're updating, inserting, or deleting a large volume of rows, you may stumble upon this error:
ERROR 1206 (HY000): The total number of locks exceeds the lock table size
InnoDB stores its lock tables in the main buffer pool. This means that the number of locks you can have at the same time is limited by the innodb_buffer_pool_size variable that was set when MySQL was started. By default, MySQL leaves this at 8MB, which is pretty useless if you're doing anything with InnoDB on your server.
Luckily, the fix for this issue is very easy: adjust innodb_buffer_pool_size to a more reasonable value. However, that fix does require a restart of the MySQL daemon. There's simply no way to adjust this variable on the fly (with the current stable MySQL versions as of this post's writing).
Before you adjust the variable, make sure that your server can handle the additional memory usage. The innodb_buffer_pool_size variable is a server wide variable, not a per-thread variable, so it's shared between all of the connections to the MySQL server (like the query cache). If you set it to something like 1GB, MySQL won't use all of that up front. As MySQL finds more things to put in the buffer, the memory usage will gradually increase until it reaches 1GB. At that point, the oldest and least used data begins to get pruned when new data needs to be present.
So, you need a workaround without a MySQL restart?
If you're in a pinch, and you need a workaround, break up your statements into chunks. If you need to delete a million rows, try deleting 5-10% of those rows per transaction. This may allow you to sneak under the lock table size limitations and clear out some data without restarting MySQL.
To learn more about InnoDB's parameters, visit the .
About a year ago, I was introduced to the joys of using and to access irc servers. Before that time, I'd usually used graphical clients like , and I always enjoyed getting notifications when someone mentioned a word or string that I set up as a trigger.
Once I started using irssi in screen, I found that the visual bell in screen didn't get my attention quickly. Luckily, someone in the channel let me know about screen's audible bell. You can flip between the visual and audible bell with CTRL-A and then CTRL-G. If you keep repeating that key combination, you'll switch back and forth between the two (with a status update at the bottom left).
You can also set up your visual bell configuration in your .screenrc via some configuration parameters:
