I sometimes enjoy living on the edge occasionally and that sometimes means I keep up with OpenStack changes commit by commit. If you're in the same boat as I am, you may save some time by using my repository of bleeding-edge Python packages from the OpenStack projects:
Python packages are updated moments after the commit is merged into the repositories under .
Although the packages will contain the latest code available, rest assured that the code has passed an initial code review (by humans), unit tests, and varying levels of functional or integrated testing. There may still be a bug or two cropping up after that, so be aware of that as you utilize these packages.
I was surprised to see coverage about on last Sunday and I was curious to know what effect the story would have on my site's overall traffic. wrote a great summary of what the site offers and how people can use it in their daily work. It's pretty obvious that icanhazip.com really only serves a niche group of internet users, but even I was surprised at the level of interest.
icanhazip.com traffic data — March 2011
The graph on the right shows some recent traffic data from March 2011. The Lifehacker story was published around 7AM on March 27th in Australia, so I first started seeing a spike on the 26th (my server's time zone is UTC-5). The yellow bar is a count of the unique visits while the other bars count page views, hits and total bandwidth.
The count of unique visitors certainly increased (by about 10-11x), but the overall hits didn't increase by much. I'd imagine that most visitors accessed the site, noticed that it displayed their public IP, and then they went on their way. As I've said before, this site is easy to re-create and will really only serve a niche segment of internet users.
On most days, I'll receive a very high number of hits from a relatively small number of unique IP addresses. There are quite a few people who check their public-facing IP address every second, but it seems like the majority stick to a more reasonable interval of 5-30 minutes. I've yet to find the value in checking my public IP address once per second, but there are obviously some folks out there who find it valuable (or they aren't good at implementing sleeps in their scripts).
Here's a bit of trivia about the site for those who are interested:
Almost 40% of the traffic to the site is from Eastern European and Asian countries
The average user on the site generates about 45 hits per day
Linux users make up 91% of the traffic on the site (based on user agent strings)
Over 88% of the hits to the site are requests made with curl or wget
Most traffic is received between 4-5PM CDT
Almost 98% of the visitors who reach the site do so via a direct link without a referrer
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
If you offer a web service that users query via scripts or other applications, you'll probably find that some people will begin to abuse the service. My site is no exception.
While many of the users have reasonable usage patterns, there are some users that query the site more than once per second from the same IP address. If you haven't used the site before, all it does is return your public IP address in plain text. Unless your IP changes rapidly, you may not need to query the site more than a few times an hour.
I added the following to my icanhazip.com virtual host definition to get the message across to those users that abuse the service:
ErrorDocument403"No can haz IP. Stop abusing this service. \
Contact major at mhtx dot net for details."RewriteEngineOnRewriteCond %{REMOTE_ADDR} ^12.23.34.45$ [OR]
RewriteCond %{REMOTE_ADDR} ^98.87.76.65$
RewriteRule .* nocanhaz [F]
The users that are caught on the business end of these 403 responses will see something like this:
$ curl -i icanhazip.com
HTTP/1.1 403 Forbidden
Date: Wed, 17 Nov 2010 13:42:55 GMT
Server: Apache
Content-Length: 84
Connection: close
Content-Type: text/html; charset=iso-8859-1
No can haz IP. Stop abusing this service. Contact major at mhtx dot net for details.
As my uptime reports have shown, and as some of you have reported, my blog's load time has increased steadily over the past few weeks. It turns out that one of my VM's was on a physical machine that had some trouble and I was reaching a point where GlusterFS's replicate functionality couldn't meet my performance needs.
Instead of using as I had before in my , I decided to use in dual-primary mode with as the clustering filesystem on top of it. The performance is quite good so far:
Pingdom Response Time Graph for rackerhacker.com
I switched over the DNS late last night and the response time has fallen from the two to three second range (during times of low load) to right around one second per request. In addition to the reduced load times, I can support higher concurrency without significant performance degradation.
Don't worry — I'll make a detailed post on this topic later along with a guide on how to set it up yourself.
Today, on my 28th birthday, I'm finally delivering on a promise to my readers which I made about two months ago. I've on how to host a web application redundantly in a cloud environment. While it's still a bit of a rough draft, it should be a good starting point for those who haven't worked in virtualized environments before. Also, it may show some of the more experienced systems administrators a new way to do things.
The guide:
As always, if you find anything in the guide that needs improvement, I'm all ears.
As many of you might have noticed from my and my , I've been working with GlusterFS in production for my personal hosting needs for just over a month. I've also been learning quite a bit from some of the folks in the channel on . On a few occasions I've even been able to help out with some configuration problems from other users.
There has been quite a bit of interest in GlusterFS as of late and I've been inundated with questions from coworkers, other system administrators and developers. Most folks want to know about its reliability and performance in demanding production environments. I'll try to do my best to cover the big points in this post.
First off, here's now I'm using it in production: I have two web nodes that keep content in sync for various web sites. They each run a GlusterFS server instance and they also mount their GlusterFS share. I'm using the to keep both web nodes in sync with client side replication.
Here are my impressions after a month:
I/O speed is often tied heavily to network throughput
This one may seem obvious, but it's not always true in all environments. If you deal with a lot of small files like I do, a 40mbit/sec link between the Xen guests is plenty. Adding extra throughput didn't add any performance to my servers. However, if you wrangle large files on your servers regularly, you may want to consider higher throughput links between your servers. I was able to push just under 900mbit/sec by using dd to create a large file within a GlusterFS mount.
Network and I/O latency are big factors for small file performance
If you have a busy network and the latency creeps up from time to time, you'll find that your small file performance will drop significantly (especially with the replicate translator). Without getting too nerdy (you're welcome to read the ), replication is an intensive process. When a file is accessed, the client goes around to each server node to ensure that it not only has a copy of the file being read, but that it has the correct copy. If a server didn't save a copy of a file (due to disk failure or the server being offline when the file was written), it has to be synced across the network from one of the good nodes.
When you write files on replicated servers, the client has to roll through the same process first. Once that's done, it has to lock the file, write to the change log, then do the write operation, drop the change log entries, and then unlock the file. All of those operations must be done on all of the servers. High latency networks will wreak havoc on this process and cause it to take longer than it should.
It's quite obvious that if you have a fast, low-latency network between your servers, slow disks can still be a problem. If the client is waiting on the server nodes' disks to write data, the read and write performance will suffer. I've tested this in environments with fast networks and very busy RAID arrays. Even if the network was very underutilized, slow disks could cut performance drastically.
Monitoring GlusterFS isn't easy
When the client has communication problems with the server nodes, some weird things can happen. I've seen situations where the client loses connections to the servers (see the next section on reliability) and the client mount simply hangs. In other situations, the client has been knocked offline entirely and the process is missing from the process tree by the time I logged in. Your monitoring will need to ensure that the mount is active and is responding in a timely fashion.
There's a which allows you to monitor GlusterFS mounts via nagios that Ian Rogers put together. Also, you can get some historical data with .
GlusterFS 3.x is pretty reliable
When I first started working with GlusterFS, I was using a version from the 2.x tree. The Fedora package maintainer hadn't updated the package in quite some time, but I figured it should work well enough for my needs. I found that the small file performance was lacking and the nodes often had communication issues when many files were being accessed or written simultaneously. This improved when I built my own RPMs of 3.0.4 (and later 3.0.5) and began using those instead.
I did some failure testing by hard cycling the server and client nodes and found some interesting results. First off, abruptly pulling clients had no effects on the other clients or the server nodes. The connection eventually timed out and the servers logged the timeout as expected.
Abruptly pulling servers led to some mixed results. In the 2.x branch, I saw client hangs and timeouts when I abruptly removed a server. This appears to be mostly corrected in the 3.x branch. If you're using replicate, it's important to keep in mind that the first server volume listed in your client's volume file is the one that will be coordinating the file and directory locking. Should that one fall offline quickly, you'll see a hiccup in performance for a brief moment and the next server will be used for coordinating the locking. When your original server comes back up, the locking coordination will shift back.
Conclusion
I'm really impressed with how much GlusterFS can do with the simplicity of how it operates. Sure, you can get better performance and more features (sometimes) from something like Lustre or GFS2, but the amount of work required to stand up that kind of cluster isn't trivial. GlusterFS really only requires that your kernel have FUSE support (it's been in mainline kernels since 2.6.14).
There are some things that GlusterFS really needs in order to succeed:
Documentation — The current documentation is often out of date and confusing. I've even found instances where the documentation contradicts itself. While there are some good technical documents about the design of some translators, they really ought to do some more work there.
Statistics gathering — It's very difficult to find out what GlusterFS is doing and where it can be optimized. Profiling your environment to find your bottlenecks is nearly impossible with the 2.x and 3.x branches. It doesn't make it easier when some of the performance translators actually decrease performance.
Community involvement — This ties back into the documentation part a little, but it would be nice to see more participation from Gluster employees on IRC and via the mailing lists. They're a little better with mailing list responses than other companies I've seen, but there is still room for improvement.
If you're considering GlusterFS for your servers but you still have more questions, feel free to leave a comment or find me on Freenode (I'm 'rackerhacker').
Typical configuration for a proxy-type load balancer
A typical load balancing configuration using hardware devices or software implementations will be organized such that they resemble the diagram at the right. I usually call this a proxy-type load balancing solution since the load balancer proxies your request to some other nodes. The standard order of operations looks like this:
client makes a request
load balancer receives the request
load balancer sends request to a web node
the web server sends content back to the load balancer
the load balancer responds to the client
If you're not familiar with load balancing, here's an analogy. Consider a fast food restaurant. When you walk up to the counter and place an order, you're asking the person at the counter (the load balancer) for a hamburger. The person at the counter is going to submit your order, and then a group of people (web nodes) are going to work on it. Once your hamburger (web request) is ready, your order will be given to the person at the counter and then back to you.
This style of organization can become a problem as your web nodes begin to scale. It requires you to ensure that your load balancers can keep up with the requests and sustain higher transfer rates that come from having more web nodes serving a greater number of requests. Imagine the fast food restaurant where you have one person taking the orders but you have 30 people working on the food. The person at the counter may be able to take orders very quickly, but they may not be able to keep up with the orders coming out of the kitchen.
LVS allows for application servers to respond to clients directly
This is where really shines. LVS operates a bit differently:
client makes a request
load balancer receives the request
load balancer sends request to a web node
the web server sends the response directly to the client
The key difference is that the load balancer sends the unaltered request to the web server and the web server responds directly to the client. Here's the fast food analogy again. If you ask the person at the counter (the load balancer) for a hamburger, that person is going to take your order and give it to the kitchen staff (the web nodes) to work on it. This time around, the person at the counter is going to advise the kitchen staff that the order needs to go directly to you once it's complete. When your hamburger is ready, a member of the kitchen staff will walk to the counter and give it directly to you.
In the fast food analogy, what are the benefits? As the number of orders and kitchen staff increases, the job of the person at the counter doesn't drastically increase in difficulty. While that person will have to handle more orders and keep tabs on which of the kitchen staff is working on the least amount of orders, they don't have to worry about returning food to customers. Also, the kitchen staff doesn't need to waste time handing orders to the person at the counter. Instead, they can pass these orders directly to the customer that ordered them.
In the world of servers, this is a large benefit. Since the web servers' responses no longer pass through the load balancer, they can spend more time on what they do best — balancing traffic. This allows for smaller, lower-powered load balancing servers from the beginning. It also allows for increases in web nodes without big changes for the load balancers.
There are three main implementations of LVS to consider:
LVS-DR: Direct Routing
The load balancer receives the request and sends the packet directly to a waiting real server to process. LVS-DR has the best performance, but all of your servers must be on the same network subnet and they have to be able to share the same router (with no other routing devices in between them).
LVS-TUN: Tunneling
This is very similar to the direct routing approach, but the packets are and sent directly to the real servers once the load balancer receives them. This removes the restriction that all of the devices must be on the same network. Thanks to encapsulation, you can use this method to load balance between multiple datacenters.
LVS-NAT: Network Address Translation
Using NAT for LVS yields the least performance and scaling of all of the implementation options. In this configuration, the incoming requests are rewritten so that they will be transported correctly in a NAT environment. This puts a bigger burden on the load balancer as it must rewrite the requests quickly while still keeping up with how much work is being done by each web server.
Looking for a Linux Virtual Server HOWTO? Stay tuned. I'm preparing one for my next post.
