Let's go ahead and get this out of the way: The following post contains only my personal opinions. These are not the opinions of my employer and should not be considered as such.
The term «cloud hosting» has become more popular over the past few years and it seems like everyone is talking about it. I'm often asked by customers and coworkers about what cloud hosting really is. Where does traditional dedicated hosting end and cloud begin? Do they overlap? Who needs cloud and who doesn't?
You can't talk about cloud hosting without defining it first. When I think of «cloud», these are the things that come to mind:
quickly add/remove resources with little or no lead time
hosting platforms that allow for quick provisioning of highly available systems
self-service adjustment of tangible and intangible resources that normally require human intervention
That list may seem a bit vague at first, but try to let it sink in just a bit. Hosting applications in a «cloud» shouldn't mean that you must have a virtual instance running on Xen, KVM or VMWare, and it shouldn't mean that you must have an account with Rackspace Cloud, Amazon EC2, or Microsoft Azure. It means that your hosting operations are highly automated and you can rapidly allocate and deallocate resources for the requirements of your current projects.
Consider this: a customer of a traditional dedicated hosting provider decides to take their applications and host them on one VPS at a leading commercial provider. That provider allows the customer to spin up new VM's in a matter of minutes and re-image the VM's whenever they like. Is that cloud hosting? I'd say yes — even if it's one single virtual instance. That customer has moved from a hosting system with manual interventions and extended lead times to a system where they have instant control over their resources.
It's not possible to talk about what cloud is without talking about what it isn't.
Cloud is not infinitely scalable. If any provider ever claims that their solution is «infinitely scalable», you should be skeptical. Regardless of the provider, everyone eventually runs out of datacenter space, servers, network bandwidth, or power. (If you know of a provider that is infinitely scalable, please let me know as I'd love to see their facilities and review their supply chain.)
Cloud isn't right for everybody. Some applications have demands that cloud hosting might not be able to meet (yet). If an application depends on proprietary hardware that is difficult to virtualize or rapidly allocate, cloud hosting is probably not the answer for that particular application.
Cloud doesn't mean VPS. VPS doesn't mean cloud. As I said before, having a virtual private server environment is not a pre-requisite for cloud hosting. Also, not all VPS solutions fit my definition of cloud as they don't allow for rapid deployments and resource adjustments.
It's important to remember that cloud hosting is a marketing term. As for the technology of cloud, it's what you make of it. You should be looking to reduce costs, solidify availability and increase performance every day. If the ideals of cloud hosting help you do that, it might be the right option for you.
Today, on my 28th birthday, I'm finally delivering on a promise to my readers which I made about two months ago. I've on how to host a web application redundantly in a cloud environment. While it's still a bit of a rough draft, it should be a good starting point for those who haven't worked in virtualized environments before. Also, it may show some of the more experienced systems administrators a new way to do things.
The guide:
As always, if you find anything in the guide that needs improvement, I'm all ears.
I've been getting requests for GlusterFS benchmarks from every direction lately and I've been a bit slow on getting them done. You may suspect that you know the , and you're probably correct.
Quite a few different sites argue that the default GlusterFS performance translator configuration from glusterfs-volgen doesn't allow for good performance. You can find other sites which say you should stick with the defaults that come from the script. I decided to run some simple tests to see which was true in my environment.
Here's the testbed:
GlusterFS 3.0.5 running on RHEL 5.4 Xen guests with ext3 filesystems
one GlusterFS client and two GlusterFS servers are running in separate Xen guests
cluster/replicate translator is being used to keep the servers in sync
the instances are served by a gigabit network
It's about time for some pretty graphs, isn't it?
The test run on the left used default stock and volume files as they come from glusterfs-volgen. The test run on the right used a (the server volume file was untouched). Between each test run, the GlusterFS mount was unmounted and remounted. I repeated this process four times (for a total of five runs) and averaged the data.
You'll have to forgive the color mismatches and the lack of labeling on the legend (that's KB/sec transferred) as I'm far from an Excel expert.
The graphs show that running without any translators at all will drastically hinder read caching in GlusterFS — exactly as I expected. Without any translators, the performance is very even across the board. Since my instances had 256MB of RAM each, their iocache translator was limited to about 51MB of cache. That's reflected in the graph on the left — look for the vertical red/blue divider between the 32MB and 64MB file sizes. I'll be playing around with that value soon to see how it can improve performance for large and small files.
Keep in mind that this test was very unscientific and your results may vary depending on your configuration. While I hope to have more detailed benchmarks soon, this should help some of the folks who have been asking for something basic and easy to understand.
As many of you might have noticed from my and my , I've been working with GlusterFS in production for my personal hosting needs for just over a month. I've also been learning quite a bit from some of the folks in the channel on . On a few occasions I've even been able to help out with some configuration problems from other users.
There has been quite a bit of interest in GlusterFS as of late and I've been inundated with questions from coworkers, other system administrators and developers. Most folks want to know about its reliability and performance in demanding production environments. I'll try to do my best to cover the big points in this post.
First off, here's now I'm using it in production: I have two web nodes that keep content in sync for various web sites. They each run a GlusterFS server instance and they also mount their GlusterFS share. I'm using the to keep both web nodes in sync with client side replication.
Here are my impressions after a month:
I/O speed is often tied heavily to network throughput This one may seem obvious, but it's not always true in all environments. If you deal with a lot of small files like I do, a 40mbit/sec link between the Xen guests is plenty. Adding extra throughput didn't add any performance to my servers. However, if you wrangle large files on your servers regularly, you may want to consider higher throughput links between your servers. I was able to push just under 900mbit/sec by using dd to create a large file within a GlusterFS mount.
Network and I/O latency are big factors for small file performance If you have a busy network and the latency creeps up from time to time, you'll find that your small file performance will drop significantly (especially with the replicate translator). Without getting too nerdy (you're welcome to read the ), replication is an intensive process. When a file is accessed, the client goes around to each server node to ensure that it not only has a copy of the file being read, but that it has the correct copy. If a server didn't save a copy of a file (due to disk failure or the server being offline when the file was written), it has to be synced across the network from one of the good nodes.
When you write files on replicated servers, the client has to roll through the same process first. Once that's done, it has to lock the file, write to the change log, then do the write operation, drop the change log entries, and then unlock the file. All of those operations must be done on all of the servers. High latency networks will wreak havoc on this process and cause it to take longer than it should.
It's quite obvious that if you have a fast, low-latency network between your servers, slow disks can still be a problem. If the client is waiting on the server nodes' disks to write data, the read and write performance will suffer. I've tested this in environments with fast networks and very busy RAID arrays. Even if the network was very underutilized, slow disks could cut performance drastically.
Monitoring GlusterFS isn't easy When the client has communication problems with the server nodes, some weird things can happen. I've seen situations where the client loses connections to the servers (see the next section on reliability) and the client mount simply hangs. In other situations, the client has been knocked offline entirely and the process is missing from the process tree by the time I logged in. Your monitoring will need to ensure that the mount is active and is responding in a timely fashion.
There's a which allows you to monitor GlusterFS mounts via nagios that Ian Rogers put together. Also, you can get some historical data with .
GlusterFS 3.x is pretty reliable When I first started working with GlusterFS, I was using a version from the 2.x tree. The Fedora package maintainer hadn't updated the package in quite some time, but I figured it should work well enough for my needs. I found that the small file performance was lacking and the nodes often had communication issues when many files were being accessed or written simultaneously. This improved when I built my own RPMs of 3.0.4 (and later 3.0.5) and began using those instead.
I did some failure testing by hard cycling the server and client nodes and found some interesting results. First off, abruptly pulling clients had no effects on the other clients or the server nodes. The connection eventually timed out and the servers logged the timeout as expected.
Abruptly pulling servers led to some mixed results. In the 2.x branch, I saw client hangs and timeouts when I abruptly removed a server. This appears to be mostly corrected in the 3.x branch. If you're using replicate, it's important to keep in mind that the first server volume listed in your client's volume file is the one that will be coordinating the file and directory locking. Should that one fall offline quickly, you'll see a hiccup in performance for a brief moment and the next server will be used for coordinating the locking. When your original server comes back up, the locking coordination will shift back.
Conclusion I'm really impressed with how much GlusterFS can do with the simplicity of how it operates. Sure, you can get better performance and more features (sometimes) from something like Lustre or GFS2, but the amount of work required to stand up that kind of cluster isn't trivial. GlusterFS really only requires that your kernel have FUSE support (it's been in mainline kernels since 2.6.14).
There are some things that GlusterFS really needs in order to succeed:
Documentation — The current documentation is often out of date and confusing. I've even found instances where the documentation contradicts itself. While there are some good technical documents about the design of some translators, they really ought to do some more work there.
Statistics gathering — It's very difficult to find out what GlusterFS is doing and where it can be optimized. Profiling your environment to find your bottlenecks is nearly impossible with the 2.x and 3.x branches. It doesn't make it easier when some of the performance translators actually decrease performance.
Community involvement — This ties back into the documentation part a little, but it would be nice to see more participation from Gluster employees on IRC and via the mailing lists. They're a little better with mailing list responses than other companies I've seen, but there is still room for improvement.
If you're considering GlusterFS for your servers but you still have more questions, feel free to leave a comment or find me on Freenode (I'm 'rackerhacker').
After I wrote a recent post on , I noticed that I forgot to mention comments for iptables rules. They can be extremely handy if you have some obscure rules for odd situations.
To make an iptables rule with a comment, simply add on the following arguments to the rule:
-m comment --comment "limit ssh access"
Depending on your distribution, you may need to load the ipt_comment or xt_comment modules into your running kernel first.
A full iptables rule to limit ssh access would look something like this:
iptables -A INPUT -j DROP -p tcp --dport 22 -m comment --comment "limit ssh access"
Typical configuration for a proxy-type load balancer
A typical load balancing configuration using hardware devices or software implementations will be organized such that they resemble the diagram at the right. I usually call this a proxy-type load balancing solution since the load balancer proxies your request to some other nodes. The standard order of operations looks like this:
client makes a request
load balancer receives the request
load balancer sends request to a web node
the web server sends content back to the load balancer
the load balancer responds to the client
If you're not familiar with load balancing, here's an analogy. Consider a fast food restaurant. When you walk up to the counter and place an order, you're asking the person at the counter (the load balancer) for a hamburger. The person at the counter is going to submit your order, and then a group of people (web nodes) are going to work on it. Once your hamburger (web request) is ready, your order will be given to the person at the counter and then back to you.
This style of organization can become a problem as your web nodes begin to scale. It requires you to ensure that your load balancers can keep up with the requests and sustain higher transfer rates that come from having more web nodes serving a greater number of requests. Imagine the fast food restaurant where you have one person taking the orders but you have 30 people working on the food. The person at the counter may be able to take orders very quickly, but they may not be able to keep up with the orders coming out of the kitchen.
LVS allows for application servers to respond to clients directly
This is where really shines. LVS operates a bit differently:
client makes a request
load balancer receives the request
load balancer sends request to a web node
the web server sends the response directly to the client
The key difference is that the load balancer sends the unaltered request to the web server and the web server responds directly to the client. Here's the fast food analogy again. If you ask the person at the counter (the load balancer) for a hamburger, that person is going to take your order and give it to the kitchen staff (the web nodes) to work on it. This time around, the person at the counter is going to advise the kitchen staff that the order needs to go directly to you once it's complete. When your hamburger is ready, a member of the kitchen staff will walk to the counter and give it directly to you.
In the fast food analogy, what are the benefits? As the number of orders and kitchen staff increases, the job of the person at the counter doesn't drastically increase in difficulty. While that person will have to handle more orders and keep tabs on which of the kitchen staff is working on the least amount of orders, they don't have to worry about returning food to customers. Also, the kitchen staff doesn't need to waste time handing orders to the person at the counter. Instead, they can pass these orders directly to the customer that ordered them.
In the world of servers, this is a large benefit. Since the web servers' responses no longer pass through the load balancer, they can spend more time on what they do best — balancing traffic. This allows for smaller, lower-powered load balancing servers from the beginning. It also allows for increases in web nodes without big changes for the load balancers.
There are three main implementations of LVS to consider:
LVS-DR: Direct Routing The load balancer receives the request and sends the packet directly to a waiting real server to process. LVS-DR has the best performance, but all of your servers must be on the same network subnet and they have to be able to share the same router (with no other routing devices in between them).
LVS-TUN: Tunneling This is very similar to the direct routing approach, but the packets are and sent directly to the real servers once the load balancer receives them. This removes the restriction that all of the devices must be on the same network. Thanks to encapsulation, you can use this method to load balance between multiple datacenters.
LVS-NAT: Network Address Translation Using NAT for LVS yields the least performance and scaling of all of the implementation options. In this configuration, the incoming requests are rewritten so that they will be transported correctly in a NAT environment. This puts a bigger burden on the load balancer as it must rewrite the requests quickly while still keeping up with how much work is being done by each web server.
Looking for a Linux Virtual Server HOWTO? Stay tuned. I'm preparing one for my next post.
When Twitter was still in its early stages, you could track certain search terms in near-realtime . It was quite popular and its performance degraded over time as more users signed up and began posting updates. Eventually, Twitter killed the jabber bot altogether. .
Well, it hasn't returned, but you can build your own replacement with ruby, a jabber account, and a few gems. While it won't do everything that the original jabber bot did, you can still track tweets mentioning certain terms very quickly.
Here's how to get started:
First, install the tweetstream and xmpp4r-simple gems:
gem install tweetstream xmpp4r-simple
Next, you'll need a jabber account. You'll probably want to make one for the exclusive use of your jabber bot. I chose to make up a quick account at for mine.
The last step is to drop a copy of this script on your server:
the username and password for the twitter account that will monitor the stream
the search terms you want to track
the destination jabber account where the messages should be sent
Ensure that your jabber account has authorized the jabber bot's account so that you'll actually receive the messages. Also, Twitter is . It's a good idea to review their to ensure that you're not going to end up having your account temporarily or permanently blacklisted.
Once everything is ready to go, you can just run the script within GNU screen or via nohup. There's still a bit more error checking to do around jabber reconnections, but the script has run non-stop for well over two weeks at a time without a failure.
My curiosity is always piqued when I find new ways to manipulate command line output in simple ways. While working on a solution to parse /proc/mdstat output, I stumbled upon the utility.
The offers a very simple description of its features:
Write lines consisting of the sequentially corresponding lines from each FILE, separated by TABs, to standard output.
Here's an example of how it works. Let's say you want to parse some software raid output that looks like this:
It would be handy if we had both on one line as that would make it easier to parse with a script. Of course, you can do this with utilities like awk and tr, but paste makes it so much easier:
High availability is certainly not a new concept, but if there's one thing that frustrates me with high availability VM setups, it's storage. If you don't mind going active-passive, you can set up , toss your favorite filesystem on it, and you're all set.
If you want to go active-active, or if you want multiple nodes active at the same time, you need to use a clustered filesystem like , or . These are certainly good options to consider but they're not trivial to implement. They usually rely on additional systems and scripts to provide reliable and capabilities.
What about the rest of us who want multiple active VM's with simple replicated storage that doesn't require any additional elaborate systems? This is where really shines. GlusterFS can ride on top of whichever filesystem you prefer, and that's a huge win for those who want a simple solution. However, that means that it has to use , and that will limit your performance.
Let's get this thing started!
Consider a situation where you want to run a WordPress blog on two VM's with load balancers out front. You'll probably want to use GlusterFS's replicated volume mode (RAID 1-ish) so that the same files are on both nodes all of the time. To get started, build two small Slicehost slices or Rackspace Cloud Servers. I'll be using Fedora 13 in this example, but the instructions for other distributions should be very similar.
First things first — be sure to set a new root password and update all of the packages on the system. This should go without saying, but it's important to remember. We can clear out the default iptables ruleset since we will make a customized set later:
# iptables -F
# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]
GlusterFS communicates over the network, so we will want to ensure that traffic only moves over the private network between the instances. We will need to add the private IP's and a special hostname for each instance to /etc/hosts on both instances. I'll call mine gluster1 and gluster2:
10.xx.xx.xx gluster1
10.xx.xx.xx gluster2
You're now ready to install the required packages on both instances:
Make the directories for the GlusterFS volumes on each instance:
mkdir -p /export/store1
We're ready to make the configuration files for our storage volumes. Since we want the same files on each instance, we will use the --raid 1 option. This only needs to be run on the first node:
# glusterfs-volgen --name store1 --raid 1 gluster1:/export/store1 gluster2:/export/store1
Generating server volfiles.. for server 'gluster2'
Generating server volfiles.. for server 'gluster1'
Generating client volfiles.. for transport 'tcp'
Once that's done, you'll have four new files:
booster.fstab — you won't need this file
gluster1-store1-export.vol — server-side configuration file for the first instance
gluster2-store1-export.vol — server-side configuration file for the second instance
store1-tcp.vol — client side configuration file for GlusterFS clients
Copy the gluster1-store1-export.vol file to /etc/glusterfs/glusterfsd.vol on your first instance. Then, copy gluster2-store1-export.vol to /etc/glusterfs/glusterfsd.vol on your second instance. The store1-tcp.vol should be copied to /etc/glusterfs/glusterfs.vol on both instances.
At this point, you're ready to start the GlusterFS servers on each instance:
/etc/init.d/glusterfsd start
You can now mount the GlusterFS volume on both instances:
mkdir -p /mnt/glusterfs
glusterfs /mnt/glusterfs/
You should now be able to see the new GlusterFS volume in both instances:
# df -h /mnt/glusterfs
Filesystem Size Used Avail Use% Mounted on
/etc/glusterfs/glusterfs.vol
9.4G 831M 8.1G 10% /mnt/glusterfs
As a test, you can create a file on your first instance and verify that your second instance can read the data:
If you remove that file on your second instance, it should disappear from your first instance as well.
Obviously, this is a very simple and basic implementation of GlusterFS. You can increase performance by making dedicated VM's just for serving data and you can adjust the default performance options when you mount a GlusterFS volume. Limiting access to the GlusterFS servers is also a good idea.
If you want to read more, I'd recommend reading the and the .
Thank you for your e-mails! I'll be expanding on this post later with some sample benchmarks and additional tips/tricks, so please stay tuned.
I'll admit it right now: I love engaging customers and learning more about how what we do at Rackspace can help their business or ideas take flight. Talking with customers can be a little nerve-wracking at first since you're not always sure what their experience level is and which products they really need. However, you can get past that initial nervousness very quickly by getting an idea of what the customer needs and what they've tried already (that didn't work).
You may not have realized it, but I covered the most important part of selling a technical product in the first paragraph without even mentioning the word «sell». That was intentional. As a technical person, you have an innate ability to interact with customers without needing to actively sell them the product.
Whenever I meet a customer at a conference, trade show, or some other relatively informal event, I try to keep a few things in mind. I'll share them with you:
Learn why your customers are seeking out your product and what they really need It's pretty obvious that this step requires more listening than talking. While the customer is explaining what they need but haven't found, try to keep a running tally in your brain of what technologies are important to them so that you can rank your suggestions for them. Don't think about which product will work best for them yet — just keep keep their general requirements in mind.
This is also a good opportunity to relate to what they've told you. If there's a certain solution that ended up working really well or one that failed miserably, and you're familiar with one of those solutions, tell them briefly about your experiences. This will re-affirm how the customer feels about that solution and it also shows them that you've been in their shoes before. They'll also appreciate that you've been listening to their concerns and looking for ways to relate to their unique situation.
Make thoughtful production suggestions and discuss implementation Some folks might say this is where the selling starts, but if you're doing it correctly, you've been selling your product and your company the whole time. This is where things can get tricky. Most technical people I've met will try to avoid being pushy when suggesting a product for a customer to use, and that's a good idea.
You need to do three things: pick the right product (or group of products), explain what needs it meets, and briefly cover some example implementations. As a technical person, this is where you really shine. Interpreting the customer's needs and turning it into a mini technical sales pitch is a piece of cake when you know the product well and you've implemented it before.
It's great to give a customer multiple options, but it's a bad idea to overwhelm them. If you find that you're talking a bit too much, there's no harm in offering to talk about details later during a formal meeting. You can say things like these:
«this product will meet all your needs, but if you want to save a little money, you can use this other product like this.»
«if you combine these two products, you can meet these needs and save some time, but you can just use one and set it up like this...»
«then later on, if you need to expand, you can start using this product by...»
Think about the customer's future growth Even if you have products that meet your customer's needs, they're going to be concerned about what's going to happen down the road. What happens when they scale to a level that they can't even comprehend right now? I don't think any customer would expect you to cover all the bases, but try to think of some basic future-proofing for the customer. Even if it might involve a product that your company doesn't sell, just mention it.
Of course, there are some things that you shouldn't do:
Don't overpromise or push hard about a future product.
Don't feel obligated to know the answer to every question.
Don't use words like «infinite», «forever», or «perfect».
Don't talk about cost constantly.
Don't force a customer to choose a product, take product literature, or take your contact information.
Don't make assumptions about the customer's technical level, needs, or purchasing power.
Don't let it bother you if the customer isn't interested in your product — it's not personal.
And that's about it. If you follow those three tips and avoid the things you shouldn't do, you'll get the confidence you need to engage the customer and create the beginnings of a relationship with them.
