Let's go ahead and get this out of the way: The following post contains only my personal opinions. These are not the opinions of my employer and should not be considered as such.
The term «cloud hosting» has become more popular over the past few years and it seems like everyone is talking about it. I'm often asked by customers and coworkers about what cloud hosting really is. Where does traditional dedicated hosting end and cloud begin? Do they overlap? Who needs cloud and who doesn't?
You can't talk about cloud hosting without defining it first. When I think of «cloud», these are the things that come to mind:
quickly add/remove resources with little or no lead time
hosting platforms that allow for quick provisioning of highly available systems
self-service adjustment of tangible and intangible resources that normally require human intervention
That list may seem a bit vague at first, but try to let it sink in just a bit. Hosting applications in a «cloud» shouldn't mean that you must have a virtual instance running on Xen, KVM or VMWare, and it shouldn't mean that you must have an account with Rackspace Cloud, Amazon EC2, or Microsoft Azure. It means that your hosting operations are highly automated and you can rapidly allocate and deallocate resources for the requirements of your current projects.
Consider this: a customer of a traditional dedicated hosting provider decides to take their applications and host them on one VPS at a leading commercial provider. That provider allows the customer to spin up new VM's in a matter of minutes and re-image the VM's whenever they like. Is that cloud hosting? I'd say yes — even if it's one single virtual instance. That customer has moved from a hosting system with manual interventions and extended lead times to a system where they have instant control over their resources.
It's not possible to talk about what cloud is without talking about what it isn't.
Cloud is not infinitely scalable. If any provider ever claims that their solution is «infinitely scalable», you should be skeptical. Regardless of the provider, everyone eventually runs out of datacenter space, servers, network bandwidth, or power. (If you know of a provider that is infinitely scalable, please let me know as I'd love to see their facilities and review their supply chain.)
Cloud isn't right for everybody. Some applications have demands that cloud hosting might not be able to meet (yet). If an application depends on proprietary hardware that is difficult to virtualize or rapidly allocate, cloud hosting is probably not the answer for that particular application.
Cloud doesn't mean VPS. VPS doesn't mean cloud. As I said before, having a virtual private server environment is not a pre-requisite for cloud hosting. Also, not all VPS solutions fit my definition of cloud as they don't allow for rapid deployments and resource adjustments.
It's important to remember that cloud hosting is a marketing term. As for the technology of cloud, it's what you make of it. You should be looking to reduce costs, solidify availability and increase performance every day. If the ideals of cloud hosting help you do that, it might be the right option for you.
Today, on my 28th birthday, I'm finally delivering on a promise to my readers which I made about two months ago. I've on how to host a web application redundantly in a cloud environment. While it's still a bit of a rough draft, it should be a good starting point for those who haven't worked in virtualized environments before. Also, it may show some of the more experienced systems administrators a new way to do things.
The guide:
As always, if you find anything in the guide that needs improvement, I'm all ears.
I've been getting requests for GlusterFS benchmarks from every direction lately and I've been a bit slow on getting them done. You may suspect that you know the , and you're probably correct.
Quite a few different sites argue that the default GlusterFS performance translator configuration from glusterfs-volgen doesn't allow for good performance. You can find other sites which say you should stick with the defaults that come from the script. I decided to run some simple tests to see which was true in my environment.
Here's the testbed:
GlusterFS 3.0.5 running on RHEL 5.4 Xen guests with ext3 filesystems
one GlusterFS client and two GlusterFS servers are running in separate Xen guests
cluster/replicate translator is being used to keep the servers in sync
the instances are served by a gigabit network
It's about time for some pretty graphs, isn't it?
The test run on the left used default stock and volume files as they come from glusterfs-volgen. The test run on the right used a (the server volume file was untouched). Between each test run, the GlusterFS mount was unmounted and remounted. I repeated this process four times (for a total of five runs) and averaged the data.
You'll have to forgive the color mismatches and the lack of labeling on the legend (that's KB/sec transferred) as I'm far from an Excel expert.
The graphs show that running without any translators at all will drastically hinder read caching in GlusterFS — exactly as I expected. Without any translators, the performance is very even across the board. Since my instances had 256MB of RAM each, their iocache translator was limited to about 51MB of cache. That's reflected in the graph on the left — look for the vertical red/blue divider between the 32MB and 64MB file sizes. I'll be playing around with that value soon to see how it can improve performance for large and small files.
Keep in mind that this test was very unscientific and your results may vary depending on your configuration. While I hope to have more detailed benchmarks soon, this should help some of the folks who have been asking for something basic and easy to understand.
As many of you might have noticed from my and my , I've been working with GlusterFS in production for my personal hosting needs for just over a month. I've also been learning quite a bit from some of the folks in the channel on . On a few occasions I've even been able to help out with some configuration problems from other users.
There has been quite a bit of interest in GlusterFS as of late and I've been inundated with questions from coworkers, other system administrators and developers. Most folks want to know about its reliability and performance in demanding production environments. I'll try to do my best to cover the big points in this post.
First off, here's now I'm using it in production: I have two web nodes that keep content in sync for various web sites. They each run a GlusterFS server instance and they also mount their GlusterFS share. I'm using the to keep both web nodes in sync with client side replication.
Here are my impressions after a month:
I/O speed is often tied heavily to network throughput This one may seem obvious, but it's not always true in all environments. If you deal with a lot of small files like I do, a 40mbit/sec link between the Xen guests is plenty. Adding extra throughput didn't add any performance to my servers. However, if you wrangle large files on your servers regularly, you may want to consider higher throughput links between your servers. I was able to push just under 900mbit/sec by using dd to create a large file within a GlusterFS mount.
Network and I/O latency are big factors for small file performance If you have a busy network and the latency creeps up from time to time, you'll find that your small file performance will drop significantly (especially with the replicate translator). Without getting too nerdy (you're welcome to read the ), replication is an intensive process. When a file is accessed, the client goes around to each server node to ensure that it not only has a copy of the file being read, but that it has the correct copy. If a server didn't save a copy of a file (due to disk failure or the server being offline when the file was written), it has to be synced across the network from one of the good nodes.
When you write files on replicated servers, the client has to roll through the same process first. Once that's done, it has to lock the file, write to the change log, then do the write operation, drop the change log entries, and then unlock the file. All of those operations must be done on all of the servers. High latency networks will wreak havoc on this process and cause it to take longer than it should.
It's quite obvious that if you have a fast, low-latency network between your servers, slow disks can still be a problem. If the client is waiting on the server nodes' disks to write data, the read and write performance will suffer. I've tested this in environments with fast networks and very busy RAID arrays. Even if the network was very underutilized, slow disks could cut performance drastically.
Monitoring GlusterFS isn't easy When the client has communication problems with the server nodes, some weird things can happen. I've seen situations where the client loses connections to the servers (see the next section on reliability) and the client mount simply hangs. In other situations, the client has been knocked offline entirely and the process is missing from the process tree by the time I logged in. Your monitoring will need to ensure that the mount is active and is responding in a timely fashion.
There's a which allows you to monitor GlusterFS mounts via nagios that Ian Rogers put together. Also, you can get some historical data with .
GlusterFS 3.x is pretty reliable When I first started working with GlusterFS, I was using a version from the 2.x tree. The Fedora package maintainer hadn't updated the package in quite some time, but I figured it should work well enough for my needs. I found that the small file performance was lacking and the nodes often had communication issues when many files were being accessed or written simultaneously. This improved when I built my own RPMs of 3.0.4 (and later 3.0.5) and began using those instead.
I did some failure testing by hard cycling the server and client nodes and found some interesting results. First off, abruptly pulling clients had no effects on the other clients or the server nodes. The connection eventually timed out and the servers logged the timeout as expected.
Abruptly pulling servers led to some mixed results. In the 2.x branch, I saw client hangs and timeouts when I abruptly removed a server. This appears to be mostly corrected in the 3.x branch. If you're using replicate, it's important to keep in mind that the first server volume listed in your client's volume file is the one that will be coordinating the file and directory locking. Should that one fall offline quickly, you'll see a hiccup in performance for a brief moment and the next server will be used for coordinating the locking. When your original server comes back up, the locking coordination will shift back.
Conclusion I'm really impressed with how much GlusterFS can do with the simplicity of how it operates. Sure, you can get better performance and more features (sometimes) from something like Lustre or GFS2, but the amount of work required to stand up that kind of cluster isn't trivial. GlusterFS really only requires that your kernel have FUSE support (it's been in mainline kernels since 2.6.14).
There are some things that GlusterFS really needs in order to succeed:
Documentation — The current documentation is often out of date and confusing. I've even found instances where the documentation contradicts itself. While there are some good technical documents about the design of some translators, they really ought to do some more work there.
Statistics gathering — It's very difficult to find out what GlusterFS is doing and where it can be optimized. Profiling your environment to find your bottlenecks is nearly impossible with the 2.x and 3.x branches. It doesn't make it easier when some of the performance translators actually decrease performance.
Community involvement — This ties back into the documentation part a little, but it would be nice to see more participation from Gluster employees on IRC and via the mailing lists. They're a little better with mailing list responses than other companies I've seen, but there is still room for improvement.
If you're considering GlusterFS for your servers but you still have more questions, feel free to leave a comment or find me on Freenode (I'm 'rackerhacker').
After I wrote a recent post on , I noticed that I forgot to mention comments for iptables rules. They can be extremely handy if you have some obscure rules for odd situations.
To make an iptables rule with a comment, simply add on the following arguments to the rule:
-m comment --comment "limit ssh access"
Depending on your distribution, you may need to load the ipt_comment or xt_comment modules into your running kernel first.
A full iptables rule to limit ssh access would look something like this:
iptables -A INPUT -j DROP -p tcp --dport 22 -m comment --comment "limit ssh access"
Typical configuration for a proxy-type load balancer
A typical load balancing configuration using hardware devices or software implementations will be organized such that they resemble the diagram at the right. I usually call this a proxy-type load balancing solution since the load balancer proxies your request to some other nodes. The standard order of operations looks like this:
client makes a request
load balancer receives the request
load balancer sends request to a web node
the web server sends content back to the load balancer
the load balancer responds to the client
If you're not familiar with load balancing, here's an analogy. Consider a fast food restaurant. When you walk up to the counter and place an order, you're asking the person at the counter (the load balancer) for a hamburger. The person at the counter is going to submit your order, and then a group of people (web nodes) are going to work on it. Once your hamburger (web request) is ready, your order will be given to the person at the counter and then back to you.
This style of organization can become a problem as your web nodes begin to scale. It requires you to ensure that your load balancers can keep up with the requests and sustain higher transfer rates that come from having more web nodes serving a greater number of requests. Imagine the fast food restaurant where you have one person taking the orders but you have 30 people working on the food. The person at the counter may be able to take orders very quickly, but they may not be able to keep up with the orders coming out of the kitchen.
LVS allows for application servers to respond to clients directly
This is where really shines. LVS operates a bit differently:
client makes a request
load balancer receives the request
load balancer sends request to a web node
the web server sends the response directly to the client
The key difference is that the load balancer sends the unaltered request to the web server and the web server responds directly to the client. Here's the fast food analogy again. If you ask the person at the counter (the load balancer) for a hamburger, that person is going to take your order and give it to the kitchen staff (the web nodes) to work on it. This time around, the person at the counter is going to advise the kitchen staff that the order needs to go directly to you once it's complete. When your hamburger is ready, a member of the kitchen staff will walk to the counter and give it directly to you.
In the fast food analogy, what are the benefits? As the number of orders and kitchen staff increases, the job of the person at the counter doesn't drastically increase in difficulty. While that person will have to handle more orders and keep tabs on which of the kitchen staff is working on the least amount of orders, they don't have to worry about returning food to customers. Also, the kitchen staff doesn't need to waste time handing orders to the person at the counter. Instead, they can pass these orders directly to the customer that ordered them.
In the world of servers, this is a large benefit. Since the web servers' responses no longer pass through the load balancer, they can spend more time on what they do best — balancing traffic. This allows for smaller, lower-powered load balancing servers from the beginning. It also allows for increases in web nodes without big changes for the load balancers.
There are three main implementations of LVS to consider:
LVS-DR: Direct Routing The load balancer receives the request and sends the packet directly to a waiting real server to process. LVS-DR has the best performance, but all of your servers must be on the same network subnet and they have to be able to share the same router (with no other routing devices in between them).
LVS-TUN: Tunneling This is very similar to the direct routing approach, but the packets are and sent directly to the real servers once the load balancer receives them. This removes the restriction that all of the devices must be on the same network. Thanks to encapsulation, you can use this method to load balance between multiple datacenters.
LVS-NAT: Network Address Translation Using NAT for LVS yields the least performance and scaling of all of the implementation options. In this configuration, the incoming requests are rewritten so that they will be transported correctly in a NAT environment. This puts a bigger burden on the load balancer as it must rewrite the requests quickly while still keeping up with how much work is being done by each web server.
Looking for a Linux Virtual Server HOWTO? Stay tuned. I'm preparing one for my next post.
High availability is certainly not a new concept, but if there's one thing that frustrates me with high availability VM setups, it's storage. If you don't mind going active-passive, you can set up , toss your favorite filesystem on it, and you're all set.
If you want to go active-active, or if you want multiple nodes active at the same time, you need to use a clustered filesystem like , or . These are certainly good options to consider but they're not trivial to implement. They usually rely on additional systems and scripts to provide reliable and capabilities.
What about the rest of us who want multiple active VM's with simple replicated storage that doesn't require any additional elaborate systems? This is where really shines. GlusterFS can ride on top of whichever filesystem you prefer, and that's a huge win for those who want a simple solution. However, that means that it has to use , and that will limit your performance.
Let's get this thing started!
Consider a situation where you want to run a WordPress blog on two VM's with load balancers out front. You'll probably want to use GlusterFS's replicated volume mode (RAID 1-ish) so that the same files are on both nodes all of the time. To get started, build two small Slicehost slices or Rackspace Cloud Servers. I'll be using Fedora 13 in this example, but the instructions for other distributions should be very similar.
First things first — be sure to set a new root password and update all of the packages on the system. This should go without saying, but it's important to remember. We can clear out the default iptables ruleset since we will make a customized set later:
# iptables -F
# /etc/init.d/iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]
GlusterFS communicates over the network, so we will want to ensure that traffic only moves over the private network between the instances. We will need to add the private IP's and a special hostname for each instance to /etc/hosts on both instances. I'll call mine gluster1 and gluster2:
10.xx.xx.xx gluster1
10.xx.xx.xx gluster2
You're now ready to install the required packages on both instances:
Make the directories for the GlusterFS volumes on each instance:
mkdir -p /export/store1
We're ready to make the configuration files for our storage volumes. Since we want the same files on each instance, we will use the --raid 1 option. This only needs to be run on the first node:
# glusterfs-volgen --name store1 --raid 1 gluster1:/export/store1 gluster2:/export/store1
Generating server volfiles.. for server 'gluster2'
Generating server volfiles.. for server 'gluster1'
Generating client volfiles.. for transport 'tcp'
Once that's done, you'll have four new files:
booster.fstab — you won't need this file
gluster1-store1-export.vol — server-side configuration file for the first instance
gluster2-store1-export.vol — server-side configuration file for the second instance
store1-tcp.vol — client side configuration file for GlusterFS clients
Copy the gluster1-store1-export.vol file to /etc/glusterfs/glusterfsd.vol on your first instance. Then, copy gluster2-store1-export.vol to /etc/glusterfs/glusterfsd.vol on your second instance. The store1-tcp.vol should be copied to /etc/glusterfs/glusterfs.vol on both instances.
At this point, you're ready to start the GlusterFS servers on each instance:
/etc/init.d/glusterfsd start
You can now mount the GlusterFS volume on both instances:
mkdir -p /mnt/glusterfs
glusterfs /mnt/glusterfs/
You should now be able to see the new GlusterFS volume in both instances:
# df -h /mnt/glusterfs
Filesystem Size Used Avail Use% Mounted on
/etc/glusterfs/glusterfs.vol
9.4G 831M 8.1G 10% /mnt/glusterfs
As a test, you can create a file on your first instance and verify that your second instance can read the data:
If you remove that file on your second instance, it should disappear from your first instance as well.
Obviously, this is a very simple and basic implementation of GlusterFS. You can increase performance by making dedicated VM's just for serving data and you can adjust the default performance options when you mount a GlusterFS volume. Limiting access to the GlusterFS servers is also a good idea.
If you want to read more, I'd recommend reading the and the .
Thank you for your e-mails! I'll be expanding on this post later with some sample benchmarks and additional tips/tricks, so please stay tuned.
I'll admit it right now: I love engaging customers and learning more about how what we do at Rackspace can help their business or ideas take flight. Talking with customers can be a little nerve-wracking at first since you're not always sure what their experience level is and which products they really need. However, you can get past that initial nervousness very quickly by getting an idea of what the customer needs and what they've tried already (that didn't work).
You may not have realized it, but I covered the most important part of selling a technical product in the first paragraph without even mentioning the word «sell». That was intentional. As a technical person, you have an innate ability to interact with customers without needing to actively sell them the product.
Whenever I meet a customer at a conference, trade show, or some other relatively informal event, I try to keep a few things in mind. I'll share them with you:
Learn why your customers are seeking out your product and what they really need It's pretty obvious that this step requires more listening than talking. While the customer is explaining what they need but haven't found, try to keep a running tally in your brain of what technologies are important to them so that you can rank your suggestions for them. Don't think about which product will work best for them yet — just keep keep their general requirements in mind.
This is also a good opportunity to relate to what they've told you. If there's a certain solution that ended up working really well or one that failed miserably, and you're familiar with one of those solutions, tell them briefly about your experiences. This will re-affirm how the customer feels about that solution and it also shows them that you've been in their shoes before. They'll also appreciate that you've been listening to their concerns and looking for ways to relate to their unique situation.
Make thoughtful production suggestions and discuss implementation Some folks might say this is where the selling starts, but if you're doing it correctly, you've been selling your product and your company the whole time. This is where things can get tricky. Most technical people I've met will try to avoid being pushy when suggesting a product for a customer to use, and that's a good idea.
You need to do three things: pick the right product (or group of products), explain what needs it meets, and briefly cover some example implementations. As a technical person, this is where you really shine. Interpreting the customer's needs and turning it into a mini technical sales pitch is a piece of cake when you know the product well and you've implemented it before.
It's great to give a customer multiple options, but it's a bad idea to overwhelm them. If you find that you're talking a bit too much, there's no harm in offering to talk about details later during a formal meeting. You can say things like these:
«this product will meet all your needs, but if you want to save a little money, you can use this other product like this.»
«if you combine these two products, you can meet these needs and save some time, but you can just use one and set it up like this...»
«then later on, if you need to expand, you can start using this product by...»
Think about the customer's future growth Even if you have products that meet your customer's needs, they're going to be concerned about what's going to happen down the road. What happens when they scale to a level that they can't even comprehend right now? I don't think any customer would expect you to cover all the bases, but try to think of some basic future-proofing for the customer. Even if it might involve a product that your company doesn't sell, just mention it.
Of course, there are some things that you shouldn't do:
Don't overpromise or push hard about a future product.
Don't feel obligated to know the answer to every question.
Don't use words like «infinite», «forever», or «perfect».
Don't talk about cost constantly.
Don't force a customer to choose a product, take product literature, or take your contact information.
Don't make assumptions about the customer's technical level, needs, or purchasing power.
Don't let it bother you if the customer isn't interested in your product — it's not personal.
And that's about it. If you follow those three tips and avoid the things you shouldn't do, you'll get the confidence you need to engage the customer and create the beginnings of a relationship with them.
Regardless of the type of hosting you're using — dedicated or cloud — it's important to take network interface security seriously. Most often, threats from the internet are the only ones mentioned. However, if you share a private network with other customers, you have just as much risk on that interface.
Many cloud providers allow you access to a private network environment where you can exchange data with other instances or other services offered by the provider. The convenience of this access comes with a price: other instances can access your instance on the private network just as easily as they could on the public interface.
Here are some security tips for your private interfaces:
Disable the private interface This one is pretty simple. If you have only one instance or server, and you don't need to communicate privately with any other instances, just disable the interface. Remember to configure your networking scripts to leave the interface disabled after reboots.
Use packet filtering The actual mechanism will vary based on your operating system, but filtering packets is the one of the simplest ways to secure your private interface. You can take some different approaches with them, but I find the easiest method is to allow access from your other instances and reject all other traffic.
For additional security, you can limit access based on ports as well as source IP addresses. This could prevent an attacker from having easy access to your other instances if they're able to break into one of them.
Configure your daemons to listen on the appropriate interfaces If there are services that don't need to be listening on the private network, don't allow them to listen on your private interface. For example, MySQL might need to listen on the private interface so the web server can talk to it, but apache won't need to listen on the private interface. This reduces the profile of your instance on the private network and makes it a less likely target for attack.
Use hosts.allow and hosts.deny Many new systems administrators forget about how handy tcpwrappers can be for limiting access. If your firewall is down in error, host.allow and hosts.deny could be an extra layer of protection. It's important to ensure that the daemons you are attempting to control are build with tcpwrappers support. Daemons like sshd support it, but apache and MySQL do not.
Encrypt all traffic on the private network Just because it's called a «private» network doesn't mean that your traffic can traverse the network privately. You should always err on the side of caution and encrypt all traffic traversing the private network. You can use ssh tunnels, stunnel, or the built-in SSL features found in most daemons.
This also brings up an important point: you should know how your provider's private network works. Are there safeguards to prevent sniffing? Could someone else possibly ARP spoof your instance's private IP addresses? Is your private network's subnet shared among many customers?
With all of that said, it's also very important to have proper change control policies so that administrators working after you are fully aware of the security measures in place and why they are important. This will ensure that all of the administrators on your instances will understand the security of the system and they should be able to make sensible adjustments later for future functionality.
