My work at Rackspace has changed a bit in the last few weeks and I've shifted from managing a team of engineers to a full technical focus on . Although it was difficult to leave my management position, I'm happy to get back to my roots and dig into the technical stuff again.
One of the first things I wanted to tackle was understanding how a build request flows through Nova to a XenServer hypervisor. Following this process through the code is a bit tricky (I'm still learning python, so that could explain it). Here are the basic steps:
Client requests a build via the API.
The API runs some checks (quotas, auth, etc) and hands the build off to the scheduler.
The scheduler figures out where the instance should go.
The scheduler drops a message in queue specific to one compute node (where the instance will be built).
The API responds to the client and the client is now unblocked and free to do other things.
The compute node updates the database with the instance details and calls to the hypervisor to assemble block devices for the instance.
A message is dropped into the network node's queue (from the compute node) to begin assembling networks for the instance. The compute node blocks and waits while this step completes.
Once the networking details come back (via the queue), the compute node does the remaining adjustments on the hypervisor and starts up the actual instance.
When the instance starts successfully (or fails to do so), the database is updated and a message is dropped onto another message queue as a notification that the build is complete.
Click on the thumbnail on the right to see the flow chart I created to explain this process.
Please note: This information should be accurate to the Nova code as of November 1, 2011. There could be some refactoring of these build processes before is released.
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
Quite a few people who couldn't make it to this year asked me to write a post summarizing my takeaways from the event. I'm not generally one to back down from peer pressure, so read on if you're interested about the discussions at this year's Summit.
The feeling I had at last year's summit is that Xen was on the verge of losing traction in the market. Very few distributions still had Xen support going forward and much of the discussion was around the lack of dom0 support in upstream Linux kernels. Distribution vendors were hesitant to drag patches forward into modern kernels and this made it much more difficult to get Xen working for many people.
This year was quite different. The number of attendees was up, the , and there was an obvious buzz of energy in the room. As many of the presenters noted, this excitement stemmed from the . This inclusion is a huge win and it helps to drive Xen forward since the developers don't have to worry about dragging patches forward. They can focus on improving performance, adding features, and tightening security.
Many of the discussions this year focused on security and performance. Ian Pratt discussed Xen's ability to view memory pages of virtual machines via an API to detect malware running inside the instance. Memory pages could be identified and marked as not executable or applications could be triggered when a VM attempts to touch a particular memory page. Also, the whole VM could be frozen if needed.
There's also a big push to bring code out of the dom0 and push it into utility VM's. Driver domains could manage the network or I/O infrastructure and this would further reduce the amount of privileged code actively running in dom0. There is already very little code required for the Xen hypervisor itself (much much less than the Linux kernel — I'm looking at you, ) and this reduces the attack surface for potential compromises of the hypervisor. Some projects even aim to restart driver domains multiple times per minute to ensure that any malicious code injected into those virtual machines can't exist for long periods.
Pradeep Vincent from talked about how Amazon uses Xen and the pain points they have with its current architecture. Much of his discussion was around scaling problems (and we see many of the same issues at ). Higher performance could easily be gained by multi-threaded operations in dom0 when attaching block devices and creating virtual network interfaces. He also saw some areas for performance gains in the pvops I/O code.
Quite a few of the talks centered on the ARM architecture and what Xen is able to do on those systems after . HVM is on the way for ARM and it might even show up in Xen 4.2. Some demos of Xen on mobile phones from Samsung were amazing. They showed how an attacker could compromise the web browser on the phone with a keylogger, but that application was running in a VM. Once the user switched back to the phone's main menu, the keylogger couldn't access the keystrokes any longer. After that, a simple close of the browser killed the VM and destroyed the malicious code.
Xen 4.2 should be available in early 2012 and the feature list is staggering. Improvements to libxenlight, pvops performance (even in HVM), and guest memory sharing should be available with the new release. Nested virtualization (run a hypervisor inside a hypervisor) is also coming in Xen 4.2 and I'm sure Xzibit will be a huge fan. This should streamline hypervisor testing, allow for embedded hypervisor options and extend the capabilities of client hypervisors. Remus should be available in 4.2 as well, but it might be marked as experimental. OVMF will be added as a BIOS option for UEFI (along with the standard SeaBIOS) and this should allow for Mac OS X guests. UEFI allows Windows to boot faster since it switches to PV mode sooner and it allows for simpler platform certification for software vendors.
Mike McClurg's presentation on was pretty important to me since Rackspace is a big consumer of . If you're not familiar with XCP, it's basically open-source XenServer which runs on bleeding edge (and sometimes unstable) components. XCP 1.5 and XenServer 6 should be available in November with Xen 4.1 and Linux 2.6.32. GPU passthrough, up to 1TB RAM, and disaster recovery will be available. Another goal for the XCP team is to work closely with OpenStack via Project Olympus. Mike's vision is to have XCP become the configuration of choice for open source clouds. was also extremely interesting. It's essentially XCP's XenAPI stack running on Debian and Ubuntu. You'd be able to install either OS on a physical server and run XCP's services on it for a fully OSS hypervisor.
Konrad Wilk gave an update on Linux pvops and it appears there is a shift to get Xen working well on a desktop. This includes 3D graphics support, S3/hibernate capabilities and various bug fixes. There's also a push to get PV functionality into HVM and get HVM functionality into PV. Driver/device domains were discussed again in Patrick Kolp's talk and he had plenty of graphs showing performance changes when regularly restarting device domains. The performance dips were almost negligible with 10 second restarts and the security gains were significant.
There were several other great presentations on other topics like , , and (from the NSA!). If these types of things interests you, keep your eyes peeled for Xen Summit 2012 next year. The is well worth the trip.
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
If you haven't noticed already, was added in the . This means there's no longer a need to drag patches forward from old kernels and work from special branches and git repositories when building a kernel for .
Something else you might not have noticed is that the Fedora kernel team has into Fedora 15's update channels in disguise. Click that link, scroll down, and you'll see «Rebase to 3.0. Version reports as 2.6.40 for compatibility with older userspace.» Although I'm not a fan of calling something what it isn't (2.6.40 doesn't exist on kernel.org), I can understand some of the reasoning behind the choice.
This change makes the Xen installation on Fedora 15 pretty trivial. To get started, update your kernel to the latest if you're not already on Fedora's 2.6.40 kernels:
yum -y upgrade kernel
We need three more packages (quite a few dependencies will roll in with them):
yum -y install xen libvirt python-virtinst
The xen package reels in the hypervisor itself along with libraries and command line tools (like xl and xm). Libvirt gives us easy access to VM management with the virsh command and python-virtinst gives us the handy virt-install command to make OS installations easy.
Once those packages are installed, we need to make some adjustments in your grub configuration. Open /boot/grub/menu.lst in your text editor of choice and add something like this at the bottom:
Ensure that the root (hd0,1) is applicable to your system (adjust it if it isn't). Also, check the kernel version to ensure it matches your installed kernel and adjust the root= portion to match your root volume. Flip the default line to a value which will boot your new grub entry and ensure the timeout is set to a reasonable number if you need to temporarily switch back to your original grub entry at boot time. (Hey, we all make mistakes.)
I take one extra precaution and change the UPDATEDEFAULT=yes line to no in /etc/sysconfig/kernel. This ensures that future kernel updates don't trample the entry you've just made. Keep in mind that you'll need to manually update your grub configuration when you do kernel upgrades later.
Cross your fingers and reboot. If your system doesn't reboot properly, reboot it again and choose your old kernel from the grub menu. Double-check your configuration for fat-fingering and give it another try. If your system boots and pings but you have no output via a monitor, don't fret. There's a for the problem which in Linux 3.0. The impatient can snag a kernel source RPM, add the patch file, and (or you can from when I did it).
You should have a VM installation underway pretty quickly and it will be visible via port 5905 on the local host. Enjoy the power and freedom of your brand new .
is a post from: Major Hayden's blog.
Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.
Diagram: OpenVPN to Rackspace Cloud Servers and Slicehost
A recent inspired me to write a quick how-to for Fedora users on using OpenVPN to talk to instances privately in the Rackspace Cloud.
The diagram at the right gives an idea of what this guide will allow you to accomplish. Consider a situation where you want to talk to the MySQL installation on db1 directly without requiring extra ssh tunnels or MySQL over SSL via the public network. If you tunnel into one of your instances, you can utilize the private network to talk between your instances very easily.
There's one important thing to keep in mind here: even though you'll be utilizing the private network between your tunnel endpoint and your other instances, your traffic will still traverse the public network. That means that the instance with your tunnel endpoint will still get billed for the traffic flowing through your tunnel.
You'll only need the openvpn package on the server side:
yum -y install openvpn
Throw down this simple configuration file into /etc/openvpn/server.conf:
port 1194
proto tcp
dev tun
persist-key
persist-tun
server 10.66.66.0 255.255.255.0
ifconfig-pool-persist ipp.txt
#push "route 10.0.0.0 255.0.0.0"
push "route 10.176.0.0 255.248.0.0"
keepalive 10 120
ca /etc/openvpn/my_certificate_authority.pem
cert /home/major/vpn_server_cert.pem
key /home/major/vpn_server_key.pem
dh /etc/openvpn/easy-rsa/2.0/keys/dh1024.pem
status log/openvpn-status.log
verb 3
Here's a bit of explanation for some things you may want to configure:
push — These are the routes that will be sent over the VPN that are pushed to the clients. If you don't use any IP addresses in the 10.0.0.0/8 network block in your office, you can probably use the commented out line above. However, you may want to be more specific with the routes if you happen to use any 10.0.0.0/8 space in your office.
server — These are the IP addresses that the VPN server will assign and NAT out through the private interface. I've used a /24 above, but you may want to adjust the netmask if you have a lot of users making tunnels to your VPN endpoint.
ca, cert, key — You will need to create a certificate authority as well as a certificate/key pair for your VPN endpoint. I already use on my Mac to manage some other CA's and certificates, but you can use scripts if you wish. They are already included with the openvpn installation.
Build your Diffie-Hellman parameters file:
cd /etc/openvpn/easy-rsa/2.0/ && ./build-dh
Tell iptables that you want to NAT your VPN endpoint traffic out to all 10.x.x.x IP addresses on the private network:
The last step on the server side is to ensure that the kernel will forward packets from the VPN endpoint out through the private interface. Ensure that your /etc/sysctl.conf looks like this:
# Controls IP packet forwarding
net.ipv4.ip_forward = 1
Adjusting your sysctl.conf ensures that forwarding is enabled at boot time, but you'll need to enable it on your VPN endpoint right now:
echo 1 > /proc/sys/net/ipv4/ip_forward
Start the openvpn server:
/etc/init.d/openvpn start
If all is well, you should see openvpn listening on port 1194:
You'll need to configure a client to talk to your VPN now. This involves three steps: creating a new certificate/key pair for the client (same procedure as making your server certificates), signing the client's certificate with your CA certificate (same one that you used above to sign your server certificates), and then configuring your client application to access the VPN.
There are many openvpn clients out there to choose from.
If you're using a Linux desktop, you may want to consider using the . For Mac users, I'd highly recommend using ($9), but there's also (free).
Let's go ahead and get this out of the way: The following post contains only my personal opinions. These are not the opinions of my employer and should not be considered as such.
The term «cloud hosting» has become more popular over the past few years and it seems like everyone is talking about it. I'm often asked by customers and coworkers about what cloud hosting really is. Where does traditional dedicated hosting end and cloud begin? Do they overlap? Who needs cloud and who doesn't?
You can't talk about cloud hosting without defining it first. When I think of «cloud», these are the things that come to mind:
quickly add/remove resources with little or no lead time
hosting platforms that allow for quick provisioning of highly available systems
self-service adjustment of tangible and intangible resources that normally require human intervention
That list may seem a bit vague at first, but try to let it sink in just a bit. Hosting applications in a «cloud» shouldn't mean that you must have a virtual instance running on Xen, KVM or VMWare, and it shouldn't mean that you must have an account with Rackspace Cloud, Amazon EC2, or Microsoft Azure. It means that your hosting operations are highly automated and you can rapidly allocate and deallocate resources for the requirements of your current projects.
Consider this: a customer of a traditional dedicated hosting provider decides to take their applications and host them on one VPS at a leading commercial provider. That provider allows the customer to spin up new VM's in a matter of minutes and re-image the VM's whenever they like. Is that cloud hosting? I'd say yes — even if it's one single virtual instance. That customer has moved from a hosting system with manual interventions and extended lead times to a system where they have instant control over their resources.
It's not possible to talk about what cloud is without talking about what it isn't.
Cloud is not infinitely scalable. If any provider ever claims that their solution is «infinitely scalable», you should be skeptical. Regardless of the provider, everyone eventually runs out of datacenter space, servers, network bandwidth, or power. (If you know of a provider that is infinitely scalable, please let me know as I'd love to see their facilities and review their supply chain.)
Cloud isn't right for everybody. Some applications have demands that cloud hosting might not be able to meet (yet). If an application depends on proprietary hardware that is difficult to virtualize or rapidly allocate, cloud hosting is probably not the answer for that particular application.
Cloud doesn't mean VPS. VPS doesn't mean cloud. As I said before, having a virtual private server environment is not a pre-requisite for cloud hosting. Also, not all VPS solutions fit my definition of cloud as they don't allow for rapid deployments and resource adjustments.
It's important to remember that cloud hosting is a marketing term. As for the technology of cloud, it's what you make of it. You should be looking to reduce costs, solidify availability and increase performance every day. If the ideals of cloud hosting help you do that, it might be the right option for you.
Regardless of the type of hosting you're using — dedicated or cloud — it's important to take network interface security seriously. Most often, threats from the internet are the only ones mentioned. However, if you share a private network with other customers, you have just as much risk on that interface.
Many cloud providers allow you access to a private network environment where you can exchange data with other instances or other services offered by the provider. The convenience of this access comes with a price: other instances can access your instance on the private network just as easily as they could on the public interface.
Here are some security tips for your private interfaces:
Disable the private interface
This one is pretty simple. If you have only one instance or server, and you don't need to communicate privately with any other instances, just disable the interface. Remember to configure your networking scripts to leave the interface disabled after reboots.
Use packet filtering
The actual mechanism will vary based on your operating system, but filtering packets is the one of the simplest ways to secure your private interface. You can take some different approaches with them, but I find the easiest method is to allow access from your other instances and reject all other traffic.
For additional security, you can limit access based on ports as well as source IP addresses. This could prevent an attacker from having easy access to your other instances if they're able to break into one of them.
Configure your daemons to listen on the appropriate interfaces
If there are services that don't need to be listening on the private network, don't allow them to listen on your private interface. For example, MySQL might need to listen on the private interface so the web server can talk to it, but apache won't need to listen on the private interface. This reduces the profile of your instance on the private network and makes it a less likely target for attack.
Use hosts.allow and hosts.deny
Many new systems administrators forget about how handy tcpwrappers can be for limiting access. If your firewall is down in error, host.allow and hosts.deny could be an extra layer of protection. It's important to ensure that the daemons you are attempting to control are build with tcpwrappers support. Daemons like sshd support it, but apache and MySQL do not.
Encrypt all traffic on the private network
Just because it's called a «private» network doesn't mean that your traffic can traverse the network privately. You should always err on the side of caution and encrypt all traffic traversing the private network. You can use ssh tunnels, stunnel, or the built-in SSL features found in most daemons.
This also brings up an important point: you should know how your provider's private network works. Are there safeguards to prevent sniffing? Could someone else possibly ARP spoof your instance's private IP addresses? Is your private network's subnet shared among many customers?
With all of that said, it's also very important to have proper change control policies so that administrators working after you are fully aware of the security measures in place and why they are important. This will ensure that all of the administrators on your instances will understand the security of the system and they should be able to make sensible adjustments later for future functionality.
