The Virtual Data Center

Once my brain starts spinning around one particular topic, it basically stays there until I’ve reached some sort of mental closure. Now that closure may be achieved when I’ve reached a personal conclusion, or it may come when I throw my arms up and say “I’m out!” Either way, I need to keep processing something until I’ve reached one of those points. This week, it’s the overlay between VMs, VMDK/VHDs, and OVF, which I started a few days ago with this post. So here I am again, and now I’m wondering if there’s even a point to OVF.

As reported at Server Virtualization, the DMTF is saying that OVF is still a few months away from a standard. Now a few months may not seem like a long time, but there are going to be some big movements between now and then, depending on which projects release on time and which are delayed, most importantly we should see Hyper-V moving out of beta. Chris Wolf has some interesting comments on that post and to be honest…I just don’t get all the fuss. Mounting VMs so any hypervisor can run an application? Telling the hypervisor what the packaged VM OS needs in order to optimize the running environment? It just seems like too many steps to get to the endgame. Here are two examples where I think OVF is just going to get in the way:

• Converting VM Disk Images: Chris states that even with OVF (right now it’s just a packaging framework standard, not a runtime standard) an interim conversion step will most likely be required. So when I grab a pre-packaged VM appliance from VMware wrapped in OVF and decide I want to run that on Hyper-V, I’m going to have to extract it, do a full conversion (which amounts to running P2V, or V2V in this case), and then re-wrap it before I drop it on Hyper-V. Hypervisors are platforms, and every hypervisor is going to run VMs in a different manner. Running 2008 in Hyper-V probably won’t take as many hypervisor resources as running it on VMware simply because 2008 shares kernel code with Hyper-V. So my app on 2008 will require X resources for Hyper-V but Y resources for VMware. Then what’s the point in packaging that data with the app? Is OVF going to have an XML switch element that contains running information for every possible hypervisor scenario? If I’m that concerned with app performance, I’m going to build the VM and app natively and not trust two translation layers (the original hypervisor the VM was built for and the OVF management metadata to allocate resources for me). To me, this is pushing OS virtualization further away from production environments.
• Lose the OS: OVF and virtual appliances deal with full-blown VMs; the OS, the disk image, and the running hypervisor. But we’re making such strides towards true application virtualization these days, I don’t see the need to focus on a solution that’s only concerned with bloated OS and disk images, pieces of the virtualization puzzle that only exist to run applications. I’d much rather see work being done on something like APS (Application Packaging Standard). Unlike VMs and VMDKs/VHDs, applications truly are portable. I’m looking forward to the day when we don’t need a full-blown OS in the data center, where we run apps directly on a hypervisor, where a packaging solution like APS can really be valuable. But even until then, something like APS has more value today because it’s “future proofing” our solutions for tomorrow. With VMs, both the OS and hypervisor have to become hardware resource managers. With true application virtualization, you only need the app hypervisor to manage your resources.

So why OVF? Why not let the DC admins worry about the hypervisor and OS installs? These are platform decisions, just like choosing HP vs. Dell. You don’t see Microsoft offering a pre-built 2003 image installed on a Dell with a conversion utility to run it on HP hardware (more on that in a few days as I start to drift into the problems with P2V…stay tuned) because that wouldn’t make any sense. OVF is the exact same thing: it’s a system to create a full-blown OS image and move it around the heterogeneous data center. But why? Every OS install is different, and it will continue to be that way until we get rid of the OS, even with major band-aids like OVF. Focus on the application and why you’re virtualizing in the first place. Right now, OVF appears to be an extra step we don’t need.

I know I’ve picked on Cisco’s Data Center blog a few times here, but they make themselves such an easy target, how can I let it slide? Case in point, this post from a few weeks ago called “The Dreaded V Word.” This posts starts on a good note: Doug jumps right into the hype of the “V Word,” although I think it surpassed SOA sometime last year both on the CIO hype scale and with companies claiming to have a buzzword of the year solution. This is one of the reasons I love answering the “Isn’t Service Virtualization just SOA?” question. “[Buzzwords] are colliding!! George is getting very upset!!”

But ironically enough, Doug actually makes the virtualization buzzword factor exponentially worse. Here’s how he defines virtualization:

“Virtualization as a technology rooted in the data center requiring network, storage and server to work together and thus drives IT collaboration. It allows the business to extend the lifecycle of capital assets they’ve already invested in and then reduce the operational expenses for remedial tasks (e.g. administrative change control, server batch moves, etc.) which allows them to free up more resources to focus on business critical applications and strategic new market entrances and such.”

Huh? Rooted in the Data Center? Drives IT collaboration? Extend capital assets? Reduce operations expenses for remedial tasks? Wow. Virtualization does all that? If I had a sales guy from a company come into my IT department and give me that answer when I asked him why I need to start looking at virtualization in my DC, I’d toss him out on his ear. That doesn’t tell me anything about what virtualization is, the problem statement, or the business benefit. Talk about using a lot of buzzwords. The term only becomes “dreaded” when you define it like that.

Wait, I just got it: now I know what Doug is trying to say:

• I call up my network guy (IT collaboration)
• Tell him to cancel the order for more Cisco switches (Extend Capital Assets)
• I’ve decided to consolidate in the DC (Free up resources)
• And move all my L2-4 switching over to all those awesome Application Delivery Controllers I just bought (Reduce OpEx for remedial tasks, ie switching)

Seriously, I couldn’t agree more that we’re still dealing with the virtualization buzzword, but to address the issue from a company like Cisco, who obviously has vested interest and virtualization technologies in the data center, is really a bad idea. And then to throw in Green IT and “Data Center 3.0″ all in the same post…a term you know I can’t stand. Did no one at Cisco cleanse this post before it went out or pass it through the Buzzword BS Meter first?

And while we’re at it, have you seen one of Cisco’s other blogs, Virtual Worlds, or basically their Second Life Marketing Blog? If I was new to data center virtualization and I wanted to get Cisco’s take, from their blogs I would think that Cisco is one big publicity company that’s more concerned with marketing names, buzzwords, and playing virtual games than the infrastructure of my Data Center. I know that’s not the case, and I know they have some deep virtualization technologies, but that’s the face their presenting through these blogs. It’s one thing to spout poetic on a personal blog; it’s something completely different when your spouting via a domain named blogs.cisco.com. I hope someone in the Technical Marketing team over there is reading this and their own blogs.

I know, it’s been quiet around here lately. I’ve been heads down in research and haven’t had a lot of time to digest new ideas and pick up old ones (or respond to Hoff ). But the Symantec+Veritas+Xen announcement today gave me good reason to poke my head up, log in, and revisit an idea I’ve been working on for a while.

When I’m not noodling virtualization and data centers, I’m a semi-professional photographer; most of my evening/weekend free time in 2008 has been spent on building a solid digital workflow from shooting to selling. One of the technology choices I’ve implemented in the middle of my workflow is converting from my camera’s proprietary RAW format to the Adobe’s open Digital Negative file format, DNG. I made this decision because I don’t want to be stuck fighting with specific RAW format support down the road, and I can edit and process files natively in DNG using Adobe tools, which I use already. So you could say I’ve “Future Proofed” my workflow for tomorrow, even if I change cameras or processing software.

So the above started me thinking a few months ago about virtual machine filesystems and what’s going on under the hood. The whole model today seems silly to me: I have a VM guest that has filesystem, say NTFS; that filesystem is packaged in a proprietary flat-file format for the virtual hypervisor platform, VMDK in VMware’s case, and that flat file is stored on top of another filesystem (VMFS, again for VMware), which is vaguely connected to the host OS filesystem (let’s say ext3 for ESX), and then layered on top of yet another file management tool with iSCSI, only to finally be stored on a real disk on a SAN. So my ‘index.html’ file hosted on my guest IIS VM has to go through approximately 6 virtual<->physical layers before it’s physically stored on a device that can manage that block data, such as VMware’s DRS. That seems excessive and very inefficient.

So that brought up two questions:

1. Why can’t we have a solution like DNG for VM filesystems that will allow me to take that flat-file and manage it as part of my virtual infrastructure on any platform? Granted we do have the OVF, but this is mostly a transport and packaging solution; it’s not a running solution. And yes, I know that disk formats are part of each hypervisor secret sauce, but that’s exactly what I’m suggesting: Let each vendor continue to refine their secret sauce (just like Nikon and Sony will continue to refine their particular flavors of RAW), but let me store and run that secret sauce in an open utility so I can simply click to push a VMDK from VMware to Hyper-V.
2. Beyond the above, do we even need that extra secret sauce filesystem layer at all? Why can’t ESX write directly to a block device in my SAN over iSCSI without storing my guest filesystem in a flat package that’s stored on VMware’s proprietary VMFS file system, only to be pushed out over an iSCSI network? If we’re going through so much trouble to virtualize the OS anyway, why can’t we simply write a translator that takes the guest block read/write request and map that to a physical block on our remote SAN/NAS disk? Basically, let’s virtualize the guest filesystem. Think about the I/O we could save…may make those VMware storage benchmarks near moot. Which leads me to…

The Symantec announcement. If it’s true, it’s exactly what we need in the VM storage space and a no-brainer. Anything that removes middleman components while also adding manageability is a great thing. We remove moving parts, which in itself can remove complexity, and then obfuscate the management (or probably integrate it an existing management platform)…we move <this much> closer to a functional VDC. And since it’s Xen based and is purported to work with Hyper-V, this could also be a driver in customers choosing one hypervisor platform over another. If it delivers and specifically doesn’t work with VMware ESX, VMFS, Virtual Center, etc, then it could be end up being a platform driver for Xen and Hyper-V. We’ll have to wait until the end of the year to see if this solution delivers as promised.

I’ve read two posts in the past few days that spin around the idea of how the network factors into a complete security solution. The first from Hoff goes after the concept of moving security management responsibility to the network; the second from Richard over at TaoSecurity covers terminating SSL at the perimeter and moving your trust network closer to the edge. Being a guy who started on the network and moved up the stack, both of these posts gave me heartburn for different reasons.

First, Hoff: No one..er, no sane one, anyway…is suggesting that all networking management tasks be relegated to the network, anymore than anyone is suggesting that virtual security solutions (such as firewall VMs) are going to be the be-all, end-all security touchdown for both virtual and physical platforms in the data center. Hoff knows better than anyone that there is no smoking gun, and no one is suggesting that In The Year 2000 the network will own all of your security. I love Hoff’s “All My Life’s A Circle” security model, but it falls victim to the same thing that he’s railing against in the first place: There is no one security solution for everything in the data center at any given time. These solutions, regardless of where and when they fall on the cyclical time scale, should work together to provide one unified security solution, with the appropriate emphasis for any one solution being placed on the appropriate technology. We may fluctuate that emphasis, but we’ll always look at the entire solution. SSL is an application security solution, yet it is an integral part of a complete network security solution and the network can play a major and critical role in managing secure application transport and policy enforcement. Which leads me to…

Second, Tao: The interesting thing to me with this post is the assumptions that are made. You don’t have to terminate SSL only at the perimeter, nor do you have to do it passively. You can terminate SSL anywhere in your network, be it at the edge in a border device, within your DMZ or at the DMZ/private edge, or even completely within the private network. It all depends on what you’re cracking SSL to look for. If you’re looking inside an SSL VPN connection coming into your network before it hits the firewall, yes, you want to look at it as soon as possible. But you can still examine the traffic in your existing “trust zone” by moving and sandboxing that connection into a secure location, terminating SSL, examining, and if clear, re-SSL’ing and passing it back into the untrusted network. There are all kinds of great examples on how to terminate and sandbox encrypted traffic for firewall inspection, even at near-line speeds with modern SSL termination devices. If you want to crack SSL for your apps, do it completely within your DMZ or trusted private network where your apps live, far from the edge. And to Ivan’s question (which I personally thing was self motivated by the company he works for), it doesn’t have to be passive. For complete trust and control, actively terminate at the device with different keys than you use on the backend. Let the user know you’re doing this, no need to be coy about it. Actively terminate SSL to inspect it, don’t simply bridge it. You end up being your MITM.

So long story short, the network is an extremely powerful tool in your complete data center security batbelt. It’s not the only tool, but it’s also not a tool that should be taken for granted. In other words, don’t commodotize your network (or anything in the DC for that matter) or the security resources you can use on your network. Make the network work for and with your security tools. Packets and data are just electricity; you can do anything you want with them, including using your network to help secure them. Take off the silo blinders and look at what you can do at every step in your DC with networking and security. You’ll find out you can do some really cool stuff…