The Virtual Data Center

Over the past week or so, my mind grapes have been wrapped around the idea that maybe the hypervisor isn’t really getting the credit it deserves, specifically for security and management. Everyone keeps saying that the hypervisor is becoming commoditized and that innovation and value-add on the virtual platform level will go away.

Part of me wants to agree. As far as the OS is concerned, and the IT folks managing the OS, the hypervisor is like the CPU was 8-10 years ago. People were actively choosing AMD Athlons or Intel P3s, and you chose one or the other for a particular reason. Windows ran on both, Linux ran on both, so eventually the CPU decision became speed vs. cost, not features. Same is true with hypervisors today; there are multiple choices, and those choices each have pros and cons. And like CPUs today, ultimately these different hypervisors will live together in perfect harmony and the IT department will move on to focusing on applications, not what those apps are running on. The decision will become about other factors, like manageability, instead of features. It’s a nice idea, to a certain degree. Should I care if I’m using a hypervisor? This is the path I’ve been treading lately with research into non-OS hypervisor implementations, such as kernel virtualization (Virtual Environments) and the ideas around application hypervisors.

But lately, most of me is standing firm in disagreeing with the idea that we either don’t need a hypervisor, or that the hypervisor is becoming a faceless resource. Hypervisors absolutely provide benefit for management and security both. They provide:

• Obfuscation: Sandboxing, if you will, that really exemplifies the value and benefit of running a virtual machine infrastructure. You need to give your guests the freedom they think they need, all the while exerting 100% control of what they can do with and to the underlying host hardware. The HAL architecture (as in NT, not as in 2001) had it right. There’s too much risk in plugging one running kernel directly into another. Hypervisors solve this by forcefully creating the obfuscation layer that keeps guests and hosts separate.

• Insight: This is what will give complete manageability to your guests and your hosts, and tie that back into your existing data center management platform. You can’t trust a guest that says it’s using 95% of non-paged memory, b/c you know the hypervisor has only given that guest 10% of the total available memory (and may, in fact, be paging that memory without the guest knowing it). You need to rely on the hypervisor for true system insight and manageability. This is a core tenant of virtualization and, to me, the hypervisor is the tool designed to do this.

So that’s where my “on the bus daydreaming about virtualization” time has been going lately. Tomorrow, “Part 2 (The Benefit),” where I’ll cover why everyone should be embracing the almost-forgotten, soon-to-possibly-be-extinct hypervisor, and extending the cool things we can do with the hypervisor that I first started at hinting at here. (HINT: I want my hypervisor to be Mother ;).

I was reading up on Montego Networks’ planned virtual switch security product yesterday, and it definitely appears to be interesting technology under the hood. From what I can deduce, which could be completely off-base given that the product doesn’t release until April, is that they’re going after the “Virtual NAC” market, plus some. Or probably more correctly stated (since NAC has such baggage these days), they’re building policies and applying logic to data as it flows from one virtual switch to another. My guess is that since they’re going to support VMware initially, they’re taking advantage of the fact that everything is running in software (and running on Linux with a Linux network stack) and adding all these small networking tools directly to the software switch. I have no idea if this is technically what they’re doing, but when I went through the above list I was thinking “They patched in iptables for the firewall, SNORT for the content inspection, LVS for the load balancing, tc for traffic shaping…” Kind of reminds of a much more robust implementation of my busybox home WAP+Router.

There’s a promo video they have posted on YouTube that goes into more of their architecture. In the most basic scenario, it looks like they’re adding a vswitch1 and vswitch2 to ESX and VLAN’ing (or possibly routing, I couldn’t tell) all data that comes into vswitch0 is forced through vswitch1, which is then VLAN’d to an application virtual image that applies the policies, then pushed back out vswich2 (or maybe back to vswitch0). There are so many unknowns from the time I’ve had to look into this, but it does look interesting, and more importantly got me thinking about other possibilities:

First off (and with no disrespect to Montego and other companies that do this), why do we need 3rd party companies to do this? Let’s look away from the model where one company provides the same solution for multiple platforms and instead look at building this into the virtual framework. For example, let’s look at Microsoft and Hyper-V. MS, IMO, is in a perfect position to build this type of policy-based secure switching from the ground up today. Sure, VMware and Citrix/Xen could do it via acquisitions, but MS has the tools and the expertise to do it today. They have products that cover all aspects of what’s required to do this from the security side: group policy on the client; NAP on the transport; Forefront on the server; AD on the backend. And of course they have Hyper-V and software switching. So if 2008 is going to run right on top of Hyper-V already, and will include the 4 products I’ve listed above, why do this from day one? Why wait for another company to build this with an API? And it wouldn’t really be anything different: it would be like treating the Hyper-V software switch like the local stack on the OS. For lack of a better way to say this, they could apply group policy “up the stack” one level and do it on Hyper-V.

And once you do that, then you get all these other goodies for free, like management. Let’s say you had 100 2008 images running in the DC, 10/Hyper-V server. You could apply security policies to each one of those individual 100 images, or you could apply software switch group policy 10 times, once to each Hyper-V. Or even cooler, think about virtual client applications (like something from SoftGrid, for example) sharing access policy information directly with a virtual server guest over the wire. It could almost be like a V-VPN over a virtual (and remote) switch fabric. Oooohhhh….

Long story short, building this kind of technology directly into the virtual software switch gets away from adding more resources to the hypervisor environment, as is needed in Montego’s solution. Adding additional vswitch devices will at best introduce another management layer and at worst introduce more troubleshooting and downtime when virtual network connections go down or the policies block good traffic.

I mean I’m all for filling a gap that doesn’t exist, but I would love to see this type of technology built in from day one. User security and access (ie contextual-based decision making for unified access) is a core requirement for the VDC, yet using single-point solutions actually violates a good bit of what a true VDC will be. Virtual switches are still switches, and their goal in life is to pass packets. Let’s build packet, connection, and user security into the switch fabric, and treat these concepts as core requirements, just like virtual CAM tables.

One of the best parts of my job is getting out in the field and talking to people about what’s possible tomorrow: the future of the data center, the future of virtualization, the future of security, the future of Second Life and Holodecks…all that way out there stuff, although none of it is really that far out there anymore. Sometimes I can sit down with other’s who think like me and we just jump straight past the what-ifs to the whens and hows and start charting a course. But most of the time, especially in regards to public speaking, I start from the beginning, a “First there was light!” kind of thing, and then build up through three acts to the dramatic peak. I love talking to people in each group, but would probably lean towards the “Light Group” group because it always challenges me to help them understand.

During this steady ascent up with the Light Group, I rely heavily on analogies. We use analogies in every day conversations with our friends, and they’re an excellent vehicle for carrying new and foreign ideas. In my presentations, there’s always a baseline starting point, a referential section relating these new concepts to ideas that are already well understood, and then the theorem “thus” moment, bringing it all together. It’s just how I think, and how I believe people listen.

• Want to talk about VLANs? Ok, think of your network as a train, and certain people have tickets for the 1st Class coach and others have tickets for the Economy coach. You know what coach you’re in before you board the train, and you stay in your coach. The coaches are VLANs and you’re a packet…
• Curious about IP Reputation spam blocking? If your brother or sister walks up to you in an airport and hands you their luggage while they find the closet Starbucks, you’ll gladly hold their luggage and watch after it. You know them, you trust them, and you trust that they’re not handing you malicious luggage. If a complete stranger does the same thing, you’ll run to the closest TSA agent. The person giving you the luggage has a known and accepted reputation based on your history with that person. Same is true for MTAs and spam.

Relation is the key to successful informative communication. If you want or need someone to understand what you’re saying, you have to phrase and present it in a way that they’re expecting and, most importantly, so they can project themselves into that situation. People want to understand you and feel comfortable; if they can’t relate to what you’re saying, then they aren’t going to listen, digest, and understand.

Now you’re probably thinking “Great Alan, but what in the world does this diatribe have to do with the Virtual Data Center?” Excellent question! Before we can get to the advanced VDC discussions, we need to find a relate-able starting point and “baby step” our way to the meaty goodness. This is why I focus so heavily on terminology and definitions.

To paraphrase Neil Peart, we need to start with a waltz; the most fundamental component of what we’re doing that everyone understands, and then we need to build on that concept and take it to new levels. As we progress, we will get more complicated, but we’ll do it one baby step at a time. Even though I’m a musician, it’s very difficult for me to relate to a Neil Peart drum solo, but I can absolutely relate to a waltz and providing a basic backing rhythm. And then I can relate to adding patterns over the basic foundation of the waltz. And once I reach that point of relation, I’m able to sit back and appreciate Neil taking me from the baseline waltz, through the relate-able sticking motions, well into the “thus, I’ll never be this accomplished as a drummer” moment.

I’m a stickler for terms and definitions. If we’re all not using the same lexicon, how can we communicate? Did Enemy Mine teach us nothing?

There are two virtualization terms that are being used in a very loose manner lately, and me personally, I think they’re being used incorrectly. I’m not knocking the sentiment of their usage, just their assigned and (becoming) generally accepted usage. At first it just appeared to be analysts using these terms in a non-standard manner, but lately I’ve started to see vendors use them incorrectly as well. Yes, “incorrectly” is subjective, but let me explain by addressing these two terms:

• Virtual Appliances: When you live and breath in a world made up of hardware appliances, you tend to know what the word “appliance” refers to in a very black-and-white manner - It’s a black box that you buy, plug in, IP, then ssh/https into. That’s it. So virtual appliances are, quite literally, virtual software versions of their hardware big brothers. Zeus builds and distributes a perfect example of a pure virtual appliance. All too often lately the term has been used to describe full-blow virtual machines that have pre-installed OS’ and applications. So basically this is being used to represent a virtual version of a Ghost image, which is a disservice to true virtual appliances. Me, I prefer to still call these virtual machines or virtual images; they just happen to be pre-built to run on a specific hypervisor. You still use an off-the-shelf OS, like Windows 2003, and still use off the shelf software, like IIS. There’s nothing stopping you from downloading one of these virtual machines and re-purposing it for another use, something you can’t do with a true virtual appliance. And that’s basically the deciding factor: If you can use it for something beyond it’s original intent, it ain’t a virtual appliance.
• OS Virtualization: Same idea as above, “OS Virtualization” already has a definition: virtualizing an entire operating system, from hardware all the way up to apps. Now this term is being used for a completely different type of virtualization technology, something I call “kernel virtualization.” Kernel virtualization is, as the name suggests, virtualizing the actual kernel of the host operating system. It’s like paravirtualization on speed. The same kernel that is hosting the bare metal OS is also hosting the virtual guests running on top of it. OpenVZ is a good example of kernel virtualization. It’s really cool technology, but the specific term “OS Virtualization” already means one thing, so why is it being co-opted for something else? Kernel virtualization can be classified as a sub-category of OS virtualization; that, I’m fine with. But leave OS virtualization alone; it didn’t do anything to you for you to start picking on its given name.

To me, the naming terms of a particular technology should describe the technology. There are (according to some estimates) 250,000+ words in the English language. Do we really need to use the same words to describe virtualization technologies over and over again? We don’t need to be creative here, let’s just be literal.

There’s been a lot of talk about VMsafe in the blogosphere these past few weeks, but there’s been no talk about what VMsafe means for the virtualization market (specifically the hypervisor market) beyond just building security-based appliances. Sure, that’s some cool stuff, but I think there are larger movements in place here with VMsafe.

Let’s look at two competing OS virtualization technologies in the data centers: hypervisor virtualization solutions - VMware/Hyper-V, typical OS virtualization solutions (don’t be confused by people using “OS virtualization” incorrectly…urgh!) - and what I call kernel virtualization solutions - Virtuozzo/OpenVZ, technologies that virtualize the host kernel rather than place their own kernel on top of a hypervisor managed by a VMM. You can lump paravirtualization solutions into the first category for the sake of this discussion.

Kernel virtualizaiton offers a ton of benefits, including performance and host-based management. There’s a lot to be said for sharing and managing one kernel. But does this architecture remove the managability of the guests? On the surface I would think “yes”, because to allow guest-based management, you would need to access and communicate with the host kernel, something that sounds very, very scary. But with a hypervisor solution, you have the hypervisor/VMM combo that obfuscates the operations from the host kernel (to varying degrees depending on the implementation). And this type of architecture allows complete management of the virtual environment independent of the host kernel. Specifically for this conversation, enter VMsafe.

VMsafe opens full access to the hypervisor and VMM, allowing management from the virtual platform up. Sure, you still have a hypervisor/VMM and the performance hit that comes with this type of solution, but VMsafe allows full access in and out to see what’s going on, and more importantly, going through, the hypervisor. It opens up the full comm channel between guests and hosts; if there’s no hypervisor and VMM, that option isn’t there. So while most of the world is gleaning in on using VMsafe (as their own name suggests) to create a secure VMM environment (which, don’t get me wrong, is freakin’ awesome!), the core technology that’s enabling the Symantecs and Blue Lanes to do what they’re going to do also allows anyone to get state and process information from the hypervisor.

This is going to allow all kinds of cool things: monitoring the software switch on a frame/packet level and how it interacts with the host stack; monitoring the load and boot processes of a VMDK; application resource monitoring - how much computing power (CPU, RAM, scheduler time) does an application consume? - etc. So VMsae is definitely more than just a tool for the malware companies; this opens the door for hypervisor-level API integration, bypassing the VMI API layer that’s always been open.

Will kernel virtualization vendors, like Parallels Virtuozzo, open up access directly to the host kernel to allow the same thing? I would hope not, however maybe they do/will have a likewise solution.

Widely reported (within virtualization circles) last Thursday, Microsoft acquired secure OS software distribution vendor Kidaro. Very similar to ACE (VMware, not Cisco, for the ADN/ADC readers out there), the goal of these technologies is to securely wrap distribution and usage policies around distributable virtual OS images. For example, let’s say that I need to test a beta version of “top secret” software via a virtual image. Kidaro/ACE allow me to check out that image and run it only on my laptop; if someone else grabs the VMDK off the SAN, tough nookies, they can’t run it.

I played around with ACE about a year ago and really didn’t like it for two reasons:

1. It was based on local software DRM technologies. In a previous life, I was a PGM for software DRM solutions for digital media, and under the hood, they ain’t pretty. Disregarding the ethical implications of DRM for any software, it carries with it too high TCO to make it worth the limited security it brings. I know there has recently been an updated ACE but I haven’t seen it yet.
2. It still viewed everything within the scope of beefy OS images running on beefy hypervisors. If all I need is the application w/in the image, why do I need to carry with it the overhead of securing the app through the OS all the way down to the hypervisor? Doesn’t make much sense and seems like overkill.

That’s why I’m excited about the Kidaro acquisition. If Microsoft can bring this technology into the Hyper-V fold (where they already provide a virtually portable hypervisor), and specifically with SoftGrid, they could create secure portable applications or application hypervisors. And that would be cool.

I’m still waiting for the technology where instead of my employer providing me with their standard laptop, they give me $2,500 to buy the computing environment of my choice, and they provide all my business software via some type of portable, transparent application virtualization. I can see this beginning to take shape somewhere between Hyper-V hypervisor model and Virtuozzo Container’s kernel virtualization model. I’m not sure that Kidaro will give MS the portability to support any OS, but it does get closer to that model. We’ll see how they use it.

WARNING: I’m all riled up today and in a mood, so this is going to get long. Read at your own peril, and grab some coffee.

In an alternate life, I was a Linux bigot. No computer was coming into my house or sitting on my desk that didn’t run Linux. My WiFi AP was a Linux laptop that bridged between two PCMCIA cards; I tried to convince my wife that Gnome had everything she needed; I spent all of my free time compiling new graphic drivers to make sure I had fluid transparency in my SSH connection windows. But Linux ultimately got it wrong: there are far fewer people that want to spend their weekends tweaking CLI arguments than want their computers to “just work,” and ultimately I realized I was wasting my life trying to figure out why my ethernet driver didn’t come back up after I resumed from hibernate. I wanted my computing (personal and professional) environment to “just work.”

So one day, relatively out of the blue, I sucked up my pride, dropped a new hard drive in the Dell Latitude, and installed XP. And oh my friends, how the sun shone that day! Within an hour, I had a fully functioning portable computer: the sound worked and didn’t phase in and out when I as accessing the network; I could access the full resolution of my graphics card and move beyond 800×600; I didn’t have to manually edit a text file when I moved from one wireless network to another; presentations and projectors miraculously starting working (Andre: I’ve been there many, many times). In fact, everything “just worked.”

And most importantly, I became productive. After tooling around with XP on the removable hard drive for a few weeks, I realized that with Linux I was wasting so much time fussing with my working environment that I was actually becoming counter-productive. And suddenly, my laptop became a tool to get my job done rather than a machine that always needed TLC. Now granted I’ve had to defend my “Windows is just better” decision to my close circle of Linux supporters (Tux tattoos and all), but it’s been so worth the inner-circle humiliation and ridicule. But it didn’t have to be that way: There was a time when Linux was <this close> to focusing on the why people use computers instead of how computers work. SuSE and Mandrake where the closest to building a plug-and-play Linux distro for the masses, but ultimately they forgot about the normal users. And don’t me started on Red Hat, who I personally blame for the Linux downfall (Full Disclosure: I am an RHCE, so I’m not just ranting and raving on this one).

And likewise, data centers are just larger tools. I’ve talked on here before about the VDC as a service, and it’s all the same thing: use a tool designed to solve a goal for that goal and then move on. So you can imagine how torqued up I get when I get in conversations where people say “Oh, well this solution would work better if it ran on Linux instead of Microsoft…” Maybe for some people, yes, but for others it would run better on Solaris, or in Java, or on z/OS. Who freakin’ cares what it runs on as long as it accomplishes the goal?! Imagine how productive we could all be with full VDCs that didn’t require us to spend all day trying to get one API to talk to another, only to find out that we have to do it again 3 times for 3 different APIs to support everyone’s virtual OS infrastructure.

If the VDC is going to emerge as the disruptive powerhouse I think it will, we all have to put our biases aside and focus on the end goal: A DC that sucks in requests and spits out responses. Sure, we’ll have the Linux team and the Windows team, the VMware team and the Hyper-V team, the Cisco team and the F5 team, the network tap team and the SPAN port team, the plenum cable team and the non-plenum cable team…you get where I’m going with this. The VDC can not become a reality if we’re all fighting religious wars. So check your biases at the door and choose a tool that solves the individual “baby step” problem you’re trying to solve in your VDC today, move on to the next problem, rinse, repeat.

Now back to my regularly schedule Apple virtualization research project: Words can not describe the size of the smile on my face when I found the Terminal in Finder and opened it in Pro mode. A 50% transparent command line with bash! ls -alFrt works! Oh sweet *nix, why did I ever leave you?

I know, it’s been quiet around here lately. What can I say? Pitching the VDC message and the future of the data center has kept me very active lately (it’s a great time to be focusing on what I focus on). Now if people would just listening to me when I rant and rave about the problems with silo’ing virtualization technologies in the data center and the problems with software switches today, I’d be such a happy person.

Speaking of happy, the MacBook Pro arrived a few days ago. Man, does Apple know how to package or what? I’ve always wanted to live in a retro-future house; you know, like the one that you used to ride through in Spaceship Earth at Epcot Center at DisneyWorld in the 80’s (long before the 90’s remodel that included the futuristic Internet). Every childhood summer included a trip to Disney, and as a budding technologist, Epcot’s vision of the future was always my destination of choice. Forget Magic Kingdom; I wanted to play in the Imagination science lab and eat exotic Japanese food. Some 70’s children grew up obsessed with Star Wars, I grew up obsessed with the lifestyle of the future.

So my first thought when unpacking the MacBook Pro was something like “It’s silver, it’s white, and it glows when I turn it on!” You guessed it, they had me at “Designed in California.” Now this is old news for the millions of people who already have a MacBook, but coming from the world of Linux, Dell, and Vista (btw I love Vista as a powerhouse production Operating System), just opening the box instantly took me back to the retro-future, and I immediately ran upstairs and looked for my ultra-sonic dishwasher. But alas, it’s only 2008 and we’re still using water and chemicals to clean our dishes.

And before we even talk about the tech details, I have to take my hat off to Apple for combining lifestyle with technology, something sorely missing from other technology companies. We all have computers, we all have HDTVs and 6.1 surround systems, and yet there is so little work in the personal technology sector (IMO) on form, it’s all function with very little thought about How it’s used vs. What it does (more on that here).

As I write this, the MacBook Pro is sitting across from me on a chrome and glass table in my very modern-looking office, lid closed, with the pulsating “I’m sleeping” white light phasing in and out on the front, sitting immediately next to a replica antique soapstone and slate Chinese chess set. I’m struck by the beauty and juxtaposition of each of these tools, each centuries apart, yet both do exactly what they’re designed to do and both look amazing.

And beyond looks, the damn thing actually works like a champ. But no time to write about that now, maybe later. Right now, I have to go do that cool magic trick where you can lower your hands above the MacBook speakers and magically turn on the keyboard light…

It is a glorious day, my virtual friends! My new “Portable Application Virtualization Test Lab” has been ordered, and I’m currently camped out on my front porch waiting for the mailman (with my dog, who informed me that he waits for the mailman every day and, thus, is a bone-afide expert on the mail and package delivery schedule). What is a PAVTL (I need to come up with a good acronym for this)? Really only two components: the Portable part (so I can work on the research while traveling) is a MacBook Pro, 2.5Ghz; the AVTL, Parallels Virtuozzo Containers 4.0/VDI Manager. Well, the MacBook is being shipped and then I’m going to start test driving a demo version of Containers. And I’ll probably throw in a bit of Parallels Desktop for good measure. A friend has been waxing poetic about running Vista on OS X with Parallels since Vista’s GA, and he’s got me all excited about it.

Having defined the 8 Types of DC Virtualization, I can safely say I’ve had hands-on experience in implementing seven of the eight, lacking only Application Virtualization, so this will be the kick in the pants I need. But beyond that, with the current trend of OS containers, meeting somewhere in the middle of OS virtualization/virtual appliances (you know how I feel about that term, but I’m stuck with it) and application hypervisors, I’m really excited to begin researching Parallels Virtuozzo (and their new templates as well as OpenVZ in general). Virtual environments here I come (take that, Second Life)!

Have no fear, there will be plenty of updating once I settle into a testing groove. But if you don’t hear from me for a few weeks, I’ll be in my basement office basking in the glow of new technology. If you need anything, just swing by dressed as a mailman, and the fierce barking of my dog will surely summon me from my Virtuozzo bliss.