The Virtual Data Center

Frequent readers will understand my love of lists, my affinity for the 4D attack plan methodology (Define, Design, Develop, Deploy), and my need to break things into small addressable (bite-sized) chunks. Over the past week I’ve been laying the groundwork for securing the cloud; not the technical “Use this VM, configure these VLANs, tether the clients this way” stuff but the larger macro business planning for techies on securing the cloud. Today follows suit in the Define category, going straight to the hardest problem first in cloud security: securing shared data plane resources: CPU, RAM, and bus.

Like it or not, we’re going to have to address and solve security of physical computing resources in the cloud sooner rather than later. And by sooner I mean now. First thing. Put down your VM security appliance and step away from your network and packets. This morning. Stop what you’re doing because I’m about ruin the image and the plan that you’re used to*. We need to figure out how to secure VM computing traffic over shared resources like CPU, RAM, and bus – the data plane implemented by virtual platforms and thus the backbone of the dynamic cloud. We’re going to deal with near-limitless attack vectors across all parts of the cloud but if we don’t secure the running environment first then we’ll be asking for someone to find an open door and take our virtual CPUs, our virtual networks, our virtual I/O.

Tools like VMware’s vShield Zones are good starts but they don’t go deep enough (at least from what about Zones today; I’ll know more after the Partner Exchange in April), managing policies and trust levels in the zones down to the bit level, not just the packet level. Exploits against the physical and virtual data planes will make network and application attacks looks like child’s play because the data plane owns the transport and storage of the targets of those attack. It’s going straight to the source. It will allow attacks from inside the cloud out through all those oh-so-useful networking and framework tools that have built up the cloud. It’s a like a microwave: attack the molecules from the inside.

Saying we’re going to do it is one thing, actually doing it is something different. Many moons ago I wrote a three-part piece about the hypervisor/platform vendors taking responsibility for their own virtual space. Their virtual CPUs, their virtual switches, their virtual IPC between host and guests; these items can all easily be secured by the vendors. But what about securing data in the hardware and the step that moves data from virtual software to the hardware? That, too, is mostly the platform vendors, but not all. With virtualization now happening in the CPU, securing that shared data in transit will require the platform vendors to work with the hardware manufacturers to address the problem and establish trust. How can the CPU trust bits from the hypervisor are safe and vise versa?

So how are the manufacturers and vendors going to do that? Easy: the platform vendors will need to create dedicated virtualization security teams that include working with hardware vendors, and start talking about this today. Get the word out that this is a critical issue and concern. Sound familiar? See, it’s all coming together to form an easily manageable plan of attack and execution for securing the cloud from the inside-out. But we have to start somewhere, and I prefer to start with the most difficult task first, the core issue and technologies at the center of the cloud, and then move out from that.

You’re reading this Microsoft and VMware, right? I thought so, just wanted to make sure.

*Yes, I did just borrow and paraphrase the opening line from The Humpty Dance, thank you very much.

I like lists. There’s no getting around my need to itemize everything, and surprisingly this is something that comes up in my every day life, every day. I even had a debate with someone recently on the proper way to structure pro and con lists: I prefer horizontal (pros listed first, then cons), she prefers vertical (pros on the left, cons on the right). Regardless of your pro/con list display preference, lists are critical to the way I think about things. This is most true when coming up with ideas about technology. Today’s list focuses on security in the cloud: How can we possibly tackle such a beast of a problem? Easy, with a list.

As I’ve talked about a good bit recently, security in the cloud is something we’re all currently thinking about and about to face head-on. But the phrase “securing the cloud” is a misnomer; we can no more easily secure the generic cloud than we can secure the entire generic internet. The cloud is made up of many, many pieces that start at a core center (the computing platform resources) and move out to the edge (the network). But that’s only one cloud; The Cloud (as we say) is actually made up of limitless smaller clouds where data is processed locally and then pushed out to another cloud for more processing.

With this in mind, let’s stop thinking about the insurmountable task of securing The Cloud and instead start looking at securing various parts of these micro-clouds. If we can secure the smaller parts then it will be easy to piece these together as we need (a jigsaw coming together if you will) to build out a complete cloud solution. So here’s the list: What smaller parts of the cloud should we start securing today?

• Secure the platforms: Microsoft, VMware, Citrix, hypervisors, virtual switching, segmentation of VM roles
• Secure the frameworks: Those wrappers around the platforms that control provisioning and resource management, tools that manage the data in and out of the cloud to the platforms
• Secure the network: Standard network security can be apply here but it needs to be managed in parallel with the other cloud delivery security solutions
• Secure the applications: The data receivers from the frameworks. Standard application security can apply here but should have the same requirements as securing the network (ie in context) and be paired with platform security
• Secure the endpoints: Doesn’t matter if an endpoint is a traditional client technology or another cloud (remember the good ol’ days of extranets? Yeah, let’s start calling them extraclouds!), anything responsible for seeding data into or receiving data out of the cloud needs to be secured and trusted
• Secure the edge: Just like the endpoints the edge needs to be secured to validate and protect data as it’s coming in and out; the Cloud Sentry
• Secure the Cloud<->Cloud connections: This is really an amalgam of edge and client security, but unlike the model today where we secure each independently, the Cloud<->Cloud security controls need to validate all data and connections in context to make sure that the data that’s supposed to be in the cloud is correct (it may be secure data before this point but now we need to look at it in context of these two clouds talking to each other)

This nice thing about breaking these items out in a list is that no single group has to tackle everything. The cloud providers are in the unique place where they can become the Secure Cloud Project Managers, but even then they’re relying on other groups to fulfill their end of the bargain by supplying secure solutions from each of their areas of expertise. Divide and conquer!

There’s no way that securing the giant cloud can be successful if we try to do it all at once and with only one solution. We need multiple solutions working together, and to get to those solutions we need to enter the first two phases of tackling a project: Define and Design.

Now that I’ve done the work of Defining the smaller, bite-sized categories for you, let’s go ahead and start securing each of those categories. Ready? ’cause this is going to take years…

My heart is truly warmed (which isn’t easy) by all the talk around cloud security. This may mark the first time in my career that I’ve seen a non-security bleeding-edge technology (c’mon, the cloud is bleeding like a sieve) hit the market coupled with concerns and ideas about security. Even if we look to the virtual foundation of the cloud, none of those technologies (hypervisors, virtual CPUs, shared RAM, storage virtualization, etc) hit the market with any care or concern about security. In this way the cloud is creating a new model of accessible computing in more ways than one.

But all the talk still isn’t enough. I know, I’m never happy. The talk needs to lead to action, and that action should be led by the big three platform vendors: Microsoft, VMware, and Citrix. Regardless of how they’re addressing the cloud in public with marketing and solutions right now, these three platforms provide the backbone (figuratively, not as in networking) for both service provider and enterprise cloud computing. There are limitless other components to the cloud I’ve talked about before, but all of those components have some reliance on solutions from one of these three vendors. Sure, you can argue that the cloud can happen without any Microsoft, VMware, or Citrix technology, but that argument would be so short it wouldn’t be worth the coffee that was ordered for the argument. So keeping in tone with most of my recent posts, this is a call to arms for the big three: Why don’t you each have very public virtual security teams canvasing the globe to gather data and offer solutions?

Here’s what I’d like to see from Microsoft, VMware, and Citrix:

1. A massive evangelical thought leadership virtual security push. I’m talking a carpet bomb attack where all you do it talk, talk, talk about the risks associated with security of virtualiztion and in the cloud. It doesn’t have to be accompanied by solutions at this stage, just spread the word and solicit feedback. I want to see deep technical security tracks at VMworld and MS TechEd. I’ll save a suggested list of topics for another post (’cause I got ‘em). At this point in the plan topics should cover all three types of virtual security.
2. Cloud security teams: It’s not enough to offer cloud services like Azure and AWS, you need to offer cloud security services as well. It (I’m generalizing here with the ‘it’ part) should be a click button when I provision a new system or service. There should be a toll-free number that I can call right now and ask Amazon what they use to secure storage calls over HTTP, or call MS and ask how they guarantee my sensitive traffic can’t leak across VLANs. I don’t want to search for it, I don’t want to submit a ticket, I want this information right in front of me and at my fingertips. And I want the people answering those calls to be security experts.
3. Behind-the-scenes security swat teams. As I’ve discussed before, virtual pentesters looking for ways to exploit hypervisors, to escape the guest, working with Intel and AMD on security risks of moving logic to the CPU, to MitM bus traffic as it moves from one CPU to another. I’m not picky on whether they publicly disclose this information (that’s not true, I would prefer they do but understand why they wouldn’t want to yet) so long as their doing the research today.
4. And finally, a single funnel-up management of all these teams. I want the hypervisor security team to work side-by-side with the cloud platform deployment teams. It does no good if these teams aren’t a single entity with weekly triage meetings. The evangelist who’s talking to an ISP in Japan needs to know the person back at HQ who’s responsible for securing traffic into the cloud data center. And no using the term ‘virtual teams’ here for the obvious reasons, and for the not-as-obvious reason that these need to be real teams that do nothing but cross-technology security research.

Not only will this plan help propel security of virtualization and the cloud, it will also do wonders for customers who are looking at the cloud for mission-critical apps. If I know how to deploy a secure vApp in my internal cloud, know how to secure the channel to move that vApp to my external cloud provider, and know that they are monitoring the security of my application data on the wire and on the bus, then I’m much more likely to move forward with a complete cloud model. Security geeks and business units unite! I want this group to explain to the world the security risks of VDI and how those compare/contrast to security risks of client virtualization.

I’ve heard from so many people in the field (partners, customers, friends) that virtual security isn’t a concern today, and that’s good news. But will you be ready when it is a concern, and who will you turn to for help getting ready? Hopefully you’ll be able to rely on your platform and cloud providers, so start asking them  your questions now.

As reported by Practical Technology yesterday, Citrix has announced that it will be giving away XenServer (the hypervisor portion of their acquired Xen solution) for free. Very similar to the VMware model, they will continue to charge for management and advanced features as add-ons to the XenServer platform but the core virtualization technology will be available for free. Will this have any impact in their market share or their stake in the virtual data center? I’m skeptical for a number of reasons:

1. Free doesn’t always mean better, and usually it’s the exact opposite. I’m reluctant to think that enterprise customers and mission-critical cloud providers will take the free route solely to save money. Maybe that’s more likely this year than last but I’m still not sold it will make that much of an impact, especially since this won’t be an OSS offering ala XenSource. They’re also competing up-stream against VMware’s ESXi and Hyper-V, both of which have value-add above and beyond the hypervisor. For Citrix to make this move work they’re going to have to prove that they have the same type of value-add with XenServer that the other two big players offer. And free doesn’t count as value where your mission critical apps are concerned.
2. They’re relying heavily on management solutions to bring in the money for XenServer customers. We all know you can’t deploy a truly reliable virtualization solution without having management. In practice free deployments of XenServer will be few and far between, instead they will be bundled with some management solution. But which one? If MS System Center will manage both Hyper-V and XenServer, then we’re back to that value-add question for MS shops. If I’ve already deployed MS SCOM/SCVMM then what’s the value in deploying a Citrix solution instead of Hyper-V — which is also integrated into my server platform? VMware shops will stick with VMware because they’re the only company that has an end-to-end solution today. So is management the golden ticket for Citrix?
3. They’re announcing a new partnership with Microsoft. Excluding their existing partnership (whether that partnership has benefitted Citrix is a decently debatable topic so let’s gloss over those particulars for now) — Has a partnership with Microsoft between competing companies ever worked out for the non-Microsoft partner? Yes, MS is an excellent partner resource for technologies that don’t compete, but really, this partnership (and the ability to manage XenServer) will drastically benefit Microsoft by extending their heterogeneous virtual management solutions with Systems Center. I don’t see it going the other way for Citrix. Again, we’re going back to the value for a complete solution and who offers the most value here: Microsoft or Citrix? My vote is very strongly printed in the Microsoft column.

Maybe this will work in Citrix’s favor by at least getting XenServer in more hands to play with, and maybe their technology value will bubble to the top. But how many times have we watched a better technology (assuming XenServer is better) go by the wayside because they didn’t offer enough business value? In this case, I question whether a free XenServer offers any business value above and beyond what Hyper-V or VMware ESXi already offer.

As I mentioned last week I attended the first in a four-part series on Cloud Computing hosted by WTIA (next meeting at Amazon btw, really looking forward to that).

While none of the presentations at the first meeting involved security, it was ever-present in my mind throughout the evening. This is how my mind works: someone talks about distributing processes across multiple CPUs and I think “New area for MitM attack? Plant one bad instruction and DoS the whole thing, or worse, siphon off all proc data?” One area that stuck with me since that evening was the security within each of the vertical cloud infrastructure levels (top down: App, Platform, Language, Hardware, according to Aaron Kimball) and more specifically how future security threats may impact each of those levels.

For example, what happens when there is an exploit in the Language layer, one that is inherited all the way up the App and spans multiple Apps? And then what happens if the Hardware layer (the IaaS component) is told to spin, down, or delete VMs based on App events? Cloud services are made up of many moving and interdependent components. An exploit in any one piece can be felt across the entire platform, or the Cloud Stack if you will. An exploit in the middle could traverse the cloud stack up one way and then down the other.

So why is this different than the risk against localized data center components today? Two reasons:

1. All four cloud stack layers are inter-related and equally shared. A fully fledged cloud system could pull any number of solutions from any layer and plug them together.
2. Traditional data center computing is linear and uni-directional: requests come in, wait, responses go out, rinse, repeat. With the cloud data becomes non-linear and multi-directional. Simply stated, there are more ways to get at the payload. It becomes more difficult to exploit an entire system when you only have access through one door. However with the multi-entry architecture of the cloud there are more doors (and windows and crawlspaces and attics). Rather than all beating down one door, attackers can flank the cloud and attack multiple entry points.

It’s these differences that give me pause. More than any other attackable group of technologies (desktop OS, server OS, network, web apps, etc) I believe attacks against any part of the cloud structure represent more of a business threat than a pure security threat. Business are going to come to rely so heavily on the cloud that the cloud will soon host a good portion of their mission critical data (to what level it’s not already today). If there’s a major public exploit against the pieces of the cloud that everyone has access to, well then that’s no good for anyone, or their proprietary data.

And what about ownership and responsibility? What liability will AWS or Azure have if there’s a breach in the App Infrastructure (outside their control) that trickles down to the Hardware level (inside their control)?  How can we expect the really large cloud providers such as Amazon and Microsoft to respond? Will we see a new market and need for Incident Responders solely focused on the cloud? Jet Fighters if you will – yes, my goal is to push these Cloud analogies as far as I can, and the sky’s the limit (see, this will be fun!).

These are the ideas I worry about with the cloud. I would love to see more time and focus put on analyzing the risks for both cloud providers and end-user businesses pushing their services into the cloud on shared platforms. Sounds like an excellent job for my first batch of security recruits: Jet Fighters in Training.

I’ve hinted in the past on my ultimate application virtualization scenario – where I want the market to be for deploying and supporting remote applications for clients in the future. I’m still working on that giant whiteboard architecture map in my basement on what AppVirt looks like from the DC computing side, but today I want to write about the client side of that architecture. And while brevity has eluded me in all parts of my lingual life recently, I’m going to try to be succinct here (expect failure).

I attended the first of a four-part series on Cloud Computing last night held by the WTIA, which included an excellent presentation by Aaron Kimball from Cloudera on the basics of the cloud from the data point of view. Having retired the engineer title for marketing a few years ago, it always makes me happy to see someone who spends their career designing complex systems stand up and give an intro presentation that also includes the business benefit. So often engineers address the How rather than the What and the Why, and Aaron did an excellent job with the latter.

His presentation, along with others last night, got me thinking about what application virtualization in the cloud would look like to the client (and I’m not talking about GMail here).  Let’s look at a real example:

I bought a Mac a few months ago primarily to run Lightroom, so I spec’d out the Mac to go high-end because it would be running a very beefy photo application (along with Photoshop in the future I’m sure). The machine also had to run VMware Fusion in parallel (no pun intended; sorry to my Parallel friends) - I have a photo stitching application that’s currently Windows-only. Standard operation keeps me stable at 75% RAM and 40% CPU on average.

But what if I didn’t need to buy local computing resources and everything was processed remotely? Let’s jump ahead 10 years (a big leap, I know) and look at how this could be different if client apps were in the cloud.

I buy a local processing machine that’s drastically stripped down from my current Mac. I boot this machine to a web browser, where I head over to Adobe and say “This is Alan; I need to run Lightroom.” Adobe says “No problem. Let me push down the secure Lightroom App Shell. Ok, now you’re ready. Here’s a list of your albums pulled from Amazon S3.” I say “I need to process the latest batch of Mt. Baker HDR images.”  HDR images take a substantial amount of computing power to process, so Adobe comes back and says “No problem. I’m going to need 2gig RAM and a dedicated CPU core for this, but your monthly subscription only covers 1gig and .5 core. I’ll charge you $0.021/minute if you’d like to burst.”  I say “Great, let’s do it.”

Amazon then pulls its own resources from AWS and start distributing my HDR processing over thousands of machines/cores/RAM, all controlled from my local Lightroom App Shell. To me it’s displayed as one 8ghz core (remember, 10 years from now:) ), but in practice that value is an aggregate of distributed resources pooled from thousands of machines.I process my HDRs, they’re stored back in my Lightroom library which is managed by Adobe.com, I take my tiny laptop over to the client’s office, rinse, repeat.

But what if I don’t want the processing done exclusively in the cloud? I mean after all the above scenario is very similar to what I can do today with a web browser. What if instead my local computing platform is moderately beefy and I process everything locally. But as my machine starts to bog down from stitching together HDR panoramas it uses the Lightroom App Shell to request raw computing resources from AWS, and I become part of the distributed cluster? I scale and process background tasks remotely and only use my local resources for rendering the images to my display. My machine is part of the cluster and the line between local and remote processing becomes blurred. That’s some cool client-based application virtualization. The application running state is spread across elastic resources, of which my local resources are a part of.

True application virtualization will be a huge undertaking and this is simply one part and one idea. But why not think big? Go for the gusto I say. Oh, and there’s more here, but I’ve long since crossed the brevity line. Maybe more later.

I’m going to start this morning with a question that I’ve been pondering lately: Should we work out all our security issues with virtualization before we start looking at cloud security? Would it take an army of security researches to work on platform and cloud threats in tandem?

My gut response is “Yes, these are linear and we should address them in a very deliberate order.” A good bit of the cloud is based on virtualization technologies that we’re all using yet no one has thoroughly addressed the security threats associated with mass adoption of these platforms. The truth is we don’t have the luxury of time to ponder all the threat vectors of virtual platforms and deliberately control our roll-outs for security: these technologies are already live and, fortunately or unfortunately, supporting a vast number of our applications and data centers today.  That shipped has sailed and it’s full of ESX and Hyper-V boxes.

Of course this scares the <you know what> out of me, and yes, it keeps me awake at night wondering when this is all going to collapse due to one silly little worm. How long before we see a virtual Sasser that has real business reproductions and sinks an entire ship?

In addition to the security concerns we have on virtual platforms, we also have different security concerns, arguably more frightening, with cloud computing. Like the three categories of virtual security I wrote about way back during RSA last year, cloud security falls into two primary categories in my head: securing the data in transit and securing access (yep, I categorize everything and make lists like you wouldn’t believe).  These differ from the virtual platform threats in that cloud security is about process rather than code. We have to implement security checks and border patrol on the Cloud Highway: everything is inspected as it comes in and out and only certain workloads are allowed on the highway. We have to look inside every freight package that traverses the Highway, and that’s going to be one mighty endeavor.

I was having a conversation with a VDC reader last week and the topic of adoption vs risk came up: If the risk is so great (as I think it ultimately is) then why isn’t it getting more attention? I believe that no matter how great the risk, the rewards associated with implementing virtualization, and eventually moving outside the data center to the cloud,  are too great to be stalled by theoreticals, and if you can’t see risk then there is no risk. The mobility, the consolidation, the new levels of management and granularity, all of these are so much more important to business process and IT agility than security threats that frankly don’t exist yet. Business moves ahead.

So what do we do? We address these security risks in parallel. We start making virtual and cloud security a more visible issue. We need more virtual security researchers — dare I say it, because I’m not a fan of the branding — more white hats out there focusing on both sides. We need PhDs looking at hypervisor and resource-based security and we need ethical hackers focusing on the cloud. We need new Cloud PenTesters.

We need giant posters in every college depicting Max Headroom (the only virtual icon I can think of) smiling, pointing, and saying “I Wa-wa-wa-wantttt You-ou-ou!” The virtual security army needs new recruits, and we need them now.

I’ll close with another question to ponder: Will these recruits even know who Max Headroom is? Were they even born yet?