The Virtual Data Center

My favorite quote from RSA: “TheVirtualDC? Your blog is about virtualization and data centers, not security. Why do you even want to come to RSA?”*

Ahhh, finally back at the home office after two weeks of conferences: VMware Partner Expo in Orlando and RSA in San Francisco (with a pinch of SAP Virtualization Week thrown in the middle for flavor). It was a tiring trip but an excellent one for getting out in the field and talking to folks about virtualization and, as much as they would let me, security. I’ll have write-ups of each of the shows over the next week beginning with RSA today.

So RSA, we’ve known each other for years and sometimes you impress and sometimes you disappoint. I’d have to lean towards the latter this year; you really didn’t feel new and exciting. I was hoping that there would be a much larger virtualizaiton (and yes, cloud) security push this year than I saw, but the majority of what I witnessed in public (admittedly this was limited to the expo floor and partner-esque conversations due to a great list of analysts meetings that kept me from the general sessions) was the same ol’ same ol’: AV, IAM, UTM, network security, application security, FOBs on your iPhone, etc.

Now don’t get me wrong: obviously these are extremely important tools and technologies, but I guess I was expecting RSA to hold form and be more than the standard security show. If this year is any judge RSA will be returning to it’s pure security roots moving forward. Much like a storage show focuses solely on drives, data, and transport, RSA may be headed back to the days when us security geeks went to dig way down into the security internals. If that’s true, we had a few great years where RSA opened its arms to everyone. There was a time when it really felt that security was leading rather than following, and I just didn’t get that feeling with the show itself this year. The show felt like a necessary evil.

In contrast, the 1:1 meetings I had throughout the week were exactly as I’d hoped: Where’s security going? How can we use these security tools to create integrated solutions for the data center? What are the threats with cloud computing? How come virtual platform providers still haven’t moved beyond securing VMs and their flat virtual networks? Why is it still so easy to create VM trojans? Those were the amazing conversations I had outside RSA; talking with people who are passionate about security. But those interesting and productive conversations felt like we were whispering behind the gym in high school, as if we’d sneaked out of Physics 101 class to build our own rocket. And that alone, that feeling of making progress while the rest of the world remained stagnant, was worth the trip alone.

I know there were some great sessions on virtual and cloud security with Hoff, McKeay, and others that I didn’t witness first hand. I look forward to hearing/reading about those once RSA is officially done. And I can’t write a post about RSA without mentioning the excellent time I had at the RSA Security Bloggers Meetup 2009, another example of moving miles ahead outside of the organized show in just a few hours. Sure the drinks were flowing, but I still took copious mental notes and left excited with a smile on my face.

Anyway I’m already looking forward to next year, hoping that 365 days from today I’ll be writing about how far we’ve come in the past year on topics like virtual platform security. Until then I’ll just need to make sure I stay busy for the cause.

*In case you’re curious, that direct quote was made by someone working the show behind the scenes, someone extremely familiar with RSA. Awesome.

After pushing my latest post, Securing the Cloud: Shared Hardware and the Data Plane, Hoff posted a series of excellent questions and responses to the post via Twitter. I thought responding via another blog post, so that his questions could be addressed alongside my last post, was the way to go. I’ve trimmed some of his questions here for brevity but all of his questions can be found on his Twitter stream. And here we go.

@thevirtualdc I hate to tell you this, but your last blog isn’t about securing “the Cloud” at all. You are interchanging cloud & virt…

You are correct that I am presumptively interchanging the cloud with virtualization within the cloud. The primary point of this series of cloud security posts is to break out all the areas that securing the cloud entails, taking a huge topic that many people are discussing and breaking it down into small bits. A very large bite of those small bits, in my opinion, is the platforms that run each individual cloud. It’s been my experience that the majority (definitely not all) of cloud providers right now, and the customers that are seeking out these cloud providers, are using some form of virtual platforms. This is an assumption I’ve discussed here before. I’m definitely not saying virtualization=the cloud, but rather that most cloud implementations rely somewhat on virtual platforms. Virtual platforms introduce a layer of transparency in cloud providers; a customer who choose a provider that’s running virtual platforms will most likely know what that platform choice and what version it’s running. To that extent, the security of those platforms is paramount to the security of the cloud itself.

@thevirtualdc …not that they aren’t related, but by lumping everything into the IaaS bucket (which is what you are essentially doing)…

I’m not necessarily lumping all cloud providers into the IaaS bucket. Non-IaaS providers, such as AWS, Azure, and Terremark, are cloud providers that build their solutions on top of virtual platforms. These are the types of cloud providers that fall into my assumptive clause above. I’m not so concerned in this post with what those providers are doing with virtual platforms or how they’re marketing their service, but rather the fact that they are running shared virtual platforms and relying on shared data plane management from companies that are outside their control. No matter how they’re implementing these technologies, the customers are trusting the providers and the providers are trusting the platforms (along with a ton of other pieces in the cloud puzzle that I’ll delve into later as part of this continuing series) to keep things secure. Basically I’m talking here about any cloud provider that’s implementing a solution on top of stadard virtual platforms.

@thevirtualdc I totally buy everything you wrote, except you decided to call it Cloud instead of Virt which will add 2 the confusion.

I completely agree with you on this one. Goodness knows I get all up in arms about terminology and definitions when it comes to technology, but the choice to lump a discussion about virtual platform and shared data security under the Cloud nameplate was intentional. I want people who are looking at the cloud, who are looking at security concerns in the cloud, to start thinking about security risks of what’s actually running most of the cloud. For example, a major cloud provider recently discussed their solution for cloud security was to deploy individually managed distributed firewalls for their customers. That’s good, but has nothing to do with the security concerns of the virtual platforms that are running those distributed firewalls. That’s the reason I want to associate virtual platform security with cloud security. Sure, there are providers and customers that won’t need to worry about this, but I believe the majority of both will. I don’t want people to think that the cloud is magical and mystical. It’s not; most of it is running some of the same software that we’re running in the enterprise, software that’s highly prone to security breaches.

Hoff concluded with this comment, which I’m unable to find in his Twitter stream but is available via Google cache:

 @thevirtualdc What about folks who use Xen derivatives…like the 800lb gorilla of Cloud, Amazon?

You are correct; I omitted Xen from my “take responsibility” list in that post. Xen introduces a different element that’s slightly harder to control: the OEM’ing and open-source nature of their solution(s). There’s no question that a provider like Amazon who’s depending on Xen as their platform foundation should be concerned about the security of that platform, however, Xen has the ability to be modified (to varying degrees). With respect to security, this makes it much more difficult for Citrix to be ultimately responsible for a secure running environment. The ESX hypervisor is always the same. The Xen hypervisor may be different across every implementation. That introduces risks to the data plane that are much harder to control. Still as critical but it’s harder to lump Citrix in the same bucket as Microsoft and VMware in this scenario for that reason. Regardless, you are correct in that I should have addressed this in my last post.

As always the feedback from Hoff is much appreciated and enjoyed. Even if I’m way off the planet on this (and most of what I wax about here) at least it contributes to the discussion and makes us think about these things. Security risks associated with virtual platforms and not controlling the data plane won’t directly impact all cloud providers or all cloud customers. But it will impact a good number of them, and the fact that we’re not looking to these technology creators (ie the platform vendors) to lead the way and create safe computing environments for shared data…well, that keeps me awake at night.

Frequent readers will understand my love of lists, my affinity for the 4D attack plan methodology (Define, Design, Develop, Deploy), and my need to break things into small addressable (bite-sized) chunks. Over the past week I’ve been laying the groundwork for securing the cloud; not the technical “Use this VM, configure these VLANs, tether the clients this way” stuff but the larger macro business planning for techies on securing the cloud. Today follows suit in the Define category, going straight to the hardest problem first in cloud security: securing shared data plane resources: CPU, RAM, and bus.

Like it or not, we’re going to have to address and solve security of physical computing resources in the cloud sooner rather than later. And by sooner I mean now. First thing. Put down your VM security appliance and step away from your network and packets. This morning. Stop what you’re doing because I’m about ruin the image and the plan that you’re used to*. We need to figure out how to secure VM computing traffic over shared resources like CPU, RAM, and bus – the data plane implemented by virtual platforms and thus the backbone of the dynamic cloud. We’re going to deal with near-limitless attack vectors across all parts of the cloud but if we don’t secure the running environment first then we’ll be asking for someone to find an open door and take our virtual CPUs, our virtual networks, our virtual I/O.

Tools like VMware’s vShield Zones are good starts but they don’t go deep enough (at least from what about Zones today; I’ll know more after the Partner Exchange in April), managing policies and trust levels in the zones down to the bit level, not just the packet level. Exploits against the physical and virtual data planes will make network and application attacks looks like child’s play because the data plane owns the transport and storage of the targets of those attack. It’s going straight to the source. It will allow attacks from inside the cloud out through all those oh-so-useful networking and framework tools that have built up the cloud. It’s a like a microwave: attack the molecules from the inside.

Saying we’re going to do it is one thing, actually doing it is something different. Many moons ago I wrote a three-part piece about the hypervisor/platform vendors taking responsibility for their own virtual space. Their virtual CPUs, their virtual switches, their virtual IPC between host and guests; these items can all easily be secured by the vendors. But what about securing data in the hardware and the step that moves data from virtual software to the hardware? That, too, is mostly the platform vendors, but not all. With virtualization now happening in the CPU, securing that shared data in transit will require the platform vendors to work with the hardware manufacturers to address the problem and establish trust. How can the CPU trust bits from the hypervisor are safe and vise versa?

So how are the manufacturers and vendors going to do that? Easy: the platform vendors will need to create dedicated virtualization security teams that include working with hardware vendors, and start talking about this today. Get the word out that this is a critical issue and concern. Sound familiar? See, it’s all coming together to form an easily manageable plan of attack and execution for securing the cloud from the inside-out. But we have to start somewhere, and I prefer to start with the most difficult task first, the core issue and technologies at the center of the cloud, and then move out from that.

You’re reading this Microsoft and VMware, right? I thought so, just wanted to make sure.

*Yes, I did just borrow and paraphrase the opening line from The Humpty Dance, thank you very much.

I like lists. There’s no getting around my need to itemize everything, and surprisingly this is something that comes up in my every day life, every day. I even had a debate with someone recently on the proper way to structure pro and con lists: I prefer horizontal (pros listed first, then cons), she prefers vertical (pros on the left, cons on the right). Regardless of your pro/con list display preference, lists are critical to the way I think about things. This is most true when coming up with ideas about technology. Today’s list focuses on security in the cloud: How can we possibly tackle such a beast of a problem? Easy, with a list.

As I’ve talked about a good bit recently, security in the cloud is something we’re all currently thinking about and about to face head-on. But the phrase “securing the cloud” is a misnomer; we can no more easily secure the generic cloud than we can secure the entire generic internet. The cloud is made up of many, many pieces that start at a core center (the computing platform resources) and move out to the edge (the network). But that’s only one cloud; The Cloud (as we say) is actually made up of limitless smaller clouds where data is processed locally and then pushed out to another cloud for more processing.

With this in mind, let’s stop thinking about the insurmountable task of securing The Cloud and instead start looking at securing various parts of these micro-clouds. If we can secure the smaller parts then it will be easy to piece these together as we need (a jigsaw coming together if you will) to build out a complete cloud solution. So here’s the list: What smaller parts of the cloud should we start securing today?

• Secure the platforms: Microsoft, VMware, Citrix, hypervisors, virtual switching, segmentation of VM roles
• Secure the frameworks: Those wrappers around the platforms that control provisioning and resource management, tools that manage the data in and out of the cloud to the platforms
• Secure the network: Standard network security can be apply here but it needs to be managed in parallel with the other cloud delivery security solutions
• Secure the applications: The data receivers from the frameworks. Standard application security can apply here but should have the same requirements as securing the network (ie in context) and be paired with platform security
• Secure the endpoints: Doesn’t matter if an endpoint is a traditional client technology or another cloud (remember the good ol’ days of extranets? Yeah, let’s start calling them extraclouds!), anything responsible for seeding data into or receiving data out of the cloud needs to be secured and trusted
• Secure the edge: Just like the endpoints the edge needs to be secured to validate and protect data as it’s coming in and out; the Cloud Sentry
• Secure the Cloud<->Cloud connections: This is really an amalgam of edge and client security, but unlike the model today where we secure each independently, the Cloud<->Cloud security controls need to validate all data and connections in context to make sure that the data that’s supposed to be in the cloud is correct (it may be secure data before this point but now we need to look at it in context of these two clouds talking to each other)

This nice thing about breaking these items out in a list is that no single group has to tackle everything. The cloud providers are in the unique place where they can become the Secure Cloud Project Managers, but even then they’re relying on other groups to fulfill their end of the bargain by supplying secure solutions from each of their areas of expertise. Divide and conquer!

There’s no way that securing the giant cloud can be successful if we try to do it all at once and with only one solution. We need multiple solutions working together, and to get to those solutions we need to enter the first two phases of tackling a project: Define and Design.

Now that I’ve done the work of Defining the smaller, bite-sized categories for you, let’s go ahead and start securing each of those categories. Ready? ’cause this is going to take years…

My heart is truly warmed (which isn’t easy) by all the talk around cloud security. This may mark the first time in my career that I’ve seen a non-security bleeding-edge technology (c’mon, the cloud is bleeding like a sieve) hit the market coupled with concerns and ideas about security. Even if we look to the virtual foundation of the cloud, none of those technologies (hypervisors, virtual CPUs, shared RAM, storage virtualization, etc) hit the market with any care or concern about security. In this way the cloud is creating a new model of accessible computing in more ways than one.

But all the talk still isn’t enough. I know, I’m never happy. The talk needs to lead to action, and that action should be led by the big three platform vendors: Microsoft, VMware, and Citrix. Regardless of how they’re addressing the cloud in public with marketing and solutions right now, these three platforms provide the backbone (figuratively, not as in networking) for both service provider and enterprise cloud computing. There are limitless other components to the cloud I’ve talked about before, but all of those components have some reliance on solutions from one of these three vendors. Sure, you can argue that the cloud can happen without any Microsoft, VMware, or Citrix technology, but that argument would be so short it wouldn’t be worth the coffee that was ordered for the argument. So keeping in tone with most of my recent posts, this is a call to arms for the big three: Why don’t you each have very public virtual security teams canvasing the globe to gather data and offer solutions?

Here’s what I’d like to see from Microsoft, VMware, and Citrix:

1. A massive evangelical thought leadership virtual security push. I’m talking a carpet bomb attack where all you do it talk, talk, talk about the risks associated with security of virtualiztion and in the cloud. It doesn’t have to be accompanied by solutions at this stage, just spread the word and solicit feedback. I want to see deep technical security tracks at VMworld and MS TechEd. I’ll save a suggested list of topics for another post (’cause I got ‘em). At this point in the plan topics should cover all three types of virtual security.
2. Cloud security teams: It’s not enough to offer cloud services like Azure and AWS, you need to offer cloud security services as well. It (I’m generalizing here with the ‘it’ part) should be a click button when I provision a new system or service. There should be a toll-free number that I can call right now and ask Amazon what they use to secure storage calls over HTTP, or call MS and ask how they guarantee my sensitive traffic can’t leak across VLANs. I don’t want to search for it, I don’t want to submit a ticket, I want this information right in front of me and at my fingertips. And I want the people answering those calls to be security experts.
3. Behind-the-scenes security swat teams. As I’ve discussed before, virtual pentesters looking for ways to exploit hypervisors, to escape the guest, working with Intel and AMD on security risks of moving logic to the CPU, to MitM bus traffic as it moves from one CPU to another. I’m not picky on whether they publicly disclose this information (that’s not true, I would prefer they do but understand why they wouldn’t want to yet) so long as their doing the research today.
4. And finally, a single funnel-up management of all these teams. I want the hypervisor security team to work side-by-side with the cloud platform deployment teams. It does no good if these teams aren’t a single entity with weekly triage meetings. The evangelist who’s talking to an ISP in Japan needs to know the person back at HQ who’s responsible for securing traffic into the cloud data center. And no using the term ‘virtual teams’ here for the obvious reasons, and for the not-as-obvious reason that these need to be real teams that do nothing but cross-technology security research.

Not only will this plan help propel security of virtualization and the cloud, it will also do wonders for customers who are looking at the cloud for mission-critical apps. If I know how to deploy a secure vApp in my internal cloud, know how to secure the channel to move that vApp to my external cloud provider, and know that they are monitoring the security of my application data on the wire and on the bus, then I’m much more likely to move forward with a complete cloud model. Security geeks and business units unite! I want this group to explain to the world the security risks of VDI and how those compare/contrast to security risks of client virtualization.

I’ve heard from so many people in the field (partners, customers, friends) that virtual security isn’t a concern today, and that’s good news. But will you be ready when it is a concern, and who will you turn to for help getting ready? Hopefully you’ll be able to rely on your platform and cloud providers, so start asking them  your questions now.

As I mentioned last week I attended the first in a four-part series on Cloud Computing hosted by WTIA (next meeting at Amazon btw, really looking forward to that).

While none of the presentations at the first meeting involved security, it was ever-present in my mind throughout the evening. This is how my mind works: someone talks about distributing processes across multiple CPUs and I think “New area for MitM attack? Plant one bad instruction and DoS the whole thing, or worse, siphon off all proc data?” One area that stuck with me since that evening was the security within each of the vertical cloud infrastructure levels (top down: App, Platform, Language, Hardware, according to Aaron Kimball) and more specifically how future security threats may impact each of those levels.

For example, what happens when there is an exploit in the Language layer, one that is inherited all the way up the App and spans multiple Apps? And then what happens if the Hardware layer (the IaaS component) is told to spin, down, or delete VMs based on App events? Cloud services are made up of many moving and interdependent components. An exploit in any one piece can be felt across the entire platform, or the Cloud Stack if you will. An exploit in the middle could traverse the cloud stack up one way and then down the other.

So why is this different than the risk against localized data center components today? Two reasons:

1. All four cloud stack layers are inter-related and equally shared. A fully fledged cloud system could pull any number of solutions from any layer and plug them together.
2. Traditional data center computing is linear and uni-directional: requests come in, wait, responses go out, rinse, repeat. With the cloud data becomes non-linear and multi-directional. Simply stated, there are more ways to get at the payload. It becomes more difficult to exploit an entire system when you only have access through one door. However with the multi-entry architecture of the cloud there are more doors (and windows and crawlspaces and attics). Rather than all beating down one door, attackers can flank the cloud and attack multiple entry points.

It’s these differences that give me pause. More than any other attackable group of technologies (desktop OS, server OS, network, web apps, etc) I believe attacks against any part of the cloud structure represent more of a business threat than a pure security threat. Business are going to come to rely so heavily on the cloud that the cloud will soon host a good portion of their mission critical data (to what level it’s not already today). If there’s a major public exploit against the pieces of the cloud that everyone has access to, well then that’s no good for anyone, or their proprietary data.

And what about ownership and responsibility? What liability will AWS or Azure have if there’s a breach in the App Infrastructure (outside their control) that trickles down to the Hardware level (inside their control)?  How can we expect the really large cloud providers such as Amazon and Microsoft to respond? Will we see a new market and need for Incident Responders solely focused on the cloud? Jet Fighters if you will – yes, my goal is to push these Cloud analogies as far as I can, and the sky’s the limit (see, this will be fun!).

These are the ideas I worry about with the cloud. I would love to see more time and focus put on analyzing the risks for both cloud providers and end-user businesses pushing their services into the cloud on shared platforms. Sounds like an excellent job for my first batch of security recruits: Jet Fighters in Training.

I’m going to start this morning with a question that I’ve been pondering lately: Should we work out all our security issues with virtualization before we start looking at cloud security? Would it take an army of security researches to work on platform and cloud threats in tandem?

My gut response is “Yes, these are linear and we should address them in a very deliberate order.” A good bit of the cloud is based on virtualization technologies that we’re all using yet no one has thoroughly addressed the security threats associated with mass adoption of these platforms. The truth is we don’t have the luxury of time to ponder all the threat vectors of virtual platforms and deliberately control our roll-outs for security: these technologies are already live and, fortunately or unfortunately, supporting a vast number of our applications and data centers today.  That shipped has sailed and it’s full of ESX and Hyper-V boxes.

Of course this scares the <you know what> out of me, and yes, it keeps me awake at night wondering when this is all going to collapse due to one silly little worm. How long before we see a virtual Sasser that has real business reproductions and sinks an entire ship?

In addition to the security concerns we have on virtual platforms, we also have different security concerns, arguably more frightening, with cloud computing. Like the three categories of virtual security I wrote about way back during RSA last year, cloud security falls into two primary categories in my head: securing the data in transit and securing access (yep, I categorize everything and make lists like you wouldn’t believe).  These differ from the virtual platform threats in that cloud security is about process rather than code. We have to implement security checks and border patrol on the Cloud Highway: everything is inspected as it comes in and out and only certain workloads are allowed on the highway. We have to look inside every freight package that traverses the Highway, and that’s going to be one mighty endeavor.

I was having a conversation with a VDC reader last week and the topic of adoption vs risk came up: If the risk is so great (as I think it ultimately is) then why isn’t it getting more attention? I believe that no matter how great the risk, the rewards associated with implementing virtualization, and eventually moving outside the data center to the cloud,  are too great to be stalled by theoreticals, and if you can’t see risk then there is no risk. The mobility, the consolidation, the new levels of management and granularity, all of these are so much more important to business process and IT agility than security threats that frankly don’t exist yet. Business moves ahead.

So what do we do? We address these security risks in parallel. We start making virtual and cloud security a more visible issue. We need more virtual security researchers — dare I say it, because I’m not a fan of the branding — more white hats out there focusing on both sides. We need PhDs looking at hypervisor and resource-based security and we need ethical hackers focusing on the cloud. We need new Cloud PenTesters.

We need giant posters in every college depicting Max Headroom (the only virtual icon I can think of) smiling, pointing, and saying “I Wa-wa-wa-wantttt You-ou-ou!” The virtual security army needs new recruits, and we need them now.

I’ll close with another question to ponder: Will these recruits even know who Max Headroom is? Were they even born yet? 

For the most part, I’ve avoided talking about the current state of security in the cloud.  For one, there are many, many smart people and great bloggers already talking about it and I love reading what they write (and almost always agree with it, almost ).  And two, I spend a good bit of my day job (and have for the past 3 years) focused on the ideas around contextual-based access and security: security has different meanings and requirements depending on the context of how a particular service is accessed or invoked.  Now that’s some cool stuff, and the cloud access model is a natural extension of contextual-based access. You’ll see much more on this idea from me over the next year-ish or so.

Over the holiday break, I spent a good bit of time thinking about how and when cloud security was really going to become a watercooler issue (while playing Super Mario World on the Wii Virtual Console, ironically enough). We’re nowhere near there yet - heck, we’re still discussing the nuances of the cloud itself; we have to build that before we can worry about who or what gets in. But there a few anecdotal things that kept coming back to me:

1. In a previous life, I used to audit the security of data centers, from the physical rack and cage installations (even testing the man gates, now that was fun) all the way down to the network and applications.  I even conducted interviews of all DC staff, both enterprise and hosting, on what they knew, how they responded in the event of a breach, where root passwords were kept, all that meaty security goodness. My goal was to verify that a particular application was installed in the most secure running environment available, primarily to limit legal exposure should there be a breach against this application.  What I took from that multi-year experience: It’s extremely expensive to conduct these types of audits, and at some point the liability baton is passed to the people actually implementing the technology, away from those who designed it.  I could interview people all day, and spend weeks walking through their network, but once I left the premises and filed my report, it was up to them to stick to those procedures. We had to trust (in our case legally) that what I saw remained in place.

2. As I’ve mentioned here before, I own a small photography company, so small in fact, I use a cloud provider (for lack of a better term) for all our business documents, including balance sheets (very basic stuff: sold 10 photos, bought a new lens, etc). Before the end of the year, I was spending a nice, cold, snowy Sunday in beautiful downtown Bellingham at a local coffee shop where, over their WEP-enabled wifi network, I was working on said balance sheet, getting ready for the fun that is tax season. Me being extremely security conscious, I immediately noticed that the cloud App provider doesn’t use SSL to access my docs, and in fact, says they don’t even offer SSL unless I pay a much higher premium. Now lucky for me simply changing the login URL manually to HTTPS worked fine and I was happily encrypted. But I’m guessing I’m in the minority and most of their users won’t notice that they’re accessing what could be sensitive documents in a completely insensitive manner, or that it could easily be forced to work. Most users who got to step two would read the Help and see they had to pay for that service and then do their own value assessment: is the risk worth paying extra?

3. Amazon’s lack of security notification in their AWS mid-December, well documented by Craig on CloudSecurity.  A weakness in Amazon’s signing algorithm opened up the possibility for more-frequent-than-normal cryptographic collisions (yep, the same core issue that’s currently plaguing SSL CA providers who use MD5). Amazon did fix the problem, but it took them 7.5 months and they kept it under wraps, a very common response in the security world.

So noodling on these three things while on vacation led me to this realization: Cloud services are going to create a new level of trust between end-users, providers, and enterprises. There’s no way around it, simply because there are so many moving pieces to a cloud service that are outside our control.  There’s nothing in the world I could have done during the 7.5 months that Amazon was fixing their signing flaw except change my transport to Amazon, and I probably wouldn’t have done that because the issue wasn’t public and I didn’t know about it. Once I start sending requests into the AWS, I have to trust that the systems in place are going to work and work securely.

Now that’s a big leap: one that I wouldn’t make with a more standard computing model. But in the cloud model, we have to trust so many new components in the stack. Of course we can have safeguards (SSL) and checks and balances (pen-tests, people who responsibly disclose security flaws) but at a minimum, those require near unfettered access to systems that are no longer in our control and require knowledgeable people to address them. In my auditing days I had unfettered access, during a specific window of time. Once I was done my access went away, so even then there was no continual checking of those systems.

So moving forward, as the security people tear apart the (in)security of cloud computing, the rest of the world will just need to take that leap of trust. A lowering of our standards for what we can control in the cloud’s outsourced data model. As an end user it kills me, but I know I have to make those sacrifices if I want to use those services. So I have to modify my level of trust, and apply new and stronger safeguards to the rest of my workflow processes (personal and professional) to make sure I’m able to recover if/when there is a massive breach that’s beyond my control.  My recovery is something I can control, and I definitely trust myself.

Ok, we have to talk about Hoff’s comments on VM sprawl, and it’s lack of security severity, for a second.  Now I’m guessing that he’s more focused on picking at the wound to spark conversation, and driving an excellent discussion on terminology, than he is in actually suggesting VM sprawl isn’t a security threat. But I’ll bite for a minute, and expand on my comments to his original post.

Yes, I do believe VM sprawl is a very serious threat, specifically due to what he calls out: lack of management. But it’s more than just a lack to me; I believe it’s an inability to even try to manage all VMs that is the problem. This same problem exists with physical servers, too. If someone can get access to the data center, bring in a box, find a switch port and know how that network is configured, then she can install a rogue physical server. But the barrier is so, so much lower with VMs because everything is “flat” once it’s virtualized. Our management choices today, such as vCenter, are doing a good job at creating virtual security barriers analogous to a keycard on the data center door, but we’re not quite there yet.

This is like managing termites. You can do everything in your power to protect your house from those little invaders by managing your perimeter and trying your best to find where they’re coming in, but if your builder “accidentally” left an exposed beam in your root cellar then built concrete blocks around it, it’s going to be very difficult to find that point of entry. Just like rogue VMs. It’s going to be extremely difficult to stop new VMs from spinning up at every point in your network.  There are ways to do this on the network level in the data center with IP registration, passive packet sniffing the virtual switches, 100% control over application traffic routing, etc.  But they’re not full-proof, and it becomes much more difficult in the desktop world.

Example: I use VMs for work on a daily basis. When I travel for work, those VMs come with me on my laptop. Inevitably, when I fire up my laptop and VMs on the hotel network and go through the “pay to play” registration process, they want to charge me for both my laptop and my VM. On a basic level, this makes sense: they charge and auth based on MAC address, I typically run my VMs in bridge mode, they ARP their own MAC over the NIC attached to the hotel network, cha-ching!.  But it’s an easy fix that I forget about until I see the 2nd $19.95 charge for the same day appear on my bill: NAT mode. Hiding the VM behind the MAC that the hotel is already aware of saves me from calling the front desk.

This is a classic example of where VM sprawl is such a threat. I could easily set up 100s of VMs on their hotel network and only pay for one access connection, and that would have no insight into what I was doing on their network with my VM farm (assuming I’m doing something malicious outside of their network and an encapsulating in SSL). Sure, it would be slow as molasses, but I’m not concerned with speed, I want power. 

This is the same threat that enterprises are currently facing with VM sprawl.  The specific threat is the ease at which users can spin up VMs: they don’t even need IP addresses, just a terse understanding of NAT. And for the most part, I don’t believe the threat of rogue VMs comes from sending traffic TO the rogue VM anyway. It comes from what that rogue VM is sending OUT on the network.  What is it spewing across a corporate or production network?  How hard is it to track and shut down if it’s NAT’d? Even if it’s only up for 5 minutes, that’s enough. VDI Bombs only take seconds.

So as always, I thank Hoff for posing an excellent question and for the lively discussion on Twitter between him, myself, and Lori. I do agree that a proplery managed VM infrastructure, with tools like vCenter and 3rd party VM network monitoring tools drastically help stave off full-blown VM sprawl. But it’s the inability of these solutions to monitor everything that scares me the most.  But if nothing, Hoff knows how to pick at the market’s shared security wounds. 

It’s been a while; content has been calling my name. I recently wrote an article for Virtualization Magazine titled Security Implications of Virtualization Platforms in the Virtual Data Center (I know, a crazy long title, but that’s how it was published ). WARNING: That page auto-launches flash video with audio enabled – gave me a bit of a pause when I heard some guy talking and interrupting my current playlist.

I like this piece because I introduce three concrete steps that IT departments can take today to help guard against security attacks tomorrow. These aren’t necessarily revolutionary ideas but they are tangible, tactical steps that can be implemented today during the planning, architecture, and roll-out phases of virtual platform installations and migrations. From the article:

In general, IT departments should focus on three virtualization areas as part of their entire virtualization security architecture:

1. Segmentation of VMs by location
2. Segmentation of VMs by service type
3. Proactive security management throughout the VM lifecycle

These three areas will help IT departments protect their virtual infrastructure against current threats as well as help mitigate the threat of future attacks.

You all know me: I’m all about baby steps and management. I don’t talk about specific threat vectors in this piece intentionally. Right now I’m very much in the design and planning stages of virtsec for IT departments. Remember my 4Ds: Define, Design, Develop, Deploy? This piece is all about the first two: know your risks and design an architecture that be used to manage those risks. Sure, if you’re keeping score, the 3 solutions above are actually in the Deploy category but I want to emphasize the planning portions of those solutions.

Start by planning today for whatever the virtsec world will throw at you tomorrow.