The Virtual Data Center

One of the oft discussed business challenges of cloud-based application deployments – or any remote app deployment where a service has to communicate over the public internet – is latency. It takes more time to fetch data when a request has to leave the LAN, and latency is usually variable and at the mercy of both the Interwebs and the cloud provider. This isn’t so much of an issue when your entire app is deployed in the cloud and users are going directly there for data; the user won’t notice any difference between accessing your app after it’s moved to AWS than they did when you had it deployed in your own data center. In fact some times it might even be faster.

The latency monster rears its ugly head when apps are spread across data centers, either in a split architecture or with bursting, where the user is first directed to your local data center and then a decision is made to move that request (and possibly that entire user session) to the cloud portion of the app. Solutions exist today to help optimize the applications and the network to provide a better user experience, but there’s a new(er) trend that’s not getting much attention to help combat geographic latency: regional cloud providers.Regional cloud providers are exactly as the name suggests: providers that offer cloud services close to your physical data center. Think of these providers as “buy local” clouds. Today these providers offer local cloud hosting services that compete with the larger players, such as Amazon and RackSpace, with local knowledge and local support.

But what about a hybrid model, the Cloud Franchise: local owners and operators that offer local cloud services but also offer branch versions of the larger cloud options. Some of this model exists in the platform levels today, in fact there is a thriving marketplace for start-ups who are offering AWS-based solutions, pre-packaged and ready to go. But those solutions, once deployed, still run in AWS data centers (although AWS doesn’t publish this information, their US data centers are rumored to be located on the East Coast). If I have latency concerns about bouncing my users from one part of my app in my private Missoula data center to another part of my app located in Baltimore, then a pre-packaged AWS solution won’t really help me with that. I want all parts of my app to be as local as possible, especially when I need to burst into or direct users to the cloud.

That’s where the cloud franchise model comes into play: regional cloud providers can offer pre-packaged AWS services as well as be a branded AWS hoster, hosting those deployed services in a local data center rather than in Baltimore. The customer is still using AWS and has access to 100% of the AWS products and features, but the end result – the hosted application — is running on an AWS platform in Missoula instead of Baltimore, run by the local provider.

AWS is just one type of architecture: Azure is another place where the cloud franchise architecture could come into play. If I write my .Net app to span between both my local data center and an Azure cloud, I want to make sure that I have the shortest path in place between my data center and where my app is actually running in the Azure cloud. Mainstreet is going to perform so much better if it only has two hops between my local DC and Azure, both in Missoula, than if it has to bounce over multiple providers to cross the country to one centralized data center in Dallas.

At the end of the day the goal is to reduce latency between my data center and my public cloud. The more I can control in my user’s experience the more likely I am to deploy into the cloud, especially for latency-sensitive apps such as VDI. One way to control access to the my cloud apps is to control location: buy local from a regional cloud provider who is also a cloud franchisee. Use the services and products of a trusted brand (AWS, Azure, etc) with local hosting, management, and support, and keep the apps local.

It works for fast food, why not the cloud?

Technorati Tags: cloud,regional cloud,regional cloud providers,intercloud,context,data center,cloud computing,aws,azure,cloud franchise,application delivery,network,latency,vdi

DISCLAIMER: This is long and the opinions are mine.I’ve written a good bit here about the various ways Microsoft and Citrix overlap in the hypervisor space, ranging from topics like shared code base through competition for the desktop space. To me, these two players have always been the underdogs battling for the right to go head-to-head against VMware in the main enterprise (and now cloud) virtual data center event. I’ve long said here that I think Microsoft is in the best position to make that move, but to be honest, Citrix currently has better technology. In other words, Microsoft has a better strategic play, Citrix a better tactical play. The announcements that came of out Synergy last week prove that. Citrix knows what it’s doing and they know how to build virtualization products to compete with VMware.As has been asked many times before, here and elsewhere: What would happen…what would be the benefit to the market…if Microsoft were to acquire Citrix and merge the best strategy and tactical solutions into one? The idea and rumor has been around for a while, so why am I revisiting it today? Since these rumors first started to really circulate in September of 2008 (around VMworld) there’s been very little advancement from the Microsoft camp on Hyper-V, and a tremendous amount of advancements from Citrix and the Xen products. We’re also seeing a few cases where the two have opted to work together. Case in point: the Essentials family for managing XenServer and Hyper-V VMs and storage. Citrix has made some excellent headway in the VDC with product announcements this year; that’s the real reason to take another look at this idea.For better or for worse, Microsoft and Citrix are already collaborating, both individually and to an extent togeter, to go after VMware. In the grand scheme of things why continue to do that on their own when they can do it together, mount one single offensive with one single goal, and bring enough technology to actually make a dent in VMware’s VDC footprint? Join forces and all that 2 against 1 stuff. Let’s look at a few categories where this makes sense, where Microsoft acquiring Citrix technology would go head to head against VMware and actually have a chance of winning:

• Networking and Application Delivery: To me recent movement from Citrix in this space is the paramount camel’s straw/tipping point for why Microsoft should finally take the leap. Citrix’s application delivery product line, NetScaler, has been a good appliance-based product for Citrix. Not a market leader, but they’ve held their own against F5 and Cisco. They manage application delivery well enough. With the announcement last week of NetScaler VPX, their virtual appliance version of MPX, NetScaler has made the leap into software-based application delivery, ala Zeus. This is huge for the acquisition discussion. First of all it could bring networking and application delivery into Microsoft’s world, something they’ve avoided with Hyper-V to date. Customers use virtualization for applications and they need to deliver those applications outside their data center. Couple VPX with the new software switch Citrix announced to compete against Cisco’s Nexus 1000v and you have the critical missing pieces for application deliver via Hyper-V (as well as another angle for Microsoft to compete against Citrix). And then add in the Citrix desktop and access-related apps for the non-MS platforms, like the iPhone, and Microsoft makes a huge push owning the application delivery stack from the VDC to the client, any client.
• VDI: Citrix has done an amazing job on virtualization geared towards the client. Going back to Metaframe and Presentation Server and then today with the work they’re doing with Xen on client virtualization, Citrix has always been focused on the client. Ironically, even though Microsoft is the de facto enterprise desktop client (in a sense), it hasn’t addressed the client virtualization markets too well. App-V is a step forward, but MED-V (with desktop virtualization code based on Virtual PC rather than Hyper-V) is a step back. VMware is making a huge push in this market with VMware View; if any player is going to win the VDC space completely they have to include a VDI solution, one that works locally and remotely, in their portfolio. Citrix could help Microsoft make that push by combining their respective solutions for hypervisor and application virtualization technologies. Many of the enterprise desktops and apps are Microsoft; the underlying technology running those desktops and apps in the data center and over the network are Citrix.
• Cloud Platforms/Providers: Xen owns a good bit of real estate in cloud and service provider data centers. Although Microsoft has good presence with customers running Windows operating systems, it doesn’t have the same exposure for Hyper-V as a platform that VMware and Xen have. I think MS is looking to change this with Azure but it will still be limited to the MS-only solution (for the short term anyway). Acquiring Citrix would give Microsoft that cloud provider mindshare by name alone. They could then take that business and technology model that Xen has built and create a best of breed service provider platform between Xen and Hyper-V for customers that want to run non-Windows apps on Xen and .Net-based apps on Hyper-V. This could drastically help Microsoft’s Oslo application lifecycle plan moving forward with cloud providers while not alienating non-.Net apps.
• Application Virtualization: As you know, I’m a huge fan of a true application virtualization model, something that I believe App-V will ultimately be able to deliver. However it will most likely be focused on .Net and Microsoft apps only and is still a few years away from full delivery and even more from adoption. In the mean time we have this bridging technology between VDI, client virtualization, and streaming apps. VMware is getting there with tools like View and ThinApp, but Citrix is staying in lockstep. Microsoft could use a Citrix acquisition to springboard App-V into a multi-focused application delivery platform, taking what’s good today with streaming apps and client virtualization and continue to work on true application virtualization for all apps.
• Customer/Device Support: And as a roll-up benefit of the above categories, we have application delivery to devices. I don’t want to place too much emphasis on supporting remote access via the iPhone, but when you look at Microsoft’s historic relationships with Apple and Linux (as a whole), of which Citrix has obvious ties into both now, that’s an appealing way for Microsoft to jump right into those groups. That doesn’t mean they’ll keep the momentum alive, but at least it would give them more opportunity than they have today. The overlap between VDI, XenApp, secure remote access, and the iPhone is an extremely appealing proposition for mobile users; a turn-key solution for Microsoft to cover a huge gap in their overall cloud and virtualization offerings.

And let’s be honest: Microsoft has had some challenges with their virtualization solutions and their overall direction. Client virtualization based on Virtual PC and no enterprise VDI solution? Hyper-V management hiccups through SCVMM/SCOM and delaying live migration for so long? Azure wanting to change the way applications run and are written on-premise? These raise questions in my mind, a lot of “Why?” questions. Citrix, on the other hand, is heading squarely in the right directly for virtualization solutions.  Citrix continues to plow ahead against VMware at a good pace, whereas Hyper-V isn’t quite at that same pace. The virtual switch announcement from Synergy last week is an excellent example; we haven’t seen any movement or advancements on virtual switching or networking for Hyper-V at all. Sophisticated virtual networking and switching management is an absolute critical component for virtual and cloud-based platforms, IMO. Moving internal roles and tasks to VMs running on the platforms is something we’ve seen for a while with VMware, even going so far as to running the full version of ESX 4.0 in a VM on top of ESXi 4.0. Citrix is doing the same with their Dazzle product. In other words both VMware and Citrix are finding optimized ways to use their own technology for their own benefit. We’re not seeing this today from Hyper-V. Again, there’s nothing to say that Microsoft acquiring Citrix would change that, but at least it might help grease the skids a bit towards internal product unification. Citrix knows how to do it well.To be clear, I am not being critical of Microsoft technologies or business practices (as any long-time readers of my blog will undoubtedly know). I am suggesting that when compared on a chart, Citrix is closer today to where the market and VMware are going for virtual platforms, and if the goal is to compete with VMware for both enterprise and cloud virtual platforms then Microsoft could benefit in leaps and bounds by acquiring Citrix for both Xen and their networking products. Microsoft would get virtual platform, application, and networking tools that they don’t have today.I’ll leave you with one final thought on how compelling a Microsoft/Citrix acquisition could be: Imagine a year from now if Azure launched out of beta running on both Xen and Hyper-V. This would be the best of both worlds: Microsoft could continue to push it’s current developer-based approach to Azure, SaaS, and application cloud computing, focusing on .Net and helping to push users to re-write their current and new apps. They could also support non-.Net customers by allowing them to run their services on Xen in Microsoft’s cloud. Customers wouldn’t have to choose based on their app needs. That would be the ultimate competitor to both Google and Amazon for cloud mindshare, bridging the two cloud models together and backed by the Microsoft brand.  Awesome. Will we ever see it? I hope so for market and customer needs.“Wish You Were Here” Image © 1975 EMI, Storm Thorgerson

I’ve been heads-down for the past few weeks in product land so I haven’t had much time to poke my head up into the clouds. I’m now standing up straight again, stretching, and getting back to thinking about the larger macro issues that cloud presents. Today I’m going to talk about one very specific issue: overlapping IP addresses and spaces by users of a common cloud.

This topic was spawned by a comment to an earlier post on the cloud. Here’s the comment:

In the Virtual Data center deployments/cloud, do you see a chance of IP address clash with the subscribers? Subscribers would want to have freedom to select the IP addresses. In such cases, how the network setup will be?

Ravi, the commenter, brings up a great point. The cloud is shared computing space so that’s got to include the network, and unlike computing resources and other parts of the network, such as MAC addresses, which are always unique until exhausted, IP addresses are such a small, finite pool of unique identifiers they have to overlapp at some point. We’ve seen this for years with companies that acquire other companies and attempt to combine and merge two different networks that share overlapping private IP addresses. And on a much smaller scale with home router users that end up using default IP addresses, typically either 192.168.0/24 or 192.168.1/24; basically any of them if they don’t change the default Linksys/Netgear/D-Link setting.

Like home users, the concern with share IP addresses in the cloud isn’t with public IP addresses; those are governed and for the most part rigid. The problem is inside the cloud, where IPv4 cloud providers are limited to one of three banks of private, non-routable IP addresses: 10/8, 172.16/12, and 192.168/16. On their own, this range encompasses almost 18 million IPv4 addresses, which will be plenty for most cloud providers. However there are two issues for pause, even with a pool of 18 million IPs to choose from:

1. Extremely large cloud providers, such as AWS. It’s not unrealistic to imagine that Amazon could support at some point more than 18 million unique workloads at once (in this case workloads meaning applications tethered to a unique IP address). Even as they were to get close to that number, management of re-usable IP addresses would be a challenge. They would have to know by the second which workload was using which IP address, when that workload relenqueshes that IP address, re-assigning the IP, and then the technical challenge of ARP management with such high-volume IP turnover.
2. Cloud providers that accept pre-configured workloads, say from internal private clouds. Company A may have a series of applications, bundled together in something like VMware’s vApp, that all use IP addresses in 192.168.22/24. Company B may have an vApp bundle with an overlapping range, such as 192.168.20/22. Given a large enough customer pool using vApp in an external cloud provider, there will be customers that use overlapping IP addresses on the same cloud network, and these customer IP ranges most likely won’t fall on easily defined subnet borders.

So what’s the solution (ruling out IPv6 since Ravi is probably asking about IPv4)? Thankfully there are a number of tools available that will virtualize networks above layer 2, virtualizing the IP space in the same way that VLANs virtualize frames. To address this issue, a cloud provider can use some network management device that will virtualize layer 3 before or at the same time customer traffic in the cloud is translated (via NAT) from a public IP address to a private IP address. As a request for a service in the cloud comes into the cloud edge, the network or application management tool will recognize the request for a specific customer and move that request over to the segmented layer 3 network isolated for that specific customer. Very much in the same way we tag frames and ports with VLAN data, a segmented layer 3 network can be created with “tagged” IP addresses that are destined for different parts of the network.

Ravi hit a core requirement for the cloud: in a true multi-tenant environment, overlapping IP addresses must be supported and cloud customers must be able to select any IP address they’d like, either during service creation or as part of a pre-configured virtual application bundle. Customers need to be able to say “This application must use this IP address” and not have to worry about the cloud provider returning a “Sorry, IP address already in use” error. So to answer his question, yes, I absolutely see a need for overlapping IP addresses inside the cloud, but as long as the cloud is built to support that architecture, overlapping IP addresses shouldn’t be a problem for cloud customers.

As reported by Practical Technology yesterday, Citrix has announced that it will be giving away XenServer (the hypervisor portion of their acquired Xen solution) for free. Very similar to the VMware model, they will continue to charge for management and advanced features as add-ons to the XenServer platform but the core virtualization technology will be available for free. Will this have any impact in their market share or their stake in the virtual data center? I’m skeptical for a number of reasons:

1. Free doesn’t always mean better, and usually it’s the exact opposite. I’m reluctant to think that enterprise customers and mission-critical cloud providers will take the free route solely to save money. Maybe that’s more likely this year than last but I’m still not sold it will make that much of an impact, especially since this won’t be an OSS offering ala XenSource. They’re also competing up-stream against VMware’s ESXi and Hyper-V, both of which have value-add above and beyond the hypervisor. For Citrix to make this move work they’re going to have to prove that they have the same type of value-add with XenServer that the other two big players offer. And free doesn’t count as value where your mission critical apps are concerned.
2. They’re relying heavily on management solutions to bring in the money for XenServer customers. We all know you can’t deploy a truly reliable virtualization solution without having management. In practice free deployments of XenServer will be few and far between, instead they will be bundled with some management solution. But which one? If MS System Center will manage both Hyper-V and XenServer, then we’re back to that value-add question for MS shops. If I’ve already deployed MS SCOM/SCVMM then what’s the value in deploying a Citrix solution instead of Hyper-V — which is also integrated into my server platform? VMware shops will stick with VMware because they’re the only company that has an end-to-end solution today. So is management the golden ticket for Citrix?
3. They’re announcing a new partnership with Microsoft. Excluding their existing partnership (whether that partnership has benefitted Citrix is a decently debatable topic so let’s gloss over those particulars for now) — Has a partnership with Microsoft between competing companies ever worked out for the non-Microsoft partner? Yes, MS is an excellent partner resource for technologies that don’t compete, but really, this partnership (and the ability to manage XenServer) will drastically benefit Microsoft by extending their heterogeneous virtual management solutions with Systems Center. I don’t see it going the other way for Citrix. Again, we’re going back to the value for a complete solution and who offers the most value here: Microsoft or Citrix? My vote is very strongly printed in the Microsoft column.

Maybe this will work in Citrix’s favor by at least getting XenServer in more hands to play with, and maybe their technology value will bubble to the top. But how many times have we watched a better technology (assuming XenServer is better) go by the wayside because they didn’t offer enough business value? In this case, I question whether a free XenServer offers any business value above and beyond what Hyper-V or VMware ESXi already offer.

Citrix has announced that they’ll be offering an alternative to traditional on-line VDI deployments (ala [Xen|Presentation]Server) in the form of a client hypervisor. In essences, a client hypervisor allows a user to run a full-blown virtual machine on their desktop — basically a throw-back to running VMware Workstation/Player in the enterprise before we had all these fancy virtual platforms like Virtual Infrastructure and Hyper-V.

Functionally, I’m a fan of this model: it allows users to work anywhere without concern about having an uplink. On planes, in coffee shops where there’s no Wifi or the connection is too small to reliably delivery a VDI experience, or even in the woods (solor panels for power optional). It also paves the way to true application virtualization, where the CPU-based hypervisor can be trimmed down to support running applications directly without the bloated guest operating system.

But this un-tethered model also brings up concerns, specifically around management. IT departments are going to have to face tough questions before deploying this type of technology. For example:

• How to deliver client VMs to users, especially remote users? Off-line via media or online?
• Change control: Will users be allowed to do anything with these images and run them like their own laptops? Will there be any type of reconciliation between the remote VM and a central policy source? How often?
• Mobility: Will users be able to transport these VMs between machines? Or will the VMs be bound to a single piece of hardware? If the latter, how will this be implemented and regulated?
• How are these VMs backed up and kept in state? What if the user drops the host (ie laptop) off the side of a ferry while finalizing her critical presentation just minutes before she’s scheduled to present (something a true VDI model would address)?
• In the case of remote users, what happens when the user “checks” their VM back into the corporate network when they’re at HQ?
• Remote access: Will these VMs be tied to a VPN solution? If so, how will that differ from how the host laptop/desktop connects?
• Policy management: How will tools like GPO apply to the host and guest simultaneously?
• Support: What happens when a user has a hypervisor-related issue? Will the support staff be trained to troubleshoot the host, the guest, and the hypervisor?

Many of these questions apply to any IT desktop policy, whether it’s a physical machine or a virtual machine. And to me, many of these issues beg the question “Are off-line client hypervisors worth the cost and effort?”  How does offering client VMs differ than just issuing everyone in the field a “normal” laptop?  In practice I’m inclined to agree with Kane Edupuganti on this issue:

We haven’t explored the idea of putting a [hypervisor] kernel on the devices, because you’d still need a regular PC.”

VDI solves many of the management problems listed above, but not all of them and also brings its own list of issues to the party (network, storage, advanced virtual platform management in the data center). There are security concerns for both models as well, but before we can address how to identify and solve security concerns, we need to be able to manage the environment first. If the goal is to make managing desktops easier and more streamlined, then I do think VDI is looking like a better alternative.

But I do think that the client hypervisor model is an important step towards pushing the computational tasks of running applications down to the client. That application hypervisor is getting closer, and steps to figure out how to manage local computing resources through a client hypervisor are all right by me. 

Douglas Gourlay wrote an excellent post yesterday over on Cisco’s Data Center blog about what doesn’t work in the current cloud model.  He had two great points. Point one:

And the most important point, about Cloud Computing is that it is in the cloud. and the cloud is the Internet.

This, to me, is one of the fundabmental problems with cloud computing that we need to address from the get-go, and why initiatives like Infrastructure 2.0 are catching on so quickly. We’re moving critical applications — and the back-end tools to mange them — out of our isolated, sadboxed, controllable enterprise data centers and on to the public internet. When someone in the office needs to access their newly-clouded sales app, they’re competing with upstream bandwidth against everyone else in the office catching up on YouTube and Twitter.

And the 2nd is in the comments:

Many people are astounded at the amount of data transmitted to support video, but that may be dwarfed by the movement of VMs.

Wow, that may be the best example to sum up the challenges with portable workloads I’ve read. Video streams content in a very deliberate way, video can buffer, video has bandwidth negotiation built in; copying straight VMs over TCP doesn’t, barring a technology on top of the connection for rate shaping. Pushing fully bundled vApps from vCloud to vCloud, with clusters of GB-sized VMDKs, across the public network is going to bring a whole new scale to “Start copy, get coffee.”  How these issues are dealt with from the DR and SLA perspective is going to be interesting.

Check out the rest of Doug’s post for the details. I’m exercising brevity on this cold Friday morning.

For the most part, I’ve avoided talking about the current state of security in the cloud.  For one, there are many, many smart people and great bloggers already talking about it and I love reading what they write (and almost always agree with it, almost ).  And two, I spend a good bit of my day job (and have for the past 3 years) focused on the ideas around contextual-based access and security: security has different meanings and requirements depending on the context of how a particular service is accessed or invoked.  Now that’s some cool stuff, and the cloud access model is a natural extension of contextual-based access. You’ll see much more on this idea from me over the next year-ish or so.

Over the holiday break, I spent a good bit of time thinking about how and when cloud security was really going to become a watercooler issue (while playing Super Mario World on the Wii Virtual Console, ironically enough). We’re nowhere near there yet - heck, we’re still discussing the nuances of the cloud itself; we have to build that before we can worry about who or what gets in. But there a few anecdotal things that kept coming back to me:

1. In a previous life, I used to audit the security of data centers, from the physical rack and cage installations (even testing the man gates, now that was fun) all the way down to the network and applications.  I even conducted interviews of all DC staff, both enterprise and hosting, on what they knew, how they responded in the event of a breach, where root passwords were kept, all that meaty security goodness. My goal was to verify that a particular application was installed in the most secure running environment available, primarily to limit legal exposure should there be a breach against this application.  What I took from that multi-year experience: It’s extremely expensive to conduct these types of audits, and at some point the liability baton is passed to the people actually implementing the technology, away from those who designed it.  I could interview people all day, and spend weeks walking through their network, but once I left the premises and filed my report, it was up to them to stick to those procedures. We had to trust (in our case legally) that what I saw remained in place.

2. As I’ve mentioned here before, I own a small photography company, so small in fact, I use a cloud provider (for lack of a better term) for all our business documents, including balance sheets (very basic stuff: sold 10 photos, bought a new lens, etc). Before the end of the year, I was spending a nice, cold, snowy Sunday in beautiful downtown Bellingham at a local coffee shop where, over their WEP-enabled wifi network, I was working on said balance sheet, getting ready for the fun that is tax season. Me being extremely security conscious, I immediately noticed that the cloud App provider doesn’t use SSL to access my docs, and in fact, says they don’t even offer SSL unless I pay a much higher premium. Now lucky for me simply changing the login URL manually to HTTPS worked fine and I was happily encrypted. But I’m guessing I’m in the minority and most of their users won’t notice that they’re accessing what could be sensitive documents in a completely insensitive manner, or that it could easily be forced to work. Most users who got to step two would read the Help and see they had to pay for that service and then do their own value assessment: is the risk worth paying extra?

3. Amazon’s lack of security notification in their AWS mid-December, well documented by Craig on CloudSecurity.  A weakness in Amazon’s signing algorithm opened up the possibility for more-frequent-than-normal cryptographic collisions (yep, the same core issue that’s currently plaguing SSL CA providers who use MD5). Amazon did fix the problem, but it took them 7.5 months and they kept it under wraps, a very common response in the security world.

So noodling on these three things while on vacation led me to this realization: Cloud services are going to create a new level of trust between end-users, providers, and enterprises. There’s no way around it, simply because there are so many moving pieces to a cloud service that are outside our control.  There’s nothing in the world I could have done during the 7.5 months that Amazon was fixing their signing flaw except change my transport to Amazon, and I probably wouldn’t have done that because the issue wasn’t public and I didn’t know about it. Once I start sending requests into the AWS, I have to trust that the systems in place are going to work and work securely.

Now that’s a big leap: one that I wouldn’t make with a more standard computing model. But in the cloud model, we have to trust so many new components in the stack. Of course we can have safeguards (SSL) and checks and balances (pen-tests, people who responsibly disclose security flaws) but at a minimum, those require near unfettered access to systems that are no longer in our control and require knowledgeable people to address them. In my auditing days I had unfettered access, during a specific window of time. Once I was done my access went away, so even then there was no continual checking of those systems.

So moving forward, as the security people tear apart the (in)security of cloud computing, the rest of the world will just need to take that leap of trust. A lowering of our standards for what we can control in the cloud’s outsourced data model. As an end user it kills me, but I know I have to make those sacrifices if I want to use those services. So I have to modify my level of trust, and apply new and stronger safeguards to the rest of my workflow processes (personal and professional) to make sure I’m able to recover if/when there is a massive breach that’s beyond my control.  My recovery is something I can control, and I definitely trust myself.

Ok, we have to talk about Hoff’s comments on VM sprawl, and it’s lack of security severity, for a second.  Now I’m guessing that he’s more focused on picking at the wound to spark conversation, and driving an excellent discussion on terminology, than he is in actually suggesting VM sprawl isn’t a security threat. But I’ll bite for a minute, and expand on my comments to his original post.

Yes, I do believe VM sprawl is a very serious threat, specifically due to what he calls out: lack of management. But it’s more than just a lack to me; I believe it’s an inability to even try to manage all VMs that is the problem. This same problem exists with physical servers, too. If someone can get access to the data center, bring in a box, find a switch port and know how that network is configured, then she can install a rogue physical server. But the barrier is so, so much lower with VMs because everything is “flat” once it’s virtualized. Our management choices today, such as vCenter, are doing a good job at creating virtual security barriers analogous to a keycard on the data center door, but we’re not quite there yet.

This is like managing termites. You can do everything in your power to protect your house from those little invaders by managing your perimeter and trying your best to find where they’re coming in, but if your builder “accidentally” left an exposed beam in your root cellar then built concrete blocks around it, it’s going to be very difficult to find that point of entry. Just like rogue VMs. It’s going to be extremely difficult to stop new VMs from spinning up at every point in your network.  There are ways to do this on the network level in the data center with IP registration, passive packet sniffing the virtual switches, 100% control over application traffic routing, etc.  But they’re not full-proof, and it becomes much more difficult in the desktop world.

Example: I use VMs for work on a daily basis. When I travel for work, those VMs come with me on my laptop. Inevitably, when I fire up my laptop and VMs on the hotel network and go through the “pay to play” registration process, they want to charge me for both my laptop and my VM. On a basic level, this makes sense: they charge and auth based on MAC address, I typically run my VMs in bridge mode, they ARP their own MAC over the NIC attached to the hotel network, cha-ching!.  But it’s an easy fix that I forget about until I see the 2nd $19.95 charge for the same day appear on my bill: NAT mode. Hiding the VM behind the MAC that the hotel is already aware of saves me from calling the front desk.

This is a classic example of where VM sprawl is such a threat. I could easily set up 100s of VMs on their hotel network and only pay for one access connection, and that would have no insight into what I was doing on their network with my VM farm (assuming I’m doing something malicious outside of their network and an encapsulating in SSL). Sure, it would be slow as molasses, but I’m not concerned with speed, I want power. 

This is the same threat that enterprises are currently facing with VM sprawl.  The specific threat is the ease at which users can spin up VMs: they don’t even need IP addresses, just a terse understanding of NAT. And for the most part, I don’t believe the threat of rogue VMs comes from sending traffic TO the rogue VM anyway. It comes from what that rogue VM is sending OUT on the network.  What is it spewing across a corporate or production network?  How hard is it to track and shut down if it’s NAT’d? Even if it’s only up for 5 minutes, that’s enough. VDI Bombs only take seconds.

So as always, I thank Hoff for posing an excellent question and for the lively discussion on Twitter between him, myself, and Lori. I do agree that a proplery managed VM infrastructure, with tools like vCenter and 3rd party VM network monitoring tools drastically help stave off full-blown VM sprawl. But it’s the inability of these solutions to monitor everything that scares me the most.  But if nothing, Hoff knows how to pick at the market’s shared security wounds. 

I just read Andreas’ post about “virtual routing” and moving VMs to match and optimize traffic flows (yes, it’s a few days old, but catching up on email and blogs takes time after vacation) and I’ll have to say my head started spinning – and not the good kind of spinning. The basic suggestion is that because we can move VMs around as needed we should move them constantly to match traffic patterns and destinations on the network. As traffic patterns and routes change, we move VMs to follow the routes Really? You want to, in essence, move data centers to follow traffic patterns over the same network that’s supporting said traffic, constantly?

Let’s look at this from another perspective: imagine you own a local chain of gas stations. These gas stations provide services for users (gas, food, coffee, etc) as they travel along various highways. They come to you when they need something, said service, otherwise they drive right past you. Your gas stations are located along major interstates at various exits as well as smaller highways and byways throughout the state. Your business accelerates during high-peak travel times (rush hour, weekends, vacation periods) and drops off during slow times (2:00 AM, Christmas, etc). In this slower economy, you’re looking for ways to maximize your revenue and you’ve noticed an interesting trend: drivers are staying closer to home on the weekends and shopping local, avoiding long travel trips in order to save money. You typically don’t have gas stations on local 2-lane roads and you can’t afford to open more stations, so you’re faced with two options:

1. Move your gas stations from the highways to local 2-lane roads: Sure, it’s an option, but you would be sacrificing valuable peek highway revenue.
2. Drive traffic from the towns to the highway and to your gas stations: Using signs and promotions such as “Free Coffee with Fill-Up!” - “2 Hot Dog Burritos for $1 with Large Coke!” - “5¢ Gas Discount for Local Residents!” - you can somewhat control how people find your gas station. You can control traffic flow to your service station.

What Andreas (along with Doug) is suggesting is a morphing of #1: Moving the gas station 2 times/day to accommodate traffic patterns and ignoring how and why traffic is driven to your station. Gas station is on exit #421 during morning and evening rush hour and then is moved to the intersection of Maple & Elm during off-peak hours. There’s no care or concern about the traffic patterns, the station is just following the flow.

This doesn’t resonate with me on many levels. First off, the basic principle of moving a server to match the traffic rather than managing traffic to the server. Traffic is smaller and easier to manage than servers, even with virtual servers. Managing constant VM movement with something like vCenter, clusters, resource pools, etc, is a surmountable task. Second, the process of moving the VMs themselves will have a cost wrt management, downtime, bandwidth (in the event Storage vMotion is used to follow the traffic). As an example, a quote from the post:

“VMotioning nodes (servers) to optimize the flow of traffic on the network”

Hmmm…How will moving a server optimize the traffic flow to that server? Again back to our gas station example: moving the gas station won’t change the traffic patterns at all. And are we talking LAN or WAN? If we’re talking LAN, then there should be traffic management devices in place already to manage optimized access to the apps running on the VMs. If you’re talking WAN, moving VMs cross-WAN on the fly has its own set of issues with traffic and storage.

Your goal when pulling into a gas station is to request a service: gas, coffee, food.  Moving the gas station just addresses cars driving into the parking lot. As always, we should be focused on the service being delivered in addition to the service station.

Don’t get me wrong: the mobility is one of the coolest benefits of VMs, and tools like vMotion and Storage vMotion are definitely changing the way we think about how and where we locate servers. But moving the servers instead of managing the traffic just because we can ain’t the way to go, IMO. Build the platforms for the services, build the infrastructure for the services, and then manage the traffic and access to the services. We have ADCs, let’s use them and manage the traffic and not micro-manage the service stations. And this quote:

“…solve for least switch hops per flow”

I think there’s a whole ‘nother blog post on how this idea becomes moot with something like DVS across multiple physical platforms. But I’ll take a break for now.

It’s been a while; content has been calling my name. I recently wrote an article for Virtualization Magazine titled Security Implications of Virtualization Platforms in the Virtual Data Center (I know, a crazy long title, but that’s how it was published ). WARNING: That page auto-launches flash video with audio enabled – gave me a bit of a pause when I heard some guy talking and interrupting my current playlist.

I like this piece because I introduce three concrete steps that IT departments can take today to help guard against security attacks tomorrow. These aren’t necessarily revolutionary ideas but they are tangible, tactical steps that can be implemented today during the planning, architecture, and roll-out phases of virtual platform installations and migrations. From the article:

In general, IT departments should focus on three virtualization areas as part of their entire virtualization security architecture:

1. Segmentation of VMs by location
2. Segmentation of VMs by service type
3. Proactive security management throughout the VM lifecycle

These three areas will help IT departments protect their virtual infrastructure against current threats as well as help mitigate the threat of future attacks.

You all know me: I’m all about baby steps and management. I don’t talk about specific threat vectors in this piece intentionally. Right now I’m very much in the design and planning stages of virtsec for IT departments. Remember my 4Ds: Define, Design, Develop, Deploy? This piece is all about the first two: know your risks and design an architecture that be used to manage those risks. Sure, if you’re keeping score, the 3 solutions above are actually in the Deploy category but I want to emphasize the planning portions of those solutions.

Start by planning today for whatever the virtsec world will throw at you tomorrow.