The Virtual Data Center

For those who follow the music business, or for anyone that just reads and/or watches the news, you’ll know that 80’s metal icons Guns n’ Roses finally released the much anticipated “Chinese Democracy” album on Sunday, Nov. 23rd, 2008. I say finally because we’ve been waiting on it for years. Every year just before Christmas rumors began to fly about a possible release date that year. And this year was no different than other years; those of us that were actually looking forward to this release hedged our bets once again and said “Sure, I’ll believe it when I see it,” including Dr. Pepper. Who even knew the Dr. was a Guns fan?

See way back in March of ‘08, Dr. Pepper ran a campaign – if Chinese Democracy came out this year, anyone who wanted one would get a free 20oz bottle of Dr. Pepper. I’m assuming Axl took that as a throwdown challenge becuase he delivered, leaving the Dr. to start giving out free bottles. The Dr. came through on their end of the bargain and opened up their website for the free 20oz requests for 24 hours, only valid on the day Chinese Democracy hit shelves, Nov. 23rd. The official word came down about 2 months ago that indeed, Democracy would actually drop as expected (with distribution and promo details to back it up), leaving the great Dr. ~2 months to plan for the onslaught of freebie requests. If we know anything about humanity it’s that no one can say no to “Free.”

I’m not sure what Dr. Pepper was expecting with their offer, but it certainly didn’t plan for the flood of internet traffic that would be directed their way beginning at 12:01 AM EST on the 23rd. Within hours their site became unresponsive, going through phases of availability: first, the intro Flash got stuck with no way to skip past it; then they finally caught on and started redirecting all users to a text-only portion of the site where the “Free Dr. Pepper” link was buried way down at the bottom which directed to a page that wouldn’t load; and finally, the coup de grace, the entire site went down and was returning nothing more than a 503 Service Unavailable error. As of this writing, we’re still at 503, and I’m still sans a Dr.

This very temporary unplanned (well, they had two months to plan, but let’s stick with un- for a second) outage is a textbook case for elastic provisioning: spinning up services infinitely to address temporary need and then spinning those services back down when traffic returns to normal levels. It’s like cable or satellite Video on Demand requests spiking during a snow storm on a Friday night. Services that normally handle X amount of traffic are suddenly forced to handle exponentially larger volumes of traffic, typically unexpectedly, but they still need to deliver that service.

To give them the benefit of the doubt, maybe Dr. Pepper didn’t anticipate the massive traffic volumes brought on by every major news source in the country picking up on the Gn’R bet, but that’s the point of elastic provisioning: accommodating service need without anticipating fixed levels. All they needed was the ability to scale for 24 hours, then things could have been returned to normal. So what went wrong? How much is this free offer (which has now been extended 18 hours, although it doesn’t matter if the site is still down) going to ultimately cost them in marketing, name, and not so good press? I’m guessing less than it would have cost to spin up a temporary site with the free offer that could scale for 24 hours.

And this one hits home: I rarely drink soda today but do have fond memories of the Pepper; I was looking forward to throwing on some 80’s hair metal and kicking back like I was in 7th grade again. But the Dr. teased me; they offered and didn’t deliver. At least Axl finally did, but I’m not waiting 17 years for a free Dr. Pepper; I’ll stick with coffee.

It’s been a while; content has been calling my name. I recently wrote an article for Virtualization Magazine titled Security Implications of Virtualization Platforms in the Virtual Data Center (I know, a crazy long title, but that’s how it was published ). WARNING: That page auto-launches flash video with audio enabled – gave me a bit of a pause when I heard some guy talking and interrupting my current playlist.

I like this piece because I introduce three concrete steps that IT departments can take today to help guard against security attacks tomorrow. These aren’t necessarily revolutionary ideas but they are tangible, tactical steps that can be implemented today during the planning, architecture, and roll-out phases of virtual platform installations and migrations. From the article:

In general, IT departments should focus on three virtualization areas as part of their entire virtualization security architecture:

1. Segmentation of VMs by location
2. Segmentation of VMs by service type
3. Proactive security management throughout the VM lifecycle

These three areas will help IT departments protect their virtual infrastructure against current threats as well as help mitigate the threat of future attacks.

You all know me: I’m all about baby steps and management. I don’t talk about specific threat vectors in this piece intentionally. Right now I’m very much in the design and planning stages of virtsec for IT departments. Remember my 4Ds: Define, Design, Develop, Deploy? This piece is all about the first two: know your risks and design an architecture that be used to manage those risks. Sure, if you’re keeping score, the 3 solutions above are actually in the Deploy category but I want to emphasize the planning portions of those solutions.

Start by planning today for whatever the virtsec world will throw at you tomorrow.

I just read Rich Miller’s excellent blog post on sites scaling up for election traffic on Data Center Knowledge. As he points out in a post this morning, traffic hit record levels through Akamai’s CDN on election night. Some companies adequately planned for the burst, other didn’t. Spike management isn’t something new, however we do deal with massively larger amounts of traffic than we have in the past, and our traffic usage is different. An election that everyone is watching is an excellent case study for these new traffic patterns. Me, I was sitting in front of the TV last night with my laptop open to MSNBC and twitter, CNN mobile on the iPhone (primarily b/c I enjoyed seeing all those 404 and 500 errors that were showing up on CNN mobile; I know, I’m evil ). And I’m guessing this was the norm for people who use the Internet as their primary news source, like me. And the company responses that Rich covers, that did plan for the election spike and anticipated this flood of traffic, are interesting to me on two fronts:

1. The lack of the V-word: Surprisingly in this day and age, none of the companies interviewed said they were relying on any virtualization solutions to scale for their traffic. All the remedies involved physical servers and physical space in a data center or with a hosting company. But with all the hype (and b/c it’s all I think about all day), I expected to see something about VMs or virtual storage as part of their spike management plans. On one hand this is encouraging that yes, the world can still spin without VMware or Microsoft virtual platforms. On other other, though, the election should serve as a perfect use case for provisioning and scaling using tools like virtual machines. This election is the best example I can imagine for “elastic computing,” and I’m surprised that it wasn’t first in responses from these companies. The ability to provision up and de-provision down as need based on real-time, immediate traffic needs is the long term bread-and-butter for virtual platforms; companies like BlueLock and Joyent know this today and have built virtual hosting solutions around provisioning scale for both infrastructure and the applications. So why not use the virtual tools available today as part of your scaling and provisioning needs, rather than having to plan for a spike by pre-ordering batches of servers and waiting weeks for them to go online?
2. Focus on the Apps: I have to say it warms my heart anytime someone mentions applications in the data center — I’m a softie for those darned apps! All of the examples in his post were customers who were expecting an increased need for their application: a political blog, a CDN that hosts political websites, Twitter, etc. Their concern isn’t with scaling core infrastructure (switches, routers, cables, trunks, etc), it’s with scaling the application platforms (servers, OS’, webservers, etc); again, a pointer to those hosting providers who have already built out virtual infrastructure platforms to allow VM and application scale and provisioning as needed.

The phrase “old school” kept popping up in my head as I was reading the post. Are these companies sticking with what works, what’s tried and true, by provisioning physical servers well in advance of the expected spike? Or does this show that virtual platforms are still in their infancy and companies that know how to plan for and manage massive amounts of application data traffic don’t yet trust virtual solutions? I would probably lean somewhere in the middle, and until virtual platforms and dynamic provisioning proves itself, we’ll continue to see dynamic provisioning in the VDC as more of a test case rather than a real-world use case.

I’m a simple guy: I like to define things in tiny, digestible chunks and build up from those bite-sized morsels. I see a pie, take a bite, and think “This must be where pies go when they die” with every amazing chew (actually I’m more of a doughnut guy than a pie guy but it works for this analogy). I’m insanely lucky in that I work with people who typically think differently than I do: in this case they think about the whole pie and how the pie impacts the rest of the meal. I’d say we’re a good balance, except on those days where I’m so stubborned I can’t move out of the basic definition phase of a problem and we discuss the gap in thinking rather than the actual problem. “Why does the way this one bite tastes have any impact on my wine choice for appetizer?”

Well over the past two weeks I’ve hit two of those issues with virtualization: How do you define and categorize cloud computing and application virtualization? Today let’s stick to the clouds (it’s a murky, rainy day in Bellingham and I have a great view of the cloud party over Bellingham Bay, so why not keep it local).

Where I’m stuck is how do you define the cloud and classify a cloud service? Let’s start with an example: Gmail. Is Gmail a cloud service? If so, why? If not, why not? I’m not concerned yet about answering whether or not Gmail is a cloud service but instead HOW to decide either way and WHY. I have a short list of cloud qualifiers that I use to determine if something is a cloud service:

• It’s got to be a service that accepts input and returns output in the form of a workload, AND
• It has to have an integration overlap between the running service and the local “management” piece, typically via a transparent management component like an API, but I’m open to other implementations.

For me the above works really well to define larger services such as those from Service Providers, such as Amazon S3. But one area where my criteria falls short is when we start talking about end-users. Is the cloud consumer-able, meaning can do/would consumers every touch the cloud? I’m leaning towards no right now. But does it matter HOW I access and use Gmail as a consumer? That’s one of the criteria I’m looking at.

My co-workers and friends are 20 steps ahead of me in defining the cloud itself; they’re talking about the cloud with terms like existential and invoking Plato, don’t get me wrong, which is great. But that doesn’t work for me; I need to start way, way down at the bottom of everything and figure out how I’m going to apply a criteria to the cloud before I can discuss it’s philosophical place in life. Lori and others have suggested that the cloud comes down to how a service is delivered rather than what is delivered, and I’m fine with that as a long term definition or categorization. I don’t think it’s narrow enough, though, to answer the question “Is Gmail a cloud service?” because if Gmail is delivered over the web, my internet connection is my work infrastructure, so therefore…Gmail is a cloud service for me? As I’ve said before, just because I send a packet outside my house doesn’t mean I’m invoking a cloud service. So does it really come down to Gmail is only a cloud service if I use it as part of the cloud? Curse you, recursion!

So what I need is a Cloud Criteria: a method for evaluating any computing service or technology in the data center to determine if it’s a cloud service or cloud-able. Maybe a Cloud Algorithm: Pass in a technology and out pops the solution. But what is that algorithm? That’s just the way my tiny brain works: I need classifications and organization (you should see how organized my records and CDs are ). I’m not opposed to non-linear classifications, but I at least need that framework. And unfortunately with cloud computing and service I’m at a stand-still until I can get that order straight in my head. Thoughts? Feel like helping a brother out? What’s the check-box criteria for a service being part of the cloud?