The Virtual Data Center

There have been two excellent analyst reports that have come out over the past few months that cover security concerns that arise from implementing virtualization in the data center; one from Burton and one from Yankee. This is in contrast to the numerous reports and articles that are available about how to secure your VDI, which is something completely different, and even one or two that suggest virtualization security isn’t an issue (Mr. Monthly Naysaying Nancy IT magazine editor, you know I’m talking about your OpEd piece mid-late 2007). Although they aren’t free, I highly recommend both reports as starting points for thinking about how implementing virtualization in your enterprise DC doesn’t come without concerns and risks, and probably introduces new ones you haven’t (yet) considered.

Both reports, however, tend to gloss over the security risks of implementing a software switching fabric, and pushing all your virtual application data through that switch fabric. I think this is an interesting omission, given that there is so much security built around physical networks in the DC. There’s a ton of research going on right now wrt poking through or infecting the hypervisor, and VDI patch management. But why is there less public research and data available for virtual networking security than there is with hypervisor security? Just on a code level, the network drivers used in some VDIs are open-source; the hypervisor code isn’t.

So why the lack of focus and research? I don’t have an answer.

Over the past few weeks, my extremely security-minded co-workers and I have been discussing the likenesses and differences of physical vs. virtual security. House key vs. HD controller password; smart card vs. passphrase. Geeky stuff like that. The last entry into this discussion was digital safe rooms. Now I’ve always wanted a real safe room. Even way back when Red Dawn used to be my favorite movie and I wore Air Jordans, I wanted a safe room. In fact I actually _like_ that Jodi Foster movie just b/c it’s about a safe room (story, plot, and acting be damned!). If I were rich, I wouldn’t have a mansion with an elevator. I would have a very discreet house in a neighborhood with secret corridors (ala Webster’s grandfather clock) leading to the safe room.

That said, I would have a real problem trusting something classified as a digital safe room with “mission critical” content. Once something is digital, IMO, it’s never safe or secure and it will live forever somewhere; the barrier to entry for physical reproduction is removed. That’s what I never understood from my DRM days and the customers that would ask “How can you guarantee this won’t be cracked?” You can’t. I would tell them “You release an album in a digital format and it will be cracked and shared. It’s the nature of humanity. The question becomes does this DRM format create a large enough barrier to entry to keep you losses to a minimum, because you will have losses.” DRM, while expensive for what you get, is great for stopping the majority of the population from stealing digital content. It will never stop or block people who are determined to steal.

Likewise, a digital safe room will never be completely safe b/c it’s still just a series of bits that can easily be reproduced somewhere. Just like any house with physical locks and a security system; it might stop your neighbors from walking in and eating your food while you’re on vacation, but it won’t stop a determined thief.

I read Shamus McGillicuddy’s discussion on Server Consolidation and I am blown away about the idea of enterprises still having servers under desks and in accessible locations due to physical lock down of buildings! Haven’t those guys heard of remote access? Ok, I’m being factitious, but if they don’t have a systems management server that can verify system health and configuration, can’t they ssh/rdp/telnet/something to those remote boxes?  Ok, even if the admins had to go to the server for administration why can’t they get a key?  

I’ve been running Vista since day 2 after it’s public release, and been using BitLocker since day 3. It’s mostly been a successful pairing (resuming from encrypted hibernate doesn’t work, encrypted sleep does). BitLocker is an excellent enterprise security too, especially when coupled with other tools like HD controller passwords.

I just read Ben’s post over on the Virtual PC Guy blog. Interesting idea, although it does put the fear in me. One of the key (pun intended) benefits of BitLocker is that the encrypted drive isn’t portable. Pull the drive and it won’t boot. Virtual machine images are highly portable. I know what you’re thinking: “Alan, you still need the key for the image.” True. But think about this in an enterprise environment, where the Vista images may be stored on a SAN along with other images belonging to other people. So access to the “drive” in this case is much easier than stealing a laptop. And they keys? Well, what admin doesn’t keep a stack of floppies (or more likely today USB keys) in their desk drawer. This level of security is no better than writing “Pencil” on a piece of paper in the Principal’s office (google it). And even worse, if the floppy is required to boot the Vista image, it’s going to stay in the floppy drive of image host, probably along with the rest of the BitLocker keys used for the other encrypted Vista images.  If you can clone the Vista image on the same host, then you should be able to boot it against the same floppy, decrypt, disable BitLocker, then you’re done.

But let’s think beyond stealing floppies and look at shared RAM. BitLocker has to keep the keys in some form of memory. If an attacker has access to enough identical Vista images that both use BitLocker and don’t, seems finding the key in RAM on the host would be a matter of binary comparison (and yes, a ton of time, but we all know that attackers have nothing but time and energy). Or maybe an attacker goes after the page file and couples this type of attack with Joanna Rutkowska’s Blue Pill.

I don’t know that running BitLocker in a virtual environment is a good thing, and may very well be a Very Bad Thing.

…and yes Betsy, they are most definitely after me.

The people in Second Life, they aren’t worth reaching. It’s just a weird place. It’s never gonna catch on. It’s a fad, not a fashion at all. — Mark Hughes

You know, I’ll be honest: I don’t get Second Life. I don’t criticize or have any issues with people that do get it, it’s just not my bag. But beyond what’s my thing and what’s your thing, I can’t even wrap my brain around fully immersing myself in an alternate reality that involves the same characteristics of my primary reality.

The SL themes are still terra-based themes that I live every day. My real life (”RL”), with a real house with real furniture, a real family, a real dog, is enough to keep me easily putting in 18 hour days. Why would I want to have 2 of those? When would I sleep? Does sitting in the park in SL constitute relaxing and unplugging in my RL? Should I spend my free time mending my SL fence instead of patching the gate on my RL fence?

I would prefer my virtual realities to be Star Trek Holodeck-style. If I’ve got time to kill and want to relax, I want to be able to say:

“Computa’: Earth, United States, San Francisco, California, 1941, A.D.”

…and I’m completely immersed. I certainly don’t want to relax by interacting with a monitor all night after doing it in my “RL” all day.

Speaking of complete immersions, why was there no “Red Light” Holodeck on the Enterprise? After years and years in space and the isolation stress, that’s all anyone would use the holodeck for anyway. So why didn’t they just program a permanent one that could be cleaned and scrubbed down every day? It’s more efficient. And it already has a built-in safe word: Exit.

And for that matter, why weren’t all living quarters just holodecks? A crew member could live in any environment they chose, and their physical living space could be no larger than a broom closet. In fact, they could have entire families and virtual lives off of the ship in their own quarters. Clock out of work, open the front door, and you’re “home”, living in a nice mountain cabin in the Cascades.

It would be like living Second Life “for real”; you would basically walk right into your monitor. Would it be called Holo-Life? Yes, the pun is intentional.

-Alan

FULL DISCLOSURE: This blog runs on software from wordpress.org, which is completely different than the software at wordpress.com that offended me below.

Prior to becoming the management and virtualization junkie that you read here, I worked in data security. I was a security freak. No, strike that: I still am a security freak. I’m overly paranoid. I shred everything. I only use one credit card for online shopping. I get in arguments with my family when I find out that they use weak passwords. We don’t even say passwords out loud in my house, it’s that bad. I have passwords for securing other passwords in password “safes”. I keep my GPG keys on write-once media in a safety deposit box. Yes, I’m a freak when it comes to security.

So you can imagine my shock and horror a few days ago when I created a WordPress blog. I thought it was about time I had a personal, non-technology related blog, so I created both WordPress and Blogspot sites to compare features and usability. So I did my thing, created blog/site names for each service, matched those to an account name, and waited for my respective confirmation emails. With Blogspot, it’s tied to one of my GMail accounts, so no problems there. Google’s SSO Dashboard environment is awesome (btw, I’m addicted to iGoogle).

WordPress has an email confirmation system, notifying me when my account was set up and ready to go. However…and it’s a HUGE HOWEVER…the confirmation email from WordPress included my username, links to manage my blog, and MY PASSWORD IN THE CLEAR! THE FULL PASSWORD…IN THE CLEAR! If I seem aggrivated, well, welcome to my world. It’s freakin’ 2008, and a site as pervasive as WordPress is sending full passwords in the clear, via the most insecure data transport system ever devised, email?! C’mon!

I promptly logged into WordPress and changed my password, and will most likely end up at Blogspot b/c of this snafu. So kudos to WordPress for making my decision much easier.

And for your viewing pleasure, here’s the email from WordPress (pertinent user information changed, obviously).

New WordPress Blog: ExampleBlog
Your New WordPress.com blog has been successfully set up.

You can log in with the following information:
Username: exampleblog
Password: alice123
at http://wordpress.com

We hope you dig your new weblog. If you have any questions or comments, please let us know!

In part 1, I waxed poetic about the too restrictive way virtualization is classified and the too loose way the term is used. For part 2, my plan was to expound on the problems with using the term virtualization for “virtually” everything. However, you’re smart, busy people, your time is at a premium, so I’ll just point you to the punchline. While most of the market is jumping straight through to OS and application virtualization, the systems that run those virtual instances, and germane to this blog, the Virtual Data Center as a concept and entity is “virtually” forgotten (I just love using “virtually” in a post about how “virtually” is over-used… ;). So to address this problem head-on from the 4Ds (Define, Design, Develop, Deploy; but more on that some other time), we first need to Define exactly how virtualization is broken up into the most basic technology definitions. You can’t solve a problem without first defining it.

And luckily for you, I’ve already addressed this in a paper titled Virtualization Defined: Eight Different Ways. So when I talk about specific virtualization technologies, those technologies will fall into one of the 8 unique definitions. For example, VLANs->Network; Service Provider Customer Partitions->Management for access, ->Hardware for implementation; Hypervisor->OS; you get the idea. Now we need to working on helping the market adopt these discrete categories so it can stop lumping everything into OS, storage, and application virtualization.

I love it when we don’t have to re-invent the wheel.

-Alan

I was thinking today about how to represent server workloads to a monitoring system and for some reason was reminded of HP’s event last year of blowing up a data center.  If you were on vacation last summer and missed it, HP, to demonstrate controlled failover, blew up five racks of operational gear.  I still think it would have been better to donate all of that gear to some starving startup, or better yet, a school district! Anyway, they blow up all the running servers and we see a monitoring and control system start to turn up machines at a duplicate data center to restore all the running services. Today, I wonder if demonstrating fail over would be any different?  Would HP still have all those physical servers? What if you had several host machines all running on a hypervisor somewhere? Would we see something different in regard to monitoring the applications? What determines if a running application has to be duplicated to restore service levels anyway? Where does the abstraction of the server services take place and how are those things represented to the end users.  Maybe the folks at Netqos have it right with their representation of overworked servers catching on fire.   What I think is more realistic is a monitoring and control system that can watch application performance levels and start to spin up services before things blow up.  Ah, the rise of the machines.

I’ve long been an opponent of Second Life in general (more on that some other time), and this is an excellent example of why:

http://blog.secondlife.com/2008/01/08/new-policy-regarding-in-world-banks/

“As of January 22, 2008, it will be prohibited to offer interest or any direct return on an investment (whether in L$ or other currency) from any object, such as an ATM, located in Second Life, without proof of an applicable government registration statement or financial institution charter.”

So apparently you still have 5 days to exploit other SL citizens. Is this what SL teaches residents, that fraud is ok until the government says it’s not? And by the very loose usage of the term “government”, I’m referring to the oppressively and ominously named Linden Research, Inc.

Does SL and Linden remind anyone else of Delta City?

As I mentioned earlier, I’m overjoyed that the industry I’ve been living in for years is finally getting the respect and attention it deserves. But the overwhelming attention does come with some problems, namely (yes, pun intended) the footloose and fancy-free way in which people have been using the term “Virtualization.” Seems that people are using it to apply to just about anything these days, which unfortunately, is much more marketing driven than technology driven. “Have an IT term you want to make money on? Slap ‘Virtual’ in front of it and the money will roll in!”

I’m pretty sure that every IT magazine over the past 6 months has carried a feature story on some form of virtualization, and most of those stories have been the cover/lead stories. But almost without fail (at least with the ones that I read) the author made very broad assumptions about what Virtualization meant and which parts of the enterprise data center that particular definition covered. As an example, I read an article a few months ago that defined virtualization in just three categories: OS, Application, and Storage. That was it. Everything virtual had to map to one of those three, extremely limiting categories. Thanks for clearing up what virtualization means. Now I need to go figure out where my Virtual LAN (VLAN) tags fit in. Talk about squeezing a square peg into a round hole. Hey, maybe if it’s a virtual round hole we’ll be ok.

-Alan