Virtually Safe?

Are you using virtualization technologies in your network environment? You are? Great. Do you know why you are using virtualization?

Most reasons I hear, which are all very viable, are things like saving energy by running less physical hardware. Improved disaster recovery planning is always a good one. Tools like VMotion are invaluable to this. But there is, fundamentally, a very good technical reason to look at using virtualization, and that is the upper limit of your server CPU. We have hit it, and it's not going to get any bigger, better, badder. Problem? Not really, along comes Multi-Core, and now we have many more CPU's to work with. Which is great, if your application and operating system can take advantage of it. And it's not always as simple as 'rewrite the application for multicore', some application jobs have to be run in a certain order, or access to a specific piece of data that limits the ability of using multiple cores to translate into any kind of benefit. You can't just re-write the application, you need to think of new ways your applications can take advantage of multiple CPU's. Let's face it, your truly multi-core aware operating systems and applications are still a long ways away. But in the meantime, we have things like VMware.

VmWare is by no means the only virtualization option out there, but for the purpose of this article, it will be the focus of discussion. Plenty of other people can give you the gory details on what works best for your needs, the concepts behind the security of virtualization stay very much the same and is the focus of this discussion. What essentially all these virtualization platforms are, is exactly that, a hardware platform, delivered as software. Did you catch that? The software is your hardware platform, and what you better be looking for in a good virtualization platform is the ability of that platform to extrapolate and abstract the hardware, such that applications written for single core/single CPU systems, start to perform that much better when you deliver the hardware through software. The technical challenge it should fundamentally address, is your ability to scale up performance, regardless of the software, or for that matter, the network, since you collapse that into the virtualized platform as well.

And everything you did to secure your applications on the network is going to change. Everything? Yes, everything. I used to think it was all the same, and don't get me wrong, there are some fundamentals that will follow you through, but let me introduce you to the 3 Stages of Virtualization, that allot of people are going through. How fast you progress through them is very unique to each person adopting virtualization as a hardware strategy.

Stage 1: Check out my cool new toy
Ever project has a stage one. Kick the tires, check the engine. You know the drill. You need to figure out if this virtualization platform will really deliver and scale your critical applications. You won't really care about security at this stage. You will think about it, and probably call your vendors in for an intense discussion about virtualization security strategies. But you don't really need anything yet, so keep your wallet in your pocket.
You can tell if you are at this stage if you have a sectioned off part of your physical network just for your virtualization platforms. It will look kind of like a virtualization DMZ. You will most likely have your usual physical firewall and IPS/IDS in front of it, maybe even mapping a few vlans across to leverage some type of 'virtualization' of security interfaces, but you won't really trust it and we will hear terms like pre-prod or staging thrown around while you work out the bugs in your applications, and build your virtual machine deployment process.

Stage 2: It's all the same, but different somehow.
You will know when you are in this stage, when things start getting serious around security choices. You will have replicated everything you have done in the physical world, in your new virtual infrastructure, and you will have probably plugged a few virtual security devices in between things. And this will probably run very well for a while, but eventually a problem will emerge. By replicating your security zones you can end up reconstructing your layer 3 routed networks, insert traditional or virtualized versions of the IPS, Firewall, DLP, etc, and off we go, doing what we always have, what we know and love, and what requires the least thought and effort from security. Don't get me wrong, this is a great way to move into virtualization, while reducing the risk imposed by a redesign of network and policy. But it won't last, and I'll tell you why.
Beyond hardware abstraction for performance, there is the possibility of leveraging a software based hardware platform (a mouthful, I know, but I had to say it) for rapid recovery, deployment and network agility. But what happens when your VmWare team starts Vmotioning whole servers around your Layer 3, policy controlled network? I'm willing to bet your physical world security policy design never considered the possibility that servers could just be picked up and dropped anywhere else in the network. And forget security policy, what about plain old network access? What happens to your web and database servers, if the DNS server is just plucked up and moved away? They stop working, along with everything else in your virtual network.
Unless of course you have your network and security team, standing by, to provision via some mechanism (virtual or real), and ensure the servers land in an authorized and safe place. You still have allot of the same problems you did in the real world, but you know how to deal with those. There are whole products around maintaining networks like we did in the physical world, keeping the separation of duty and overall control, with the respective groups.

Stage 3: One step backward, then two steps forward.
You have to think of how we want a virtualized network to act. If we want to be able to move server applications around, without impact to the network, and from that build an endless flow of hardware scalability, as you add virtualization servers to the mix, you are going to have to build a radically different type of network. But it's not a new idea, it's a very old one, and an idea we thought of as very bad, not too long ago. Just build one big large Layer 2 network, and drop all your server apps in together.
At this point your network and security team just resigned over this idea with a letter that reads something along the lines of " . . . good luck with that." I'm all for understanding how others are dealing with the challenges today, but I don't think it is a design we can continue to ignore, if we re-evaluate our expectations about virtualization security.
And this is where you enter Stage 3, virtualization nirvana, but you have to change your expectation of a security solution. It should have all the features of your traditional security, but we need to work at the hardware layer of your software. Remember our software is now the hardware platform? You need to plug your security into the hypervisor. By inserting a solution that captures all traffic on all virtual ports, you can develop some unique security policies that are as dynamic as the nature of virtualization. What about having a policy that not only attaches to an application virtual machine, but follows that virtual machine as it moves around the various physical systems used to scale up at an elastic speed? It doesn't stop there. Take a fresh look at your old problems. Setting a default security policy so new machines are required to be authorized, monitored and audited when they come online. Are you worried about IP spoofing? Sure, but a simple interface policy across the hypervisor could take care of that. What about virtual machine spoofing? If someone replicates a virtual machine but injects something new into it, what policy will this virtual machine inherit?

Stage 3 is a new and exciting opportunity for us to fix some things at *(not really!) a basic level. It's almost a do over, and if you are going all the way, make sure you have considered all your options when it comes to virtualization security. I'm not saying I have all the answers, we all have allot to learn still about the new security challenges this presents. I only recently came into Stage 3 myself, and have been re-writing this blog over and over recently, because of that. I am also sure at some point we will be discussing the impending Stage 4. This is the stage we all get together in about 10 years to discuss how to get our applications and data out of large virtual data farms in the sky. But that is a story told many times before, and will be again. Take a macro look at computer evolution, we are moving back to the 'mainframe', and I suspect, will move away again one day.

Or maybe Stage 4 is us really taking a new look at the underlying protocols we are using. Time to strip the OS and reduce our virtual machines to one click applications. Dissolve the network and build a new system of access. It really could be a fresh start for everything.

*(I don't mean to imply that running and managing virtualization infrastructures are trivial at all. They are not, and I hate all this marketing of virtualization like it's somehow magically simple. When humans interact with these systems, expect some complex designs to be set in motion to support real world needs.)