Veeam Stories: My first Veeam encounter > ProVirtualzone

As promised, I will write some blog posts about Veeam in this Veeam Stories series. These are histories that I have from all the years I have worked with Veeam since they launched their v1.0 back in 2008.

In this first Veeam Stories: My first Veeam encounter, I will talk about the first time I used or propose a Veeam solution for a customer after a full disaster recovery.

Note: Before I start this history, of course, I will provide any names or hints that anyone could match the real one.

This history is, of course, almost in the “pre-historic” era of the IT and Cloud environments. Veeam officially launched its first version in 2008 and announcing v1.0 at the VMworld 2008. Veeam launched v2.0 in the middle of that year.

A roadmap from Veeam versions.

© Veeam

By that time, I was already doing some testing with this new fantastic Backup tool that everyone was talking about and using an old desktop with some local disks as a backup repository.

© Veeam

This was the time of VMware ESX 2.5 and 3.0 and in 2008 ESX 3.5. VMware Hypervisor was incredible technology back then, and running VMs(servers) inside another server was great. But there was a big problem in that Hypervisor environment, backups. There were some scripts, some small tools(mainly script base) where we backup our VMs. We had many problems most of the time, especially if you were using raw disks for Exchange and SQL Microsoft Clusters.

So most of the customers that used VMware did not care about VMs backups, and they just did what they were accustomed to do, backup the data inside the servers(in this case, VMs), and that was it. If there were a total loss of data in the Storage, they would lose everything in their virtual environment.

That was how companies and IT departments work because they always did in the past for physical servers. But we know it is not the same, and that was what we need to do? Inform, clarify, and explain that a Virtual Environment is not the same as a physical Environment and needs to be backed up differently.

So this story is base on a similar environment. It was a government department that managed all schools in the country, exams, classes, schedules everything. The environment was base on several Web Server, some Databases, base an Active Directory, some File Servers, and Exchange Server running in a Cluster(back then, running Exchange Cluster in ESX was a big struggle).

Honestly, I can’t remember how many VMs were, but it was around 15/20 VMs. Running on two ESX 2.5 or 3.0(can’t remember for sure). Storage was an HP StorageWorks providing some Fibre Channel LUNs.

I did pick up this new customer on my support tasks very recently. The customer was doing standard backups every day/week to tapes but only of the data from the Web Server, DBs, and File Servers files, nothing more. I provided full support to their IT team on VMware/Storage as an external consultant from a company with a support contract.

But as always I do, I was teaching some of the team members about ESX and how to use it, how to install, etc. Small tasks that they can do by themselves without the need of asking for support all the time.

We had discussions before about the Virtual Backups(and I was updating him about this excellent Backup Tool from a company called Veeam). Still, they decided(also because of budget restrictions) that they would continue to backup everything like before, with no need to use extra tools to backup additional data(in this case, VMs).

One day they wanted to add two extra ESX to the environment(they were migrated to better servers and I think moving to ESX 3.0), and they did not request support for this. We have talked about it, and the manager for IT said he wants to do it by himself and install the new servers(maybe saving some money in the next contract support).

Customer starts to install the two VMware ESX server on a Friday(end of the day) and then panic happen.

Back then, there was an option (still can’t remember if it was 2.5 or until 3) that if you had your fibre channel connections, ESX would recognize all the LUNs(that were set for the old ESX server and also for the new ones) on it and then on the install ask if you want to destroy all data. Even the option was enabled by default, so if you do something like next, next, next, all data in the Storage will be deleted.

Was something similar like this.

The customer continued the install in both servers(not checking what was happening in the production environment), and now he had wiped out all information from those LUNs and had a Full Data Loss.

As I said, this was on a Friday end of the day, a government department, so we had a huge problem here.

The customer called me in a panic, saying that he deleted all LUNs, and I was like: “no, you are wrong, maybe you are missing something”. Then he confirmed the option and answered yes to the delete, so since the ESXi was already connected to the fibre channel and seeing the used LUNs, all was destroyed.

Back then, recovering a deleted LUN was not the same as today, when we can more or less recover. Asking for backups, of course, he only confirmed that he only had backup of all data from the servers. My question then was, “How can you restore that data if now you don’t have any servers to restore the data? That data is pointless now without the VMs”.

I thought, “I need to help this guy”. If not, Monday, he will be fired, that is for sure. My only thinking is that I would have someone help me fix and recover the best we could do.

So I decided that I would help him to rebuild all the VMs, domain, Exchange, Web Servers, SQL Servers, everything. At least the most urgent ones that need to be running by Monday morning so that schools, teachers, students could work.

We spend Friday night, Saturday, Sunday until Monday morning building everything from scratch. Almost without sleep (one sleeps some hours while the other continue the work, then the other go to sleep).

After the servers were rebuilt, we could use then that useless data to restore DB, Exchange, File Servers, and some web servers data.

Monday morning, Active Directory and domain were recovered, Exchange was up, File Server and the most urgent Web Servers were up and online.

A couple of weeks later, after the storm passed, he managed to get the budget to buy Veeam licenses(my first Veeam sale), and we built a server with local disks. After that day, all VMs were backed up by this great tool to quickly backup and restore VMs.

To finalize the history, I received a call from the CEO to go to her office one day. Talking to the CEO, there was a letter from the customer appreciating me and thanking me for my work (during an entire weekend) to help him and save his “life”. I smile and only answer, “this means I will get a raise?” 🙂

Final comments:

As I said, back then, in the beginning, few customers did pay attention to backup VMs or any of the Virtual Infrastructure. They were not used to this, so for them(IT teams and management) was just extra work and extra expenses since they already had a tool to backup data.

Only sometime after I saw customers worry about this part, consulting companies were informing/teaching their customers that they were using Hypervisor about the new reality. After now, they need to think differently about backups.

Personally, even a couple of years later, I was still struggling with some customers to try to open their minds and explain to them what was the new reality and that they need to think about backup their virtual environment and always, always, use this example to show how a complete data loss was so easy and quick to fix by just doing a full restore of all the VMs.

Hypervisor and virtual environments were here, with all the advantages but also how to recover. A virtual recovery plan now needs to be added to the existing physical disaster recovery plan. It was a brand new day, was a brand new era.

So this was the first time I used Veeam in a customer(only after the disaster) and where Veeam did show what it was and what it could be in the future.

Thanks to spending some minutes reading this first Veeam Stories and this example of a disaster that could be prevented, at least minimized.

Share this article if you think it is worth sharing. If you have any questions or comments, comment here or contact me on Twitter.