Worldwide Distributor Dramatically Improves RTO/RPO with Dell AppAssure – from 12 hours to 15 Minutes

Note: although this worldwide distribution company has a strict policy against giving product testimonials, they willingly agreed to share their story, because of their great experience with Dell AppAssure Backup, Replication & Recovery, under the pseudonym, “WWDC”

Environmental Snapshot:

• Exchange 2007 mailboxes: 300 (two mailbox roll servers, four hub transport/client access servers
• SQL servers: 3
• Windows file servers: 4
• Virtualization: VMWare hosts (ESX3.5 and ESXi 4.x) connected to iSCSI SAN, plus VMWare’s Virtual Center. VM Guests: 25; Physical Servers: 30
• Total data: approximately 15 TB

Business Challenges

A subsidiary company of a European conglomerate, WWDC is a worldwide distribution company. As a global enterprise, their IT environment is complex, with both physical and virtual servers handling the ever-increasing data needs for the 250+ employees worldwide.

To deal with IT complexities, WWDC’s IT staff work in a team model allowing for both IT expertise as well as human redundancy – the company doesn’t just rely on one person to get its important IT work done. “In our team work, we try not to compartmentalize,” notes HK, the Network Administrator. “We don’t want staff to wait for someone in IT if they need something now, so we cross-train, do offline experimentation, and put things in test environments so that folks who don’t do things every day have the opportunity to know all aspects of IT. We want the transition from one person to another to be as transparent as possible. We have collaboration and redundancy.”

Unfortunately, their amazing redundancy and integrated IT team approach wasn’t an aspect of the backup and disaster recovery solution the company was using. “The ever-increasing backup window we experienced with Symantec’s Backup Exec was slowing down performance as the data increased,” remembers HK. “Not only that, but the in-house backup methods of only capturing some data and then taking it home on tape left us unprepared for a disaster.”

HK’s team did backups from 6:00pm to 6:00am every night. However, the team could only back up the changeable data and not the whole server. Periodically the team would use an imaging software for a full snap, but found that the hardware-specific issues made this process nearly impossible if they didn’t have the same box. “The backups started encroaching into the business day,” says HK. “There were times when we were forced to go into business hours because the backup was affecting performance on LAN, so we had to stop the backup. This meant that we then had to get creative and accommodate the increasing data. We would do backup only on critical things, and do the non-critical maybe every other day or so. But of course it then gets confusing because we weren’t always 100% sure when everything was backed up, and what was backed up.”

In addition to the backup windows, performance issues, and lack of backup, the team had some out-and-out failures. “With tape, there’s always a certain percentage of failure that we had to accept,” says HK. “That’s the nature of tapes and tape drives, but we weren’t happy with it. Ninety-six percent of the time, failures happened during the day and not really on full servers, but just regular issues. And when there was a problem, it took several people to fix it, so our staff time was affected. I really didn’t like that we couldn’t adequately respond to inquiries when data was lost or there was a failure.”

And then there was the insanely long recovery point objective (RPO). The company reluctantly was accepting an RPO of 24 hours or longer, even though there were far too many occasions when someone needed an important file or email that was deleted from two hours ago. “We keep careful records of help desk inquiries that are difficult or that we can’t fulfill,” says HK. “We analyze them quarterly, and there were enough file deletion occurrences that we kept thinking we could do better. Our goal was to find a backup and disaster recovery product that would let us always respond affirmatively to an item-level deletion and not rely on the 24-hours-ago backup. That’s the tool we wanted to have.”

The catalyst for making the decision to ditch the under-performing solution they’d had for years was when the company went virtual, which necessitated a strategy change for HK and his team. “At that point, we wanted to back up everything – OS and all the data – which would of course increase our capacity need,” he explained. “We realized that we had a wide range of recovery needs and we needed a tool for all of them. We migrated to more virtualization, but there just was not enough time or media space to adequately back up everything we needed and provide for a suitable off-site rotation schedule for disaster recovery. We sought to improve our RPO/RTO, and wanted new options to respond to non-site disaster recoveries like ‘I deleted/messed up a file two hours ago… can you get it back?’ and so forth. All of this forced us to look for alternatives.”

That’s when HK found Dell AppAssure as his one solution for the company’s combination of backup and disaster recovery needs.

Solution

After evaluating other products including other Symantec offerings, HK and his team knew that they had the right solution in Dell AppAssure Backup, Replication & Recovery, with its ease of setup, variety of recovery option settings, and sophisticated virtualized disaster recovery features. “We now have the ability to have periodic recovery points for critical servers, so we can address non-site disaster issues like lost data,” says HK. “Also, we now ‘export’ physical servers to virtual (like P2V) and maintain these as standbys for disaster recovery and testing.” Dell AppAssure allows HK and his team to take periodic snapshots of his servers with no noticeable performance impact on the protected server, and with easy-to-restore data, whether from file servers or Exchange.

“When we first started with Dell AppAssure Backup, Replication & Recovery,” explains HK, “we thought it was just going to be a ‘belt-and-suspenders’ to our environment – in other words, something extra. But as time went on, and we learned how to effectively use more features, Dell AppAssure became a critical component of our data protection strategy; not so much for ordinary backups, but to cover a wide range of solutions for on-site and off-site recovery. We started with Exchange to get the 15-minute recovery points. We thought, wow this was great! Then we shifted our file/print from Netware to Windows, and went from physical to virtual for these servers, so we bought a few more licenses with Dell AppAssure and set up the console with both the Exchange physical servers’ and the virtual file servers’ RPO set for 30 minutes. We then added an FTP and threw that on. And we also added even more virtual machines! But the foundational purpose of this was always the recovery points, so now we can say ‘we got it’ – if someone has deleted a file or email. With Dell AppAssure Backup, Replication & Recovery, we got it.”

Results

This evolution of Dell AppAssure into the core of this firm’s IT environment has meant that HK and his team can make very specific comparisons about the tangible benefits that Dell AppAssure is giving the company:

RPO

• Before: “If someone lost file server data, changed data, deleted folders/files, encountered corruption, we were only able to restore data from tape from last night’s backup – provided there was no tape failure. That’s a 12-hour RPO.”
• Dell AppAssure Backup, Replication & Recovery: “We can respond to such an inquiry by restoring the requested data from as recently as 15 or 30 minutes ago (or from whatever recovery point is requested, based on our retention period maintained). It only takes seconds to do this, and I watch the task manager when Dell AppAssure is taking a snapshot – there’s not even a hiccup. It doesn’t impact the parent server at all.”

Item Level and Full Server Restores

• Before: “With Backup Exec and Exchange, restoring mailbox data was always a challenge and took hours to accomplish. We had to stage the entire database before we could restore individual mailbox data. It rarely worked as advertised and took easily over an hour. For a full server during testing, it took us easily 5-6 hours.”
• Dell AppAssure Backup, Replication & Recovery: “Recovering specific data or mail with Dell AppAssure takes less than 5 minutes compared to an hour or more with our previous solution. Recovering an entire server with Dell AppAssure took only a bit more than an hour.”

Disaster Recovery Testing

• Before: “We do our DR Tests at SunGard twice a year. We used to only be able to recover a handful of servers during a 16-hour test.”
• Dell AppAssure Backup, Replication & Recovery: “We now recover 16-20 servers during that same period of DR testing.”

In addition to concrete comparisons, HK and his team are getting amazing storage space savings with Dell AppAssure Backup, Replication & Recovery’s compression and deduplication. “We get fabulous compression,” says HK. Our TevoRepository is on a 2TB partition and we get well over 60% compression.”

In many ways, HK finds himself rather glad that his IT team had to go through some of the challenges they did to get to where they are now with Dell AppAssure Backup, Replication & Recovery. “Problems are the only things that teach you what your weaknesses are and force you to look in another direction,” explains HK. “This is how we got to Dell AppAssure and now we rely on Dell AppAssure as our ‘silver bullet’ and honestly, I can’t imagine living without it. It’s now our DR solution for Exchange, as well as getting our recovery points. What Dell AppAssure gives us that others have not, is helping IT look good in the eyes of users and management. We are able to respond quickly to data loss with a proactive solution rather than a reactive one. I love being able to say, “No problem … I can get that back for you.”