Don’t Rely on Your Cloud Provider for Disaster Recovery

It surprises some to hear this, but cloud doesn’t come with disaster recovery built in. Creating and implementing a disaster recovery strategy for your cloud infrastructure is your responsibility.

Disaster Recovery-like Features in the Cloud and Their Shortcomings

There are some capabilities in the cloud that sound like disaster recovery. Cloud has high availability, and if the hardware underlying your instance goes down, your virtual machine will readily and automatically be migrated to a new physical device. That sounds like just what you need for DR, but there are several ways this falls short:

  • If there’s a widespread outage, due to network, power, or other issues, you probably won’t be automatically switched over to a different region that isn’t experiencing that problem. Even if the cloud provider can do this, it may not meet your recovery objectives.
  • The data backed up by the cloud provider and used for your recovery may not meet the recovery objectives for your business and your workloads. Unless you’ve got a backup and recovery agreement with your cloud provider, they may only have a backup of your most recent data. This may not enable recovery in instances like ransomware.
  • If you’re using Software as a Service, you may not have access to a copy of the data to store in a different location for safety. Even with infrastructure or platform as a service, moving data out of the cloud can be difficult and expensive. Remember the 3-2-1 backup rule requires having copies of data in multiple locations! 

Build Your Own Disaster Recovery Solution

The net impact of those shortcomings is that to achieve true disaster recovery capability, you need to plan and implement a solution yourself, even when your infrastructure is in the cloud. To do this, you need to add a few capabilities:

  • Secondary location in another region, cloud, or data center. If there’s a network outage that prevents you from accessing data in your primary cloud, you need another site where you can connection. This can be another region of the same cloud provider, a different cloud provider, or a data center you still maintain and support.
  • A complete backup and archiving strategy. You need to create and store copies of your data that will allow you to load data at a secondary site, recover from ransomware, or load historical data for analytics or compliance purposes.
  • Offsite storage. Keep a copy of your data outside of your cloud to ensure you have access in case of a cloud failure.
  • Unify your cloud recovery strategy with your on-premises recovery strategy. Because most businesses are not cloud-only and have a hybrid IT infrastructure, you’ll simplify your recovery solution if you have one set of tools and one strategy that covers your systems whether they’re running in the cloud or at your own site.

CCS Technology Services develops comprehensive disaster recovery solutions that protect your business wherever your data resides. Contact us to learn how we can help you develop a disaster recovery strategy for your cloud.

Additional Disaster Recovery Resources

5 Disaster Recovery Disasters to Avoid

Craft An Effective Disaster Recovery Plan

5 Changes to Make When You Switch to Disaster Recovery in the Cloud

Summer Storms Shouldn’t Take Down Your Servers

Summer means power outages. That can mean a data center outage; but it shouldn’t. All businesses should have appropriate disaster recovery plans to keep them functioning through power outages and other incidents that take down systems.

A disaster recovery plan includes the steps needed to bring information systems back online, but it isn’t just a copy of the daily runbook. The plan needs to document:

  • Inventory of systems affected. Both hardware and software resources should be identified.
  • Risk assessment and prioritization. Some systems can have downtime without major impact on the business; others serve critical business functions and need minimal downtime. Analysis should rate each system’s level of risk and its importance to the business.
  • Recovery objectives. “As soon as possible” is not specific enough guidance for the IT team. In order to appropriately design a recovery procedure, the business should define a recovery time objective (RTO) and recovery point objective (RPO) for each application. These numbers tell the IT team how long an application can be down and how much data the business can afford to lose. With those numbers in mind, the technology team can implement high availability and backup solutions appropriate to the business needs. Without those numbers, IT has no choice but to overspend and provide high availability to all applications or underspend and fail to provide applications the support they need.
  • Recovery procedures. Because teams shouldn’t need to scramble to figure out what to do in the middle of a crisis, the plan should include specific details of the recovery process. It’s particularly vital to include dependencies to ensure systems are brought up in the appropriate sequence. Also critical is documenting the process to check out the restored servers and verify that they’re up and operational with the correct data.
  • Recovery personnel. Include a list of key contacts and their backups. Also document responsibilities, including who has the all-important authority to invoke the recovery plan.
  • Fallback process. Recovery may include bringing systems up at another location; eventually, they need to be restored to the normal production servers. In many ways, this process is the same as the recovery process, just to a different set of machines, but any special considerations should be noted.
  • Impacts on business processes. It’s possible that some recovery procedures will change the way the business needs to perform certain operations. For instance, you may opt not to have secondary servers for a low-priority process and to switch to a manual process in case of failure.

Once the recovery plan is developed, it needs to be tested to ensure that it works. It’s surprising how easy it is to leave important systems and important steps out of the plan! Only testing can provide the reassurance that the plan will be effective. Tests can be as simple as a tabletop read-through, but full-scale disaster simulations that execute the documented processes are the most robust way to test a disaster recovery plan.

Finally, the plan needs to be kept up to date to reflect changes in IT resources and business processes. It’s a good idea to update the plan as part of your change management process whenever a new system or device is deployed in production. Annual reviews, coordinated with an annual test, are also effective.

For more guidance on developing an effective disaster recovery strategy, contact CCS Technology Group.

Additional Disaster Recovery Resources

5 Disaster Recovery Disasters to Avoid

Make Sure Your Disaster Recovery Plan Isn’t Just Words on Paper

Craft An Effective Disaster Recovery Plan

Choosing the Right Offsite Location for Backups

It’s well-known that one of the best strategies for backups is to follow the 3-2-1 rule: have a least three copies of data, use two different storage media, and keep one copy offsite.

When it comes to deciding where to keep the offsite copy, cloud is an obvious choice today. However, cloud isn’t the only choice. The backup copy can be stored at your secondary data center, or at some storage facility.

How do you choose? The cost of the storage is one factor, but other factors should be considered as well. While the primary reason for keeping the copy offsite is to ensure you won’t lose it if your primary site it totally destroyed, there are other considerations as well.

The things to think about in addition to cost are the level of risk and the impact on recovery time objectives (RTO) and recovery point objectives (RPO).

Offsite storage can impact your RTO depending on how long it takes to access the data. If you use cloud for offsite storage, this will be impacted by both data access times (backups stored on a less expensive storage tier will take longer to access) as well as the time to transfer the data. That data transfer process will in turn be affected by network bandwidth. If you choose to store data offsite at a storage facility your RTO will be impacted by the time to locate the backup as well as either the network bandwidth or the time to physically ship the media to your data center. If your offsite location is a secondary data center and you’re failing over there, you may be able to recover almost immediately; otherwise, transferring the data back to your primary site requires either network bandwidth or physical transport capability.

The time delay in delivering offsite media also affects RPO. You’ll lose any data between the last backup and the outage. If it takes a day for a tape to be delivered and processed, you won’t be able to recover yesterday’s data; you’ll be recovering the day before yesterday, while yesterday’s is still in transit.

The final consideration is risk. Offsite storage that’s nearby is conveniently accessible by your onsite staff, but it’s vulnerable to being damaged by the same natural disaster that’s taken out your primary site. A more remote storage site reduces that risk but can increase delays or errors in accessing the data you need to restore.

Many times, if you’re following the 3-2-1 rule, you’ll have a second copy in your data center and won’t have to worry about accessing the remote copy. But if that onsite backup turns out to be bad, getting access to the remote copy will be extremely important. CCS Technology Group helps businesses develop comprehensive business continuity solutions that ensure you’ll have access to your backups when you need them. Contact us to learn more about what you should consider when developing your backup strategy.

Additional Backup Resources

Don’t Let Ransomware Destroy the Backups You Need to Recover from Ransomware

Effective Backups Need to Address These Challenges

The Differences Between Backups, Disaster Recovery, and Archiving Matter

How Many Purposes Are Your Backups Supporting?

Backups are not a new practice, but backups today face new challenges. They need to serve multiple purposes and meet multiple requirements. Businesses that are still creating their backups the same way they always did should review their strategy to make sure it meets today’s backup requirements.

Many Backup Purposes

Backups aren’t just about making copies of files. Backups need to create the duplicates in ways that allow them to be used for multiple purposes. These purposes include:

  1. Data retention. Depending on your industry, there may be compliance mandates to retain all data for some period of time. In addition, new application development and analytics projects need access to historical data. When planning for this need, it’s important to recognize that backups are not archives.
  2. Disaster recovery. Backups are the primary method businesses uses to recover from outages and other disasters that make servers inaccessible.
  3. Prevent tampering. Data tampering is a kind of disaster, but deserves mention on its own. Backed up data provides an audit trail and historic record of data changes that can identify whether data has been tampered with and restore it to the correct value.
  4. Comply with e-discovery and data privacy laws. Backup files provide support for e-discovery requests that mandate searches through historical data. Backups also provide support for new data privacy laws that require businesses share all the information they have about a consumer with that individual.

Many Backup Requirements

Along with being able to support multiple purposes, backup strategies need to meet multiple requirements. These include:

  1. Completeness. It’s easy to omit critical systems from backup scripts or to overlook alerts that backups are failing to complete successfully. At the same time, some data may change infrequently and not need daily backups. Matching the level of protection to data criticality requires analysis.
  2. Time to create. Creating application consistent backups generally requires applications to be quiescent. The windows available for shutting applications down continue to shrink as business continues around the clock.
  3. Time to restore. Different backup approaches offer different levels of flexibility and take different amounts of time to complete the file restore process.
  4. Security. In order to perform speedy restorations, backups need to be immediately available, but they also need to be protected from loss, tampering, and other unauthorized access.
  5. Scalability. Backup and recovery processes that work well on smaller volumes may fail when systems grow larger.
  6. Different sources. Backup strategies need to address a wide variety of different sources, including cloud and data centers, virtual machines and bare metals.

Does your backup strategy meet all of today’s challenges? If you need help updating your backup tools and procedures, contact CCS Technology Group. Our business continuity solutions apply best practices to protecting your data.

Additional Backup and Disaster Recovery Resources

7 Critical Factors to Consider When Developing Your Backup Strategy

5 Disaster Recovery Disasters to Avoid

Choose the Right Backup Strategy to Meet Time and Space Requirements

7 Critical Factors to Consider When Developing Your Backup Strategy

Backing up without a strategy, or with an ineffective strategy, is likely to generate backups that don’t protect your business. After all, the point of backups isn’t to create the backup; it’s to create a copy of data that your business can restore from when the primary copy is damaged or unavailable. Creating a backup strategy to meet that goal requires identifying key backup concerns and selecting appropriate backup technologies.

Critical Factors in Developing Backup Strategies

When you start thinking about your backup strategy, keep these considerations in mind. You’ll need to balance these factors to come up with a strategy that truly protects your business.

  1. Cost. Like everything else, backups cost money. You may have to buy hardware and software, pay for a maintenance agreement, and train your staff.
  2. Backup location. Today, many default their backups to the cloud. However, you should still consider potentially keeping a copy of your data in another location as well. Cloud outages are rare but do happen.
  3. Backup method. You can choose from different kinds of backups. Each backup method requires a different amount of storage, impacting costs, and a different amount of time, impacting both the length of the backup procedure and the length of the recovery procedure.
  4. Backup (and recovery) flexibility. When creating backups, you generally want to backup everything, but that’s not true for recovery. Recovery needs to be able to scale from restoring a single file to restoring an entire server.
  5. Backup schedule. Your backups should be automated and run on a schedule, not rely on someone remembering to execute them manually. They should be scheduled to run frequently enough that you’ll capture data that changes often as well as data that changes rarely. They should be scheduled around production workflow needs. Your recovery point objective and recovery time objective come into play here; note those targets shouldn’t be global but should be tailored to the needs of each system. Your backup schedule may be unique to each system as well
  6. Scalable. You can expect your data to grow and your backup needs to grow along with it. Your backup process should be able to handle expected volumes of new data. You should have a process that ensures new servers, applications, and data stores are added to your backups.
  7. Backup security. Backups need to be accessible when needed, but they shouldn’t be accessible by just anyone. Making sure backups are safe from tampering is vital to protect your business.

Backups are part of a comprehensive business continuity solution. Contact CCS Technology Group to learn more about how these critical factors can shape your backup process.

Don’t Let Ransomware Destroy the Backups You Need to Recover from Ransomware

Backups are the primary means a business can use to recover from a ransomware attack. It’s no wonder, then, that many forms of ransomware now attempt to destroy any backup files they encounter. Protecting your backups against ransomware is an important part of your defensive strategy.

The Ransomware Threat Against Backups

Ransomware is a form of malware that encrypts system and data files with an unknown encryption key. This encryption makes the files unreadable by their owner. The only way to recover the data is to pay a ransom and receive the encryption key or restore the files from an unencrypted backup.

Some malware implementations attempt to recognize backups by file extensions and will delete those files. On Windows systems, ransomware can detect and delete shadow copies that support file recovery. Ransomware will also attempt to spread through the network, accessing mounted file systems containing backup, and encrypt those files as well. Ransomware may even be able to reach and corrupt backup files stored in the cloud.

Ways to Protect Backups Against Ransomware

The methods to protect backups against ransomware rely on making multiple copies of backups and taking steps to make them inaccessible to any ransomware.

Make Multiple Backups

It’s a good idea to use specialized third-party backup software rather than (or in addition to) built-in backup solutions. Ransomware can’t know how to target every vendor’s backup files.

Keep multiple versions of your backups. There are good reasons for this that have nothing to do with ransomware, but if your latest backup is encrypted, you can restore an older version of your files from before the ransomware attack.

Keep Backups Inaccessible to Ransomware

There are several ways to make backups inaccessible to ransomware:

  • Store at least one copy of your backups in an offsite location.
  • Dismount backup devices after the backup process is complete.
  • Make backup files read-only, or store on write-once media.
  • Use access controls such as Windows Controlled Folder Access to prevent unauthorized processes from accessing backup files.

Note that backing up to cloud does not make those backups inaccessible to ransomware, unless the only access to the backup is via an API rather than mounting the cloud as a drive.

Test Your Backups

It’s important to test your backup files periodically to verify that the data is complete and that you know how to access it and use it to restore your data. You should conduct a full disaster recovery test at least annually and continuously monitor your backup process and address any alerts or failures.

CCS Technology Group helps businesses implement comprehensive business continuity solutions to protect against ransomware and other causes of IT outages. Contact us to learn more about implementing a backup solution that protects your backups as well as your data.

Additional Ransomware Resources

Take These Steps to Avoid Expensive Ransomware Recovery Costs

Don’t Lose Your Files to Ransomware

Ransomware 101: Keeping Your Organization Safe

5 Disaster Recovery Disasters to Avoid

No one likes thinking about bad news. Maybe that’s why planning for disaster recovery often doesn’t get the attention it requires. But without a solid plan that’s been documented and tested, your disaster recovery process can turn into a disaster of its own. Take the steps you need to ensure you don’t experience these disaster recovery disasters:

1. Recovery site not ready

There are two backup sites you need to think about when you create your disaster recovery strategy.

The first backup site is where your backups are stored. You need your backups to be stored securely, but also to be available quickly when they’re needed. It often makes sense to keep backups in at least two locations—one, onsite at your primary data center for use in small outages, and two, at your secondary location or in the cloud for use when the primary site is unavailable.

The second backup site is where you bring up your systems when your production site is down. Whether in the cloud or a secondary data center, it’s a good idea to keep this site relatively current with applications and data to allow the recovery process to happen more quickly.

2. Backups not available

You can’t restore systems to their production state without backup copies of systems and data. The problems go beyond potential inaccessibility of the site where backups are stored. Common backup problems include data that was never backed up or backup media that has been corrupted. Another potential problem is that the data backup doesn’t let you easily isolate the exact elements to be restored, forcing you to spend time restoring files that haven’t changed.

3. Recovery process not known

In the middle of a crisis is a bad time to discover you don’t know how to restore your systems. Not being documented at all is a worst-case scenario, but even a thick binder of recovery procedures doesn’t guarantee the process will run smoothly. This isn’t a process your staff is familiar with; maybe they remember it from a once-per-year test scenario, or maybe they didn’t participate in the test and have never executed the process.

In addition, no matter how detailed the recovery documents are, they may not be a match for your situation. Sometimes you’ll need to recover just a single file or a single server, not your entire data center, and comprehensive disaster recovery plans typically focus on the biggest possible outage. You’ll have to figure out a smaller-scale recovery process on the fly.

4. Recovery process takes too long

Every minute your business isn’t operational costs money, so restoring systems as quickly as possible is critical to minimizing the impact of a disaster. You should have a recovery time objective for every system, and conduct tests that verify you can meet those requirements.

5. Recovery process is error-prone

A manual recovery process is vulnerable to user error, and every recovery process is vulnerable to poor communication.

Take steps to avoid disaster recovery disasters before you experience a crisis. Let CCS Technology Group develop a disaster recovery solution that gets you through your disaster without creating new disasters along the way. Contact us to learn more about effective disaster recovery planning.

Additional Disaster Recovery Resources

Craft An Effective Disaster Recovery Plan

The Differences Between Backups, Disaster Recovery, and Archiving Matter

5 Changes to Make When You Switch to Disaster Recovery in the Cloud

Make Sure Your Disaster Recovery Plan Isn’t Just Words on Paper

A written disaster recovery (DR) plan is a good start towards making sure your business can resume operations after an outage, but you won’t know how good those words are until you put them into action. Because you don’t want to find out your plan is incomplete or incorrect during a crisis, it’s important to schedule periodic disaster recovery tests to try out your plan before you need to execute it for real.

Types of Disaster Recovery Tests

There are several different ways you can test your plan:

  • Circulate for comment. Distribute the plan to everyone who would participate in it and solicit their comments and feedback.
  • Walkthrough the plan. Gather everyone who would participate in the plan in a conference room or on a conference call. Read through the plan as a group—out loud, not silently. Because there is group interaction in this approach, you’re likely to surface issues that won’t be identified when individuals read through the plan separately.
  • Tabletop testing. Similar to a walkthrough, the participants are gathered together. Rather than read through the plan in isolation, they are presented with a typical failure situation and called upon to resolve it. This can identify planning gaps and failures that are not addressed by the DR plan. It’s important to choose realistic failure scenarios and that the participants are not informed of the scenario in advance.
  • Parallel test the plan. Bring up the disaster recovery systems and test whether they can execute a day’s work. The production systems run in parallel, so the only impact on routine business is that some personnel have to perform tasks on the disaster recovery systems.
  • Failover test. Simulate a production outage by gracefully shutting down the primary servers and failing over to the secondary site. This test method impacts ordinary production work so it may be better to execute this process on a weekend or other low volume time period. This process requires additional work to bring the primary servers back online after the test is complete.

Learn more in Craft An Effective Disaster Recovery Plan.

Disaster Recovery Test Follow-up

Whichever test strategy you choose, the test process isn’t over when the final system is brought back online. After the test, the DR plan needs to be updated to reflect:

  • missing applications. It’s not uncommon for applications to be overlooked when the DR plan is written.
  • missing or incorrect steps. The processes for bringing up applications may be missing some steps, miss some dependencies, have steps in the incorrect sequence, or contain errors in the details of the commands to be executed.
  • incorrect timings. Every application should have a recovery time objective which the recovery plan attempts to meet. If the test shows recovery can’t meet those objectives, the plan needs to be revisited to determine how it can be altered.
  • missing communication. Plans often fail because important notification steps are omitted.

In addition, you should always consider how the plan would have worked if this was an actual, unscheduled outage.

Learn more in Don’t Improvise Your Way Through Disaster Recovery.

Repeat the Test

If there were major failures during the test, take time to revise the plan to reflect those problems and then schedule another test to verify the corrections. If the recovery process mostly worked as planned, you can wait until your next regularly scheduled test—usually annually, though some prefer twice annually or even quarterly—to test the update.

CCS Technology group offers disaster recovery planning services. Disaster recovery testing is an important part of your business continuity strategy. Contact CCS Technology Group to learn more about writing and testing your DR plan.

Choose the Right Backup Strategy to Meet Time and Space Requirements

There are multiple reasons businesses need to backup their data. You need the ability to restore data if it gets lost or corrupted, or if a disaster requires shifting processing to an alternate site. Compliance policies may require retaining copies of data for a lengthy period of time. Analytics projects may need years’ worth of history, and new software projects often require copies of production data for development and testing.

Given all these reasons for making backups, implementing an effective backup process is a critical IT function.

Types of Backups

All critical systems need to be backed up daily, but not every piece of data needs to be backed up every day. There are different kinds of backups that allow the process to run more efficiently.

  • Full backup. Every system needs a full backup to be made at least once. This is a complete copy of the data, and serves as a baseline state for the system.
  • Differential backup. A differential backup includes all the data that changed since the last full backup.
  • Incremental backup. An incremental backup includes only data that changed since the last incremental backup.

Once you’ve made a full backup, you can use either differential or incremental backups to copy only the changed data. This makes the backup process faster and requires less storage space. However, it makes the recovery process longer, as recovering means first restoring the full backup and then applying the changes on top of that.

Creating a synthetic full backup every week or month allows you to use incremental backups and shorten the recovery process. On a regular schedule, the incremental changes are applied to the last full backup. This effectively creates a current full backup that can be restored rapidly.

Backup Capabilities

In addition to the different types of backups described above, there are some backup features that can help speed recovery in specific scenarios.

  • Snapshots. A snapshot is a copy of a dataset at a specific point in time. Unlike backups, snapshots are typically stored on the same device as the original data. This makes them suitable for recovering rapidly, but you can’t recover from them if the device fails.
  • Replication. Replication copies data changes to a second site nearly instantaneously. This allows recovery with almost no downtime if the primary device fails. However, the second site only has the latest copy of the data, so it doesn’t support recovery if data is corrupted or deleted or if an older version is required.
  • Deduplication. Much data is stored in multiple places throughout an organization. Deduplication reduces the size of backups by identifying and reducing this duplicate data. However, recovery times are made longer by the need to reverse this process.

With all these options, choosing an appropriate backup strategy requires careful consideration. Contact CCS Technology Group to develop and implement a backup solution that protects your data and your business.

Additional Backup Resources

Effective Backups Need to Address These Challenges

The Differences Between Backups, Disaster Recovery, and Archiving Matter

Understand the Different Cloud Options for Your Backup and Disaster Recovery Strategy

Don’t Improvise Your Way Through Disaster Recovery

Given the importance of disaster recovery (DR), you don’t want to improvise through the planning—or worse, through the execution. Here are some best practices to make sure your disaster recovery follows an effective script:

1. Assign staff to disaster recovery

It sounds obvious, but if you don’t have staff assigned to disaster recovery, it isn’t anybody’s job, and it won’t get done. You need staff who are dedicated and empowered to make sure disaster recovery is properly planned. This isn’t limited to technology staff either; business employees have roles and responsibility in disaster recovery as well.

2. Develop a detailed plan

If you don’t want to improvise, you need a documented plan. The full contents of a DR plan are beyond the scope of this short blog post, but you need to start by identifying all of your IT resources. Evaluate the impact of an outage on each application and use that to determine your DR priorities. Then assess how much time you can tolerate the application being down and how much data you can afford to lose. Use those numbers to guide you in developing a cost-effective recovery strategy. Document the recovery steps in detail, and make sure the recovery plan will be available in case of a disaster.

3. Test your recovery plan

It’s far better to discover your DR plan won’t work during a test rather than during a disaster. Schedule time to test your plan, at least annually. There are different ways of approaching testing, ranging from a table read-through of the documentation to fully executing the steps to failover and resume operations at a secondary site. The more your test simulates a real disaster, the more reliable results you’ll get. Track the time it takes to recover as well as the accuracy of the documented procedures. After the test, collect feedback from all participants on what worked and what didn’t, and use it to update the document.

4. Update the plan

Changes in your business and your technology mean the plan that worked last year may not work this year. Allocate time to review and update your plan every year—even better, make updating the plan part of your change management process and don’t sign off on deployments until the recovery process is documented.

5. Don’t go it alone

For many businesses, leveraging Disaster Recovery as a Service (DRaaS) is a good choice that makes disaster recovery faster and more reliable. With DRaaS, you get a high level of automation and support from the provider to help guide you through the process of defining and implementing a recovery strategy.

Another way to avoid going it alone is to work with an IT services firm like CCS Technology Group. Our disaster recovery and business continuity services help you protect your data, reduce downtime, and survive a crisis. Contact us to learn how CCS Technology Group can help you write your disaster recovery script.

Additional Disaster Recovery Resources

Craft An Effective Disaster Recovery Plan

5 Changes to Make When You Switch to Disaster Recovery in the Cloud

Backups Are Not A Disaster Recovery Solution