Blog / Municipal IT Budget Planning Part 2: Municipality Backups and Disaster Recovery
This is part 2 of a 9-part series that will look at selected areas of interest to Municipalities, pertaining to IT Budget Planning. Please contact me if you would like copies of other articles.
It seems we’re starting at the end of the series, when we talk about Backups and Disaster Recovery. But of all of the possible topics, it’s our experience that is the least understood and most ignored aspect of all IT areas. Most Municipalities are doing a LOUSY job of Backups and Disaster Recovery (BDR for our purposes.)
It’s not hard to understand why. It has the least payoff in terms of cost-benefit of any function or system, or at least that’s the conventional wisdom. For every $100 you invest in BDR, you get exactly ZERO in increased functionality, capacity, and performance. In fact, it’s viewed as a drag on your IT operations, as it takes careful management and constant testing to ensure it’s working properly.
Until you need it, then it’s like GOLD. At least one Alberta Municipality has learned this painful lesson every year for the last 5. You need to have an Insurance mindset to appreciate and properly plan for BDR.
Let’s start by defining a few terms:
Backup is the process of taking your critical IT systems and copying them to a media that is removed from the main system using portable cartridges or by storing them to an offsite location. Backups are used to recover select data (such as files or Emails), to archive old seldom-used data, and for Disaster Recovery. Backup media can also be Cloud-based, in that backups are written directly to a Cloud Storage Provider. However, good Cloud systems copy data to a local storage device and then to the Cloud.
Backup Retention is the term used to define how many days you can go back in time, before the backup media is overwritten with fresh data. Retention times can be a week, a month, or even a year. Longer retention times take more space on the media.
Backup Software is the application that controls the process of copying the IT system data to the backup media. Modern software has sophisticated features that expedite, compress, and manage the backup process. Backup software also works in reverse, in that it manages the process of recovering files from the backup media to be put back on the server. Most modern backup software is subscription-based; you pay a monthly fee to use the software, instead of purchasing it outright.
Disaster Recovery (DR) is the execution of a plan that outlines how backups will be used to recover systems in the event of a failure; often catastrophic or system-wide. We tend to think in terms of fire and flood, but Cyber Ransomware Attacks have caught out more than one Municipality in the last 18 months. The plan often provisions resources (servers, network infrastructure, Internet, etc.), so DR often involves discussions about computer equipment.
DR plans need to answer one fundamental question: How long can you be without one or more IT systems? The answer could range from several days to hours. Some mission-critical systems (like banks and hospitals) are measured in minutes or are even fully redundant (i.e.: they never go down.) Obviously, the shorter the allowable downtime, the more DR costs.
Books have been written about the nuances of BDR, but we’ll focus on these simple definitions to help with our budget work.
Let’s start Budget Planning with a typical scenario*:
- New System – or one that’s 5 years old and needs to be replaced (Little Idea #2)
- IT Systems comprised of 3 (virtual) servers and 2TB of TOTAL storage requirements
- Internet Connection: Cable (75/10Mbps – Down/Up)
- Backup Retention: 5 Business Days (1 calendar week)
- DR Allowable Downtime: 48 hours
- Budget Plan: 12 months
Option #1: Locally-stored and managed backup media
The simplest approach is to use a cartridge-based system with provision to store the 2nd oldest cartridge securely offsite (i.e.: bank vault); the CFO’s desk drawer, the Receptionist’s purse, or the filing cabinet are NOT secure and offsite. Thus, someone needs to make a trip to the bank’s safety deposit box every day.
Capital (equipment) Costs:
- 1 x Backup Host Drive: $250
- 1 x Backup (seed) Hard Drive: $300 (initialize the system – retained onsite)
- 5 x 3TB Backup cartridges: $2,500 (3TB needed to accommodate growth)
- Total: $3,050
Operating Costs:
- 3 x Backup Software: $35/month x 12 months = $420 (3 required – 1 for each server)
- 3 x Backup monitoring support: $150 x 12 months = $1,800 (random testing, resolving errors, etc.)
- 1 x Safety Deposit Box Rental: $25 x 12 months = $300
- Total: $2,520
Total Budget Option #1: $5,570 + 10% contingency = $6,127 -> $6,200
The only difficult part of this scenario might be achieving the 48 hour DR time, as it implies that server and related network hardware can be readily replaced or accommodated in an alternate location, within 24 to 36 hours. In addition, recovery costs will be higher, as it will require more onsite Technical help. You can also see that if the Backup Retention time is pushed to 1 month or 1 year, the cost of backup cartridges escalates dramatically (1 month requires 9 cartridges; 1 year requires 21 cartridges – these are called “Grandfather-Father-Child Rotations.”)
Option #2: Cloud-hosted Offsite backup – Limited DR capability
This uses a Cloud-based backup Provider to transfer the files DIRECTLY to the Cloud system. Cloud backup systems require high speed Internet; cable speeds (50/10 Mbps) or better. There are some caveats; national Cloud Providers often store redundant copies of backups in locales other than Canada, so this may violate legislative requirements. Recovery capabilities may be compromised; many national Cloud Suppliers have trouble transferring entire backup copies BACK to your hardware, in the event of a full restoration. Sometimes they revert to shipping portable hard drives back and forth across the country; hardly fast or secure. Cloud Providers usually supply their own backup software, and they should monitor backups, perform random restore tests, and troubleshoot backup errors, as part of their service… though you need to verify this.
Capital (equipment) Costs:
- 1 x Backup (seed) Hard Drive: $300
- Total: $300
Operating Costs:
- 3 x Cloud-hosted Backup environments – 14 day retention @ $250/month/Server:$750 x 12 months = $9,000
- Total: $9,000
Total Budget Option #2: $9,300 + 10% contingency = $10,230 -> $10,250
So, for an extra $4,050/year, you get the convenience of not having to manage Cartridge Security or rotation – or physically transfer cartridges. The retention period has been extended to 14 days, and you have TRUE offsite backup that is located at a significant distance from your facilities. However, meeting the 48 hour DR time-frame can still be a challenge; in fact, it may be worse – as to perform a full DR requires that you transfer upwards of 2 TB of data back over the Internet to your local facility – and you still have to find suitable equipment to host the restored files!
Option #3: Cloud-hosted Offsite backup – Full DR capability
This uses a Cloud-based backup Provider to transfer the files DIRECTLY to the Cloud system – AND the Cloud Provider is offering to remote-host your backed-up servers on their infrastructure, in the event of a Disaster. This means that your servers can be activated on the Cloud Supplier’s system, that would allow Users to login remotely to conduct operations. While there are some limitations, it means that DR times can be reduced to 24 hours or less – in some cases, as short as 8 hours. You also don’t need local server infrastructure to be operational in DR mode.
Capital (equipment) Costs:
- 1 x Backup (seed) Hard Drive: $300
- Total: $300
Operating Costs:
- 3 x Cloud-hosted Backup environments – 14 day retention and full DR @ $500/TB + $25/Server):$1075 x 12 months = $12,900
- Total: $12,900
Total Budget Option #3: $12,200 + 10% contingency = $13,420 -> $13,500
So, for a little more than double the cost of Option #1, the BDR capabilities have been greatly enhanced.
There are other soft costs that you should account for, regardless of what option you use. More than likely, you don’t have a written or tested DR plan. That may require a Consultant to help you develop one ($2,000 – $4,000.) Testing DR in a full Emergency Operations Plan table-top exercise can also add costs. Archiving data to reduce the amount of data actively being backed up, may require a file re-organization, which could mean some external help and additional archive-storage media ($1,500 – $7,500.)
Of course, many factors will impact the budget for your individual BDR scenario: number of servers, amount of data, allowable down time, and Internet speed, are a few. The locally-managed solution (Option #1 – or a variation of it) is the only practical backup method, if you have poor Internet.
Finally, a caution about Cyber Security (Little Idea #3): An effective BDR is the ONLY proven method to guard against a Cyber Ransomware Attack. While Firewalls, Anti-Virus, and User Training are good Cyber Security countermeasures, BDR is your last line of defense against an attack. When all else fails, you can wipe the servers clean and restore the last known (good) backups – but only if you have them.
Next week, we’ll explore Proactive Maintenance. Until then, please contact me or your Account Manager, if you would like some personalized help with some stress-free BDR budgeting.
Thanks!
Dave White
TRINUS
stress-free IT
trinustech.com
* – Full Disclosure: We’re going to use equipment and prices for TRINUS-supplied Systems & Services. Other Provider costs and equipment may vary.
