Backup Essentials Part Four

We have now finally arrived to the conclusion of the Backup Essentials series of posts. Part four may be the most important one to consider, as it is the factor that will have the single largest impact on your customers. For our conclusion we'll be covering testing and expectations management, and while these things don't sound as technically exciting as the previous parts they are critical to your business. In covering this topic we'll hit four key areas: Bench marking, availability, retention compliance, and configuration auditing.

Bench mark testing is critical in gathering data for you to use to set reasonable expectations for your customers. Periodic testing of exactly how long your backup solution takes to conduct operations is imperative if you wish to be able to speak credibly to a client about your capabilities. I am always pushing clients at my work to conduct test restores from random client machines at least a couple of times a month in order to keep track of restore time frames. Taking that data and coming up with an average of sorts will allow you to say with some confidence the amount of downtime a client could expect in a disaster. If your clients are attempting to form a service level agreement (SLA) it's especially prudent to have this data available to you.

Just as important is to check the integrity of your data. We gain hark back to conducting test restores, but in this case we have to take a step further than just testing I/O speeds. For example if you are backing up a SQL database the step has to be taken to attempt to do a RESTORE from the dump file (on a dev/test environment). The ideal test gets as close to a real world disaster scenario as possible, and even better if you train your associates to document practices and problems along the way. The data gathered from these tests can form the bedrock of a larger business continuity strategy as you figure out what resources need to go where, and the best placement for your experts. I generally suggest conducting some form of disaster test once every month or two, and at the minimum once every ninety days.

Let us also not forget about retention policies. It is important to double check your backup solution to make sure that any storage retention policies that you have set up are being enforced properly. The last thing you need is to either have your space taken up needlessly (driving up the cost of the solution and decreasing performance), or not have a set of data due to a retention policy that is too tight and unchecked.

Lastly we have configuration auditing. It's useful from time to time to go back through the rules that define what data is being backed up in order to make sure they are still valid. Over time companies move files around, migrate databases and email stores, and well ultimately things are always changing. It would be rather embarrassing to find that one was backing up an empty directory because the backup parameters haven't been checked in months. This is something that should be checked at a weekly or biweekly rate and requires close cooperation with leaders in other parts of the business in order to ensure accuracy.

The conclusion you should draw from this is that testing, auditing and compliance are terribly important. Conducting these properly will allow you to set reasonable expectations with your customers, give you confidence in your technical solutions, and will give your customer reason to have confidence in you.

Backup Essentials Part Three

It's time for part three of this series. What we're discussing here is what to keep in mind when selecting your backup solution. The factors we'll be discussing include: data availability, administrative cost of the solution, portability, and security.

Data Availability
This factor is always a balance that has to be struck between how quickly you need to get at your backed up data, and how resilient you want your solution to be. For example on one end you could merely have your data backed up to a drive attached to the source machine. This would give you the fastest access to it, but at the cost of a single point of failure. On the other end of the spectrum you can have the data backed up to offsite location(s). This would give you the most protection as any harm to your office is far from your data, but also increases the amount of time required for access. Ultimately you either want to have some form of both, a solution that is highly portable (physically moving storage and compatible with your equipment) or even better has all of the above.

Portability
Perhaps one of the more overlooked factors is how portable your backup solution is physically. This becomes a point of particular importance when you have backups that are at an offsite location. For most organizations that aren't in the 'large' category it is unlikely that you'll have a fiber link running across to your offsite location. So if/when an emergency strikes your main office you need to be able to physically move the data and then make use of local link speeds. Remember this when choosing your solution as the ability to do this can vary greatly depending on what sort of hardware requirements are involved, and the manner in which the software operates.

Administrative Cost
One of the banes of the small IT shop is the near impossibility of its associates to truly specialize. The needs of the business, and likely size of available staff tend to demand that the technician/administrator/engineer in this role be a generalist. It's important to keep this mind when picking your backup solution, as it would be a terrible situation to need to conduct a restore, but have no experts in its use to conduct it. If the product requires a significant amount of additional training to operate it can end up costing the business significant resources. The cost is incurred doubly so if the training itself isn't of a certain caliber as well. The long and short is that 'simple' and 'intuitive' should be your watch words here.

Security
The manner in which your data is stored and transmitted is obviously important. Looking around you are likely to find that most if not all of the notable solutions out there at the very least transmit data in an encrypted manner between target and source. The differentiator here is going to be the manner in which its stored after the job is complete. Do you have the option to encrypt the data? If you use third party encryption does it effect their storage format? Checking this functionality is especially important for those of you in the medical field who have to comply with HIPAA (The Health Insurance Portability and Accountability Act). Also consider the discrimination of data here. For example if I log onto one target machine am I capable of pulling data down to it that belongs to another? While it may sound convienent if the answer is yet, it does create a bit of a security vulnerability if one of your target machines were compromised. If your business in a field with confidential data you certainly want to make sure that you have the ability to discriminate and control which clients can access which data.

Truth be told an entire article could be written about each of these factors. These basic thoughts act as an effective guide in researching backup solutions and implementing them into your infrastructure. If anyone would like to discuss the topic in greater detail with me, feel free to comment or shoot me an email: koch.ryan@gmail.com.

Backup Essentials Part Two

For part two of this series we will begin the planning phase. In this phase we'll take an inventory, categorize our data, decide backup types, figure out our backup window, and figure out storage requirements. For this example we will use a small test environment representing the size of a 10 employee company as mentioned in the introduction.

First we'll knock out the inventory. The test infrastructure has a mix of desktops, servers, and remote laptops. I've gone ahead and included a list of these below:

6 desktops of a similar configuration (general user machines)
1 SQL Server (HR)
1 Exchange Server
1 Web server
1 Domain Controller
3 Remote laptops

Having this list we will now need to set priorities. To do this we need to break down an analyze what this business would need to operate, and what it has that merely makes it operate better/more efficiently. While simple, the prioritization below will help us in making decisions on backup types as well as scheduling and storage.

Need to operate:
Domain Controller
Exchange Server
SQL Server
Web Server

Makes business more effective:
Desktops
Remote laptops

Next we need to discuss what backup types are available to us. In general the various backup solutions will give you the ability to do file level backups, service/application backups, and image level backups. In general the decision of which level to use with which machine depends on what expectation of recovery time you have, the location of the machines, and what the machine is being used for. In our case the desktops, and the remote laptops will require a file level backup. The servers will all require an image level backup, a file level backup, and a service level backup.

After deciding what backup types we're using we also need to decide how this fits into our disaster recovery strategy. Do we need any of this data to be backed up to an offsite location? In our scenario it seems clear that the SQL Server,Exchange Server, Domain Controller, and Web Server all need to be backed up to a remote location. These services should be able to be brought backup at a remote office or home office in the event that the main office is brought down by a disaster of some sort for any extended period of time.

The last two factors to consider are scheduling and storage. For scheduling you have to figure out what your organization's 'production' hours are. This will allow you to create a backup window that will exist to minimize the impact your backup jobs have on the ability of  users to operate. For example if the office is open from 8am-5pm then an effective backup window might be from 7pm-6am. This will give a 2 hour buffer both on the beginning and ending side of the window in case someone stays late, or a backup job runs too long. A lot of the decision is going to be unique to your organization and its needs. Storage is also going to be a bit unique as the requirements are going to be different from backup solution to backup solution. Depending on if you choose to go with an appliance, or software you may or may not have to purchase your own hardware. To fully implement a backup and recovery policy however you are going to want to arrange for some sort of storage system on site at your office, as well as another one at some off site location be it an off site data center, or even someone's home office.

That about covers are basic planning stage. The next article will cover what we need to consider in order to pick and implement a solution. There may even be pictures in that one (screenshots).

Backup Essentials: A 4 part series

As many of you aware one of the topics I fancy writing about is backup and recovery. To appease that desire I've decided to write a 4 part set on what's involved in planning, picking and implementing a backup solution. While you've seen this talked about over and over again, we're at it here because it is truly important. Data loss and downtime is the same as tossing cash out the window, and most businesses can't afford it. Besides let's face it, now matter how well your infrastructure runs, Moore's law will eventually strike and it's best to be prepared.

The series starts with this introduction, an outline of what we'll be trying to accomplish. For our walk-through we will use the example of a company made up of 10 employees. Our goal will be to come up with a backup strategy to propose to this 'company', once they've accepted begin implementation, and then finish off by conducting some testing and concluding on its effectiveness. Our scenario will also involve two separate customer sites, and with machines ranging from standard Windows 7 desktops to an Exchange server and a SQL server.

Check back for the next part where we begin the planning phase. In the mean time I need to finish building up the test environment to use as screenshot materials. This should an interesting set of articles and hopefully will help some of you out there in the course of your careers.

Users and Security

I do apologize for the delay in getting a new article out. Between the power outages here in Columbus last week and the catch up work that followed the plate has been rather full. Today I started giving some thought to security, and specifically how the actions of Users can effect your policies and planning. It seems that no matter how much care and caution one puts into a great security set up, there is always one weakness to root out and that is User behavior. These beings seem to be able to defeat the greatest of security infrastructure practices and are somehow able to throw a wrench in the most finely configured of ecosystems. And so here is a bit of an overview of some thoughts on how to manage user behavior.

But how to do you prevent the user from accidentally breaching your security? It's not so much a question of control, as it is a question of influence and education. Ultimately the majority of user mistakes are due to a lack of instruction, or knowledge of good practices. It's understandable, most of these users be they internal or external customers have other things to do and have other concerns that have been given a greater priority. As the IT Engineer/Analyst/Manager it ends up falling to you to be the one that breaks this shell, and instructs them on what to do. The importance of taking the time to do this is only going to grow over time, especially since these users are now even bringing in their own devices and conducting business on them. You now not only have the possibility of company infrastructure being mucked up, but also of corporate data leaking through external devices.

How do you accomplish an educational role? It's about the soft skills here. You need to schmooze a bit with other departments and employees in order to gain their trust and cooperation. The idea of IT education needs to be sold as a value added piece, something that will ultimately save the company time and money. I find that his process is very similar to that of starting a new workout program or lifestyle change. It's best to start off just getting the first session/meeting/etc and then the next one. Once you can get a routine going the ride is much smoother, and you've accomplished an institutional change. An example of this might be to send out a weekly email newsletter and partner that with a monthly 'class'. Once you get solid practices in your users' minds you'll start to see improvements in their behavior and perhaps less security oriented incidents.

Outside of purely IT type education, you also need to make sure that policies are clear and published in as many places as humanly possible. In crafting your policies I would suggest taking heed of the manner in which intelligence agencies operate. Users should only have access to infrastructure pieces they have both 'clearance' for and a 'need to access'. This is very similar to the 'clearance' and 'need to know' principal of intelligence that is used to keep information leak to a minimum. Documentation for this principal is critical as the rest of your organization needs to understand the structure for how access is granted, as well as procedures to gain clearance when needed. Along with policies on access a proper acceptable use policy is recommended. The document should detail exactly what behaviors are frowned upon, and what behaviors are considered acceptable practice.

Lastly, and perhaps the most important piece of this is user involvement. If you want to maintain a credible IT department you need to make the users feel involved and keep them in the loop. Regular and predictable communication with as much of your organization as possible will create an environment in which users don't see the IT department as just some strange offshoot of the company that just tells them they can't do things with their system. By creating these venues of communication you'll be able to create a situation where you users aren't following described practices because you said so, but because you convinced them that they *want* to. This type of shift can only lower risk to both your data and your infrastructure.

Backup 101

I swear I'll post something with some substance tomorrow, but today I wanted to promote the recording of the Backup 101 web seminar I hosted today. My presentation covered an overview of backup essentials that are required for any effective IT knowledge set. I hope to do more sessions like this on a range of topics as time moves forward. If you guys have any requests please feel free to drop me a line.

Backup 101 Seminar

Just quick blurb for you guys today. I'm hosting a Backup 101 seminar/class on Thursday 6/28 that you might find fruitful. It will have a quick slideshow presentation followed by what I hope will be an energetic and fruitful conversation. Please click here and register for the webex event.