Linux server performance

In my daily tasks I deal with a lot of Linux servers, and from time to time decide to tweak them for performance reasons, depending on what task they are executing. A lot of the units I'm dealing with tend to be operating a postgres database and some sort of data store for a custom application that is being run (usually via tomcat). I've found that there are three easy things to play around with in order to get the most out of the system, especially if the resources on the box are fairly limited. Those things are the swappiness value, the I/O scheduler, and use of the renice command implemented with a script called via crontab.

Swappiness

The Swappiness value is what systems administrators and engineers use to instruct the linux kernel on how aggressive the system should be in storing pages of memory on disk, as opposed to in memory. Most default installations have this value at 60 which is supposed to represent a balanced number (the range is: 0-100). In my situation where I'm running a lot of database operations I've found that a higher value seems to help free up memory for use in postgres related processes, where otherwise idle system processes may have been holding on to that memory. This has been particularly effective in situations where I have application servers with just barely enough memory to get by.

You can adjust the swappiness value two ways. The first is more of a testing/temporary measure and can be done by using the following command (via the terminal):

sysctl -w vm.swappiness=(value you'd like to set it to)

You can also make this change by editing the following file: /proc/sys/vm/swappiness . One should exercise caution when editing this file though as it does require a bit of monitoring to make sure that you aren't breaking vital processes when changing memory allocation settings.

I/O Scheduler

CFQ (Completely Fair Queuing)
If my memory is still serving me well, on most Linux distributions this is the default setting. This scheduler serves as a sort of general use setting as it has decent performance on a large number of configurations and uses ranging from servers to desktops. This scheduler attempts to balance resources evenly for multiple I/O requests, and across multiple I/O devices. It's great for things like desktops or general purpose servers.

Deadline
This one is particularly interesting as it more or less takes 5 different queues and reorders tasks in order to maximize I/O performance and minimize latency. It attempts to get near real time results with this method. It also attempts to distribute resources in a manner that avoids having a process lose out entirely. This one seems to be great for things like database servers, assuming that the bottle neck in the particular case isn't CPU time.

Noop
This is a particularly lightweight scheduler and attempts to reduce CPU latency by reducing the amount of sorting occurring in the queue. It assumes that the device(s) you are using have a scheduler of their own that is optimizing the order of things.

Anticipatory
This scheduler uses a slight delay on I/O operations in order to sort them in a manner that is most efficient based on the physical location of the data on disk. This tends to work out well for slower disks, and older equipment. The delay can cause a higher level of latency as well.

In choosing your scheduler you have to consider exactly what the system is doing. In my case as I stated before I am administering application/database servers with a fair amount of load, so I've chosen the deadline scheduler. If you'd like to read into these with a bit more detail I'd check out this Redhat article (it's old but still has decent information: http://www.redhat.com/magazine/008jun05/features/schedulers/)

You can change your scheduler either on the fly by using:
echo <scheduler> > /sys/block/<disk>/queue/scheduler

Or in a more permanent manner (survives reboot) by editing the following file:
/boot/grub.conf
You'll need to add 'elevator=<scheduler> to the kernel line.

Using renice

Part of what my boxes do is serve up a web interface for users to interact with. When there are other tasks going on and the load spikes access to this interface can become quite sluggish. In my scenario I'm using tomcat as the webservices application and it launches with a 0 nice value (the normal user priority level in a range from -15-15 with lower being more important). The problem with this is that postgres also operates on the same priority and if it is loaded up with queries they are both on equal footing when fighting for CPU time. So in order to increase the quality of the user experience I've decided to set the priority for the tomcat process to -1, allowing it to take CPU time as needed when users interact with the server. I've done this using a rather crude bash script, and an entry on crontab (using crontab -e).

The script
--

#!/bin/bash
tomcatString="$(ps -eaf|grep tomcat|cut -c10-15)"
renice -1 -P $tomcatString

--

The crontab entry:

--

*/10 * * * * sh /some/path/here/reniceTomcat

--

All the above uses are the ps,grep, and cut commands to pull the process ID and then run the renice command on that ID by streaming it in. The crontab entry just calls it on a periodic basis to make sure the process stays at that priority. In the case of the above it's doing it every 10 minutes, but it can be set to just about any sort of scheduling. To read more on how to use cron scheduling check out this article: http://www.debian-administration.org/articles/56.

Analytics in Columbus?

I read something rather interesting in the paper this morning. It would seem that IBM is putting a new analytics center right in my backyard here in Columbus Ohio. This is big news for the city as it's supposed to bring in around 500 new tech jobs as well add credibility to the region as a tech center. Data/Business Analytics is a fascinating field, and is certainly worth a gander as it represents something significant for the future of the technology sector.

You see right now all the talk is about 'big data' and how it is stored, where it is served from, how its collected. But the lingering question that a lot of companies are now answering is 'what do you do with it once it's there?'. Companies such as IBM are taking this data boiling it down and using it to formulate strategies, see patterns of behavior that might be uncouth, and where consumer interest is going. This trend has caused a new type of IT job to exist that is an interesting mix of both technological, and business savvy.

What all of this means for those of us here in the Midwest is that its a step toward breaking the assumption that all of the IT talent is on either the east or west coast of the US. So all in all this should be a positive sign for the economy here, plus I must add it will be interesting to see the sort of talent that Ohio State is able to churn out for this field. To that end the Fisher College is opening up a new Graduate program for it, and the college itself is looking into something in the undergrad arena.

There should be some interesting times ahead for the Tech sector in Columbus.

Deficit Hawk: A cool federal budget app

So due to my being a bit of a public policy nerd on top of my enjoyment of technology I started playing around with an app on the Google Play Store called 'Deficit Hawk'. A friend of mine had suggested it, and I must say its a neat little app. It takes CBO projections, and possible choices that cover new revenues as well as cuts and allows you to attempt to set a budget plan. It's incredibly easy to use, and gives you a nice graph so you can see how you are doing.

The plan I created when messing around with it is below:

----- NEW SPENDING CUTS -----

$-88.0 billion over ten years - Add a Public Plan to the Health Insurance Exchanges

$-88.5 billion over ten years - Apply the Social Security Benefit Formula to Individual Years of Earnings

$-112.0 billion over ten years - Base Social Security Cost-of-Living Adjustments on an Alternative Measure of Inflation

$-2.0 billion over ten years - Charge transactions fees to fund the Commodity Futures Trading Commission

$-4.8 billion over ten years - Drop Wealthier Communities from the Community Development Block Grant Program

$-20.8 billion over ten years - Increase Fees for Aviation Security

$-26.5 billion over ten years - Increase Guarantee Fees Charged by Fannie Mae and Freddie Mac

$-241.2 billion over ten years - Increase the Basic Premium for Medicare Part B to 35 Percent of the Program's Costs

$-85.6 billion over ten years - Limit Highway Funding to Expected Highway Revenues

$-62.4 billion over ten years - Limit Medical Malpractice Torts

$-84.6 billion over ten years - Link Initial Social Security Benefits to Average Prices Instead of Average Earnings|Implement progressive price indexing

$-124.8 billion over ten years - Raise the Age of Eligibility for Medicare to 67

$-119.9 billion over ten years - Raise the Full Retirement Age in Social Security

$-642.0 billion over ten years - Reduce Growth in Appropriations for Agencies Other Than the Department of Defense|Freeze Funding at 2011 Level

$-610.7 billion over ten years - Reduce the Growth in Appropriations for the Department of Defense|Freeze Funding at 2011 Level

$-112.0 billion over ten years - Require Manufacturers to Pay a Minimum Rebate on Drugs Covered Under Medicare Part D for Low-Income Beneficiaries

$-3.6 billion over ten years - Transfer the Tennessee Valley Authority's Electric Utility Functions and Associated Assets and Liabilities

----- NEW REVENUE -----

$309.5 billion over ten years - Accelerate and Modify the Excise Tax on High-Cost Health Care Coverage

$96.1 billion over ten years - Expand Social Security Coverage to Include Newly Hired State and Local Government Employees

$241.4 billion over ten years - Extend the Period for Depreciating the Cost of Certain Investments

$70.9 billion over ten years - Impose a Fee on Large Financial Institutions

$456.8 billion over ten years - Increase the Maximum Taxable Earnings for the Social Security Payroll Tax

$1.2 trillion over ten years - Limit the Tax Benefit of Itemized Deductions to 15 Percent

$48.7 billion over ten years - Raise Tax Rates on Capital Gains

--------

In any case if you have an interest in public policy, and you enjoy playing around with neat apps on your phone or tablet I suggest giving this a go.

What is data protection really?

Data protection is a vague term that I've seen being thrown about. I've observed it being used in reference to data backup software, security software, network security devices, and well really to all sorts of software and hardware platforms and devices. I've come to the conclusion that data protection is some bit of all of these things and ultimately is about three factors: protection from loss, protection against leak, and the ability to ensure the data can be trusted.

Data loss is perhaps the most fiscally costly, and best known portion of data protection. It is here that you have your nightmare scenarios about all the customer data being gone, and your IT staff is rung up in the middle of the night to rush around to save what they can. These days most are conducting some form of data backup to cover themselves here, and the wiser of us are doing so to off site locations. Obviously this one is something that every organization should have covered at the minimum, though if you aren't sure this is taken care of go ahead and take a look at this post that kicks off a series on backup solutions, and this one which is a webinar about backup solutions.

Data leak is perhaps equally dangerous, though not as often thought about. This area of data protection involves the act of a malicious party gaining access to your sensitive information for some sort of nefarious purpose. A leak can harm not only those of whom the data belongs to or references, but also the reputation of the organization that has been breached.

The risks of leak are present in three stages, transmission, storage, and actions by the internal users themselves. It is important then that all transmitted communications have some form of encryption enabled, be they emails, backups, or other web transactions between customers or internal users. Sending anything across in plain text is just asking for a breach. These days most sites and services provide this, and just about every email service be it hosted or self hosted is capable of some level of protection here. Encryption of stored data is also rather important, and is increasingly so for those with laptops on the road. I can say that I've had my personal information exposed to the world on two occasions due to a laptop being stolen. It's a frustrating circumstance, and can cause all sorts of havoc for a business. It's important to do some for of encryption on your laptops, and its easy with the free solutions out there (like truecrypt).

Trust is important as well. When you are backing something up, or interacting with a web service how can you be assured that you are accessing data that is without malicious content. There are solutions that do this in some sort of piecemeal format be it scanning your computer to make sure it's up to date, or doing general scans to ensure OS integrity, but I'm not aware of a true comprehensive point to point solution. When interacting with questionable web services I might suggest running some sort of sandbox utility, such as that offered by avast, or sandboxie.

Why your SMB needs a private cloud

For small to medium size enterprises (SMB) the private cloud is the next natural step for their IT infrastructure. Their employees need access to data no matter where they are, and the employer needs them to be always on, and always connected. The organization is going to want to accomplish this goal in a manner that is cost effective, secure, and able to be owned.

Employees need their data. Almost every job these days from sales to engineering requires access to either some form of CRM (customer relations management) interface, ticketing system, knowledge base, or some other database oriented solution. Setting up some form of private cloud environment that is accessible from anywhere is the key to giving them what they need. It's becoming easier and easier to set up an infrastructure these days as well. You really just need some blade servers and some sort of virtualization platform tossed on top in order to deploy virtual machines to serve your purposes. Ultimately your deployment becomes less about hardware and more about services and software.

Your infrastructure cost should go down over time as well. While you wont be able to layoff your IT team, they will be able to automate more tasks, centralize more of the infrastructure, and spend time on things that develop the business instead of fighting fires. If you set up more centralized infrastructure with a proper disaster recovery and business continuity plan utilizing backup essentials you can create a resilient and accessible set of services for your employees and customers.

Ownership is also an issue of great import. One of the next big things in IT is going to be the 'how' in determining what filters to place on all this data that has been collected on users. The answer to that question is going to bring up a lot of privacy concerns, as well as the issue of user rights v. owner rights. Through the use of a private cloud you can avoid some of those pitfalls that you would run straight into by going to a public provider. The key portion of this is that in a private cloud scenario your data is housed in something you physically own. If you need to pull something, or migrate the data away you can and without being impeded by the governance of another organization outside of your own.

In owning the infrastructure yourself you can also lessen the risk of data leak. You have the opportunity to set your own strict testing and security standards. In the hands of another company your are subject to what ever policies they have dreamt up, be they for better or worse. This sort of thing is especially important if you are storing proprietary data, or personally identifiable customer/employee information.

The move to the private cloud is a natural step forward. We've witnessed over the past twenty years the empowerment of the personal desktop, which lead to the beginnings of a collaborative office environment. Now we're moving to each organization having it's own private cloud of computing power, giving them further capabilities and control. Your organization can move from a collaborative office environment to being that of a collaborative organization. No matter where your employees are, they can communicate and in a manner that is under your control. 

Backup Essentials Part Four

We have now finally arrived to the conclusion of the Backup Essentials series of posts. Part four may be the most important one to consider, as it is the factor that will have the single largest impact on your customers. For our conclusion we'll be covering testing and expectations management, and while these things don't sound as technically exciting as the previous parts they are critical to your business. In covering this topic we'll hit four key areas: Bench marking, availability, retention compliance, and configuration auditing.

Bench mark testing is critical in gathering data for you to use to set reasonable expectations for your customers. Periodic testing of exactly how long your backup solution takes to conduct operations is imperative if you wish to be able to speak credibly to a client about your capabilities. I am always pushing clients at my work to conduct test restores from random client machines at least a couple of times a month in order to keep track of restore time frames. Taking that data and coming up with an average of sorts will allow you to say with some confidence the amount of downtime a client could expect in a disaster. If your clients are attempting to form a service level agreement (SLA) it's especially prudent to have this data available to you.

Just as important is to check the integrity of your data. We gain hark back to conducting test restores, but in this case we have to take a step further than just testing I/O speeds. For example if you are backing up a SQL database the step has to be taken to attempt to do a RESTORE from the dump file (on a dev/test environment). The ideal test gets as close to a real world disaster scenario as possible, and even better if you train your associates to document practices and problems along the way. The data gathered from these tests can form the bedrock of a larger business continuity strategy as you figure out what resources need to go where, and the best placement for your experts. I generally suggest conducting some form of disaster test once every month or two, and at the minimum once every ninety days.

Let us also not forget about retention policies. It is important to double check your backup solution to make sure that any storage retention policies that you have set up are being enforced properly. The last thing you need is to either have your space taken up needlessly (driving up the cost of the solution and decreasing performance), or not have a set of data due to a retention policy that is too tight and unchecked.

Lastly we have configuration auditing. It's useful from time to time to go back through the rules that define what data is being backed up in order to make sure they are still valid. Over time companies move files around, migrate databases and email stores, and well ultimately things are always changing. It would be rather embarrassing to find that one was backing up an empty directory because the backup parameters haven't been checked in months. This is something that should be checked at a weekly or biweekly rate and requires close cooperation with leaders in other parts of the business in order to ensure accuracy.

The conclusion you should draw from this is that testing, auditing and compliance are terribly important. Conducting these properly will allow you to set reasonable expectations with your customers, give you confidence in your technical solutions, and will give your customer reason to have confidence in you.

Backup Essentials Part Three

It's time for part three of this series. What we're discussing here is what to keep in mind when selecting your backup solution. The factors we'll be discussing include: data availability, administrative cost of the solution, portability, and security.

Data Availability
This factor is always a balance that has to be struck between how quickly you need to get at your backed up data, and how resilient you want your solution to be. For example on one end you could merely have your data backed up to a drive attached to the source machine. This would give you the fastest access to it, but at the cost of a single point of failure. On the other end of the spectrum you can have the data backed up to offsite location(s). This would give you the most protection as any harm to your office is far from your data, but also increases the amount of time required for access. Ultimately you either want to have some form of both, a solution that is highly portable (physically moving storage and compatible with your equipment) or even better has all of the above.

Portability
Perhaps one of the more overlooked factors is how portable your backup solution is physically. This becomes a point of particular importance when you have backups that are at an offsite location. For most organizations that aren't in the 'large' category it is unlikely that you'll have a fiber link running across to your offsite location. So if/when an emergency strikes your main office you need to be able to physically move the data and then make use of local link speeds. Remember this when choosing your solution as the ability to do this can vary greatly depending on what sort of hardware requirements are involved, and the manner in which the software operates.

Administrative Cost
One of the banes of the small IT shop is the near impossibility of its associates to truly specialize. The needs of the business, and likely size of available staff tend to demand that the technician/administrator/engineer in this role be a generalist. It's important to keep this mind when picking your backup solution, as it would be a terrible situation to need to conduct a restore, but have no experts in its use to conduct it. If the product requires a significant amount of additional training to operate it can end up costing the business significant resources. The cost is incurred doubly so if the training itself isn't of a certain caliber as well. The long and short is that 'simple' and 'intuitive' should be your watch words here.

Security
The manner in which your data is stored and transmitted is obviously important. Looking around you are likely to find that most if not all of the notable solutions out there at the very least transmit data in an encrypted manner between target and source. The differentiator here is going to be the manner in which its stored after the job is complete. Do you have the option to encrypt the data? If you use third party encryption does it effect their storage format? Checking this functionality is especially important for those of you in the medical field who have to comply with HIPAA (The Health Insurance Portability and Accountability Act). Also consider the discrimination of data here. For example if I log onto one target machine am I capable of pulling data down to it that belongs to another? While it may sound convienent if the answer is yet, it does create a bit of a security vulnerability if one of your target machines were compromised. If your business in a field with confidential data you certainly want to make sure that you have the ability to discriminate and control which clients can access which data.

Truth be told an entire article could be written about each of these factors. These basic thoughts act as an effective guide in researching backup solutions and implementing them into your infrastructure. If anyone would like to discuss the topic in greater detail with me, feel free to comment or shoot me an email: koch.ryan@gmail.com.

Backup Essentials Part Two

For part two of this series we will begin the planning phase. In this phase we'll take an inventory, categorize our data, decide backup types, figure out our backup window, and figure out storage requirements. For this example we will use a small test environment representing the size of a 10 employee company as mentioned in the introduction.

First we'll knock out the inventory. The test infrastructure has a mix of desktops, servers, and remote laptops. I've gone ahead and included a list of these below:

6 desktops of a similar configuration (general user machines)
1 SQL Server (HR)
1 Exchange Server
1 Web server
1 Domain Controller
3 Remote laptops

Having this list we will now need to set priorities. To do this we need to break down an analyze what this business would need to operate, and what it has that merely makes it operate better/more efficiently. While simple, the prioritization below will help us in making decisions on backup types as well as scheduling and storage.

Need to operate:
Domain Controller
Exchange Server
SQL Server
Web Server

Makes business more effective:
Desktops
Remote laptops

Next we need to discuss what backup types are available to us. In general the various backup solutions will give you the ability to do file level backups, service/application backups, and image level backups. In general the decision of which level to use with which machine depends on what expectation of recovery time you have, the location of the machines, and what the machine is being used for. In our case the desktops, and the remote laptops will require a file level backup. The servers will all require an image level backup, a file level backup, and a service level backup.

After deciding what backup types we're using we also need to decide how this fits into our disaster recovery strategy. Do we need any of this data to be backed up to an offsite location? In our scenario it seems clear that the SQL Server,Exchange Server, Domain Controller, and Web Server all need to be backed up to a remote location. These services should be able to be brought backup at a remote office or home office in the event that the main office is brought down by a disaster of some sort for any extended period of time.

The last two factors to consider are scheduling and storage. For scheduling you have to figure out what your organization's 'production' hours are. This will allow you to create a backup window that will exist to minimize the impact your backup jobs have on the ability of  users to operate. For example if the office is open from 8am-5pm then an effective backup window might be from 7pm-6am. This will give a 2 hour buffer both on the beginning and ending side of the window in case someone stays late, or a backup job runs too long. A lot of the decision is going to be unique to your organization and its needs. Storage is also going to be a bit unique as the requirements are going to be different from backup solution to backup solution. Depending on if you choose to go with an appliance, or software you may or may not have to purchase your own hardware. To fully implement a backup and recovery policy however you are going to want to arrange for some sort of storage system on site at your office, as well as another one at some off site location be it an off site data center, or even someone's home office.

That about covers are basic planning stage. The next article will cover what we need to consider in order to pick and implement a solution. There may even be pictures in that one (screenshots).

Backup Essentials: A 4 part series

As many of you aware one of the topics I fancy writing about is backup and recovery. To appease that desire I've decided to write a 4 part set on what's involved in planning, picking and implementing a backup solution. While you've seen this talked about over and over again, we're at it here because it is truly important. Data loss and downtime is the same as tossing cash out the window, and most businesses can't afford it. Besides let's face it, now matter how well your infrastructure runs, Moore's law will eventually strike and it's best to be prepared.

The series starts with this introduction, an outline of what we'll be trying to accomplish. For our walk-through we will use the example of a company made up of 10 employees. Our goal will be to come up with a backup strategy to propose to this 'company', once they've accepted begin implementation, and then finish off by conducting some testing and concluding on its effectiveness. Our scenario will also involve two separate customer sites, and with machines ranging from standard Windows 7 desktops to an Exchange server and a SQL server.

Check back for the next part where we begin the planning phase. In the mean time I need to finish building up the test environment to use as screenshot materials. This should an interesting set of articles and hopefully will help some of you out there in the course of your careers.

Users and Security

I do apologize for the delay in getting a new article out. Between the power outages here in Columbus last week and the catch up work that followed the plate has been rather full. Today I started giving some thought to security, and specifically how the actions of Users can effect your policies and planning. It seems that no matter how much care and caution one puts into a great security set up, there is always one weakness to root out and that is User behavior. These beings seem to be able to defeat the greatest of security infrastructure practices and are somehow able to throw a wrench in the most finely configured of ecosystems. And so here is a bit of an overview of some thoughts on how to manage user behavior.

But how to do you prevent the user from accidentally breaching your security? It's not so much a question of control, as it is a question of influence and education. Ultimately the majority of user mistakes are due to a lack of instruction, or knowledge of good practices. It's understandable, most of these users be they internal or external customers have other things to do and have other concerns that have been given a greater priority. As the IT Engineer/Analyst/Manager it ends up falling to you to be the one that breaks this shell, and instructs them on what to do. The importance of taking the time to do this is only going to grow over time, especially since these users are now even bringing in their own devices and conducting business on them. You now not only have the possibility of company infrastructure being mucked up, but also of corporate data leaking through external devices.

How do you accomplish an educational role? It's about the soft skills here. You need to schmooze a bit with other departments and employees in order to gain their trust and cooperation. The idea of IT education needs to be sold as a value added piece, something that will ultimately save the company time and money. I find that his process is very similar to that of starting a new workout program or lifestyle change. It's best to start off just getting the first session/meeting/etc and then the next one. Once you can get a routine going the ride is much smoother, and you've accomplished an institutional change. An example of this might be to send out a weekly email newsletter and partner that with a monthly 'class'. Once you get solid practices in your users' minds you'll start to see improvements in their behavior and perhaps less security oriented incidents.

Outside of purely IT type education, you also need to make sure that policies are clear and published in as many places as humanly possible. In crafting your policies I would suggest taking heed of the manner in which intelligence agencies operate. Users should only have access to infrastructure pieces they have both 'clearance' for and a 'need to access'. This is very similar to the 'clearance' and 'need to know' principal of intelligence that is used to keep information leak to a minimum. Documentation for this principal is critical as the rest of your organization needs to understand the structure for how access is granted, as well as procedures to gain clearance when needed. Along with policies on access a proper acceptable use policy is recommended. The document should detail exactly what behaviors are frowned upon, and what behaviors are considered acceptable practice.

Lastly, and perhaps the most important piece of this is user involvement. If you want to maintain a credible IT department you need to make the users feel involved and keep them in the loop. Regular and predictable communication with as much of your organization as possible will create an environment in which users don't see the IT department as just some strange offshoot of the company that just tells them they can't do things with their system. By creating these venues of communication you'll be able to create a situation where you users aren't following described practices because you said so, but because you convinced them that they *want* to. This type of shift can only lower risk to both your data and your infrastructure.

Backup 101

I swear I'll post something with some substance tomorrow, but today I wanted to promote the recording of the Backup 101 web seminar I hosted today. My presentation covered an overview of backup essentials that are required for any effective IT knowledge set. I hope to do more sessions like this on a range of topics as time moves forward. If you guys have any requests please feel free to drop me a line.

Backup 101 Seminar

Just quick blurb for you guys today. I'm hosting a Backup 101 seminar/class on Thursday 6/28 that you might find fruitful. It will have a quick slideshow presentation followed by what I hope will be an energetic and fruitful conversation. Please click here and register for the webex event.

SMB IT Continuity

You might be thinking that as a small or medium sized organization that business continuity isn't that important to you. In particular with your small IT department you may not realize just how important this sort of planning can be to your organization's well being. The truth of the matter is that planning for business continuity from an IT perspective isn't horribly complicated, but does require thought and careful consideration.

In making a plan there's only a couple basic categories you need to worry about. You need to consider your personnel, your physical infrastructure, and your data. Depending on your size a list made of these portions of your organization may fit on a single page, or it may take an entire book. In either case it is important to take an inventory of what you have. For example a company of 5-10 employees might have a list like this:

People:
Fred, Systems Administrator
Joe, HR/Accounting/etc
Linda, Sales
April, Sales
Alvin, Sales Manager
George, Owner
...

IT Equipment:
1 Server (Virtualization host)
6 Desktop Computers (Model: xxxx)
1 Color Printer (Model: xxxx)
1 B&W Printer (Model: xxxx)
...

Data:
1 Email server VM
1 Domain controller VM
Local desktop data (Users)
1 Webserver VM
1 Financial Management VM (Terminal services)

 As you can see, even in a company that small there are a lot of things to worry about if something goes wrong. And if the size of that company were to increase over time, that list would only grow in size and complexity.

Once we have some semblance of and idea of what we have, we then have to figure out what our priorities are. By this I mean we need to decide what people and things are absolutely required for the business to run, and what after that assists in making it perform more efficiently. Again I suggest coming up with some sort of list but this time ordering by priority, or adding a priority designation. In the case of the company above it looks like as far as equipment goes the most important bit is the server. For the data they likely would need at minimum the Domain controller VM, Email VM and the Financial Management VM data in order to begin operating. A list describing would look like this:

Critical Needs:

IT Equipment:
Virtualization host
2 Desktop computers

Data:
Domain Controller VM
Email VM
Financial Management VM

After that you would list the other stuff in increasing categories of importance. What this allows your organization to do is prioritize its time in what it's working to bring back online in the event of a catastrophe. You avoid spending time on portions of your infrastructure that may be able to wait and thus could end up saving the organization a substantial sum of capital.

So knowing what's important, and what you have is all well and good but what are we going to do about preserving it? Emergencies are unpredictable (obviously) and you need a strategy in place that allows you to get back to an operating status in hours (not days). You need to have an offsite location to store a backup of your data, and perhaps a few pieces of hardware to get you by in a pinch. When I say offsite location that doesn't necessarily mean a sophisticated data center or even another office itself. If you are a small organization your cash flow might not allow for something that elaborate, and in that event you could even use a residence. It just has to be somewhere that  you outfit with a decent broadband connection, and reliable electricity.

Once you've picked and outfitted a proper offsite location you have to come up with a data backup plan. Using the backup solution of your choice you will need to maintain an offsite copy of anything deemed important. I suggest using a two tiered approach to going about this. First start by taking a local backup of everything at the file level, image level, and application level (Database, mail, etc) giving you one solid copy. After this you'll want to set up file level backups (and smaller application level) that occur across to your offsite. After you have started moving data off site regularly you will want to pair that with an image backup (and any larger application level backups) that is dumped to a local spot. This scenario gives you the fall back you need in the event of a disaster, but also gives you the easy access you need for smaller incidents.

As you may be aware by reading this blog in the past I do have a bit of a favorite when it comes to products that help out with business continuity and disaster recovery planning. The 3X Appliance is an excellent solution for this sort of thing and certainly deserves a look. They've posted an article here that talks a bit about its usefulness for business continuity.

Quick Microsoft SQL restore how to

When you have to conduct a restore of a Microsoft SQL database it can be a rather scary time wrought with stress and angry calls from a client or IT manager. Sometimes you just need a quick look at a how to just to make you calm down and realize its not so bad. So to that end, I present you a quick how to on restoring a Microsoft SQL database from a backed up BAK file. The first portion of this how to will cover restoring from a 3X Appliance as that's what we use here at the office, but the second half covers the use of the Restore command. So, if you have no 3X appliance and backed it up some other way you can obviously skip down to the second part.

The Steps:

1. Complete a Restore of your backed up BAK from your 3X Appliance.

To do this we just need to go into the web manager, to 'client management' then click on 'manage clients' and wait for the page to load. Once there we can select the client in question and click 'Restore'. The page just before clicking restore should look like this:



 Next we'll want to go through the restore wizard. There are a couple of ways we could go about the restore. For our purposes I suggest we go ahead and do an SCP restore. This will allow us to create a user name and password on the vault with which we can use to download the BAK file. So we'll select Microsoft SQL Server as the restore type, and select SCP under 'Restore To'. Then you'll just select your backup set, and what date you want to go back to, and then which databases you wish to restore. Once complete you'll get this lovely screen.




Note the user name and password from this screen as we'll need it for the SCP session. On your client machine open up any SCP client of your choice. Your login screen should be filled out in a manner similar to this:

 

For the port number you'll use what ever you have configured the Backup data port to. In my case this is port 5589. Also note that I have selected SCP as the file protocol. Once successful you'll be greeted with a simple file system screen. You will also want to adjust the Server wait timeout under the connections section that appears when you select 'Advanced Options.' I suggest setting it to a value of 999. All you need to do is download your bak file.


Step 2: Restore the BAK to your database using command line. This is the part I imagine a large group of you have skipped to. Now this is actually a fairly simple process but must still be completed with care. While I know that it is possible to conduct a restore using the management studio UI, I prefer to do this using osql and the command interface.

To complete the restore you will use either 'osql', or in the case of 08R2 'sqlcmd' to connect to your database. It is suggestible to go ahead and use the -S and -U flags. Also bare in mind that if you are on a cluster situation that you'll want to use the name for the cluster as opposed to the computer name of the server you are on to resolve the instance.

Once connected we'll be making use of the restore command. This command is actually very simple to use and is pretty self explanatory. The only thing we need to clarify is that since you are restoring from a BAK you'll need to use the 'restore from disk' option. Here is an example:

RESTORE DATABASE ImportantStuff
   FROM DISK = 'C:\Users\Administrator\Desktop\ImportantStuff.bak'

Now if the backup solution you are using supports versions and  you aren't conducting manual differentials of the database the above should suit you just fine as is. This will conduct the restore with recovery. If you are using a manual dump with differentials you will want to specify which file in the BAK you are restoring from. To get more information on this I suggest reading the msdn article here that covers the restore command in greater detail. That about covers a simple restore.