Maintaining Your Technology - Don't Be "That Guy"

You buy a car and you regularly change the oil and take it in for its mileage-appropriate checkups. Why? Because you don’t want it to die on you on the highway on the way to closing your biggest deal. Well, the same is true for your IT infrastructure – – your servers, desktops, firewalls, switches, wireless access points, etc. – – they all need regular maintenance or they will, per Mr. Murphy and his law, break down at the most inopportune moments.

In my nearly 20 years as an IT professional, I’ve seen numerous instances where some regular maintenance and monitoring would have caught minor issues before they escalated to the level of a business disaster. And, in many cases, things started off in the right direction with the purchase of good hardware, a proper warranty and backups in all the right places. But this is where many businesses stop – – just because you got things off to a good start doesn’t mean you can ignore the maintenance. Is that server getting too long in the tooth to be trusted with your business’s valuables? Is your firewall out of warranty and no longer receiving critical security updates? Remember that car in the first paragraph? It may be a Mercedes and it came with the best oil money could buy but it won’t be performing very well after 100,000 miles without regularly replacing the oil and having a mechanic look at it periodically.

One story in particular stands out in my mind – and you don’t want to be this guy, trust me. He started off on the right foot – – he bought a solid server for his small business, with a good warranty and the right redundant storage hardware (a RAID configuration, in technical terms). He also bought a solid backup drive and changed the backup cartridges on a regular basis. But what he didn’t plan for was regular maintenance or monitoring. He just wanted to call us when he needed us.

Well, 6 years later, he needed us. He called our helpdesk and said that his server was dead and his business was completely at a standstill. A few hours later when we were onsite, we diagnosed the issue – – two of the three redundant hard drives had failed which caused the server to crash. A server with that kind of redundant hard drive configuration can withstand the loss of one hard drive but it cannot survive two hard drive failures. If the server had been properly monitored, we would have been alerted to the first hard drive’s failure and replaced it before the second one failed.

“But, what about his backups?”, you say. Well, he was dutifully swapping the cartridges every day but he wasn’t checking the backup job logs and he didn’t notice that they’d been failing for over 4 months due to a separate hardware failure.

So, to recover from this, we had to send the failed RAID5 hard drive set out to a data recovery service to the tune of about $3000 and, even at that cost, his data was only partially recoverable and he was without his data for almost 3 weeks. The damage to his business (and his blood pressure) was far greater than the cost of having a proper maintenance and monitoring in place.

While this case may sound a little extreme, it’s really not that rare. Information technology that gets ignored is information technology that breaks – – it’s just a matter of time. The combination of an aging, out of warranty server and lack of any proactive maintenance or monitoring can cause a lot of headaches and worse.

Here are some suggestions for things that you can do that may not eliminate ALL downtime but they will greatly reduce the chance of a true disaster:

Proactive Monitoring – – all critical equipment and services should be properly monitored so that when any alerts are created, for example if a failed backup occurs, they can be responded to.
Regular Maintenance – – The firmware on your network device and operating system updates should be applied as often as recommended by the manufacturers. There’s nothing worse than finding out that a fix was available six months ago for the issue that took your business offline for two days.
Active Warranty – – All of your critical hardware (and software!) should have an active support warranty that is properly sized for your business needs. When the server is down, you don’t want to have to go scouring e-bay for parts.

If you want to know if your business is properly protected, let us come by to perform a free assessment.