• Ignoring the patch

A few weeks ago I was watching the news when a reporter started talking about a new computer virus that was spreading through the internet. Just think, main stream news media talking about a virus that some wacky kid had probably created to impress his friends. A few years ago, the idea of a reporter talking about computer exploits on the six o’clock local news was unheard of. But, ever since the Zombie Denial of Service attacks in the late 90’s, news agencies have realized that people actually care about what hackers are doing. People are interested in keeping their own networks secure and at the same time have a morbid fascination about the havoc that has been wreaked on others. You would think with this kind of publicity, security would be at an all time high. Hackers would be on the run, knowing their worms and viruses would fail due to the media superheroes that were there to warn the public ala batman signal in the sky.

So, why have worms and viruses caused more damage this year then in any other year? Security technology is obviously better, user education is far greater and public awareness is unprecedented. Yet, a computer virus breaks out and suddenly we’re pulling game plays from medieval medicine. Whole networks are amputated from the internet to stem the infection. “Kill your email without opening anything!” is the cry from management. Sure you might lose a few important documents, but it’s in the name of security!

What happened? Well it turns out that instead of becoming more vigilant, users have become complacent. Users feel that the idea of viruses and worms spreading through a fortune 100 company with more money spent on security then the gross national average of some small countries is just preposterous. Anti-Virus, Firewalls and IDS have all but insured users that they have nothing more to worry about. Basically these users are riding the Titanic of the internet and it’s full steam ahead. Add in the news media who realized that stories about new patches really don’t grab the viewers the way network mayhem does and now your media superhero is playing a very twisted role. Sure if people were made aware of a patch and actually knew how devastating things could be if it is not applied, there would be a lot less virus and worm issues-but what kind of ratings come from that? On the other hand, a network outbreak that brings down several large corporations, freezes communication between critical servers and causes traffic lights to fail in 5 states, now that’s some news. To go back to the Titanic, it’s seeing the iceburg, getting a cup of joe and positioning your camera for the impact photo.

Now don’t get me wrong, I am not saying that it is the news media’s fault. Certainly they could do a lot more to warn people about potential security issues, but they are just one part of the puzzle. Ultimately it comes back to the IT department itself. Look at each virus and worm that comes out. In almost every single case, there was a patch that had been released by the manufacturer at least three weeks prior and in some cases up to a year. Code Red for example, a worm that wreaked havoc a couple years back, exploited a vulnerability that had a patch available for just under a year. Unbelievably, that worm is still running on the internet to this day exploiting additional systems. The most recent attack which brought down thousands of networks had a patch released just under a month prior to the outbreak. In other words, had these systems just been patched, they would not have been vulnerable and the outbreak would have been thwarted.

The majority of the vulnerabilities discovered and made public come from security companies. These issues are brought to the manufacturers attention and then the public is notified and given a patch to resolve the issue. At that point it becomes the responsibility of the IT department to do what it takes to resolve the issue on their network. If you are the IT manager and you’re thinking to yourself, we have over 5000 computers in our network and the idea of patching everyone of these systems with each new patch is beyond the realm of possibility, you are not alone. On the average, when I conduct a security assessment, there are at least 5 critical vulnerabilities found on each system tested. In fact the number seems to increase the larger the organization. It is obvious that as corporations continue to tighten their budgets, the IT staff is left having to juggle what is important and what can be put on the back burner. Often times, patches are the first thing to be shelved and once out of sight, they are easily forgotten. User training is next to slip, ultimately adding a one two punch to the otherwise secure network. Even if you have the budget, the logistics of knowing what systems have issues, what patch is needed and then actually applying the patch to all these systems can seem overwhelming.

Though there is no perfect solution to keeping up with the patches needed to secure a network, there are several options available. Over the past few years I have come across some organizations that just astounded me; they were current with all patches on over 95% of their network and in most cases were 100% secured against high level vulnerabilities. These companies ranged in size from small to large, limited budget to VC funding gone mad and limited IT staff to IT staff playing Quake to pass the time away. So, what makes them different then the rest of the corporations getting worked over? Simple plans with strict enforcement.

Option One (Manufacturer auto patch) Manufacturers such as Microsoft and Redhat offer automated patching. Though this can be a cause of concern, when done properly this can be a no cost way of maintaining a secured environment. Because so many defective patches have been released throughout the years that ultimately turned out to break other applications once applied, engineers have become gun shy at just applying a newly released patch. I personally couldn’t agree more. On the other hand, you can’t just ignore the patch and wait weeks or months to hear if others had a problem with it. The solution more and more organizations are turning to is the sacrificial systems. There are two types of sacrificial systems, production and test. Production systems are currently being used by employees in the organization. These systems are never mission critical and should be used by employees with low level job functions. A test system is built for testing purposes only and has no real purpose other then to be beat up. Choosing what’s best for your organization can be difficult. While test systems are nice and insure no down time, they sometimes can give a false sense of normalcy as some application may be overlooked during testing. A production system has the risk of going down but on the other hand, the user will go about their daily business and it will become very clear if the patch has caused a problem. Once you have established what’s best for your organization, then it becomes just a simple matter of testing the patch on your sacrificial system. After the test is a success, send an email to all users with specific instructions for auto patching their system. Larger organizations do this in groups to reduce confusion and support calls. The key to the success is detailed instructions in the email or memo. The more information and step by step instructions, the less chance for mistakes. Strict enforcement is the key to this particular options success. It must be understood by all users that this is not optional and that when requested, they must follow through with the patch.

Option Two (Patch Notification) Many organizations are running software that does not fall under the auto patch features of the large manufacturers. These third party products still have vulnerabilities and require just as many patches. Some organizations also do not like the idea of anything being auto applied and want complete control. For these organizations there is Patch Notification. There are many levels of Patch notification from free services that send out emails with every vulnerability that comes out, to very sophisticated fee based services that allow complete management of the patch process down to each specific system. Selecting the service best for your organization comes down to the amount of time available to maintain the process. If you are a single IT guy trying to keep up with email notifications that are about every exploit that comes out, you might have a very hard time figuring out just what systems are vulnerable to what exploits and what patches are required to fix those vulnerabilities. Picking the service best for your organization should be based on the hours per day the IT department spends looking for vulnerabilities and patches compared to the amount of salary paid to this staff to perform these functions. Now obviously this calculation will be incorrect if no time is currently being spent looking for vulnerabilities. In that case, the calculation becomes the cost of compromise times the risks associated to the network. This number is always far greater and is the reason why it is so important to secure the network in the first place. If you are finding that you’re spending thousands of dollars a year just trying to keep informed of the latest security issues that effect your network, you might want to consider some form of Patch Notification. Patch notification does not apply patches for you. Simply being told about the patch and systems vulnerable is a large step but it still comes down to applying the patch. If you are an organization that wants the control of what software is loaded on what system, you have to make the commitment to follow through. Scheduled time each week to address the necessary patches is mandatory. It should be clear how long it will take to apply a single patch to every system. Sometimes a team is created for the sole purpose of keeping patches up to date. As with auto patching, it is still a good idea to test any patch before applying on a large scale.

Option Three (Automated Patching) Some large and small organizations have found that manual patch application is not realistic. Going from system to system would take too long, users are unable or unwilling to help with the load or the level of work has just gotten out of hand. In these cases organizations turn to centralized automated patching. Automated patching allows a system administrator to review from his console all systems on the network, apply patches to those systems as needed and keep an audit trail of each systems patch level. As with any type of remote maintenance there are pluses and minuses. The reviews on companies offering these services seem to range from great to awful and depending on who you believe those reviews can be about the same company. The obvious advantage is ease of use. To apply a single patch to 5000 systems all at the click of a button is a dream come true. To be able to look at a single screen and see all systems on the network and know who is currently at risk and what needs to be applied to fix it makes sense. The limitation to this ease of use generally comes down to limited applications. Obviously Microsoft is covered but many Centralized Patching Applications do not cover unix and those that do generally have very specific software that is covered. Some arguments are made the many of the vendors that are covered by Centralized Automated Patching already have their own auto patching capabilities built in and therefore it is an unnecessary expense. Obviously this is going to come down to the organizations total needs. Centralized Automated Patching is not for every organization any more than the other options I have laid out. It simply comes down to choosing what’s best for yours.

All of the above options are just that, options. There is no sure fire answer that will resolve your network woes. As you have probably noticed by now, I never touched on mission critical servers, as that is a whole separate issue where in my opinion the only possible answer is manual application of patches. New vulnerabilities will continue to be discovered on a daily basis, manufacturers will continue to release patches to resolve these issues, hackers will continue to write worms and viruses to exploit these issues and the main stream media will continue to hype the chaos that ensues. Firewalls, Anti Virus, IDS, and all the other security devices available have been unable to stop these issues. Patches are released for a reason. Patches stop the vulnerability completely. If a patch has been applied, the vulnerability will not be exploited. Security doesn’t get much simpler then that.