Because of the way software and firmware are developed and released, they sometimes, if not often contain bugs or dysfunctional code that creates problems with functionality and security. That’s why Patch Management is critical. In fact, between the years 2003 and 2005, more than 2,000 vulnerabilities were identified per year in an average system, which resulted in approximately 7 vulnerabilities per day! This means even a single server business will be dealing with several bugs each month. To find out more about specific bugs or vulnerabilities that have been found and published in the systems you administer, check out the National Vulnerability Database.
When the developers become aware of such bugs or vulnerabilities, they hurry to release patches or changes in the code that fix the problems. Some developers require IT personnel to keep up to date with patches by checking in to their website periodically, while others automatically notify the user that an update is needed. Either way, the patch must be downloaded and applied in a timely manner because hackers are eager to learn of these bugs and back doors into systems before patches can be applied. This creates a contest between those who must perform patch management safely and the hackers who seek to exploit the holes.
The decision-making process surrounding whether and when to patch can be somewhat complex, and it is difficult to keep up with patches that are released as frequently as every day. This results in patch management being a wide-spread area of weakness for a lot of companies and IT departments, which is why it is important to at least know the basics of patch management.
Though this is a large and complex subject that can be difficult to master, knowledge of a few simple procedures will go a long way in securing your company’s resources.
Assess your Patch Management program
Accountability
The National Institute of Standards and Technology (NIST) recommends creating at least one Patch and Vulnerability Group (PVG) within even a small company and having several hierarchically structured PVGs in a large company who are responsible for executing the patch management program. The designation of individuals to this team creates accountability such that patch management is made a priority and doesn’t fall through the cracks on some low-level system administrator’s head, even though that person may ultimately be responsible for implementing the patches.
The PVG is charged with a number of responsibilities not least of which is taking inventory of all the IT resources in your company’s system. It’s important to capture hardware, which may require firmware updates, and software, including applications and operating systems. This puts them in a good position to also be responsible for monitoring for updates and patches to this equipment.
Prioritizing the order of implementation of the patches is also PVG responsibility. Hopefully, your company has the resources for the PVG to create a database as a repository for this information so all the patches/remediations can be logged somewhere accessible to those who need to know about them. Usually an enterprise patch management tool will come with this kind of database.
Procuring Patches
When procuring the actual patches via download through a vendor’s website or other source of distribution, the PVG must be vigilant in determining the authenticity of the patch. There are methods that are automatically used, but often the PVG will have to compare, for example, a checksum to that listed on a vendor’s website to ensure integrity of the data.
The PVG should also run a virus scan of the patch to make sure it isn’t infected. This means the virus signatures must be up to date on your antivirus software. It’s still possible for a brand new piece of malware to infect your system, but doing this will add one more layer of security to your patching procedures.
Pre-Deployment Patch Testing
Patches often replace integral system files and alter system security settings in a way that is too complex to determine before implementation by merely examining the code. That is why you must test the patch in a NON-production environment before going live with an update. Patches are usually released as quickly as possible without the usual testing that production releases go through. They can cause unintended consequences like crashing other programs on your system or uninstalling other patches.
Using standardized configurations is paramount for proper testing to be performed. The PVG is responsible for this pre-deployment testing on equipment that has the same configuration. Imagine how complex this would be if everyone in your company had a customized machine! And the vendor cannot possibly test every configuration their patch may be deployed to, especially when time is of the essence. So, make sure you test the patch in a sandboxed environment with the same configuration as your system before you start deploying it.
Pre-Deployment Research
What if you really don’t have time to test the patch yourself? Even if you do, it is wise to search online for others’ experiences in deploying the same patch. They may not share your environment, but it will at least give you an idea of what to prepare for. If there is a fix for a problem, it will likely be published. If there isn’t, you will have to decide whether the advantage to patching outweighs the problem it creates. Often, waiting for an updated patch release can solve the initial problems presented.
Factors to consider when determining whether to go ahead with patching or not include the threat level, the risk of compromise, and the consequences of compromise. Public facing servers like web servers or government owned equipment is likely to have a lot more exposure to attacks than say your company’s internal intranet server. So, take care of your Internet facing and sensitive data devices first. Also, you have to take into account how easily the vulnerability can be exploited. And finally there are the consequences of not patching, which could lead to a compromise. Determine the actual monetary cost of a breach as well as the damage to your company’s reputation when you decide whether to patch or not.
Patch Implementation
The PVG must supervise the implementation of patches that must be done manually, and they are also to execute automatic deployment of patches using enterprise patch management tools when possible. To that end, they should also automate the updates of software and firmware where possible. Regardless of the size of your company, you should be migrating over to automated patching as much as possible, as patching each system by hand is onerous and time-consuming.
When implementing the patches, a phased approach works best. Deploying the patches to a smaller group of individuals first prevents widespread mayhem if the patch causes a dysfunction in the system. The order of operations is usually standardized desktop systems and single platform server farms first. Multi platform environments, nonstandard desktop systems, legacy computers, and computers with nonstandard configurations are next.
Also, the patch should be applied to ALL systems that require it, not just the ones for which its absence represents the greatest threat to security. Sometimes it is not a patch that is required, though. You may need to change a configuration or remove bad software from your system. All remediations must be made in the proper sequence indicated to avoid known issues, as well.
Post Implementation Process
The PVG distributes information about what vulnerabilities exist and remediation efforts taken to local administrators, so they are prepared for any negative outcomes.They also train local administrators in how to apply the vulnerability patches by hand when necessary. This is critical for obvious reasons. The PVG may only be able to focus on the major applications of your company due to budget constraints. Local administrators are aware of all applications they are installing on the system and are in a better position to monitor and patch those sometimes myriad of minor applications.
Patch and Remediation Verification
Finally, the PVG is also responsible for making sure remediation and patches were successful. They can do this by testing the vulnerabilities on non-production systems, but only an experienced security officer or administrator should do this as the testing could cause issues itself. You can outsource this part if you do not have the expertise in-house to do it.
PVG should check the vendor documentation and verify that all the files and configuration changes have been made correctly as specified. They need to perform a vulnerability scan of the network to make sure no new vulnerabilities have emerged and that existing vulnerabilities have been patched. The PVG must also manually review the patch logs as part of ensuring the patches were installed correctly and conduct penetration testing on the system.
At What Cost?
As stated previously, it is a difficult decision to determine whether or not to go forward with remediations like patches considering the potential dangers the security vulnerability might cause compared with the potential damage a patch can cause to a system. It may also be difficult for a company to determine that patching the system is worth the money. Beyond that, is it cheaper to patch manually using human resources or to use automated enterprise systems that can be quite costly?
To Patch or Not to Patch
To address the first issue, it’s always less expensive to patch than to not patch. If we consider the number of endpoints affected multiplied by the number of hours, multiplied by the hourly rate it will cost to repair damages, you will see how quickly it adds up. If the number of endpoints is 1,000, the number of hours required to rebuild the computer plus the number of hours the employee is without the workstation at an average rate of 70 dollars an hour, we come up with an estimate of 560,000 dollars. And that’s just for workstations.
Now what is the cost of patching? That can be determined by multiplying the time it takes to monitor the patches by the hourly rate of the employee monitoring them plus the product of the time it takes to implement the patch itself, the number of devices, and the hourly rate of the employee patching the endpoints.
So, let’s use the same hourly rate from our previous example of 70 dollars an hour (this includes benefits) and say that it takes a maximum of 10 minutes a day to monitor for endpoint patches. This amounts to 60.8 hours of monitoring a year, which results in an annual cost of 4,256 dollars per year to monitor. It also takes about 10 minutes per machine to install the patches which results in an annual cost of 11,200 dollars per year to install the patches. The end result is 15,456 dollars per year, which is 2.76% of the cost to repair.
To Automate or Not to Automate
And what about the cost of automation compared to the cost of manually repairing patches annually? The previous cost determined not to patch was 560,000 PER INCIDENT. It cost 11,200 dollars per year to apply that same patch by hand. But what about the reality of the thousands of patches that must be applied every year? If you multiply the number of patches that need to be installed each year mentioned at the beginning of this article, by the cost of not patching, you get 112,000,000 dollars. If you multiply the cost per patch manually, you get 22,400,000 per year.
What is this compared to the cost of maybe one dedicated system administrator’s salary plus the up-front cost of the automated patching solution plus the product of the number of workstation endpoints to be updated and the cost to maintain each one automatically? If we assume the system administrator is paid well and makes 150,000 dollars each year and that the cost of the automated patching solution is 15,000 dollars plus 20 dollars per machine it has to update, you get 180,600 for the first year, and 165,600 for each year thereafter. Even to the naked eye this is an incredible savings, but for the sake of precision, it is 0.739% of the cost of doing it by hand.
Even Lower Costs
Let’s say you do not have the 165,600 dollars per year it costs to implement an automated patch management solution. Given the importance of protecting your brand and reputation along with the material costs of potential damage or a breach of your system, it makes sense to look for another way to make sure you can protect your assets. RSI Security has the solution. We provide Managed Security Services to small to medium sized businesses who cannot afford to pay for their own personnel and equipment at a fraction of the cost. Contact RSI Security today to schedule a free consultation and find out how.