Fallout from a network security breach could affect everything from machine performance and output to worker safety and environmental protections. Critical infrastructures and vital supplies also could be endangered. Any approach to staving off these threats should include proactive patch management at the infrastructure level.
Patching is notoriously complex, and organizations may shy away from it because of fears around downtime and potential operational impacts. But when a security vulnerability is revealed – as with the recent Meltdown and Spectre flaws – organizations should respond quickly to protect their assets and operations.
When automation vendors release patches, manufacturers and industrial organizations should have patch management processes and policies in place. However, developing and maintaining a patch management program includes many facets and variables. Consider these best practices when developing your program.
About the author: Umair Masud
Umair Masud is a consulting services product manager for Rockwell Automation.
Determine frequency. When developing a patch-management schedule, frequency should be dictated by the risk level associated with each vulnerability. Weigh the amount of risk and the severity of consequences if it were exploited. The higher the risk, the sooner a vulnerability should be patched. Also consider the likelihood of something being exploited, along with the ease of exploitation. Again, the higher the probability, the sooner it should be patched.
When determining the patching schedule, it’s important to keep automation and infrastructure assets separate from the enterprise assets to avoid overlap. Create a separate grouping that is specifically and explicitly tied to the control system and apply a patch cadence that fits the industrial control requirements. Typically, that means utilizing a separate Microsoft Windows domain or group within the enterprise domain, but the key is to manage them separately.
Lastly, patches should always be scheduled during a downtime or maintenance event so as not to disrupt production schedules. For operations that run 24/7, start by applying patches on a backup system.
Test the patch. Even if a patch has been approved by an automation vendor, manufacturers and industrial organizations should test it before applying it in production. Nuances among applications, customizations, and other dependencies may interfere with patches.
Creating a test bed that mimics the environment can be done within a virtual environment. Then Microsoft WSUS or SCCM can be utilized to issue patches to the test environments first. Once the test environment is set up and the patch is installed, run through typical usage scenarios to validate proper functionality and performance of your application. This will help prevent any issues arise when the patch is deployed in the production environment.
If a test environment doesn’t exist, consider choosing a noncritical asset for the analysis. That way, if issues do occur with the patches, the consequences won’t have a major impact on operations.
Once the patches have been tested and are ready to be rolled out systemwide, backup all running applications before applying the patches in a live environment. Despite best efforts for testing and preparations, issues do occur, and it’s a good idea to have a copy of the last known functioning operations of the automation servers.
Maintain policies. Patch management is only as good as the policies and procedures that support it. While each manufacturer and industrial organization will have its own unique needs, it’s important to document the processes and best practices for patching. This can help establish a disciplined governance process that outlines the risks and consequences associated with failing to maintain patches. In addition to supplying the patching updates, many automation vendors can help by developing procedures, enacting the program, and supplying remote support services.
