Overcome the obstacles of network security in the age of IT-OT convergence

PS: Let me ask you about the responsibilities among plant teams for these kinds of functions. I think it can be an open question on how security responsibilities should be divided up between the OT and the IT teams. Our readers often report up through the operations function, maintenance and reliability especially, plus the operators themselves. So they’re curious to hear your thoughts: What are these teams on the hook for, for security? What do you see people on the hook for, and what would you say they should let IT take care of, or ask them to take care of?

RK: It’s different for every single organization, depending on how mature IT is, and how much collaboration there is between the two departments. But if I stick to a purely philosophical perspective, maybe that might help. You and I talked about the notion that cybersecurity in OT isn’t necessarily around honeypots and catching the bad guy and doing the stuff we see in the movies, like tracing back to the evil guys. The reality is, we want to keep safe, reliable, expected operations running as much as possible, and so the ultimate responsibility lies with operations, maintenance, and the entire OT team.

Now, the challenge there is that we need to have this knowledge and awareness of the toolsets on the IT side. Let’s face it, you could pick an entire career just on networking, or just on email, or just on databases. There’s too much for any one person to know, so I can’t find someone with all operational and security knowledge. I will need a team, right?

What Verve Industrial advocates is something we call “think global, act local.” What that means is, if you do it right, a small team of a few IT and a few OT professionals can work together to first assess risk. If Blue Keep comes out, for example, we can drop that into our database and see exactly where that lights us up. But then you need the OT context. I can’t simply say it’s a critical risk on 400 assets. I need the OT site. I need the OT guy or an OT toolset to say, “Okay, here’s everywhere that it is. But within these assets, some of these are critical to safe operations.”

For example, maybe it’s a safety system, maybe it’s a lab information management system, maybe it’s something that’s working on a product spec or safety system. Those are going to have a very different priority, but also a different handling than redundant corporate domain controllers in the DMZ. So right there, I can tell you that same Blue Keep risk has a different context for me in an operations role on one versus the other. Where am I going to test first? What am I going to start with? And the only way to come up with smart decisions like that is to have the context, and I challenge my OT readers and followers that are listening to this: inventory is not just IP and Mac addresses; it’s everything about that device, plus its operational context.

Carrying on the example: You take that context, and then you work with IT and say, “Okay, what’s this Blue Keep all about?” Well, I can’t necessarily apply the patch because either the vendor hasn’t approved it yet, or they looked at it and they don’t approve it. What we need to do in OT, which is the hallmark of anything, is what I call “compensating controls.” If I can apply the patch, what is my plan B, C, or D? If the IT guy that’s reading the vulnerability can tell me, “Well, just disable the remote desktop or the guest account in as many places as possible,” we as an organization have now gone and said, “Okay, rather than a raw risk indicator, we’ve taken a notice and we’ve understood it through this combination of IT and OT, and we’re now going to provide compensating controls to most of our systems and full patch where possible.”

I’ve done what I always claim for IT people: I will get the OT team there as close as I can and as fast as we can, but we may take a different path to get there. That’s really the only way to have meaningful risk reduction and a sustainable, collaborative environment.

PS: You mentioned that these teams often share the process of assessing the risk in the first place and building out that plan. Are there other areas of overlap that you’ve noticed where the teams would share responsibilities, such as proving out the ROI on these kinds of investments?

RK: Where I’d like to see more, and I don’t see as much: forward-looking teams are when we get into proactive maintenance. Verve had a client where IT came up with a list of system hardening things they’d like to see in all of their systems, and we threw that in group policy and just let it turn on. And, of course, that scares every operator everywhere – small things they’re unaware of on the IT side, like a banner login. Well, if you’re at a pipeline or a transmission line, an auto-reboot with an auto-login in case it goes offline is a fundamental need, or somebody has to be physically dispatched to this distant thing in a Skidoo or a quad out in the middle of nowhere.

PS: We scheduled this conversation to talk about IT/OT specifically as a network strategy. In the seven years that I’ve been with Plant Services, it has gone from being an emerging strategy to something which is being deployed day-to-day. What are some of the things that you can share with our readers on what the reality of IT/OT convergence looks like on a day-to-day level?

RK: Great question. Everybody does it to differing levels of success. The ones that do it right are using this “think global, act local” perspective. For example, one of Verve’s manufacturing clients leveraged our Verve Security Center technology, and the result was they were able to see all of their risks. They found five systems that were dual-homed around the firewall. So, we saw violations of their network architecture. On those five systems, four of them were running TeamViewer. It turns out, they were bypassing the remote access policy and expectation. They did find a lot of things that were not patched for WannaCry, but they also found exploitable firmware on PLC levels.

That particular organization took this information and said, “This is our new standard. We’re going to put this into Plant No. 1.” And they had 50 or 60 of them in different levels of criticality, and then their results produced a plan. They said, “Let’s work together to build what we’re doing day-one versus month-one versus year-one and build a standard.” That same standard gets rolled out to the next plant and the next plant, and eventually they’re building this corporate collaborative environment. It’s wonderful. I can’t say enough about it.

But the real challenges come from the environments that don’t do it well. Verve has another client where the CISO said, “I will do what I need to do, and I have enough work to do on the IT side that we’ll get to OT later.” There are two things wrong with that. Number one, OT is being left behind. And number two, if I make a decision on the IT side and 18 months later I try to implement it into OT, I’m pretty baked-in with that solution. If it doesn’t work on the OT side, I’m either going back to the drawing board, or I’m only doing half the job.

What I would implore your team to do is to fight for that seat at the table, not because you’re going to take over running whitelisting tools or a SOC, but because you have very specific requirements that can or can’t happen when IT makes those decisions. And IT, for the most part, they’re not doing it out of malice. They’re doing it arguably out of ignorance. I can’t tell you the number of times I’ve sat with a big-four OT specialist and said, “You can’t do scan-based vulnerabilities. In OT, you just don’t. You knock stuff off. And so, you’re going to miss systems, or you’re going to miss openings or opportunities.” So, let’s create a vulnerability plan that matches corporate’s requirement, but let’s not put the tool first and hope we can come back later and retrofit it to OT.

To answer your question, we need to have a louder voice. We need to say, “I’m on board. I want to get where you need to, but you must believe me that we have to take a different path or maybe use different toolsets.”

PS: We’ve heard that’s a challenge too because a lot of people on the OT side may not have the similar vocabulary that IT would expect out of them. In some ways, it’s different worlds, different goals, even though the goal of each department is to help make money for the company. They’ve got different missions to go about doing that. On the OT side, it’s asset care, asset operation. On the IT side, make sure the systems are maintained.

RK: The other real challenge is that in IT, the general trend is that for any size or shape of organization, there’s more than one or two people in IT. There’s a lot of specialization within IT, so the person who does the vulnerability analysis sends it to somebody else who does vulnerability management or risk reduction in a different department. If you can get your budget to an advocate that goes to the network team, the endpoint team, and the CISO and advocates on your behalf for all operations, they’re able to navigate those otherwise silos of information.

I find that’s a huge challenge in trying to get things done in OT, again because of that compensating control. I may not be able to patch, but I need to know that I can do other compensating controls, so we know what we have and what actions we’ve taken. In the IT side, this would span multiple divisions. On the OT side, we don’t have that luxury. If we could at least dedicate somebody who is responsible for advocating on behalf of operations to navigate amongst the different departments, we’d have a much better success rate.

PS: Let’s go back to something you said about the way that IT teams have a lot of specialization in them, because in some ways that’s not so dissimilar from maintenance and reliability teams.

You may have a vibration specialist on hand to monitor the motors, you might have a thermal technician who is in charge of electrical panel maintenance, and yet each department can look monolithic to the others. I’m bringing it up as a possible point of sympathy between the two teams that, you know what, you may appear to be a monolith, but each side has specializations, and the more one learns about the other, the more they can appreciate, okay, what are the data demands on the reliability side and how might this specialist in IT help out with that problem?

RK: You’re absolutely right, and that goes both ways. I’d advocate on the OT side, and I do so because I’m biased. I have seen organizations where IT has designated an OT outreach person. In both cases, it helps to navigate multiple divisions on both sides.

If I come in and say I’m the OT advocate, and we’re going to do something around vulnerability scanning or something else, I’m going to need to know my options from IT. Then I’ll explain it to the vibration guys, the operators, etc., because they’re all going to have different needs for different systems.

This story originally appeared in the September 2021 issue of Plant Services. Subscribe to Plant Services here.