How system integrators are helping maintenance teams overcome a widening skills gap

This is the last of our 2024 look back / look ahead articles on changes in industry over the past 10 years. For this article, Plant Services spoke with three CSIA-member system integrators about how their work has changed in an era of “big maintenance data,” and what they see happening five years from now.

• Dave Bader is vice president of business development (NA) for Eurotech, and his previous work includes roles at Amazon Web Services (AWS), Rockwell Automation, and Fetch Robotics.
• G Brooks-Zak is a co-founder of Outlier Automation, a control systems integrator that helps manufacturers use automation and technology to improve operational efficiency and profitability.
• Heath Stephens is the digitalization leader for Hargrove Controls & Automation, and has more than 27 years of experience in chemical engineering, process safety, automation, and controls.

1. In your opinion, what have been the biggest changes for system integrators and their customers over the past 10 years?

G Brooks-Zak

The biggest thing (and what we see in general with technology) is that the breadth of technology and the layers that are involved has shifted and come more into the space. For me, what that means is that there are more types of people involved in maintaining, building, and continuing these systems. There's different skill sets – networking, cyber security software development – and we have to be able to interface with IT on the business side too. 

I'd also say simultaneously, as most people know, there there's a lot of in-house knowledge that traditionally had been in the operational technology space that was making equipment run and installing equipment, and those people are retiring. I think systems integration is bridging that gap and playing more of a role in managing both aspects of that change, and helping manufacturers adopt new technology and keep it going.

David Bader

Infrastructure is changing, and the convergence of IT and OT is an especially big change. This shift has enabled traditional OT people to speak in ways that the IT team can understand. When I started my career, and even in more recent years, that wasn't the case – there was a clear delineation between those two teams, so I think that that's a huge change. 

The other big change is the shift to pushing plant floor data to the cloud. The cloud is changing all aspects of business, and is providing the ability for us to be able to deal with the amount of data that many companies want to process. Practically, it wasn't available in the past, and even when it became possible at the edge, security was a problem. Additionally, the total amount of compute was also not available. Now we’re able to take information from the plant floor, and are able to do really special things in the cloud with increasingly larger amounts of data, as well as making it secure at the edge and to enable a secure pipeline to the cloud.

I also think manufacturers much like many other segments of business are looking to be able to analyze their operations more efficiently and are looking to do predictive analytics, visual inspections, or other highly compute intensive workloads at the edge. For example, being able to do AI model training and inference at the edge for manufacturing is a strong area of focus. As these things happen, it also expands the roles of people that are on the plant floor, who are now empowered to help provide this data

Heath Stephens

Organizations have all gotten leaner and younger with retirements, and there's not as much of a skilled workforce available. Many companies have open positions they just can't fill, and they're trying to fill in the gaps with automation. 

When I started as a system integrator, the focus was really on your classic DCS/PLC-type implementations. We were working with the automation teams that the owners had. We were working with the production engineers, those sorts of roles. Now, those teams have shrunk, and because of the increased variety of projects we implement we're also dealing with a lot more types of clients and customers. We're dealing more with maintenance personnel, with reliability teams, and with people outside of the classic production roles. 

We're also doing a lot more with automation – we’re tying the process control system into the ERP system, we're tying it into the laboratory data system, into the reliability system. We’re connecting all these different sources of information so that our clients can get a whole picture of their operations. And they need that because they don't have the people to walk the floors and do everything they used to do. They don't have time to search for information. They need information brought to them, and suggested solutions brought to them by those automation platforms.

2. The Plant Services audience includes many maintenance and reliability workers, who are being challenged to adopt digital technologies to better understand machine health in real time. How are integrators helping these teams achieve their goals?

Heath Stephens

We're doing a lot with my digitalization team to make reliability a focus and assist our clients with their proactive reliability initiatives, so we're doing things like multivariate predictive analysis for equipment downtime. Several tools are available from multiple vendors, that we can feed process data, and they will give us a big-picture look at what's going on with the process and what equipment will fail. 

Instead of having a lead time of a few minutes or a few hours from that single high alarm or trip sensor in your control system, with a multivariate view, you can get a much better fingerprint of what the process should look like and whether or not that process is on course. Then you could have these systems alert you, “Hey, you're going to have a failure of this pump in two days,” just unbelievable lead times compared to what we had in the past. With that information, maintenance and reliability teams can plan for downtime, get their repair teams organized, and order any spare parts they need so they can be a lot more proactive and efficient at what they do. 

The flip side is those same tools can monitor equipment after you've done service to them, and they can tell you, “Hey, that service was effective.  You did a good job there. We were looking at an imminent failure you prevented, and now everything is back to nominal.” Or vice versa, they could say, “Hey, you did some maintenance, but it wasn't effective. It didn't address whatever the root cause was, and we still have a pending issue.”

We also do a lot of work with reliability, availability, & maintainability tools (or RAM modeling tools), where we do statistical modeling for what are predicted to be the biggest sources of downtime in the process. Then you can either make design changes to your process, or you can make operational and maintenance changes so that you can better adapt to the equipment you have. Tools like that really take a scientific, mathematical approach to reliability. In the past, you might have depended on a lot of experienced people and their best instincts, but sometimes it was hard to prove whether  they made the most effective decisions.

It does still take some time to get familiar with these tools and configure them. There are some things that you can do, especially if you have similar equipment, where you can learn and templatize one instance and then copy it over, but it does still require a little initial setup. All these systems also require some ongoing care and maintenance. Otherwise, their effectiveness eventually starts to decay, and that's something that I always try to be upfront with my clients about. Just like any other system they have in their facility, whether it's a piece of equipment or a piece of software if you don't maintain it, it's not going to continue to perform for you.

G Brooks-Zak

A lot of machinery that we work on is custom equipment that might only be one of its kind or very few have been built. If it has not been maintained over time from a software perspective, people might not even know what's programmed in there, so if you're talking about trying to get data from those types of systems, it might not even be clear how to connect to it. 

I think that systems integrators can help there because there's communication protocols on all of this equipment that is very specific to whatever the controller was in it, or how someone programmed it. A system integrator would be able to help give that context as to what is available and what even physically might make sense to get from a data perspective and then gain insight from that machinery. You might have to add hardware in order to communicate with it, you might have to add software, or you could even add sensors on the side in order to get basic data like “is this machine running” and go up from there. If you don't have that kind of templatized or consistent process control, you might not be able to get to this machine learning / AI area without a lot more effort. 

David Bader

There is a new line of business available for OT integrators who are able to adopt modern practices. Cyber security and remote device connectivity is something that is underserved from an OT integrator perspective; it’s something that if it is done right and if the OT integrators are now able to give customers insights in a rapid way – by enabling the data management at the edge and providing customers with deep and secure insights into their business, while minimizing the possibilities of a cyber security attach from the OT side – then these systems integrators will be in high demand.

G Brooks-Zak

Bringing it back to the workforce, I think one thing that's really important too is being able to make these analysis tools understandable to folks that might not have a technical background, and that ability actually requires engineering. Making things simple is actually a lot harder and requires a lot more effort than you'd think, and control systems integrators have a unique insight in that because they understand that control systems are physical, they're a physical asset and we have to understand the physics, but at the same time we have that software background.

3. Any thoughts on what to tell maintenance and reliability folks when it comes to bringing in OT equipment. Are they opening themselves up to unnecessary risk?

Heath Stephens

The number one thing that the production and maintenance reliability teams can do is have an open line of communication with their IT department. There was a time in my early career when there was a clear dividing line between IT and operations, and that just doesn't work anymore. 

We're in a blended environment. A lot of our new applications and hardware are cloud-native, whether they're going through a VPN tunnel that the IT team has to set up to get to the cloud or whether they've got a cellular antenna and they're going straight to the cloud (which is a big risk with IT departments all over). Even if you're not going to the cloud, for most of the things we do now on the plant floor, you want to get that data to your desktop at least, and again that will require IT's involvement. 

I implore all the plant teams out there to make friends with IT and collaborate with them. Whenever I go to a client, IT needs to be in the room and aware of what we need and what we're planning to do. Anything we do presents some sort of risk, and we can identify and manage risk, but let's talk about how we're managing that risk together and talk about the upside of why we want to do these things.

David Bader

Let's face it, if the IT and OT teams start a project on the same page, there's a lot less friction along the way. Using the IT team to help maintenance and reliability teams facilitate a project or help the OT team move faster is just good business. It doesn't have to be a combative kind of relationship.

4. What changes or advances in systems integration do you expect over the next 5-10 years?

David Bader

I think the convergence of IT and OT will come together in an even deeper way, which will make it super important to be able to transfer information back and forth in a secure way. Even if companies do not want to send information to the cloud, any data transfer should be managed securely. Enhance your security by implementing methods like security by design, conduct a cybersecurity analysis of your network, understand where there are potential growth areas or potential vulnerabilities, and be able to mitigate those vulnerabilities using modern technology. These are big areas for growth going forward, and areas where systems integrators can help. 

I also think building solutions to meet regulatory compliance is another growth area, maintaining things in a way that is proven and standards-based is critical. We have ISA and IEC standards for areas control systems, but for some reason when we start to talk about cybersecurity and the convergence of IT and OT, we don't think about that in those terms. There are some really great standards out there, such as IEC 62443, a worldwide standard for ICS networks that cuts across many different verticals. In Europe, for instance, it's mandated for companies to build to those standards. The faster the systems integrator community learns about this and about building solutions to that standard, it's going to make it so much easier. 

I like to look at it from North America's perspective. Rather than a mandate, it is more of an ROI discussion. – when an integrator deploys a job and there's a potential cyber threat or an actual cyber threat, it's going to cost somebody money and time. Mitigating these possibilities creates a really great opportunity in the next 5 to 10 years for companies to build to those cyber standards up front and by design.

G Brooks-Zak

The reality is that things are only going to get more connected. There's going to be more devices on the manufacturing floor, and more things we have to connect, so it will be more difficult to maintain and even know where things are unless there is more standardization. I recently wrote an article that focused on software-defined automation, and the concept is to be able to understand how hardware and software is laid out, and then be able to maintain that through the life cycle of creating it, deploying it, and changing it. There are things that I think that we can learn from software engineering and apply it in our space, and that will be more and more important as the complexity increases but also as the value and the capabilities of our manufacturing increases.

Heath Stephens

We're seeing a lot of changes right now with generative AI, and it's coming for your control system – it's going to be a good thing, but it's also going to be a transition. We've gone from casually speaking to Alexa at home to ChatGPT and Microsoft Copilot on our desktops at work. That sort of interaction is going to come to the process control system side, and it's going to allow many more people who aren't automation experts or full-time operators to be able to talk to their control systems and get information out of them. It will mean great things for maintenance and reliability teams and the technicians out there that aren't necessarily trained to work with a control system and don’t know how to look through code, but have process questions that need answers.

All the big control system vendors are working on tools, and there are a bunch of add-on tools that will be out there, too, that are more specific to data processing. One of the basic things all of my clients do if they're running a new facility is, from the start, they look at whole facility Wi-Fi, because they want those operators to be able to go around with their mobile devices and have data at their fingertips and to be able to feed data back in real-time. All of this is going to be a game changer for the workforce.