Industry 4.0 Blog #3: Hardening the OT Network

We can learn a lot about building an operational technology (OT) security program from how employee safety programs are run.

For employee safety, your top focus isn’t how to monitor for accidents and have emergency responders on standby. No, your program is focused first on PREVENTING accidents and injury. And that allows you to respond with great urgency and effectiveness when an unfortunate event does occur. Shouldn't our approach to cybersecurity be similar? Our first focus should be to PREVENT cybersecurity incidents, so that you can respond with urgency upon that rare occurrence when your network is accessed by a “bad actor”.

A successful program has key elements, each covered in this blog post:

• As simple and as consistent as possible without losing effectiveness

• Manageable

• Empowering of “front line” workers

• Step-by-Step plan based upon industry best practices

Simplicity & Consistency

There are a number of ways that OT differs from IT. Recognizing two of these differences is key to how you approach the design of your OT security:

Safety - the most fundamental driver of operations, keeping free from harm

Uptime - the critical nature of keeping the devices operating 

Support - the characteristics of the devices & people, and how they are supported and serviced

Safety - The tangible and intangible costs of an injury or catastrophic failure are high. National and state bad actors are known to be planning for cyberattacks on critical infrastructure and OT operations across a wide range of industries.

Uptime

The priorities for securing IT and OT networks are: 

IT   

1. Confidentiality 

2. Integrity 

3. Availability

   
   

OT

1. Availability 

2. Integrity 

3. Confidentiality

It’s obvious that the priorities are reversed. And that can lead to misaligned  planning and execution of projects.

For instance if we consider ‘availability’ in IT.  If a PC fails, the user can move to another computer, or the PC can be quickly replaced or repaired. The business keeps going.

But for OT, availability is everything.  If a critical piece of machinery on the line fails, it can quickly escalate to work queuing up and production coming to a grinding halt. Getting the equipment back in operation is, generally speaking, critically more important and time-sensitive than getting a PC in IT back online.

Support

When an IT user has a computer, network or security issue, they typically call a support line, where a team of highly knowledgeable experts is ready to resolve what is usually a routine issue.

When a person in operations experiences an equipment malfunction or failure, the resolution process does not involve first calling a support line with specialists standing by to resolve it. The specialized nature of the equipment, troubleshooting, and resolution processes are substantially different for OT. The differences include:

• Verifying that a problem exists it's generally confirmed by multiple individuals on the floor in a collaborative process.

• The first escalation is to an individual, internal or external, onsite or off, who has highly specialized training on that specific device

• If diagnosis is needed, it is performed via a combination of video, voice, and remote access to the device.

• PCs and printers have fairly consistent troubleshooting procedures. For OT,  it sometimes seems as if no two devices are exactly the same.

• For sensors and other simple devices, troubleshooting is easy… swap it out. But replacing the device may still require a person with specialized skills or knowledge.

• For more sophisticated pieces of equipment, the fix is generally replacing a component. Replacing an entire robot, CNC, generators or machine controllers can take days, and in some cases much longer.

In any case, as we apply these factors to OT security, these are best practices:

1. Design consistency and simplicity into how devices on the network are set up and accessed.

2. Devote more time than you initially think necessary to define the roles, access rules, and groupings of assets and people. The result will be maintaining consistent access controls (e.g.; security permissions) over time.  Problem resolution, adds/moves/changes, security audits and system administration will take less time and resources far into the future - helping you to keep your “total life-cycle cost" to a minimum.

3. To accelerate day-to-day administration, design your processes and select the enabling technology so that it is performed by operating engineers. The IT support model (call support, open a ticket, escalate to the network team, wait for the network security admin, etc.) doesn't fit OT’s need for rapid changes to keep production moving at the consistent pace that the business demands.

4. Develop the processes and technology to support ad hoc access to a specific device from remote specialists. That remote access should be easily and reliably implemented so that permission is granted only to the minimum access needed to perform the troubleshooting and resolution required.

Simple, Consistent Setups Across Multiple Plants

State-of-the-art network security solutions provide the ability to create “overlay networks”.   Think of an overlay network as being like an invisible network running on top of another network.  The devices cannot be “seen” on the network, unless a specific user or device has been given express and very limited permission to know that a device on the “invisible network” is available to them.

When the design of an overlay network is done correctly, the devices on the network can be set up so that the consistency in the naming, addressing, and access is always performed exactly the same way regardless of where a device is located in the network.  The implication is that once setup for a given type of device, the setup will be exactly the same for every other similar device type.  Some manufacturers have said improved visibility alone justified the expense of the entire security project.  Essentially, the cost for security was free.

Illustration 1: How Byos creates an overlay network to address all aspects of the OT/IT/remote/cloud secure infrastructure.

Prioritizing the Controls that are Part of a Mature OT Security Program

If your organization is beginning to build a systematic approach to hardening your OT network, this section will be a high-level overview about where to start. If you are already down that path, then you may find a few ideas here that might help you fine tune your OT security program.

There are seven primary areas of security that you need to address in building a comprehensive OT security program:

• Inventory & Asset Management

• Network Security

• Application Security

• Identity & Access Management

• Behavior Monitoring (MDR/XDR)

• Logging & Alerting (SIEM)

• Incident Response

With so many areas of security that need to be addressed, you obviously want to focus first on doing the things that will have the biggest impact, as well as set the foundation for the things to come as your OT security program matures.  And yet, the demands from the business and the fact that technology is advancing so quickly, this is uncharted territory for many organizations.

The most successful initiatives have been the ones that “start small, learn fast, make progress in phases”.

Chart 1: Leverage your organization's understanding of the product development process to plan & execute your OT Security initiative.

The Chart above is a model for “starting small” (Phase 1, prototyping) where the three areas are done in parallel with each other. Before moving to Phase 2 (the initial rollout, which you can think of as similar to a product launch), assess your “lessons learned" from Phase 1, make adjustments, and plan the implementation of this phase with the confidence gained in Phase 1. IT will lend its expertise in the areas of technology assessment, technology project management, documentation, requirements definition, policies and procedures, and implementation planning. 

Phase 2 will provide you with more insight into refining the policies and processes, roles and responsibilities, and how to scale into the broad deployment during Phase 3 efficiently and effectively.

Phase 3 is an area where you want to leverage IT’s deep experience in risk management, vulnerability management, Incident response, and business continuity planning.  The collaboration of OT & IT in the first two stages will help you make the most of the capabilities and knowledge of both disciplines as you move into the production phase and build a culture of continuous improvement far into the future. 

An iterative approach ensures a successful rollout. If your organization is just starting its OT security program, anyone who has already traveled this path will advise you to crawl before you walk with each step.

Administration & Management

Blog #2 in this series covered the biggest obstacles to undertaking an Industry 4.0 digital transformation initiative while avoiding the risks of exposing your operations to “bad actors” on the internet:

• Securing devices you have previously kept “air gapped” - legacy devices and other equipment that could not be adequately secured
• The cost and difficulty of cabling and deploying secure wi-fi
• Finding the personnel with the expertise in IT and manufacturing who can lead this kind of project with confidence

Where to Start - Hardening your OT Network

To launch your OT security project there are some things to do as you begin planning and then deploying OT.  If you haven’t already begun the process, successful initiates follow a path similar to this.

1. Put in place the organizational & support structure

• Executive/board buy-in

• Get the key contributors on board (OT team, IT network & security teams, plant floor operators/supervisors/chain-of-command, vendor management team, etc.)

• Roles & responsibilities

• Governance & compliance

2. Develop the implementation plan

3. Inventory - know what assets you need to secure, and know what you need to secure first. You can also get a better understanding of your network security maturity posture here. 

4. Secure the network and your devices

   • Isolation & Segmentation to prevent adversaries for being able to “move laterally” to access all the devices on your network

   • Learn about zero trust and microsegmentation and how they are properly implemented for OT.  Caution that OT zero trust is similar to IT zero trust, but sometimes, what happens in IT should stay in IT.

   • Third party access should not be an add-on (If 3^rd party access is not built into your solution from the ground up, your OT security execution will get complicated really quick) Hint: If it’s a different module, or licensed as an add-on, beware!)

• Build toward consistency across the enterprise. Avoid complexity in how machines & data are accessed

• Harden your OT network and your devices

5. Secure the applications and data

• Look toward your vendors and application providers for this

• They’ve already done a lot of that work

• Know what questions to ask and have your requirements list for your baseline of what securing the data looks like (know what you need, don’t blindly let the vendors tell you what you need)

6. Develop programs for risk management, vulnerability management, and incident response. Anticipate resources that will be needed in support of these additional security-related functions, most of them likely already in place that you will be able to leverage from your IT security function.

7. Like you do on your manufacturing operations, always strive for continuous improvement.  “Rinse & repeat.”

Learning from Others’ Experiences will Lead You to Success

• Manufacturing understands how to execute the product development cycle process better than anyone. Leverage your experience in managing your OT security initiative as described in Chart #1 above.

• The design process should be built to empower front-line operations personnel to perform day-to-day cyber security administration.

• It is important to build your access control cyber security rules with forethought about how the security will look 18 months from now.  If you don’t have the expertise in-house, bring in the expertise from outside your organization, if necessary.

• Building your network security with a security overlay virtual network that creates simple, consistent visibility across your enterprise will accelerate your OT digital transformation efforts significantly.

• Even if remote and third party access are not a part of the initial phases of the OT roll out, you will serve yourself well to build your processes & technology with this front of mind.

What’s Next?

This is the third of a six-part series laying the groundwork for a successful, secure “Industry 4.0 Digital Transformation” initiative. This blog explores the issues and tradeoffs between ease of access and visibility, and provides a roadmap to prioritize how you build your operational security plan. Expect practical, real world insight from successful initiatives that you can use in your initiative.

#1 Industry 4.0 Digital Transformation

#2 How Visibility & Security Can Coexist in Manufacturing

#3 Hardening the OT Network  ← this blog

#4 Controlling Third Party & Remote Access into OT

#5 Conclusion and Resources for Your Success

Our objective is to provide you with information that you can use to communicate inside your organization where Industry 4.0 stands today, how to set the foundation for a successful program, and how to get started.

Follow this blog series for a roadmap for creating a secure and manageable OT network. In our next piece, we address how to “build your OT network for remote & third party access” - putting the controls and protections in place to make the OT network available from outside your organization, with the protections that essentially make your assets completely invisible to anyone without the specific credentials to see it. To learn more about how you can prevent malicious actors from gaining access and doing harm to your operations, contact one of our experts, or follow Byos on LinkedIn to receive updates as the series continues.