In our previous articles of this OT cybersecurity monitoring series (Cybersecurity monitoring for OT / Cybersecurity tooling strategy), we explained the current state of OT detection capabilities and discussed the right tooling strategy. 

This third article focuses on a key question: how do you measure the efficiency of your OT detection? 

From compliance to efficiency: a KPI paradigm shift 

KPI stands for Key Performance Indicator. However, we tend to create KPIs to monitor progress against our plans, not real performance. While useful, monitoring only deployment or coverage (number of sites connected to the SOC, EDR deployment on OT machines, number of probes registered to the management console) tells you very little about the actual ability of your SOC to detect a real attacker. 

So, how confident are you in your detection tools, use cases, and processes? The only way to be sure is simple: test them. And the best way to test them is through Purple Team exercises. 

What is Purple Teaming in OT? 

A Purple Team exercise is a collaborative mission between the Red Team (attackers) and the Blue Team (defenders). Unlike a traditional Red Team assessment, where the defenders are kept in the dark and evaluated afterward, a Purple Team exercise is an iterative, joint effort. 

This collaborative approach allows both teams to: 

• Share assumptions about the OT environment 
• Validate detection logic in real time 
• Understand blind spots 
• Improve playbooks and detection pipelines 
• Align everyone around a realistic threat model 

Performing a Purple Team Exercise 

A Purple Team operation can be summarized in three main phases: 

1. Preparation

The preparation phase is often the most challenging, especially in OT environments, where safety, process continuity, and vendor constraints must be considered. 

Depending on the maturity of the organization, preparation can range from basic to highly sophisticated: 

• Unit Tests 
  Small, isolated tests of specific detection rules (e.g., “Detect Modbus function code 90”). 
• Feared Scenario-based Testing 
  Build scenarios around the organization’s crown jewels and failure modes (e.g., “Unauthorized remote program upload on a PLC controlling a critical process”). 
• CTI-Infused Testing 
  Integrate threat intelligence: test techniques used by real OT-focused attackers (e.g. TTPs from Volt Typhoon, Sandworm, Xenotime, or ransomware groups targeting industrial environments). 

To structure the preparation phase, two elements are essential: 

• A good knowledge of your OT environment 
  Planning an exercise that will be relevant to both the business risks & OT detection without impacting the process requires a deep knowledge of the site and its automation. 
• Mapping to the MITRE ATT&CK for ICS matrix 
  Mapping your tests to the ATT&CK matrix allows you to have a common language with the detection teams. This allows you to select relevant techniques, avoid blind spots, and ensure coverage across multiple layers: OT workstations, PLCs, network interactions, engineering actions… 

2. D-day (Execution)

Execution is performed jointly: 

• The Red Team launches controlled and authorized actions 
• The Blue Team monitors detections in real time 
• Both teams adjust, document, and validate findings as the exercise unfolds 

Depending on the scope and complexity of the tests, the Purple Team operation can last from a few hours to a few days. 

Ensuring Reproducibility with Caldera 

To ensure repeatability and consistency across Purple Team exercises, automation becomes key.  Caldera, an open-source Breach & Attack Simulation (BAS) framework developed by MITRE, is a powerful tool for this. 

As a former pentester, I’ve always disliked the term “automated pentest”—but BAS tools are the closest thing we have to repeatable, safe attack execution. 

Why use Caldera instead of performing tests manually? 

Caldera enables you to: 

• Prepare and validate a controlled list of tests on a controlled list of assets 
• Ensure only authorized actions are executed 
• Guarantee reproducibility across environments 
• Replay the exact same actions to measure improvements after configuration changes 

Some OT-specific plugins already exist in the Caldera-OT module, supporting Modbus, Profinet, DNP3, and others. 

Recently, Wavestone released two additional OT plugins: 

• Siemens S7 protocol support 
• OPC-UA communications actions 

Caldera in a nutshell 

Caldera usage relies on: 

• Abilities: atomic technical actions (e.g., reading coils, writing tags, scanning a PLC) 
• Adversaries: collections of abilities that form a scenario 
• Operations: real-time execution of those adversaries against a target 

Fact sources: parameters provided for an operation; you can launch the same operations against different environments by just changing the fact source. 

The following video (French with English subtitles) will walk you through a demonstration of Caldera on our small ICS demo setup: 

3. Debriefing

The debrief is where most of the value is extracted. The following types of Key Performance Indicators might be used: 

• Detection Coverage – what percentage of executed stimuli were detected? 
• Alert Quality – were alerts actionable, precise, and intelligible? 
• Reaction Time – how long before an alert is raised and acknowledged? 
• Playbook Efficiency – were the right actions taken in the expected time frame? 

These might phase results in: 

• Updated detection rules 
• Improved SIEM/SOC playbooks 
• Better monitoring architecture 
• Training material for analysts and engineers 

Start Testing Now! 

Purple Team testing brings value immediately, no matter what your current maturity level is: 

• It validates your tools in real-world conditions 
• It trains your SOC and OT teams 
• It reveals blind spots early in the program 
• It provides quantitative KPIs to drive detection improvements 

And yes, it is possible, in most production environments, under the following conditions: 

• Strictly controlled scope 
• Vendor-approved actions 
• No disruptive functions executed 
• Involvement of operations and safety teams 
• Continuous monitoring of system behavior during testing 

In short: start small, stay safe, and iterate. 

Do not wait for your OT security program to be “finished” before you start testing its effectiveness!