A quiet revolution is underway in European SOCs. Faced with ever-growing volumes of security events and a persistent shortage of skilled experts, a new generation of AI-powered security tools is emerging, designed to identify correlations that human teams can no longer process alone. AI is not replacing analysts but accelerating and enhancing their work. Between ambitions of hyper‑automation, challenges around model transparency, and the growing push for European digital sovereignty, the landscape of detection and incident-response solutions is rapidly evolving.  

To support this ongoing market transformation, the French National Cybersecurity Agency (ANSSI) and the French National Cyber Coordination Center (NCC‑FR), hosted by ANSSI, have launched an ambitious initiative to provide a detail overview of how IA is used for SOC by conducting a thorough study [1] with major European players specializing in SOC‑oriented security solutions. 

The study had two main objectives: 

1. Identify European players developing solutions for SOCs that integrate AI-based features [2]. 
2. Build an overview of the use cases available on the market, including those offered by leading US vendors operating in Europe. 

This article summarises the key insights drawn from our study conducted among 48 detection and response solution vendors. 

Geographical distribution of the vendors interviewed

A booming European market undergoing consolidation  

The study covered 48 vendors. Among them, 34 are European companies (out of an initial pool of 72 European actors identified), while the remaining 14 are major US‑based vendors firmly established in Europe.  

The market shows clear signs of consolidation, marked by numerous acquisitions, most often involving European companies being acquired by US firms. These acquisitions primarily aim at reinforcing detection and response capabilities, expanding protection coverage, or, more marginally, integrating AI components directly dedicated to detection. Thus, vendors are converging towards a unified platform approach capable of addressing the full spectrum of SOC needs. 

 
Some European initiatives, such as the OPEN XDR alliance, aim at providing a collective response to platform‑related challenges without relying on acquisition strategies between vendors. 

Meetings held with vendors revealed several key insights. 

First, GenAI, or Generative AI (AI capable of generating original content from instructions), is starting to appear within SOC solutions, primarily through chatbots integrated into analysis interfaces; however, their capabilities remain highly limited and inconsistent. These chatbots almost always rely on external technologies, particularly LLMs provided by a small group of major players such as OpenAI, Google, Meta, Anthropic, or Mistral AI, who largely dominate the market. This reliance on third‑party solutions, which often involves transferring data to the environments of these providers, raises significant concerns regarding the protection of sensitive information handled within SOCs. 
To reduce this dependency, several vendors are now considering adopting open‑source LLMs that can be deployed directly within their own environments, enabling greater control over their data and keeping sensitive flows internally.

Overview of the LLMs used by the vendors 

Besides, the use of PredAI, or Predictive AI (AI capable of predicting or classifying an input based on “knowledge” acquired during a training phase), is considerably more mature. Some European vendors have been relying on such approaches for more than 15 years to support use cases ranging from behavioral detection to alert prioritization, demonstrating genuine maturity and established expertise. Most of these use cases focus on the detection phase, where predictive models are widely used, well mastered, and most relevant. 

In addition, several vendors are beginning to explore agentic approaches, with the ambition of gradually delegating part of the repetitive or time‑consuming tasks, particularly the initial qualification of alerts and some steps of the investigation process. 

Finally, these findings should be interpreted with caution: the vendors included in the study represent only a sample of this fast-evolving market.  

Overview of European vendors in Detection & Incident Response solutions using AI  

Overview of AI use cases in detection and incident response tools  

Overview of AI use cases in the SOC operations chain 

The study identified around 50 use cases that can fall under 2 main categories:  

• Use cases based on Predictive AI models, primarily designed for incident detection; 
• Use cases relying on Generative AI, which focus mainly on investigation and incident response tasks. 

Even though the use cases are diverse and hard to list exhaustively, several major categories can nonetheless be identified. Each of these categories is designed to address similar challenges and support the same objective.  

For incident detection, the following AI use case categories can be identified: 

• Detection of abnormal behaviour from users or assets; 
• Detection of anomalies in network traffic; 
• Detection of events suggesting a possible attack; 
• detectionof phishing attempts; 
• and detection of malicious files. 

A new category, regrouping usecases fully addressed by Generative AI, is currently emerging and often addressed by chatbot assistant. Vendors are currently concentrating most of their efforts on these analyst‑oriented assistants, into which they are progressively integrating a wide range of use cases. Their priority is to simplify access to documentation and provide answers to operational questions, as well as extend these capabilities towards more advanced qualification or investigation tasks. 

To achieve this, nearly all vendors follow the same approach by: 

• leveraging a third-party foundation model; 
• applying prompt engineering to make the best use of the model’s capabilities by guiding it towards specific topics; 
• and using RAG (Retrieval‑Augmented Generation), which customizes and enriches the model’s output by supplying it with an authoritative documentation base to create its responses. 

Last, some agentic use cases, based on autonomous agents, are beginning to appear even if they still remain limited. They are currently being addressed by the most advanced and mature vendors in the sector, as well as by start-ups seeking to disrupt the market. 

Unlike most vendors, who are gradually integrating AI use cases into an existing cybersecurity platform, these newcomers are betting on specialized AI-driven solutions designed to address a specific cybersecurity task. Among these use cases are agents dedicated to threat hunting, advanced malware analysis (including automated reverse engineering), as well as the initial qualification of alerts.  

Agentic use cases, however, remain only marginally deployed to date.  

To go deeper… 

ANSSI has published a comprehensive report detailing all the results of the study: https://cyber.gouv.fr/enjeux-technologiques/intelligence-artificielle/etude-de-marche-lia-au-service-de-la-detection-et-de-la-reponse-a-incident/ 

This document now serves as a key reference for understanding current trends and the future evolution of AI’s role in detection and incident response.  

Ultimately, the study highlights a European cybersecurity market that is undergoing rapid restructuring, driven by the rise of AI but also marked by a strong consolidation dynamic. Within this shifting landscape, AI continues to gain maturity across SOC tooling: from Predictive‑AI‑based detection use cases, to GenAI‑powered analytical assistants, all the way to early but promising agentic approaches. This trajectory confirms that intelligent automation will become a major lever for increasing operational efficiency and strengthening organizations’ ability to defend against tomorrow’s threats. 

References

[1] Study conducted from October 2024 to July 2025 – https://cyber.gouv.fr/enjeux-technologiques/intelligence-artificielle/etude-de-marche-lia-au-service-de-la-detection-et-de-la-reponse-a-incident/ 

[2] Artificial intelligence-based features : Set of features using machine learning models (ML, deep learning, LLM) capable of learning from data and producing new analyses, predictions or content.

Cet article Integrating AI into SOC tools: Global overview and current trends in the European market  est apparu en premier sur RiskInsight.

Figure 1 : Example of the Leaking exploit found in ChatGPT in December 

Scandals like these highlight a deeper truth: the core architecture of Large Language Models (LLMs) such as GPT and Google’s Gemini is inherently prone to data leakage. This leakage can involve Personally Identifiable Information (PII) or confidential company data. The techniques used by attackers will continue to evolve in response to improved defenses from tech giants, the underlying vectors remain unchanged. 

Today, three main vectors exist through which PIIs (Personally Identifiable Information) or sensitive data might be exposed to such attacks:  

• The use of publicly available web content in training datasets 
• The continuous re-training of models using user prompts and conversations 
• The introduction of persistent memory features in chatbots 
   

LLM Pre-Training Data Leakage  

Most models available right now are transformer models, specifically GPTs or Generative Pre-Trained Transformers. The Pre-Trained in GPT refers to the initial training phase, where the model is exposed to a massive, diverse corpus of data unrelated to its final application. This helps the model learn foundational knowledge such as grammar, vocabulary, and factual information. When GPTs were first released, companies were transparent on where this training data came from, but currently the largest models on the web have datasets that are too large and too diverse and are often kept confidential.  

A major source of the data used in GPT pre-training are online forums such as Reddit (for Google’s models), Stack Overflow, and other social media platforms. This poses a significant risk since these social media forums often contain PIIs . Although companies claim to filter out PII during training, there have been many instances where LLMs have leaked personal data from their pre-training data corpus to users after some prompt engineering and jail breaking. This danger will become ever more present as companies race to gather more data through web scraping to train larger and more sophisticated models.  

Known leaks of this type are mostly uncovered by researchers who develop more and more creative methods to bypass the defenses of chatbots. The example mentioned earlier is one such case. By prompting the chatbot to repeat forever a word, it “forgets” its task and begins to exhibit a behavior known as memorization. In this state, the chatbot regurgitates data from its training set. While this attack has been patched, new prompt techniques continue to be found to change the behavior of the chatbot.

User Input Re-Usage and Re-Training  

User Inputs re-training is the process of continuously improving the LLM by training it on user inputs. This can be done in several ways, the most popular of which is RLHF or Reinforcement Learning from Human Feedback.   

Figure 3 : The feedback buttons used for RLHF in ChatGPT 

This method is built on top of collecting user feedback on the LLM’s output. Many users of LLMs might have seen the “Thumbs Up” or “Thumbs Down” buttons in ChatGPT or other LLM platforms.  

These buttons collect feedback from the user and use the feedback to re-train the model. If the user signifies the response as positive, the platform takes the user input / model output pair and encourages the model to replicate the behavior. Similarly, if the user indicates that the model performed poorly, the user input / model output pair will be used to discourage the model from replicating the behavior.  

However, continuous re-training can also occur without any user interaction. Models may occasionally use user input / model output to re-train in seemingly random ways. The lack of transparency from model providers and developers makes it difficult to pinpoint exactly how this happens. However, many users across the internet have reported models gaining new knowledge through re-training from other users’ chats all the way back to 2022. For example, OpenAI’s GPT 3.5 should not be able to know any information after Sept 2021, its cut-off date. Yet, asking it about recent information such as Elon Musk’s new position as CEO of Twitter (now X) will provide you with a different reality as it confidently answers your question with accuracy.   

Essentially, what this means for end-users is that their chats are not kept confidential at all and any information given to the LLM through internal documents, meeting minutes or development codebases may show up in the chats of other users thus leaking it. This poses significant privacy risks not only for individuals but also for companies, many of which have already taken action, like Samsung. In April 2023, Samsung banned the use of ChatGPT and similar chatbots after a group of employees used the tool for coding assistance and summarizing meeting notes. Although Samsung has no concrete evidence that the data was used by OpenAI, the potential risk was deemed too high to allow employees to continue using the tool. This is a classic example of Shadow AI, where unauthorized use of AI tools leads to the possible leakage of confidential or proprietary information. 

Many companies globally are waiting for stricter AI and data regulations before using LLMs for commercial use. We are seeing certain industries such as consulting open up but at an incredibly slow pace. Other companies, however, are tightening their control over internal LLM use to avoid leaking confidential data and client information.  

Memory Persistence 

While the two precedent risks have been recognized to exist for a few years, a new threat has emerged with the introduction of a feature by ChatGPT in September 2024. This feature enables the model to retain long-term memory of user conversations. The idea is to reduce redundancy by allowing the chatbot to remember user preferences, context, and previous interactions, thereby improving the relevance and personalization of responses.  

However, this convenience comes at a significant security cost. Unlike earlier cases, where leaked information was more or less random, persistent memory introduces account-level targeting. Now, attackers could potentially exploit this memory to extract specific details from a particular user’s history, significantly raising the stakes. 

Security researcher Johannes Rehberger demonstrated how this vulnerability could be exploited through a technique known as context poisoning. In his proof-of-concept, he crafted a site with a malicious image containing instructions. Once the targeted chatbot views the URL, its persistent memory is poisoned. This covert instruction allows the chatbot to be manipulated into extracting sensitive information from the victim’s conversation history and transmitting it to an external URL. 

This attack is particularly dangerous because it combines persistence and stealth. Once it infiltrates the chatbot, it remains active indefinitely, continuously exfiltrating user data until the memory is cleaned. At the same time, it is subtle enough to go unnoticed, requiring careful human analysis of the memory to be detected. 

LLM Data Privacy and Mitigation  

LLM developers often intentionally make it hard to disable re-training since it benefits their LLM development. If your personal information is already out in public, it has probably been scraped and used for pre-training an LLM. Additionally, if you gave ChatGPT or another LLM a confidential document in your prompt (without manually turning re-training OFF), it has most probably been used for re-training.  

Currently, there is no reliable technique that allows an individual to request the deletion of their data once it has been used for model training. Addressing this challenge is the goal of an emerging research area known as Machine Unlearning. This field focuses on developing methods to selectively remove the influence of specific data points from a trained model, thus deleting those data from the memory of the model. The field is evolving rapidly, particularly in response to GDPR regulations that enforce the right to erasure. For this reason, it is important to mitigate and minimize these risks in the future by controlling what data individuals and organizations put out on the internet and what information employees add to their prompts.  

It is vital for many business operations to stay confidential. However, the productivity boost that LLMs add to employee workflows cannot be overlooked. For this reason, we constructed a 3-step framework to ensure that organizations can harness the power of LLMs without losing control over their data.  

Choose the most optimal model, environment and configuration  

Ensure that the environment and model you are using are well-secured. Check over the model’s data retention period and the provider’s policy on re-training on user conversations. Ensure that you have “Auto-delete” as ON when available and “Chat History” to OFF.   

At Wavestone we made a tool that compares the top 3 closed-source and open-source models in terms of pricing, data retention period, guard rails, and confidentiality to empower organizations in their AI journey. 

Raise employee awareness on best practices when using LLMs  

Ensure that your employees know the danger of providing confidential and client information to LLMs and what they can do to minimize including corporate or personal information in an LLM’s pre-training and re-training data corpus.  

Implement a robust AI policy   

Forward-looking companies should implement a robust internal AI policy that specifies:  

• What information can and can’t be shared with LLMs internally  
• Monitoring of AI behavior  
• Limiting their online presence  
• Anonymization of prompt data  
• Limiting use to secure AI tools only  

Following these steps, organizations can minimize the digital risk they face by using the latest GenAI tools while also benefiting from their productivity increases.  

Moving Forward  

Although the data privacy vulnerabilities mentioned in this article impact individuals like you and me, their cause is the LLM developers’ greed for data. This greed produces higher-quality end products but at the cost of data privacy and autonomy.  

New regulations and technologies have come out to combat this issue such as the EU AI Act and OWASP top 10 LLM checklist. However, relying solely on responsible governance is not enough. Individuals and organizations must actively recognize the critical role PIIs play in today’s digital landscape and take proactive steps to protect them. This is especially important as we move toward more agentic AI systems, which autonomously interact with multiple third-party services. Not only will these systems process an increasing amount of personal and sensitive data, but this data will also be transmitted and handled by numerous different services, complicating oversight and control. 

References and Further Reading  

[1] D. Goodin, “OpenAI says mysterious chat histories resulted from account takeover,” Ars Technica, https://arstechnica.com/security/2024/01/ars-reader-reports-chatgpt-is-sending-him-conversations-from-unrelated-ai-users/ (accessed Jul. 13, 2024). 

[2] M. Nasr et al., “Extracting Training Data from ChatGPT,” not-just-memorization , Nov. 28, 2023. Available: https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html 

[3] “What Is Confidential Computing? Defined and Explained,” Fortinet. Available: https://www.fortinet.com/resources/cyberglossary/confidential-computing#:~:text=Confidential%20computing%20refers%20to%20cloud 

[4] S. Wilson, “OWASP Top 10 for Large Language Model Applications | OWASP Foundation,” owasp.org, Oct. 18, 2023. Available: https://owasp.org/www-project-top-10-for-large-language-model-applications/ 

[5] “Explaining the Einstein Trust Layer,” Salesforce. Available: https://www.salesforce.com/news/stories/video/explaining-the-einstein-gpt-trust-layer/ 

[6] “Hacker plants false memories in ChatGPT to steal user data in perpetuity” Ars Technica , 24 sept. 2024 Available: https://arstechnica.com/security/2024/09/false-memories-planted-in-chatgpt-give-hacker-persistent-exfiltration-channel/

[7] “Why we’re teaching LLMs to forget things” IBM, 07 Oct 2024 Available: https://research.ibm.com/blog/llm-unlearning

Cet article Leaking Minds: How Your Data Could Slip Through AI Chatbots  est apparu en premier sur RiskInsight.