Figure 1 : Example of the Leaking exploit found in ChatGPT in December 

Scandals like these highlight a deeper truth: the core architecture of Large Language Models (LLMs) such as GPT and Google’s Gemini is inherently prone to data leakage. This leakage can involve Personally Identifiable Information (PII) or confidential company data. The techniques used by attackers will continue to evolve in response to improved defenses from tech giants, the underlying vectors remain unchanged. 

Today, three main vectors exist through which PIIs (Personally Identifiable Information) or sensitive data might be exposed to such attacks:  

• The use of publicly available web content in training datasets 
• The continuous re-training of models using user prompts and conversations 
• The introduction of persistent memory features in chatbots 
   

LLM Pre-Training Data Leakage  

Most models available right now are transformer models, specifically GPTs or Generative Pre-Trained Transformers. The Pre-Trained in GPT refers to the initial training phase, where the model is exposed to a massive, diverse corpus of data unrelated to its final application. This helps the model learn foundational knowledge such as grammar, vocabulary, and factual information. When GPTs were first released, companies were transparent on where this training data came from, but currently the largest models on the web have datasets that are too large and too diverse and are often kept confidential.  

A major source of the data used in GPT pre-training are online forums such as Reddit (for Google’s models), Stack Overflow, and other social media platforms. This poses a significant risk since these social media forums often contain PIIs . Although companies claim to filter out PII during training, there have been many instances where LLMs have leaked personal data from their pre-training data corpus to users after some prompt engineering and jail breaking. This danger will become ever more present as companies race to gather more data through web scraping to train larger and more sophisticated models.  

Known leaks of this type are mostly uncovered by researchers who develop more and more creative methods to bypass the defenses of chatbots. The example mentioned earlier is one such case. By prompting the chatbot to repeat forever a word, it “forgets” its task and begins to exhibit a behavior known as memorization. In this state, the chatbot regurgitates data from its training set. While this attack has been patched, new prompt techniques continue to be found to change the behavior of the chatbot.

User Input Re-Usage and Re-Training  

User Inputs re-training is the process of continuously improving the LLM by training it on user inputs. This can be done in several ways, the most popular of which is RLHF or Reinforcement Learning from Human Feedback.   

Figure 3 : The feedback buttons used for RLHF in ChatGPT 

This method is built on top of collecting user feedback on the LLM’s output. Many users of LLMs might have seen the “Thumbs Up” or “Thumbs Down” buttons in ChatGPT or other LLM platforms.  

These buttons collect feedback from the user and use the feedback to re-train the model. If the user signifies the response as positive, the platform takes the user input / model output pair and encourages the model to replicate the behavior. Similarly, if the user indicates that the model performed poorly, the user input / model output pair will be used to discourage the model from replicating the behavior.  

However, continuous re-training can also occur without any user interaction. Models may occasionally use user input / model output to re-train in seemingly random ways. The lack of transparency from model providers and developers makes it difficult to pinpoint exactly how this happens. However, many users across the internet have reported models gaining new knowledge through re-training from other users’ chats all the way back to 2022. For example, OpenAI’s GPT 3.5 should not be able to know any information after Sept 2021, its cut-off date. Yet, asking it about recent information such as Elon Musk’s new position as CEO of Twitter (now X) will provide you with a different reality as it confidently answers your question with accuracy.   

Essentially, what this means for end-users is that their chats are not kept confidential at all and any information given to the LLM through internal documents, meeting minutes or development codebases may show up in the chats of other users thus leaking it. This poses significant privacy risks not only for individuals but also for companies, many of which have already taken action, like Samsung. In April 2023, Samsung banned the use of ChatGPT and similar chatbots after a group of employees used the tool for coding assistance and summarizing meeting notes. Although Samsung has no concrete evidence that the data was used by OpenAI, the potential risk was deemed too high to allow employees to continue using the tool. This is a classic example of Shadow AI, where unauthorized use of AI tools leads to the possible leakage of confidential or proprietary information. 

Many companies globally are waiting for stricter AI and data regulations before using LLMs for commercial use. We are seeing certain industries such as consulting open up but at an incredibly slow pace. Other companies, however, are tightening their control over internal LLM use to avoid leaking confidential data and client information.  

Memory Persistence 

While the two precedent risks have been recognized to exist for a few years, a new threat has emerged with the introduction of a feature by ChatGPT in September 2024. This feature enables the model to retain long-term memory of user conversations. The idea is to reduce redundancy by allowing the chatbot to remember user preferences, context, and previous interactions, thereby improving the relevance and personalization of responses.  

However, this convenience comes at a significant security cost. Unlike earlier cases, where leaked information was more or less random, persistent memory introduces account-level targeting. Now, attackers could potentially exploit this memory to extract specific details from a particular user’s history, significantly raising the stakes. 

Security researcher Johannes Rehberger demonstrated how this vulnerability could be exploited through a technique known as context poisoning. In his proof-of-concept, he crafted a site with a malicious image containing instructions. Once the targeted chatbot views the URL, its persistent memory is poisoned. This covert instruction allows the chatbot to be manipulated into extracting sensitive information from the victim’s conversation history and transmitting it to an external URL. 

This attack is particularly dangerous because it combines persistence and stealth. Once it infiltrates the chatbot, it remains active indefinitely, continuously exfiltrating user data until the memory is cleaned. At the same time, it is subtle enough to go unnoticed, requiring careful human analysis of the memory to be detected. 

LLM Data Privacy and Mitigation  

LLM developers often intentionally make it hard to disable re-training since it benefits their LLM development. If your personal information is already out in public, it has probably been scraped and used for pre-training an LLM. Additionally, if you gave ChatGPT or another LLM a confidential document in your prompt (without manually turning re-training OFF), it has most probably been used for re-training.  

Currently, there is no reliable technique that allows an individual to request the deletion of their data once it has been used for model training. Addressing this challenge is the goal of an emerging research area known as Machine Unlearning. This field focuses on developing methods to selectively remove the influence of specific data points from a trained model, thus deleting those data from the memory of the model. The field is evolving rapidly, particularly in response to GDPR regulations that enforce the right to erasure. For this reason, it is important to mitigate and minimize these risks in the future by controlling what data individuals and organizations put out on the internet and what information employees add to their prompts.  

It is vital for many business operations to stay confidential. However, the productivity boost that LLMs add to employee workflows cannot be overlooked. For this reason, we constructed a 3-step framework to ensure that organizations can harness the power of LLMs without losing control over their data.  

Choose the most optimal model, environment and configuration  

Ensure that the environment and model you are using are well-secured. Check over the model’s data retention period and the provider’s policy on re-training on user conversations. Ensure that you have “Auto-delete” as ON when available and “Chat History” to OFF.   

At Wavestone we made a tool that compares the top 3 closed-source and open-source models in terms of pricing, data retention period, guard rails, and confidentiality to empower organizations in their AI journey. 

Raise employee awareness on best practices when using LLMs  

Ensure that your employees know the danger of providing confidential and client information to LLMs and what they can do to minimize including corporate or personal information in an LLM’s pre-training and re-training data corpus.  

Implement a robust AI policy   

Forward-looking companies should implement a robust internal AI policy that specifies:  

• What information can and can’t be shared with LLMs internally  
• Monitoring of AI behavior  
• Limiting their online presence  
• Anonymization of prompt data  
• Limiting use to secure AI tools only  

Following these steps, organizations can minimize the digital risk they face by using the latest GenAI tools while also benefiting from their productivity increases.  

Moving Forward  

Although the data privacy vulnerabilities mentioned in this article impact individuals like you and me, their cause is the LLM developers’ greed for data. This greed produces higher-quality end products but at the cost of data privacy and autonomy.  

New regulations and technologies have come out to combat this issue such as the EU AI Act and OWASP top 10 LLM checklist. However, relying solely on responsible governance is not enough. Individuals and organizations must actively recognize the critical role PIIs play in today’s digital landscape and take proactive steps to protect them. This is especially important as we move toward more agentic AI systems, which autonomously interact with multiple third-party services. Not only will these systems process an increasing amount of personal and sensitive data, but this data will also be transmitted and handled by numerous different services, complicating oversight and control. 

References and Further Reading  

[1] D. Goodin, “OpenAI says mysterious chat histories resulted from account takeover,” Ars Technica, https://arstechnica.com/security/2024/01/ars-reader-reports-chatgpt-is-sending-him-conversations-from-unrelated-ai-users/ (accessed Jul. 13, 2024). 

[2] M. Nasr et al., “Extracting Training Data from ChatGPT,” not-just-memorization , Nov. 28, 2023. Available: https://not-just-memorization.github.io/extracting-training-data-from-chatgpt.html 

[3] “What Is Confidential Computing? Defined and Explained,” Fortinet. Available: https://www.fortinet.com/resources/cyberglossary/confidential-computing#:~:text=Confidential%20computing%20refers%20to%20cloud 

[4] S. Wilson, “OWASP Top 10 for Large Language Model Applications | OWASP Foundation,” owasp.org, Oct. 18, 2023. Available: https://owasp.org/www-project-top-10-for-large-language-model-applications/ 

[5] “Explaining the Einstein Trust Layer,” Salesforce. Available: https://www.salesforce.com/news/stories/video/explaining-the-einstein-gpt-trust-layer/ 

[6] “Hacker plants false memories in ChatGPT to steal user data in perpetuity” Ars Technica , 24 sept. 2024 Available: https://arstechnica.com/security/2024/09/false-memories-planted-in-chatgpt-give-hacker-persistent-exfiltration-channel/

[7] “Why we’re teaching LLMs to forget things” IBM, 07 Oct 2024 Available: https://research.ibm.com/blog/llm-unlearning

Cet article Leaking Minds: How Your Data Could Slip Through AI Chatbots  est apparu en premier sur RiskInsight.

Today, the impacts are limited because the latitude given to AI systems is still relatively low. Tomorrow, with the rise of agentic AI, accelerated adoption of generative AI, and the multiplication of use cases, the impacts will grow. Just as the ransomware WannaCry exploited vulnerabilities on a massive scale in 2017, major cyberattacks are likely to target AI systems and could result in injuries or financial bankruptcies.

These risks can be anticipated. One of the most pragmatic ways to do this is to take on the role of a malicious individual and attempt to manipulate an AI system to study its robustness. This approach highlights system vulnerabilities and how to fix them. Specifically for generative AI, this discipline is called AI RedTeaming. In this article, we offer insight into its contours, focusing particularly on field feedback regarding the main vulnerabilities encountered.

To stay aligned with the market practices, this article exclusively focuses on the RedTeaming of generative AI systems.

Back to basics, how does genAI work ?

GenAI relies on components that are often distributed between cloud and on-premise environments. Generally, the more functionalities a generative AI system offers (searching for information, launching actions, executing code, etc.), the more components it includes. From a cybersecurity perspective, this exposes the system to multiple risks :

Diagram of a Generative AI System and Issues Raised by Component

In general, an attacker only has access to a web interface through which they can interact (click, enter text into fields, etc.). From there, they can:

• Conduct classic cybersecurity attacks (inserting malicious scripts – XSS, etc.) by exploiting vulnerabilities in the AI system’s components;
• Perform a new type of attack by writing in natural language to exploit the functionalities provided by the generative AI system behind the web interface: data exfiltration, executing malicious actions using the privileges of the generative AI system, etc.

Technically, each component is protected by implementing security measures defined by Security Integration Processes within Projects. It is then useful to practically assess the effective level of security through an AI RedTeam audit.

RedTeaming IA, Art of findings AI vulnerabilities

AI RedTeam audits are similar to traditional security audits. However, to address the new challenges of GenAI, they rely on specific methodologies, frameworks, and tools. Indeed, during an AI RedTeam audit, the goal is to bypass the generative AI system by either attacking its components or crafting malicious instructions in natural language. This second type of attack is called prompt injection, the art of formulating malicious queries to an AI system to divert its functionalities.

During an AI RedTeam audit, two types of tests in natural language attacks (specific to AI) are conducted simultaneously:

• Manual tests. These allow a reconnaissance phase using libraries of malicious questions consolidated beforehand.
• Automated tests. These usually involve a generative AI attacking the target generative AI system by generating a series of malicious prompts and automatically analyzing the coherence of the chatbot’s responses. They help assess the system’s robustness across a wide range of scenarios.

These tests typically identify several vulnerabilities and highlight cybersecurity risks that are often underestimated.

What are the main vulnerabilities we found ?

We have covered three main deployment categories with our clients:

1. Simple chatbot : these solutions are primarily used for redirecting and sorting user requests;
2. RAG (Retrieval-Augmented Generation) chatbot : these more sophisticated systems consult internal document databases to enrich their responses;
3. Agentic chatbot : these advanced solutions can interact with other systems and execute actions.

The consolidation of vulnerabilities identified during our interventions, as well as their relative criticality, allows us to define the following ranking:

Diversion of the model and generation of illegitimate content 

This concerns the circumvention of the technical safeguards put in place during the development of the chatbot in order to generate offensive, malicious, or inappropriate content. Thus, the credibility and reputation of the company are at risk of being impacted since it is responsible for the content produced by its chatbot. 

It is worth noting that the circumvention of the model’s security mechanisms can lead to a complete unlocking. This is referred to as a jailbreak of the model, which shifts it into an unrestricted mode. In this state, it can produce content outside the framework desired by the company.

Access to the preprompt

The term preprompt refers to the set of instructions that feed the model and shape it for the desired use. All models are instructed not to disclose this preprompt in any form. 

An attacker gaining access to this preprompt has their attack facilitated, as it allows them to map the capabilities of the chatbot model. This mapping is particularly useful for complex systems interfaced with APIs or other external systems. Furthermore, access to this preprompt by an attacker enables them to visualize how the filters and limitations of the chatbot have been implemented, which allows them to bypass them more easily.

Web integration and third-party integration

GenAI solutions are often presented to users through a web interface. AI RedTeaming activities regularly highlight classic issues of web applications, particularly the isolation of user sessions or attacks aimed at trapping them. In the case of agentic systems, these vulnerabilities can also affect third-party components interconnected with the GenAI system.

Sensitive data leaks

If the data feeding the internal knowledge base of a RAG chatbot is insufficiently consolidated (selection, management, anonymization, …), the models may inadvertently reveal sensitive or confidential information. 

This issue is related to aspects of rights management, data classification, and hardening the data preparation and transit pipelines (MLOps).

Stored injection

In the case of stored injection, the attacker is able to feed the knowledge base of a model by including malicious instructions (via a compromised document). This knowledge base is used for the chatbot’s responses, so any user interacting with the model and requesting the said document will have their session compromised (leak of users’ conversation history data, malicious redirections, participation in a social engineering attack, etc.). 

Compromised documents may be particularly difficult to identify, especially in the case of large or poorly managed knowledge bases. This attack is thus persistent and stealthy.

Mention honorable: parasitism and cost explosion

We talk about parasitism when a user is able to unlock the chatbot to fully utilize the model’s capabilities and do so for free. Coupled with a lack of volumetric restrictions, a user can make a prohibitive number of requests, unrelated to the initial use case, and still be charged for them.

In general, some of the mentioned vulnerabilities concern relatively minor risks, whose business impact on information systems (IS) is limited. Nevertheless, with advances in AI technologies, these vulnerabilities take on a different dimension, particularly in the following cases:

• Agentic solutions with access to sensitive systems
• RAG applications involving confidential data
• Systems for which users have control over the knowledge base documents, opening the door to stored injections

The tested GenAI systems are largely unlockable, although the exercise becomes more complex over time. This persistent inability of the models to implement effective restrictions encourages the AI ecosystem to turn to external security components.

What are the new attack surfaces ?

The increasing integration of AI into sensitive sectors (healthcare, finance, defense, …) expands the attack surfaces of critical systems, which reinforces the need for filtering and anonymization of sensitive data. Where AI applications were previously very compartmentalized, agentic AI puts an end to this compartmentalization as it deploys a capacity for interconnection, opening the door to potential threat propagation within information systems. 

The decrease in the technical level required to create an AI system, particularly through the use of SaaS platforms and Low/no code services, facilitates its use for both legitimate users and attackers. 

Finally, the widespread adoption of “co-pilots” directly on employees’ workstations results in an increasing use of increasingly autonomous components that act in place of and with the privileges of a human, accelerating the emergence of uncontrolled AI perimeters or Shadow IT AI. 

Towards increasingly difficult-to-control systems

Although appearing to imitate human intelligence, GenAI models (LLMs, or Large Language Models) have the sole function of mimicking language and often act as highly efficient text auto-completion systems. These systems are not natively trained to reason, and their use encounters a “black box” operation. It is indeed complex to reliably explain their reasoning, which regularly results in hallucinations in their outputs or logical fallacies. In practice, it is also impossible to prove the absence of “backdoors” in these models, further limiting our trust in these systems. 

The emergence of agentic AI complicates the situation. By interconnecting systems with opaque functioning, it renders the entire reasoning process generally unverifiable and inexplicable. Cases of models training, auditing, or attacking other models are becoming widespread, leading to a major trust issue when they are integrated into corporate information systems.

What are the perspectives for the future ?

The RedTeaming AI audits conducted on generative AI systems reveal a contrasting reality. On one hand, innovation is rapid, driven by increasingly powerful and integrated use cases. On the other hand, the identified vulnerabilities demonstrate that these systems, often perceived as intelligent, remain largely manipulable, unstable, and poorly explainable. 

This observation is part of a broader context of the democratization of AI tools coupled with their increasing autonomy. Agentic AI, in particular, reveals chains of action that are difficult to trace, acting with human privileges. In such a landscape, the risk is no longer solely technical: it also becomes organizational and strategic, involving continuous governance and oversight of its uses. 

In the face of these challenges, RedTeaming AI emerges as an essential lever to anticipate possible deviations, adopting the attacker’s perspective to better prevent drifts. It involves testing the limits of a system to design robust, sustainable protection mechanisms that align with new uses. Only by doing so can generative AI continue to evolve within a framework of trust, serving both users and organizations. 

Cet article Red Teaming IA : State of play of AI risks in 2025 est apparu en premier sur RiskInsight.

Today, there’s every reason to believe so!

After a decade of massive investment in cybersecurity, we are a period of consolidation. Optimization is becoming the watchword: automate repetitive tasks, rationalize resources, detect ever faster and respond ever better.

AI, among other things, is a response to these objectives.

But in concrete terms, what changes has it already brought? What use cases are transforming the daily lives of cyber teams? And how far can we go?

Let’s explore together how AI will revolutionize cybersecurity.

Raising awareness: AI is changing the game!

In a nutshell: 20% of cyber incidents are related to phishing and the use of stolen accounts (according to the CERT-Wavestone 2024 report: trends, analyses and lessons for 2025).

Training teams is therefore essential. But it’s an onerous task, requiring time, resources and the right approach to capture attention and guarantee real impact. AI is changing the game by automating awareness campaigns, making them more interactive and engaging.

There’s no longer any excuse for excluding an entity from your campaign because they don’t speak English, or for failing to tailor your communications to the issues faced by different departments (HR, Finance, IT…).

With a little background on the different teams targeted, and an initial version of your awareness campaign, GenAI¹ templates can quickly break down your campaigns into customized copies for each target group. AI makes it possible to create, with minimal effort, content tailored to the issues of the awareness program’s targets, increasing employee engagement and interest thanks to a message that is fully addressed to them and deals with their own issues. This saves time, performance and quality, enabling you to transform massive, generic awareness campaigns into targeted, personalized campaigns that are undeniably more relevant.

Two possibilities are emerging for implementing this use case:

• Use your company’s trusted GenAI templates to help you generate your campaign elements. The advantage here is, of course, the low costs involved.
• Use an external supplier. Many service providers who assist companies with standard phishing campaigns use GenAI internally to deliver a customized solution quickly.

In short, AI will reduce the cost and time taken to roll out awareness programs, while improving their adherence and effectiveness to make safety a responsibility shared by all.

These same AI models can also be customized and used by cybersecurity teams for other purposes, such as facilitating access to cybersecurity repositories.

CISO GPT: simplified access to the cyber repository for the business

Internal cybersecurity documents and regulations are generally comprehensive and well mastered by the teams involved in drawing them up. However, they remain little known to other company departments.

These documents are full of useful information for the business, but due to a lack of visibility, policies are not applied. Cyber teams are called upon to respond to recurring requests for information, even though these are well documented.

With AI chatbots, this information becomes easily accessible. No need to scroll through entire pages: a simple question provides clear, instant answers, making it easier to apply best practices and react quickly in the event of an incident

More and more companies are adopting chatbots based on generative AI to answer users’ questions and guide them to the right information. These tools, powered by models such as ChatGPT, Gemini or LLaMA, access up-to-date, high-quality internal data.

Result: users quickly find the answers they need.

At Wavestone, we have developed CISO GPT. This chatbot, connected to internal security repositories, becomes a veritable cybersecurity assistant. It answers common questions, facilitates access to best practices and relieves cyber teams of repetitive requests

Answering business questions with AI is all well and good. But it’s possible to do so much more!

As well as providing rapid access to information, AI can also automate time-consuming tasks. Incident management, alert analysis, reporting… these are all processes that consume time and resources. What if AI could speed them up, or even take them over?

Save time with AI: Automate time-consuming tasks

Everyday business life is full of time-consuming tasks. AI can certainly automate many of them, but which ones should you focus on first for maximum value?

Automating data classification with AI

Here’s a first answer with another figure: 77% of recorded cyber-attacks resulted in data theft. (According to the CERT-Wavestone 2024 report: trends, analyses and lessons for 2025

And this trend is unlikely to slow down. The explosion in data volumes, accelerated by the rise of AI, makes securing them more complex.

Faced with this challenge, Data Classification remains an essential pillar in building effective DLP (Data Loss Prevention) rules. The aim: to identify and categorize data according to its sensitivity, and apply the appropriate protection measures.

But classifying data by hand is impossible on a large scale. Fortunately, machine learning can automate the process. No need for GenAI here: specialized algorithms can analyze immense volumes of documents, understand their nature and predict their level of sensitivity.

These models are based on several criteria:

• The presence of sensitive indicators (bank numbers, personal data, strategic information, ).
• User behavior to detect anomalies and report abnormally exposed files.

By combining Data Classification and AI, companies can finally regain control of their data and drastically reduce the risk of data leakage.

This is where DSPM (Data Security Posture Management) comes in. These solutions go beyond simple classification, offering complete visibility of data exposure in cloud and hybrid environments. They can detect poorly protected data, monitor access and automate compliance.

And compliance is another time-consuming process!

Simplify compliance: automate it with AI

Complying with standards and regulations is a tedious task. With every new standard comes a new compliance process!

For an international player, subject to several regulatory authorities, it’s a never-ending loop.

Good news: AI can automate much of the work. GenAI-based solutions can verify and anticipate compliance deviations.

AI excels at analyzing and comparing structured data. For example, a GenAI model can compare a document with an internal or external repository to validate its compliance. Need to check an ISP against NIST recommendations? AI can identify discrepancies and suggest adjustments.

Simplify vulnerability management

AI has no shortage of solutions when it to vulnerability management. It can automate several key tasks:

• Verification of firewall rules: GenAI can analyze a flow matrix and compare it with the rules actually implemented. It detects inconsistencies and can even anticipate the impact of a rule change.
• Code review: AI scans code for security flaws and suggests optimizations. With these tools, teams reduce the risk of error, speed up processes and free up time to concentrate on higher value-added tasks.

Automating compliance and vulnerability management reinforces upstream security and anticipates threats. But sometimes it’s already too late!

Faced with ever more innovative attackers, how can AI help to better detect and respond to incidents?

Incident detection and response: AI on the front line

Let’s start with a clear observation: cyberthreats are constantly evolving!

Attackers are adapting and innovating, and it is imperative to react quickly and effectively to increasingly sophisticated incidents. Security Operations Centers (SOCs) are at the forefront of incident management.

With the AI on their side, they now have a new ally!

AI at the heart of the SOC: detect faster….

One of the most widely used and damaging attack vectors in recent years is phishing, and the attempts are not only more recurrent, but also more elaborate than in the past: QR-Code, BEC (Business Email Compromise) …

As mentioned above, awareness-raising campaigns are essential to deal with this threat, but it is now possible to reinforce the first lines of defense against this type of attack thanks to deep learning.

NLP language processing algorithms don’t just analyze the raw content of e-mails. They also detect subtle signals such as an alarmist tone, an urgent request or an unusual style. By comparing each message with the usual patterns, AI can more effectively spot fraud attempts. These solutions go much further than traditional anti-spam solutions, which are often based solely on indicators of compromise.

Apart from this very specific case, AI will become indispensable for the detection of deviant behavior (UEBA). The ever-increasing size and diversity of IS makes it impossible to build individual rules to detect anomalies. Thanks to machine learning, we can continuously analyze the activities of users and systems to identify significant deviations from normal behavior. This makes it possible to detect threats that are difficult to identify with static rules, such as a compromised account suddenly accessing sensitive resources, or a user adopting unusual behavior outside his or her normal working hours.

These solutions are not new: as early as 2015, solution vendors were proposing the incorporation of behavioral analysis algorithms into their solutions!

AI also plays a key role in accelerating and automating response. Faced with ever faster and more sophisticated attacks, let’s see how AI enables SOC teams to react with greater efficiency and precision.

… answer louder

SOC analysts, overwhelmed by a growing volume of alerts, have to deal with ever more of them, with teams that are not growing. To help them, new GenAI assistants dedicated to SOC are emerging on the market, optimizing the entire incident processing chain. The aim is to do more with less, by redirecting analysts towards higher value-added tasks and limiting the well-known syndrome of “alert fatigue”

Starting with prioritization, operational teams are overwhelmed by alerts, and must constantly distinguish between true and false, priority and low priority. On a list of 20 alerts in front of me, which ones represent a real attack on my IS? AI’s strength lies precisely in ensuring better alert processing by correlating current events. In an instant, AI excludes false positives and returns the list of priority incidents to be investigated

The analyst can then rely on this feedback to launch his investigation. And here again, the AI supports him in his research. The GenAI assistant is capable of generating queries based on natural language, making it easy to interrogate all network equipment. Based on its knowledge, the AI can also suggest the steps to follow for the investigation: who should I question? What should I check?

The results returned will not be comparable to the analysis  an expert SOC engineer. On the other hand, they will enable more junior analysts to begin their investigation before escalating it in the event of difficulties.

But the job doesn’t stop there: you need to be able to take the necessary remediation actions following the discovery of an attack. Once again, the AI assistant keeps the focus on the decision-making process, and quickly provides the user with a set of actions to take to contain the threat: hosts to isolate, IPs to block…

The power of these use cases also lies in the ability of AI assistants to provide structured feedback, which makes it much easier not only for analysts to understand, but also to archive and explain incidents to a third party.

Of course, these are not the only use cases to date, and many more will emerge in the years to come. For incident response teams, the next step is clear: automate remediation and protection actions. We are already seeing this for our most mature customers, and the arrival of AI agents² will only accelerate this trend.

The next use cases are clear: AI active rights over corporate resources to enable a real-time response to block the spread of a threat. Following an autonomous investigation, the AI will be able to decide on its own whether to adapt firewall rules, revoke a user’s access on the fly, or initiate a new strong authentication request. Of course, such advanced autonomy is still some way off, but it’s clear that we’re heading in that direction…

Finally, integrating these use cases raises another major challenge: price. Adding these use cases has a cost. In a tense economic climate, the budgets of security teams are not being revised upwards – quite the contrary. The next step will be to find a compromise between security gains and financial costs.

Conclusion

Cybersecurity teams are faced with a plethora of AI solutions on offer, making the choice a complex one. To move forward effectively, it’s essential to adopt a pragmatic and structured approach. Our recommendations:

• Get trained in AI to better assess the added value of certain products, and avoid ‘gimmicky’ solutions.
• Choose the right use cases according to their added value (optimization of resources, economies of scale, improved risk coverage) and complexity (technology base, data management, HR and financial costs).
• Define the right development strategy, choosing between an in-house approach or using existing market solutions.
• Focus on impact rather than completeness, aiming for efficient deployment of use cases.
• Anticipate the challenges of securing AI, including model robustness, bias management and resistance to adversarial attacks.

Ten years ago, DARPA launched a challenge on autonomous cars. What was then science fiction is now reality. In 2025, AI will transform cybersecurity. We’re only at the beginning: how far will AI agents go in 10 years’ time?

–

1: GenAI (Generative Artificial Intelligence) refers to a branch of AI capable of creating original content (text, images, code, etc.) based on models trained on large datasets.
2: AI agent refers to an artificial intelligence capable of acting autonomously to achieve complex goals, by planning, making decisions and interacting with its environment without constant human supervision.

Cet article AI4Cyb: how will AI improve your company’s cyber capabilities? est apparu en premier sur RiskInsight.