The challenge of choosing a guardrails solution

As guardrails solutions multiply, organizations face a practical challenge: selecting protection mechanisms that effectively reduce risk without compromising performance, user experience, or operational feasibility.

Choosing guardrails is not limited to blocking malicious prompts. It requires balancing detection accuracy, false positives, latency, and the ability to adapt filtering to the specific context, data sources, and threat exposure of each application. In practice, no single solution addresses all use cases equally well, making guardrail selection a contextual and risk-driven decision.

An important diversity of solutions

In 2025, the AI security and LLM guardrails landscape experienced significant consolidation. Major cybersecurity vendors increasingly sought to extend their portfolios with protections dedicated to generative AI, model usage, and agent interactions. Rather than building these capabilities from scratch, many chose to acquire specialized startups to rapidly integrate AI-native security features into their existing platforms, such as SentinelOne with Prompt Security or Check Point with Lakera.

This trend illustrates a broader shift in the cybersecurity market: protections for LLM-based applications are becoming a standard component of enterprise security offerings, alongside more traditional controls. Guardrails and runtime AI protections are no longer niche solutions, but are progressively embedded into mainstream security stacks to support enterprise-scale AI adoption

The main criteria to choose your guardrails

With so many guardrails’ solutions, choosing the right option becomes a challenge. The most important criteria to focus on are:

• Filtering effectiveness, to reduce exposure to malicious prompts while limiting false positives
• Latency, to ensure a user-friendly experience
• Personalisation capabilities, to adapt filtering to business-specific contexts and risks
• Operational cost, to support scalability over time

Key Results & Solutions Profiles

To get an idea of the performances the guardrails in the market, we tested several solutions across these criteria and a few profiles stood out:

• Some solutions offer rapid deployment and effective baseline protection with minimal configuration, making them suitable for organizations seeking immediate risk reduction. These solutions typically perform well out of the box but provide limited customization.
• Other solutions emphasize flexibility and fine-grained control. While these frameworks enable advanced filtering strategies, they often exhibit poor default performance and require significant configuration effort to reach good protection levels.

As a result, selecting a guardrails solution depends less on raw detection scores and more on the expected level of customization, operational maturity, and acceptable setup effort.

Focus on Cloud Providers’ guardrails

As most LLM-based applications are deployed in cloud environments, native guardrails offered by cloud providers represent a pragmatic first layer of protection. These solutions are easy to activate, cost-effective, and integrate seamlessly into existing cloud workflows.

Using automated red-teaming techniques, we observed that cloud-native guardrails consistently blocked most of the common prompt injection and jailbreak attempts. The overall performance of the guardrails available on Azure, AWS and GCP were similar, confirming their relevance as baseline protection mechanisms for production workloads.

Sensitivity Configuration

The configuration of several of the Cloud provider’s solutions allows us to set a sensitivity level to the guardrails configured in order to adapt the detection to the required level for the considered use-case.

Customization

Beyond sensitivity tuning, fine-grained customization is essential for effective guardrails protections. Each application has specific filtering requirements, driven by business context, regulatory constraints, and threat exposure.

Personalization is required at multiple levels:

• Business context: blocking application-specific forbidden topics, such as competitors, confidential projects, or regulated information
• Threat mitigation: adapting filters to address high-impact attacks, including indirect prompt injection
• Data flow awareness: within a single application, different data sources require different filtering strategies. User inputs, retrieved documents, and tool outputs should not be filtered identically.

Applying uniform filtering across all inputs significantly limits effectiveness and may create blind spots. Guardrails must therefore be designed as part of the application architecture, not as a single monolithic filter.

Key Insights

This study highlights several key insights:

• No single guardrails solution fits all use cases, trade-offs exist between ease of deployment, performance, and customization
• Cloud-native guardrails provide an effective and low-effort baseline for most cloud-hosted applications
• Advanced use cases require configurable solutions capable of adapting filtering logic to application context and data flows

Guardrails should be selected based on risk exposure, operational maturity, and long-term maintainability rather than raw detection scores alone.

Guardrails have become a necessary component of LLM-based applications, and a wide range of solutions is now available. Selecting the right guardrails requires identifying the solution that best aligns with an organization’s specific risks, constraints, and application architecture.

Depending on your profile we have several suggestions for you:

• If your application is already deployed in a cloud environment, using the guardrails provided by the cloud provider is a good solution.
• If you want better control over the filtering solution, deploying one of the open-source guardrails solutions may be the most suitable option.
• You want the best and have the capacity, you can issue an RFI or RFP to compare different solutions and select the most tailored to your needs.

Finally, guardrails alone are not sufficient to protect your applications. Secure LLM applications also rely on properly configured tools, strict IAM policies, and robust security architecture to prevent more severe exploitation scenarios.

Cet article GenAI Guardrails – Why do you need them & Which one should you use? est apparu en premier sur RiskInsight.

The projected impact of agentic AI is significant. By 2028, it could automate 15% of routine[1] decision-making and be embedded in a third of enterprise applications, up from virtually none today. At the same time, perceptions of risk are shifting. In early 2024, Gartner surveyed 345 senior risk executives and identified malicious AI-driven activity and misinformation as the top two emerging threats[2]. Yet despite these concerns, organisations are accelerating adoption. By 2029, agentic AI could autonomously resolve up to 80% of common customer service issues, reducing costs by as much as 30%[3]. This tension, between the growing promise of agentic AI and the expanding risk surface it introduces, raises a critical question:

“How can organisations securely deploy agentic AI at scale, balancing innovation with accountability, and automation with control?”

This article explores that question, outlining key risks, security principles, and practical guidance to help CISOs and technology leaders navigate the next wave of AI adoption.

An AI agent is an autonomous AI system in the decision-making process

In AI systems, agents are designed to process external stimuli and respond through specific actions. The capabilities of these agents can vary significantly, especially depending on whether they are powered by LLMs.

Figure 1: A diagram to show the different constituent parts of an LLM-enabled agent, showing 1) external stimuli, 2) the agents core processes (reasoning and tools) and 3) the agent’s actions

Traditional agents typically follow a rule-based or pre-programmed workflow: they receive input, classify it, and execute a predefined action. In contrast, agentic AI introduces a new dimension by incorporating LLMs to perform reasoning and decision-making between perception and action. This, with only few words to configure it. This enables more flexible, context-aware responses, and in many cases, allows AI agents to behave more like human intermediaries.

As illustrated in Figure 1, the agentic AI workflow unfolds in several stages:

1. Perception: The AI agent receives external stimuli, such as text, images, or sound.
2. Reasoning: These inputs are processed through an orchestration layer, which transforms them into structured formats using classification rules and machine learning techniques.

Here, the LLM plays a central role. It adds a layer of adaptive thinking that enables the agent to analyse context, select tools, query external data sources, and plan multi-step actions.

3. Action: With refined data and a reasoning layer applied, the agent executes complex tasks, often with greater autonomy than traditional systems.

This architecture gives agentic AI the ability to operate across dynamic environments, adapt in real time, and coordinate with other agents or systems, a key differentiator from earlier, more static automation.

In summary, AI agents with LLM capabilities can perform more complex actions by applying “AI reasoning” to transformed and refined data, making them more powerful and versatile than traditional agents.

Field insights on Agentic AI use-cases in client environments

Businesses have rightfully recognised the potential of these AI agents in a variety of use cases, ranging from the simple, to the more complex. We will now take a deeper look at some of the different common use cases across these different levels of agent autonomy.

Basic Use Cases: Chatbot/Virtual Agents

AI agents can be configured to provide instant answers to complex questions and can be designed to only answer from certain information repositories. This allows them to smoothly and effectively guide users through extensive SharePoint libraries or other document repositories. Acting as both a search function and an assistant, these agents can dramatically improve the productivity of employees by reducing the time spent searching for information and ensuring that users have quick access to the data they need. For example, a chatbot integrated into SharePoint can help employees locate specific documents, understand company policies, or even assist with onboarding processes by providing relevant information and resources. These agents have no autonomy, and only directly respond to requests as they are made by users.

Intermediate Use Cases: Routine Task Automation

Agents can be used to streamline repetitive tasks such as managing scheduling, processing customer enquiries, and handling transactions. These agents can be designed to follow specified processes and workflows, offering significant advantages over humans by reducing human error and increasing productivity. For instance, an AI agent can automatically schedule meetings by coordinating with participants’ calendars, send reminders, and process routine customer service requests such as order tracking or account updates. This automation not only saves time but also ensures consistency and accuracy in task execution. Additionally, by handling routine tasks, AI agents free up human employees to focus on more complex and strategic activities, thereby contributing to higher efficiency and productivity within the organisation.

Advanced Use Cases: Complex data analysis & vulnerability management

Agents can also be used for more complex use cases, specifically in a security context. For example, Microsoft has recently announced the release of AI agents as part of their security copilot offering, with previews releasing in April 2025. One particularly interesting use case is regarding vulnerability remediation agents. These agents will work within Microsoft Intune to monitor endpoints for vulnerabilities, assess these vulnerabilities for potential risks and impacts, and then produce a prioritised list of remediation actions. This provides a large increase in productivity for security teams, as they can then focus on the most critical issues and streamline the decision-making process. By automating the identification and prioritisation of vulnerabilities, these agents help ensure that security teams can address the most pressing threats promptly, reducing the risk of security breaches and improving overall security posture.

The promise of intelligent automation and cost efficiency is compelling, but it also introduces a strategic trade-off. CISOs will face the growing challenge of securing increasingly autonomous systems. Without robust guardrails, organisations expose themselves to operational disruption, governance failures, and reputational damage. Transparency, asset visibility, and cloud security are areas which will also require heightened vigilance and a proactive security posture. The benefits are clear, but so are the risks. Without a security-first approach, agentic AI could quickly become a liability for organisations as much as an asset.

Risks mainly known but with increased likelihood and impact

Agentic AI introduces a new level of security complexity. Unlike traditional AI systems, where threat surfaces are generally limited to inputs, model behaviour, outputs, and infrastructure, agentic AI systems operate across dynamic, autonomous chains of interaction. This covers exchanges such as agent-to-agent, agent-to-human, and human-to-agent, many of which are difficult to trace, monitor, or control in real time. As a result, the security perimeter expands beyond static models to encompass unpredictable behaviours and interactions.

Recent work by OWASP on Agents’ security[4] highlights the breadth of threats facing AI systems today. These risks span multiple domains:

• Some are traditional cybersecurity risks (e.g., data extraction, and supply chain attacks),
• Others are general GenAI risks (e.g., hallucinations, model poisonning),
• A third emerging category relates specifically to agents’ autonomy in realising actions in real world.

In addition to traditional risks, agentic AI systems introduce new security threats, such as data exfiltration through agent-driven workflows, unauthorised or unintended code execution, and “agent hijacking,” where agents are manipulated to perform harmful or malicious actions. These risks are amplified by the way many agentic AI applications are built today. Around 90% of current AI agent use cases rely on low-code platforms, prized for their speed and flexibility. However, these platforms often depend heavily on third-party libraries and components, introducing significant supply chain vulnerabilities and further expanding the overall attack surface.

Agentic AI represents a shift from passive prediction to action-oriented intelligence, enabling more advanced automation and interactive workflows. As organisations deploy networks of interacting agents, the systems become more complex, and their exposure to security risks increases. With more interfaces and autonomous exchanges, it becomes essential to establish strong security foundations early. A critical first step is mapping agent activities to maintain transparency, support effective auditing, and enable meaningful oversight.

Security Best Practices

1. Activity Mapping & Security Audits

Since AI agents operate autonomously and interact with other systems, mapping all agent activities, processes, connections, and data flows is crucial. This visibility enables the detection of anomalies and ensures alignment with security policies.

Regular audits are vital for identifying vulnerabilities, ensuring compliance, and preventing shadow AI where agents act without oversight. Unauthorised agents can expose systems to significant risks, and shadow AI, especially unsanctioned models, pose major data security threats. Auditing decision-making processes, data access, and agent interactions, along with maintaining an immutable audit trail, supports overall accountability and traceability.

To mitigate these risks, organisations should adopt clear governance policies, comprehensive training, and effective detection strategies. These practices should be backed by a strong library of AI controls and data governance policies. However, audits and governance alone aren’t enough. Robust access controls for AI agents are necessary to restrict actions and protect the system’s integrity.

      2. AI Filtering

To avoid the agent performing inappropriate actions, the first step is to ensure that its decision-making system is protected. One of the most efficient ways is by filtering potentially malicious inputs and outputs of the Decision-Maker, often composed of an orchestrator & an LLM.

Several technical ways to perform AI filtering:

Keyword filtering – Medium-Low Efficiency: Prevent the LLM from considering any input containing specified keywords and from generating any output containing these keywords.

• Pro: Quick win, particularly on the outputs, for example preventing a chatbot from generating any rude words.
• Con: Can easily be bypassed by using obfuscated inputs or requiring obfuscated outputs. For example, “p@ssword” or “p,a,s,s,w,o,r,d” can be ways to bypass the keyword “password”

LLM as-a-judge – High Efficiency: Ask to the LLM to analyse both inputs & outputs and identify if they are malicious.

• Pro: Extend the analysis to the whole answer.
• Con: Can be bypassed by overflowing the agent’s inputs, so it has trouble dealing with the whole input.

AI Classification – Very-High Efficiency: Define categories of topic that the LLM can answer or not. It can be done through whitelisting (the LLM can answer to only some categories of topics) and blacklisting (the LLM cannot answer to some precise categories of topics). Use a specialised AI system to analyse each input and output.

• Pro: Ensure the agent’s alignment by not letting it receive inputs on topics it should not be able to answer.
• Con: High cost, as it requires additional LLM analysis.

These filtering actions need to be performed for the users’ inputs, but sometimes also for the data retrieved from external sources (they can be poisoned).

      3. AI-specific Security Measures

Human-in-the-loop (HITL) oversight is essential for ensuring the responsible and secure operation of agentic AI. While AI agents can autonomously perform tasks, human review in high-risk or ethically sensitive situations provides an extra layer of judgment and accountability. This oversight helps prevent errors, biases, and unintended consequences, while allowing organisations to intervene when AI actions deviate from guidelines or ethical standards. HITL also fosters trust in AI systems and ensures alignment with business objectives and regulatory requirements. To maximise the benefits of automation, a hybrid AI-human approach is critical, supported by ongoing training to address compliance and inherent risks.

Some actions may be strictly forbidden to the agent, some should require human validation, and some could be done without human supervision. These actions should be determined through classical risk analysis, based on the agent’s impact & autonomy.

Triggers should be set-up to determine if and when human validation is needed. This can be set-up in the LLM Master Prompt, and access can be restricted by using an appropriate IAM model.

      4. Access Controls & IAM

As AI agents take on more active roles in enterprise workflows, they must be managed as non-human identities (NHIs), with their own identity lifecycle, access permissions, and governance policies. Accordingly, this requires integrating agents into existing identity and IAM frameworks, applying the same rigor used for human users.

Managing AI agents introduces new requirements. When acting on behalf of end-users, agents must be constrained to operate strictly within the permissions of those users, without exceeding or retaining elevated privileges. To achieve this, organisations should enforce key IAM principles:

• Just Enough Access (JEA): Limit agents to the minimum set of permissions required to complete specific tasks.
• Just in Time (JIT) access: Provision access temporarily and contextually to reduce standing privileges and exposure.
• Segregation of duties and scoped credentials: Define clear boundaries between roles and prevent unauthorised privilege escalation.

In addition, to further enhance control, security teams should implement real-time anomaly detection to monitor agent behaviour, flag policy violations, and automatically remediate or escalate issues when necessary.

Access to sensitive data must also be tightly restricted. Violations should trigger immediate revocation of privileges and deny lists should be used to block known malicious patterns or endpoints.

Ultimately, while technical controls are essential, they should be supported by human oversight and governance mechanisms, particularly when agents operate in high-impact or sensitive contexts. IAM for agentic AI must evolve in step with these systems’ increasing autonomy and integration into critical business functions.

      5. AI Crisis Response & Red teaming

While AI-specific controls are essential, traditional measures like crisis management must also extend into the AI landscape. As cyberattacks become more sophisticated, organisations should consider crisis management strategies for potential AI failures or compromises; by ensuring all teams such as AI scientists, operational teams, and security teams are equipped to respond quickly and effectively to minimise disruption.

Concrete guidelines for CISOs

This year CISOs will be exposed to increased threats introduced by agentic AI alongside ongoing regulatory pressure from complex regulations such as DORA, NIS 2 and the AI Act. Both CISOs and CTOs will collaborate closely, with CISOs overseeing the secure deployment of AI systems to ensure that agent interactions are carefully mapped and secured to safeguard the security of their organisations, workforce and customers.

Key starting points for CISOs:

• Limit access to AI agents by enforcing strong access controls and aligning with existing IAM policies.
• Monitor agent behaviour by tracking activity and conducting regular audits to identify vulnerabilities.
• Filter the agent’s inputs and outputs to ensure that the decision-maker does not launch any unwilled action.
• Implement Human-in-the-Loop oversight to validate AI outputs for critical decisions/tasks.
• Provide agentic AI awareness training to educate employees on the risks, security best practices and identifying potential attacks.
• Perform AI red teaming on the agent, to identify potential weaknesses.
• Despite all security measures, AI operates on probabilistic principles rather than deterministic ones. This means that the agent might occasionally behave inappropriately. Therefore, it’s crucial to establish clear accountability for any wrongful actions taken by AI agents.
• Prepare for AI crises early by initiating discussions with relevant teams to ensure a coordinated response if an incident occurs.

Over the past several years, Wavestone has observed a marked increase in client maturity around AI security. Many organisations have already implemented robust processes to assess the sensitivity of AI initiatives and to manage associated risks. These early efforts have proven valuable in reducing exposure and strengthening governance.

While agentic AI does not fundamentally rewrite the AI security playbook, it does introduce a meaningful shift in the risk landscape. Its inherently autonomous, interconnected nature increases both the impact and likelihood of certain threats. The complexity of these systems can be challenging at first, but they are manageable. With a clear understanding of these dynamics and the emergence of new market standards and security protocols, agentic AI can deliver on its transformative potential.

As this transition unfolds, we remain committed to helping CISOs and their teams navigate the evolving risk environment with confidence.

References

[1] Orlando, Fla., Gartner Identifies the Top 10 Strategic Technology Trends for 2025, October 21, 2024. https://www.gartner.com/en/newsroom/press-releases/2024-10-21-gartner-identifies-the-top-10-strategic-technology-trends-for-2025

[2] Stamford, Conn., Gartner Predicts Agentic AI Will Autonomously Resolve 80% of Common Customer Service Issues Without Human Intervention by 2029, March 5, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-03-05-gartner-predicts-agentic-ai-will-autonomously-resolve-80-percent-of-common-customer-service-issues-without-human-intervention-by-20290

[3] Stamford, Conn. Gartner Survey Shows AI-Enhanced Malicious Attacks Are a New Top Emerging Risk for Enterprises, May 22, 2024. https://www.gartner.com/en/newsroom/press-releases/2024-05-22-gartner-survey-shows-ai-enhanced-malicious-attacks-are-new0

[4] OWASP, OWASP Top 10 threats and mitigation for AI Agents, 2025. OWASP-Agentic-AI/README.md at main · precize/OWASP-Agentic-AI · GitHub

Thank you to Leina HATCH for her valuable assistance in writing this article.

Cet article Agentic AI: typology of risks and security measures est apparu en premier sur RiskInsight.

This type of platform can be used to create applications based on generative AI models such as LLMs (GTP-3.5, Mistral, etc.). 

Nevertheless, the adoption of this technology is not without risk: from virtual assistants criticizing their companies [3] to data leaks [4]; there is no shortage of examples. 

To support the many deployments currently underway, you need to think quickly about your security, particularly when sensitive data is being used. In this article, we take a look at the risks and mitigations associated with using these platforms. 

Which model is right for you? 

Three types of generative AI can be used to create an application. The difference lies in the precision of the answers provided:  

1. Simple: generic AI model (GPT-4, Mistral, etc.) plugged in as such, with a user interface. It is an internal GPT. 
2. Boosted: generic AI model that leverages the company’s data, for example via RAG (Retrieval Augmented Generation). These are specialized companions for a particular use, HR GPT, Operations GPT, CISO GPT…). 
3. Specialized: the AI model retrained for a particular use. For example, India has retrained Llama 3 for its 22 official languages to make it a specialized translator. 

All three deployment methods entail risks. We will begin by describing the different modes. We will then look at the risks, and the associated mitigations. 

Risks and models 

Simple model 

This model is the simplest to deploy. It allows users to interact with the AI models proposed by the platforms. It simplifies the integration of sending prompts and receiving responses in an application. It is an internal ChatGPT, with the advantage of limiting the leakage of sensitive data inserted into a prompt, unlike the web version. Also, in this case, exchanges with users are not used to re-train and improve the model. Your data is protected. The Cloud platforms offered by Azure, AWS or GCP enable these solutions to be deployed rapidly. 

Examples of use: text summary, development assistant. 

How the simple model works

Boosted model 

This model remains generic, but will have access to selected company data. The AI could, for example, consult the group’s PSSI to provide the password policy. 

Examples of use: enterprise chatbot, data analysis. 

How the boosted model works

Specialized model 

The application is no longer based on a generic model (GPT-4, Mistral, etc.). Before using it, you will need to train your own model on your company’s data. It will always be able to consult the company’s data and will have a better understanding of it to generate its response. 

Examples of applications: fault detection on a production line, medical diagnostics. 

How the specialized model works

What risks are you exposed to? 

Regardless of the model selected, there are a number of transversal or specific risks. It is important to take these into account to ensure that the solution is securely integrated. 

Hijacking the model 

AI models are exposed to the risk of misuse. Imagine a scenario where someone uses this technology to generate harmful content. This could lead to real consequences such as the propagation of toxic content. One known attack for this purpose is Prompt Injection [5]. 

Example – Model hijacking (Prompt Injection)

Hallucination 

When AI asserts information that is false, it hallucinates. Think of it as “daydreaming”: if it doesn’t have the answer, it will “invent” things to fill the void. This can be particularly problematic in situations where accuracy is crucial: generating reports, making decisions, etc. Users could unknowingly spread this false information, or make bad decisions.  

Example – Model hallucination

Data leakage 

There are several ways in which data can be leaked. An attacker can inject a malicious prompt to retrieve it, or an employee can be given more rights than necessary and access sensitive information (e.g. strategic minutes of an executive committee meeting). The security of the underlying database must therefore be proportional to the amount of data stored. 

The model has access to certain company data. If, for example, its rights are too extensive, it will be able to consult confidential data. These responses will therefore include sensitive information that should not be disclosed. 

Example – Data leak

Model theft 

If the model is specialized, it is now your company’s intellectual property. As such, it could be a target for attackers. Confidential training data, for example, could be targeted. The question of trust in the Cloud host may also arise: wouldn’t it be better to host it locally? 

 Example – Model theft

Poisoning the model 

Without claiming to steal the model, the attacker’s aim could be to make it unreliable. The responses generated could then no longer be used by the teams. 

Poisoning can occur in two ways:  

• Boosted model: the attacker accesses the RAG and modifies the information. The model then relies on poisoned data to provide its answers.  

• Specialized model: the attacker poisons the model’s training data. Either directly on the database that he makes available on a public platform (Hugging face type), or by accessing the training database hosted in your information system. 

 Example – Poisoning the model

Main risks: what mitigations? 

Of the 5 risks presented, 3 dominate in the risk analyses carried out by our teams. We suggest you study the associated mitigations. 

The novelty of the technology provides an opportunity to build a solid security foundation. Several iterations will be necessary to achieve an effective and secure solution. 

Risk #1: Hijacking the model 

Hijacking the model and the key to remediation

We recommend the following measures to prevent the model from being hijacked: 

#1 – Toughen the configuration in two ways. Firstly, management of the master prompt (discussion window with the model). Certain keywords, for example, can be banned to prevent abuse. Secondly, the number of tokens and therefore the size of responses. A less verbose model will have less chance of being hijacked. Other parameters can be taken into account: temperature, language used, etc. 

#2 – Filter responses by applying, for example, a simple response filtering algorithm. To go further, it is possible to deploy specialised LLM firewalls. This would make it possible, for example, to prevent potential abuse (this is known as abuse monitoring). 

#3 – Limit the sources to which the model has access to generate its responses. If the model is given access to company data, it can be limited to this data only. In this way, it will not be able to search for other information on the Internet, for example.  

Risk #2: Hallucination 

 Hallucination and the key to remediation

To deal with hallucinations, we recommend the following measures: 

#1 – Train and educate users on how models work, their limitations and best practices. This enables users to use Large Language Models responsibly and to recognise misuse or potential security threats. 

#2 – Toughen the configuration in two ways. Firstly, adjusting the parameters, including setting the model temperature (how creative the model is) and limiting the number of tokens (number of words per question/answer). Secondly, the use of a more recent model (GPT-4 rather than GPT 3.5 for example). 

#3 – Optional – Re-training the model gives it a context. This will have a positive impact on the reliability of responses. Using a wide range of training data can help to cover more scenarios and reduce bias, which helps AI to better understand and generate appropriate responses. Similarly, eliminating errors and inconsistencies in training data can reduce the likelihood of the AI learning and repeating these same errors. 

Risk #3: Data leakage 

 Data leakage and the key to remediation

To deal with leaks of sensitive data, we recommend the following measures: 

#1 – Ensuring compliance with data protection laws and protocols by involving the Data Protection Officer (DPO) in projects accessing Large Language Model platforms is important to protect personal and sensitive data. By adhering to these standards, organizations not only protect individual privacy but also strengthen their defense against data breaches and misuse. 

#2 – Manage rights and access to all components interacting with the model. Understanding which data can be accessed by the model is not trivial. Auditing and recertifying this data over time helps to limit potential discrepancies. 

#3 – Reduce the verbosity of the model by limiting the number of output tokens. The less verbose a model is, the lower the probability that it will inadvertently share confidential data. 

#4 – Anonymize the data, or make it generic, if the use case allows. For example, AI will be able to work on population trends without an explicit name being cited. As well as greatly reducing the risk of data leakage, this will reduce the standards to be complied with (e.g. RGPD). 

#5 – Limit the amount of sensitive data used. Here we need to think about what data is necessary and sufficient for the model to work. The data can be processed beforehand to remove or modify sensitive data and thus reduce exposure (e.g. data anonymization). 

Cross-disciplinary mitigations 

Certain measures apply to all the risks listed above. Two of them are fundamental.  

#1 – Integrate security into projects via, for example, contextualized security analysis. This enables organizations to preventively identify and mitigate potential vulnerabilities, ensuring that only secure and verified projects access generative AI applications.  

#2 – Document each application to establish an operational framework that not only facilitates easier supervision and management, but also reduces the risk of unauthorized or malicious use.  

The development of AI applications is accelerated by the platforms available. However, the sophistication it brings is not without risk.  

Recognizing these challenges, the priority is to establish robust governance for the platform. This involves delineating roles and responsibilities, ensuring a structured approach to managing and mitigating risks. 

Governance extends beyond the platform itself. Securing the myriads of AI application use cases is just as important. It’s about ensuring that the application of this AI technology is both responsible and aligned with ethical standards, guarding against misuse and unintended consequences. 

This calls for a model of shared responsibility, where all stakeholders – developers, users and governance bodies – work together to maintain the integrity and security of AI applications. 

References 

1. https://synthedia.substack.com/p/microsoft-azure-ai-users-base-rose 
2. https://www.usine-digitale.fr/article/amazon-fait-son-entree-sur-le-marche-de-l-ia-generative-avec-bedrock.N2121081  
3. https://www.theguardian.com/technology/2024/jan/20/dpd-ai-chatbot-swears-calls-itself-useless-and-criticises-firm 
4. https://openai.com/blog/march-20-chatgpt-outage 
5. https://www.riskinsight-wavestone.com/2023/10/quand-les-mots-deviennent-des-armes-prompt-injection-et-intelligence-artificielle/ 

Cet article Generative AI applications: risks and mitigations  est apparu en premier sur RiskInsight.