{"id":24514,"date":"2024-11-06T17:22:04","date_gmt":"2024-11-06T16:22:04","guid":{"rendered":"https:\/\/www.riskinsight-wavestone.com\/?p=24514"},"modified":"2024-11-07T16:08:49","modified_gmt":"2024-11-07T15:08:49","slug":"generative-ai-applications-risks-and-mitigations","status":"publish","type":"post","link":"https:\/\/www.riskinsight-wavestone.com\/en\/2024\/11\/generative-ai-applications-risks-and-mitigations\/","title":{"rendered":"Generative AI applications: risks and mitigations\u00a0"},"content":{"rendered":"\n

Microsoft has announced that in Q2 2024 <\/span>“more than half of Fortune 500 companies will be using Azure OpenAI”<\/span><\/i>. [1<\/a>] At the same time, AWS is offering Bedrock [2<\/a>], a direct competitor to Azure OpenAI.<\/span>\u00a0<\/span><\/p>\n

This type of platform can be used to create applications based on generative AI models such as LLMs (GTP-3.5, Mistral, etc.).<\/span>\u00a0<\/span><\/p>\n

Nevertheless, the adoption of this technology is not without risk: from virtual assistants criticizing their companies [3<\/a>] to data leaks [4<\/a>]; there is no shortage of examples.<\/span>\u00a0<\/span><\/p>\n

To support the many deployments currently underway, you need to think quickly about your security, particularly when sensitive data is being used. In this article, we take a look at the risks and mitigations associated with using these platforms.<\/span>\u00a0<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Which model is right for you?<\/span>\u00a0<\/span><\/h2>\n

Three types of generative AI can be used to create an application. The difference lies in the precision of the answers provided:\u00a0<\/span>\u00a0<\/span><\/p>\n

    \n
  1. Simple<\/span><\/b>: generic AI model (GPT-4, Mistral, etc.) plugged in as such, with a user interface. <\/span>It is an internal GPT.<\/span>\u00a0<\/span><\/li>\n
  2. Boosted<\/span><\/b>: generic AI model that leverages the company’s data, for example via RAG (<\/span>Retrieval Augmented Generation). <\/span><\/i>These are specialized companions for a particular use, HR GPT, Operations GPT, CISO GPT…).<\/span>\u00a0<\/span><\/li>\n
  3. Specialized<\/span><\/b>: the AI model retrained for a particular use. For example, India has retrained Llama 3 for its 22 official languages to make it a specialized translator.<\/span>\u00a0<\/span><\/li>\n<\/ol>\n

    All three deployment methods entail risks. We will begin by describing the different modes. We will then look at the risks, and the associated mitigations<\/span>.<\/span>\u00a0<\/span><\/p>\n

    \"\"<\/span><\/p>\n

    Risks and models<\/span><\/i>\u00a0<\/span><\/p>\n

    \u00a0<\/p>\n

    Simple model<\/span>\u00a0<\/span><\/h3>\n

    This model is the simplest to deploy. It allows users to interact with the AI models proposed by the platforms. It simplifies the integration of sending prompts and receiving responses in an application. <\/span>It is an internal ChatGPT, with the advantage of limiting the leakage of sensitive data inserted into a prompt, unlike the web version. Also, in this case, exchanges with users are not used to re-train and improve the model. Your data is protected. The Cloud platforms offered by Azure, AWS or GCP enable these solutions to be deployed rapidly.<\/span>\u00a0<\/span><\/p>\n

    Examples of use: text summary, development assistant.<\/span>\u00a0<\/span><\/p>\n

    \"\"<\/span><\/p>\n

    How the simple model works<\/span><\/i><\/p>\n

    \u00a0<\/span><\/p>\n

    Boosted model<\/span>\u00a0<\/span><\/h3>\n

    This model remains generic, but will have access to selected company data. The AI could, for example, consult the group’s PSSI to provide the password policy.<\/span>\u00a0<\/span><\/p>\n

    Examples of use: enterprise chatbot, data analysis.<\/span>\u00a0<\/span><\/p>\n

    \"\"<\/span><\/p>\n

    How the boosted model works<\/span><\/i><\/p>\n

    \u00a0<\/p>\n

    Specialized model<\/span>\u00a0<\/span><\/h3>\n

    The application is no longer based on a generic model (GPT-4, Mistral, etc.). Before using it, you will need to train your own model on your company’s data. It will always be able to consult the company’s data and will have a better understanding of it to generate its response.<\/span>\u00a0<\/span><\/p>\n

    Examples of applications: fault detection on a production line, medical diagnostics.<\/span>\u00a0<\/span><\/p>\n

    \"\"<\/span><\/p>\n

    How the specialized model works<\/span><\/i><\/p>\n

    \u00a0<\/span><\/p>\n

    What risks are you exposed to?<\/span>\u00a0<\/span><\/h2>\n

    Regardless of the model selected, there are a number of transversal or specific risks. It is important to take these into account to ensure that the solution is securely integrated.<\/span>\u00a0<\/span><\/p>\n

    Hijacking the model<\/span>\u00a0<\/span><\/h3>\n

    AI models are exposed to the risk of misuse. Imagine a scenario where someone uses this technology to generate harmful content. This could lead to real consequences such as the propagation of toxic content. <\/span>One known attack for this purpose is <\/span>Prompt Injection <\/span><\/i>[5<\/a>].<\/span>\u00a0<\/span><\/p>\n

    \"\"<\/span><\/p>\n

    Example – Model hijacking (Prompt Injection)<\/span><\/i><\/p>\n

    \u00a0<\/span><\/p>\n

    Hallucination<\/span>\u00a0<\/span><\/h3>\n

    When AI asserts information that is false, it hallucinates. Think of it as “daydreaming”: if it doesn’t have the answer, it will “invent” things to fill the void. This can be particularly problematic in situations where accuracy is crucial: generating reports, making decisions, etc. Users could unknowingly spread this false information, or make bad decisions.\u00a0<\/span>\u00a0<\/span><\/p>\n

    \"\"<\/span><\/p>\n

    Example – Model hallucination<\/span><\/i><\/p>\n

    \u00a0<\/span><\/p>\n

    Data leakage<\/span>\u00a0<\/span><\/h3>\n

    There are several ways in which data can be leaked. An attacker can inject a malicious prompt to retrieve it, or an employee can be given more rights than necessary and access sensitive information (e.g. strategic minutes of an executive committee meeting). The security of the underlying database must therefore be proportional to the amount of data stored.<\/span>\u00a0<\/span><\/p>\n

    The model has access to certain company data. If, for example, its rights are too extensive, it will be able to consult confidential data. These responses will therefore include sensitive information that should not be disclosed.<\/span>\u00a0<\/span><\/p>\n

    \"\"<\/span><\/p>\n

    Example – Data leak<\/span><\/i><\/p>\n

    \u00a0<\/p>\n

    Model theft<\/span>\u00a0<\/span><\/h3>\n

    If the model is specialized, it is now your company’s intellectual property. As such, it could be a target for attackers. Confidential training data, for example, could be targeted. The question of trust in the Cloud host may also arise: wouldn’t it be better to host it locally?<\/span>\u00a0<\/span><\/p>\n

    \"\"<\/span><\/p>\n

    \u00a0Example – Model theft<\/span><\/i><\/p>\n

    \u00a0<\/span><\/p>\n

    Poisoning the model<\/span>\u00a0<\/span><\/h3>\n

    Without claiming to steal the model, the attacker’s aim could be to make it unreliable. The responses generated could then no longer be used by the teams.<\/span>\u00a0<\/span><\/p>\n

    Poisoning can occur in two ways:\u00a0<\/span>\u00a0<\/span><\/p>\n