As part of its commitment to supporting the digital transformation of its clients while limiting the risks involved, Wavestone’s Cyber teams invites you to discover how cyber-attacks can be carried out on an AI system and how to protect against them.

Attacking an internal AI system (our CISO hates us)

Approach and objectives

As demonstrated by recent work on AI[1] systems by ENISA[2] and NIST[3], AI is vulnerable to a number of cyber threats. These threats can be generic or specific, but impact all AI systems based on Machine Learning.

To check the feasibility of such threats, we wanted to test Evasion and Oracle threats on one of our low-impact internal applications: Artistic, a tool for classifying employee tickets for IT support.

To do this, we put ourselves in the shoes of a malicious user who, knowing that ticket processing is based on an Artificial Intelligence algorithm, would try to carry out Evasion or Oracle-type attacks.

Obviously, the impact of such attacks is very low, but our AI is a great playground for experimentation.

Application overview

Application architecture

Evasion attack

Approach overview

An evasion attack consists of hijacking the artificial intelligence by providing it with contradictory examples (also known as “adversarial examples”) in order to create inaccurate predictions. An adversarial example is an input with intentional mistakes or changes that cause a machine learning model to make a false prediction. These mistakes or changes can easily go unnoticed by a human, such as a typo in a word, but radically alter the model’s output data.

For our example, we will try to build different contradictory examples using three techniques:

• Deleting and changing characters
• Replacing words using a dedicated technique (Embedding)
• Changing the position of words

The contradictory examples in our use case are slightly modified written requests (see example 1 below) which will be categorised in the Artistic ticketing tool.

To do this, we’re going to use a dedicated tool: TextAttack. TextAttack is a Python framework for performing evasion attacks (interesting for our case), training an NLP model with contradictory examples, and performing data augmentation in the NLP domain.  

Results

Consider a sentence correctly classified by our Artificial Intelligence with a high probability. Let’s now apply the TextAttack Framework and use it to generate contradictory examples based on our correctly classified sentence.

We have observed that sentences which are (more or less) comprehensible to a person can confuse the Artificial Intelligence to the point of misclassifying them. In addition, we can see that with a multitude of contradictory examples created, it is possible for the model to assign the same message to each of the classification categories with varying accuracy rates.

By extension, with more critical Artificial Intelligence models, these poor predictions cause a number of problems:

• Security breaches: the model in question is compromised and it becomes possible for attackers to obtain inaccurate predictions
• Reduced confidence in AI systems: such an attack reduces confidence in AI and the choice of adopting such models, calling into question the potential of this technology

However, according to ENISA, a number of measures can be implemented to be protected against this type of attack:

• Define a model that is more robust against evasion attacks. Artistic’s AI system is not particularly robust to these attacks and is very basic in its operation (as we shall see later). A different model would certainly have been more resistant to evasion attacks.
• Adversarial training during the model learning phase. This consists of adding examples of attacks to the training data so that the model improves its ability to classify “strange” data correctly.
• Implement checks on the model’s input data to ensure the ‘quality’ of the words entered.

Oracle Attack

Definition

Oracle attacks involve studying AI models and attempting to obtain information about the model by interacting with it via queries. Unlike evasion attacks, which aim to manipulate the input data of an AI model, Oracle attacks attempt to extract sensitive information about the model itself and the data it has manipulated (the type of training data used, for example).

In our use case, we are simply trying to understand how the model works. To do this, we sought to understand the model’s behaviour by analysing the input-output pairs provided by our contradictory examples.

Results

By going through several trials, the attacker may be able to detect the sensitivity of the model to changes in the input data. From the example above, we can see that the algorithm used by the application predicts the class of a message by assigning a score to each word and then determines the category. By analysing these various results, the attacker may be able to deduce the model’s vulnerabilities to evasion attacks.

By extension, on more critical Artificial Intelligences, Oracle-type attacks pose several problems:

• Infringement of intellectual property: as mentioned, the Oracle attack can allow the theft of the model architecture, hyperparameters, etc. Such information can be used to create a replica of the model.
• Attacks on the confidentiality of training data: this attack may reveal sensitive information about the training data used to train the model, which may be confidential.

A few measures can be implemented to protect against this type of attack:

• Define a model that is more robust to Oracle-type attacks. Artistic’s AI system is very basic and easy to understand.
• For AI more broadly, ensure that the model respects differential privacy. Differential privacy is an extremely strong definition of privacy that guarantees a limit to what an attacker with access to the results of the algorithm can learn about each individual record in the dataset.

Getting to grips with the subject in your organisation today

We have observed that even without precise knowledge of the parameters of an Artificial Intelligence model, it is relatively easy to carry out Evasion or Oracle-type attacks.

In our case, the impact is limited. However, the consequences of an evasion attack on an autonomous vehicle or an Oracle-type attack on a model used with health data are far more serious for individuals: physical damage in one case and invasion of privacy in the other.

A number of our customers are already starting to deploy initial measures to deal with the cyber risks created by the use of AI systems. In particular, they are developing their risk analysis methodology to take account of the threats outlined above, and most importantly they are putting in place relevant countermeasures, based on security guides such as those proposed by ENISA or NIST.

[1] An artificial intelligence system, in the AI Act legislative proposal, is defined as “software developed using one or more of the techniques and approaches listed in Annex I of the proposal and capable, for a given set of human-defined goals, of generating results such as content, predictions, recommendations, or decisions influencing the environments with which they interact.” In our paper, we consider that AI systems have been trained via Machine Learning, as is generally the case on modern use cases such as ChatGPT.

[2] https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms

[3] https://csrc.nist.gov/publications/detail/white-paper/2023/03/08/adversarial-machine-learning-taxonomy-and-terminology/draft

[4] A ticket represents a sequence of words (in other words, a sentence) in which the employee expresses his or her need.

Cet article Attacking AI? A real-life example! est apparu en premier sur RiskInsight.

But how to strengthen the detection of an intrusion into one’s company? Penetration testing allows us to evaluate the level of protection/hardening on a precise and supervised perimeter, which does not necessarily represent the reality experienced during a real situation. Crisis management simulations help to improve the response… Red Team operations can be a good element of response, allowing the information system and detection capabilities to be evaluated (and therefore improved) from the beginning to the end of a cyberattack.

What is a Red Team operation? It is simply a realistic attack without the negative effects. The objective is to determine, today, what malicious actions a group of attackers can carry out on my company and when am I able to detect them?

In this article, we will see what the key steps are in such an operation and how to ensure that we reap the benefits.

STEP 1: DEFINE THE CORE TEAM AND IDENTIFY THE TROPHY

Defining a restricted control team is essential to limit the leakage of information to the Blue Team (detection team, i.e. SOC), to make communications more fluid and to facilitate decision-making by the client. The control team must therefore be clearly separated from the Blue Team. Often the choice is made between the CISO or a representative of the ExCom, with whom the limits of the perimeter and the operating mode (as wide as possible) will have to be set, in order to avoid any unfortunate incident!

“That’s out of scope” – Said no attacker ever

Then, a particularity of a Red Team operation is to define a “trophy”, the final target of the operation. Indeed, an attacker is often motivated by an objective (gain, destruction, data theft, …) and it is advisable to copy the Red Team’s objectives on it. The Red Team generally already has some good ideas but the best trophies (i.e. the strongest impact during the restitution) are closely linked to the business stakes of the company and its current events.

The target must be the scenarios that are the most chilling for the managers: a remote takeover of the SWIFT infrastructure? A compromise of the payment terminals? The leakage of the VIP customer list? Positions taken in recent months abroad? The ideas may be numerous, but it is necessary to restrict oneself to one or two targets to keep the most critical ones visible. It will always be possible to identify another trophy for the next operation: it is even advisable to modify the trophies from one year to the next in order to test different parts of the information system.

During our latest operations, ExComs have chosen scenarios that have already taken place during real cyber attacks they have been subjected to. This also enables them to assess the effectiveness of the new security measures implemented.

STEP 2: PREPARE AND LAUNCH THE ASSAULT BY MIXING EFFICIENCY AND STEALTH

The credibility of the attack is one of the key factors in the success of the operation, particularly during the restitution phase. Once the trophy has been arrested: we build an approach based on the techniques used by the attacking groups.

It is at this point that field returns are particularly useful! At Wavestone, we rely heavily on our Incident Response Team (CERT-W) and its Threat Intelligence capacity to identify the latest trends, on the technical know-how of our audit team and the creativity of our CTF (Capture The Flag) team.

Thus, the Red Team will use all possible and necessary means to penetrate the IS (phishing campaign, telephone phishing, physical intrusion, compromise of components exposed on the internet…) and then bounce back to the trophy. This phase is the most creative and exciting for the listeners (as well as for the attackers), and can potentially last several weeks, just like the most high-profile cyber-attacks.

However, the key word must remain: “stealth”! Indeed, the slightest detection by the Blue Team can totally derail or set back the operation. Particular attention must be paid to customised attack tools and infrastructures to avoid panicking the detection systems. For the first point, we have developed internally the Abaddon tool, now open-source, which allows us to build and deploy the necessary infrastructures in just a few clicks.

Two questions always come up when it comes to the conduct of a Red Team operation.

The first one is “What to do in case of detection? ». That an action may be detected can happen during a Red Team operation: a good SIEM correlation rule, an informed user who shares an abnormal behaviour, etc… First of all, in the basic organisation, the control team must supervise incidents within the SOC in order to avoid an “over-escalation” in crisis of an incident related to the Red Team. The control unit will then be able to request a report from the Blue Team (summary, detected behaviour, timeline of actions, remedial actions, etc.) and then define the recovery scenario with the Red Team: ignore the detection because it is too old, start from the penultimate compromised asset that has not been detected, etc. Moreover, the work will become more complicated for the Red Team, which will have to completely change its C2 infrastructure and henceforth succeed in deceiving a Blue Team on alert.

The second “If the trophy(s) are obtained very quickly, what can be done?” Let’s imagine the worst case scenario: the application administrator of the trophy gets trapped by a phishing email and allows us to take complete control of the application from the very beginning of the operation (Anyone can make a mistake). The point will be shared with the control team and the procedure to be followed will be defined jointly: add trophies to test the robustness of another perimeter, start from scratch and identify another compromise path, … This somewhat caricatural example is there to remind us that the objective of a Red Team operation is to durably improve the level of security via Blue Team training and not just to obtain a trophy.

STEP 3: PROVIDE A CLEAR RESPONSE TO A CRITICAL BUSINESS RISK

The objective remains to provide sponsors with a clear vision of the real security status of their IS, the attack scenarios that will give them access to their critical resources (identified as “trophies” of the operation), as well as their detection capabilities. Quite simply, the Red Team operation must make it possible to answer the question “Is the trophy accessible and with what level of expertise? ». However, it should be remembered that Operation Red Team will highlight an exploitable path that may not be the only or the simplest one.

From then on, we return to the stakeholders (CISO, SOC, COMEX…) with a high-level synthesis in order to present the conclusions of the Red Team operation, the attack scenario followed and the most priority worksites. The results are generally compared with typical attacker profiles (Maze, REvil/Sodinoki…) on the MITRE ATT&CK reference frame to be more meaningful.

In a second stage, the Blue Team will obviously have to be given a detailed account of the technical stages of the operation, with a view to defining areas of progress on detection.

“If we win, we lose ” Said a good red teamer

This second phase is fundamental for the operation to have the expected added value: pedagogy and clarity are needed to get the right messages across! Let’s not hesitate to hold additional workshops to explain the problems raised by the operation and to find solutions together. A joint interpretation of the findings by the Blue Team and the Red Team allows us to take a step back from the vulnerabilities and identify concrete actions for improvement.

The Red Team operation should not be reduced to correcting a few vulnerabilities on the IS, but should make it possible to obtain the effective level of security (even if it is not exhaustive, as a Red Team will never be an audit).

In a few words, a Red Team operation makes it possible to test its defence strategy on a large scale and to train (improve) its defence team. The very concrete nature of the trophies allows an understanding and awareness of the cyber risk of the decision-makers.

Cet article How to take advantage of a red team operation? est apparu en premier sur RiskInsight.