共找到 20 条结果
We investigate the scale of attack and defense mathematically in the context of AI's possible effect on cybersecurity. For a given target today, highly scaled cyber attacks such as from worms or botnets typically all fail or all succeed. Here, we consider the effect of scale if those attack agents were intelligent and creative enough to act independently such that each attack attempt was different from the others or such that attackers could learn from their successes and failures. We find that small increases in the number or quality of defenses can compensate for exponential increases in the number of independent attacks and for exponential speedups.
This paper is based on a three year project during which we studied attackers' behavior, reading military planning literature, and thinking on how would we do the same things they do, and what problems would we, as attackers, face. This research is still ongoing, but while participating in applications for other projects and talking to cyber security experts we constantly face the same issues, namely attackers' behavior is not well understood, and consequently, there are a number of misconceptions floating around that are simply not true, or are only partially true. This is actually expected as someone who casually follows news about incidents easily gets impression that attackers and attacks are everywhere and every one is under attack. Our goal in this paper is to debunk these myths, to show what attackers really can and can not, what dilemmas they face, what we don't know about attackers and attacks, etc. The conclusion is that, while attackers do have upper hand, they don't have absolute advantage, i.e. they also operate in an uncertain environment. Knowing this, means that defenses could be well established.
Cooperative multi-agent reinforcement learning (CMARL) has shown to be promising for many real-world applications. Previous works mainly focus on improving coordination ability via solving MARL-specific challenges (e.g., non-stationarity, credit assignment, scalability), but ignore the policy perturbation issue when testing in a different environment. This issue hasn't been considered in problem formulation or efficient algorithm design. To address this issue, we firstly model the problem as a limited policy adversary Dec-POMDP (LPA-Dec-POMDP), where some coordinators from a team might accidentally and unpredictably encounter a limited number of malicious action attacks, but the regular coordinators still strive for the intended goal. Then, we propose Robust Multi-Agent Coordination via Evolutionary Generation of Auxiliary Adversarial Attackers (ROMANCE), which enables the trained policy to encounter diversified and strong auxiliary adversarial attacks during training, thus achieving high robustness under various policy perturbations. Concretely, to avoid the ego-system overfitting to a specific attacker, we maintain a set of attackers, which is optimized to guarantee the attackers
As collaborative learning and the outsourcing of data collection become more common, malicious actors (or agents) which attempt to manipulate the learning process face an additional obstacle as they compete with each other. In backdoor attacks, where an adversary attempts to poison a model by introducing malicious samples into the training data, adversaries have to consider that the presence of additional backdoor attackers may hamper the success of their own backdoor. In this paper, we investigate the scenario of a multi-agent backdoor attack, where multiple non-colluding attackers craft and insert triggered samples in a shared dataset which is used by a model (a defender) to learn a task. We discover a clear backfiring phenomenon: increasing the number of attackers shrinks each attacker's attack success rate (ASR). We then exploit this phenomenon to minimize the collective ASR of attackers and maximize defender's robustness accuracy by (i) artificially augmenting the number of attackers, and (ii) indexing to remove the attacker's sub-dataset from the model for inference, hence proposing 2 defenses.
In the prior literature on multi-agent area defense games, the assignments of the defenders to the attackers are done based on a cost metric associated only with the interception of the attackers. In contrast to that, this paper presents an Inter-Defender Collision-Aware Interception Strategy (IDCAIS) for defenders to intercept attackers in order to defend a protected area, such that the defender-to-attacker assignment protocol not only takes into account an interception-related cost but also takes into account any possible future collisions among the defenders on their optimal interception trajectories. In particular, in this paper, the defenders are assigned to intercept attackers using a mixed-integer quadratic program (MIQP) that: 1) minimizes the sum of times taken by defenders to capture the attackers under time-optimal control, as well as 2) helps eliminate or delay possible future collisions among the defenders on the optimal trajectories. To prevent inevitable collisions on optimal trajectories or collisions arising due to time-sub-optimal behavior by the attackers, a minimally augmented control using exponential control barrier function (ECBF) is also provided. Simulation
In the era of a data-driven society with the ubiquity of Internet of Things (IoT) devices storing large amounts of data localized at different places, distributed learning has gained a lot of traction, however, assuming independent and identically distributed data (iid) across the devices. While relaxing this assumption that anyway does not hold in reality due to the heterogeneous nature of devices, federated learning (FL) has emerged as a privacy-preserving solution to train a collaborative model over non-iid data distributed across a massive number of devices. However, the appearance of malicious devices (attackers), who intend to corrupt the FL model, is inevitable due to unrestricted participation. In this work, we aim to identify such attackers and mitigate their impact on the model, essentially under a setting of bidirectional label flipping attacks with collusion. We propose two graph theoretic algorithms, based on Minimum Spanning Tree and k-Densest graph, by leveraging correlations between local models. Our FL model can nullify the influence of attackers even when they are up to 70% of all the clients whereas prior works could not afford more than 50% of clients as attacke
This work investigates the problem of synthesizing obfuscated supervisors against covert actuator attackers. For a non-resilient supervisor S, for which there exist some covert actuator attackers that are capable of inflicting damage, we propose an algorithm to compute all the obfuscated supervisors, with the requirements that: 1) any obfuscated supervisor S' is resilient against any covert actuator attacker, and 2) the original closed-behavior of the closed-loop system under S is preserved, that is, any obfuscated supervisor S' is control equivalent to the original non-resilient supervisor S. We prove that the designed algorithm to synthesize obfuscated supervisors against covert actuator attack is sound and complete.
This paper is concerned with the optimal allocation of detection resources (sensors) to mitigate multi-stage attacks, in the presence of the defender's uncertainty in the attacker's intention. We model the attack planning problem using a Markov decision process and characterize the uncertainty in the attacker's intention using a finite set of reward functions -- each reward represents a type of the attacker. Based on this modeling framework, we employ the paradigm of the worst-case absolute regret minimization from robust game theory and develop mixed-integer linear program (MILP) formulations for solving the worst-case regret minimizing sensor allocation strategies for two classes of attack-defend interactions: one where the defender and attacker engage in a zero-sum game, and another where they engage in a non-zero-sum game. We demonstrate the effectiveness of our framework using a stochastic gridworld example.
In this work, we address the problem of synthesis of covert attackers in the setup where the model of the plant is available, but the model of the supervisor is unknown, to the adversary. To compensate the lack of knowledge on the supervisor, we assume that the adversary has recorded a (prefix-closed) finite set of observations of the runs of the closed-loop system, which can be used for assisting the synthesis. We present a heuristic algorithm for the synthesis of covert damage-reachable attackers, based on the model of the plant and the (finite) set of observations, by a transformation into solving an instance of the partial-observation supervisor synthesis problem. The heuristic algorithm developed in this paper may allow the adversary to synthesize covert attackers without having to know the model of the supervisor, which could be hard to obtain in practice. For simplicity, we shall only consider covert attackers that are able to carry out sensor replacement attacks and actuator disablement attacks. The effectiveness of our approach is illustrated on a water tank example adapted from the literature.
This paper presents a cooperative target defense strategy using nonlinear model-predictive control (NMPC) framework for a two--targets two--attackers (2T2A) game. The 2T2A game consists of two attackers and two targets. Each attacker needs to capture a designated target individually. However, the two targets cooperate to lure the attackers into a collision. We assume that the cooperative target pair do not have perfect knowledge of the attacker states, and hence they estimate the attacker states using an extended Kalman filter (EKF). The NMPC scheme computes closed- loop optimal control commands for the targets while respecting imposed state and control constraints. Theoretical analysis is carried out to determine regions that will lead to the targets' survival, given the initial positions of the attacker and target agents. Numerical simulations are carried out to evaluate the performance of the proposed NMPC- based strategy for different scenarios.
Distributed backdoor attacks (DBA) have shown a higher attack success rate than centralized attacks in centralized federated learning (FL). However, it has not been investigated in the decentralized FL. In this paper, we experimentally demonstrate that, while directly applying DBA to decentralized FL, the attack success rate depends on the distribution of attackers in the network architecture. Considering that the attackers can not decide their location, this paper aims to achieve a high attack success rate regardless of the attackers' location distribution. Specifically, we first design a method to detect the network by predicting the distance between any two attackers on the network. Then, based on the distance, we organize the attackers in different clusters. Lastly, we propose an algorithm to \textit{dynamically} embed local patterns decomposed from a global pattern into the different attackers in each cluster. We conduct a thorough empirical investigation and find that our method can, in benchmark datasets, outperform both centralized attacks and naive DBA in different decentralized frameworks.
We consider a variant of the target defense problem in a planar conical environment where a single defender is tasked to capture a sequence of incoming attackers. The attackers' objective is to breach the target boundary without being captured by the defender. As soon as the current attacker breaches the target or gets captured by the defender, the next attacker appears at the boundary of the environment and moves radially toward the target with maximum speed. Therefore, the defender's final location at the end of the current game becomes its initial location for the next game. The attackers pick strategies that are advantageous for the current as well as for future engagements between the defender and the remaining attackers. The attackers have their own sensors with limited range, using which they can perfectly detect if the defender is within their sensing range. We derive equilibrium strategies for all the players to optimize the capture percentage using the notions of capture distribution. Finally, the theoretical results are verified through numerical examples using Monte Carlo type random trials of experiments.
The technological advancements facilitating the rapid development of cyber-physical systems (CPS) also render such systems vulnerable to cyber attacks with devastating effects. Supervisory control is a commonly used control method to neutralize attacks on CPS. The supervisor strives to confine the (symbolic) paths of the system to a desired language via sensors and actuators in a closed control loop, even when attackers can manipulate the symbols received by the sensors and actuators. Currently, supervisory control methods face limitations when effectively identifying and mitigating unknown, broad-spectrum attackers. In order to capture the behavior of broad-spectrum attacks on both sensing and actuation channels we model the plant, supervisors, and attackers with finite-state transducers (FSTs). Our general method for addressing unknown attackers involves constructing FST models of the attackers from spectral analysis of their input and output symbol sequences recorded from a history of attack behaviors observed in a supervisory control loop. To construct these FST models, we devise a novel learning method based on the recorded history of attack behaviors. A supervisor is synthesi
Graph Neural Network-based methods face privacy leakage risks due to the introduction of topological structures about the targets, which allows attackers to bypass the target's prior knowledge of the sensitive attributes and realize membership inference attacks (MIA) by observing and analyzing the topology distribution. As privacy concerns grow, the assumption of MIA, which presumes that attackers can obtain an auxiliary dataset with the same distribution, is increasingly deviating from reality. In this paper, we categorize the distribution diversity issue in real-world MIA scenarios as an Out-Of-Distribution (OOD) problem, and propose a novel Graph OOD Membership Inference Attack (GOOD-MIA) to achieve cross-domain graph attacks. Specifically, we construct shadow subgraphs with distributions from different domains to model the diversity of real-world data. We then explore the stable node representations that remain unchanged under external influences and consider eliminating redundant information from confounding environments and extracting task-relevant key information to more clearly distinguish between the characteristics of training data and unseen data. This OOD-based design m
Communication networks are essential for our economy and our everyday lives. This makes them lucrative targets for attacks. Today, we see an ongoing battle between criminals that try to disrupt our key communication networks and security professionals that try to mitigate these attacks. However, today's networks, like the Internet or peer-to-peer networks among smart devices, are not controlled by a single authority, but instead consist of many independently administrated entities that are interconnected. Thus, both the decisions of how to interconnect and how to secure against potential attacks are taken in a decentralized way by selfish agents. This strategic setting, with agents that want to interconnect and potential attackers that want to disrupt the network, was captured via an influential game-theoretic model by Goyal, Jabbari, Kearns, Khanna, and Morgenstern (WINE 2016). We revisit this model and show improved tight bounds on the achieved robustness of networks created by selfish agents. As our main result, we show that such networks can resist attacks of a large class of potential attackers, i.e., these networks maintain asymptotically optimal welfare post attack. This imp
Modern AI-powered Integrated Development Environments (AI-IDEs) are increasingly defined by an Agent-centric architecture, where an LLM-powered Agent is deeply integrated to autonomously execute complex tasks. This tight integration, however, also introduces a new and critical attack surface. Attackers can exploit these components by injecting malicious instructions into untrusted external sources, effectively hijacking the Agent to perform harmful operations beyond the user's intention or awareness. This emerging threat has quickly attracted research attention, leading to various proposed attack vectors, such as hijacking Model Context Protocol (MCP) Servers to access private data. However, most existing approaches lack stealth and persistence, limiting their practical impact. We propose the Cuckoo Attack, a novel attack that achieves stealthy and persistent command execution by embedding malicious payloads into configuration files. These files, commonly used in AI-IDEs, execute system commands during routine operations, without displaying execution details to the user. Once configured, such files are rarely revisited unless an obvious runtime error occurs, creating a blind spot f
Numerous blockchain applications are designed with tasks that naturally have finite durations, and hence, a double-spending attack (DSA) on such blockchain applications leans towards being conducted within a finite timeframe, specifically before the completion of their tasks. Furthermore, existing research suggests that practical attackers typically favor executing a DSA within a finite timeframe due to their limited computational resources. These observations serve as the impetus for this paper to investigate a time-restricted DSA (TR-DSA) model on Proof-of-Work based blockchains. In this TR-DSA model, an attacker only mines its branch within a finite timeframe, and the TR-DSA is considered unsuccessful if the attacker's branch fails to surpass the honest miners' branch when the honest miners' branch has grown by a specific number of blocks. First, we developed a general closed-form expression for the success probability of a TR-DSA. This developed probability not only can assist in evaluating the risk of a DSA on blockchain applications with timely tasks, but also can enable practical attackers with limited computational resources to assess the feasibility and expected reward of
Generative models are subject to overfitting and thus may potentially leak sensitive information from the training data. In this work. we investigate the privacy risks that can potentially arise from the use of generative adversarial networks (GANs) for creating tabular synthetic datasets. For the purpose, we analyse the effects of re-identification attacks on synthetic data, i.e., attacks which aim at selecting samples that are predicted to correspond to memorised training samples based on their proximity to the nearest synthetic records. We thus consider multiple settings where different attackers might have different access levels or knowledge of the generative model and predictive, and assess which information is potentially most useful for launching more successful re-identification attacks. In doing so we also consider the situation for which re-identification attacks are formulated as reconstruction attacks, i.e., the situation where an attacker uses evolutionary multi-objective optimisation for perturbing synthetic samples closer to the training space. The results indicate that attackers can indeed pose major privacy risks by selecting synthetic samples that are likely repr
While image-to-text models have demonstrated significant advancements in various vision-language tasks, they remain susceptible to adversarial attacks. Existing white-box attacks on image-to-text models require access to the architecture, gradients, and parameters of the target model, resulting in low practicality. Although the recently proposed gray-box attacks have improved practicality, they suffer from semantic loss during the training process, which limits their targeted attack performance. To advance adversarial attacks of image-to-text models, this paper focuses on a challenging scenario: decision-based black-box targeted attacks where the attackers only have access to the final output text and aim to perform targeted attacks. Specifically, we formulate the decision-based black-box targeted attack as a large-scale optimization problem. To efficiently solve the optimization problem, a three-stage process \textit{Ask, Attend, Attack}, called \textit{AAA}, is proposed to coordinate with the solver. \textit{Ask} guides attackers to create target texts that satisfy the specific semantics. \textit{Attend} identifies the crucial regions of the image for attacking, thus reducing the
Decentralized Gradient Descent (D-GD) allows a set of users to perform collaborative learning without sharing their data by iteratively averaging local model updates with their neighbors in a network graph. The absence of direct communication between non-neighbor nodes might lead to the belief that users cannot infer precise information about the data of others. In this work, we demonstrate the opposite, by proposing the first attack against D-GD that enables a user (or set of users) to reconstruct the private data of other users outside their immediate neighborhood. Our approach is based on a reconstruction attack against the gossip averaging protocol, which we then extend to handle the additional challenges raised by D-GD. We validate the effectiveness of our attack on real graphs and datasets, showing that the number of users compromised by a single or a handful of attackers is often surprisingly large. We empirically investigate some of the factors that affect the performance of the attack, namely the graph topology, the number of attackers, and their position in the graph.