搜索结果：risks

In this study, we investigate system-level emergent risks of interacting AI agents. The core contribution of this work is an exploratory scenario-based identification of these risks as well as their categorization. We consider a multitude of systemic risk examples from existing literature and develop two scenarios demonstrating emergent risk patterns in domains of smart grid and social welfare. We provide a taxonomy of identified risks that categorizes them in different groups. In addition, we make two other important contributions: first, we identify what emergent behavior types produce systemic risks, and second, we develop a graphical language "Agentology" for visualization of interacting AI systems. Our study opens a new research direction for system-level risks of interacting AI, and is the first to closely investigate them.

Large language models (LLMs) are evolving into autonomous decision-makers, raising concerns about catastrophic risks in high-stakes scenarios, particularly in Chemical, Biological, Radiological and Nuclear (CBRN) domains. Based on the insight that such risks can originate from trade-offs between the agent's Helpful, Harmlessness and Honest (HHH) goals, we build a novel three-stage evaluation framework, which is carefully constructed to effectively and naturally expose such risks. We conduct 14,400 agentic simulations across 12 advanced LLMs, with extensive experiments and analysis. Results reveal that LLM agents can autonomously engage in catastrophic behaviors and deception, without being deliberately induced. Furthermore, stronger reasoning abilities often increase, rather than mitigate, these risks. We also show that these agents can violate instructions and superior commands. On the whole, we empirically prove the existence of catastrophic risks in autonomous LLM agents. We release our code to foster further research.

Emerging extended reality technologies are reshaping how children play, learn, and socialize. Yet, they also present serious safety risks. Gaming, a primary form of entertainment for children, is also one of the key applications of XR. While XR platforms offer immersive and engaging gaming experiences, recent news has highlighted safety concerns such as car accidents, lower judgment for real-world situations, and exposure to disturbing content like virtual rape. This research examines how XR game design may lead to online safety risks for children. Through analysis of player forums, game developer forums, and interviews with child players, we identify harmful XR design patterns, explore how developers collaboratively generate and implement risky game ideas, and document children's firsthand experiences of online safety risks. Existing ethical frameworks often fail to address the immersive and socially dynamic nature of XR games. We advocate for a child-centered, design-aware approach to ethical considerations in XR games, urging platforms and policymakers to prioritize children's developmental needs. Our work aims to help shape safer, more inclusive XR environments through research

Artificial intelligence (AI) is often presented as a key tool for addressing societal challenges, such as climate change. At the same time, AI's environmental footprint is expanding increasingly. This report describes the systemic environmental risks of artificial intelligence, in particular, moving beyond direct impacts such as energy and water usage. Systemic environmental risks of AI are emergent, cross-sector harms to climate, biodiversity, freshwater, and broader socioecological systems that arise primarily from AI's integration into social, economic, and physical infrastructures, rather than its direct resource use, and that propagate through feedbacks, yielding nonlinear, inequitable, and potentially irreversible impacts. While these risks are emergent and quantification is uncertain, this report aims to provide an overview of systemic environmental risks. Drawing on a narrative literature review, we propose a three-level framework that operationalizes systemic risk analysis. The framework identifies the structural conditions that shape AI development, the risk amplification mechanisms that propagate environmental harm, and the impacts that manifest as observable ecological

Accurate time-to-event prediction is integral to decision-making, informing medical guidelines, hiring decisions, and resource allocation. Survival analysis, the quantitative framework used to model time-to-event data, accounts for patients who do not experience the event of interest during the study period, known as censored patients. However, many patients experience events that prevent the observation of the outcome of interest. These competing risks are often treated as censoring, a practice frequently overlooked due to a limited understanding of its consequences. Our work theoretically demonstrates why treating competing risks as censoring introduces substantial bias in survival estimates, leading to systematic overestimation of risk and, critically, amplifying disparities. First, we formalize the problem of misclassifying competing risks as censoring and quantify the resulting error in survival estimates. Specifically, we develop a framework to estimate this error and demonstrate the associated implications for predictive performance and algorithmic fairness. Furthermore, we examine how differing risk profiles across demographic groups lead to group-specific errors, potential

The rapid advancement in building large language models (LLMs) has intensified competition among big-tech companies and AI startups. In this regard, model evaluations are critical for product and investment-related decision-making. While open evaluation sets like MMLU initially drove progress, concerns around data contamination and data bias have constantly questioned their reliability. As a result, it has led to the rise of private data curators who have begun conducting hidden evaluations with high-quality self-curated test prompts and their own expert annotators. In this paper, we argue that despite potential advantages in addressing contamination issues, private evaluations introduce inadvertent financial and evaluation risks. In particular, the key concerns include the potential conflict of interest arising from private data curators' business relationships with their clients (leading LLM firms). In addition, we highlight that the subjective preferences of private expert annotators will lead to inherent evaluation bias towards the models trained with the private curators' data. Overall, this paper lays the foundation for studying the risks of private evaluations that can lead

This paper addresses the issue of blockchain protocol risks, a foundational category of risks affecting Distributed Ledger Technology (DLT) which underpins digital assets, smart contracts, and decentralised applications. It presents a comprehensive risk management framework developed in collaboration with financial institutions, blockchain development teams and regulators that applies a traditional risk management taxonomy to address certain overlooked blockchain protocol risks. The approach offers a structured way to identify, measure, monitor and report blockchain protocol risks. The paper provides real-world use cases to demonstrate the practicality and implementation of the proposed framework. The findings of this work contribute to the evolving understanding of blockchain protocol risks and provide valuable insights on how these risks affect the adoption of DLT by financial institutions.

The speed and scale at which machine learning (ML) systems are deployed are accelerating even as an increasing number of studies highlight their potential for negative impact. There is a clear need for companies and regulators to manage the risk from proposed ML systems before they harm people. To achieve this, private and public sector actors first need to identify the risks posed by a proposed ML system. A system's overall risk is influenced by its direct and indirect effects. However, existing frameworks for ML risk/impact assessment often address an abstract notion of risk or do not concretize this dependence. We propose to address this gap with a context-sensitive framework for identifying ML system risks comprising two components: a taxonomy of the first- and second-order risks posed by ML systems, and their contributing factors. First-order risks stem from aspects of the ML system, while second-order risks stem from the consequences of first-order risks. These consequences are system failures that result from design and development choices. We explore how different risks may manifest in various types of ML systems, the factors that affect each risk, and how first-order risks

In this paper, we will show that under certain conditions, associated to any fixed distortion function $g$, the distortion risk measure of a sum of two counter-monotonic risks can be expressed as the sum of two related distortion risk measures of the marginals involved, one associated to the original distortion function $g$ and the other associated to the dual distortion function of $g$. This result extends some of the work in \cite{Chaoubi et al. (2020)} and \cite{HLD} since the class of distortion risk measures includes the risk measure of VaR and TVaR as special cases.

Post-traumatic stress disorder (PTSD) is associated with sudden, uncontrollable, and intense flashbacks of traumatic memories. Trauma exposure psychotherapy has proven effective in reducing the severity of trauma-related symptoms. It involves controlled recall of traumatic memories to train coping mechanisms for flashbacks and enable autobiographical integration of distressing experiences. In particular, exposure to visualizations of these memories supports successful recall. Although this approach is effective for various trauma types, it remains available for only a few. This is due to the lack of cost-efficient solutions for creating individualized exposure visualizations. This issue is particularly relevant for the treatment of Complex PTSD (CPTSD), where traumatic memories are highly individual and generic visualizations do not meet therapeutic needs. Generative Artificial Intelligence (GAI) offers a flexible and cost-effective alternative. GAI enables the creation of individualized exposure visualizations during therapy and, for the first time, allows patients to actively participate in the visualization process. While GAI opens new therapeutic perspectives and may improve ac

We study a general risk measure called the generalized shortfall risk measure, which was first introduced in Mao and Cai (2018). It is proposed under the rank-dependent expected utility framework, or equivalently induced from the cumulative prospect theory. This risk measure can be flexibly designed to capture the decision maker's behavior toward risks and wealth when measuring risk. In this paper, we derive the first- and second-order asymptotic expansions for the generalized shortfall risk measure. Our asymptotic results can be viewed as unifying theory for, among others, distortion risk measures and utility-based shortfall risk measures. They also provide a blueprint for the estimation of these measures at extreme levels, and we illustrate this principle by constructing and studying a quantile-based estimator in a special case. The accuracy of the asymptotic expansions and of the estimator is assessed on several numerical examples.

Large language models (LLMs) have become increasingly sophisticated, leading to widespread deployment in sensitive applications where safety and reliability are paramount. However, LLMs have inherent risks accompanying them, including bias, potential for unsafe actions, dataset poisoning, lack of explainability, hallucinations, and non-reproducibility. These risks necessitate the development of "guardrails" to align LLMs with desired behaviors and mitigate potential harm. This work explores the risks associated with deploying LLMs and evaluates current approaches to implementing guardrails and model alignment techniques. We examine intrinsic and extrinsic bias evaluation methods and discuss the importance of fairness metrics for responsible AI development. The safety and reliability of agentic LLMs (those capable of real-world actions) are explored, emphasizing the need for testability, fail-safes, and situational awareness. Technical strategies for securing LLMs are presented, including a layered protection model operating at external, secondary, and internal levels. System prompts, Retrieval-Augmented Generation (RAG) architectures, and techniques to minimize bias and protect pri

A typical situation in competing risks analysis is that the researcher is only interested in a subset of risks. This paper considers a depending competing risks model with the distribution of one risk being a parametric or semi-parametric model, while the model for the other risks being unknown. Identifiability is shown for popular classes of parametric models and the semiparametric proportional hazards model. The identifiability of the parametric models does not require a covariate, while the semiparametric model requires at least one. Estimation approaches are suggested which are shown to be $\sqrt{n}$-consistent. Applicability and attractive finite sample performance are demonstrated with the help of simulations and data examples.

In this paper, we propose a novel axiomatic approach to evaluating the joint risk of multiple insurance risks under dependence uncertainty. Motivated by both the theory of expected utility and the Cobb-Dauglas utility function, we establish a joint risk measure for non-negative multivariate risks, which we refer to as a scalar distortion joint risk measure. After having studied its fundamental properties, we provide an axiomatic characterization of it by proposing a set of new axioms. The most novel axiom is the component-wise positive homogeneity. Then, based on the resulting distortion joint risk measures, we also propose a new class of vector-valued distortion joint risk measures for non-negative multivariate risks. Finally, we make comparisons with some vector-valued multivariate risk measures known in the literature, such as multivariate lower-orthant value at risk, multivariate upper-orthant conditional-tail-expectation, multivariate tail conditional expectation and multivariate tail distortion risk measures. It turns out that those vector-valued multivariate risk measures have forms of vector-valued distortion joint risk measures, respectively. This paper mainly gives some t

In this paper, we investigate risk measures such as value at risk (VaR) and the conditional tail expectation (CTE) of the extreme (maximum and minimum) and the aggregate (total) of two dependent risks. In finance, insurance and the other fields, when people invest their money in two or more dependent or independent markets, it is very important to know the extreme and total risk before the investment. To find these risk measures for dependent cases is quite challenging, which has not been reported in the literature to the best of our knowledge. We use the FGM copula for modelling the dependence as it is relatively simple for computational purposes and has empirical successes. The marginal of the risks are considered as exponential and pareto, separately, for the case of extreme risk and as exponential for the case of the total risk. The effect of the degree of dependency on the VaR and CTE of the extreme and total risks is analyzed. We also make comparisons for the dependent and independent risks. Moreover, we propose a new risk measure called median of tail (MoT) and investigate MoT for the extreme and aggregate dependent risks.

Value at risk (VaR) and expected shortfall (ES) are common high quantile-based risk measures adopted in financial regulations and risk management. In this paper, we propose a tail risk measure based on the most probable maximum size of risk events (MPMR) that can occur over a length of time. MPMR underscores the dependence of the tail risk on the risk management time frame. Unlike VaR and ES, MPMR does not require specifying a confidence level. We derive the risk measure analytically for several well-known distributions. In particular, for the case where the size of the risk event follows a power law or Pareto distribution, we show that MPMR also scales with the number of observations $n$ (or equivalently the length of the time interval) by a power law, $\text{MPMR}(n) \propto n^η$, where $η$ is the scaling exponent. The scale invariance allows for reasonable estimations of long-term risks based on the extrapolation of more reliable estimations of short-term risks. The scaling relationship also gives rise to a robust and low-bias estimator of the tail index (TI) $ξ$ of the size distribution, $ξ= 1/η$. We demonstrate the use of this risk measure for describing the tail risks in fina

Air transportation has been becoming a major part of transportation infrastructure worldwide. Hence the study of the Airports Networks, the backbone of air transportation, is becoming increasingly important. In complex systems domain, airport networks are modeled as graphs (networks) comprising of airports (vertices or nodes) that are linked by flight connectivities among the airports. A complex network analysis of such a model offers holistic insight about the performance and risks in such a network. We review the performance and risks of networks with the help of studies that have been done on some of the airport networks. We present various network parameters those could be potentially used as a measure of performance and risks on airport networks. We will also see how various risks, such as break down of airports, spread of diseases across the airport network could be assessed based on the network parameters. Further we review how these insights could possibly be used to shape more efficient and safer airport networks.