搜索结果：Delays

In standard reinforcement learning (RL) settings, the interaction between the agent and the environment is typically modeled as a Markov decision process (MDP), which assumes that the agent observes the system state instantaneously, selects an action without delay, and executes it immediately. In real-world dynamic environments, such as cyber-physical systems, this assumption often breaks down due to delays in the interaction between the agent and the system. These delays can vary stochastically over time and are typically unobservable when deciding on an action. Existing methods deal with this uncertainty conservatively by assuming a known fixed upper bound on the delay, even if the delay is often much lower. In this work, we introduce the interaction layer, a general framework that enables agents to adaptively handle unobservable and time-varying delays. Specifically, the agent generates a matrix of possible future actions, anticipating a horizon of potential delays, to handle both unpredictable delays and lost action packets sent over networks. Building on this framework, we develop a model-based algorithm, Actor-Critic with Delay Adaptation (ACDA), which dynamically adjusts to

This paper develops methods for numerically solving stochastic delay-differential equations (SDDEs) with multiple fixed delays that do not align with a uniform time mesh. We focus on numerical schemes of strong convergence orders $1/2$ and $1$, such as the Euler--Maruyama and Milstein schemes, respectively. Although numerical schemes for SDDEs with delays $τ_1,\ldots,τ_K$ are theoretically established, their implementations require evaluations at both present times such as $t_n$, and also at delayed times such as $t_n-τ_k$ and $t_n-τ_l-τ_k$. As a result, previous simulations of these schemes have been largely restricted to the case of divisible delays. We develop simulation techniques for the general case of indivisible delays where delayed times such as $t_n-τ_k$ are not restricted to a uniform time mesh. To achieve order of convergence (OoC) $1/2$, we implement the schemes with a fixed step size while using linear interpolation to approximate delayed scheme values. To achieve OoC $1$, we construct an augmented time mesh that includes all time points required to evaluate the schemes, which necessitates using a varying step size. We also introduce a technique to simulate delayed it

This paper studies active automata learning (AAL) in the presence of stochastic delays. We consider Mealy machines that have stochastic delays associated with each transition and explore how the learner can efficiently arrive at faithful estimates of those machines, the precision of which crucially relies on repetitive sampling of transition delays. While it is possible to naïvely integrate the delay sampling into AAL algorithms such as $L^*$, this leads to considerable oversampling near the root of the state space. We address this problem by separating conceptually the learning of behavior and delays such that the learner uses the information gained while learning the logical behavior to arrive at efficient input sequences for collecting the needed delay samples. We put emphasis on treating cases in which identical input/output behaviors might stem from distinct delay characteristics. Finally, we provide empirical evidence that our method outperforms the naïve baseline across a wide range of benchmarks and investigate its applicability in a realistic setting by studying the join order in a relational database.

Reinforcement learning (RL) is challenging in the common case of delays between events and their sensory perceptions. State-of-the-art (SOTA) state augmentation techniques either suffer from state space explosion or performance degeneration in stochastic environments. To address these challenges, we present a novel Auxiliary-Delayed Reinforcement Learning (AD-RL) method that leverages auxiliary tasks involving short delays to accelerate RL with long delays, without compromising performance in stochastic environments. Specifically, AD-RL learns a value function for short delays and uses bootstrapping and policy improvement techniques to adjust it for long delays. We theoretically show that this can greatly reduce the sample complexity. On deterministic and stochastic benchmarks, our method significantly outperforms the SOTAs in both sample efficiency and policy performance. Code is available at https://github.com/QingyuanWuNothing/AD-RL.

In discrete-event systems, to save sensor resources, the agent continuously adjusts sensor activation decisions according to a sensor activation policy based on the changing observations. However, new challenges arise for sensor activations in networked discrete-event systems, where observation delays and control delays exist between the sensor systems and the agent. In this paper, a new framework for activating sensors in networked discrete-event systems is established. In this framework, we construct a communication automaton that explicitly expresses the interaction process between the agent and the sensor systems over the observation channel and the control channel. Based on the communication automaton, we can define dynamic observations of a communicated string. To guarantee that a sensor activation policy is physically implementable and insensitive to non-deterministic control delays and observation delays, we further introduce the definition of delay feasibility. We show that a delay feasible sensor activation policy can be used to dynamically activate sensors even if control delays and observation delays exist. A set of algorithms are developed to minimize sensor activation

Since 2018, there has been a consistent decline in the distance traveled by U.S. manufacturing imports, reaching a level not observed since 2008. This trend is the result of the substitution away from imports from China and towards imports from closer countries. At the same time, U.S. manufacturing inventory-to-sales ratio has continued to rise. These trends are at odds with the literature, which finds that reductions in the distance of imports are associated with a decline in inventories. We argue that a rise in delivery time risk, driven by longer and more frequent delays and supply disruptions, can reconcile these trends. We do so in the context of a model of global sourcing with stochastic delivery times and inventories. Firms trade off the lower price of farther inputs with the increase in exposure to demand volatility and longer delays. In response, firms increase their inventories. Yet, as delivery delays rise, firms need to carry more inventories per unit of the input used. We calibrate the model for the period from 2018 to 2024 using data on the increase in tariffs for inputs from China, and the rise in inventories over sales. We find an increase in delivery delays for for

In the context of the Laser Interferometer Space Antenna (LISA), the laser subsystems exhibit frequency fluctuations that introduce significant levels of noise into the measurements, surpassing the gravitational wave signal by several orders of magnitude. Mitigation is achieved via time-shifting individual measurements in a data processing step known as time-delay interferometry (TDI). The suppression performance of TDI relies on accurate knowledge and consideration of the delays experienced by the interfering lasers. While considerable efforts have been dedicated to the accurate determination of inter-spacecraft ranging delays, the sources for onboard delays have been either neglected or assumed to be known. Contrary to these assumptions, analog delays of the phasemeter front end and the laser modulator are not only large but also prone to change with temperature and heterodyne frequency. This motivates our proposal for a novel method enabling a calibration of these delays on-ground and in-space, based on minimal functional additions to the receiver architecture. Specifically, we establish a set of calibration measurements and elucidate how these measurements are utilized in data

Time-delay interferometry (TDI) is a data processing technique for space-based gravitational-wave detectors to create laser-noise-free equal-optical-path-length interferometers virtually on the ground. It relies on the interspacecraft signal propagation delays, which are delivered by intersatellite ranging monitors. Also delays due to onboard signal propagation and processing have a nonnegligible impact on the TDI combinations. However, these onboard delays were only partially considered in previous TDI-related research; onboard optical path lengths have been neglected so far. In this paper, we study onboard optical path lengths in TDI. We derive analytical models for their coupling to the second-generation TDI Michelson combinations and verify these models numerically. Furthermore, we derive a compensation scheme for onboard optical path lengths in TDI and validate its performance via numerical simulations.

We propose a new best-of-both-worlds algorithm for bandits with variably delayed feedback. In contrast to prior work, which required prior knowledge of the maximal delay $d_{\mathrm{max}}$ and had a linear dependence of the regret on it, our algorithm can tolerate arbitrary excessive delays up to order $T$ (where $T$ is the time horizon). The algorithm is based on three technical innovations, which may all be of independent interest: (1) We introduce the first implicit exploration scheme that works in best-of-both-worlds setting. (2) We introduce the first control of distribution drift that does not rely on boundedness of delays. The control is based on the implicit exploration scheme and adaptive skipping of observations with excessive delays. (3) We introduce a procedure relating standard regret with drifted regret that does not rely on boundedness of delays. At the conceptual level, we demonstrate that complexity of best-of-both-worlds bandits with delayed feedback is characterized by the amount of information missing at the time of decision making (measured by the number of outstanding observations) rather than the time that the information is missing (measured by the delays).

Delayed processes are ubiquitous throughout biology. These delays may arise through maturation processes or as the result of complex multi-step networks, and mathematical models with distributed delays are increasingly used to capture the heterogeneity present in these delayed processes. Typically, these distributed delay differential equations are simulated by discretizing the distributed delay and using existing tools for the resulting multi-delay delay differential equations or by using an equivalent representation under additional assumptions on the delayed process. Here, we use the existing framework of functional continuous Runge-Kutta methods to confirm the convergence of this common approach. Our analysis formalizes the intuition that the least accurate numerical method dominates the error. We give a number of examples to illustrate the predicted convergence, derive a new class of equivalences between distributed delay and discrete delay differential equations, and give conditions for the existence of breaking points in the distributed delay differential equation. Finally, our work shows how recently reported multi-delay complexity collapse arises naturally from the converg

Here we investigate the synchronization of networks of FitzHugh-Nagumo neurons coupled in scale-free, small-world and random topologies, in the presence of distributed time delays in the coupling of neurons. We explore how the synchronization transition is affected when the time delays in the interactions between pairs of interacting neurons are non-uniform. We find that the presence of distributed time-delays does not change the behavior of the synchronization transition significantly, vis-a-vis networks with constant time-delay, where the value of the constant time-delay is the mean of the distributed delays. We also notice that a normal distribution of delays gives rise to a transition at marginally lower coupling strengths, vis-a-vis uniformly distributed delays. These trends hold across classes of networks and for varying standard deviations of the delay distribution, indicating the generality of these results. So we conclude that distributed delays, which may be typically expected in real-world situations, do not have a notable effect on synchronization. This allows results obtained with constant delays to remain relevant even in the case of randomly distributed delays.

The Sparse Identification of Nonlinear Dynamics (SINDy) framework is a robust method for identifying governing equations, successfully applied to ordinary, partial, and stochastic differential equations. In this work we extend SINDy to identify delay differential equations by using an augmented library that includes delayed samples and Bayesian optimization. To identify a possibly unknown delay we minimize the reconstruction error over a set of candidates. The resulting methodology improves the overall performance by remarkably reducing the number of calls to SINDy with respect to a brute force approach. We also address a multivariate setting to identify multiple unknown delays and (non-multiplicative) parameters. Several numerical tests on delay differential equations with different long-term behavior, number of variables, delays, and parameters support the use of Bayesian optimization highlighting both the efficacy of the proposed methodology and its computational advantages. As a consequence, the class of discoverable models is significantly expanded.

Classic reinforcement learning (RL) frequently confronts challenges in tasks involving delays, which cause a mismatch between received observations and subsequent actions, thereby deviating from the Markov assumption. Existing methods usually tackle this issue with end-to-end solutions using state augmentation. However, these black-box approaches often involve incomprehensible processes and redundant information in the information states, causing instability and potentially undermining the overall performance. To alleviate the delay challenges in RL, we propose $\textbf{DEER (Delay-resilient Encoder-Enhanced RL)}$, a framework designed to effectively enhance the interpretability and address the random delay issues. DEER employs a pretrained encoder to map delayed states, along with their variable-length past action sequences resulting from different delays, into hidden states, which is trained on delay-free environment datasets. In a variety of delayed scenarios, the trained encoder can seamlessly integrate with standard RL algorithms without requiring additional modifications and enhance the delay-solving capability by simply adapting the input dimension of the original algorithms

Delays are inherent to most dynamical systems. Besides shifting the process in time, they can significantly affect their performance. For this reason, it is usually valuable to study the delay and account for it. Because they are dynamical systems, it is of no surprise that sequential decision-making problems such as Markov decision processes (MDP) can also be affected by delays. These processes are the foundational framework of reinforcement learning (RL), a paradigm whose goal is to create artificial agents capable of learning to maximise their utility by interacting with their environment. RL has achieved strong, sometimes astonishing, empirical results, but delays are seldom explicitly accounted for. The understanding of the impact of delay on the MDP is limited. In this dissertation, we propose to study the delay in the agent's observation of the state of the environment or in the execution of the agent's actions. We will repeatedly change our point of view on the problem to reveal some of its structure and peculiarities. A wide spectrum of delays will be considered, and potential solutions will be presented. This dissertation also aims to draw links between celebrated framewo

We consider the effect of distributed delays in neural feedback systems. The avian optic tectum is reciprocally connected with the nucleus isthmi. Extracellular stimulation combined with intracellular recordings reveal a range of signal delays from 4 to 9 ms between isthmotectal elements. This observation together with prior mathematical analysis concerning the influence of a delay distribution on system dynamics raises the question whether a broad delay distribution can impact the dynamics of neural feedback loops. For a system of reciprocally connected model neurons, we found that distributed delays enhance system stability in the following sense. With increased distribution of delays, the system converges faster to a fixed point and converges slower toward a limit cycle. Further, the introduction of distributed delays leads to an increased range of the average delay value for which the system's equilibrium point is stable. The enhancement of stability with increasing delay distribution is caused by the introduction of smaller delays rather than the distribution per se.

Delayed interactions are a common property of coupled natural systems and therefore arise in a variety of different applications. For instance, signals in neural or laser networks propagate at finite speed giving rise to delayed connections. Such systems are often modeled by delay differential equations with discrete delays. In realistic situations, these delays are not identical on different connections. We show that by a componentwise timeshift transformation it is often possible to reduce the number of different delays and simplify the models without loss of information. We identify dynamic invariants of this transformation, determine its capabilities to reduce the number of delays and interpret these findings in terms of the topology of the underlying graph. In particular, we show that networks with identical sums of delay times along the fundamental semicycles are dynamically equivalent and we provide a normal form for these systems. We illustrate the theory using a network motif of coupled Mackey-Glass systems with 8 different time delays, which can be reduced to an equivalent motif with three delays.

We report measurements of energy-dependent attosecond photoionization delays between the two outer-most valence shells of N$_2$O and H$_2$O. The combination of single-shot signal referencing with the use of different metal foils to filter the attosecond pulse train enables us to extract delays from congested spectra. Remarkably large delays up to 160 as are observed in N$_2$O, whereas the delays in H$_2$O are all smaller than 50 as in the photon-energy range of 20-40 eV. These results are interpreted by developing a theory of molecular photoionization delays. The long delays measured in N$_2$O are shown to reflect the population of molecular shape resonances that trap the photoelectron for a duration of up to $\sim$110 as. The unstructured continua of H$_2$O result in much smaller delays at the same photon energies. Our experimental and theoretical methods make the study of molecular attosecond photoionization dynamics accessible.