Deploying Large Language Models (LLMs) requires exploring a large configuration space spanning parallelization strategies, batching techniques, and scheduling policies. Exhaustive measurement across this space is impractical, making latency prediction essential for system optimization. While NPUs have emerged as accelerators designed for LLM inference, no prediction methodology has been established for them. Specifically, applying prior work to LLM inference latency prediction on NPUs faces three challenges: undisclosed microarchitecture of commercial NPUs, unpredictable compiler optimizations, and latency non-linearity induced by bucketing. We present LENS, a latency estimator that predicts NPU inference latency without information on the microarchitecture or compiler, and captures the non-linear latency induced by bucketing. LENS profiles each bucket with two end-to-end (E2E) measurements and composes the results to predict latency for arbitrary input-output length combinations. We validate LENS across NPUs from multiple vendors, several LLMs, and diverse workloads, achieving a mean prediction error of 2.15\%. We further compare LENS against two methodologically related baselines
In the modelling of social systems, opinion latency is the idea that once an agent changes its opinion, there will be a period of time where it is immune to other changes. When added to the voter model this leads to a situation where no matter how low the latency is or how many opinions are considered, all opinions end up in a coexistence where they are equally represented. In this work, we examine what happens when latency is added to the Sznajd model. What we find is that for low latency, the model behaves roughly like it does in the absence of latency, where one opinion will always eventually dominate. For high latency, the possibility for a symmetric coexistence of 2 opinions arises, but contrary to the voter model, a coexistence of more than 2 opinions is never stable. We provide evidence of this phenomenon with computer simulations of the model in Barabási-Albert networks, together with a mean field treatment that is able to capture the observed behavior. We argue that this could hint at an explanation for the prevalence of two opinion splits in the real world.
End-to-end (e2e) latency in head-mounted displays (HMD) is the time delay between a physical change in the world (e.g., a user's head movement) and the moment the display updates to reflect that change. Tracking, rendering, and other computation in real systems invariably introduce some amount of e2e latency to all HMDs. In modern devices this latency is usually in the range of 12-60 milliseconds which is partially addressed through pose prediction and late stage reprojection which means that perceptual studies and user experience evaluations cannot explore latencies below these values. Here, we introduce a video passthrough HMD, called Camsicle, which is capable of 2-millisecond e2e latency and, additionally, uses a catadioptric design to achieve perspective-correct passthrough without reprojection. This platform enables naturalistic user studies to interrogate the impacts of latency on user experience, preference, and performance. Across two user studies and 57 participants we find that 2 and 14.3 millisecond latencies are preferred over 23 and 29 milliseconds when attempting to catch a ball. Additionally, we compare individual latency preferences in this naturalistic ball-catchi
End-to-end blockchain latency has become a critical topic of interest in both academia and industry. However, while modern blockchain systems process transactions through multiple stages, most research has primarily focused on optimizing the latency of the Byzantine Fault Tolerance consensus component. In this work, we identify key sources of latency in blockchain systems and introduce Zaptos, a parallel pipelined architecture designed to minimize end-to-end latency while maintaining the high-throughput of pipelined blockchains. We implemented Zaptos and evaluated it against the pipelined architecture of the Aptos blockchain in a geo-distributed environment. Our evaluation demonstrates a 25\% latency reduction under low load and over 40\% reduction under high load. Notably, Zaptos achieves a throughput of 20,000 transactions per second with sub-second latency, surpassing previously reported blockchain throughput, with sub-second latency, by an order of magnitude.
Hybrid cloud-edge infrastructures now support latency-critical workloads ranging from autonomous vehicles and surgical robotics to immersive AR/VR. However, they continue to experience crippling long-tail latency spikes whenever bursty request streams exceed the capacity of heterogeneous edge and cloud tiers. To address these long-tail latency issues, we present Latency-Aware, Predictive In-Memory Routing and Proactive Autoscaling (LA-IMR). This control layer integrates a closed-form, utilization-driven latency model with event-driven scheduling, replica autoscaling, and edge-to-cloud offloading to mitigate 99th-percentile (P99) delays. Our analytic model decomposes end-to-end latency into processing, network, and queuing components, expressing inference latency as an affine power-law function of instance utilization. Once calibrated, it produces two complementary functions that drive: (i) millisecond-scale routing decisions for traffic offloading, and (ii) capacity planning that jointly determines replica pool sizes. LA-IMR enacts these decisions through a quality-differentiated, multi-queue scheduler and a custom-metric Kubernetes autoscaler that scales replicas proactively -- be
A recent advance in networking is the deployment of path-aware multipath network architectures, where network endpoints are given multiple network paths to send their data on. In this work, we tackle the challenge of selecting paths for latency-sensitive applications. Even today's path-aware networks, which are much smaller than the current Internet, already offer dozens and in several cases over a hundred paths to a given destination, making it impractical to measure all path latencies to find the lowest latency path. Furthermore, for short flows, performing latency measurements may not provide benefits as the flow may finish before completing the measurements. To overcome these issues, we argue that endpoints should be provided with a latency estimate before sending any packets, enabling latency-aware path choice for the first packet sent. As we cannot predict the end-to-end latency due to dynamically changing queuing delays, we measure and disseminate the propagation latency, enabling novel use cases and solving concrete problems in current network protocols. We present the Global Latency Information Dissemination System (GLIDS), which is a step toward global latency transparenc
Spiking neural networks (SNNs) offer a biologically inspired computing paradigm with significant potential for energy-efficient neural processing. Among neural coding schemes of SNNs, Time-To-First-Spike (TTFS) coding, which encodes information through the precise timing of a neuron's first spike, provides exceptional energy efficiency and biological plausibility. Despite its theoretical advantages, existing TTFS models lack efficient training methods, suffering from high inference latency and limited performance. In this work, we present a comprehensive framework, which enables the efficient training of deep TTFS-coded SNNs by employing backpropagation throuh time (BPTT) algorithm. We name the generalized TTFS coding method in our framework as latency coding. The framework includes: (1) a latency encoding (LE) module with feature extraction and straight-through estimators to address severe information loss in direct intensity-to-latency mapping and ensure smooth gradient flow; (2) relaxation of the strict single-spike constraint of traditional TTFS, allowing neurons of intermediate layers to fire multiple times to mitigating gradient vanishing in deep networks; (3) a temporal adap
Low Latency, Low Loss, and Scalable Throughput (L4S), as an emerging router-queue management technique, has seen steady deployment in the industry. An L4S-enabled router assigns each packet to the queue based on the packet header marking. Currently, L4S employs per-flow queue selection, i.e. all packets of a flow are marked the same way and thus use the same queues, even though each packet is marked separately. However, this may hurt tail latency and latency-sensitive applications because transient congestion and queue buildups may only affect a fraction of packets in a flow. We present SwiftQueue, a new L4S queue-selection strategy in which a sender uses a novel per-packet latency predictor to pinpoint which packets likely have latency spikes or drops. The insight is that many packet-level latency variations result from complex interactions among recent packets at shared router queues. Yet, these intricate packet-level latency patterns are hard to learn efficiently by traditional models. Instead, SwiftQueue uses a custom Transformer, which is well-studied for its expressiveness on sequential patterns, to predict the next packet's latency based on the latencies of recently received
Low-latency communication plays an increasingly important role in delay-sensitive applications by ensuring the real-time information exchange. However, due to the constraint on the maximum instantaneous power, guaranteeing bounded latency is challenging. In this paper, we investigate the reliability-latency-rate tradeoff in low-latency communication systems with finite-blocklength coding (FBC). Specifically, we are interested in the fundamental tradeoff between error probability, delay-violation probability (DVP), and service rate. Based on the effective capacity (EC), we present the gain-conservation equations to characterize the reliability-latency-rate tradeoffs in low-latency communication systems. In particular, we investigate the low-latency transmissions over an additive white Gaussian noise (AWGN) channel and a Nakagami-$m$ fading channel. By defining the service rate gain, reliability gain, and real-time gain, we conduct an asymptotic analysis to reveal the fundamental reliability-latency-rate tradeoff of ultra-reliable and low-latency communications in the high signal-to-noise-ratio (SNR) regime. To analytically evaluate and optimize the quality-of-service-constrained thr
This paper introduces a novel, fast atomic-snapshot protocol for asynchronous message-passing systems. In the process of defining what ``fast'' means exactly, we spot a few interesting issues that arise when conventional time metrics are applied to long-lived asynchronous algorithms. We reveal some gaps in latency claims made in earlier work on snapshot algorithms, which hamper their comparative time-complexity analysis. We then come up with a new unifying time-complexity metric that captures the latency of an operation in an asynchronous, long-lived implementation. This allows us to formally grasp latency improvements of our atomic-snapshot algorithm with respect to the state-of-the-art protocols: optimal latency in fault-free runs without contention, short constant latency in fault-free runs with contention, the worst-case latency proportional to the number of active concurrent failures, and constant, amortized latency.
Latency Based Tiling provides a systems based approach to deriving approximate tiling solution that maximizes locality while maintaining a fast compile time. The method uses triangular loops to characterize miss ratio scaling of a machine avoiding prefetcher distortion. Miss ratio scaling captures the relationship between data access latency and working set size with sharp increases in latency indicating the data footprint exceeds capacity from a cache level. Through these noticeable increases in latency we can determine an approximate location for L1, L2, and L3 memory sizes. These sizes are expected to be under approximations of a systems true memory sizes which is in line with our expectations given the shared nature of cache in a multi process system as described in defensive loop tiling. Unlike auto tuning, which can be effective but prohibitively slow, Latency Based Tiling achieves negligible compile time overhead. The implementation in Rust enables a hardware agnostic approach which combined with a cache timing based techniques, yields a portable, memory safe system running wherever Rust is supported. The tiling strategy is applied to a subset of the polyhedral model, where
As digital media consumption shifts toward large-scale Over-the-Top (OTT) platforms, the efficiency of the control plane, specifically entitlement and identity verification, has become a critical factor in user experience. Current architectures often rely on synchronous cloud-tethered validation flows that introduce significant latency, especially on resource-constrained consumer electronics. This paper proposes a Hybrid Edge-Cloud Entitlement Framework designed to minimize user-perceived friction. By implementing a secure, local caching layer within device middleware and utilizing an Adaptive Entitlement Cache with Proactive Refresh (AEC-PR) algorithm, we decouple the user interaction from backend network variability. We evaluate the performance on ARM Cortex-A series hardware, demonstrating that localized cryptographic verification reduces authorization latency from a mean of 422.8ms to 18.4ms (a 95.6% reduction) while mitigating implementation-level side-channel risks through deterministic Ed25519 arithmetic and TEE isolation.
Since the advent of ultra-reliable and low-latency communications (URLLC), the requirements of low-latency applications tend to be completely characterized by a single pre-defined latency-reliability target. That is, operation is optimal whenever the pre-defined latency threshold is met but the system is assumed to be in error when the latency threshold is violated. This vision is severely limited and does not capture the real requirements of most applications, where multiple latency thresholds can be defined, together with incentives or rewards associated with meeting each of them. Such formulation is a generalization of the single-threshold case popularized by URLLC and, in the asymptotic case, approximates to defining a cost for each point in the support of the latency distribution. In this paper, we explore the implications of defining multiple latency targets on the design of access protocols and on the optimization of repetition-based access strategies in orthogonal and non-orthogonal multiple access scenarios with users that present heterogeneous traffic characteristics and requirements. We observe that the access strategies of the users can be effectively adapted to the req
Low latency and high data rate performance are essential in wireless communication systems. This paper explores trade-offs between latency and data rates for optical wireless communication. We introduce a latency-optimized model utilizing compound codes as one corner case and a data rate-optimized model employing channel estimation via pilot signals and feedback before data transmission. Trade-offs between the two extremes are displayed. Most importantly, we detail operating points that can only be reached when the receiver side of the link employs optimal quantum measurement strategies. Furthermore, we propose an IoT application in a robot factory as an example scenario. Our findings reveal a trade-off between latency and data rate driven by two basic algorithms: compound codes reduce latency at the cost of data rates, while channel estimation enhances data rates at the cost of latency.
Efficient deployment of neural networks (NN) requires the co-optimization of accuracy and latency. For example, hardware-aware neural architecture search has been used to automatically find NN architectures that satisfy a latency constraint on a specific hardware device. Central to these search algorithms is a prediction model that is designed to provide a hardware latency estimate for a candidate NN architecture. Recent research has shown that the sample efficiency of these predictive models can be greatly improved through pre-training on some \textit{training} devices with many samples, and then transferring the predictor on the \textit{test} (target) device. Transfer learning and meta-learning methods have been used for this, but often exhibit significant performance variability. Additionally, the evaluation of existing latency predictors has been largely done on hand-crafted training/test device sets, making it difficult to ascertain design features that compose a robust and general latency predictor. To address these issues, we introduce a comprehensive suite of latency prediction tasks obtained in a principled way through automated partitioning of hardware device sets. We the
Objective: This study aims to understand the cognitive impact of latency in teleoperation and the related mitigation methods, using functional Near-Infrared Spectroscopy (fNIRS) to analyze functional connectivity. Background: Latency between command, execution, and feedback in teleoperation can impair performance and affect operators mental state. The neural underpinnings of these effects are not well understood. Method: A human subject experiment (n = 41) of a simulated remote robot manipulation task was performed. Three conditions were tested: no latency, with visual and haptic latency, with visual latency and no haptic latency. fNIRS and performance data were recorded and analyzed. Results: The presence of latency in teleoperation significantly increased functional connectivity within and between prefrontal and motor cortexes. Maintaining visual latency while providing real-time haptic feedback reduced the average functional connectivity in all cortical networks and showed a significantly different connectivity ratio within prefrontal and motor cortical networks. The performance results showed the worst performance in the all-delayed condition and best performance in no latency
Future vehicles are expected to dynamically deploy in-vehicle applications within a Service-Oriented Architecture (SOA) while critical services continue to operate under hard real-time constraints. Time-Sensitive Networking (TSN) on the in-vehicle Ethernet layer is dedicated to ensure deterministic communication between critical services; its Credit-Based Shaper (CBS) supports dynamic resource reservations. However, the dynamic nature of service deployment challenges network resource configuration, since any new reservation may change the latency of already validated flows. Standard methods of worst-case latency analysis for CBS have been found incorrect, and current TSN stream reservation procedures lack mechanisms to signal application layer Quality-of-Service (QoS) requirements or verify deadlines. In this paper, we propose and validate a QoS negotiation scheme that interacts with the TSN network controller to reserve resources while ensuring latency bounds. For the first time, this work comparatively evaluates reservation schemes using worst-case analysis and simulations of a realistic In-Vehicle Network (IVN) and demonstrates their impact on QoS guarantees, resource utilizatio
Background: The computationally intensive task of real-time rendering can be offloaded to remote cloud systems. However, due to network latency, interactive remote rendering (IRR) introduces the challenge of interaction latency (IL), which is the time between an action and response to that action. Objectives: to model sources of latency, measure it in a real-world network and to use this understanding to simulate latency so that we have a controlled platform for experimental work in latency management. Method: we present a seven-parameter model of latency for a typical IRR system; we describe new, minimally intrusive software methods for measuring latency in a 3D graphics environment and create a novel latency simulator tool in software. Results: We demonstrate our latency simulator is comparable to real-world behavior and confirm that real-world latency exceeds the interactive limit of 70ms over long distance connections. We also find that current approaches to measuring IL are not general enough for most situations and therefore propose a novel general-purpose solution. Conclusion: to ameliorate latency in IRR systems we need controllable simulation tools for experimentation. In
Despite efforts from cloud and content providers to lower latency to acceptable levels for current and future services (e.g., augmented reality or cloud gaming), there are still opportunities for improvement. A major reason that traffic engineering efforts are challenged to lower latency is that the Internet's inter-domain routing protocol, the Border Gateway Protocol, is oblivious to any performance metric, and circuitous routing is still pervasive. In this work, we propose two implementation modifications that networks can leverage to make BGP latency-aware and reduce excessive latency inflation. These proposals, latency-proportional AS prepending and local preference neutralization, show promise towards providing a method for propagating abstract latency information with a reasonable increase in routing overhead.
Caching is crucial for system performance, but the delayed hit phenomenon, where requests queue during lengthy fetches after a cache miss, significantly degrades user-perceived latency in modern high-throughput systems. While prior works address delayed hits by estimating aggregate delay, they universally assume deterministic fetch latencies. This paper tackles the more realistic, yet unexplored, scenario where fetch latencies are stochastic. We present, to our knowledge, the first theoretical analysis of delayed hits under this condition, deriving analytical expressions for both the mean and variance of the aggregate delay assuming exponentially distributed fetch latency. Leveraging these insights, we develop a novel variance-aware ranking function tailored for this stochastic setting to guide cache eviction decisions more effectively. The simulations on synthetic and real-world datasets demonstrate that our proposed algorithm significantly reduces overall latency compared to state-of-the-art delayed-hit strategies, achieving a $3\%-30\%$ reduction on synthetic datasets and approximately $1\%-7\%$ reduction on real-world traces.