搜索结果：sink

Diffusion Language Models (DLMs) incur high inference cost due to iterative denoising, motivating efficient pruning. Existing pruning heuristics largely inherited from autoregressive (AR) LLMs, typically preserve attention sink tokens because AR sinks serve as stable global anchors. We show that this assumption does not hold for DLMs: the attention-sink position exhibits substantially higher variance over the full generation trajectory (measured by how the dominant sink locations shift across timesteps), indicating that sinks are often transient and less structurally essential than in AR models. Based on this observation, we propose ${\bf \texttt{Sink-Aware Pruning}}$, which automatically identifies and prunes unstable sinks in DLMs (prior studies usually keep sinks for AR LLMs). Without retraining, our method achieves a better quality-efficiency trade-off and outperforms strong prior pruning baselines under matched compute. Our code is available at https://github.com/VILA-Lab/Sink-Aware-Pruning.

Attention sinks are tokens, often the beginning-of-sequence (BOS) token, that receive disproportionately high attention despite limited semantic relevance. In this work, we identify a class of attention sinks, which we term secondary sinks, that differ fundamentally from the sinks studied in prior works, which we term primary sinks. While prior works have identified that tokens other than BOS can sometimes become sinks, they were found to exhibit properties analogous to the BOS token. Specifically, they emerge at the same layer, persist throughout the network and draw a large amount of attention mass. Whereas, we find the existence of secondary sinks that arise primarily in middle layers and can persist for a variable number of layers, and draw a smaller, but still significant, amount of attention mass. Through extensive experiments across 11 model families, we analyze where these secondary sinks appear, their properties, how they are formed, and their impact on the attention mechanism. Specifically, we show that: (1) these sinks are formed by specific middle-layer MLP modules; these MLPs map token representations to vectors that align with the direction of the primary sink of that

The goal of this paper is to strengthen the reasoning of Omnimodal Large Language Models (Omni-LLMs) at inference time, without additional training. These models jointly process video, audio, and text, and given the large number of tokens they consume, how attention is routed across them is central to their behaviour. We focus specifically on attention sinks, tokens that absorb a disproportionate share of attention mass regardless of their semantic content, to understand how this routing unfolds. To this end, we conduct a systematic analysis of sink behaviour in Omni-LLMs. Our analysis yields two key findings: (i) high sink attention does not solely indicate head redundancy, suggesting that sink value representations play additional functional roles; (ii) the sink value vector acts as a shared bias added to every token's output, serving as a global signal that organises the representation as a whole. Building on this, we propose OutRo, which correspondingly aligns non-sink token representations with the sink in feature space, and relaxes the causal mask for sink tokens at an early layer to sharpen this bias before the rest of decoding proceeds. This design enhances the reasoning pr

Harmful fine-tuning can invalidate safety alignment of large language models, exposing significant safety risks. In this paper, we utilize the attention sink mechanism to mitigate harmful fine-tuning. Specifically, we first measure a statistic named \emph{sink divergence} for each attention head and observe that \emph{different attention heads exhibit two different signs of sink divergence}. To understand its safety implications, we conduct experiments and find that the number of attention heads of positive sink divergence increases along with the increase of the model's harmfulness when undergoing harmful fine-tuning. Based on this finding, we propose a separable sink divergence hypothesis -- \emph{attention heads associating with learning harmful patterns during fine-tuning are separable by their sign of sink divergence}. Based on the hypothesis, we propose a fine-tuning-stage defense, dubbed Surgery. Surgery utilizes a regularizer for sink divergence suppression, which steers attention heads toward the negative sink divergence group, thereby reducing the model's tendency to learn and amplify harmful patterns. Extensive experiments demonstrate that Surgery improves defense perfor

Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this issue, including Sink Attention in GPT-OSS and Gated Attention in Qwen3-Next. However, a comprehensive analysis of the relationship among these attention mechanisms is lacking. In this work, we provide both theoretical and empirical evidence demonstrating that the sink in Vanilla Attention and Sink Attention naturally construct a Mixture-of-Experts (MoE) mechanism within attention layers. This insight explains the head collapse phenomenon observed in prior work, where only a fixed subset of attention heads contributes to generation. To mitigate head collapse, we propose a sink-aware training algorithm with an auxiliary load balancing loss designed for attention layers. Extensive experiments show that our method achieves effective head load balancing and improves model performance across Vanilla Attention, Sink Attention, and Gated Attention. We hope this study offers a new perspective on attention mechanisms and encourages further exploration of the inherent MoE structure within attention layers.

Diffusion Language Models (DLMs) have emerged as a compelling alternative to autoregressive approaches, enabling parallel text generation with competitive performance. Despite these advantages, there is a critical instability in DLMs: the moving sink phenomenon. Our analysis indicates that sink tokens exhibit low-norm representations in the Transformer's value space, and that the moving sink phenomenon serves as a protective mechanism in DLMs to prevent excessive information mixing. However, their unpredictable positions across diffusion steps undermine inference robustness. To resolve this, we propose a simple but effective extra sink token implemented via a modified attention mask. Specifically, we introduce a special token constrained to attend solely to itself, while remaining globally visible to all other tokens. Experimental results demonstrate that introducing a single extra token stabilizes attention sinks, substantially improving model performance. Crucially, further analysis confirms that the effectiveness of this token is independent of its position and characterized by negligible semantic content, validating its role as a robust and dedicated structural sink.

Attention mechanisms are central to the success of large language models (LLMs), enabling them to capture intricate token dependencies and implicitly assign importance to each token. Recent studies have revealed the sink token, which receives disproportionately high attention despite their limited semantic role. In this paper, we first expand the relationship between the sink token and other tokens, moving beyond attention to explore their similarity in hidden states, considering the layer depth. We observe that as the layers get deeper, the cosine similarity between the normalized hidden states of the sink token and those of other tokens increases, and that the normalized hidden states of the sink token exhibit negligible changes. These imply that other tokens consistently are directed toward the sink token throughout the layers. Next, we propose a dynamic token selection method, called OrthoRank, using these findings to select important tokens. Specifically, in a certain layer, we define token importance by the speed at which the token moves toward the sink token. This is converted into orthogonality with the sink token, meaning that tokens that are more orthogonal to the sink to

Vision Transformers (ViTs) face severe computational bottlenecks due to the quadratic complexity of self-attention at high resolutions. Existing token reduction methods rely on local metrics - such as single-layer attention scores - that are inherently vulnerable to the attention sink phenomenon, where uninformative tokens are paradoxically preserved over salient foreground objects. We propose ASAP (Attention Sink Anchored Pruning), a training-free framework that recasts this sink as a feature. Modeling ViT information flow as a Lazy Random Walk, ASAP identifies the sink as a dominant accumulator of probability mass. By computing the diffusion distance to the sink within the cumulative transition matrix, ASAP partitions tokens via Radial Diffusion Clustering and compresses background redundancy through Transition Weight Pooling in a single shot. Extensive experiments across image, video, and vision-language tasks demonstrate ASAP outperforms state-of-the-art methods, accelerating throughput by up to 48% while maintaining - or even exceeding - baseline accuracy.

We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel, they improve inter-layer routing flexibility but also exacerbate attention sinks, activation outliers, and the resulting degradation in inference stability and quantization robustness. OASIS addresses this issue by introducing a Softmax1-based null space and coupling token-level null evidence to depth routing through an inter-layer null signal, thereby reducing sink-dominated routing and improving structural robustness. Theoretically, we show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness. Experimentally, we compare OASIS against five baselines on three real-world datasets and observe consistent improvements in both attention sink and post-quantization performance. Notably, OASIS achieves an average reduction of 9.26% in maximum infinity norm and 2.60% in average kurtosis across the evaluated settings, while lowering perplexity by 75.85% under W8A8 and improving GSM8K Pass@1 by 12.42% under W4A4.

Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in higher layers, are more redundant has remained elusive. In this work, we identify the BOS sink phenomenon as a key mechanism driving this layer-wise sensitivity. We show that attention heads with high BOS sink scores are strongly associated with functional redundancy: such heads, especially in deeper layers, contribute little to predictive performance and effectively serve as \emph{dumping grounds} for superfluous attention weights. This provides a concrete functional explanation for the structural redundancy reported in prior studies. Leveraging this insight, we introduce a simple pruning strategy that removes high-BOS sink heads. Experiments on Gemma-3, Llama-3.1, and Qwen3 demonstrate that this approach identifies redundant transformer components more reliably than weight- or activation-based criteria, while preserving performance close to dense baselines even under aggressive pruning. Moreover, we find that the behavior of sink heads remains stable across different sequence lengths. Overall, our results suggest that structural proper

Radiation-induced segregation (RIS) and chemical redistribution in structural alloys can significantly degrade material performance, ultimately leading to failure. In this study, building on previous work by the authors [1], we investigate how the dimensional characteristics of sinks influence solute concentration distributions and segregation behavior. Specifically, we utilize a kinetic Monte Carlo (KMC) model to simulate atomic-scale diffusion and analyze segregation processes in an Fe-3Cr alloy. Our analysis includes three representative sink geometries: one-dimensional (1D), two-dimensional (2D), and three-dimensional (3D) planar sinks to capture the effects of sink dimensionality on Cr segregation at grain boundaries (GBs). We also found solutions of concentration and segregation profiles in these cases as well as for a 3D spherical sink. KMC simulations are performed over a range of temperatures to assess thermal effects on Cr redistribution. The results reveal distinct segregation profiles and concentration gradients, although the dependence with sink density seems to remain linear in all cases with planar sinks. The analytical results show that this is not the case in spher

Subtraction games are a classical topic in Combinatorial Game Theory. A result of Golomb~(1966) shows that every subtraction game with a finite move set has an eventually periodic nim-sequence, but the known proof yields only an exponential upper bound on the period length. Flammenkamp~(1997) conjectures a striking classification for three-move subtraction games: non-additive rulesets exhibit linear period lengths of the form ``the sum of two moves'', where the choice of which two moves displays fractal-like behavior, while additive sets $S=\{a,b,a+b\}$ have purely periodic outcomes with linear or quadratic period lengths. Despite early attention in Winning Ways~(1982), the general additive case remains open. We introduce and analyze a dual winning convention, which we call {\sc sink subtraction}. Unlike the standard {\em wall} convention, where moves to negative positions are forbidden, the sink convention declares a player the winner upon moving to a non-positive position. We show that {\sc additive sink subtraction} admits a complete solution: the nim-sequence is purely periodic with an explicit linear or quadratic period formula, and we conjecture a duality between additive sin

Click-Through Rate (CTR) prediction, a core task in recommendation systems, estimates user click likelihood using historical behavioral data. Modeling user behavior sequences as text to leverage Language Models (LMs) for this task has gained traction, owing to LMs' strong semantic understanding and contextual modeling capabilities. However, a critical structural gap exists: user behavior sequences consist of discrete actions connected by semantically empty separators, differing fundamentally from the coherent natural language in LM pre-training. This mismatch causes semantic fragmentation, where LM attention scatters across irrelevant tokens instead of focusing on meaningful behavior boundaries and inter-behavior relationships, degrading prediction performance. To address this, we propose $\textit{CTR-Sink}$, a novel framework introducing behavior-level attention sinks tailored for recommendation scenarios. Inspired by attention sink theory, it constructs attention focus sinks and dynamically regulates attention aggregation via external information. Specifically, we insert sink tokens between consecutive behaviors, incorporating recommendation-specific signals such as temporal dist

For an element $g$ of a group $G$, a right Engel sink of $g$ is a subset of $G$ containing all sufficiently long commutators $[...[[g ,x],x],\dots ,x]$ for all $x\in G$. A left Engel sink of $g$ is a subset of $G$ containing all sufficiently long commutators $[...[[x ,g ],g ],\dots ,g]$ for all $x\in G$. Using the classification of finite simple groups we prove that if a finite group $G$ has an element $g$ such that $G=[G,g]$, then the order of $G$ is bounded in terms of a right Engel sink of $g$, as well as in terms of a left Engel sink of $g$. Earlier Guralnick and Tracey proved this in the case where $g$ is an involution without using the classification.

Language Models (LMs) assign significant attention to the first token, even if it is not semantically important, which is known as attention sink. This phenomenon has been widely adopted in applications such as streaming/long context generation, KV cache optimization, inference acceleration, model quantization, and others. Despite its widespread use, a deep understanding of attention sink in LMs is still lacking. In this work, we first demonstrate that attention sinks exist universally in LMs with various inputs, even in small models. Furthermore, attention sink is observed to emerge during the LM pre-training, motivating us to investigate how optimization, data distribution, loss function, and model architecture in LM pre-training influence its emergence. We highlight that attention sink emerges after effective optimization on sufficient training data. The sink position is highly correlated with the loss function and data distribution. Most importantly, we find that attention sink acts more like key biases, storing extra attention scores, which could be non-informative and not contribute to the value computation. We also observe that this phenomenon (at least partially) stems from

Recent works on WSNs show that use of mobile sink can prolong network lifetime. This paper demonstrates the advantages of the mobile sink in WSNs for increasing their lifetime than static sink. A novel sink mobility with coverage algorithm has been proposed here. During the movement of the sinks in the network, they sojourn at different points in the network and collect data over there. The direction of the movement of the sinks is determined by the algorithm and sinks are mobile inside the network. This algorithm can also be used for multiple sinks partitioned network. Simulations has been carried out for both mobile and static sinks to determine the average energy consumption of the network. At the same time, it also determines the network lifetime in terms of number of rounds neighbor nodes of sink is alive and first node die of the network. Our experiments show that mobile sinks outperform static sink in all scenarios. Furthermore, the proposed algorithm results in a good balancing of energy depletion among the sensor nodes.

Characterizing the limit behavior -- that is, the attractors -- of learning dynamics is one of the most fundamental open questions in game theory. In recent work on this front, it was conjectured that the attractors of the replicator dynamic are in one-to-one correspondence with the sink equilibria of the game -- the sink strongly connected components of a game's preference graph -- and it was established that they do stand in at least one-to-many correspondence with them. Here, we show that the one-to-one conjecture is false. We disprove this conjecture over the course of three theorems: the first disproves a stronger form of the conjecture, while the weaker form is disproved separately in the two-player and $N$-player ($N>2$) cases. By showing how the conjecture fails, we lay out the obstacles that lie ahead for characterizing attractors of the replicator, and introduce new ideas with which to tackle them. All three counterexamples derive from an object called a local source -- a point lying within the sink equilibrium, and yet which is `locally repelling'; we prove that the absence of local sources is necessary, but not sufficient, for the one-to-one property to be true. We c

A Dark Sink uses dark-sector interactions to siphon energy from dark matter to lighter dark degrees of freedom, i.e. dark radiation. Here, we extend dark matter models containing a Dark Sink to sub-MeV masses. We consider a Dark Sink model where the dark matter is charged under a light dark photon that has kinetic mixing with the Standard Model. For sub-MeV dark matter masses, plasmon decays are the dominant mechanism for transferring energy to the dark sector. Relative to a standard freeze-in cosmology, reproducing the observed dark matter density in a Dark Sink structure requires an increase in the dark matter couplings to the Standard Model, and hence increased direct detection cross sections. These models provide benchmarks for current and upcoming direct detection experiments. Accounting for plasmon effects, we derive the range of possible dark matter masses and cross sections for Dark Sink models in the sub-MeV regime. We make code available to reproduce our benchmarks; it may be of use for other freeze-in scenarios, including those where plasmon decays to the dark matter are important.

We describe a simple dark sector structure which, if present, has implications for the direct detection of dark matter (DM): the Dark Sink. A Dark Sink transports energy density from the DM into light dark-sector states that do not appreciably contribute to the DM density. As an example, we consider a light, neutral fermion $ψ$ which interacts solely with DM $χ$ via the exchange of a heavy scalar $Φ$. We illustrate the impact of a Dark Sink by adding one to a DM freeze-in model in which $χ$ couples to a light dark photon $γ'$ which kinetically mixes with the Standard Model (SM) photon. This freeze-in model (absent the sink) is itself a benchmark for ongoing experiments. In some cases, the literature for this benchmark has contained errors; we correct the predictions and provide them as a public code. We then analyze how the Dark Sink modifies this benchmark, solving coupled Boltzmann equations for the dark-sector energy density and DM yield. We check the contribution of the Dark Sink $ψ$'s to dark radiation; consistency with existing data limits the maximum attainable cross section. For DM with a mass between $\text{MeV} -\mathcal{O}(10\text{ GeV})$, adding the Dark Sink can increa