搜索结果：expand

Hyper-Connections (HC) expand the residual stream of Transformers into $N$ parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from $N{=}1$ to $N{=}4$ suggest residual-stream expansion as a promising scaling axis. However, existing HC-family methods typically stop at $N{=}4$. Our experiments reveal why: scaling mHC beyond this point yields diminishing performance gains and rapidly increasing training cost. We attribute this limitation to two bottlenecks: insufficient write-back information for an expanding number of streams and residual-mixing generation whose cost scales cubically with $N$. To address both bottlenecks, we propose xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond $N{=}4$. xHC combines temporal feature augmentation for richer write-back with a sparse residual-stream architecture that updates only $k=4$ of the $N=16$ streams while retaining dense access to the full residual state. Across 18B and 28B MoE models, xHC delivers strong and consistent downstream improvements. On an 18B MoE model, xHC improves t

In this paper, we show how to expand Euclidean/Hermitian self-orthogonal code preserving their orthogonal property. Our results show that every $k$-dimension Hermitian self-orthogonal code is contained in a $(k+1)$-dimensional Hermitian self-orthogonal code. Also, for $k< n/2-1$, every $[n,k]$ Euclidean self-orthogonal code is contained in an $[n,k+1]$ Euclidean self-orthogonal code. Moreover, for $k=n/2-1$ and $p=2$, we can also fulfill the expanding process. But for $k=n/2-1$ and $p$ odd prime, the expanding process can be fulfilled if and only if an extra condition must be satisfied. We also propose two feasible algorithms on these expanding procedures.

We construct the first explicit two-sided vertex expanders that bypass the spectral barrier. Previously, the strongest known explicit vertex expanders were given by $d$-regular Ramanujan graphs, whose spectral properties imply that every small subset of vertices $S$ has at least $0.5d|S|$ distinct neighbors. However, it is possible to construct Ramanujan graphs containing a small set $S$ with no more than $0.5d|S|$ neighbors. In fact, no explicit construction was known to break the $0.5 d$-barrier. In this work, we give an explicit construction of an infinite family of $d$-regular graphs (for large enough $d$) where every small set expands by a factor of $\approx 0.6d$. More generally, for large enough $d_1,d_2$, we give an infinite family of $(d_1,d_2)$-biregular graphs where small sets on the left expand by a factor of $\approx 0.6d_1$, and small sets on the right expand by a factor of $\approx 0.6d_2$. In fact, our construction satisfies an even stronger property: small sets on the left and right have unique-neighbor expansion $0.6d_1$ and $0.6d_2$ respectively. Our construction follows the tripartite line product framework of Hsieh, McKenzie, Mohanty & Paredes, and instanti

We give an efficient algorithm that transforms any bounded degree expander graph into another that achieves almost optimal (namely, near-quadratic, $d \leq 1/λ^{2+o(1)}$) trade-off between (any desired) spectral expansion $λ$ and degree $d$. Furthermore, the algorithm is local: every vertex can compute its new neighbors as a subset of its original neighborhood of radius $O(\log(1/λ))$. The optimal quadratic trade-off is known as the Ramanujan bound, so our construction gives almost Ramanujan expanders from arbitrary expanders. The locality of the transformation preserves structural properties of the original graph, and thus has many consequences. Applied to Cayley graphs, our transformation shows that any expanding finite group has almost Ramanujan expanding generators. Similarly, one can obtain almost optimal explicit constructions of quantum expanders, dimension expanders, monotone expanders, etc., from existing (suboptimal) constructions of such objects. Another consequence is a "derandomized" random walk on the original (suboptimal) expander with almost optimal convergence rate. Our transformation also applies when the degree is not bounded or the expansion is not constant. We

Early weakly supervised video grounding (WSVG) methods often struggle with incomplete boundary detection due to the absence of temporal boundary annotations. To bridge the gap between video-level and boundary-level annotation, explicit-supervision methods, i.e., generating pseudo-temporal boundaries for training, have achieved great success. However, data augmentations in these methods might disrupt critical temporal information, yielding poor pseudo boundaries. In this paper, we propose a new perspective that maintains the integrity of the original temporal content while introducing more valuable information for expanding the incomplete boundaries. To this end, we propose EtC (Expand then Clarify), first use the additional information to expand the initial incomplete pseudo boundaries, and subsequently refine these expanded ones to achieve precise boundaries. Motivated by video continuity, i.e., visual similarity across adjacent frames, we use powerful multimodal large language models (MLLMs) to annotate each frame within initial pseudo boundaries, yielding more comprehensive descriptions for expanded boundaries. To further clarify the noise of expanded boundaries, we combine mutu

In this survey we aim to give a comprehensive overview of results using sublinear expanders. The term sublinear expanders refers to a variety of definitions of expanders, which typically are defined to be graphs $G$ such that every not-too-small and not-too-large set of vertices $U$ has neighbourhood of size at least $α|U|$, where $α$ is a function of $n$ and $|U|$. This is in contrast with linear expanders, where $α$ is typically a constant. :We will briefly describe proof ideas of some of the results mentioned here, as well as related open problems.

Activation functions are core components of all deep learning architectures. Currently, the most popular activation functions are smooth ReLU variants like GELU and SiLU. These are self-gated activation functions where the range of the gating function is between zero and one. In this paper, we explore the viability of using arctan as a gating mechanism. A self-gated activation function that uses arctan as its gating function has a monotonically increasing first derivative. To make this activation function competitive, it is necessary to introduce a trainable parameter for every MLP block to expand the range of the gating function beyond zero and one. We find that this technique also improves existing self-gated activation functions. We conduct an empirical evaluation of Expanded ArcTan Linear Unit (xATLU), Expanded GELU (xGELU), and Expanded SiLU (xSiLU) and show that they outperform existing activation functions within a transformer architecture. Additionally, expanded gating ranges show promising results in improving first-order Gated Linear Units (GLU).

We propose a definition of expander representations of quivers, generalizing dimension (or linear algebra) expanders, as a qualitative refinement of slope stability. We prove existence of uniform expander representations for any wild quiver over an algebraically closed base field, using the concept of general subrepresentations and spectral properties of Cartan matrices.

This work demonstrates how increasing the number of neurons in a network without increasing its total number of non-zero parameters improves performance. We show that this gain corresponds with a decrease in interference between multiple features that would otherwise share the same neurons. On symbolic Boolean tasks, splitting each neuron into sparser sub-neurons with knowledge of the clauses systematically reduces polysemanticity metrics and yields higher task accuracy. Notably, even random splits of neuron weights approximate these gains, indicating that reduced collisions, not precise assignment, are a primary driver. Consistent with the superposition hypothesis, the benefits of this framework grow with increasing interference: when polysemantic load is high, accuracy improvements are the largest. Transferring these insights to more realistic models, including classifiers over CLIP embeddings, convolutional neural networks, and deeper multilayer networks, we find that widening networks while maintaining a constant non-zero parameter count consistently increases accuracy. These results identify an interpretability-grounded mechanism to leverage width against superposition, improv

We study the problem of constructing explicit sparse graphs that exhibit strong vertex expansion. Our main result is the first two-sided construction of imbalanced unique-neighbor expanders, meaning bipartite graphs where small sets contained in both the left and right bipartitions exhibit unique-neighbor expansion, along with algebraic properties relevant to constructing quantum codes. Our constructions are obtained from instantiations of the tripartite line product of a large tripartite spectral expander and a sufficiently good constant-sized unique-neighbor expander, a new graph product we defined that generalizes the line product in the work of Alon and Capalbo and the routed product in the work of Asherov and Dinur. To analyze the vertex expansion of graphs arising from the tripartite line product, we develop a sharp characterization of subgraphs that can arise in bipartite spectral expanders, generalizing results of Kahale, which may be of independent interest. By picking appropriate graphs to apply our product to, we give a strongly explicit construction of an infinite family of $(d_1,d_2)$-biregular graphs $(G_n)_{n\ge 1}$ (for large enough $d_1$ and $d_2$) where all sets $

In this paper, we use geometric tools to study the structure of asymptotic expanders and show that a sequence of asymptotic expanders always admits a "uniform exhaustion by expanders". It follows that asymptotic expanders cannot be coarsely embedded into any $L^p$-space, and that asymptotic expanders can be characterised in terms of their uniform Roe algebra. Moreover, we provide uncountably many new counterexamples to the coarse Baum--Connes conjecture. These appear to be the first counterexamples that are not directly constructed by means of spectral gaps. Finally, we show that vertex-transitive asymptotic expanders are actually expanders. In particular, this gives a $C^*$-algebraic characterisation of expanders for vertex-transitive graphs.

Expanding existing learning systems to provide high-quality customized models for more domains, such as new users, is challenged by the limited labeled data and the data and device heterogeneities. While knowledge distillation methods could overcome label scarcity and device heterogeneity, they assume the teachers are fully reliable and overlook the data heterogeneity, which prevents the direct adoption of existing models. To address this problem, this paper proposes a framework, HaT, to expand learning systems. It first selects multiple high-quality models from the system at a low cost and then fuses their knowledge by assigning sample-wise weights to their predictions. Later, the fused knowledge is selectively injected into the customized models based on the knowledge quality. Extensive experiments on different tasks, modalities, and settings show that HaT outperforms state-of-the-art baselines by up to 16.5% accuracy and saves up to 39% communication traffic.

We introduce a new concept, which we call partition expanders. The basic idea is to study quantitative properties of graphs in a slightly different way than it is in the standard definition of expanders. While in the definition of expanders it is required that the number of edges between any pair of sufficiently large sets is close to the expected number, we consider partitions and require this condition only for most of the pairs of blocks. As a result, the blocks can be substantially smaller. We show that for some range of parameters, to be a partition expander a random graph needs exponentially smaller degree than any expander would require in order to achieve similar expanding properties. We apply the concept of partition expanders in communication complexity. First, we give a PRG for the SMP model of the optimal seed length, n+O(log k). Second, we compare the model of SMP to that of Simultaneous Two-Way Communication, and give a new separation that is stronger both qualitatively and quantitatively than the previously known ones.

In this paper, we propose Vocab-Expander at https://vocab-expander.com, an online tool that enables end-users (e.g., technology scouts) to create and expand a vocabulary of their domain of interest. It utilizes an ensemble of state-of-the-art word embedding techniques based on web text and ConceptNet, a common-sense knowledge base, to suggest related terms for already given terms. The system has an easy-to-use interface that allows users to quickly confirm or reject term suggestions. Vocab-Expander offers a variety of potential use cases, such as improving concept-based information retrieval in technology and innovation management, enhancing communication and collaboration within organizations or interdisciplinary projects, and creating vocabularies for specific courses in education.

A two-dimensional simplicial complex is called $d$-{\em regular} if every edge of it is contained in exactly $d$ distinct triangles. It is called $ε$-expanding if its up-down two-dimensional random walk has a normalized maximal eigenvalue which is at most $1-ε$. In this work, we present a class of bounded degree 2-dimensional expanders, which is the result of a small 2-complex action on a vertex set. The resulted complexes are fully transitive, meaning the automorphism group acts transitively on their faces. Such two-dimensional expanders are rare! Known constructions of such bounded degree two-dimensional expander families are obtained from deep algebraic reasonings (e.g. coset geometries). We show that given a small $d$-regular two-dimensional $ε$-expander, there exists an $ε'=ε'(ε)$ and a family of bounded degree two-dimensional simplicial complexes with a number of vertices goes to infinity, such that each complex in the family satisfies the following properties: * It is $4d$-regular. * The link of each vertex in the complex is the same regular graph (up to isomorphism). * It is $ε'$ expanding. * It is transitive. The family of expanders that we get is explicit if the one-skele

Adversarial examples are vital to expose the vulnerability of machine learning models. Despite the success of the most popular substitution-based methods which substitutes some characters or words in the original examples, only substitution is insufficient to uncover all robustness issues of models. In this paper, we present AdvExpander, a method that crafts new adversarial examples by expanding text, which is complementary to previous substitution-based methods. We first utilize linguistic rules to determine which constituents to expand and what types of modifiers to expand with. We then expand each constituent by inserting an adversarial modifier searched from a CVAE-based generative model which is pre-trained on a large scale corpus. To search adversarial modifiers, we directly search adversarial latent codes in the latent space without tuning the pre-trained parameters. To ensure that our adversarial examples are label-preserving for text matching, we also constrain the modifications with a heuristic rule. Experiments on three classification tasks verify the effectiveness of AdvExpander and the validity of our adversarial examples. AdvExpander crafts a new type of adversarial e

A cross-bifix-free code of length $n$ over $\mathbb{Z}_q$ is defined as a non-empty subset of $\mathbb{Z}_q^n$ satisfying that the prefix set of each codeword is disjoint from the suffix set of every codeword. Cross-bifix-free codes have found important applications in digital communication systems. One of the main research problems on cross-bifix-free codes is to construct cross-bifix-free codes as large as possible in size. Recently, Wang and Wang introduced a family of cross-bifix-free codes $S_{I,J}^{(k)}(n)$, which is a generalization of the classical cross-bifix-free codes studied early by Lvenshtein, Gilbert and Chee {\it et al.}. It is known that $S_{I,J}^{(k)}(n)$ is nearly optimal in size and $S_{I,J}^{(k)}(n)$ is non-expandable if $k=n-1$ or $1\leq k<n/2$. In this paper, we first show that $S_{I,J}^{(k)}(n)$ is non-expandable if and only if $k=n-1$ or $1\leq k<n/2$, thereby improving the results in [Chee {\it et al.}, IEEE-TIT, 2013] and [Wang and Wang, IEEE-TIT, 2022]. We then construct a new family of cross-bifix-free codes $U^{(t)}_{I,J}(n)$ to expand $S_{I,J}^{(k)}(n)$ such that the resulting larger code $S_{I,J}^{(k)}(n)\bigcup U^{(t)}_{I,J}(n)$ is a non-expan

This paper proposes the transition-net, a robust transition strategy that expands the versatility of robot locomotion in the real-world setting. To this end, we start by distributing the complexity of different gaits into dedicated locomotion policies applicable to real-world robots. Next, we expand the versatility of the robot by unifying the policies with robust transitions into a single coherent meta-controller by examining the latent state representations. Our approach enables the robot to iteratively expand its skill repertoire and robustly transition between any policy pair in a library. In our framework, adding new skills does not introduce any process that alters the previously learned skills. Moreover, training of a locomotion policy takes less than an hour with a single consumer GPU. Our approach is effective in the real-world and achieves a 19% higher average success rate for the most challenging transition pairs in our experiments compared to existing approaches.

Let $G=(V,E)$ be a finite graph. For $v\in V$ we denote by $G_v$ the subgraph of $G$ that is induced by $v$'s neighbor set. We say that $G$ is $(a,b)$-regular for $a>b>0$ integers, if $G$ is $a$-regular and $G_v$ is $b$-regular for every $v\in V$. Recent advances in PCP theory call for the construction of infinitely many $(a,b)$-regular expander graphs $G$ that are expanders also locally. Namely, all the graphs $\{G_v|v\in V\}$ should be expanders as well. While random regular graphs are expanders with high probability, they almost surely fail to expand locally. Here we construct two families of $(a,b)$-regular graphs that expand both locally and globally. We also analyze the possible local and global spectral gaps of $(a,b)$-regular graphs. In addition, we examine our constructions vis-a-vis properties which are considered characteristic of high-dimensional expanders.