搜索结果：billion

Graph-structured data underpins many critical applications. While foundation models have transformed language and vision via large-scale pretraining and lightweight adaptation, extending this paradigm to general, real-world graphs is challenging. In this work, we present Graph Billion-Foundation-Fusion (GraphBFF): an end-to-end recipe for building billion-parameter Graph Foundation Models (GFMs) for large-scale heterogeneous graphs. Central to the recipe is the GraphBFF Transformer, a flexible and scalable architecture designed for practical billion-scale GFMs. Using the GraphBFF, we present neural scaling laws for heterogeneous graphs and show that loss decreases predictably as either model capacity or training data scales, depending on which factor is the bottleneck. The GraphBFF framework provides concrete methodologies for data batching, pretraining, and fine-tuning for building GFMs at scale. We demonstrate the effectiveness of the framework over a real-world billion-scale graph, with an evaluation of a billion-parameter GraphBFF Transformer following the proposed recipe. Across ten diverse, real-world downstream tasks on graphs unseen during training, spanning node- and link-

The merger history of the Galaxy has been traced back firmly to redshift 2 (10 Billion years ago). While there have been claims of the existence of at least one more significant merger before this time, supporting evidence has been indirect and contentious. Here we show that the population of globular clusters around the Galaxy depicts three distinct age-metallicity sequences, one associated with the merger with Gaia-Enceladus 10 billion years ago, one to the progenitor of the Milky Way and a third intermediate sequence associated to at least one merger which we estimate took place merely 1.5 billion years after the Big Bang. This discovery has been possible thanks to exquisite Hubble Space Telescope data and sophisticated analysis that enables very precise relative age determination of globular clusters. The newly identified sequence reveals that this merger took place with an object of stellar mass similar to that of Gaia-Enceladus (~5x10$^8$ M$_{\odot}$), and which deposited most of its mass in the inner 6 kpc of the Milky Way. The unambiguous identification of a third merger event in the inner Galaxy puts to rest earlier debates, and honoring previous work we name the progenito

These notes present material from lectures given at the 54th Saas-Fee Advanced Course of the Swiss Society of Astrophysics and Astronomy in January 2025, entitled "Galaxies and Black Holes in the First Billion Years as seen by the JWST", and are intended for early career researchers or those new to the sub-field. My lectures covered the theory of galaxy formation with a focus on the first billion years of cosmic evolution. In these notes, I discuss cosmological structure formation, properties of dark matter halos at $z\gtrsim 6$, and whether any of the JWST observations to date present a serious and fundamental challenge for the $Λ$ Cold Dark Matter Paradigm. I then give an overview of physical processes and modeling techniques, including translating simulation-based quantities to observables, and discuss recent progress and future directions in galaxy formation modeling. The closing section presents a summary of some of the theoretical puzzles and challenges raised by the first three years of high redshift observations with JWST, and how our models of galaxy formation may need to be revised to accommodate them.

We provide an empirical investigation of the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples. We find that model performance tends to saturate at this scale on many common Western-centric classification and retrieval benchmarks, such as COCO Captions. Nevertheless, tasks of cultural diversity achieve more substantial gains from the 100-billion scale web data, thanks to its coverage of long-tail concepts. Furthermore, we analyze the model's multilinguality and show gains in low-resource languages as well. In addition, we observe that reducing the size of the pretraining dataset via quality filters like using CLIP, typically used to enhance performance, may inadvertently reduce the cultural diversity represented in large-scale datasets. Our results highlight that while traditional benchmarks may not benefit significantly from scaling noisy, raw web data to 100 billion examples, this data scale is vital for building truly inclusive multimodal systems.

Predicting Click-Through Rate (CTR) in billion-scale recommender systems poses a long-standing challenge for Graph Neural Networks (GNNs) due to the overwhelming computational complexity involved in aggregating billions of neighbors. To tackle this, GNN-based CTR models usually sample hundreds of neighbors out of the billions to facilitate efficient online recommendations. However, sampling only a small portion of neighbors results in a severe sampling bias and the failure to encompass the full spectrum of user or item behavioral patterns. To address this challenge, we name the conventional user-item recommendation graph as "micro recommendation graph" and introduce a more suitable MAcro Recommendation Graph (MAG) for billion-scale recommendations. MAG resolves the computational complexity problems in the infrastructure by reducing the node count from billions to hundreds. Specifically, MAG groups micro nodes (users and items) with similar behavior patterns to form macro nodes. Subsequently, we introduce tailored Macro Graph Neural Networks (MacGNN) to aggregate information on a macro level and revise the embeddings of macro nodes. MacGNN has already served Taobao's homepage feed f

Graph Neural Network (GNN) inference on billion-scale graphs is critical for domains like fintech and recommendation systems. Full-graph inference on these large graphs can be challenging due to high communication costs in distributed settings and high I/O costs in disk-backed Out-of-Core (OOC) settings. Existing OOC systems, operating across disk and memory, primarily focus on GNN training and perform poorly for full-graph inference due to massive read amplification, irregular I/O, and memory pressure. We present ATLAS, a disk-based GNN inference framework that enables efficient full-graph, layer-wise inference on graphs whose topologies, features and intermediate embeddings exceed the available memory on single machines. ATLAS replaces gather-based execution with a broadcast-based model that enables sequential, single-pass streaming reads of features and embeddings per layer. A tiered memory-disk hierarchy with minimum-pending-message eviction, graph reordering and a GPU-accelerated pipeline sustains high throughput within $128$ GiB RAM and $2$ TiB SSD. Across out-of-core graphs with up to $4$B edges and $550$ GiB features and multiple GNN architectures, ATLAS improves end-to-end

The problem of combining multiple forecasts of related quantities that obey expected equality and additivity constraints, often referred to a hierarchical forecast reconciliation, is naturally stated as a simple optimization problem. In this paper we explore optimization-based point forecast reconciliation at scales faced by large retailers. We implement and benchmark several algorithms to solve the forecast reconciliation problem, showing efficacy when the dimension of the problem exceeds four billion forecasted values. To the best of our knowledge, this is the largest forecast reconciliation problem, and perhaps on-par with the largest constrained least-squares-problem ever solved. We also make several theoretical contributions. We show that for a restricted class of problems and when the loss function is weighted appropriately, least-squares forecast reconciliation is equivalent to share-based forecast reconciliation. This formalizes how the optimization based approach can be thought of as a generalization of share-based reconciliation, applicable to multiple, overlapping data hierarchies.

Large Language Model (LLM)-based cold-start recommendation systems continue to face significant computational challenges in billion-scale scenarios, as they follow a "Text-to-Judgment" paradigm. This approach processes user-item content pairs as input and evaluates each pair iteratively. To maintain efficiency, existing methods rely on pre-filtering a small candidate pool of user-item pairs. However, this severely limits the inferential capabilities of LLMs by reducing their scope to only a few hundred pre-filtered candidates. To overcome this limitation, we propose a novel "Text-to-Distribution" paradigm, which predicts an item's interaction probability distribution for the entire user set in a single inference. Specifically, we present FilterLLM, a framework that extends the next-word prediction capabilities of LLMs to billion-scale filtering tasks. FilterLLM first introduces a tailored distribution prediction and cold-start framework. Next, FilterLLM incorporates an efficient user-vocabulary structure to train and store the embeddings of billion-scale users. Finally, we detail the training objectives for both distribution prediction and user-vocabulary construction. The proposed

Scaling up contrastive language-image pretraining (CLIP) is critical for empowering both vision and multimodal models. We present EVA-CLIP-18B, the largest and most powerful open-source CLIP model to date, with 18-billion parameters. With only 6-billion training samples seen, EVA-CLIP-18B achieves an exceptional 80.7% zero-shot top-1 accuracy averaged across 27 widely recognized image classification benchmarks, outperforming its forerunner EVA-CLIP (5-billion parameters) and other open-source CLIP models by a large margin. Remarkably, we observe a consistent performance improvement with the model size scaling of EVA-CLIP, despite maintaining a constant training dataset of 2-billion image-text pairs from LAION-2B and COYO-700M. This dataset is openly available and much smaller than the in-house datasets (e.g., DFN-5B, WebLI-10B) employed in other state-of-the-art CLIP models. EVA-CLIP-18B demonstrates the potential of EVA-style weak-to-strong visual model scaling. With our model weights made publicly available, we hope to facilitate future research in vision and multimodal foundation models.

While large transformer models have been successfully used in many real-world applications such as natural language processing, computer vision, and speech processing, scaling transformers for recommender systems remains a challenging problem. Recently, Generative Recommenders framework was proposed to scale beyond typical Deep Learning Recommendation Models (DLRMs). Reformulation of recommendation as sequential transduction task led to improvement of scaling properties in terms of compute. Nevertheless, the largest encoder configuration reported by the HSTU authors amounts only to ~176 million parameters, which is considerably smaller than the hundreds of billions or even trillions of parameters common in modern language models. In this work, we present a recipe for training large transformer recommenders with up to a billion parameters. We show that autoregressive learning on user histories naturally decomposes into two subtasks, feedback prediction and next-item prediction, and demonstrate that such a decomposition scales effectively across a wide range of transformer sizes. Furthermore, we report a successful deployment of our proposed architecture on a large-scale music platfo

Understanding the dynamic evolution of complex social phenomena requires both high-fidelity modeling of human behavior and large-scale simulations. Traditional agent-based models (ABMs) have been employed to study these dynamics, but are constrained by simplified agent behaviors. Recent advances in large language models (LLMs) enable agents to exhibit sophisticated social behaviors, yet face significant scaling challenges. We present Light Society, an agent-based simulation framework that advances both fronts. Light Society formalizes social processes as structured transitions of agent and environment states, governed by a set of LLM-powered simulation operations. Joint algorithmic and system optimizations, particularly a mixture-of-models engine that combines full LLMs with distilled surrogates, enable Light Society to efficiently simulate societies with over one billion agents. Grounded in real-world demographic profiles from the World Values Survey, simulations of Trust Games and opinion diffusion at up to one billion agents demonstrate Light Society's high fidelity and efficiency in modeling diverse social phenomena, providing researchers with a practical foundation for hypothe

Searching a multi-biometric database of a billion records for a country-level identity system requires pushing the limits of all aspects of a biometric system, including acquisition, preprocessing, feature extraction, accuracy, matching speed, presentation attack detection, and handling of special cases (e.g., missing finger digits). This is the first paper that gives insights into such a large-scale multimodal biometric search system, called Bharat ABIS, based on open-source architectures. The end-to-end pipeline of Bharat ABIS processes fingerprint, face and iris modalities through modality-specific stages of preprocessing (segmentation), quality assessment, presentation attack detection, and learning an embedding (feature extraction), producing a concatenated template of 13.5KB per person. We present a detailed analysis of the modalities and how they are integrated to create an efficient and effective solution for 1:N search (de-duplication). Evaluations on a demographically stratified gallery of 220 million identities, randomly sampled from 1.55 billion records in India's Aadhaar database, yield an FNIR of 0.3% at an FPIR of 0.5%, for adult probes (over 18 years). We also compa

The Collatz conjecture, also known as the 3n+1 problem, is one of the most popular open problems in number theory. In this note, an algorithm for the verification of the Collatz conjecture is presented that works on a standard PC for numbers with up to ten billion decimal places.

Moiré and super-moiré materials provide exceptional platforms to engineer exotic correlated quantum matter. The vast number of sites required to model moiré systems in real space remains a formidable challenge due to the immense computational resources required. Super-moiré materials push this requirement to the limit, where millions or even billions of sites need to be considered, a requirement beyond the capabilities of conventional methods for interacting systems. Here, we establish a methodology that allows solving correlated states in systems reaching a billion sites, that exploits tensor-network representations of real-space Hamiltonians and self-consistent real-space mean-field equations. Our method combines a tensor-network kernel polynomial method with quantics tensor cross interpolation algorithm, enabling us to solve exponentially large models, including those whose single particle Hamiltonian is too large to be stored explicitly. We demonstrate our methodology with super-moiré systems featuring spatially modulated hoppings, many-body interactions and domain walls, showing that it allows access to self-consistent symmetry broken states and spectral functions of real-spac

Understanding how supermassive black holes (SMBHs) form in the early universe is one of the most challenging problems in astrophysics. Their high abundance in the first billion years, as observed by the James Webb Space Telescope, hints towards black hole seeds that accrete mass rapidly. The origin of this accreted mass is not known. Here, we consider a billion solar mass clumpy galaxy at z=5.48 with a 30 million solar mass black hole in the center. We show that the clumps should migrate to the central region because of torques from dynamical friction with the halo, funneling in at least 14 solar masses per year. This is fast enough to grow the observed SMBH, with only 1% of the accreted mass getting in and the rest going to a bulge. Clump-fed accretion could explain most young SMBHs because young galaxies are highly irregular with massive star-forming clumps.

Observational studies showed that galaxy disks are already in place in the first few billion years of the universe. The early disks detected so far, with typical half-light radii of 3 kiloparsecs at stellar masses around 10^11 M_sun for redshift z~3, are significantly smaller than today's disks with similar masses, in agreement with expectations from current galaxy models. Here, we report observations of a giant disk at z=3.25, when the universe was only 2 billion years old, with a half-light radius of 9.6 kiloparsecs and stellar mass of 3.7^+2.6_-2.2x10^11 M_sun. This galaxy is larger than any other kinematically-confirmed disks at similar epochs and surprisingly similar to today's largest disks regarding size and mass. JWST imaging and spectroscopy reveal its spiral morphology and a rotational velocity consistent with local Tully-Fisher relation. Multi-wavelength observations show that it lies in an exceptionally dense environment, where the galaxy number density is over ten times higher than the cosmic average and mergers are frequent. The discovery of such a giant disk suggests the presence of favorable physical conditions for large-disk formation in dense environments in the e

In the video game "7 Billion Humans", the player is requested to direct a group of workers to various destinations by writing a program that is executed simultaneously on each worker. While the game is quite rich and, indeed, it is considered one of the best games for beginners to learn the basics of programming, we show that even extremely simple versions are already NP-Hard or PSPACE-Hard.

With stunning clarity, JWST has revealed the Universe's first billion years. The scientific community is analyzing a wealth of JWST imaging and spectroscopic data from that era, and is in the process of rewriting the astronomy textbooks. Here, 1.5 years into the JWST science mission, we provide a snapshot of the great progress made towards understanding the initial chapters of our cosmic history. We highlight discoveries and breakthroughs, topics and issues that are not yet understood, and questions that will be addressed in the coming years, as JWST continues its revolutionary observations of the Early Universe. While this compendium is written by a small number of authors, invited to ISSI Bern in March 2024 as part of the 2024 ISSI Breakthrough Workshop, we acknowledge the work of a large community that is advancing our collective understanding of the evolution of the Early Universe.

The field of Artificial Intelligence has witnessed remarkable progress in recent years, especially with the emergence of powerful large language models (LLMs) based on the transformer architecture. Cloud-based LLMs, such as OpenAI's ChatGPT, offer impressive capabilities but come with concerns regarding latency and privacy due to network dependencies. This article presents an innovative approach to LLM inference, envisioning a future where LLMs with billions of parameters can be executed directly on mobile devices without network connectivity. The article showcases a fine-tuned GPT LLM with 3 billion parameters that can operate smoothly on devices with as low as 4GB of memory. Through the integration of native code and model quantization techniques, the application not only serves as a general-purpose assistant but also facilitates seamless mobile interactions with text-to-actions features. The article provides insights into the training pipeline, implementation details, test results, and future directions of on-device LLM inference. This breakthrough technology opens up possibilities for empowering users with sophisticated AI capabilities while preserving their privacy and elimina