The ability to programmatically retrieve vast quantities of data from online sources has given rise to increasing usage of web-scraped datasets for various purposes across government, industry and academia. Contemporaneously, there has also been growing discussion about the statistical qualities and limitations of collecting from online data sources and analysing web-scraped datasets. However, literature on web-scraping is distributed across computer science, statistical methodology and application domains, with distinct and occasionally conflicting definitions of web-scraping and conceptualisations of web-scraped data quality. This work synthesises technical and statistical concepts, best practices and insights across these relevant disciplines to inform documentation during web-scraping processes, and quality assessment of the resultant web-scraped datasets. We propose an integrated framework to cover multiple processes during the creation of web-scraped datasets including 'Plan', 'Retrieve', 'Investigate', 'Transform', 'Evaluate' and 'Summarise' (PRITES). The framework groups related quality factors which should be monitored during the collection of new web-scraped data, and/or
We investigate the contents of web-scraped data for training AI systems, at sizes where human dataset curators and compilers no longer manually annotate every sample. Building off of prior privacy concerns in machine learning models, we ask: What are the legal privacy implications of web-scraped machine learning datasets? In an empirical study of a popular training dataset, we find significant presence of personally identifiable information despite sanitization efforts. Our audit provides concrete evidence to support the concern that any large-scale web-scraped dataset may contain legally defined personal data. We use these findings of a real-world dataset to inform our legal analysis with respect to existing privacy and data protection laws. We surface various legal risks of current data curation practices that may propagate personal information to train downstream models. Based on our empirical and legal analyses, we argue for reorientation of current frameworks of "publicly available" information to meaningfully limit the development of AI built upon indiscriminate scraping of the internet.
The increasing adoption of econometric and machine-learning approaches by empirical researchers has led to a widespread use of one data collection method: web scraping. Web scraping refers to the use of automated computer programs to access websites and download their content. The key argument of this paper is that naïve web scraping procedures can lead to sampling bias in the collected data. This article describes three sources of sampling bias in web-scraped data. More specifically, sampling bias emerges from web content being volatile (i.e., being subject to change), personalized (i.e., presented in response to request characteristics), and unindexed (i.e., abundance of a population register). In a series of examples, I illustrate the prevalence and magnitude of sampling bias. To support researchers and reviewers, this paper provides recommendations on anticipating, detecting, and overcoming sampling bias in web-scraped data.
Web-scraped, in-the-wild datasets have become the norm in face recognition research. The numbers of subjects and images acquired in web-scraped datasets are usually very large, with number of images on the millions scale. A variety of issues occur when collecting a dataset in-the-wild, including images with the wrong identity label, duplicate images, duplicate subjects and variation in quality. With the number of images being in the millions, a manual cleaning procedure is not feasible. But fully automated methods used to date result in a less-than-ideal level of clean dataset. We propose a semi-automated method, where the goal is to have a clean dataset for testing face recognition methods, with similar quality across men and women, to support comparison of accuracy across gender. Our approach removes near-duplicate images, merges duplicate subjects, corrects mislabeled images, and removes images outside a defined range of pose and quality. We conduct the curation on the Asian Face Dataset (AFD) and VGGFace2 test dataset. The experiments show that a state-of-the-art method achieves a much higher accuracy on the datasets after they are curated. Finally, we release our cleaned versi
Web scraping has historically required technical expertise in HTML parsing, session management, and authentication circumvention, which limited large-scale data extraction to skilled developers. We argue that large language models (LLMs) have democratized web scraping, enabling low-skill users to execute sophisticated operations through simple natural language prompts. While extensive benchmarks evaluate these tools under optimal expert conditions, we show that without extensive manual effort, current LLM-based workflows allow novice users to scrape complex websites that would otherwise be inaccessible. We systematically benchmark what everyday users can do with off-the-shelf LLM tools across 35 sites spanning five security tiers, including authentication, anti-bot, and CAPTCHA controls. We devise and evaluate two distinct workflows: (a) LLM-assisted scripting, where users prompt LLMs to generate traditional scraping code but maintain manual execution control, and (b) end-to-end LLM agents, which autonomously navigate and extract data through integrated tool use. Our results demonstrate that end-to-end agents have made complex scraping accessible - requiring as little as a single p
The increasing demand for accelerated scientific discovery, driven by global challenges, highlights the need for advanced AI-driven robotics. Deploying robotic chemists in human-centric labs is key for the next horizon of autonomous discovery, as complex tasks still demand the dexterity of human scientists. Robotic manipulation in this context is uniquely challenged by handling diverse chemicals (granular, powdery, or viscous liquids), under varying lab conditions. For example, humans use spatulas for scraping materials from vial walls. Automating this process is challenging because it goes beyond simple robotic insertion tasks and traditional lab automation, requiring the execution of fine-granular movements within a constrained environment (the sample vial). Our work proposes an adaptive control framework to address this, relying on a low-level Cartesian impedance controller for stable and compliant physical interaction and a high-level reinforcement learning agent that learns to dynamically adjust interaction forces at the end-effector. The agent is guided by perception feedback, which provides the material's location. We first created a task-representative simulation environmen
Bibliographic catalogues store millions of data. The use of computer techniques such as web-scraping allows the extraction of data in an efficient and accurate manner. The recent emergence of ChatGPT is facilitating the development of suitable prompts that allow the configuration of scraping to identify and extract information from databases. The aim of this article is to define how to efficiently use prompts engineering to elaborate a suitable data entry model, able to generate in a single interaction with ChatGPT-4o, a fully functional web-scraper, programmed in PHP language, adapted to the case of bibliographic catalogues. As a demonstration example, the bibliographic catalogue of the National Library of Spain with a dataset of thousands of records is used. The findings present an effective model for developing web-scraping programs, assisted with AI and with the minimum possible interaction. The results obtained with the model indicate that the use of prompts with large language models (LLM) can improve the quality of scraping by understanding specific contexts and patterns, adapting to different formats and styles of presentation of bibliographic information.
This paper presents the design and implementation of a user-friendly, automated web application that simplifies and optimizes the web scraping process for non-technical users. The application breaks down the complex task of web scraping into three main stages: fetching, extraction, and execution. In the fetching stage, the application accesses target websites using the HTTP protocol, leveraging the requests library to retrieve HTML content. The extraction stage utilizes powerful parsing libraries like BeautifulSoup and regular expressions to extract relevant data from the HTML. Finally, the execution stage structures the data into accessible formats, such as CSV, ensuring the scraped content is organized for easy use. To provide personalized and secure experiences, the application includes user registration and login functionalities, supported by MongoDB, which stores user data and scraping history. Deployed using the Flask framework, the tool offers a scalable, robust environment for web scraping. Users can easily input website URLs, define data extraction parameters, and download the data in a simplified format, without needing technical expertise. This automated tool not only en
An improved time delay estimation method is used to calculate the velocity of cross-field blob motion in the scrape-off layer of Alcator C-Mod for an ohmic and two high confinement (H-mode) plasmas; an edge localized mode free and an enhanced D-alpha H-mode. The gas puff imaging data analysis results are interpreted in the framework of a stochastic model that describes the fluctuations as a super-position of uncorrelated blob-like structures. In all confinement modes investigated, the scrape-off layer is dominated by large amplitude, blob-like filaments moving radially outwards with velocities in the range from 400 to 1000 m/s. Blobs in high confinement mode plasmas have similar velocities and sizes as in ohmic plasma, which is consistent with the close similarity of conditionally averaged burst shapes and frequency spectra for the confinement modes investigated.
This technical note presents a reproducible workflow for converting a legacy archaeological image collection into a structured and segmentation ready dataset. The case study focuses on the Lower Palaeolithic hand axe and biface collection curated by the Archaeology Data Service (ADS), a dataset that provides thousands of standardised photographs but no mechanism for bulk download or automated processing. To address this, two open source tools were developed: a web scraping script that retrieves all record pages, extracts associated metadata, and downloads the available images while respecting ADS Terms of Use and ethical scraping guidelines; and an image processing pipeline that renames files using UUIDs, generates binary masks and bounding boxes through classical computer vision, and stores all derived information in a COCO compatible Json file enriched with archaeological metadata. The original images are not redistributed, and only derived products such as masks, outlines, and annotations are shared. Together, these components provide a lightweight and reusable approach for transforming web based archaeological image collections into machine learning friendly formats, facilitati
GenAI companies are strip-mining the web. Their scraping bots harvest content at an unprecedented scale, circumventing technical barriers to fuel billion-dollar models while creators receive nothing. Courts have enabled this exploitation by misunderstanding what property rights protect online. The prevailing view treats websites as mere repositories of intellectual property and dismisses trespass claims absent server damage. That framework grants AI companies presumptive access while ignoring the economic devastation they inflict. But the content is severable from the website itself. This paper reframes the debate: websites are personal property as integrated digital assets subject to the same exclusionary rights as physical chattels. When scrapers bypass access controls and divert traffic that sustains a website's value, they commit actionable trespass. The law need not create new protections; it need only apply existing property principles to digital space. Courts and litigants have struggled to police unwanted, large-scale scraping because copyright preemption often narrows available claims, leaving copyright and its fair use defense as the primary battleground. Trespass to chat
The "Tete de moine" Swiss cheese is generally served by scraping the surface of a cylindrical loaf with a sharp tool. This produces thin sheets of cheese that are strongly wrinkled at the edge, resembling frilly flowers and enhancing the tasting experience. In this work we unveil the physical mechanisms at play in this scraping-induced morphogenesis. We measure the deformation of the cheese during scraping and show that plastic deformation occurs everywhere, but find a larger plastic contraction in the inner part of the flower, causing its buckling into shape. We show that it surprisingly derives from the lower friction coefficient evidenced on the cheese close to its crust. Our analysis provides the tools for a better control of chip morphogenesis through plasticity in the shaping of other delicacies, but also in metal cutting.
In jamming systems like colloids, emulsions, foams, and biological tissues, significant deformation is essential for processes such as material scraping or wound self-healing. To adequately spread a foam or cream over a surface, external force must be applied to artificially scrape it. The scraping of foam using a rigid plate has been observed to exhibit complex behavior distinct from that of simple liquids. In this study, we quantitatively analyzed the transition between partial and slender scraping regimes by examining changes in internal structure and partial spreading lengths. Our findings reveal that the sequential propagation of bubble rearrangement in the foam's internal structure leads to the partial scraping. Moreover, the scraping length in the partial scraping regime shows divergence near the transition point, characterized by a critical exponent of approximately 0.61. These results imply that foam scraping is governed by directional percolation theory, supported by the agreement between the experimentally observed critical exponent and theoretical predictions. This research significantly advances the understanding of macroscopic kinetics and rheological behavior in jamm
Scientists across disciplines often use data from the internet to conduct research, generating valuable insights about human behavior. However, as generative AI relying on massive text corpora becomes increasingly valuable, platforms have greatly restricted access to data through official channels. As a result, researchers will likely engage in more web scraping to collect data, introducing new challenges and concerns for researchers. This paper proposes a comprehensive framework for web scraping in social science research for U.S.-based researchers, examining the legal, ethical, institutional, and scientific factors that researchers should consider when scraping the web. We present an overview of the current regulatory environment impacting when and how researchers can access, collect, store, and share data via scraping. We then provide researchers with recommendations to conduct scraping in a scientifically legitimate and ethical manner. We aim to equip researchers with the relevant information to mitigate risks and maximize the impact of their research amidst this evolving data access landscape.
The scrape-off layer parallel heat flux decay lengths measured at ST40, a high field, low aspect ratio spherical tokamak, have been observed to bifurcate into two groups. The wide group matches closely with the scale of ion poloidal Larmour radius and follows existing H-mode scalings, while the narrow group falls up to 10 times below scalings, on the scale of ion total Larmour radius. The onset of the narrow scrape-off layer width is observed to be associated with suppressed magnetic fluctuations, suggesting reduced electromagnetic turbulence levels in the SOL.
The emergence and merging of high-density coherent structures - plasma blobs - is a recurrent phenomenon in the Scrape-off layer (SOL) of a tokamak plasma that has a significant impact on the rate of convective transport in that region. We report on a model study of the merging of two electromagnetically interacting blobs in a high beta plasma. Our detailed numerical simulations show that the merging process is akin to the coalescence instability between two magnetic islands but with important differences due to the density perturbation. The blobs are found to rotate about each other during merging and the merging occurs with an acceleration in the poloidal direction that is directly proportional to the square of the current density of the blobs and inversely proportional to its density. The separation distance between two high current density blobs is also seen to oscillate indicating a sloshing behavior.
As the exploration of digital behavioral data revolutionizes communication research, understanding the nuances of data collection methodologies becomes increasingly pertinent. This study focuses on one prominent data collection approach, web scraping, and more specifically, its application in the growing field of research relying on web browsing data. We investigate discrepancies between content obtained directly during user interaction with a website (in-situ) and content scraped using the URLs of participants' logged visits (ex-situ) with various time delays (0, 30, 60, and 90 days). We find substantial disparities between the methodologies, uncovering that errors are not uniformly distributed across news categories regardless of classification method (domain, URL, or content analysis). These biases compromise the precision of measurements used in existing literature. The ex-situ collection environment is the primary source of the discrepancies (~33.8%), while the time delays in the scraping process play a smaller role (adding ~6.5 percentage points in 90 days). Our research emphasizes the need for data collection methods that capture web content directly in the user's environmen
Large Language Models (LLMs) demonstrate remarkable capabilities in replicating human tasks and boosting productivity. However, their direct application for data extraction presents limitations due to a prioritisation of fluency over factual accuracy and a restricted ability to manipulate specific information. Therefore to overcome these limitations, this research leverages the knowledge representation power of pre-trained LLMs and the targeted information access enabled by RAG models, this research investigates a general-purpose accurate data scraping recipe for RAG models designed for language generation. To capture knowledge in a more modular and interpretable way, we use pre trained language models with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus. We utilised RAG model architecture and did an in-depth analysis of their capabilities under three tasks: (i) Semantic Classification of HTML elements, (ii) Chunking HTML text for effective understanding, and (iii) comparing results from different LLMs and ranking algorithms. While previous work has developed dedicated architectures and training procedures for HTML unde
The turbulence characteristics of the scrape-off-layer (SOL) plasma in the W7-X stellarator are investigated using a gas-puff-imaging (GPI) diagnostic, newly installed and operated during the OP 2.1 campaign. The SOL plasma on W7-X features a set of island divertors for heat and particle exhaust and provides a unique environment for studying SOL turbulence and transport. This paper focuses on the O-point region of the magnetic island divertor SOL in the standard magnetic configuration. Fourier and cross-correlation analyses show that turbulence flows are predominantly in the poloidal direction (i.e., direction tangent to the last closed flux surface) with significantly weaker radial motion. This suggests dominant ExB convection and suppressed filamentary transport compared to those observed in the far scrape-off-layer region of tokamaks, as further supported by high-resolution skewness and kurtosis data that show the absence of intermittent, bursty filamentary events. Additionally, a relationship between the radial profile of the connection length and the sheared poloidal flow structure is reported, suggesting a possible linkage among magnetic topology, turbulence dynamics, and tur
[WhitePaper] The TelegramScrap tool provides a robust and versatile solution for extracting and analyzing data from Telegram channels and groups, addressing the increasing demand for efficient methods to study digital ecosystems. This white paper outlines the tool's development, capabilities, and applications in academic and scientific research, including studies on disinformation, political communication, and thematic patterns in online communities. Built with flexibility and user accessibility in mind, the tool allows researchers to customize scraping parameters, handle large datasets, and produce structured outputs in formats such as Excel and Parquet. Its modular architecture, real-time progress tracking, and error-handling mechanisms ensure reliability and scalability for diverse research needs. Emphasizing ethical data collection, the tool aligns with Telegram's terms of service and data privacy regulations, encouraging responsible use. Released under an open-source license, TelegramScrap invites the academic community to explore, adapt, and improve the tool while providing appropriate credit. This paper demonstrates the tool's impact through its application in multiple studi