搜索结果：Cleaning

Multivariate time series (MTS) are frequently affected by co-occurring quality issues, such as missing values, outliers, and constraint violations, which significantly undermine downstream analytics. Existing cleaning approaches fix only a limited set of such issues, making them ill-suited for scenarios where multiple quality problems arise simultaneously. Furthermore, these methods commonly depend on the availability of ground truth data or domain-specific rules, both of which are rarely accessible in real-world applications. In this paper, we introduce AegisTS, an agent system with reinforcement learning designed to clean multiple data quality issues in MTS. We cast the cleaning process as a joint optimization problem that simultaneously handles quality issue order and cleaning model selection, allowing efficient navigation of the large space of possible cleaning pipelines. Our framework relies on a hierarchical agent architecture, where a high-level agent determines the order in which data quality issues should be processed, while a low-level agent identifies the most suitable cleaning method for each issue. To guide the agent toward an optimal cleaning pipeline, we propose a du

Data quality is crucial in machine learning (ML) applications, as errors in the data can significantly impact the prediction accuracy of the underlying ML model. Therefore, data cleaning is an integral component of any ML pipeline. However, in practical scenarios, data cleaning incurs significant costs, as it often involves domain experts for configuring and executing the cleaning process. Thus, efficient resource allocation during data cleaning can enhance ML prediction accuracy while controlling expenses. This paper presents COMET, a system designed to optimize data cleaning efforts for ML tasks. COMET gives step-by-step recommendations on which feature to clean next, maximizing the efficiency of data cleaning under resource constraints. We evaluated COMET across various datasets, ML algorithms, and data error types, demonstrating its robustness and adaptability. Our results show that COMET consistently outperforms feature importance-based, random, and another well-known cleaning method, achieving up to 52 and on average 5 percentage points higher ML prediction accuracy than the proposed baselines.

Context: Machine Learning (ML) is integrated into a growing number of systems for various applications. Because the performance of an ML model is highly dependent on the quality of the data it has been trained on, there is a growing interest in approaches to detect and repair data errors (i.e., data cleaning). Researchers are also exploring how ML can be used for data cleaning; hence creating a dual relationship between ML and data cleaning. To the best of our knowledge, there is no study that comprehensively reviews this relationship. Objective: This paper's objectives are twofold. First, it aims to summarize the latest approaches for data cleaning for ML and ML for data cleaning. Second, it provides future work recommendations. Method: We conduct a systematic literature review of the papers published between 2016 and 2022 inclusively. We identify different types of data cleaning activities with and for ML: feature cleaning, label cleaning, entity matching, outlier detection, imputation, and holistic data cleaning. Results: We summarize the content of 101 papers covering various data cleaning activities and provide 24 future work recommendations. Our review highlights many promisi

This study introduces a time-based cleaning method for H.E.S.S. using CT5 in monoscopic mode and presents an optimization workflow for image-cleaning algorithms to enhance telescope sensitivity while minimizing systematic biases. We evaluate three methods - tail-cut cleaning and two flavours of time-based cleaning TIME3D and TIME4D - and find best-cut configurations for two cases: optimal overall sensitivity and minimal energy threshold. TIME3D achieves a $\sim 15\%$ improvement with regard to the standard tail-cut cleaning for $E < 300$ GeV, with a $\sim 200\%$ improvement for the first energy bin (36.5 GeV < E < 64.9 GeV), providing a more stable performance across a wider energy range by preserving more signal. TIME4D achieves a $\sim 20\%$ improvement at low energies due to superior NSB noise suppression, allowing for an enhanced capability of detecting sources at the lowest energies. We demonstrate that using first-order estimations of the performance of a cleaning, such as image size retaining or NSB pixel reduction, cannot provide a full picture of the expected result in the final sensitivity. Beyond expanding the effective area at low energies, sensitivity improvem

Commercial lunar activity is accelerating the need for reliable surface infrastructure and routine operations to keep it functioning. Maintenance tasks such as inspection, cleaning, dust mitigation, and minor repair are essential to preserve performance and extend system life. A specific application is the cleaning of lunar solar arrays. Solar arrays are expected to provide substantial fraction of lunar surface power and operate for months to years, supplying continuous energy to landers, habitats, and surface assets, making sustained output mission-critical. However, over time lunar dust accumulates on these large solar arrays, which can rapidly degrade panel output and reduce mission lifetime. We propose a small mobile robot equipped with a long-reach, lightweight deployable boom and interchangeable cleaning tool to perform gentle cleaning over meter-scale workspaces with minimal human involvement. Building on prior vision-guided long-reach manipulation, we add a compliant wrist with distal force sensing and a velocity-based admittance controller to regulate stable contact during surface cleaning. In preliminary benchtop experiments on a planar surface, the system maintained appr

There has been extensive research on automating and scaling data cleaning, i.e., the detection and correction of erroneous values in tabular data. Yet, existing approaches often perform well only within controlled environments. One of the major bottlenecks in data cleaning research is the lack of real-world datasets. In this paper, we address this gap by providing a large, dirty dataset with postal entries and their corresponding ground truth. We discuss the design decisions and challenges for obtaining the dataset. We demonstrate the limitations of existing cleaning approaches when faced with our proposed datasets and derive guidelines for future research.

Data cleaning is a crucial yet challenging task in data analysis, often requiring significant manual effort. To automate data cleaning, previous systems have relied on statistical rules derived from erroneous data, resulting in low accuracy and recall. This work introduces Cocoon, a novel data cleaning system that leverages large language models for rules based on semantic understanding and combines them with statistical error detection. However, data cleaning is still too complex a task for current LLMs to handle in one shot. To address this, we introduce Cocoon, which decomposes complex cleaning tasks into manageable components in a workflow that mimics human cleaning processes. Our experiments show that Cocoon outperforms state-of-the-art data cleaning systems on standard benchmarks.

Streaming data can arise from a variety of contexts. Important use cases are continuous sensor measurements such as temperature, light or radiation values. In the process, streaming data may also contain data errors that should be cleaned before further use. Many studies from science and practice focus on data cleaning in a static context. However, in terms of data cleaning, streaming data has particularities that distinguish it from static data. In this paper, we have therefore undertaken an intensive exploration of data cleaning of data streams. We provide a detailed analysis of the applicability of data cleaning to data streams. Our theoretical considerations are evaluated in comprehensive experiments. Using a prototype framework, we show that cleaning is not consistent when working with data streams. An additional contribution is the investigation of requirements for streaming technologies in context of data cleaning.

Solar energy is used for many mission-critical applications including space exploration, sensor systems to monitor wildfires, etc. Their operation can be limited or even terminated if solar panels are covered with dust or hit by space debris. To address this issue, we designed panel cleaning mechanisms and tested protective materials. For cleaning mechanisms, we designed and compared a wiper system and a rail system. For protective materials, we found through collision tests that polycarbonate was very promising, though the most important factor was layering a soft material between the panel's surface and a hard material. In the cleaning system comparisons, the wiper-based system was more efficient than the rail-based system in terms of cost, cleaning speed, and total power consumption.

Investigative drilling (ID) is an innovative measurement while drilling (MWD) technique that has been implemented in various site investigation projects across Australia. While the automated drilling feature of ID substantially reduces noise within drilling data streams, data cleaning remains essential for removing anomalies to enable accurate strata classification and prediction of soil and rock properties. This study employed three machine learning algorithms--IsoForest, one-class SVM, and DBSCAN--to automate the data cleaning process for ID data in rock drilling scenarios. Two data cleaning contexts were examined: (1) removing anomalies in rock drilling data, and (2) removing both anomalies and soil drilling data in mixed rock drilling data. The analysis revealed that all three machine learning algorithms outperformed traditional statistical methods (the 3-sigma rule and IQR method) in both data cleaning tasks, achieving a good balance between true positive rate and false positive rate, though hyperparameter tuning was required for one-class SVM and DBSCAN. Among them, IsoForest was proven to be the best-performing algorithm, capable of removing anomalies effectively without the

Human feedback plays a pivotal role in aligning large language models (LLMs) with human preferences. However, such feedback is often noisy or inconsistent, which can degrade the quality of reward models and hinder alignment. While various automated data cleaning methods have been proposed to mitigate this issue, a systematic evaluation of their effectiveness and generalizability remains lacking. To bridge this gap, we introduce the first comprehensive benchmark for evaluating 13 preference data cleaning methods in the context of LLM alignment. PrefCleanBench offers a standardized protocol to assess cleaning strategies in terms of alignment performance and generalizability across diverse datasets, model architectures, and optimization algorithms. By unifying disparate methods and rigorously comparing them, we uncover key factors that determine the success of data cleaning in alignment tasks. This benchmark lays the groundwork for principled and reproducible approaches to improving LLM alignment through better data quality-highlighting the crucial but underexplored role of data preprocessing in responsible AI development. We release modular implementations of all methods to catalyze

Traffic classification, a technique for assigning network flows to predefined categories, has been widely deployed in enterprise and carrier networks. With the massive adoption of mobile devices, encryption is increasingly used in mobile applications to address privacy concerns. Consequently, traditional methods such as Deep Packet Inspection (DPI) fail to distinguish encrypted traffic. To tackle this challenge, Artificial Intelligence (AI), in particular Machine Learning (ML), has emerged as a promising solution for encrypted traffic classification. A crucial prerequisite for any ML-based approach is traffic data cleaning, which removes flows that are not useful for training (e.g., irrelevant protocols, background activity, control-plane messages, and long-lived sessions). Existing cleaning solutions depend on manual inspection of every captured packet, making the process both costly and time-consuming. In this poster, we present an unsupervised framework that automatically cleans encrypted mobile traffic. Evaluation on real-world datasets shows that our framework incurs only a 2%~2.5% reduction in classification accuracy compared with manual cleaning. These results demonstrate th

Despite the observable benefit of Natural Language Processing (NLP) in processing a large amount of textual medical data within a limited time for information retrieval, a handful of research efforts have been devoted to uncovering novel data-cleaning methods. Data cleaning in NLP is at the centre point for extracting validated information. Another observed limitation in the NLP domain is having limited medical corpora that provide answers to a given medical question. Realising the limitations and challenges from two perspectives, this research aims to clean a medical dataset using ensemble techniques and to develop a corpus. The corpora expect that it will answer the question based on the semantic relationship of corpus sequences. However, the data cleaning method in this research suggests that the ensemble technique provides the highest accuracy (94%) compared to the single process, which includes vectorisation, exploratory data analysis, and feeding the vectorised data. The second aim of having an adequate corpus was realised by extracting answers from the dataset. This research is significant in machine learning, specifically data cleaning and the medical sector, but it also un

This paper describes the problem of drift of solid non-interacting particles in a microchannel, which can stick to its walls under the action of the van der Waals forces and break away from the wall due to thermal noise and viscous stresses arising from the flow. The pressure drop is given between the channel inlet and outlet. At the initial moment of time, the channel walls are contaminated with adhered particles, i.e. the walls are uneven, which affects the formation of the flow structure through the channel. Over time, under the action of viscous stresses and thermal noise, the particles break away from the channel walls, causing its cleaning. The interaction of the detached particles with the flow is taken into account in the Stokes approximation. In addition, the model takes into account random particle motion caused by diffusion. The problem is solved numerically within the framework of the random walk model. The evolution of the liquid flow in the channel during its cleaning is obtained: stream function, pressure, and vorticity fields. The dependencies of the volume occupied by settled particles, the flow rate through the channel and the channel gap on time are determined fo

Though data cleaning systems have earned great success and wide spread in both academia and industry, they fall short when trying to clean spatial data. The main reason is that state-of-the-art data cleaning systems mainly rely on functional dependency rules where there is sufficient co-occurrence of value pairs to learn that a certain value of an attribute leads to a corresponding value of another attribute. However, for spatial attributes that represent locations on the form of <latitude, longitude>, there is very little chance that two records would have the same exact coordinates, and hence co-occurrence would unlikely to exist. This paper presents Sparcle~(SPatially-AwaRe CLEaning); a novel framework that injects spatial awareness into the core engine of rule-based data cleaning systems as a means of boosting their accuracy. Sparcle injects two main spatial concepts into the core engine of data cleaning systems: (1) Spatial Neighborhood, where co-occurrence is relaxed to be within a certain spatial proximity rather than same exact value, and (2) Distance Weighting, where records are given different weights of whether they satisfy a dependency rule, based on their relativ

This paper presents a novel methodology for characterizing soiling losses through experimental measurements. Soiling predictions were obtained by calibrating a soiling model based on field measurements from a 50 MW modular solar tower project in Mount Isa, Australia. The study found that the mean predicted soiling rate for horizontally fixed mirrors was 0.12 percentage points per day (pp/d) during low dust seasons and 0.22 pp/d during high seasons. Autoregressive time series models were employed to extend two years of onsite meteorological measurements to a 10-year period, enabling the prediction of heliostat-field soiling rates. A fixed-frequency cleaning heuristic was applied to optimise the cleaning resources for various operational policies by balancing direct cleaning resource costs against the expected lost production, which was computed by averaging multiple simulated soiling loss trajectories. Analysis of resource usage showed that the cost of fuel and operator salaries contributed 42 % and 35 % respectively towards the cleaning cost. In addition, stowing heliostats in the horizontal position at night increased daily soiling rates by 114 % and the total cleaning costs by 51

Recently, as the demand for cleaning robots has steadily increased, therefore household electricity consumption is also increasing. To solve this electricity consumption issue, the problem of efficient path planning for cleaning robot has become important and many studies have been conducted. However, most of them are about moving along a simple path segment, not about the whole path to clean all places. As the emerging deep learning technique, reinforcement learning (RL) has been adopted for cleaning robot. However, the models for RL operate only in a specific cleaning environment, not the various cleaning environment. The problem is that the models have to retrain whenever the cleaning environment changes. To solve this problem, the proximal policy optimization (PPO) algorithm is combined with an efficient path planning that operates in various cleaning environments, using transfer learning (TL), detection nearest cleaned tile, reward shaping, and making elite set methods. The proposed method is validated with an ablation study and comparison with conventional methods such as random and zigzag. The experimental results demonstrate that the proposed method achieves improved traini

Imaging atmospheric Cherenkov telescopes, such as the VERITAS array, are subject to the Night Sky Background (NSB) and electronic noise, which contribute to the total signal of pixels in the telescope camera. The contribution of noise photons in event images is reduced with the application of image cleaning methods. Conventionally, high thresholds must be employed to ensure the removal of pixels containing noise signal. On that account, low-energy gamma-ray showers might be suppressed during the cleaning. We present here the application of an optimised next neighbour image cleaning for the VERITAS array. With this technique, differential noise rates are estimated for each individual observation and thus changes in the NSB and afterpulsing are consistently being accounted for. We show that this method increases the overall rate of reconstructed gamma-rays, lowers the energy threshold of the array and allows the reconstruction of low energy (E > 70 GeV) source events which were suppressed by the conventional cleaning method.

We formulate a Bayesian framework to analyze ringdown gravitational waves from colliding binary black holes and test the no-hair theorem. The idea hinges on mode cleaning -- revealing subdominant oscillation modes by removing dominant ones using newly proposed ${\it rational~filters}$. By incorporating the filter into Bayesian inference, we construct a likelihood function that depends only on the mass and spin of the remnant black hole (no dependence on mode amplitudes and phases) and implement an efficient pipeline to constrain the remnant mass and spin without Markov chain Monte Carlo (MCMC). We test ringdown models by cleaning combinations of different modes and evaluating the consistency between the residual data and pure noise. The model evidence and Bayes factor are used to demonstrate the presence of a particular mode and to infer the mode starting time. In addition, we design a hybrid approach to estimate the remnant black hole properties exclusively from a single mode using MCMC after mode cleaning. We apply the framework to GW150914 and demonstrate more definitive evidence of the first overtone by cleaning the fundamental mode. This new framework provides a powerful tool