搜索结果：checks

Agentic data science (ADS) pipelines have grown rapidly in both capability and adoption, with systems such as OpenAI Codex now able to directly analyze datasets and produce answers to statistical questions. However, these systems can reach falsely optimistic conclusions that are difficult for users to detect. To address this, we propose a pair of lightweight sanity checks grounded in the Predictability-Computability-Stability (PCS) framework for veridical data science. These checks use reasonable perturbations to screen whether an agent can reliably distinguish signal from noise, acting as a falsifiability constraint that can expose affirmative conclusions as unsupported. Together, the two checks characterize the trustworthiness of an ADS output, e.g. whether it has found stable signal, is responding to noise, or is sensitive to incidental aspects of the input. We validate the approach on synthetic data with controlled signal-to-noise ratios, confirming that the sanity checks track ground-truth signal strength. We then demonstrate the checks on 11 real-world datasets using OpenAI Codex, characterizing the trustworthiness of each conclusion and finding that in 6 of the datasets an a

Empirical researchers often use diagnostic checks to assess the plausibility of their modeling assumptions, such as testing for covariate balance in RCTs, pre-trends in event studies, or instrument validity in IV designs. While these checks are traditionally treated as external hurdles to estimation, we argue they should be integrated into the estimation process itself. In particular, we propose residualizing one's baseline estimator against the vector of diagnostic check statistics to remove the component of baseline sampling variation explained by the diagnostic checks. This residualized estimator offers researchers a "free lunch," delivering three properties simultaneously: (i) eliminating inference distortions from check-based selective reporting; (ii) reducing variance without changing the estimand when the baseline model is correctly specified; and (iii) minimizing worst-case bias under bounded local misspecification within the class of linear adjustments. We apply our method to the RCT in Kaur et al. (2024) and find that, even in a setting where all balance checks pass comfortably, residualization increases the magnitude of the baseline point estimate and reduces its standar

Purpose: Accurate assessment of the Critical View of Safety (CVS) during laparoscopic cholecystectomy is essential to prevent bile duct injury, a complication associated with significant morbidity and mortality. While large vision-language models (LVLMs) offer flexible reasoning, their predictions remain difficult to audit and unreliable on safety-critical surgical tasks. Methods: We introduce Sum-of-Checks, a framework that decomposes each CVS criterion into expert-defined reasoning checks reflecting clinically relevant visual evidence. Given a laparoscopic frame, an LVLM evaluates each check, producing a binary judgment and justification. Criterion-level scores are computed via fixed, weighted aggregation of check outcomes. We evaluate on the Endoscapes2023 benchmark using three frontier LVLMs, comparing against direct prompting, chain-of-thought, and sub-question decomposition, each with and without few-shot examples. Results: Sum-of-Checks improves average frame-level mean average precision by 12--14% relative to the best baseline across all three models and criteria. Analysis of individual checks reveals that LVLMs are reliable on observational checks (e.g., visibility, tool o

Robustness checks are routine in empirical work, but there is no standard statistical procedure to formally measure what one can learn from them. I propose a "robustness radius" measure to quantify the amount by which the robustness checks estimands differ from the main specification estimand. I do so by framing robustness checks as explicitly biased regressions, clarifying what exactly the estimands are when comparing multiple regressions with slightly different samples, and applying a test from the moment inequalities literature. The robustness radius is easily interpretable and adapts to sampling uncertainty and correlation across regressions. An application shows that, although assessing overall robustness is context-specific, the robustness radius guides those judgments and improves transparency.

Safety-critical prediction systems, such as autonomous vehicles, weather forecasters, and medical monitors, commonly rely on probabilistic forecasters. These forecasters make predictions about possible future outcomes, and their quality and robustness needs to be validated and certified. Often, only accuracy -- the mean of the predictions -- is evaluated against true outcomes. However, for safety-critical scenarios and decision making under uncertainty, the full distributional properties of the forecasts should be checked: do the observed prediction errors actually follow the forecasted probability distributions? To this end, we introduce a framework for calibration checks: statistical tests that validate distributional properties of forecasts when measured over many samples. In order to support ease-of-use in real-world operations, these checks produce a single accept/reject decision for data collected from a forecaster. This contrasts typical calibration calculations which produce one or multiple continuous calibration scores and require expertise to implement in a validation workflow. We further support operationalization by introducing modifications to calibration testing that

Joint models for longitudinal and time-to-event data are widely used in many disciplines. Nonetheless, existing model comparison criteria do not indicate whether a model adequately fits the data or which components may be misspecified. We introduce a Bayesian posterior predictive checks framework for assessing a joint model's fit to the longitudinal and survival processes and their association. The framework supports multiple settings, including existing subjects, new subjects with only covariates, dynamic prediction at intermediate follow-up times, and cross-validated assessment. For the longitudinal component, goodness-of-fit is assessed through the mean, variance, and correlation structure, while the survival component is evaluated using empirical cumulative distributions and probability integral transforms. The association between processes is examined using time-dependent concordance statistics. We apply these checks to the Bio-SHiFT heart failure study, and a simulation study demonstrates that they can identify model misspecification that standard information criteria fail to detect. The proposed methodology is implemented in the freely available R package JMbayes2.

In data analysis, unexpected results often prompt researchers to revisit their procedures to identify potential issues. While some researchers may struggle to identify the root causes, experienced researchers can often quickly diagnose problems by checking a few key assumptions. These checked assumptions, or expectations, are typically informal, difficult to trace, and rarely discussed in publications. In this paper, we introduce the term *analysis validation checks* to formalize and externalize these informal assumptions. We then introduce a procedure to identify a subset of checks that best predict the occurrence of unexpected outcomes, based on simulations of the original data. The checks are evaluated in terms of accuracy, determined by binary classification metrics, and independence, which measures the shared information among checks. We demonstrate this approach with a toy example using step count data and a generalized linear model example examining the effect of particulate matter air pollution on daily mortality.

Quantum error correction typically requires repeated syndrome extraction due to measurement noise, which results in substantial time overhead in fault-tolerant computation. Single-shot error correction aims to suppress errors using only one round of syndrome extraction. However, for most codes, it requires high-weight checks, which significantly degrade, and often eliminate, single-shot performance at the circuit level. In this work, we introduce local single-shot checks, where we impose constraints on check weights. Using a dynamic measurement scheme, we show that the number of required measurement rounds can be reduced by a factor determined by this constraint. As an example, we show through numerical simulation that our scheme can improve decoding performance compared to conventional checks when using sliding-window decoding with a reduced window size under circuit-level noise models for toric codes. Our work provides a new direction for constructing checks that can reduce time overhead in large-scale fault-tolerant quantum computation.

A key step in the Bayesian workflow for model building is the graphical assessment of model predictions, whether these are drawn from the prior or posterior predictive distribution. The goal of these assessments is to identify whether the model is a reasonable (and ideally accurate) representation of the domain knowledge and/or observed data. There are many commonly used visual predictive checks which can be misleading if their implicit assumptions do not match the reality. Thus, there is a need for more guidance for selecting, interpreting, and diagnosing appropriate visualizations. As a visual predictive check itself can be viewed as a model fit to data, assessing when this model fails to represent the data is important for drawing well-informed conclusions. We present recommendations and diagnostic tools to mitigate ad-hoc decision-making in visual predictive checks. These contributions aim to improve the robustness and interpretability of Bayesian model criticism practices. We offer recommendations for appropriate visual predictive checks for observations that are: continuous, discrete, or a mixture of the two. We also discuss diagnostics to aid in the selection of visual metho

Long range, multi-qubit parity checks have applications in both quantum error correction and measurement-based entanglement generation. Such parity checks could be performed using qubit-state-dependent phase shifts on propagating pulses of light described by coherent states $\vertα\rangle$ of the electromagnetic field. We consider "flying-cat" parity checks based on an entangling operation that is quantum non-demolition (QND) for Schrödinger's cat states $\vertα\rangle\pm \vert-α\rangle$. This operation encodes parity information in the phase of maximally distinguishable coherent states $\vert\pm α\rangle$, which can be read out using a phase-sensitive measurement of the electromagnetic field. In contrast to many implementations, where single-qubit errors and measurement errors can be treated as independent, photon loss during flying-cat parity checks introduces errors on physical qubits at a rate that is anti-correlated with the probability for measurement errors. We analyze this trade-off for three-qubit parity checks, which are a requirement for universal fault-tolerant quantum computing with the subsystem surface code. We further show how a six-qubit entangled "tetrahedron" sta

We study linear codes that maximize minimum distance subject to arbitrary support constraints on the parity-check matrix. Such constraints arise naturally in the design of LDPC codes, locally repairable codes, and hardware-constrained systems where each parity check must involve only a limited number of code symbols. They are also essential in quantum error correction, where sparse stabilizers reduce measurement noise and respect the connectivity constraints of physical qubit architectures. We derive the optimal minimum distance possible given support constraints on the parity-check matrix and show it is achievable over sufficiently large fields. When this maximum distance coincides with the Singleton bound for unconstrained parity check matrices, the dual GM-MDS construction yields generalized Reed--Solomon codes obeying the mask. In the generator-matrix setting, the GM-MDS theorem guarantees that the optimal distance can always be achieved by a subcode of a generalized Reed--Solomon code while satisfying arbitrary support constraints. We show that this is not true for the parity-check setting. We exhibit a set of support constraints, derived from the vertex-edge incidence of $K_{

Recently, a lot of effort has been devoted towards designing erasure qubits in which dominant physical noise excites leakage states whose population can be detected and returned to the qubit subspace. Interest in these erasure qubits has been driven by studies showing that the requirements for fault-tolerant quantum error correction are significantly relaxed when noise in every gate operation is dominated by erasures. However, these studies assume perfectly accurate erasure checks after every gate operation which generally come with undesirable time and hardware overhead costs. In this work, we investigate the consequences of using an imperfect but overhead-efficient erasure check for fault-tolerant quantum error correction with the surface code. We show that, under physically reasonable assumptions on the imperfect erasure checks, the threshold error rate is still at least over twice that for Pauli noise. We also study the impact of imperfect erasure checks on the effective error distance and find that it degrades the effective distance under a general error model in which a qubit suffers from depolarizing noise when interacting with a leaked qubit. We then identify a more restric

Pauli Check Sandwiching (PCS) is an error detection scheme that protects quantum circuits by inserting pairs of parity checks and discarding runs that signal errors. However, each additional check introduces noise and exponentially increases sampling costs. To address these limitations, we propose Pauli Check Extrapolation (PCE), an error mitigation technique that obtains measured expectation values from circuits with different numbers of checks and, analogous to ZNE, extrapolates to the ``maximum check'' limit -- the theoretical number of checks required for unit fidelity. We test linear and exponential ansatzes, deriving the exponential form from the Markovian error model. Benchmarking PCE against ZNE on random Clifford circuits with simulated depolarizing noise shows PCE outperforming ZNE for larger circuits. On real IBM hardware, PCE achieves an accuracy of up to 99.2% (56.2% improvement over baseline), compared to ZNE's 82% accuracy (29.1% improvement over baseline), for 4-qubit circuits. To demonstrate a practical use case, we then apply PCE towards mitigating errors in classical shadow measurements. Our results show that PCE can achieve fidelities greater than the state-of-t

Not everything on the internet is true. This unfortunate fact requires both humans and models to perform complex reasoning about credibility when working with retrieved information. In NLP, this problem has seen little attention. Indeed, retrieval-augmented models are not typically expected to distrust retrieved documents. Human experts overcome the challenge by gathering signals about the context, reliability, and tendency of source documents - that is, they perform source criticism. We propose a novel NLP task focused on finding and summarising such signals. We introduce a new dataset of 6,709 "media background checks" derived from Media Bias / Fact Check, a volunteer-run website documenting media bias. We test open-source and closed-source LLM baselines with and without retrieval on this dataset, finding that retrieval greatly improves performance. We furthermore carry out human evaluation, demonstrating that 1) media background checks are helpful for humans, and 2) media background checks are helpful for retrieval-augmented models.

Visualizations play a critical role in validating and improving statistical models. However, the design space of model check visualizations is not well understood, making it difficult for authors to explore and specify effective graphical model checks. VMC defines a model check visualization using four components: (1) samples of distributions of checkable quantities generated from the model, including predictive distributions for new data and distributions of model parameters; (2) transformations on observed data to facilitate comparison; (3) visual representations of distributions; and (4) layouts to facilitate comparing model samples and observed data. We contribute an implementation of VMC as an R package. We validate VMC by reproducing a set of canonical model check examples, and show how using VMC to generate model checks reduces the edit distance between visualizations relative to existing visualization toolkits. The findings of an interview study with three expert modelers who used VMC highlight challenges and opportunities for encouraging exploration of correct, effective model check visualizations.

Explanations for machine learning models can be hard to interpret or be wrong. Combining an explanation method with an uncertainty estimation method produces explanation uncertainty. Evaluating explanation uncertainty is difficult. In this paper we propose sanity checks for uncertainty explanation methods, where a weight and data randomization tests are defined for explanations with uncertainty, allowing for quick tests to combinations of uncertainty and explanation methods. We experimentally show the validity and effectiveness of these tests on the CIFAR10 and California Housing datasets, noting that Ensembles seem to consistently pass both tests with Guided Backpropagation, Integrated Gradients, and LIME explanations.

Global navigation satellite systems (GNSSs) are implementing security mechanisms: examples are Galileo open service navigation message authentication (OS-NMA) and GPS chips-message robust authentication (CHIMERA). Each of these mechanisms operates in a single band. However, nowadays, even commercial GNSS receivers typically compute the position, velocity, and time (PVT) solution using multiple constellations and signals from multiple bands at once, significantly improving both accuracy and availability. Hence, cross-authentication checks have been proposed, based on the PVT obtained from the mixture of authenticated and non-authenticated signals. In this paper, first, we formalize the models for the cross-authentication checks. Next, we describe, for each check, a spoofing attack to generate a fake signal leading the victim to a target PVT without notice. We analytically relate the degrees of the freedom of the attacker in manipulating the victim's solution to both the employed security checks and the number of open signals that can be tampered with by the attacker. We test the performance of the considered attack strategies on an experimental dataset. Lastly, we show the limits of

Saliency methods are frequently used to explain Deep Neural Network-based models. Adebayo et al.'s work on evaluating saliency methods for classification models illustrate certain explanation methods fail the model and data randomization tests. However, on extending the tests for various state of the art object detectors we illustrate that the ability to explain a model is more dependent on the model itself than the explanation method. We perform sanity checks for object detection and define new qualitative criteria to evaluate the saliency explanations, both for object classification and bounding box decisions, using Guided Backpropagation, Integrated Gradients, and their Smoothgrad versions, together with Faster R-CNN, SSD, and EfficientDet-D0, trained on COCO. In addition, the sensitivity of the explanation method to model parameters and data labels varies class-wise motivating to perform the sanity checks for each class. We find that EfficientDet-D0 is the most interpretable method independent of the saliency method, which passes the sanity checks with little problems.