搜索结果：Effectiveness

Sequential multiple assignment randomized trials (SMARTs) have grown in popularity in recent years, and many of their study protocols propose conducting a cost effectiveness analysis of the adaptive strategies embedded within them. The cost effectiveness of these regimes is often proposed to be assessed using incremental cost effectiveness ratios (ICERs). In this paper, we present an estimation and inference procedure for such cost effectiveness measures for the embedded dynamic treatment regimes within a SMART design. In particular, we describe a targeted maximum likelihood estimator for the ICER of a SMART's embedded regimes with influence curve-based inference. We illustrate the performance of these methods using simulations. Throughout, we use as illustration a cost effectiveness analysis for the Adaptive Strategies for Preventing and Treating Lapses of Retention in HIV Care (ADAPT-R; NCT02338739) trial, presenting estimated ICERs (with inference) for embedded regimes aimed at increasing HIV care adherence. This manuscript is one of the first to present cost effectiveness analysis results from a SMART.

Robustness and Effectiveness are critical aspects of developing dense retrieval models for real-world applications. It is known that there is a trade-off between the two. Recent work has addressed scaling laws of effectiveness in dense retrieval, revealing a power-law relationship between effectiveness and the size of models and data. Does robustness follow scaling laws too? If so, can scaling improve both robustness and effectiveness together, or do they remain locked in a trade-off? To answer these questions, we conduct a comprehensive experimental study. We find that:(i) Robustness, including out-of-distribution and adversarial robustness, also follows a scaling law.(ii) Robustness and effectiveness exhibit different scaling patterns, leading to significant resource costs when jointly improving both. Given these findings, we shift to the third factor that affects model performance, namely the optimization strategy, beyond the model size and data size. We find that: (i) By fitting different optimization strategies, the joint performance of robustness and effectiveness traces out a Pareto frontier. (ii) When the optimization strategy strays from Pareto efficiency, the joint perfor

In this paper, we provide an overview and evaluation of different types of age assurance technologies (AAT). We describe and analyse 1) different approaches to age assurance online (age verification, age estimation, age inference, and parental control and consent), as well as 2) different age assurance architectures (online, offline device-based, offline credential-based), and assess their various combinations with regards to their respective a) effectiveness, b) side effects, and c) acceptance. We then discuss general limitations of AAT's effectiveness stemming from the possibility of circumvention and outline the most important side effects, in particular regarding privacy and anonymity of all users; bias, discrimination, and exclusion; as well as censorship and related concerns. We conclude our analyses by offering some recommendations on which types of AAT are better or less suited to protect minors online. Guiding our assessment is a weighing of effectiveness against side effects, resulting in a graduated hierarchy of acceptable AAT mechanisms.

Benchmark collections have long enabled controlled comparison and cumulative progress in Information Retrieval (IR). However, prior meta-analyses have shown that reported effectiveness gains often fail to accumulate, in part due to the use of weak or outdated baselines. While large language models are increasingly used in retrieval pipelines, their impact on established IR benchmarks has not been systematically analyzed. In this study, we analyze 143 publications reporting results on the TREC Robust04 collection and the TREC Deep Learning 2020 (DL20) passage retrieval benchmark to examine longitudinal trends in retrieval effectiveness and baseline strength. We observe what we term an \emph{LLM effect}: recent systems incorporating LLM components achieve 8.8\% higher nDCG@10 on DL20 compared to the best result from TREC 2020 and approximately 20\% higher on Robust04 since 2023. However, adapting a data contamination detection approach to reranking reveals measurable contamination in both benchmarks. While excluding contaminated topics reduces effectiveness, confidence intervals remain wide, making it difficult to determine whether the LLM effect reflects genuine methodological advan

The coupon incentive is one of the most common tools marketers use to court users to engage with a business at various stages of the customer life cycle. A variety of factors can affect the effectiveness of a coupon incentive on users, timing being one of them. We hypothesize that coupons can be more effective when delivered at critical times in the customer journey, right when a user is engaging with the platform. Verifying such a hypothesis would typically require real time event-triggered coupon distribution software that may be too expensive to implement. In this paper, we propose a framework in which we apply causal inference on "natural randomized control trial experiments" to measure the effectiveness of sending coupons at the right time to users without requiring a dedicated AB test. We demonstrate the usefulness of our framework in the case of a user onboarding coupon campaign held in our company and show how the results can lead to correct data-driven decisions for the business. Furthermore, in order to test the generalizability of our framework, and to make our research more reproducible, we apply our framework on a user retention campaign with a publicly available datas

Numerous studies have investigated the underlying mechanisms of in-context learning (ICL) effectiveness to inspire the design of related methods. However, existing work predominantly assumes the effectiveness of the demonstrations provided within ICL, while many research indicates that not all demonstrations are effective, failing to yielding any performance improvement during ICL. Therefore, in this paper, we investigate the reasons behind demonstration ineffectiveness. Our analysis is based on gradient flow and linear self-attention models. By setting the gradient flow to zero, we deduce that a demonstration becomes ineffective if its information has either been learned by the model or is irrelevant to the user query. Furthermore, we demonstrate that in multi-layer models, the disparity in effectiveness among demonstrations is amplified with layer increasing, causing the model to focus more on effective ones. Considering that current demonstration selection methods primarily focus on the relevance to the user query while overlooking the information that the model has already assimilated, we propose a novel method called GradS, which leverages gradient flow for demonstration selec

Open science is increasingly recognised worldwide, with preprint posting emerging as a key strategy. This study explores the factors influencing researchers' adoption of preprint publication, particularly the perceived effectiveness of this practice and research intensity indicators such as publication and review frequency. Using open data from a comprehensive survey with 5,873 valid responses, we conducted regression analyses to control for demographic variables. Researchers' productivity, particularly the number of journal articles and books published, greatly influences the frequency of preprint deposits. The perception of the effectiveness of preprints follows this. Preprints are viewed positively in terms of early access to new research, but negatively in terms of early feedback. Demographic variables, such as gender and the type of organisation conducting the research, do not have a significant impact on the production of preprints when other factors are controlled for. However, the researcher's discipline, years of experience and geographical region generally have a moderate effect on the production of preprints. These findings highlight the motivations and barriers associat

This study compared the effectiveness of COVID-19 control policies, including wearing masks, and the vaccine rates through proportional infection rate in 28 states of the United States using the eSIR model. The effective rate of policies was measured by the difference between the predicted daily infection proportion rate using the data before the policy and the actual daily infection proportion rate. The study suggests that both mask and vaccine policy had a significant impact on mitigating the pandemic. We further explored how different social factors influenced the effectiveness of a specific policy through the linear regression model. Out of 9 factors, the population density, number of hospital beds per 1000 people, and percent of the population over 65 are the most substantial factors on mask policy effectiveness, while public health funding per person, percent of immigration have the most significant influence on vaccine policy effectiveness. This study summarized the effectiveness of different policies and factors they associated with. It can be served as a reference for future covid-19 related policy.

The field of Quantum Computing has gathered significant popularity in recent years and a large number of papers have studied its effectiveness in tackling many tasks. We focus in particular on Quantum Annealing (QA), a meta-heuristic solver for Quadratic Unconstrained Binary Optimization (QUBO) problems. It is known that the effectiveness of QA is dependent on the task itself, as is the case for classical solvers, but there is not yet a clear understanding of which are the characteristics of a problem that makes it difficult to solve with QA. In this work, we propose a new methodology to study the effectiveness of QA based on meta-learning models. To do so, we first build a dataset composed of more than five thousand instances of ten different optimization problems. We define a set of more than a hundred features to describe their characteristics, and solve them with both QA and three classical solvers. We publish this dataset online for future research. Then, we train multiple meta-models to predict whether QA would solve that instance effectively and use them to probe which are the features with the strongest impact on the effectiveness of QA. Our results indicate that it is poss

Generative patent language models can assist humans to write patent text more effectively. The question is how to measure effectiveness from a human-centric perspective and how to improve effectiveness. In this manuscript, a simplified design of the autocomplete function is proposed to increase effectiveness by more than 10%. With the new design, the effectiveness of autocomplete can reach more than 60%, which means that more than 60% of keystrokes can be saved by autocomplete. Since writing patent text does not necessarily start from the beginning to the end, a question is whether the generative model can assist a user no matter where to start writing. To answer the question, the generative models in this manuscript are pre-trained with training data in both directions. The generative models become bidirectional. Since text generation is bidirectional, the calculation of autocomplete effectiveness can be bidirectional and starts from anywhere in the text. After thorough experiments, a key finding is that the autocomplete effectiveness of a model for the same text remains similar no matter where the calculation starts. The finding indicates that such bidirectional models can assist

Context. Nowadays there is a great deal of uncertainty surrounding the effects of experience on Requirements Engineering (RE). There is a widespread idea that experience improves analyst performance. However, there are empirical studies that demonstrate the exact opposite. Aim. Determine whether experience influences requirements analyst performance. Method. Quasi-experiments run with students and professionals. The experimental task was to elicit requirements using the open interview technique immediately followed by the consolidation of the elicited information in domains with which the analysts were and were not familiar. Results. In unfamiliar domains, interview, requirements, development, and professional experience does not influence analyst effectiveness. In familiar domains, effectiveness varies depending on the type of experience. Interview experience has a strong positive effect, whereas professional experience has a moderate negative effect. Requirements experience appears to have a moderately positive effect; however, the statistical power of the analysis is insufficient to be able to confirm this point. Development experience has no effect either way. Conclusion. Exper

Action detection and understanding provide the foundation for the generation and interaction of multimedia content. However, existing methods mainly focus on constructing complex relational inference networks, overlooking the judgment of detection effectiveness. Moreover, these methods frequently generate detection results with cognitive abnormalities. To solve the above problems, this study proposes a cognitive effectiveness network based on fuzzy inference (Cefdet), which introduces the concept of "cognition-based detection" to simulate human cognition. First, a fuzzy-driven cognitive effectiveness evaluation module (FCM) is established to introduce fuzzy inference into action detection. FCM is combined with human action features to simulate the cognition-based detection process, which clearly locates the position of frames with cognitive abnormalities. Then, a fuzzy cognitive update strategy (FCS) is proposed based on the FCM, which utilizes fuzzy logic to re-detect the cognition-based detection results and effectively update the results with cognitive abnormalities. Experimental results demonstrate that Cefdet exhibits superior performance against several mainstream algorithms

Autoregressive transformers exhibit adaptive learning through in-context learning (ICL), which begs the question of how. Prior works have shown that transformers represent the ICL tasks as vectors in their representations. In this paper, we leverage the encoding-decoding framework to study how transformers form task vectors during pretraining and how their task encoding quality predicts ICL task performance. On synthetic ICL tasks, we analyze the training dynamics of a small transformer and report the coupled emergence of task encoding and decoding. As the model learns to encode different latent tasks (e.g., "Finding the first noun in a sentence.") into distinct, separable representations, it concurrently builds conditional decoding algorithms and improves its ICL performance. We validate this phenomenon across pretrained models of varying scales (Gemma-2 2B/9B/27B, Llama-3.1 8B/70B) and over the course of pretraining in OLMo-7B. Further, we demonstrate that the quality of task encoding inferred from representations predicts ICL performance, and that, surprisingly, finetuning the earlier layers can improve the task encoding and performance more than finetuning the latter layers. Ou

Session-based recommendation predicts users' future interests from previous interactions in a session. Despite the memorizing of historical samples, the request of unlearning, i.e., to remove the effect of certain training samples, also occurs for reasons such as user privacy or model fidelity. However, existing studies on unlearning are not tailored for the session-based recommendation. On the one hand, these approaches cannot achieve satisfying unlearning effects due to the collaborative correlations and sequential connections between the unlearning item and the remaining items in the session. On the other hand, seldom work has conducted the research to verify the unlearning effectiveness in the session-based recommendation scenario. In this paper, we propose SRU, a session-based recommendation unlearning framework, which enables high unlearning efficiency, accurate recommendation performance, and improved unlearning effectiveness in session-based recommendation. Specifically, we first partition the training sessions into separate sub-models according to the similarity across the sessions, then we utilize an attention-based aggregation layer to fuse the hidden states according to

A primary goal of remote collaboration tools is to provide effective and inclusive meetings for all participants. To study meeting effectiveness and meeting inclusiveness, we first conducted a large-scale email survey (N=4,425; after filtering N=3,290) at a large technology company (pre-COVID-19); using this data we derived a multivariate model of meeting effectiveness and show how it correlates with meeting inclusiveness, participation, and feeling comfortable to contribute. We believe this is the first such model of meeting effectiveness and inclusiveness. The large size of the data provided the opportunity to analyze correlations that are specific to sub-populations such as the impact of video. The model shows the following factors are correlated with inclusiveness, effectiveness, participation, and feeling comfortable to contribute in meetings: sending a pre-meeting communication, sending a post-meeting summary, including a meeting agenda, attendee location, remote-only meeting, audio/video quality and reliability, video usage, and meeting size. The model and survey results give a quantitative understanding of how and where to improve meeting effectiveness and inclusiveness and

In 2017, GitHub was the first online open source platform to show security alerts to its users. It has since introduced further security interventions to help developers improve the security of their open source software. In this study, we investigate and compare the effects of these interventions. This offers a valuable empirical perspective on security interventions in the context of software development, enriching the predominantly qualitative and survey-based literature landscape with substantial data-driven insights. We conduct a time series analysis on security-altering commits covering the entire history of a large-scale sample of over 50,000 GitHub repositories to infer the causal effects of the security alert, security update, and code scanning interventions. Our analysis shows that while all of GitHub's security interventions have a significant positive effect on security, they differ greatly in their effect size. By comparing the design of each intervention, we identify the building blocks that worked well and those that did not. We also provide recommendations on how practitioners can improve the design of their interventions to enhance their effectiveness.

Any program that is designed to accomplish certain objectives, needs to establish program level controls pertaining to the overall goal. A critical aspect that determines the success of a program is the quality of the controls and their effectiveness in accomplishing the goal. Traditional Control Maturity Models primarily focus on the efficiency, management, and optimization of controls, while only indirectly measuring control effectiveness which neglects an essential aspect of control efficacy. In this paper, we highlight the ineffectiveness of these models, outline an adaptable program controls framework, and provide an approach to define measurable attributes (metrics) that enable a zero-defect program. To the best of our knowledge, we believe this is the first paper that provides a structured approach to defining a controls measurement framework and creation of effectiveness metrics that can be adopted by a variety of program use cases.

Honeypots are a deceptive technology used to capture malicious activity. The technology is useful for studying attacker behavior, tools, and techniques but can be difficult to implement and maintain. Historically, a lack of measures of effectiveness prevented researchers from assessing honeypot implementations. The consequence being ineffective implementations leading to poor performance, flawed imitation of legitimate services, and premature discovery by attackers. Previously, we developed a taxonomy for measures of effectiveness in dynamic honeypot implementations. The measures quantify a dynamic honeypot's effectiveness in fingerprinting its environment, capturing valid data from adversaries, deceiving adversaries, and intelligently monitoring itself and its surroundings. As a step towards developing automated effectiveness testing, this work introduces a tool for priming a target honeypot for evaluation. We outline the design of the tool and provide results in the form of quantitative calibration data.