During the training of large Transformer models, attention masks regulate the scope and direction of information flow across a sequence. Numerous mask variants exist, and operators such as FlexAttention already support arbitrary attention masks. Nevertheless, a systematic formal analysis of the information-flow structure induced by arbitrary masks has been missing. This paper develops a complete theoretical framework. We prove that, with sufficient depth, the information flow of a multi-layer Transformer converges to a Hasse diagram -- a directed acyclic graph representing a partial order. Building on this, we recast the design of parallel training tasks as the problem of finding a minimal common supergraph of Hasse diagrams, and we establish a criterion for the minimal common supergraph. This yields a constructive method to derive attention masks directly from a family of tasks. Applying the framework, we design two novel masks: a block-generation attention mask that ensures training-inference consistency (Block Two-Stream Attention), and a fully supervised bidirectional attention mask (Butterfly Attention). These results demonstrate the framework's capacity to discover new struct
Recent open-vocabulary segmentation methods adopt mask generators to predict segmentation masks and leverage pre-trained vision-language models, e.g., CLIP, to classify these masks via mask pooling. Although these approaches show promising results, it is counterintuitive that accurate masks often fail to yield accurate classification results through pooling CLIP image embeddings within the mask regions. In this paper, we reveal the performance limitations of mask pooling and introduce Mask-Adapter, a simple yet effective method to address these challenges in open-vocabulary segmentation. Compared to directly using proposal masks, our proposed Mask-Adapter extracts semantic activation maps from proposal masks, providing richer contextual information and ensuring alignment between masks and CLIP. Additionally, we propose a mask consistency loss that encourages proposal masks with similar IoUs to obtain similar CLIP embeddings to enhance models' robustness to varying predicted masks. Mask-Adapter integrates seamlessly into open-vocabulary segmentation methods based on mask pooling in a plug-and-play manner, delivering more accurate classification results. Extensive experiments across
Snapshot compressive imaging (SCI) refers to the recovery of three-dimensional data cubes-such as videos or hyperspectral images-from their two-dimensional projections, which are generated by a special encoding of the data with a mask. SCI systems commonly use binary-valued masks that follow certain physical constraints. Optimizing these masks subject to these constraints is expected to improve system performance. However, prior theoretical work on SCI systems focuses solely on independently and identically distributed (i.i.d.) Gaussian masks, which do not permit such optimization. On the other hand, existing practical mask optimizations rely on computationally intensive joint optimizations that provide limited insight into the role of masks and are expected to be sub-optimal due to the non-convexity and complexity of the optimization. In this paper, we analytically characterize the performance of SCI systems employing binary masks and leverage our analysis to optimize hardware parameters. Our findings provide a comprehensive and fundamental understanding of the role of binary masks - with both independent and dependent elements - and their optimization. We also present simulation
Snapshot compressive imaging (SCI) systems have gained significant attention in recent years. While previous theoretical studies have primarily focused on the performance analysis of Gaussian masks, practical SCI systems often employ binary-valued masks. Furthermore, recent research has demonstrated that optimized binary masks can significantly enhance system performance. In this paper, we present a comprehensive theoretical characterization of binary masks and their impact on SCI system performance. Initially, we investigate the scenario where the masks are binary and independently identically distributed (iid), revealing a noteworthy finding that aligns with prior numerical results. Specifically, we show that the optimal probability of non-zero elements in the masks is smaller than 0.5. This result provides valuable insights into the design and optimization of binary masks for SCI systems, facilitating further advancements in the field. Additionally, we extend our analysis to characterize the performance of SCI systems where the mask entries are not independent but are generated based on a stationary first-order Markov process. Overall, our theoretical framework offers a comprehe
Transformers are widely used across various applications, many of which yield sparse or partially filled attention matrices. Examples include attention masks designed to reduce the quadratic complexity of attention, sequence packing techniques, and recent innovations like tree masking for fast validation in MEDUSA. Despite the inherent sparsity in these matrices, the state-of-the-art algorithm Flash Attention still processes them with quadratic complexity as though they were dense. In this paper, we introduce Binary Block Masking, a highly efficient modification that enhances Flash Attention by making it mask-aware. We further propose two optimizations: one tailored for masks with contiguous non-zero patterns and another for extremely sparse masks. Our experiments on attention masks derived from real-world scenarios demonstrate up to a 9x runtime improvement. The implementation will be publicly released to foster further research and application.
We introduce a new task, Referring and Reasoning for Selective Masks (R2SM), which extends text-guided segmentation by incorporating mask-type selection driven by user intent. This task challenges vision-language models to determine whether to generate a modal (visible) or amodal (complete) segmentation mask based solely on natural language prompts. To support the R2SM task, we present the R2SM dataset, constructed by augmenting annotations of COCOA-cls, D2SA, and MUVA. The R2SM dataset consists of both modal and amodal text queries, each paired with the corresponding ground-truth mask, enabling model finetuning and evaluation for the ability to segment images as per user intent. Specifically, the task requires the model to interpret whether a given prompt refers to only the visible part of an object or to its complete shape, including occluded regions, and then produce the appropriate segmentation. For example, if a prompt explicitly requests the whole shape of a partially hidden object, the model is expected to output an amodal mask that completes the occluded parts. In contrast, prompts without explicit mention of hidden regions should generate standard modal masks. The R2SM ben
Face masks muffle speech and make communication more difficult, especially for people with hearing loss. This study examines the acoustic attenuation caused by different face masks, including medical, cloth, and transparent masks, using a head-shaped loudspeaker and a live human talker. The results suggest that all masks attenuate frequencies above 1 kHz, that attenuation is greatest in front of the talker, and that there is substantial variation between mask types, especially cloth masks with different materials and weaves. Transparent masks have poor acoustic performance compared to both medical and cloth masks. Most masks have little effect on lapel microphones, suggesting that existing sound reinforcement and assistive listening systems may be effective for verbal communication with masks.
We present a comparative study of the manufacture of binary pupil masks for coronagraphic observations of exoplanets. A checkerboard mask design, a type of binary pupil mask design, was adopted, and identical patterns of the same size were used for all the masks in order that we could compare the differences resulting from the different manufacturing methods. The masks on substrates had aluminum checkerboard patterns with thicknesses of 0.1/0.2/0.4/0.8/1.6$μ$m constructed on substrates of BK7 glass, silicon, and germanium using photolithography and chemical processes. Free-standing masks made of copper and nickel with thicknesses of 2/5/10/20$μ$m were also realized using photolithography and chemical processes, which included careful release from the substrate used as an intermediate step in the manufacture. Coronagraphic experiments using a visible laser were carried out for all the masks on BK7 glass substrate and the free-standing masks. The average contrasts were 8.4$\times10^{-8}$, 1.2$\times10^{-7}$, and 1.2$\times10^{-7}$ for the masks on BK7 substrates, the free-standing copper masks, and the free-standing nickel masks, respectively. No significant correlation was concluded
We focus on addressing the dense backward propagation issue for training efficiency of N:M fine-grained sparsity that preserves at most N out of M consecutive weights and achieves practical speedups supported by the N:M sparse tensor core. Therefore, we present a novel method of Bi-directional Masks (Bi-Mask) with its two central innovations in: 1) Separate sparse masks in the two directions of forward and backward propagation to obtain training acceleration. It disentangles the forward and backward weight sparsity and overcomes the very dense gradient computation. 2) An efficient weight row permutation method to maintain performance. It picks up the permutation candidate with the most eligible N:M weight blocks in the backward to minimize the gradient gap between traditional uni-directional masks and our bi-directional masks. Compared with existing uni-directional scenario that applies a transposable mask and enables backward acceleration, our Bi-Mask is experimentally demonstrated to be more superior in performance. Also, our Bi-Mask performs on par with or even better than methods that fail to achieve backward acceleration. Project of this paper is available at \url{https://gith
Face masks have been widely used as a protective measure against COVID-19. However, pre-pandemic empirical studies have produced mixed statistical results on the effectiveness of masks against respiratory viruses. The implications of the studies' recognized limitations have not been quantitatively and statistically analyzed, leading to confusion regarding the effectiveness of masks. Such confusion may have contributed to organizations such as the WHO and CDC initially not recommending that the general public wear masks. Here we show that when the adherence to mask-usage guidelines is taken into account, the empirical evidence indicates that masks prevent disease transmission: all studies we analyzed that did not find surgical masks to be effective were under-powered to such an extent that even if masks were 100% effective, the studies in question would still have been unlikely to find a statistically significant effect. We also provide a framework for understanding the effect of masks on the probability of infection for single and repeated exposures. The framework demonstrates that more frequently wearing a mask provides super-linearly compounding protection, as does both the susce
While most masks have a limited effect on personal protection, how effective are they for collective protection? How to enlighten the design of masks from the perspective of collective dynamics? In this paper, we assume three preferences in the population: (i) never wearing a mask; (ii) wearing a mask if and only if infected; (iii) always wearing a mask. We study the epidemic transmission in an open system within the Susceptible-Infected-Recovered (SIR) model framework. We use agent-based Monte Carlo simulation and mean-field differential equations to investigate the model, respectively. Ternary heat maps show that wearing masks is always beneficial in curbing the spread of the epidemic. Based on the model, we investigate the potential implications of different mask designs from the perspective of collective dynamics. The results show that strengthening the filterability of the mask from the face to the outside is more effective in most parameter spaces, because it acts on individuals with both preferences (ii) and (iii). However, when the fraction of individuals always wearing a mask achieves a critical point, strengthening the filterability from outside to the face becomes more e
Phase retrieval with pre-defined optical masks can provide extra constraint and thus achieve improved performance. The recent progress in optimization theory demonstrates the superiority of random masks in phase retrieval algorithms. However, traditional approaches just focus on the randomness of the masks but ignore their non-bandlimited nature. When using these masks in the reconstruction process for phase retrieval, the high frequency part of the masks is often removed in the process and thus leads to degraded performance. Based on the concept of digital halftoning, this paper proposes a green noise binary masking scheme which can greatly reduce the high frequency content of the masks while fulfilling the randomness requirement. The experimental results show that the proposed green noise binary masking scheme outperform the traditional ones when using in a DMD-based coded diffraction pattern phase retrieval system.
Increasing spatial image resolution is an often required, yet challenging task in image acquisition. Recently, it has been shown that it is possible to obtain a high resolution image by covering a low resolution sensor with a non-regular sampling mask. Due to the masking, however, some pixel information in the resulting high resolution image is not available and has to be reconstructed by an efficient image reconstruction algorithm in order to get a fully reconstructed high resolution image. In this paper, the influence of different sampling masks with a reduced randomness of the non-regularity on the image reconstruction process is evaluated. Simulation results show that it is sufficient to use sampling masks that are non-regular only on a smaller scale. These sampling masks lead to a visually noticeable gain in PSNR compared to arbitrary chosen sampling masks which are non-regular over the whole image sensor size. At the same time, they simplify the manufacturing process and allow for efficient storage.
The use of facial masks in public spaces has become a social obligation since the wake of the COVID-19 global pandemic and the identification of facial masks can be imperative to ensure public safety. Detection of facial masks in video footages is a challenging task primarily due to the fact that the masks themselves behave as occlusions to face detection algorithms due to the absence of facial landmarks in the masked regions. In this work, we propose an approach for detecting facial masks in videos using deep learning. The proposed framework capitalizes on the MTCNN face detection model to identify the faces and their corresponding facial landmarks present in the video frame. These facial images and cues are then processed by a neoteric classifier that utilises the MobileNetV2 architecture as an object detector for identifying masked regions. The proposed framework was tested on a dataset which is a collection of videos capturing the movement of people in public spaces while complying with COVID-19 safety protocols. The proposed methodology demonstrated its effectiveness in detecting facial masks by achieving high precision, recall, and accuracy.
Joint synthesis of images and segmentation masks with generative adversarial networks (GANs) is promising to reduce the effort needed for collecting image data with pixel-wise annotations. However, to learn high-fidelity image-mask synthesis, existing GAN approaches first need a pre-training phase requiring large amounts of image data, which limits their utilization in restricted image domains. In this work, we take a step to reduce this limitation, introducing the task of one-shot image-mask synthesis. We aim to generate diverse images and their segmentation masks given only a single labelled example, and assuming, contrary to previous models, no access to any pre-training data. To this end, inspired by the recent architectural developments of single-image GANs, we introduce our OSMIS model which enables the synthesis of segmentation masks that are precisely aligned to the generated images in the one-shot regime. Besides achieving the high fidelity of generated masks, OSMIS outperforms state-of-the-art single-image GAN models in image synthesis quality and diversity. In addition, despite not using any additional data, OSMIS demonstrates an impressive ability to serve as a source o
Recently, we introduced a class of shaped pupil masks, called spiderweb masks, that produce point spread functions having annular dark zones. With such masks, a single image can be used to probe a star for extrasolar planets. In this paper, we introduce a new class of shaped pupil masks that also provide annular dark zones. We call these masks starshaped masks. Given any circularly symmetric apodization function, we show how to construct a corresponding starshaped mask that has the same point-spread function (out to any given outer working distance) as obtained by the apodization.
The introduction of face masks during COVID-19 presents a potential challenge for human face perception and recognition. Face masks possibly hinder the holistic processing of faces leading to difficulty in facial recognition. Our present study aims to investigate this issue by probing the neuropsychological mechanisms of face recognition, while also exploring a possible learning effect observed in regularly seen (personally familiar) masked faces. Our stimuli consisted of personally familiar, famous, and unfamiliar Indian faces in masked and unmasked conditions. Subjects participated in a 2-back task wherein trials were balanced within and across blocks to represent all conditions identically, while behavioral and EEG responses were recorded. Statistical analyses revealed significant main effects of familiarity and mask-conditions on performance accuracy and reaction time (RT). The highest performance accuracy was observed for familiar and unmasked faces and the least for unfamiliar and masked ones, whereas RTs followed expected reverse trends. Notably, the difference in performance accuracy between unmasked and masked famous faces was more prominent than that for personally famili
Remote photoplethysmography (rPPG), a family of techniques for monitoring blood volume changes, may be especially useful for widespread contactless health monitoring using face video from consumer-grade visible-light cameras. The COVID-19 pandemic has caused the widespread use of protective face masks. We found that occlusions from cloth face masks increased the mean absolute error of heart rate estimation by more than 80\% when deploying methods designed on unmasked faces. We show that augmenting unmasked face videos by adding patterned synthetic face masks forces the model to attend to the periocular and forehead regions, improving performance and closing the gap between masked and unmasked pulse estimation. To our knowledge, this paper is the first to analyse the impact of face masks on the accuracy of pulse estimation and offers several novel contributions: (a) 3D CNN-based method designed for remote photoplethysmography in a presence of face masks, (b) two publicly available pulse estimation datasets acquired from 86 unmasked and 61 masked subjects, (c) evaluations of handcrafted algorithms and a 3D CNN trained on videos of unmasked faces and with masks synthetically added, an
During a pandemic in which aerosol and droplet transmission is possible, the demand for masks that meet medical or workplace standards can prevent most individuals or organizations from obtaining suitable protection. Cloth masks are widely believed to impede droplet and aerosol transmission but most are constructed from materials with unknown filtration efficiency, airflow resistance and water resistance. Further, there has been no clear guidance on the most important performance metrics for the materials used by the general public (as opposed to high-risk healthcare settings). Here we provide data on a range of common fabrics that might be used to construct masks. None of the materials were suitable for masks meeting the N95 NIOSH standard, but many could provide useful filtration (>90%) of 3 micron particles (a plausible challenge size for human generated aerosols), with low pressure drop. These were: nonwoven sterile wraps, dried baby wipes and some double-knit cotton materials. Decontamination of N95 masks using isopropyl alcohol produces the expected increase in particle penetration, but for 3 micron particles, filtration efficiency is still well above 95%. Tightly woven th
This paper proposes a mask optimization method for improving the quality of object removal using image inpainting. While many inpainting methods are trained with a set of random masks, a target for inpainting may be an object, such as a person, in many realistic scenarios. This domain gap between masks in training and inference images increases the difficulty of the inpainting task. In our method, this domain gap is resolved by training the inpainting network with object masks extracted by segmentation, and such object masks are also used in the inference step. Furthermore, to optimize the object masks for inpainting, the segmentation network is connected to the inpainting network and end-to-end trained to improve the inpainting performance. The effect of this end-to-end training is further enhanced by our mask expansion loss for achieving the trade-off between large and small masks. Experimental results demonstrate the effectiveness of our method for better object removal using image inpainting.