Multimodal Large Language Models (MLLMs) suffer from hallucinations. Existing hallucination evaluation benchmarks are often limited by over-simplified tasks leading to saturated metrics, or insufficient diversity that fails to adequately assess the hallucination extent in state-of-the-art multimodal models. To address this gap, we propose FREAK, a comprehensive multimodal benchmark designed for fine-grained hallucination assessment in MLLMs. Through high-quality photorealistic images featuring fine-grained counter-commonsense edits, FREAK innovatively evaluates hallucination phenomena in detailed visual perception of MLLMs. Extensive experiments on FREAK show severe hallucination issues in SOTA models regarding detailed visual perception. To enable deeper investigation, we curate a controlled subset to indirectly evaluate the model's ability to perceive target detailed information. Through systematic evaluation of prevailing Chain-of-Thought (CoT) prompting techniques within this task, we reveal critical insights regarding hallucination patterns and model reasoning processes.
Achieving high-fidelity lip-speech synchronization in audio-driven talking portrait synthesis remains challenging. While multi-stage pipelines or diffusion models yield high-quality results, they suffer from high computational costs. Some approaches perform well on specific individuals with low resources, yet still exhibit mismatched lip movements. The aforementioned methods are modeled in the pixel domain. We observed that there are noticeable discrepancies in the frequency domain between the synthesized talking videos and natural videos. Currently, no research on talking portrait synthesis has considered this aspect. To address this, we propose a FREquency-modulated, high-fidelity, and real-time Audio-driven talKing portrait synthesis framework, named FREAK, which models talking portraits from the frequency domain perspective, enhancing the fidelity and naturalness of the synthesized portraits. FREAK introduces two novel frequency-based modules: 1) the Visual Encoding Frequency Modulator (VEFM) to couple multi-scale visual features in the frequency domain, better preserving visual frequency information and reducing the gap in the frequency spectrum between synthesized and natural
The purpose of this short note is to explain how the existing results on the validity of the NLS approximation can be extended from Sobolev spaces $H^s(\mathbb{R})$ to the spaces of functions $u = v + w$ where $v \in H_{per}^s$ and $w \in H^s(\mathbb{R})$. This allows us to use the Peregrine solution of the NLS equation to find freak or rogue wave dynamics in more complicated systems.
Waves propagating through weakly disordered smooth linear media undergo a universal phenomenon called branched flow. Branched flow has been observed and studied experimentally in various systems by considering coherent waves. Recent experiments have reported the observation of optical branched flow by using an incoherent light source, thus revealing the key role of coherent phase-sensitive effects in the development of incoherent branched flow. By considering the paraxial wave equation as a generic representative model, we elaborate a stochastic theory of both coherent and incoherent branched flow. We derive closed-form equations that determine the evolution of the intensity correlation function, as well as the value and the propagation distance of the maximum of the scintillation index, which characterize the dynamical formation of incoherent branched flow. We report accurate numerical simulations that are found in quantitative agreement with the theory without free parameters. Our theory highlights the important impact of coherence and interference on branched flow, thereby providing a framework for exploring branched flow in nonlinear media, in relation with the formation of fre
Fisheye lens, which is suitable for panoramic imaging, has the prominent advantage of a large field of view and low cost. However, the fisheye image has a severe geometric distortion which may interfere with the stage of image registration and stitching. Aiming to resolve this drawback, we devise a stronger stitching algorithm for fisheye images by combining the traditional image processing method with deep learning. In the stage of fisheye image correction, we propose the Attention-based Nonlinear Activation Free Network (ANAFNet) to deblur fisheye images corrected by Zhang calibration method. Specifically, ANAFNet adopts the classical single-stage U-shaped architecture based on convolutional neural networks with soft-attention technique and it can restore a sharp image from a blurred image effectively. In the part of image registration, we propose the ORB-FREAK-GMS (OFG), a comprehensive image matching algorithm, to improve the accuracy of image registration. Experimental results demonstrate that panoramic images of superior quality stitching by fisheye images can be obtained through our method.
In this paper we investigate the frequency sensitivity of Deep Neural Networks (DNNs) when presented with clean samples versus poisoned samples. Our analysis shows significant disparities in frequency sensitivity between these two types of samples. Building on these findings, we propose FREAK, a frequency-based poisoned sample detection algorithm that is simple yet effective. Our experimental results demonstrate the efficacy of FREAK not only against frequency backdoor attacks but also against some spatial attacks. Our work is just the first step in leveraging these insights. We believe that our analysis and proposed defense mechanism will provide a foundation for future research and development of backdoor defenses.
The ``ugly duckling'' of the Segré-Plebański-Hawking-Ellis classification of stress-energy tensors is believed to be either impossible or extremely difficult to realise in Einstein gravity. Effective stress-energy tensors in alternative gravity offer a wider range of possibilities. We report a class of type III realisations in ``first-generation'' scalar-tensor and in Horndeski gravity, and their physical interpretation. The ugly duckling may be a freak of nature of limited importance but it is not physically impossible.
Pair soliton interactions play a significant role in the dynamics of soliton turbulence. The interaction of solitons with different polarities is particularly crucial in the context of abnormally large wave formation, often referred to as freak or rogue waves, as these interactions result in an increase in the maximum wave field. In this article, we investigate the features and properties of bipolar soliton interactions within the framework of the non-integrable Schamel equation, contrasting them with the integrable modified Korteweg-de Vries equation. We examine variations in moments and extrema of the wave fields. Additionally, we identify scenarios in which, in the bipolar solitary wave interaction, the smaller solitary wave transfers a portion of its energy to the larger one, causing an increase in the amplitude of the larger solitary wave and a decrease in the amplitude of the smaller one, returning them to their pre-interaction state. Notably, we observe that non-integrability can be considered a factor that triggers the formation of rogue waves.
In this paper, we propose a simple way to utilize stereo camera data to improve feature descriptors. Computer vision algorithms that use a stereo camera require some calculations of 3D information. We leverage this pre-calculated information to improve feature descriptor algorithms. We use the 3D feature information to estimate the scale of each feature. This way, each feature descriptor will be more robust to scale change without significant computations. In addition, we use stereo images to construct the descriptor vector. The Scale-Invariant Feature Transform (SIFT) and Fast Retina Keypoint (FREAK) descriptors are used to evaluate the proposed method. The scale normalization technique in feature tracking test improves the standard SIFT by 8.75% and improves the standard FREAK by 28.65%. Using the proposed stereo feature descriptor, a visual odometry algorithm is designed and tested on the KITTI dataset. The stereo FREAK descriptor raises the number of inlier matches by 19% and consequently improves the accuracy of visual odometry by 23%.
Formation of giant waves in sea states with two spectral maxima, centered at close wave vectors ${\bf k}_0\pmΔ{\bf k}/2$ in the Fourier plane, is numerically simulated using the fully nonlinear model for long-crested water waves [V. P. Ruban, Phys. Rev. E {\bf 71}, 055303(R) (2005)]. Depending on an angle $θ$ between the vectors ${\bf k}_0$ and $Δ{\bf k}$, which determines a typical orientation of interference stripes in the physical plane, rogue waves arise having different spatial structure. If $θ\lesssim\arctan(1/\sqrt {2})$, then typical giant waves are relatively long fragments of essentially two-dimensional (2D) ridges, separated by wide valleys and consisting of alternating oblique crests and troughs. At nearly perpendicular ${\bf k}_0$ and $Δ{\bf k}$, the interference minima develop to coherent structures similar to the dark solitons of the nonlinear Shroedinger equation, and a 2D freak wave looks much as a piece of a 1D freak wave, bounded in the transversal direction by two such dark solitons.
We construct and discuss a semi-rational, multi-parametric vector solution of coupled nonlinear Schrödinger equations (Manakov system). This family of solutions includes known vector Peregrine solutions, bright-dark-rogue solutions, and novel vector unusual freak waves. The vector freak (or rogue) waves could be of great interest in a variety of complex systems, from optics to Bose-Einstein condensates and finance.
We describe a general N-solitonic solution of the focusing NLSE in the presence of a condensate by using the dressing method. We give the explicit form of one- and two- solitonic solutions and study them in detail. We distinguish a special class of solutions that we call regular solitonic solutions. Regular solitonic solutions do not disturb phases of the condensate at infinity by coordinate. All of them can be treated as localized perturbations of the condensate. We find a broad class of superregular solitonic solutions which are small perturbations at certain a moment of time. Superregular solitonic solutions are generated by pairs of poles located on opposite sides of the cut. They describe the nonlinear stage of the modulation instability of the condensate and play an important role in the theory of freak waves.
We review recent progress in modeling the probability distribution of wave heights in the deep ocean as a function of a small number of parameters describing the local sea state. Both linear and nonlinear mechanisms of rogue wave formation are considered. First, we show that when the average wave steepness is small and nonlinear wave effects are subleading, the wave height distribution is well explained by a single "freak index" parameter, which describes the strength of (linear) wave scattering by random currents relative to the angular spread of the incoming random sea. When the average steepness is large, the wave height distribution takes a very similar functional form, but the key variables determining the probability distribution are the steepness, and the angular and frequency spread of the incoming waves. Finally, even greater probability of extreme wave formation is predicted when linear and nonlinear effects are acting together.
The defining feature of chaos is its hypersensitivity to small perturbations. However, we report a stability of branched flow against large perturbations where the classical trajectories are chaotic, showing that strong perturbations are largely ignored by the quantum dynamics. The origin of this stability is accounted for by the piecewise nature of the interference, which is largely ignored by the traditional theory of scattering. Incorporating it into our theory, we introduce the notion of piecewise classical stable paths(PCSPs). Our theory shall have implications for many different systems, from electron transport in nanostructures, light propagation in nonhomogeneous photonic structures to freak wave formations in oceans.
In the article, we describe three-phase finite-gap solutions of the focusing nonlinear Schrödinger equation and Kadomtsev-Petviashvili and Hirota equations that exhibit the behavior of almost-periodic "freak waves". We also study the dependency of the solution parameters on the spectral curves.
Waves propagating through a weakly scattering random medium show a pronounced branching of the flow accompanied by the formation of freak waves, i.e., extremely intense waves. Theory predicts that this strong fluctuation regime is accompanied by its own fundamental length scale of transport in random media, parametrically different from the mean free path or the localization length. We show numerically how the scintillation index can be used to assess the scaling behavior of the branching length. We report the experimental observation of this scaling using microwave transport experiments in quasi-two-dimensional resonators with randomly distributed weak scatterers. Remarkably, the scaling range extends much further than expected from random caustics statistics.
We propose an alternative anthropic probability for calculating the probabilities in eternal inflation. This anthropic probability follows naturally from the weak anthropic principle, and does not suffer the freak observer or the typicality problems. The problem that our observed cosmological constant is not at the peak of the usual anthropic probability distribution is also solved using this proposal.
In this paper we evaluate performance of data-dependent hashing methods on binary data. The goal is to find a hashing method that can effectively produce lower dimensional binary representation of 512-bit FREAK descriptors. A representative sample of recent unsupervised, semi-supervised and supervised hashing methods was experimentally evaluated on large datasets of labelled binary FREAK feature descriptors.
We consider the modulational instability of nonlinearly interacting two-dimensional waves in deep water, which are described by a pair of two-dimensional coupled nonlinear Schroedinger equations. We derive a nonlinear dispersion relation. The latter is numerically analyzed to obtain the regions and the associated growth rates of the modulational instability. Furthermore, we follow the long term evolution of the latter by means of computer simulations of the governing nonlinear equations and demonstrate the formation of localized coherent wave envelopes. Our results should be useful for understanding the formation and nonlinear propagation characteristics of large amplitude freak waves in deep water.
Recently, the Fisher vector representation of local features has attracted much attention because of its effectiveness in both image classification and image retrieval. Another trend in the area of image retrieval is the use of binary features such as ORB, FREAK, and BRISK. Considering the significant performance improvement for accuracy in both image classification and retrieval by the Fisher vector of continuous feature descriptors, if the Fisher vector were also to be applied to binary features, we would receive similar benefits in binary feature based image retrieval and classification. In this paper, we derive the closed-form approximation of the Fisher vector of binary features modeled by the Bernoulli mixture model. We also propose accelerating the Fisher vector by using the approximate value of posterior probability. Experiments show that the Fisher vector representation significantly improves the accuracy of image retrieval compared with a bag of binary words approach.