搜索结果：Text-to-Speech

Extended Reality (XR) technologies offer transformative potential for language education, yet current platforms largely neglect the accessibility needs of deaf and hard-of-hearing individuals. Existing solutions typically operate in single-language environments and lack integrated support for sign language and multimodal communication. There is a critical need for inclusive platforms that serve both deaf and hearing learners through cross-modal AI services embedded in immersive environments. This study presents a modular platform integrating six AI services: speech-to-text transcription (OpenAI Whisper), multilingual translation (Meta NLLB), text-to-speech synthesis (AWS Polly), sentiment analysis (RoBERTa), session summarisation (flan-t5-base-samsum), and International Sign (IS) translation via Google MediaPipe. An IS dataset of 750 gesture videos was processed to extract hand landmark coordinates mapped to 3D avatar animations within a Unity-based VR environment on Meta Quest 3 headsets. The system was validated through technical benchmarking of AI service performance, including comparative evaluation of text-to-speech services and multilingual translation models (NLLB-200 and EuroLLM 1.7B variants), load testing to assess platform. scalability, and end-to-end pipeline latency measurement for both the hearing and the deaf user pathways. The educational scenario was additionally evaluated in a companion pilot study, 50 which shares the same underlying AI services and provides complementary user-perception evidence. Technical benchmarking confirmed the platform's viability for real-time XR deployment. TTS benchmarking confirmed AWS Polly's lowest latency (50-100 ms first byte) at competitive cost. The EuroLLM 1.7B Instruct model achieved a BLEU score of 84.34, outperforming NLLB's 79.25. Load testing with 1,000 simulated concurrent users demonstrated average response times under 800 milliseconds with no critical failures. Avatar animation latency for IS sign rendering remained consistently under 300 milliseconds. End-to-end pipeline latency averaged 2.05 ± 0.31 s for the hearing pathway and 2.32 ± 0.34 s for the deaf (IS) pathway, both within accepted thresholds for conversational educational use. The companion pilot (N = 10) reported a mean overall experience rating of 4.6/5.0, 92% user satisfaction and unanimous (100%) demand for expanded language and sign-language support. 50. The results presented in this study focus on the technical feasibility of integrating cross-modal AI services within XR environments for accessible, multilingual language learning. The modular architecture enables independent scaling and adaptation to diverse contexts, laying the groundwork for equitable educational solutions aligned with EU digital accessibility objectives. Learning a new language can be challenging, and it is even more difficult for deaf individuals who rely on sign language. This study addresses the challenge by creating a virtual reality (VR) learning environment where a digital 3D character (avatar) can speak, translate, and perform sign language in real time. The system uses several artificial intelligence tools working together: one that converts speech into text, another that translates text between multiple languages, a third that converts text back into spoken language, and a fourth that translates text into International Sign Language gestures performed by the avatar. Users wear a VR headset and interact with the avatar in a virtual classroom where they can select their preferred language and receive immediate translations in both spoken and signed forms. The system’s technical performance was validated through benchmarking of AI translation models, text-to-speech services, end-to-end pipeline latency measurement, and scalability testing, confirming its suitability for real-time educational applications. Complementary user feedback gathered in a companion pilot study, which uses the same underlying AI services, is cross-referenced where relevant. 50 This research is a step toward creating virtual learning environments where language barriers and hearing limitations no longer prevent people from accessing education.

This study proposed the integration of a laser Doppler vibrometer sensing with a Variational Inference with adversarial learning for Text-to-Speech-based voice conversion system to enhance automatic speech recognition for individuals with dysarthria in noisy environments. The proposed framework combines the noise robustness of laser Doppler vibrometer and generative modeling capabilities of Variational Inference with adversarial learning for Text-to-Speech to transform dysarthric speech into intelligible acoustic outputs. Experimental results demonstrated significant gains in automatic speech recognition accuracy compared with conventional acoustic methods, even at low signal-to-noise ratios. These findings establish a foundation for future clinical applications of augmentative and alternative communication systems.

Voice-based authentication is widely used today but remains vulnerable to spoofing attacks like text-to-speech and voice conversion, threatening privacy and its authenticity. To mitigate such threats, various detection techniques ranging from conventional machine learning models to contemporary deep learning frameworks have been used in the past. Yet, it is difficult to choose an ideal solution, as model performance typically varies with dataset attributes, computational power, and deployment environments.This paper is a comparative experimental analysis of four spoofing detection models as controllers: Support Vector Machine (SVM), Gaussian Mixture Model (GMM), ResNet, and Wav2Vec 2.0 with any transformer. The models are tested with one experiment through the usage of the automatic speaker verification spoof 2019 Logical Access dataset. Equal error rate and tandem detection cost function measures the effectiveness of the detection, whereas training and inference time measure the effectiveness of the computation. Results, under the standardized set up, are that ResNet is typified with a good detection, at a comparatively low computational cost, and Wav2Vec 2.0 gives good error rates, at a significantly larger cost of fine-tuning. The GMM and classical SVM models are lower in their cost of computation and greater in error rates, respectively.Overall, the results reveal that more complex models do not always improve, and using the detection performance and runtime analysis, the study gives appropriate guidance on how to choose the spoof detection models based on the deployment needs, not only the accuracy.

The objective of this study is to develop and assess a wearable assistive device, enabled by the Internet of Things (IoT), that improves obstacle perception, mobility, and interaction with the environment for those with visual impairments. The technology amalgamates real-time object detection, facial recognition, and obstacle sensing into a unified framework to facilitate autonomous and secure navigation. An experimental design-and-build methodology was employed to create the prototype utilising an ESP32-CAM module, Arduino microcontroller, ultrasonic sensor, and text-to-speech feedback interface. Deep learning and machine learning algorithms-Support Vector Machine (SVM) for facial detection, ResNet-50 Convolutional Neural Network (CNN) for facial identification, and YOLOv4 for object detection-were employed to facilitate precise, low-latency performance. An evaluation of the system was performed under various environmental conditions utilising conventional criteria such as accuracy, precision, recall, and F1-score. The implemented system attained a recognition accuracy surpassing 90% with negligible computational latency, making it appropriate for real-time applications. The hybrid SVM-YOLOv4 model exhibited a harmonious mix of precision and efficiency, facilitating dependable multi-object recognition under various environmental and lighting situations. The quantitative results confirm the strength and flexibility of the proposed wearable device for supportive purposes. The IoT-enabled wearable assistive technology demonstrates considerable promise to enhance mobility, environmental awareness, and independence in those with visual impairments. The device combines hardware sensors with advanced computer vision models, linking theoretical innovation to practical rehabilitative applications. Future endeavours will encompass extensive user validation to improve usability, accessibility, and user comfort. Consequences for rehabilitation: Wearable devices driven by IoT can enhance mobility and situational awareness for those with visual impairments. The incorporation of machine learning-derived feedback facilitates safer, autonomous navigation in fluctuating situations. The modular system architecture facilitates adaption for a wider range of assistive and rehabilitation applications. The research enhances the design of cost-effective, intelligent assistive technologies that align with rehabilitative objectives and the creation of inclusive technology. Wearable assistive technology powered by the Internet of Things has significant ramifications for rehabilitation, mainly because it gives visually impaired people more mobility and independence through voice feedback, real-time object and face recognition, and enhanced spatial awareness through an ultrasonic sensor and buzzer system. In addition to helping with personal navigation, this multifaceted assistive support—which combines computer vision, machine learning, and real-time feedback—has the potential for wider applications in fields like security and surveillance. Ultimately, by tackling the major obstacles that visually impaired people encounter when navigating and engaging with their surroundings, this technology helps to create a more thorough approach to rehabilitation.

Recent advancements in visual speech recognition (VSR) have promoted progress in lip-to-speech synthesis, where pre-trained VSR models enhance the intelligibility of synthesized speech by providing valuable semantic information. The success achieved by cascade frameworks, which combine pseudo-VSR with pseudo-text-to-speech (TTS) or implicitly utilize the transcribed text, highlights the benefits of leveraging VSR models. However, these methods typically rely on mel-spectrograms as an intermediate representation, which may introduce a key bottleneck: the domain gap between synthetic mel-spectrograms, generated from inherently error-prone lip-to-speech mappings, and real mel-spectrograms used to train vocoders. This mismatch inevitably degrades synthesis quality. To bridge this gap, we propose Natural Lip-to-Speech (NaturalL2S), an end-to-end framework that jointly trains the vocoder with the acoustic inductive priors. Specifically, our architecture introduces a fundamental frequency (F0) predictor to explicitly model prosodic variations, where the predicted F0 contour drives a differentiable digital signal processing (DDSP) synthesizer to provide acoustic priors for subsequent refinement. Notably, the proposed system achieves satisfactory performance on speaker similarity without requiring explicit speaker embeddings. Both objective metrics and subjective listening tests demonstrate that NaturalL2S significantly enhances synthesized speech quality compared to existing state-of-the-art methods. Audio samples are available on our demonstration page: https://yifan-liang.github.io/NaturalL2S/.

A persistent challenge for Deaf and Hard-of-Hearing individuals is the communication gap between sign language users and the hearing community, particularly in regions with limited automated translation resources. In Saudi Arabia, this gap is amplified by the reliance on Saudi Sign Language (SSL) and the scarcity of real-time, sentence-level translation systems. This paper presents a real-time system for sentence-level recognition of continuous SSL and direct mapping to natural spoken Arabic. The proposed system operates end-to-end on live video streams or pre-recorded content, extracting spatio-temporal landmark features using the MediaPipe Holistic framework. For classification, the input feature vector consists of 225 features derived from hand and body pose landmarks. These features are processed by a Bidirectional Long Short-Term Memory (BiLSTM) network trained on the ArabSign (ArSL) dataset to perform direct sentence-level classification over a vocabulary of 50 continuous Arabic sign language sentences, supported by an idle-based segmentation mechanism that enables natural, uninterrupted signing. Experimental evaluation demonstrates robust generalization: under a Leave-One-Signer-Out (LOSO) cross-validation protocol, the model attains a mean sentence-level accuracy of 94.2%, outperforming the fixed signer-independent split baseline of 92.07%, while maintaining real-time performance suitable for interactive use. To enhance linguistic fluency, an optional post-recognition refinement stage is incorporated using a large language model (LLM), followed by text-to-speech synthesis to produce audible Arabic output; this refinement operates strictly as post-processing and is not included in the reported recognition accuracy metrics. The results demonstrate that direct sentence-level modeling, combined with landmark-based feature extraction and real-time segmentation, provides an effective and practical solution for continuous SSL sentence recognition in real-time.

Repeated exposure to information increases the likelihood that people will judge it as accurate. This phenomenon - known as the illusory truth effect - is robust and widely replicated, and particularly relevant in an age of unfiltered information dissemination. In four experiments, we test whether perceptual fluency is necessary for the effect to occur, whether auditory stimuli generated by text-to-speech (TTS) systems can elicit the effect, and whether voice similarity between speaker and recipient increases the perceived truth of statements. The results of our first three experiments indicate that perceptual fluency is likely the primary driver of the effect, and that repeated exposure to TTS-generated auditory stimuli enhances credibility, albeit to a lesser extent than written statements. Building on this, our fourth experiment shows that - although voice similarity only marginally amplifies the illusory truth effect - the credibility of a statement increases when presented in a voice similar to the listener's, even without repetition.

The objective of this scoping review was to explore the use of assistive technology incorporating smart camera features in the rehabilitation of people living with disabilities. This review was conducted in accordance with the Joanna Briggs Institute methodology for scoping reviews. Articles were eligible if they considered assistive technology that incorporated at least one smart camera feature. Rehabilitation in any setting was considered. Research including participants of any age with any form of disability, both self-reported and diagnosed, was considered. 25 studies were included in the final synthesis. Most studies investigated assistive technology for visual impairment (n = 23). The most explored devices were smartphones (n = 16) and/or smart glasses (n = 13). Most studies focused on text-to-speech (n = 22) and/or object recognition features (n = 15). One study focused on facial recognition for people with Alzheimer's disease and one on reading for children with dyslexia. Assistive technology incorporating smart camera features has been used mainly in the rehabilitation of people living with visual disabilities. Reported rehabilitative outcomes included enhanced reading ability and improved daily living skills. However, there were technical limitations, usability issues and high financial costs associated with various devices. Assistive technology incorporating smart camera features may enhance reading ability and daily living skills for people living with visual impairment.This technology is available in the form of smartphone apps and wearable devices.Its potential uses for people living with other types of disabilities remain under-explored in the literature.

Neural audio codecs (NACs) based on end-to-end neural networks and vector quantization have recently achieved high-fidelity speech compression and reconstruction. However, most existing NACs learn entangled latent codes that mix speaker timbre, prosody, and phonetic information, which limits interpretability and controllability. Although several attempts introduce attribute-aware objectives, they often lack an explicit decomposition mechanism and a principled information-theoretic constraint to encourage independent factorization. We propose OACodec, a novel neural audio disentanglement codec specifically designed to learn disentangled representations of speech attributes. Unlike conventional neural audio codec systems that treat audio as undifferentiated data, OACodec introduces a multi-stage orthogonal disentanglement network that explicitly separates timbre, prosody and phonetic information. Each latent attribute is extracted via a residual separation mechanism, guided by orthogonality constraints and supervised learning. To further promote independent factorization, we employ mutual information estimators both within and across attribute components, minimizing their mutual information. Additionally, we adopt a smoothed Tchebycheff optimization strategy to achieve a Pareto-optimal balance between reconstruction fidelity and disentanglement objectives. Experimental results demonstrate the effectiveness of OACodec in both zero-shot voice conversion and speech reconstruction, where it outperforms VC baselines and FACodec in zero-shot VC tasks and achieves reconstruction quality comparable to EnCodec and HiFi-Codec while surpassing FACodec. We further demonstrate, by constructing a two-stage TTS model, that the disentangled codes effectively improve performance on the TTS task. Ablation studies further validate the contribution of each proposed module. OACodec lays a strong foundation for interpretable and controllable speech modeling, with promising implications for applications such as text-to-speech and speech-based language modeling. Audio samples are available at https://hamidun123.github.io/OACodecDemo/.

Silent speech interfaces (SSIs) offer a viable alternative to traditional microphones in capturing clear audio in noisy environments. We propose a reconceptualized SSI that reproduces voice by monitoring continuous multiaxial strain maps induced by throat muscle movements. The system integrates a computer vision-based optical strain (CVOS) sensor with deep learning-based voice reconstruction, enabling clear alphabetic communication under extreme noise conditions. The CVOS sensor-comprising a soft silicone substrate with micromarkers and a tiny camera-achieves high-sensitivity marker detection and captures complex strain patterns with higher scalability and reliability compared to conventional wearable sensors. The inference pipeline of the CVOS-based SSI incorporates physics-based automated baseline calibration and content-adaptive temporal attention, enabling robust analysis of the captured strain patterns. Based on the inference results, a personalized text-to-speech model subsequently reconstructs the speaker's voice. These algorithmic features ensure robustness under dynamic conditions by employing real-time adaptive signal processing that compensates for inter- and intrasubject anatomical variability. Alphabet-based communication is achieved through the synergy between optimized algorithms and interface design. The performance of the CVOS-based SSI was validated in real-world noisy scenarios, confirming its practical applicability.

Background/Objectives: The objective of this scoping review was to map and critically describe emerging speech-in-noise assessment tools developed over the last decade for the evaluation of hearing loss beyond conventional audiological measures. Methods: This review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines. A comprehensive literature search was performed in the PubMed/MEDLINE, Scopus, and Embase databases. A comprehensive review of studies describing novel or emerging SIN-based assessment tools was conducted, with a particular emphasis on those including adult participants with normal hearing and hearing loss. Results: Nine studies met the inclusion criteria and were included in the review. The identified tools cover a range of methodological innovations, including advanced digits-in-noise paradigms, antiphasic and binaural presentation modes, optimized adaptive procedures, and digital or automated testing platforms. Several studies also incorporated artificial intelligence-based approaches, such as machine learning, text-to-speech, and automatic speech recognition, to enhance test development, administration, and hearing loss classification. Across all studies, SIN measures demonstrated the ability to reliably differentiate between normal hearing listeners and individuals with hearing loss and to provide complementary information beyond pure-tone audiometry. Conclusions: Emerging speech-in-noise tools show considerable potential to improve the functional assessment of hearing loss and to support more sensitive, accessible, and scalable approaches for hearing evaluation. Further research is required to assess their clinical integration and long-term impact on hearing screening and diagnostic pathways.

Creating engaging language stimuli suitable for children can be difficult and time-consuming. To simplify and accelerate the process, we developed an automated pipeline that combines existing audio generation and animation tools to generate customizable audiovisual stimuli from text input. The pipeline consists of two components: the first uses Google Cloud Text-to-Speech to generate audio stimuli from text, and the second uses Adobe Character Animator to create video stimuli in which an animated character "speaks" the audio with speech-aligned mouth movements. We evaluated the pipeline with two stimulus sets, including an acoustic comparison between generated audio stimuli and existing human-recorded stimuli. The pipeline is efficient, taking less than 2 min to generate each audiovisual stimulus, and fewer than 9 % of stimuli needed to be regenerated. The audio generation component is particularly fast, taking less than 1 s per stimulus. By leveraging automated tools for language stimulus creation, this pipeline can facilitate developmental research on language and other domains of cognition, especially in cognitive neuroscience studies that require large numbers of stimuli.