This article suggests ways in which a cultural sign language frame for use by and with people with congenital deafblindness might be supported. It proposes including an approach to tactile signing informed by insights from the protactile movement, which grew out of attempts to rework the visual nature of American Sign Language, thereby creating a new tactile language. Protactile language represents a profound commitment to a co-constructive, participatory and enactive form of tactile languaging that goes far beyond improved access to cultural linguistic signs and signing in the tactile modality, extending instead into tactile collaborative meaning-making at a deeper cognitive (conceptual-exploratory) level. Within an enactive embodiment perspective, both cognition and language are activities pursued by embodied subjects in real time, rather than abstract, propositional processes. Language of any kind is always with and from the body, and the sensory modalities of its use must be accessible to speakers and listeners. This article uses perspectives from Material Engagement Theory. We describe two examples from our own (non-protactile) practice and examine these through the lens of protactile language. We suggest that attending to issues of tactile salience and reciprocity that are grammatically enacted in protactile language is valuable for co-constructive signing with people with congenital deafblindness. Considering the principles of protactile language can enable conversation partners to move from a signal-based, information-relaying frame toward a reciprocal, dialogical communicative frame.
Body image disturbance (BID) is a key symptom of anorexia nervosa (AN) and involves body size overestimation. Although this overestimation might be related to perceptual deficits, little is known about tactile perception in AN. To clarify the role of putative tactile deficits in BID, the present study investigated 36 adolescent female AN patients and 41 matched healthy controls (HC) using a tactile oddball paradigm during parallel EEG and MEG (EMEG) measurement. Tactile perception was behaviorally tested via the Touch Test (tactile perception threshold), a deviant count task, and a tactile stimulus discrimination task. Compared to HC participants, AN patients had similar tactile perception thresholds, but performance in the deviant count task and the tactile stimulus discrimination task was poor. Non-parametric cluster permutation tests on estimated neural source activity of evoked EMEG responses revealed that AN patients and HC did not differ regarding the neural differentiation between deviant and standard stimuli (oddball effect). However, estimated neural activity was globally reduced in the inferior temporal cortex and (by trend) in the posterior parietal cortex. These findings argue against the idea that behavioral discrimination deficits of tactile stimuli in AN are grounded in neural stimulus discrimination deficits. Instead, the overall reduced neural activity in the inferior temporal and posterior parietal cortex might reflect aberrant integration of neural tactile representations into a coherent multisensory body representation.
Tactile sensation and hand dexterity decline with age and are associated with later-life cognition, yet it remains unclear whether tactile sensation moderates the strength of hand motor-cognitive associations. This cross-sectional study examines whether tactile sensation moderates the association between hand dexterity and higher-level cognition (executive function and processing speed) in cognitively normal (CN) older adults and those with mild cognitive impairment (MCI), and whether moderation patterns differ between these groups. Participants were 132 community-dwelling older adults aged ≥ 60 (female = 79; mean age = 73.3 ± 5.2). MCI was defined by MMSE ≥ 24 and MoCA ≤ 25. Executive function was assessed with Trail Making Test-B (TMT-B) and letter fluency (LFT); processing speed with TMT-A, Digit Symbol, and category fluency (CFT). Hand dexterity and tactile sensation were measured using the Purdue Pegboard Test and Weinstein-Semmes monofilaments. Moderation analyses tested the interaction between dexterity and tactile sensation, adjusting for demographic and health-related covariates. Tactile sensation significantly moderated the association between hand dexterity and executive function and processing speed, with stronger dexterity-cognition associations among older adults with intact tactile sensation (TMT-B p < 0.05; LFT p < 0.05; TMT-A p < 0.01). No moderation was observed for Digit Symbol or CFT. The three-way interaction with cognitive group was not significant, indicating comparable patterns across groups. Stratified analyses showed a significant LFT interaction in the CN group (p < 0.05). Older adults with intact tactile sensation showed stronger hand dexterity-cognitive associations, highlighting sensorimotor integrity as relevant for early screening and functional evaluation. Stratified patterns suggest possible cognitive group differences that merit further study.
In environments with multiple talkers, humans can 'tune in' to a speaker of interest while ignoring competing voices. In such conditions, however, auditory cortices track the attended speech envelope rhythms (cortical tracking of speech, CTS) less accurately than in quiet, hindering intelligibility. Visual speech cues (e.g., lip movements) can enhance this CTS, but it remains unclear whether other non-auditory sensory cues, such as tactile input, provide comparable benefits through similar neural mechanisms. Here, using magnetoencephalography, we quantified CTS as the coherence between the speech temporal envelope of the attended speaker and brain responses in supratemporal auditory areas at syllabic (4-8 Hz), word (1-4 Hz) and phrasal/sentential (<1 Hz) frequencies. Participants listened to connected speech presented alone, together with synchronous or asynchronous speech-based vibrations, or with the corresponding speaker video, in both quiet and a multi-talker background. We hypothesized that, in the presence of competing background speakers, speech-based vibrotactile stimulation improves comprehension by enhancing CTS of the attended speaker and modulating auditory-seeded functional brain connectivity with extra-auditory neocortical brain areas. Results revealed that synchronous vibrotactile stimulation improved comprehension in the multi-talker background and increased syllabic CTS at the right auditory cortex, with this CTS increase magnitude correlating with comprehension performance. Audio-tactile CTS enhancement was accompanied by stronger beta-band auditory cortex connectivity with ipsilateral angular and ventral inferior temporal gyri, alongside reduced alpha-band coupling with the precuneus. These findings suggest that vibrotactile input can support speech-in-noise processing by impacting both local auditory cortical activity and auditory-seeded long-range functional connectivity.
The perception of tactile locations is an important function of human's somatosensory system during body movements and its interactions with the surroundings. Our previous study on the perception of locations found that the gamma-frequency band provides better decoding accuracy than all the lower frequencies on the legs. In the present study, we recorded electroencephalography (EEG) responses evoked by four vibrotactile stimulators placed on the arms of 18 human subjects. Human subjects were instructed to sit in a chair while somatosensory-evoked potentials were obtained using a 64-channel EEG. A linear classifier and an artificial neural network with 10 hidden-layer neurons were separately used to predict tactile locations based on EEG power obtained from various frequency bands. We found that the beta (13-30 Hz) and high-gamma (50-100 Hz) bands can best predict the tactor locations on the arms. Interestingly, power information carried by the high-gamma band was uncorrelated to information contained in the lower frequency bands. Consequently, combining the beta and high-gamma bands significantly improved the prediction accuracy. Our findings prove that tactile location information can be decoded from EEG signals, which agrees with previous studies regarding the importance of the gamma band during tactile perception.
It is not known how the brain extracts vibrotactile stimulus frequency from the peripheral afferent population whose afferents exhibit a wide range of response patterns to a pure frequency stimulus, including afferents which do not respond to every vibration cycle of the stimulus and thus have lower firing rates. One hypothesis is a winner-takes-all mechanism based on the peripheral afferents with the highest or most regular firing rate, which discounts those firing at lower rates. We used a high-resolution fingertip electrode array and activated some electrodes at 40 Hz and others at 20 Hz to investigate integration of tactile afferent population input signalling frequency information. Participants (n = 12) gave ratings of frequency for these mixed-frequency stimuli against a standard pure frequency. Participants reported perceiving a unified holistic frequency for the mixed-frequency stimuli. This perceived frequency did not match any inter-pulse intervals in the stimulus; instead, it appeared to represent a weighted measure of the two presented frequencies. Our findings add to previous reports of frequency averaging for multiple mechanical stimuli, which suggest that activity across the whole population of activated afferents determines frequency perception. That frequency perception may be based on weighted averaging across all afferents has implications for applications of electrical nerve stimulation for sensory feedback. KEY POINTS: Vibration of the skin causes a wide range of response patterns in tactile afferents of the responding neural population. It has been proposed that a winner-takes-all mechanism favouring afferents with the highest or most regular firing rate enables the nervous system to extract the vibration frequency of the applied stimulus. We stimulated fingertip tactile afferents via an electrical array, with a variable proportion of electrodes activated at 40 Hz and the rest at 20 Hz, and asked participants to make judgements about the stimulus frequency. The perceived frequency was not locked to the highest presented frequency, but instead increased gradually in all participants as the proportion of electrodes delivering 40 Hz increased. This shows that a winner-takes-all mechanism does not operate; instead, the entire population of responding afferents contribute with some weighting factor to vibration frequency perception.
Patients with traumatic brain injury (TBI) are subjected to multiple stressors, including environmental changes, medical treatment, and surgical procedures. Hospitalization in an unfamiliar setting, especially when separated from family members, can increase anxiety and stress, which may adversely influence vital signs and levels of consciousness. Tactile stimulation, when integrated with routine medical management, may contribute to improving patient outcomes. Therefore, the present study aimed to determine the effect of tactile stimulation on the level of consciousness and vital signs among patients with traumatic brain injury. A quasi-experimental study was conducted in the neurosurgery wards and intensive care units of a tertiary care hospital. A total of 40 patients with mild (Glasgow Coma Scale [GCS] score 13-15) and moderate (GCS score 9-12) traumatic brain injury were selected using a consecutive sampling technique and equally allocated into experimental and control group (n = 20 each). The experimental group received tactile stimulation in addition to conservative care daily at 8 a.m., 12 pm, and 4 p.m. for three consecutive days, with each session lasting 15-20 minutes. The control group received only conservative care. Data were collected using a sociodemographic and clinical profile, a vital sign monitoring sheet, and the Glasgow Coma Scale. Statistical analysis was performed using the Chi-square test, Fisher's exact test, Friedman test, and Mann-Whitney U test. Both groups were comparable at baseline, with no statistically significant differences in socio-demographic and most clinical variables. The majority of patients had a GCS score of 9-12 at admission, and road traffic accidents were the predominant cause of injury. A statistically significant difference was observed only in the type of brain injury (p < 0.001). No statistically significant differences were found between the groups in vital signs or GCS scores across day 1, day 2, and day 3 following the intervention, indicating similar clinical outcomes. The findings indicate that no statistically significant differences were observed between experimental and control groups, indicating that the intervention had no significant effect on the measured clinical outcomes. The limited duration of intervention may have constrained its effectiveness.
Artificial tactile systems increasingly use independent transduction of slow-adapting (SA) signals from static pressure and rapid-adapting (RA) signals from dynamic vibrations, aiming to mimic the human skin's mechanosensory pathways for enhanced perceptual processing. Here, we present a tactile sensing system based on hybrid materials that integrate an Fe3O4-based piezomagnetic elastomer and a poly(vinyl chloride) (PVC)-based iontronic gel in a unified layered architecture, enabling the orthogonal encoding of SA and RA mechanotransduction. The Fe3O4 elastomer exhibits piezomagnetic coupling, yielding a magnetic flux density of ∼1.5 mT and a peak voltage modulation of ∼15 mV at 3.2 N load, while effectively capturing RA signals over a wide bandwidth up to 1 kHz. Concurrently, the iontronic PVC gel is self-driven by the potential of the Fe3O4 elastomer and delivers stable SA signal outputs with a sensitivity of 1.6/0.48 mV N- 1 (whereas the RA channel exhibits 11.4/0.75 mV N-1). By combining these decoupled signal modalities, we construct a haptic mapping framework that generates distinctive tactile fingerprints of object surfaces. This multimodal self-driven sensing approach enables accurate classification of material texture, reliable slip detection, and identification of surface anomalies with different groove widths. This work offers a scalable materials strategy for intelligent robotics and human-machine interfaces.
Background: Early-life sensory experiences play a crucial role in the maturation of neural circuits underlying sensorimotor development. The present study investigated whether the effects of neonatal tactile stimulation (NTS) on sensorimotor development depend on the developmental timing of the intervention in WAG/Rij rat pups, a well-established genetic model of absence epilepsy. Methods: Seventy rat pups were randomly assigned to five groups: Wistar control, WAG/Rij control, and WAG/Rij rats receiving NTS during the first (PN Week 1-NTS), second (PN Week 2-NTS), or third (PN Week 3-NTS) postnatal week. Tactile stimulation was administered three times daily. Sensorimotor development was assessed on postnatal day 22 using a modified rung-bridge task that evaluated orientation, flexor/extensor activity, distal control, gait development, postural control, sensorimotor responses, tail use, total sensorimotor performance, and crossing time. Results: WAG/Rij control pups exhibited impaired sensorimotor performance compared with Wistar controls, demonstrating lower distal control, gait development, sensorimotor response, and total sensorimotor scores, together with prolonged crossing times. The effects of the NTS protocol were dependent on the developmental window in which it was administered. The NTS protocol applied during the second postnatal week was associated with the most pronounced improvements, including significantly higher distal control, sensorimotor response, tail performance, and total sensorimotor scores, as well as significantly shorter crossing times compared with untreated WAG/Rij controls. In contrast, orientation, flexor/extensor activity, and postural control were not significantly affected. Discussion and Conclusions: These findings indicate that sensorimotor abnormalities are detectable in WAG/Rij rats during the early postnatal period, before the typical adult expression of absence seizures. The effects of the NTS protocol were strongly dependent on developmental timing, with the second postnatal week representing a particularly sensitive developmental window for sensorimotor maturation. Overall, the NTS protocol was associated with improved early sensorimotor performance in WAG/Rij rats, particularly when administered during the second postnatal week. Because a sham-handled control group was not included, these findings should be interpreted as reflecting the NTS protocol as a whole rather than tactile stimulation alone.
The sensory function of the tongue is critical for mastication and swallowing. However, its assessment methods remain insufficiently established. In this study, we developed a pneumatic multi-point stimulation system capable of controlling the location, timing, and pressure of small balloon inflations to simulate the movement and elasticity of food within the oral cavity. Using this system, we established a new method to quantify lingual tactile sensitivity, specifically the ability to detect stimulus point movement under passive touch and pressure differences under active touch. We evaluated the age-related differences in lingual tactile sensitivity in 16 young men (mean age ± standard deviation, 21.9 ± 1.2 years) and 20 older men (74.1 ± 4.1 years) with no history of tongue or related disorders. The results demonstrated significant age-related declines in lingual tactile sensitivity under both passive and active touch conditions. Furthermore, these measures showed no significant correlations with conventional sensory tests, including the Semmes-Weinstein monofilament test and the two-point discrimination test. This suggests that the proposed method captures aspects of lingual sensory function not assessed by existing approaches. By enabling the quantitative evaluation, using a single device, of not only dynamic sensory perception during passive touch but also active touch, which involves the integration of sensory input (superficial and deep sensation) and motor output, this method has the potential to serve as a novel tool for screening oral hypofunction in older adults and for supporting the assessment of dysphagia, including masticatory dysfunction.
Soft dual-modal tactile sensors capable of simultaneously sensing force and temperature are essential for enhancing human-like perception and interaction in robots, particularly in the functional sense of concurrent mechanical and thermal perception. However, achieving self-decoupled and high-fidelity dual-modal sensing remains a significant challenge due to intrinsic signal crosstalk, structural complexity, and limited flexibility in existing designs. Here, we present a soft robotic tactile (RoboTac) skin that intrinsically decouples force and temperature using an ionic conductive film within a minimalist architecture, featuring an ultralight weight and an ultralow cost. Ionic conductivity enables independent readouts without algorithmic compensation by allowing thickness compression to modulate capacitance (force) and lateral ionic transport under thermal stimuli to modulate resistance (temperature). Moreover, the RoboTac skin demonstrates its practical utility for robots in object perception, specialized tasks, and human-robot interaction. This work establishes a general principle for intrinsically self-decoupling modalities in tactile sensors, advancing multimodal sensing, intelligent perception, and embodied robotics.
We demonstrate a multimodal, wearable device with haptics-based communication that enables wearers to perceive environmental hazards through vibrations. The device monitors gaseous, aerosolized, and aqueous contaminants and conveys threshold events via distinct tactile codes. Energy harvesting with low power sensing methods yield a high-fidelity system with day long operation. Communication can be extended from humans to robots by engineering a soft electronic skin (e-skin) incorporating an array of transducers embedded in silicone that resolves the temporal structure of the tactile codes. On a quadrupedal robot, the e-skin decodes haptic sequences to trigger adaptive re-routing upon detecting chemical hazards, bypassing the need for wireless communication. Our approach introduces a framework in which chemical awareness is communicated physically rather than electronically, opening opportunities for embodied intelligence, distributed sensing, and human-robot interactions.
[This corrects the article DOI: 10.1093/pnasnexus/pgag164.].
Multisensory integration (MSI) is crucial to interact adaptively with our environment and create a coherent representation of the body. Body illusion techniques, such as the rubber hand illusion, provide useful tools to study MSI since they produce a conflict between different sensory modalities and alter body perception. While previous research revealed that body illusions modulate neural activity in unisensory and multisensory cortical areas during multisensory stimulation, cortical activity during unisensory stimulation has barely been studied. The present EEG study investigated brain mechanisms involved in MSI using unisensory (visual, tactile) and multisensory (visuotactile) stimulation. Alpha- and beta-band power, as well as functional connectivity, were assessed in 36 subjects during visual, tactile, and visuotactile stimulation in the rubber hand illusion paradigm and a control condition. The illusion was associated with widespread reductions in alpha and beta power, alongside increased beta-band connectivity between unisensory and multisensory cortical regions. Notably, alpha-band effects revealed cross-modal influences: visual stimulation modulated activity in somatosensory regions, while tactile stimulation affected occipital areas. Unisensory stimulation also elicited beta connectivity changes across modalities, with visual input enhancing sensorimotor coupling and tactile input increasing connectivity involving visual regions. The subjective strength of the illusion was associated with beta power reductions during visual stimulation, linking neural modulation to individual perceptual experience. These findings demonstrate that the illusory state is characterized by a global reconfiguration of sensory processing, extending beyond multisensory contexts and shaping unisensory processing through cross-modal interactions.
The tactile consequences of self-initiated movements are thought to be predicted by a forward model, yet the precise neural implementation of these predictions remains unclear. In non-motor contexts, expectations are thought to activate sensory neurons tuned towards the expected stimulus. This acts as a predictive template against which afferent sensory input is compared. It is unclear whether forward model predictions have a similar neural instantiation. Here we employed time-resolved multivariate decoding on human electroencephalography (EEG) during self-generated movements to examine the content of predictive neural activity. Human participants (males and females) performed index finger movements which were predictably paired with a vibration to either the index or ring finger of the opposite, passive hand. On some trials the tactile stimulus was unexpectedly omitted. Results revealed above-chance finger decoding in the pre-movement period supporting a predictive representation of expected stimulation location. As the movement approached, this predictive activity became similar to late-stage processing of a physical tactile stimulus. On omission trials, we found that despite the absence of afferent input, finger location could be decoded ∼120 ms after expected stimulus onset. This shows a stimulus-specific omission response. Together these findings indicate that self-generated movement pre-activates neurons tuned towards expected tactile consequences.Significance Statement Engaging effectively with the world relies on our ability to anticipate the sensory consequences of our own movements. To characterise how the brain encodes action-driven tactile predictions we recorded EEG while participants performed finger movements paired with vibrations to either the index or ring finger of the opposite hand. Crucially, we introduced unexpected omissions, where no stimulus followed the movement. Applying time-resolved EEG-decoding methods, we show that pre-movement neural activity encodes expected stimulation location. In addition, on omission trials, despite the absence of bottom-up stimulation, we could decode expected stimulation location. Such predictions of self-produced tactile events likely ensure accurate dissociation of self from other and promote a sense of agency over our own motor actions.
Skin perceives touch by transmitting mechanical stimuli through a hierarchically organized, gradient-modulus architecture that concentrates stress and localizes deformation to activate mechanoreceptors. Inspired by the hierarchical architecture of skin, modulus engineering has been introduced into artificial tactile systems. However, how stress transmission governs internal electric-field distributions-and how the coupling can be exploited for intelligent tactile perception-remain largely unexplored. Here we report a skin-inspired, gradient-modulus electroluminescent pressure display that directly converts mechanical inputs into spatially resolved optical signals. Guided by Hertzian contact theory, we uncover gradient-modulus-induced stress focusing and deformation localization, which in turn redistributes the internal electric field and sensitizes the emissive layer to pressure. This mechano-electro-optical coupling enables intuitive visualization of tactile stimuli. Furthermore, the resulting spatiotemporal luminescence patterns provide rich information for intelligent tactile recognition. Our results establish a general strategy to bridge mechanical perception and optical output, offering a conceptual route toward visualized electronic skin and next-generation human-machine interfaces.
For animals to interact coherently with the external world, their brains must integrate object location information across sensory modalities, an inherently complex process. Each sensory modality samples distinct regions of external space, with object location encoded in fundamentally different reference frames: for example, visual representation of space begins with retinotopic maps, and tactile representation of space in the rodent whisker system starts with somatotopic maps.1,2,3,4,5,6 To form an integrated representation of objects in space, the brain must reconcile differences in spatial coverage and reference frames. The posterior parietal cortex (PPC) has emerged as a key neural substrate for multimodal integration in mammals.7 Within mouse PPC, rostrolateral area (RL), located between primary visual and barrel cortex, serves as a hub for visuo-tactile integration in both supralinear and sublinear ways.8,9,10,11,12,13 A recent study has shown that RL neurons are tuned to binocular disparities, with a preference for very close objects, likely within whisker reach.14 Further, the coherent visual and tactile representation of upper and lower space in RL described recently implies the possible emergence of a unified multimodal framework for near space.11 However, whether this reflects a true three-dimensional near-space representation or simply the aligned arrangement of retinotopic and somatotopic maps remains unresolved. Here, using high-resolution multimodal receptive field mapping in awake mice, we reveal a primarily co-aligned egocentric reference framework across modalities, which is also reflected in cortical topographical maps. We show that visuo-tactile integration is profoundly distance-dependent, affecting not only modality preference but also the mechanisms and linearity of integration.
Central nervous system development is a rapid and highly plastic process during the first years of life. Tactile stimuli have been shown to induce cortical changes, but potential sex-related differences remain unexplored. This study aimed to investigate sex-specific differences in cortical activity and cerebral oxygenation in response to tactile stimulation via body massage. Four healthy full-term infants (two females and two males), all aged 11 weeks, were included in this prospective exploratory study. Each infant received a standardized 5 min massage protocol. Cortical activity and cerebral oxygenation were assessed using an 8-channel electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) before, during, and after the intervention, with a 5 min pre-intervention resting period used as the baseline. EEG analysis focused on a single spectral band (4 Hz-30 Hz). This range was selected to capture the main cortical oscillations in infants, including theta, alpha, and beta activity, while delta activity below 4 Hz was partially excluded to reduce movement and physiological artifacts. Standard infant EEG bands were considered when defining this range. Data shows for the female subject an average PSD of -6.726 (± -4.075), and for the male subject, -12.594 (± -10.741). Although babies are of the same gestational age, they exhibited distinct basal cortical activity, which prevented comparisons from being made. Nevertheless, massage induced similar activity patterns in all subjects with increased cortical electrical activity in the left parietal region relative to baseline. fNIRS data showed that comparable HbO concentration patterns between participants were observed only during the second minute of recording. Relative to baseline, pre-intervention HbO responses displayed an opposite distribution, and the effects of the intervention differed by sex. The female participant exhibited a slight reduction in activation in the right hemisphere accompanied by a modest increase in the most ventral region of the left hemisphere. Conversely, the male participant showed an inverse response pattern, characterized by a marked increase in right hemispheric activation and a pronounced decrease in the left hemisphere during the intervention period. These preliminary observations suggest the presence of early variations in cortical processing that warrant further investigation in larger samples, although they cannot be considered conclusive. While baseline response patterns differed between participants, both showed increased left parietal activity during tactile stimulation. The inversion of HbO responses between the pre-intervention and intervention phases points to potential sex-related differences in hemodynamic trajectories. Nevertheless, these results remain preliminary, and larger, well-powered studies are required to determine whether these patterns reflect stable, sex-dependent developmental changes.
Intelligent perception with closed-loop information acquisition, processing, and feedback is critical for humanoid robots and embodied intelligence systems. Ionochromic transistors hold great potential for on-site signal processing and visual feedback. Here, we report a bioinspired ionochromic neuromorphic device with integrated signal-processing capabilities for intelligent perception and display. The transistor unit consisting of poly(3-hexylthiophene) (P3HT) and [EMIM][TFSI] (1-ethyl-3-methylimidazolium bis(trifluoromethylsulfonyl)imide) ion gel, achieves synchronous conductance modulation and reversible color change via voltage-controlled ion doping effects, mimicking biological synaptic response, and color regulation in chameleons. This dual-functional behavior originates from the generation of polarons/bipolarons that reconfigure the P3HT energy levels and modify optical transitions. The device exhibits a high-contrast electrochromic transition, with absorbance at 520 nm decreasing from 36.3% to 18.9%. Furthermore, a uniform electrolyte-gated transistors array enables tactile signal visualization, verified by a visual Morse code system and robotic hand integration, realizing in situ tactile perception and accurate object recognition. This work provides an alternative solution for integrated intelligent perception, advancing the development of next-generation human-machine interaction (HMI) platforms.
Sensory substitution devices (SSDs) translate information from one sensory modality into another to assist individuals with sensory impairments. This paper presents the first direct usability comparison of a visual-to-tactile SSD (BrainPort) and a visual-to-auditory SSD (Colorophone). Two studies were conducted with visually impaired participants. The pilot study used the BrainPort V100, and the main study used the BrainPort Vision Pro; the Colorophone mobile 1.1 was used in both. Participants completed comparable training protocols for each device, followed by usability testing based on scavenger-hunt tasks in naturalistic settings (kitchen, bedroom, toilet, and park). Usability was assessed via the System Usability Scale (SUS), task completion rates, task completion times, and qualitative structured interviews. In both studies, the Colorophone obtained significantly higher usability scores and satisfaction ratings than the BrainPort across nearly all participants, and showed better task-level time performance. The BrainPort showed better efficacy for some kitchen-based tasks requiring greater spatial precision. Using both devices simultaneously did not enhance usability and introduced device-incoherence issues. The Colorophone outperformed the BrainPort on most usability metrics. Suggestions for improving usability include redesigning the devices for hygiene and aesthetics, adding automatic camera adjustments, and enhancing integration between devices for combined use. Human-centered UX methods can inform the development of more usable and widely adoptable sensory substitution solutions. Prioritise user-centered, context-aware design and training. Select SSDs to match task demands and user preferences: deploy auditory Colorophone for rapid colour identification and categorisation; favour tactile BrainPort for spatially precise, gestalt-dependent tasks (e.g., shape discrimination, navigation). Provide targeted, scenario-based training that builds sensorimotor contingencies and manages cognitive load.Avoid simultaneous multi-SSD use. Combining BrainPort and Colorophone in parallel increased cognitive burden and reduced usability; rehabilitation protocols should favour single-device workflows with clear task handovers rather than concurrent streams.Address social, aesthetic, and equity factors as core design requirements. SSDs must complement existing non-visual adaptations, combat ableism, be customisable, discreet/acceptable in public use, and remain affordable and accessible to disadvantaged users to support real-world adoption and independence.Embed iterative usability evaluation into rehabilitation. Use standardised measures (e.g., workload, satisfaction, task accuracy/time) to refine device settings and training plans, tailoring to individual differences and real-life goals to sustain long-term functional gains without surgical intervention.