Hearing aid through skin sensory for profound deaf people

Introduction

The World Health Organization (WHO) has adopted a grading system based on audiometric measurements to standardize the way in which severity of hearing loss of a person is reported. According to WHO,¹ a profound deafness is defined as a person who has a hearing threshold of 80 to 95 decibels (dB). A person with complete or total hearing loss has a threshold of 95dB or more. On the other hand, a normal hearing person has a hearing threshold of less than 20dB. Furthermore, they stated that almost 30 million people worldwide have profound or complete hearing loss in both ears. Based on their reported statistics on number of people with moderate or higher levels of hearing loss, 80% are from low income and middle-income countries as opposed to 20% that are from high-income countries.¹

A hearing aid or a cochlear implant is the common solution for individual that are profoundly or severely deaf.² The purpose of hearing aid is sound amplification; therefore, it is more suitable for mild hearing impairment.^3,4 This is due to the fact that a mild hearing impairment person can hear sound to a certain loudness but soft sound is hard to hear. On the other hand, a profound hearing lost person would not be able to hear anything. Besides that, the treatment based on cochlear implant is too costly and involves complex and risky surgery. In addition, hearing-impaired users still encounter substantial practical and social challenges with the use of this aid. At present, sign language^5-7 and speech-reading⁸ are commonly applied in schools for the deaf.

Recent studies show that the human ear and the auditory nerve are not the only channel for gaining audio signals.^10,11 There are several processes such as obtaining sound signals, translating voice signals to electrical signals, and producing frequency resolution can be achieved through the advancement of electronic devices. In other words, this additional tactile stimulus could help the profoundly deaf person wearing the electronic device to perceive sounds.

A study on multi-channel array skin-hearing technology was proposed which involved the stimulation of the skin by the electrical signal based on the sound.⁹ This enabled the profoundly deaf person to receive sound signals through the skin to accomplish the purpose of hearing the sound. The advantages of their technology were that it does not rely on intact hearing, and it was cost-effective. Furthermore, Li, J et al.¹⁰ showed that voice signal can be transferred using the cutaneous sensory nerves and it can also distinguish different speech signals. Their findings demonstrate that skin sensory nerves are a reliable replacement for the auditory nerve in the challenges facing individuals with severe-to-profound hearing loss.

A recent study¹¹ has shown that these individuals can learn to identify sounds that are algorithmically translated into patterns of vibration on the skin on the wrist. Furthermore, the users of the device could identify different sounds from the vibrations alone, which gradually improved over the course of one month. The analysis on the wearable sensory device unlocks the gateway to auditory stimuli, which could be interpreted through skin.¹¹ The most recent literature review demonstrates on the potential of haptic devices, which applies tactile stimulation to the user, are capable of replacing sensory information especially on people with partial or complete loss of hearing.¹² These devices send perceptual signals in the forms of forces or through electrical inputs to the user’s skin and body as skin plays an important role in perceiving and interacting with the environment. They elaborated the significance of developing these devices are worthwhile as it benefits the hearing impaired society in many ways such as experiencing music, alert the individual to critical sounds in their surroundings, etc. The aim of this study was to develop a hearing aid that stimulates “hearing” through skin sensory, which is more affordable and accessible for the profoundly deaf or total hearing loss community. Hence, we embarked into this research as studies on hearing through skin sensory seems to be a promising area for the beneficial of the society of profound hearing people in terms of safety and affordability.

Methods

The first phase of this study was to design a skin sensory stimulator by using a hand phone coin vibrator or micro actuator (MEMS) that allows the sound vibrations to be detected through the user’s skin. Figure 1 briefly demonstrates the stages of designing the human sensory stimulator.

Figure 1. Flow design of human sensory stimulator.

A Sound Transducer uses electrical energy to produce mechanical vibrations to disturb the surrounding air to produce sound regardless of whether the frequency is audible or inaudible. Audio Sound Transducer includes an input sensor that transforms sound into an electrical signal such as a microphone, and an output actuator that transforms the electrical signals back into sound, such as a loudspeaker. A preamplifier converts a weak electrical signal into an output signal, which is powerful enough to be noise tolerant. Absence of this will cause the output signal to be noisy or misleading. Due to this, the preamplifier is frequently located near to the sensor which reduces the effects of noise and other interference. A frequency filter is an electrical circuit that either stops or passes frequency from electrical signal with respect to specific frequency needed for further process. There are two types of filters, passive filter which uses passive components, and active filter that uses active components. Audio spectrum analysers are commonly used by sound engineers for various applications. Measuring the frequency response and analysing distortion characteristics of different types of audio tools are some of these applications. Micro actuators are active devices proficient of producing mechanical motion of solids or fluids. Hand phone coin vibrator is taken from the concept of cell phone vibration motor. We developed a simple single audio signal to vibration converter, which is based on the LM358 Driver, electret microphone and coin vibrator.

Sound sensor module gives a simple method to detect sound and it is usually used for the purpose of detecting sound intensity. It uses a microphone as a transducer which supplies the input audio signal to an amplifier, peak detector, and buffer. This module is made of three functional elements, the sensor on the front of the module will perform the pick-up (transducer), then the analogue signal is sent to the amplifier to amplify the signal. As an initial test device, a transducer vibrating device circuit is designed to study the response of subjects to vibrating motor or actuator to skin sensory. The circuit (Figure 2) utilises a LM358 operational amplifier. An electret microphone preamplifier will pick-up the audible signal from the surrounding and amplifies it through a first stage operational amplifier to a process able signal. The signal will then pass through a second stage amplifier comparator to increase the voltage level high enough to drive the vibrator or actuator. A transistor interface is used to drive the vibrator or actuator in accordance with the sound received from the electret microphone.

Figure 2. Single sound to vibration actuator driver.

This research consists of 3 phases which consists of design, development and testing for each phase that will be carried out at different level. In this paper, the initial testing model designed and developed based on LM358 operational amplifier works as pre-microphone amplifier and driving amplifier. This circuit functions as transducer to convert sound to electrical signal. This vibrating device will then vibrate according to the sound received or pick-up through the condenser microphone (CN1). The output (CN2) then will be amplified using a power amplifier to a higher power to drive the handphone coin vibrating motor/actuator. The circuit is then simulated to produce a PCB as shown in Figure 3.

Figure 3. PCB Single sound to vibration actuator driver.

The Sound to Vibration Converter (SBC) is build and encapsulated in a wearable plastic box. A total of 4 unit of prototype being built for this device. The prototype of the device as shown in Figure 4.

Figure 4. Prototype: Sound to Vibration Converter.

The next level improvement of the device will be using LM3915 audio level indicator (Figure 5) which the LEDs will be converted to hand phone coin vibrating devices. Accommodated with the adjustable voltage reference and an accurate ten-step voltage divider is the IC LM3915. The advantages of substituting conservative meters with an LED bar graph are providing a quicker respond, more rugged display with high visibility that provides the ease of interpretation of an analog display. The circuit design development and drawing done with NI Multisim Electronic CAD software. The schematic circuit diagram is as shown in Figure 5. Then there will be a follow up study where the initial design in Figure 5 will be upgraded to an audio spectrum analyzer (Figure 6) which can be designed by using Arduino processor for a multichannel wearable spectrum vibrator.

Figure 5. Schematic circuit diagram.

Figure 6. Audio spectrum analyser.

The spectrum analyser measures and displays the amplitude of a given input signal to a full frequency ranges with led matrix indicators. Its main purpose is to measure the strength of the spectrum of both known and unknown signals. This audio spectrum analyser enables us to see the frequencies present in audio or sound spectrum and produces the graph of all the frequencies that are present in a sound source in real time. The Arduino is a microcontroller board that contains everything needed to support the microcontroller to create a led audio spectrum matrix. The board is equipped with sets of digital and analogue input/output (I/O) pins that may be interfaced to various expansion boards (shields) and other circuits.

The initial experiment used Sound to Vibration Converter (SBC), which is a single overall sound to vibrate the device which will be used to stimulate the skin sensory. The electret microphone will pick up audio or sound signal which will feed to a preamplifier with adjustable gain of sound. This is to test the subject whether they can perceive sound through the vibrating motor or actuator. Upon completion of the second level design, the single vibrating device will then be upgraded to single rows of vibrators and then to full audio spectrum visible led matrix. The Audio spectrum led matrix would then be replaced with coin vibrator or actuator to give more precise vibrating pattern. The sound sensor can detect the frequency range from 20hz -20khz as this is the normal human hearing range. By using this type of sound sensor, it will enable the profoundly deaf people to “hear” the sound through their skin. The hand phone coin vibrators or micro actuator acts as an audio spectrum vibrator to create a vibrating pattern. This vibrating pattern will then be used to stimulate the human sensory system such as the skin for the profoundly deaf person, so they can use the vibration pattern as information. This repeating vibrating pattern stimulation will create a memory of a particular vibrating pattern for future recognition when similar sound is “heard” through the microphone.

The LED Matrix will have different types of frequency column representation and its own vibrating devices with its own sound intensity (dB) levels. Each column will indicate the particular frequency level representation based on how it is programmed. For example, if there is a sound present, the particular frequency of the led matrix column will light up to the maximum dB level picked up by the electret microphone. By changing the LEDs to vibrating stimulation, the profoundly deaf person can feel and “hear” the sound through his or her skin sensory system.

Results

Due to the recent outbreak of covid-19 pandemic, profoundly deaf individuals from different institutions, schools, or hospitals were not able to participate in our study.

Therefore, we randomly selected 30 people: 20 normal, seven partially deaf and three profoundly deaf (Underlying data),¹³ who were mostly friends and relatives that lived within a 10 km radius of the authors residential area. All the 30 participants have given a written consent and they voluntarily agreed to participate in this study. We developed four initial vibrating transducers with single spectrum, which were strapped to a belt. The main purpose of this experiment is to test them on a stimulus and if they feel a vibration on the stimulated direction they should give a hand signal which indicates the direction of the source of sound. The subject was given a passive hearing protector to reduce the direct audible sound from the speaker. This process is mainly to test the functionality of the device as a ground work for this research. Once the individual wears the belt on his/her waist (Figure 7) and the hearing protectors, the four transducers will be adjusted to be placed at the front, back, left, and right side of the individual’s waist (Figure 8). Audible sound will be played (stimulus) and the individual will be asked whether he/she felt a stimulation at the point of the transducer and if so, state the direction of the sound source verbally or through hand signal. Based on the experiments, both normal and the partial or profoundly deaf individuals felt the vibrating stimulation on their skin and accurately stated the directions of the sounds. This initial experiment was successful, which is an important direction for the future development of a hearing aid through human sensory skin. Further advancement of a more comprehensive device using the concept of spectrum analyser is now in progress. The sound signals will then be fed to multiple audio filters and audio spectrum analyser to produce sound audio spectrum to drive multiple vibration transducers or actuators.

Figure 7. Belt strapped to subject’s waist.

Figure 8. Position of the four transducers.

Conclusions

Despite the advancement of various hearing devices that improve the life of profoundly deaf people around the globe, there are still many limitations and drawbacks, which emphasizes the need to further improve such technologies. We have developed four initial vibrating transducers with single spectrum, which were strapped to a belt. Upon wearing the belt, the profoundly deaf persons are able to feel the vibrating stimulation on their skin and accurately stated the directions of the sounds. The results from this study have been promising for the first phase of the design, development and tests. This serves as a motivation for further development to the second phase of this hearing aid which is currently on going. The study of “hearing through the skin” is very promising and leads to affordable and safe solutions for the profoundly deaf people. As far as we are aware, adopting the concept of audio spectrum analyser for developing hearing aid has not been previously studied. Future studies should investigate and address how various types of sound stimulated through the skin by devices such as the belt used in this study, can be interpreted and identified by the profoundly deaf individual.

Ethical approval

The Research Ethics committee of Technology Transfer Office, Multimedia University has granted ethics approval for this study. Ethical Approval Number: EA1992021. All participants of this study gave informed written consent to take part in the research.

Data availability

Author contributions

Yasothei Suppiah: Conceptualization, investigation, supervision. M Chandran Maruthan: investigation, methodology, resources. Fazly Salleh Abas: validation, visualization, writing-original draft.