Audiority: An AI mobile app for the hearing impaired

A free, convenient, and robust AI mobile app that detects and classifies a variety of outdoor sounds for hearing-impaired users to use to navigate outdoors safely.

As 430 million in the world are hearing-impaired and hearing aids often do not work well on non-speech sounds, when the hearing impaired navigate outside, especially around busy streets and roads, this can impact their safety negatively. For example, on December 27th of 2016, a hearing-impaired woman was killed by an SUV as she was crossing the street. Devastatingly, according to researchers from JAMA otolaryngology, people who have trouble hearing are 90% more likely to be injured outside their homes. And two of the most dangerous sounds that are critical for humans to detect in order to move to safety are vehicle sounds and gunshots:1.35 million people each year are killed on roadways globally and 18,089 people in 2020 were killed by US firearm homicides. Hence, it’s imperative for the hearing-impaired to need a solution to detect these types of environmental sounds especially. However, most high-quality hearing aids are expensive: costing between $6,000 and $8,000, most hearing aids are virtually inaccessible to the low-income hearing-impaired community. In addition, they are usually designed to magnify human voices, not detect background noises like car horns or gunshots; in fact, hearing aids amplify and reduce noise and human speech simultaneously, making it difficult to hear each sound type separately. Further, ear wax buildup can block sound outlets of hearing aids, needing frequent cleaning, and ear moisture can easily damage the microphone portion of a hearing aid. Additionally, because a hearing aid is an electronic system based on acoustics, its performance will inevitably deteriorate over time. Besides hearing aids, there are not many alternative solutions that serve to detect non-word sounds. But seeing the progress that AI and machine learning (ML), new niche areas of technology, have already made for numerous other fields such as transportation, healthcare, etc., I turned to an AI-based solution to provide an alternative for the hearing impaired to detect non-word sounds, especially car horns and gunshots, during their outdoor navigation. In the area of environmental sound classification (ESC) in machine learning, I saw that many existing ML architectures are either inaccurate, naming a large number of sounds incorrectly, and/or too large with too many parameters to be deployed into a mobile app for users for real-time usage. For example, the AudioCLIP Transformer model has a 90.07% classification accuracy for outdoor sounds, but it has a whopping 30 million parameters) total. Therefore, developing an AI-based framework lightweight enough to be deployed into a portable application to identify outdoor sounds would be a more reliable and affordable solution for the hearing impaired in order to aid their safety. This app would be a free, accessible, and convenient solution for hearing-impaired members around the world.

Audiority is a machine learning model+mobile app that ensures hearing-impaired users will always have up-to-date information on different sounds surrounding them. When a user opens Audiority, they are prompted to press the “start recording” button to prompt their phone’s microphone to start detecting surrounding background noises around them. The audio is then processed and passed through a custom deep convolutional neural network(CNN) that's part of the MLSoundClassifier model. The model is trained on 3 critical outdoor sounds: ambulance siren, car horn, and gunshot, so when the phone’s microphone detects one of these 3, the model matches the sound heard to one of the 3 sounds it has learned and then outputs to the user the name and picture of the sound that is heard. The picture is included along with the sound label in case the user does not speak English. Afterward, the user can choose to record the history of the sound-that is, the exact date and time the sound occurred, in case they want to send out a police report to report accidents/incidents, etc. The Audiority mobile app is built using Apple’s frameworks XCode, Swift, and CreateML. XCode is the IDE platform used to host and write Swift code for the UI and design of the mobile app. CreateML includes a custom MLSoundClassifier model I built with, tweaked, trained, and used to learn to detect and classify outdoor sounds. I trained the model on a robust Kaggle audio dataset created by a Google Engineer, UrbanSound8K, in .wav format, including 929 sirens, 429 car horns, and 374 gunshot sounds. Because the dataset includes hundreds of different versions of these sounds, including of various lengths, pitches, and speeds, any slight variation in the real-life version of the sound will still be picked up and detected by the machine learning model.

Six years ago, my grandma lost part of her hearing: however, she still has trouble hearing sounds around her even when wearing the latest conventional hearing aid available in her country that she could afford with the money she had. When outside, a family member would need to walk with her to make sure she avoids incoming people, different vehicles, etc. But the trouble comes when no family member is available to aid her. 430 million people around the world are hearing impaired, and a lot of them are underserved, as they do not have the privilege of having a family member or friend to guide them when they navigate outside and/or can afford the most robust hearing aid. Hence, hearing aids are not accessible to the underserved portion of the hearing-impaired community. Therefore, I primarily aim to serve hearing-impaired people like my grandma: hearing-impaired users that live alone, need to navigate outside alone, do not have access to buy high-quality hearing aids, and have access to a phone, no internet connection needed. Audiority, my solution, is a free, accessible, and easy-to-use mobile app that hearing-impaired members anywhere around the world can download after I plan to publish it on the app store after testing and developing an Android version.

Throughout my 4-year journey in exploring technology and machine learning, whether through learning coding concepts in Youtube tutorials (such as HTML, CSS, Python, Swift) and school courses, creating skin cancer detection AI models in InspiritAI programs (taught by MIT students!:)), creating trash classification AI projects in the MIT FutureMakers Summer Program, conducting cancer segmentation research at the Harvard AIM Lab, and conducting research in detecting seizures with AI models in the Stanford EEGML Lab, I am awed every time I see code being applied to and improving progress in different fields. These discoveries and projects blew my mind, as I never knew something so esoteric as technology can be applied to such niche areas that can aid humanity. Therefore, I developed a strong passion to conduct research and create inventions to impact not just fields but also society. I plan to pursue a computer science major, learn more about coding, and further explore machine learning research and projects to create a truly unique innovation to serve the communities around me. To also provide others with knowledge and opportunities in tech, I became the president of my school technology clubs and founded a student organization, Bytes & Pieces (https://bytesandpieces.techno-jules.repl.co), to increase diversity, inclusion, and participation in the tech field, teaching both in-person and online Zoom lectures in topics ranging from web development concepts to how to build an AI solution. Currently, I grew my school tech clubs to 120+ members total and Bytes & Pieces has impacted over 400+ students. In the projects I've worked on, many of my AI projects stemmed from the struggles of people around me. After I traveled around to different beaches in the US, the amount of trash I saw shocked me. Passionate about environmental conservation, I created an AI model in the MIT FutureMakers program and later a web app to detect and classify types of trash that users of the app can use and record trash data they see to provide data scientists with more insight on littering behaviors of the people in each location. After my mom told a story of how a student in her high school class fell into bouts of seizures periodically, I decided to work on an AI solution in the Stanford EEGML to help detect seizures to determine their patterns to provide insight on how and when they would occur. After my great uncles passed away from lung cancer, I created 2 AI models to detect lung cancer from miRNA human data to provide an early and quick diagnosis of patients before their cancer worsens. After a few years, after I gained enough technical knowledge, especially in coding and AI, I created Audiority to provide an alternative solution for my grandma and other hearing-impaired people to aid their navigation and safety. Through my experience in teaching students in Bytes and Pieces and school clubs, many of the students I taught/hosted workshops for were from underserved communities (racial minority groups, gender minority groups, earing-impaired, mobility-impaired, Title 1 schools, international nations, etc), and many of my projects have target audiences that are underserved (people with cancers, people who are seizure-prone, the hearing-impaired), helping me be more diverse with my approaches and leading me to do more research for each individual group to deliver the best possible lecture format and/or technology project. For example, I added Zoom and Youtube recording captions to aid hearing-impaired students, recruited a diversity of workshop hosts and lecturers from Nigeria, India, Singapore, etc. to teach coding in Bytes and Pieces, and observed and interviewed my grandma on her experiences when walking or traveling outside her home with her hearing aids.

Currently, I have done research with and gathered information from my grandma, who is a prime candidate and potential hearing-impaired user of Audiority. As I am still in the prototype, testing, and optimization stage of my project, I plan to further interview the hearing-impaired students in my Bytes & Pieces organization to ask them about their experiences when navigating outside and suggestions they have about Audiority's design and development. In addition, afterward, I plan to have each potential user test out Audiority and provide feedback.

As mentioned previously, hearing aids are usually very expensive, do not detect non-word sounds like car horns well, and can deteriorate due to frequent ear moisture. On the other hand, a technological solution is free, can be trained to detect specific sounds accurately, and can last forever. In the area of environmental sound classification (ESC) in machine learning, many existing ML architectures that detect and classify sounds are either inaccurate, naming a large number of sounds incorrectly, and/or too large with too many parameters to be deployed into a mobile app for users for real-time usage. For example, the AudioCLIP Transformer model has a 90.07% classification accuracy for outdoor sounds, but it has a whopping 30 million parameters) total. Therefore, developing an AI-based framework lightweight enough to be deployed into a portable application to identify outdoor sounds would be a more reliable and affordable solution for the hearing impaired in order to aid their safety. Audiority combines the latest software of both the machine learning and mobile application development portions of tech creation, as it includes both a CNN machine learning model in CreateML that I trained to detect and classify non-word sounds and a mobile app using Swift and XCode that is UI-friendly for users to use easily. I also utilized an external Kaggle audio dataset, inputting its audio files into the CNN model for training and later into the app for sound detection testing. Lastly, I've optimized the model enough that it could detect and output a sound category within approximately 1 second for the user and can detect and classify siren, car horn, and gunshot sounds with 90% accuracy each time. Therefore, Audiority is a free, robust, accessible, convenient, and accurate technological solution for hearing-impaired members around the world. I do expect Audiority to change the AI solutions market, as it can inspire more machine learning engineers and programmers to create and publish unique, innovative products to serve niche communities using combined AI-app development software, perhaps to develop a solution to detect objects for the visually impaired, for example. Therefore, Audiority can help increase the number of hybrid AI-app development software solutions in the market that serve a certain community, which is a market that is currently very small and niche. Audiority can also enable broader positive impacts from others in this space since other technologists can also be inspired to utilize technology for social good and create similar products to Audiority, therefore increasing the number of socially good technological products for a variety of communities to try out and use.

After further testing and optimization on Audiority, such as adding more sounds it can detect such as engine idling and footsteps, my impact goals for next year are to (1) publish Audiority as a free iPhone app on the Apple App Store to serve at least 100 potential hearing-impaired users who have iPhones, (2) further expand my customer base by creating and publishing an Android version of the app and serve at least 100 users, (3) partner with the Hearing Loss Association of America (HLAA) to advertise Audiority and have at least 20-40 HLAA members test out my app, and (4) expand my customer base to hearing-impaired veterans especially, another underserved and large hearing-impaired community who may not be able to afford expensive hearing aids. I plan to gain at least 20-30 hearing-impaired veteran users. I plan to achieve the first point by researching how to publish an app on the App Store and advertise my app through Facebook and Youtube ads (popular social media platforms), achieve the second point by learning Java for Android app development and advertise it the same way, achieve the third point by reaching out to HLAA, providing a business proposal I wrote detailing the mission, purpose, descriptions, and benefits of Audiority, and pitching Audiority to HLAA members, and achieve the fourth point by reaching out to and pitching Audiority to members of the HLAA Veterans Across America Virtual Chapter.

The Audiority machine learning mobile app is built using the latest Apple frameworks, XCode, Swift, and CreateML. XCode is the IDE platform used to host and write Swift code for the UI and design of the mobile app, such as the check history, start, and stop recording buttons. CreateML includes a custom MLSoundClassifier model I built with, tweaked, trained, and used to learn to detect and classify outdoor sounds. I trained the model on a robust Kaggle.com audio dataset (UrbanSound8K) in the .wav format, including 929 siren sounds, 429 car horn sounds, and 374 gunshot sounds. Because the dataset includes hundreds of different versions of these sounds, including of various lengths, pitches, and speeds, any slight variation in the real-life version of the sound will still be picked up and detected by the model.

Audiority is currently still under testing, but I plan to launch this mobile app as soon as possible on both the Android and Apple app stores. In the next year, I expect my first customer to be my grandma, the next 3-5 customers to be hearing-impaired students of my organization, Bytes & Pieces (a student-run initiative where we teach coding workshops), and my next 40-70 customers to be HLAA members and members of the HLAA Veterans Across America Virtual Chapter. In addition, I expect to serve 200 iPhone and Android customers that I reach through online advertising. Therefore, in total, I expect to directly and meaningfully affect 240-280 customers within the first year of launch.

Although I have experience in the area of coding (4 years of experience) and machine learning (2.5 years of experience), I am currently still new to app development, so I plan to take a few weeks and/or months to learn Java for app development specifically to create an Android version of Audiority. In addition, I do not have a huge budget to increase the reach of Facebook or Youtube advertising even more, so I also plan to obtain sponsorships from various local and/or well-established technology companies.

I currently do not partner with any organizations, but I plan to partner with the Hearing Loss Association of America (HLAA) to learn about their organization's mission, how they are serving the hearing-impaired community, how my technological solution, Audiority, could provide a solution for HLAA members to try out as a tech "hearing aid," and how we can gain fundraising and sponsorships to further improve and advertise Audiority and advertise AI/app development tech solutions in general for the hearing impaired.

I plan to have Audiority be a social enterprise Limited Liability Company (LLC), as my primary goal is not to gain profit but to maximize the well-being and navigational safety of the hearing impaired. I also plan to establish a Freemium model, where my customers can use parts of the Audiority app for free but must pay for access to more advanced features. This latter model is more common for software products, such as Spotify. For example, I would provide the general audio detection feature in Audiority for free-mode users and open access to more detailed features in the app for premium subscribers (future features I would implement such as automatic messaging, location tagging, and reports to the police and/or friends and family). Potential users may want to consider Premium if they would like to file a report about location and sound information of dangers around them, such as a nearby gunman that fired a shot Audiority detected. This would also help hearing-impaired users gain more insight into certain patterns of dangerous sounds detected in their area.

My expected expenses include fees to publish Audiority to Apple and Android app stores and fees to pay for Youtube and Facebook Ads, and perhaps even advertising sales via email notifications. I plan to gain startup capital by utilizing a current Bold technology college scholarship of $1,015 I received last year, which can be utilized for initial funding, branding, and advertising of Audiority. After gaining the first 100 users, I hope to gain financial support/grants from tech companies such as Google, Meta, and Microsoft and the Department of Veteran Affairs. Afterward, I plan to gain sustained donations by partnering with a variety of hearing-impaired and technology nonprofits. 

If the revenue above does not happen to cover my expected expenses, I plan to add a premium (paid) mode to Audiority for users to gain access to more specific features, while the free version would contain the general audio detection features. We expect that at least 10% of our users would use the premium mode of our app, as they will be granted access to more varieties of sounds and features such as automatic messaging, location tagging, and reports to the police and/or friends and family. For non-premium subscribers, the revenue model would be periodic image and video ads online that would generate revenue.