HearMe

Web App that gives deaf/mute people access to inclusive education by allowing them to speak with sign languages with teachers on the platform and it will translate these movements into audio/text thanks to AI

In Madagascar, people with hearing disabilities are sidelined in society. According to statistics from the Federation of Deaf and Dumb in Madagascar, only 10% of these people have access to further education and only 6% are in the workforce. This problem is due to the marginalization of deaf and dumb people in Malagasy society, due to difficult communication with these people. Indeed, only 7% of the population has knowledge of Malagasy sign language. People with hearing disabilities are challenged to communicate daily, making even daily actions difficult.

But first we're just going to focus on Madagascar and then we're going to spread globally with ASL which is easier than Malagasy Sign Language.

HearMe offers a unique solution (in Madagascar and worldwide) using artificial intelligence to automatically translate sign language into text or audio. We will offer two services that will improve the community and professional life of deaf and/or dumb people. The first is a web-based artificial intelligence, which translates in real time through video capture, sign language movements into text or audio. The second is an online professional training module in Graphic design, Web writing and accounting for deaf and dumb people.

HearMe will help change the lives of the 600,000 deaf and mute people in Madagascar who are marginalized by society and the workplace. Our solution will be an indispensable and free tool for their daily life while offering a better professional future. A quality education that will be adapted for them, offering a better future perspective.

Our main clients are deaf and dumb people and their families, associations, NGOs, and institutions like the UN. As well as schools and companies.

The HearMe team is composed of 3 engineering students from the Ecole Superieur Polytechnique of Antananarivo, with expertise in artificial intelligence and machine learning.

1) Raharimalala Onitina Iavotriniaina, project leader, and president of a non-profit association helping marginalized people in society were able to work for one year with deaf and mute people. With two years of leadership experience, she has more than enough knowledge to carry out this project.
2) Rafaliniaina Radolaza Leondaris, the project's technical manager, is a Python teacher and junior data scientist in finance and has excellent technical skills to carry out such a large project. He was having worked and cooperated on different projects, and his skills in project realization are not to be denied.
3) Ranaivoarison Hariniaina Tsilavina, data manager, specializes in data manipulation and blockchain technologies. His data collection and cleaning knowledge is the main asset for artificial intelligence using millions of image data.

In order to understand the demands of deaf and dumb people in Madagascar, we have established contact with the federation of deaf and dumb people in Madagascar. As a result, we have found the main bases of the Malagasy sign language, a video database although weak but essential and the main difficulties that these people do daily. Thanks to this existing database, we implemented a prototype and tested it with the young deaf, and dumb people of the NGO AkanyFaly. Their reaction to HearMe has moved us all, although only a few words could be recognized by artificial intelligence, we believe we are on the right track for the future.

What makes HearMe innovative is:

First, we are the first to offer such a solution in Madagascar and build a comprehensive video database of Malagasy sign language.
HearMe improves accessibility: We can enhance accessibility for people who are deaf or hard of hearing, making it easier for them to communicate with people who do not know sign language.
Second, we improve education: we provide never-before-seen educational settings to help deaf or hard-of-hearing students better understand our courses because they are specifically designed for them.
Third, HearMe facilitates communication between people who do not share a commonly spoken language by providing a real-time translation of spoken language into sign language. And with future support for French and English sign language, HearMe will move closer to a global version.
Finally, we are using a brand new technology that can improve the accuracy of sign language translation by using machine learning algorithms to recognize and interpret a wide range of signs and gestures.

Our goal for the year 2023 is to reach the point of deployment of the number one version of HearMe. During the first 3 months of deployment, we expect 10,000 users per week.

Several core technologies are used to power HearMe, including

Computer vision: we use computer vision (OpenCV) to analyze and recognize signs and gestures from a video stream, such as from a webcam or smartphone camera. This involves using algorithms to analyze hand, arm, and facial movements, shapes, and positions, as well as other facial features, to accurately identify and translate signs and gestures into spoken or written language.

Machine Learning: HearMe uses machine learning algorithms to improve its accuracy over time. This involves training the algorithm on a large dataset of Malagasy sign language examples, allowing it to recognize and classify different signs and gestures more accurately.

Natural Language Processing: (NLP) techniques for recognizing and interpreting spoken language. This involves the use of algorithms to analyze the structure, syntax, and semantics of the spoken language, and to generate a translation into sign language or written language.

Overall, HearMe relies on a combination of these technologies to accurately recognize and translate Malagasy sign language in real-time.

Currently, HearMe has been tested with 5 deaf and mute people, tests that were successful with basic text recognition. The market is 600 000 deaf and mute people in Madagascar, we plan to reach 10 000 users every week in the first three months of launch.

Currently, there are two barriers that prevent us from realizing the solution for next year. The first one is financial, as the creation of the video database of the Malagasy sign language requires a high-capacity server. We are in a financial impasse, as the rental of such a server represents a significant investment and an adequate infrastructure. Also to mention only the funding needed for testing and deployment of hardware. Secondly, the technical barrier, because we will have to proceed to countless optimization of the algorithm so that the specific sign is recognized, this aspect we will need expert advice.

We are currently partnering with:

. Orange Digital Center Madagascar, a startup incubator that offers us training on entrepreneurship

. UN in Madagascar, being our first contact supports us from a legal and governmental point of view.

To achieve financial independence first, we will sell our translation services to our main clients, NGOs, associations, and large organizations such as the UN, as well as to deaf and dumb people and their families. Secondly, through the sale of professional training for people with hearing loss, we will be able to realize a stable source of income. In the future expansion of HearMe, English and French sign language will be added and will allow us to expand the solution worldwide.