NAMAQAL

Shaqodoon’s Beneficiary Feedback system receives an average of 1500-3000 calls per day, with 125-800 callers leaving voicemail messages. Due to low literacy and internet connectivity, this telephone service is the most convenient way for beneficiaries to give feedback to humanitarian organizations about their on-the-ground programmes.

Continuous monitoring and manual review of calls can add considerable delays to the effective triaging of suggestions, inquiries and complaints, ultimately impacting on the delivery of core services.

Currently, there are no voice recognition systems that operate in Somali at the accuracy required for actual use. Developing a new Interactive Voice Response (IVR) tool, that can automatically process and categorize incoming voice calls, will enable Shaqodoon to monitor and triage calls, at scale, and ultimately serve the Somali project beneficiaries more effectively.

With appropriate language modifications, this tool could be scaled up to serve the needs of Somali communities elsewhere, and by other organisations globally.

Due to low literacy (38%) and internet penetration (1.2%) across Somalia, Shaqodoon’s Beneficiary Feedback System is the simplest way for beneficiaries to give a direct feedback to aid agencies about their programmes. 

The system receives between 1500-3000 voice calls per day, of which 125-800 are recorded and need manual review. An average of 460 recorded 30 second messages means 4 hours of voice recording per day. Adding an estimated 2 minutes, per message, to filter spam call (60-70% of total calls), transcribe and categorize each message, can mean it takes 15 hours to process a single days worth of messages.

Managing this scale of messages would require at least two full-time Management Information Systems (MIS) officers, at a cost of 2000 USD/month. In reality, due to limited resources, the system is impacted by response delays, vital calls being missed and a loss of trust by beneficiaries.

The platform has huge potential to monitor small and large scale projects, promoting accountability and transparency in program delivery. Capturing metrics such as age/ gender/ location also provide a valuable tool for organizations to understand and ensure that they are effectively reaching people who need the most help and addressing community needs. 

Shaqodoon provides ‘Technology for Development’ services to humanitarian and development agencies operating in Somalia/land. Our services reach rural, scattered and hard to reach communities that urgently need humanitarian and developmental assistance from aid agencies and government(s).

Since founding in 2011, over one million individual beneficiaries and over 20 implementing agencies, donors, government agencies and local organizations have engaged Shaqodoon’s services.

Our proposal will address core issues relating to effectively and timely addressing needs and concerns of communities. It will provide beneficiaries with greater access to implementing organizations and give them greater say and input on proposals for local projects and the allocation of funds, ultimately reaching those people who most need to be heard and included in the decision-making process. 

This will  mean that donor agencies are better placed to distribute limited resources to the most vulnerable. In addition, implementing organization/contractors can better monitor the effects of their projects on the ground.

  Finally, we continually seek feedback and monitor how our own services are taken up, gaining valuable insights to further improve our support to our clients

What makes our solution innovative:

The onslaught of daily recorded messages would require several full-time employees, which Shaqodoon and its client organisations cannot afford. We will develop an automated triaging system to prioritize voicemails on the IVR feedback service. Inspired by triaging practices in hospitals, this will ensure the focus goes on the most pressing needs, in a scalable way.

This system should:

• Detect the gender of the caller (females found to provide more reliable feedback).
• Detect recordings that are complaints (estimates to be around 2% of voicemails).
• Categorize each call by: complaints, appreciations, inquiry, suggestion and request.
• Transcribe the calls automatically.
• Prioritise / highlight the most urgent/important calls
• Where possible, translate the transcribed calls to English automatically.

Somali is completely ignored by existing automatic transcription systems. Of 1.6 million existing on Automatic Speech Recognition, only 3 consider Somali -- and the best is only trained on 1h30 of transcribed audio, with an abysmal 50% error rate. Even the largest multilingual system of word embeddings from Google (BERT, 104 languages) ignores Somali. Our unique dataset of IVRs make us uniquely placed to fill that ignorance.

How community and technology are integral to our solution:

The community built this opportunity! Indeed, our archive of 20k IVRs is a unique dataset, incredibly on-topic: real people, talking about real issues, with the real vocabulary. This is an opportunity to advance not just Somali transcription but to scale the response of multiple organisations to distress on real problems. And this is only possible thanks to the community who reached out to us.

On the technology side, we will apply machine learning and deep learning (DL) to automate the feedback categorization, transcription, and prioritization, in five steps:

1. Gender Classification: we know from our feedback score cards that feedback from women tends to be more accurate and trustworthy;
2. Automatic Speech Recognition: transcription from audio to somali text;
3. Word embeddings from Somali text to semantic representations;
4. Classification by category based on these representations; and finally
5. Translation for auditing by non-Somali-speaking partners.

Machine Learning, and more precisely Deep Learning over temporal sequences, is the current state of the art for such tasks in more widespread languages. Transcription is the real challenge, but our unique 20,000 IVRs will allow us to train Dilated Convolutional Neural Networks, Bidirectional LSTMs, Time-delay Neural Networks. This can be combined with a phoneme-detection intermediate step for extra efficiency.

Somali is completely ignored by existing automatic transcription systems. Of 1.6 million existing on Automatic Speech Recognition, only 3 consider Somali -- and the best is only trained on 1h30 of transcribed audio, with an abysmal 50% error rate. Even the largest multilingual system of word embeddings from Google (BERT, 104 languages) ignores Somali. Our unique dataset of IVRs make us uniquely placed to fill that ignorance.

This is the first time cutting edge technology in very low tech communities in Somalia will be used to enhance services to contribute to saving lives. The solution to solve the problems described are innovative and use the latest and new technologies. 

No Somali transcription models exist apart from early efforts in 2006 and 2017. The most recent work on automatic speech recognition of Somali, by researchers at Stellenbosch University and at UN Global Pulse Lab Kampala, while salutable and cutting edge, is limited by the extremely small amount of data available: 1.57 hours transcribed data, and 17.55 hours untranscribed. 

Our archive of 20k IVRs is a unique dataset, incredibly on-topic: real people, talking about real issues, with the real vocabulary. This is an opportunity to advance not just Somali transcription but to scale the response of multiple organisations to distress on real problems. And this is only possible thanks to the community who reached out to us.

We will apply machine learning and deep learning to automate the feedback categorization, transcription, and prioritization, in five steps:

1. Gender Classification: we know from our feedback score cards that feedback from women tends to be more accurate and trustworthy;
2. Automatic Speech Recognition: transcription from audio to somali text;
3. Word embeddings from Somali text to semantic representations;
4. Classification by category based on these representations; and finally
5. Translation for auditing by non-Somali-speaking partners.

Machine Learning, and more precisely Deep Learning over temporal sequences, is the current state of the art for such tasks in more widespread languages.

The transcription is the real challenge, but our unique 20,000 IVRs, totalling an estimated 166 hours, will allow us to train Dilated Convolutional Neural Networks, Bidirectional LSTMs, Time-delay Neural Networks. This can be combined with a phoneme-detection intermediate step for extra efficiency, as published at ICML 2018 by the French startup SNPS for privacy-preserving, data-efficient speech recognition pipelines. Similarly to the efforts from the team at Stellenbosch University, we can enrich this dataset by collected text from Somali news sites and social networks, to help learning the language model.

Once the transcription achieved, the classification by category is pretty straightforward: we can either use another Deep Learning approach, or fall back on classical topic models such as Latent Dirichlet Analysis.

With proper embeddings learned, the translation could be added as an extra task to existing multi-language translation models.

Somalia, has been ranked bottom of Transparency International’s Corruption Perceptions Index every year since 2006. According to Transparency International, corruption is a  leading cause, and consequence, of Somalia’s endemic political instability. It affects virtually every aspect of the Somali society: from public officials’ misuse of public goods for private gain and the solicitation of bribes in exchange for basic services to the clan-based patronage networks used to obtain employment and political appointments.

According to the World Bank, the likelihood of violent conflict increases when governments do not adequately prevent corruption or ensure justice. Corruption, and impunity for corruption, undermines the legitimacy of state institutions, and the impact of corruption on job opportunities and social cohesion can also lead to instability: corruption fuels grievances which can spill over into violence.

The Theory of Change (ToC) underpinning the ‘Triaging Interactive Voice Responses in the Somali language’ project is based on the notion that if beneficiaries and communities have direct access to implementing agencies and authorities, if donors have access to real time feedback from beneficiaries in an efficient and cost effective way, if donors are more accountable and responsive, if donors and implementing agencies take timely action to address the concerns raised by beneficiaries and communities, then mismanagement of funds are reduced, most needed will receive support, economic wellbeing of communities are revived, then tension are reduced, leading to stability and overall peace and prosperity in Somalia.

Shaqodoon beneficiary feedback service has been active since 2011. The feedback service currently serves 5-10 NGOs, donors and implementing agencies that provide humanitarian and development assistance to the most needy communities in Somalia; with beneficiaries accessing the service totalling around 30,000, annually.

In recent years, demand has grown for agencies to be more accountable and transparent to the communities that they serve. As a result, we expect five additional new clients a year to take up the service, leading to an estimated 20,000 beneficiaries accessing the services.

Over the next five years we expect to serve an additional 1,000,000 new beneficiaries with plans to scaled up to other countries with similar context as Somalia such as Ethiopia, Kenya, Djibouti, Yemen and Sudan with view of more new agencies benefiting from this advanced feedback mechanism to serve their beneficiaries effectively. 

In the next one to five years, we expect to have enhanced the existing feedback mechanism and user experience. We will grow the number of new service users and impact the lives of millions of people in Africa and the Middle East. In the first few years, we will achieve our primary goals by completing a number of key milestones.

In the first year, Shaqodoon will recruit qualified translators who will listen and translate the 20,000+ voice messages. They will also categorise and label them, with guidance from Element AI experts. Once complete it will be handed over to Element AI to train the system.

During this same period, inefficiencies will be addressed with feedback processing by reducing the time it takes to process each voice feedbacks from 6 feedback's per hour per agency to 10 per hour per agency.

  The user experience of the system will be enhanced for both beneficiaries and agencies, creating more trust in the system. As a result of this steady growth in clients interesting the services will be realised: increasing from 5-10 per year to 8-15 per year. We also expect more vulnerable communities heard and assisted, as a direct result of the solution. Year 2-5 of the solution being active will focus on expanding the service to at least 2 other Somali speaking countries such as Kenya northern regions and Ethiopia easten regions and Arabic to cover middle east countries in year 3 onwards

A specific barrier for the five years plan to scaling our operations is the sheer amount of human time required to listen and transcribe the 300-600 calls per client organisation per day -- currently 1500-3000 per day for our five client organisations. This is precisely what this AI solution aims to solve.

For this, the one-year barrier is that we need to label the 20,000 archived IVRs, each ranging from 1 second to 120 seconds of audio, with an average of 30 seconds. Estimating 5-10 minutes per voicemail to produce a full, reliable set of labels, will therefore involve a substantial effort of either a dedicated team working full-time for 2-3 months or a larger group of part-time labellers.

A final barrier at one-year is that  Natural Language Processing (NLP) is in full expansion, and still a research domain. This comes with a discovery risk. The current state of the art in the only 3 articles trying to transcribe Somali is an abysmal 50% word-error rate. 

Our plan to overcome the five-year barrier of scalability is precisely to develop and deploy the automated triage system described in this application.

Our plan to overcome the labeling effort is to enrol local translators. We can benefit from the proximity of a large English-speaking native population in the neighbouring Hargeisa University as a pool of talent, and use the potential prize money to fund labeling.

Finally, our plan to overcome the NLP challenges is the combination of our unique dataset and our partner Element AI’s technical expertise in machine learning -- both their AI for Good team, but also their dedicated NLP team and their research centre. If needs be, they can also count on the support from their Pr Bengio’s laboratory (Mila), with which they have successfully partnered in the past, with a joint team topping the leaderboard of a machine learning competition from the European Space Agency.

Shaqodoon is partnering with  Element AI’s AI for Good team to develop the solution for  this challenge. AI for Good is a dedicated group within Element AI, an AI software product company founded in 2016. While part of a for profit company, the AI for Good team applies its deep expertise in machine learning to help solve humanitarian, human rights and environmental problems on a non-profit basis.. 

Shaqodoon and Element AI recognise the social need for this solution to impact millions of lives. 

There will be two phases to this project;

• Phase 1: Labelling the achieve of voices files:
  • 30 Somali translators (familiar with major Somali dialects) working full-time to transcribe, translate and categorise for 3 months.

• Phase 2: Developing the AI and ML solution

Mustafa Othman, Communications and Technology Manager, Shaqodoon, and Julien Cornebise, Director of Research in the AI for Good team, Element AI, met in February 2019 at the Dagstuhl AI for Social Good seminar  This seminar united ML practitioners and organizations from the humanitarian, human rights and development sectors. Mustafa and Julien drafted a detailed proposal for this project. Element AI has continued to support Shaqodoon to find resources and solutions. 

Shaqodoon is a leading NGO with years of experience and local knowhow. Shaqodoon is the only organization that offers dedicated and customization solutions to aid agencies operating in Somalia and the only organization that provides SMS and IVR aggregators services as well partnerships with all Somali telecom companies. Shaqodoon has been providing feedback mechanism to humanitarian and development agencies since 2011 and has learnt first hand about the challenges our proposed initiative intends to address.

Element AI is an AI solutions provider that gives organizations unparalleled access to cutting-edge technology. They bring expertise in machine learning into the hands of domain experts such as Shaqodoon. They have a track record of delivering impact through partnerships with domain experts. Some of their work in NLP includes a resounding automated study of abuse against women on Twitter, done with Amnesty International. This received heavy media coverage and caused a 2.5 Billion USD shift in Twitter’s market capitalization overnight. Dr Cornebise was an early researcher at DeepMind, and is Honorary Associate Professor at University College London.

Existing clients for the feedback services include: Danish Refugee Council, BBC Media Action, Concern WorldWide, African Voices Foundation, Media Support, UNICEF, Spark and Oxfam Novib. Our solution would directly be added to the existing feedback service platform. Therefore, as existing clients, these organisations, and most importantly, their beneficiaries, would all immediately benefit from this automation.

Element AI -- a Canadian startup co-founded by Pr. Yoshua Bengio, Turing Award recipient -- and their AI for Good team is currently the technical partner that Shaqodoon is collaborating with for this solution. They bring their expertise on bringing machine learning into the hands of domain experts such as Shaqodoon. They have a track record of delivering impact through deep partnerships with domain experts. Some of their work includes a resounding automated study of abuse against women on Twitter, done with Amnesty International, which was heavily covered in the media and caused a 2.5 Billion USD shift in Twitter’s market capitalization overnight.

We are partnering with them through the design of this project by close communication with their machine learning team. We started this collaboration at the Dagstuhl AI for Social Good seminar in February 2019.

Shaqodoon is registered charity, registered in Somalia and Somaliland with registration no: L.43.13.327. The Technology for Development department charges services by providing technology services to development and humanitarian agencies operating in Somalia/land. All income is used to sustain Shaqodoon’s core services.

Shaqodoon offers the following services:

• Developing or customizing existing solutions.
• SMS and IVR services
• Dedicated reversed-charge short codes that are secure for client from partner telecoms and integrated to the system to receive feedback.
• SMS and Call minutes are charged to clients making it free for beneficiaries to call or SMS.
• Technical support and hosting maintenance
• Supporting  dedicated crowdfunding platforms for clients to raise match-funding for community projects (www.sokaab.com) integrated to mobile payments services.
• Money payment gateways to clients to receive and send money. 

Shaqodoon sells solutions and services to clients operating in the field of humanitarian and development sectors. An enhanced service and with lower cost feedback-processing is likely to generate more long-term clients. Currently, we have some clients who have used our service continuously for the last seven years, while others try it out for six months and leave. We want all of our clients to use our service for more than a year as this new enhanced service is rolled out.

For the initial investment for labeling and development of the solution, we will seek financing and support from donors and existing partners.

Once the initial support with labelling of voice message and development of the solution has been completed, we anticipate a steady growth in clients.

The proposed expansion of our services to Ethiopia, Yemen and Sudan will bring further revenue which will be reinvested into continuous service improvements.

With Solve by our side we believe it is worth more than the price money. By joining a supportive community of like minded people for expertise and mentorship will go a long way in making our solution a reality. We believe that by Shaqodoon getting the right media coverage and exposure from solve will also lead to many more partners coming onboard and using this service to enhance their service. Other areas Solve can assist is through networking events to countries where hope to expand these services such as event where governments, humanitarian and development agencies come together. Introduction and linkages to telecommunication companies operating these new countries that is of target.

n/a

We would like to partner with Stellenbosch University and the UN Global Pulse Kampala groups. To date these are the two institutions who published in 2018 and 2019 on Somali language transcription. Our partner Element AI can serve as a domain-translator, helping to bridge between our domain expertise and Pulse and Stellenboch’s specialised technical expertise (in addition to their own technical expertise).

By thus uniting multiple research groups in this joint effort, our solution and efforts could help reach critical mass for leadership in the applications of NLP to oft-overlooked African languages. If successful, this would be an important impact to this work, beyond the immediate beneficiaries discussed above.

We will be using AI and ML in our solutions which parts of it are new. The prize money will be used to label data, develop the solutions and purchase necessary hardware.

We are partnering with Element AI which is a very experienced company that has a specific department that focuses on AI for social good.

They will be contracted to develop the solutions working closely with Shaqodoon developers and our technical teams to bring this solution to the market.

Parts of the funds will also be used to promote the work to existing partners and potential new partners in Somalia.

It will also be used to expand at much faster past than if we did not have the funds to Ethiopia, Yemen and Sudan who are in similar status as Somalia and who can benefit from these services.