Expansion of TWB’s Multilingual conversational AI in Nigeria

TWB will expand the engagement for chatbots for COVID-19, by increasing the number of languages, using SMS, and including elements that increase engagement and referral rates.  The solution will particularly target women and will support the expansion of the solution to new contexts and prepare for future health challenges.

Sylvia Kaawe, Program Lead 

Arjun Thomas, Product Manager and Technical Lead

A recent TWB survey of humanitarian staff in over 30 emergency contexts found around 75% felt accurate content on COVID-19 was lacking in local languages and accessible formats to meet communities’ information needs. 

The 1.5 million displaced people who live in camps in northeast Nigeria speak 30 to 40 languages. The UN’s pre-COVID needs assessment found that over 41% of households couldn’t understand written information in Hausa, the main language of humanitarian communication.  Even before COVID, TWB found that 245,000 people struggled to get any information they understood. Official communication is, mostly, not in the first languages of local people, and trust is adversely affected by a negative history of relations between the region and the center of government. It is particularly difficult for women to access reliable information. 

Most people prefer face-to-face communications; COVID restrictions have made this nearly impossible. People turn to WhatsApp groups, friends and family, local radio, social media platforms such as Facebook, SMS messages by various actors and (often poorly informed) health communicators. 

That’s a problem because reliable communication is the foundation of controlling disease transmission and spread. The COVID-19 pandemic has led to an urgent need for new ways to communicate. 

TWB’s chatbots are designed to target people who speak marginalized languages. Shehu, for example, targets the 1.5 million displaced people in northeast Nigeria; it currently speaks 3 languages, making the content accessible to a large number of the displaced population. However, we have found that, because it relies on smartphone access, Shehu is mostly used by men.  In addition, it does not currently include one of the major languages, Fulfulde, of the population of the northeast, nor does it include the language that five million women in the east speak, Igbo.  Most of these women do have access to mobile phones and use SMS messaging. This grant would allow us to reach almost 500,000 more people in the northeast, in addition to about 5 million women who speak Igbo.

The team working on Shehu is based in NE Nigeria and speak the languages of the bot; we are designing interactions that engage people in a way that they are familiar, listening to them and responding to their questions in their own language in a way that feels natural.  Further engagement efforts will be developed by expanding this team, with support from the global team.

Communicating effectively in the languages and formats people understand is central to controlling the spread of diseases.  This innovation ultimately attempts to address the needs of, and take into account, the voices of the most marginalized and hard to reach communities by engaging them in languages they understand. The chatbot will answer their questions and address misinformation. It is available free of charge to anyone, focusing on people who speak marginalized languages and women. By enabling vulnerable populations to speak their mind directly and hear their voices, we will be better able to address their concerns, assisting them to make the best decisions for themselves, their families and their communities. 

One of the goals of the TWB chatbot program is to open up the insights derived from user conversations (in the form of information gaps, access to services, etc) to other organizations in the region to improve their services. By aggregating and analyzing information about chatbot user concerns and misinformation and providing it to health communicators, the solution supports policy makers to make data-driven decisions.  By developing and testing the technology now, we can also prepare the ground to respond to future disease outbreaks.

People who use TWB’s bots overwhelmingly find them useful and believe that it would be helpful for others.  They can ask questions they might be hesitant to ask an “official.”  Studies have found people are more likely to ask sensitive questions to a bot. In DRC, Uji was asked questions about mother-to-child transmission. TWB quickly added this information and other organizations adapted their messages to address this concern.  

In the short term, displaced people and all women throughout northeast and eastern Nigeria will have a reliable source of information. They’ll be able to remotely and confidentially get answers to their questions and feel free to ask questions they might be hesitant to ask others.

Their questions will be aggregated and made available to health communicators, along with answers based on content from WHO, CDC, and the Nigerian Ministry of Health. Because this will be in near real-time, the bots will become one of the go-to sources of information about disease prevention and will inform communications strategies.

In the medium term, TWB plans to deploy bots in over 15 countries where information in local languages is scarce, helping people to get verified information.

TWB purposely designed the chatbot to be agile.  The content back-end is based on FAQs.  TWB can use FAQs for any context, translate them and deploy them in any crisis.  We have also developed them so that they are easily updated by our trained linguists. 

During each deployment, we learn more about:

• What information people want and how to engage them

• Who uses the bots and in what ways

• What content is most useful 

• How to simplify the management of the bot and use tools to gain better insights. 

We plan to use funding from Trinity to 

• Expand the number of languages, targeting women. 

• Pilot a  new female “personality” for the bot, to determine if this increases women’s engagement.  

• Make Shehu available via SMS

• Add new features and improve reporting. 

In addition, we will determine if local conversational designers can help improve uptake and engagement. 

Using this learning, our plan is to scale the disease prevention bots, improving in Nigeria and expanding them to Bangladesh and three other countries. In total, we aim to cover 16 languages in 6 countries, reaching over 100 million people. The Trinity funding will also help TWB expand to expand to new countries.

The monitoring and evaluation framework for this project assesses the underlying technology and how well the chatbot tools contribute to improving two-way communication.  We measure effectiveness by tracking the number of unique views of our dashboard and corresponding reports. 

If successful, the chatbot will 

• understand people’s questions (measured by language identification and F1), 

• give relevant and appropriate responses (measured by user satisfaction), 

• instantly process and analyze high volume, high-quality data, resulting in timely and actionable recommendations for improving COVID-19 communication strategies (measured by new content responses)

• Measure, through quizzes and surveys, how people’s opinions change or knowledge increases by interacting with the chatbot (next phase)

In DRC, our pilot bot had the following target for key metrics (the bot’s current performance is in parentheses) 

• Number of users - 3000 (3180) 

• Repeat users - 600 (919)

• User Satisfaction - 70% (73.1%) 

• Language Identification Accuracy - 90% (95.1%) 

• Intent weighted F1 Score - 70% (60%) - (fluctuating) 

• # of new content responses produced and translated which directly address concerns or misinformation identified in the chatbot analysis - (periodic offline analysis)

The dashboard gives more granular information: language preference, channel preferences, most popular questions and topics, demographic data. 

• Users cannot access because of low connectivity: research best channels to ensure reach e.g. sms 

• High cost of messaging: TWB will work with local mobile network operators (MNO) to reduce fees and ensure that the messages are free to users

• Women do not have access to mobile phones or do not trust them: we have researched this and decided to expand to other languages and areas of the countries with higher mobile penetration rates.

• Financial: our business model assumes three years before the product is sustainable

• Finding or developing partnerships locally to maintain and develop the bots after initial set up.  We have struggled to find reliable organizations with which to partner in DRC.  In Nigeria and Bangladesh, because of  higher education and technological literacy levels, we believe that it will be easier to find and develop partnerships

Translators without Borders - Ireland

While we have a good plan, achieving those targets and having a funding model by building real partnerships with local organizations and local technology organizations are the biggest barriers to sustainability.  TWB can develop and deploy the bots, but over time, this isn’t sustainable.  A local organization should take on the maintenance and updating of the bot, in addition to developing bots for other purposes. Putting us in touch with Nigerian (and Kenyan, Ugandan, etc) organizations or with organizations that have successful partnerships would put us on a good footing.  In addition, support with the financial model - and with finances - is a boon.  We estimate that it will take 3 years before the model is sustainable.

We would like to deploy the solution in as many contexts and languages as possible in the very long term.  We need partners with use cases and content who can support the development of the bots.  We currently work closely with Internews, a leading community facing NGO,  in a number of contexts and projects.  We also have regular interaction with a number of other Trinity Challenge members.

In particular, we would like to work more closely with Gates Foundation partners.  We believe that part of solution for the “last mile” health care lays with ensuring that information is in the right language and ensuring that community health workers have access to information in local languages, making it easier for them to communicate complex information without having to interpret learning in one language when the speak to the people in their community.  One of the unintended impacts of the chatbot in DRC was that Red Cross volunteers found it easier to ask Uji questions in local languages so they could more easily convey concepts to the community members.