Accelerating crowd-sourced fact-checking chatbot system

Cofacts is a collaborative fact-cecking project with a chatbot in closed messaging app. It relies on volunteer editors to populate a database, so that the chatbot can send fact-checking replies to the chatbot users automatically.

While this system works in normal times, during the elections, the system was flooded by disinformation going viral so quickly that volunteers could hardly handle.

We plan to tackle this challenge with these measures:

1. Design an objective moderation standard to award participants in the volunteer editor community with reasonable renumerations for each new, eligible fact-checking report.

2. Form a compact, official fact-checking task force to produce fact-chekcing reports complying with International Fact Checking Network’s Code of Principles so that it provides an example for other editors to follow.

3. New software features like collaborative & automatic topic labeling to help editors efficiently reply user-reported messages.

The system is open-source so that it benefits everyone.

Disinformation propagated in closed messaging app is a global issue. Unlike public posts in Facebook or Twitter, disinformation in WhatsApp, Facebook Messenger and LINE is not traceable, can affect millions of people, and costs lives in harsh situations.

To make matters worse, disinformation can be used to undermine democracy of countries in the free world. According to the latest Varieties of Democracy dataset released in 2019, Taiwan ranked 1st in “foreign governments dissemination of false information”. The risk of Taiwanese democracy being affected by foreign disinformation is higher than Latvia’s, Kosovo’s and the US’s.

The collaborative fact-checking chatbot, Cofacts, tries to tackle this with chatbots in closed messaging app, and crowd-sourcing fact-checking reports. We have an Facebook group of 3,000 volunteer editors, and about 10 to 20 weekly active editors fact-checking and debunking user-reported instant messages.

However, during the 5 months before 2018 Taiwanese election, Cofacts chatbot received 6,000 user-reported messages, overwheming the volunteer editors. In 2018, the chatbot has only around 5,000 subscribers; As in 2019, the chatbot’s subscriber count has doubled, but not our volunteer editors. We are eager to empower volunteers to resist the incoming unprecedented wave of disinformation before 2020 Taiwanese election.

As a crowd-sourced fact-checking platform equipped with a chatbot, it serves two populations. First, the system empowers closed messaging app users to have better access to different opinions and fact-checking reports. Secondly, such platform also helps fact-checkers to deliver their work to readers that have just received hoax in closed messaging apps.

There are over 21 million people using LINE, a closed messaging app. That is over 90% of Taiwanese population. Many of them are digital immigrants that have little ability to find information sources by themselves. On the other hand, Taiwanese fact-checking organizations and individuals produce more than 1 fact-checking reports every day.

As a platform, Cofacts receives over 20,000 inquiries and delivers fact-checking reports to thousands of our chatbot users every month, whenever the chatbot user is in need. According to the user feedback collected inside the chatbot, users think the chatbot's reply is useful. If we can successfully establish a solid cooperative fact-checking model, such methodology can be applied to other closed messaging apps and countries to benefit more people worldwide.

We plan to deal with disinformation during elections by bulding a strongly motivated editor group. For now, we have Cofacts chatbots that delivers fact-checking replies to the hand of users who just received a piece of disinformation. We also have a voluntary editor community that is comprised of hundreds of netizens contributing fact-checking replies tackling with disinformation collected in closed messaging apps.

On top of the foundation we currently have, we will put all the resource to support our editor community and to facilitate the generation of solid replies. This includes:

1. Providing remuneration to the author of replies qualified by specified standards. This will facilitate the fact-checking work that is currently done by hundreds of unpaid volunteers on Cofacts. Qualification standards may include metrics like coverage of external references in the reply, number of chatbot users who think the reply is useful, and so on.
2. Organize a small task force (2~3 people) of skilled, proficient editors that composes fact-checking reports and commits to International Fact-Checking Network (IFCN)'s Code of Principles. The output of the task force will set example for the Cofacts editor community. We will also apply for IFCN verified signatories, so that we can gain access to more fact-checking resources world-wide.

We will also keep pushing on the development of our open-source crowd-sourced editor’s website and chatbot to maximize the influence of the editorial work mentioned above. This includes:

1. Crowd-sourced and machine-learning based tagging system that categorizes reported messages inside Cofacts editors’ website. By categorizing the messages into different topics such as “China related”, “HongKong related” and “Food safety”, volunteers with different expertise can get notified when latest message of their interest gets sent into the database, so that they can react quickly.

2. Cofacts chatbot users that has searched for a message will be able to receive updates of the message’s counter-narrative, as we implement the new notification mechanism in the chatbot.

Hundreds of the volunteer editors of Cofacts are doing a great job fighting disinformation in closed messaging app, and it is the time we bring the editor community to the next level. We believe that the above measures in the community and improvements made to the Cofacts system can greatly motivate volunteers to stand out and fight disinformation effectively and efficiently.

Traditionally, fact-checking work is done by professional fact-checkers with journalism backgorund. Although fact-checking reports from these institutions are highly credible and has high quality, it may come too late and is unlikely to reach the people in need.

Cofacts is an innovative solution in this field because it combines chatbots with a crowd-sourced fact-checking system. The chatbot acts as a probe to collect disinformation circulated in closed messaging network, also it is a direct channel to the people that are exposed to disinformation. Crowd-sourced replies enables public participation of countering disinformation, which also speeds up the process yo generate effective counter-narrative.

The core of Cofacts is a chatbot that informs its user about  crowd-sourced counter-narratives of online disinformation. Our solution leverages the following technologies:

• Chatbot: provides natural and native user experience inside closed messaging apps, where disinformation gets circulated and very hard to observe.
• Crowd-sourcing: provides the content of the chatbot in a diverse and reactive approach.
• Full text search and web scraping: when chatbot user receives a message from the user, it will match against similar messages inside the database. If the message contains hyperlinks, the textual content of the link is also considered, in order to adapt to the nature of how disinformation is propagated in closed messaging apps.
• Machine learning (text classification): we plan to build a topic classifier so that each user reported message can be categorized under a certain topic. This enables volunteer editors to subscribe to a certain topic and react more efficiently.

We have operated Cofacts chatbot since 2017 and build the voluntary editor community since 2018. As we shape the product, we learned that  timely and high quality replies really helps chatbot users a lot. On the other hand, editors' community are most motivated as product team provides rewards and leads the community by example. Thus, the effectiveness of this proposal can be shown in this Theory of Change:

Activity 1: Provide remuneration to the Cofacts editor that provides qualified replies

Output 1: Fact-checking narratives and corrections generated more quickly. Cofacts chatbot users reporting disinformation gets answered more timely.

Activity 2: Organize a fact-checking task force that works under IFCN Code of Principles. Apply for IFCN signed signatories.

Output 2: Sets standards for Cofacts editors' community.

Activity 3: Implement a crowd-sourced and machine-learning based tagging system

Output 3: Voluntary editors of Cofacts can focus on the topic they are good at answering

Activity 4: Implement notification mechanism to notify chatbot users of updates of the message’s counter-narrative

Output 4: Chatbot users are kept informed and equipped with latest counter-narratives to disinformation

Objective 1:
Cofacts chatbot users are willing to consult Cofacts and spread the counter-narratives

Objective 2:
Cofacts editors commnuity are motivated to provide more timely fact-checking replies.

Goal:
Counter disinformation in closed messaging app via crowd-sourced fact-checking chatbot system

Currently we are serving 100,000 chatbot users in Taiwan, while all Taiwanese people can come to contribute their reply to disinformation.

In 1 year, the number of chatbot users should increase to at least 150,000, according to our statistics in the past 12 months.

In 5 years, we expect to reach half a million of chatbot users in Taiwan.

On the other hand, we will be actively sharing the methodology and design of our solution to any country in need. Hopefully the outcome can affect democratic societies worldwide.

In order to deal with disinformation in closed messaging apps, merely fact-checking is not enough. We need counter-narratives that answers the doubt in people’s mind right on the spot, finds the right references so the audience can see for themselves, and it should be succinct enough to be consumed and circulated in closed messaging apps. The ultimate goal of Cofacts as a collaborative platform is to stop disinformation from spreading by facilitating the creation of counter-narratives, and efficiently distributing them.

Currently, Cofacts has put together a unilateral loop against disinformation. It pulls circulated messages outside of closed chatrooms and sends counter-narrative back to the closed messaging app.

Within the next year, we plan to accelerate the loop by implementing:

• A tagging system that allows editors to focus on topics they are interested in.
• A notification and subscription mechanism for volunteer editors to know about newly reported messages and replies that are marked “not useful”.
• A notification mechanism to notify chatbot users of updated replies.
• Motivation for the volunteers to keep contributing replies. This can be achieved by showing them how many people they have helped by their reply composed, provide a gamified leveling-up experience, or giving them adequate renumeration for composing a qualified reply.

For the next 5 years, we will keep improving the design of the above feedback loop as we evaluate its performance in each democratic election. We will also actively sharing the methodology and design of our solution to any country in need.

Currently all the developers, reply editors and even the product team are unpaid volunteers. All work, including writing counter-narratives, software development, communications with like-minded partners, are done using the free time after work.

In the following year, as the Cofacts chatbot user grows, more messages are being reported to the database. Although the editor community is growing as well, it can hardly catch up with the rising number of reported messages.

In the next five years, the impact of this project may be large enough to attract malicious editors to come to the platform and spread disinformation, leveraging the openess of editorial community. The volunteer editor community may be overwhelmed by the user-reported messages, and also possible conflicts between different editors. At this scale, the system can no longer be administered by unpaid part-time members in their free time.

The current capacity of our product team can only mitigate issues in tactical level. Knowing our limits, we designed our solution in overcome the short-term barriers. That is, we try to provide reasonable motivation for our volunteer editors, and ship out new functionality to Cofacts system to enhance the efficiency of Cofacts editors. The solution in this proposal is our plan to overcome the barrier at hand.

The barrier in the next five years can hardly be overcome without turning Cofacts product team into a stable organization. Although we do not have a schedule for this yet, we do consider running the community as an NGO, receives fundings and donations to have full-time members to work out the obstacles we can foresee.

N/A

2.___

Johnson has over decade experience to develop software program,also he is the full-stack developer of Cofacts project.

Billion has done most of the fact-checking progress in Cofacts projects and she has devoted her time in fighting against disinformation for about 3 years. She has edited most of the fact-checking reports.

Johnson has master degree in computer science.Billion was major in political science; as they combined their professional fields and gather more volunteers to cooperate together, it is efficient to harmful propaganda.Moreover, they have weekly offline meeting and daily meeting online, and would held bi-monthly editors meet up.

There is no other fake news system research innovation especially in closing chatroom, and as China-United State trade war become more and more severe, there are plenty of fake news and dangerous propaganda written in Simplified Chinese, from Chinese content farms or Wechat platform then received by Taiwanese people. Almost every Taiwanese people recognize Mandarin characters, resulting in all the fake news related to China would be disseminated in Taiwan social platform.

Johnson and Billion has facilitated the first ecosystem to reach disinformation hided in closing chat-room.They not only observe the phenomenon caused by disinformation but try to verify the false information sent by the users especially elders and average person who doesn't have enough expertise to realize messages from relatives might not be real.

Currently we are not belong to any organization, but when we apply for official account from LINE (a messaging app like what's app) social platform we rely on Open Culture Foundation's legal entity to fulfill contract progress.

Taiwan Fact-Check Center center is another organization established in 2018, some of their fact-checker would login Cofacts website to post their fact-checking report and collect popular suspicious message inside Cofacts website reported by LINE users. Billion was invited to the Tackling Misinformation workshop hold by Google and Taiwan Fact-Check Center.

Cooperated with Media Watch, being their lecturer to teach media literacy, like Taipei library community lectures for senior age and Media Literacy Camp for adolescent.

Recognized as partner of LINE company for Digital Responsibly Program. Help Taiwanese people having doubtful messages clarification.

Currently there is no business model. Cofacts system is developed and maintained as an open-source civic tech project with all unpaid volunteers, including the product team and editors.

The project recruits volunteers in offline events like bi-monthly hackathons held by g0v community and editors’ meetup.

Although we do consider forming an NGO, it probably will not happen within a year or two. When we do so, we may want to form a foundation like Wikipedia, or derive revenue from advertising like Snopes does, or sell hosted Cofacts system as a service to sustain, like Meedan does.

We are not a organization, and our team relies on voluntary work from people's skill and kindness, therefore, people has no responsibility to accomplish the progress by schedule.

First, we would pay for chatbot optimization progress, like automatic topic labeling to help editors reply user-reported hoax messages. The developer would develop a machine-learning based tagging system and new notification mechanism in the chatbot.

Second, the counter-narrative can be remunerated. We could encourage editor from profession field contribute expertise fact-checking reports on propaganda, medicine press, news and public administration.

Third, we could have more chance to share this open-source project to people might be interested in. Cofacts, as an open-project, more people use the open data, it might be the most productive outcome for us. Since Solve is welcome to the whole wide world, we hope any participant who is interested in solving problem related to fake news and misinformation could get benefit from our project.

N/A

Build international fact-checking network.

- DUKE REPORTERS' LAB and PunditFact leaded by Bill Adair, researching fake news, fact checking, and artificial intelligence application, 

 - FactCheck.org (A Project of The Annenberg Public Policy Center of the University of Pennsylvania),  

• Poynter, championing freedom of expression and democracy, having the International Fact-Checking Network, 

• The Nevada Independent , disseminating public information , 

•  Snopes, the oldest and largest fact-checking site online, having their own community like Cofacts in Taiwan, 
  

TruthOrFiction.com, debunking eRumors, classifying in a categorized way, 

N/A

N/A

N/A