Humans in the Loop

Iva Gumnishka is the founder and CEO of the social enterprise Humans in the Loop. HITL connects refugees and conflict-affected people to remote work opportunities in the field of data annotation for Artificial Intelligence. Since 2017, the company has provided digital work opportunities to more than 300 refugees and IDPs in Bulgaria, Turkey, Syria and Iraq. 

Iva is passionate about using technology and AI for social good and she is also the Director of the Impact Tech Foundation where she has worked to organize pro bono programs at IT companies and hackathons. She is currently working on establishing standards for eliminating bias in AI through better annotation.

She holds a degree in Human Rights from Columbia University where she graduated summa cum laude and received honors for her thesis on refugee employment. She was also named Forbes 30 under 30 in 2018 and is a member of Ashoka's ChangemakerXChange.

65 million people worldwide have been displaced by conflict or violence, giving up their homes, education and work (Source:UNHCR). Due to language and administrative barriers, many are unable to access education or work in their new homes. 

Humans in the Loop provides conflict-affected people with training and employment opportunities in the Artificial Intelligence Industry. We are building the workforce of the future which annotates datasets and corrects model predictions, helping AI learn to think like a human. This provides much-needed income as well as new skills and experience as professional "humans in the loop".

Our vision is that conflict-affected communities could be transformed into "digital nomads" and could be made an active part of the latest technological advancements. By opening up opportunities in AI, there is a potential to "leapfrog" into the future and establish the whole Balkans and Middle East region as a leader in the field.

We refer to the target community that we are working with as "conflict-affected" people, which includes not only refugees and asylum-seekers but also internally displaced people and those currently living in places of armed conflict. Our target region are the Balkans and the Middle East, with 3.5 million refugees living in Turkey alone and 6 million people who are internally displaced in Syria and Iraq (source: UN OCHA). 

Armed conflict has caused a rise in unemployment levels in Syria and Iraq (currently at 8%) while refugees and asylum-seekers in Turkey and Balkan countries are facing severe administrative restrictions (only 50,000 people out of 2 million working age ones have a work permit in Turkey). Youth unemployment is even higher, reaching 20%, while female unemployment has been an enduring challenge in the whole region.

Due to work permit issues or lack of opportunities, many conflict-affected people end up working in the informal sector and are vulnerable to exploitation. Unfortunately, they have also been among the most hit by the COVID-19 economic downturn, with 200,000 people reverting to using cash support (UNHCR). This reliance on humanitarian assistance, even if extremely necessary in the short-term, can be harmful in the long term.

Our solution is to provide conflict-affected people with training and work opportunities that are entirely digital and can be accessed remotely. The work that we offer is extremely easy but it taps into one of the most innovative industries: Artificial Intelligence. 

We work as a B2B outsourcing company which connects directly with computer vision startups and researchers and supplies human input in order to help them improve their models. That includes activities such as dataset collection and annotation, edge case handling, and output verification.

Essentially, we train our workers to understand how AI works and to teach it how to think like a human by providing examples and correcting its mistakes. Our professional "humans in the loop" are trained to use a variety of annotation tools and platforms and to understand machine learning bias and how to avoid it.

The work that we offer does not require English capabilities and can be done from home so it's very appropriate for women and is resilient to situations like the COVID-19 lockdown during which our workers had uninterrupted access to work. In addition, it helps workers acquire essential computer and workplace skills that increase their professional opportunities in the future.

We primarily work with conflict-affected people living in Bulgaria (104 refugees and asylum-seekers), Turkey (39 refugees, including people with disabilities), Syria (80 internally displaced people and locals) and Iraq (95 internally displaced people and locals). The overall ratio of women in our workforce is 52% and the average age is 26, even though we have annotators who are 50+ as well and we do not focus on a particular age group. 

Thanks to our work with local NGOs, primarily the Roia Foundation and WorkWell, we are able to adjust our offering to the local context in each country, addressing issues such as internet connectivity and access to technology, payment processing, and training needs. We conduct monthly measurements of hours of work, wages paid, and trainings received across our entire workforce to make sure we are hitting our impact targets. In addition, most of our supervisors come from our target group and ensure a feedback loop.

In order to better understand the needs of our community, we conduct a yearly impact assessment which includes qualitative research through interviews and quantitative KPI evaluation. This year, the research is conducted in partnership with the Refugee Digital Livelihoods project at the University of Edinburgh.

Our project is focused on elevating a vulnerable community which comprises displaced and conflict-affected people. We like the term "digital nomads" because our beneficiaries are already facing displacement and a nomadic lifestyle but through digital remote work we are able to turn this into an empowering condition.

As the world has shifted to remote work, we want vulnerable people, no matter their skills level, to be made a central part of this tendency instead of being left out of it. By using our program as a springboard, we aim to enable them to reach many more opportunities in the future.

During my studies in the US between 2013 and 2017, I was watching the so-called refugee crisis in my home country Bulgaria from afar and I was really ashamed by the response of the government and the media. Refugees were stigmatized and their presence was framed as a threat, which led to the appearance of "migrant hunters" patrolling the Turkish-Bulgarian border.

In my last year of college, I became increasingly excited about returning back home and helping to re-frame the situation and create a semantic connection between "refugees" and "innovation", "technology", and "opportunity". Inspired by coding schools like Re:coded and ReDi and impact sourcing companies like Samasource and DDD, I set out to create a blended project where trainees would receive coding training for half a day and work on outsourcing projects the other half.

Naturally, the initial idea went through a lot of transformations, such as our change of focus from coding to more basic digital competencies due to the general skill level. We decided to go for the AI annotation niche which was perfect because it offered the right balance between accessibility and innovation. But the core spirit and the mission have remained the same ever since.

During my studies, I never imagined I would go back home and I was always dreaming of traveling the world and contributing to international development projects. However, as I learned more and more about postmodern views on "development" and the "white savior" industry, I became skeptical about changing the world, especially in locations and communities that are not "mine".

That's why my decision was to come back home after graduation to a community that I know and would be better positioned to navigate and change. However, for personal reasons, I have a very deep passion for the Middle East as well, and I've always been trying to steer and scale the project to neighboring countries where my approach is to partner with organizations working on the ground.

Currently, the most exciting development for me is the work that we are doing in Syria, which is severely underfunded by humanitarian organizations in comparison with Iraqi Kurdistan, Lebanon or Jordan. The political sanctions and the complex context make it a very challenging environment but I trust that providing work opportunities there can have a much more transformative effect than almost anywhere in the region.

One could say that I was pretty clueless when I started my project: I had never worked in any company and I became the founder of a startup where I had to reinvent the wheel and learn from scratch every time when something had to be done; I had never worked in any NGO and I became the director of a foundation where I had to acquire all of the knowledge as I went about what impact is and how it can be managed and scaled.

I started out with zero budget and zero experience, and managed to secure layer after layer of support and resources: we started out with a small training for 10 people taught entirely by me, we then secured some laptop donations and an office room, we began our first pilots with clients - again supervised directly by me, etc. etc. Very slowly, I began to step back as we expanded our operations and the core team grew but I still know each small piece of the company as the palm of my hand.

I could talk about my education, my mentors, the programs I have been part of - but nothing prepares you better to navigate the industry than the countless hours of calls with clients in order to understand their needs; and nothing is more valuable than the tens of conversations with our annotators in order to understand each person's story and strife and how you can be of real help.

One of the most important factors in the first year of the enterprise was a state subsidy that we were able to secure for the employment of 5 refugees. At that time, I imagined that job security, full-time work, and a community of colleagues would be something many refugees would appreciate after so much instability.

As we were growing, we had to hire more refugees on a freelance basis to help out with larger projects. So the full-time team was seeing that freelancers were working flexible hours from home and then earned a similar wage. At the end of the year, the entire full-time team dropped out, dissatisfied with the conditions.

For me, it was quite a commotion that led me to question the whole project and whether it was even worth it to continue. However, I accepted that my assumptions had been wrong and I resolved to shift our entire workforce to a freelance basis, since I realized that was much more attractive to workers. Little did we know that this set us up for success in terms of our business model, our scalability, and our resilience in the face of unexpected adversities in the future - like COVID-19.

As I was starting to develop my project in 2017 and I got to know the NGO ecosystem, I realized many of these organizations were feeling like competitors even though they were fighting for the same mission. So I thought it would be worth it to gather all of them for a month-long series of events called "The Month of Refugees" where each organization would be responsible for one event. 

That first edition was a blast. We brought together refugees and locals for movie screenings, concerts, shared dinners, poetry readings, a belly dance night, a hackathon, workshops, a "speed dating" session between refugees and locals... I ended up exhausted after the month and then resolved that I needed to focus on Humans in the Loop if I wanted it to become a reality and stop spreading myself too thin.

And then, in 2018, NGOs started calling me and asking about the Month of Refugees. They were excited about making it a tradition and another organization was happy to take the lead. This year will be the fourth edition and whoever the main coordinator is, it has been lovely seeing this "legacy" take on a life of its own.

We are building a completely new job category: a professional "human in the loop" who works to improve AI systems and teach them what it means to be human. Currently, Artificial Intelligence requires a lot of manual input in the form of annotations, such as bounding boxes or image tags, in order to recognize objects in an image. However, as AI advances, such tasks will be automated and humans will be required to teach even more advanced skills to AI, such as decision-making, reasoning, and emotions. 

Using the theory of “leapfrogging” that communities can make a quick jump in economic development by harnessing technological innovation, we aim to train those who are in the biggest need of job opportunities and make them an active part of the latest technological advancements. For example, together with our partners at Roia we are opening Syria's first Annotation Hub in Aleppo in July 2020.

There are many initiatives in the region doing amazing work in the livelihoods field which are focusing on other types of remote work such as Natakallam and Chatterbox (language teaching). There are also two similar refugee annotation platforms and apps like Taqaddam and WorkAround. What distinguishes us is that we couple training and employment through a unique partnership model with NGOs on the ground. In addition, while most similar livelihoods initiatives focus on refugees only, we also provide employment opportunities in the countries of origin and are currently the only organisation providing such work opportunities in Syria.

Goal:

To help conflict-affected communities be self-sufficient through digital employment and training in the field of "human-in-the-loop" jobs for AI

Objectives:

Over the next 5 years, at least 3000 conflict-affected persons (refugees, asylum-seekers, IDPs, locals, 50% of whom women) across the Balkans and the Middle East:

1. are working on digital "human-in-the-loop" jobs and earning a living wage for part-time work

2. achieve proficiency in transferable "human-in-the-loop" skills, including basic English, basic computer literacy and data labeling skills, different annotation platforms, and knowledge of the AI industry 

Outputs:

At least 3000 conflict-affected persons (at least 50% women) across the Balkans and the Middle East:

1. are working 40 hours per month on average on "human-in-the-loop" projects for continuous AI improvement through human input

2. have received 40 hours per month on average in trainings in basic English, basic computer literacy and data labeling skills, different annotation platforms, and knowledge of the AI industry

Activities:

1.1. Create partnerships and work actively with NGO partners in 8 countries across the Balkans and the Middle East (Bulgaria, Turkey, Iraq, Syria, Palestine, Greece, Jordan, Lebanon) 

1.2. Onboard 3000 workers who qualify as conflict-affected (either refugees, asylum-seekers, internally displaced or living in conflict zones) over the next 5 years 

1.3. Implement measures for gender inclusion in order to ensure at least 50% women representation in all recruitment and training activities

1.4. Secure a minimum of $1,000,000 per year from clients through business development activities and partnerships with research institutions

1.5. Provide at least 120,000 hours of work per year to conflict-affected people on a variety of platforms and tools for AI model training and verification

2.1. Develop and deliver 20 hours/week of trainings for basic English and general computer skills for the entire workforce in offline and online modules through NGO partners

2.2. Deliver 20 hours/week of trainings for specific AI annotation projects, including data labeling skills, working with annotation platforms, and understanding project guidelines 

2.3. Deliver a series of trainings to ensure a deep understanding of the AI industry among the workforce, including AI basics, understanding and recognizing bias, and the "human-in-the-loop" model

In our lifetime, since the beginning of our operations in March 2018, we have served 321 conflict-affected people across 4 countries in total. In 2020 only (up until end of May), we have provided 154 people with training and work.

In 2021, our goal is to provide training and employment opportunities for 400 people and to expand our partnerships to Greece and Palestine, where we are already in touch with potential partners.

In 2025, we will work with 1000 people in eight countries, scaling to Jordan and Lebanon.

Our aim is to provide a minimum of 3000 unique beneficiaries with work throughout this five year period.

We have had partnership inquiries around the world but we are keen on balancing expansion with maximizing our impact in the countries that we already work in, recognizing our role in contributing to the the post-conflict reconstruction of Syria and Iraq.

In 2021, our plan is to:

- Provide training and employment opportunities for 400 people (at least 50% of them women), impacting up to 1200 indirect beneficiaries such as dependents and family members

- Work with partners in 5 countries (Turkey, Syria, Iraq, Palestine, Greece)

Within the next five years, we will:

- Provide employment and training for 3000 people (at least 50% of them women), impacting up to 9000 indirect beneficiaries such as dependents and family members

- Work with partners in 7 countries (based on comprehensive needs assessment and partner selection processes)

In terms of our business model, we are planning to continue structuring our operations around remote work, which allows a great degree of flexibility to workers. However, we are exploring the idea of setting up Annotation hubs, such as our HQ in Bulgaria and the hub opening soon in Syria, where workers are able to attend in-person trainings and socialize, as well as access computers and reliable internet.

In terms of technology, we are extremely interested in exploring more ways in which humans can be plugged into the AI training pipeline and we are not ruling out the option to develop our own software if we see that it would bring a considerable business advantage.

1. We face a variety of challenges to our work on the ground which are influenced by the local economic situation and infrastructure, such as:

- Payment processing and legal constraints to employment, in addition to freelance work not being recognized officially

- Access to internet, computer equipment and reliable electricity

- Language barriers and low levels of skills, education and digital awareness among our target group

2. Given these challenges, we require very reliable partners on the ground which is a very hard thing to ensure. We have reviewed more than 10 NGOs in our target region but have so far only established a partnership with two, because we have very high standards and very specific requirements for the partners to fulfill.

3. Business Development is also a challenge because the industry that we are working in is very dynamic and growing very fast, and it is important for us to keep pace. We are competing against massive crowdsourcing platforms like Amazon's Mechanical Turk or Crowdflower, and also against large delivery centers who come from a traditional BPO background and are now shifting to AI data annotation. 4. In addition, companies are looking into ways to automate the work that our annotators do in order to reduce their costs for manual pre-processing, so we are developing ways in which our annotators can still be kept in the loop of developing technology and we can make sure their work is relevant. This will also create more long-term opportunities for them.

1. We believe the most effective way to address challenges on the ground is to work in partnership with local NGOs who know their context and people well. This is why the NGO partnership model is part and parcel of our work and central to our vision going forward.

2. We are continuously looking to identify new appropriate partners, and we believe that if we are successful in the SOLVE program, this will give us exposure and partnership opportunities to reach a number of other suitable NGOs in our region. 

3. In terms of business development, we are planning to approach more and more the academic sector, where we have seen a much more open reception of our social impact model and where we think we are well positioned to replace Mechanical Turk as the source of reference for data annotation.  Through Solve, we are hoping to extend our network in the academic community and to become involved in exciting new research projects.

4. Our focus is currently on developing ways in which our annotators can be involved in every stage of the AI training process, from collecting data and annotating it, to ongoing review and correction of machine-produced results.

This is a promising development not only because it will keep our workers relevant to our client's business by providing continuous improvement of their models through human supervision, but it will also ensure more long-term employment for our workers and more opportunities for us as an organisation.

We currently partner with two organisations for project delivery and work on the ground in the Middle East. These partners are responsible for recruitment, training and supervision of annotators; project management; and distributing payments.

- Roia, based in Istanbul and with operations in two cities in Turkey (Gaziantep and Istanbul) and three in the Syrian Arabic Republic (Aleppo, Raqqa, Idlib). They focus on ICT trainings and together we are opening the first ever annotation hub in Syria, based in Aleppo.

- Workwell, a programme of the Premptive Love Coalition in Iraq, with operations in three cities across Iraqi Kurdistan (Erbil, Suleimaniyah, Duhok) as well as Mosul. They also focus on ICT and microwork trainings.

In addition, we have established partnerships with several of the companies offering annotation tools, such as:

- Hasty

- Diffgram

- Trainingdata.io

- Darwin V7

- Supervise.ly

- Annotate.online

Since the services that we offer are complementary to theirs (we offer workforce, they offer software), these partnerships are a win-win situation for both parties. Through these partnerships, we receive free access to their annotation tools as well as introductions to prospects. We regularly publish reviews of annotation tools and we actively engage with the annotation landscape so as to keep up with the latest trends in the AI training data space. With many of these partners we have set up custom infrastructures for clients where our workforce has been incorporated in a pipeline for training data annotation and verification.

We work primarily as an outsourcing company, providing B2B services. Our clients are start-ups, medium sized companies and corporations, working in the artificial intelligence and computer vision fields. The majority of our clients are based in Western Europe and the United States, but also have clients in Australia, Singapore and Dubai.

We provide a complete suite of services for the machine learning model development process, including: collecting the ground truth data set; annotating data; verification of machine outputs; and edge case handling.

Many of our projects are one-off, because they depend on our partners' machine learning development cycle, but many of our clients have recurring needs as they improve and hone their models, and they need more batches of work to be prepared and labelled by us.

Our clients appreciate the fact that we work with dedicated teams who are trained specifically for their projects and are available on demand. Our teams are also scalable and can go from 2-3 people up to 40-50 depending on the needs of the client. We are also very flexible in terms of tooling and can use the client's in-house tools if they prefer.

When we start the project, we provide a free trial during which we evaluate the pricing and difficulty, after which we reach out to our partners, identify the most appropriate team and a dedicated project manager, and conduct trainings. We deliver projects in several iterations to allow clients to give ongoing feedback on our work.

Our goal is to be completely self-sustaining, and to rely on earned revenue through services. We have a very clear revenue stream by charging clients for every unit of service delivered (e.g. pricing per image, label, or hour of work). This is then broken down in the following way:

- 50% goes directly to the worker

- 5-10% goes to the partner NGO to cover project management and supervision.

- 40-45% remains within HITL to finance overheads and business development opportunities.

Because we hire annotators on a freelance basis, we have the capacity to take on a variety of projects at short notice, and our aim is to focus on business development to be able to offer work to as many people as possible. In addition, by working with several partners and our in-house team in Bulgaria, we are able to manage multiple projects in parallel without an excessive operations burden.

In the short-to medium- term we hope to support our business development work by securing grant funding and prize money to cover the time and costs of our core staff to develop new partnerships and scope new possibilities for scaling our activities. However, even without external funding our organization is already self-sustaining and able to cover all of its operations and sales costs.

We received a grant of $100,000 in 2019 from Dubai Expo Live Global Innovators towards the core costs of our organisation. This has given us the capacity to focus on developing new business opportunities and partnerships.

We have seen significant growth in income secured from clients over the three years since Humans in the Loop was founded:

- We started in September of 2017. The first months were focused on developing partnerships, understanding the needs of our beneficiary group and offering trainings together with the Red Cross. We also secured our first contracts worth $1,700

- In 2018 we started out with a small full-time team of annotators and we had a total revenue of $42,000 

- In 2019 we shifted to working on a fully freelance basis with our workforce and our total revenue more than doubled to $109,000

- So far in the first five months of 2020 we have secured $90,000 in deals and we are expecting to double our revenue from last year

We want our core business to be self-sustaining and to be independent of grants, and our target for revenue from clients in 2020 is $200,000, which is double our revenue for 2019.

In addition, we are looking for funding in order to be able to develop our training program, explore innovative ways to support our people and to increase the number of people that we support.

We are realistic about the fact that developing new partnerships and moving into new areas of growth can be time-intensive, and so we are looking for additional funds to support our core team to focus on this period of growth, with a focus on partnerships, business development and sales.

Our goal is to raise $100,000 of grant funding towards these costs by the end of 2020. We still have not considered debt or equity funding but we are open to exploring that option as well.

Our estimated expenses for 2020 are $160,000, with the following distribution:

- 35 % payments to annotators in partner countries, including NGO management fees

- 20% payments to annotators in Bulgaria

- 10% team salaries, including local supervisors

- 10% software and outside services

- 10 % social security/taxes

- 5% overheads, including rent, electricity, and other materials

Being selected as a Global Hero would give a great deal of recognition not only to me and my work but also to the great team behind Humans in the Loop and to the partnership network that we have created. 

Here are some of the most important potential impacts:

- As I think about our impact beyond our direct beneficiaries, it would be very relevant for us to connect to other organizations in the region and work together to turn it into an AI hub. Our contribution would be in the field of data generation but hopefully we can collaborate with institutions, startups, academia, NGOs, etc and contribute in our own ways to a shared agenda.

- The most direct benefit, even beyond prize money, would always be giving us more visibility, raising our profile, connecting with more clients, and ultimately securing more paid projects. That would increase our organizational capacity and contribute to our financial sustainability.

- Being a relatively new organisation, we would also hugely value the opportunity for mentorship, expert advice and guidance from the Elevate network, which will help us to plan the future direction and development of Humans in the Loop.

- The prize money would most likely be directed entirely into our partnership, business development and sales activities, with a particular focus on developing new partnerships in different geographical regions, and moving into new areas of growth

We are looking for mentorship on our revenue and business model in order to ensure more financial security for the organization and pick the best combination of funding sources. We would also love to discuss ways in which we can maximize the relevance of our work to companies working in the computer vision domain with subject matter experts.

We currently do not have a board of directors or an advisory board and we would love to set up one with the help of the Elevate network and community. We would love to establish one with a balance of experiences in scaling social impact solutions and launching and solidifying strong companies.

Finally, we would really appreciate the opportunity for marketing, media and exposure that Elevate provides in order to spread the word about our work and to help channel more digital remote work towards conflict-affected people, especially post-COVID-19.

We have identified a number of members of the Solve community who might be potential partners as we look to expand our work into different countries, such as Dignify (Columbia), Moringa School (Kenya), D2 (Bangladesh), Skills and Entrepreneurship (Nigeria), Butterframe (Uganda), Establishment of Digital Skill Centre (Tanzania) and Douar Tech (Morocco). We would be interested to initiate conversations to see whether our activities and goals align.

In terms of MIT Faculty and Initiatives, we are really interested in collaborating with the MIT Media Lab on exploring human input in AI, human-AI interaction, diversity in computer vision, and avoiding harmful biases. We would love to collaborate with computer vision and AI researchers across the MIT faculty, especially at the Computer Science & Artificial Intelligence Lab and the Center for Brains, Minds and Machines. 

We have been working on a project for cardiac CT and Xray annotation with the University of Leeds (UK) and we are negotiating a project for prostate lesions detection with UCL. We have a dedicated team of Syrian doctors who are refugees in Turkey and we are excited to explore more ways in which we can use their capabilities to advance medical research. That is why we are especially interested in connecting with the Medical Vision Group at MIT.