Intangible software pulse data

In summary, the problem that we are trying to solve is that, in most low- and middle-income country (LMIC) health systems, performance measurement systems do not capture data from healthworkers themselves. This means that their voices, perspectives, needs, and challenges are not taken into consideration in routine performance management processes. This contributes to unconducive “intangible software”,  characterized by demotivated workers, weak teamwork, and low levels of trust. This undermines the performance of health systems and interventions introduced to improve it.

Accelerating progress towards Universal Health Coverage and the Sustainable Development goals, at a time of increasingly constrained resources, requires increasing the performance of healthcare providers, which is shown to be inadequate in many contexts. More than 33 million disability adjusted life years are lost annually in LMICs through inadequate care by providers (Rowe et al 2019).

However, it is increasingly understood that existing approaches to performance measurement and management in primary healthcare systems in LMICs are unsuccessful at driving organizational and population-level outcomes. This is confirmed by a recent evidence-gap map (Munar et al 2019). This mirrors findings in the broader public sector management and human resource management literature, which also report unintended and sometimes negative effects from performance management systems (Franco-Santos & Otley, 2018; Tweedie et al 2019).   

Recent thinking – including that published by the lead of this application (Newton-Lewis et al 2021) -  argues that current approaches to performance measurement and management in primary health care systems are ineffective because they ignore how the social environment of workers impacts on their performance. Data systems for performance measurement tend only to focus on “hard” data (resource availability and service delivery outputs), without data on the softer things that influence employee performance, such as motivation, teamwork, and trust (Das et al 2022). Indeed, they rarely collect any data on or from healthworkers at all.

Healthworkers, like all workers, are social creatures, working in social systems. The social environment impacts on how workers behave and interact with one another. This is particularly important in health systems, where workers need to work together in teams, use their discretion over how to be responsive to patients, and make complex decisions about how to make the best of scare resources. The social environment has an enormous impact on the quality and responsiveness of care provided by healthworkers (Nxumalo et al 2018).

As such, health systems need to create the enabling social conditions for positive behaviors and interactions. This requires what the health systems literature calls conducive “intangible software” – the informal institutions that influence actors within social systems (Sheikh et al 2011). Conducive intangible software is characterized by organizational cultures with high levels of trust, teamwork, motivation, shared commitment and values, and distributed leadership where teams and workers at the front line take collective responsibility for delivering responsive, high-quality services. Healthy intangible software is increasingly understood to be key for the resilience and shock-responsiveness of health systems, including in the response to COVID-19 (Barasa et al 2021).

Conducive intangible software can be fostered by appropriate performance management processes. The ways in which leaders behave and manage have a huge influence on the motivation and teamwork of workers; appropriate leadership styles can promote feelings of organizational reciprocity, leverage the intrinsic motivation of healthworkers, and promote teams to take shared responsibility for finding solutions to health system challenges (Orgill et al 2021).

However, in many LMICs, performance management looks like the complete opposite (Madlabana et al 2020). Managers tend to be overly directive, reliant on targets, carrot-and-stick based incentives, and the use of data to monitor compliance through audit-style approaches (Braithwaite et al 2017). There is overwhelming evidence that this can be counterproductive to performance, as it can demotivate workers, undermine teamwork, and promote empty compliance and data falsification (Hewko and Cummings 2016). This contributes to the high levels of burnout of healthworkers that have become painfully apparent during the COVID-19 pandemic.

Inappropriate performance management partly arises because data on “intangible software” is largely absent from performance measurement systems. Routine data review meetings at facility and administrative levels therefore miss the opportunity to review, make sense of, and problem solve for, data on the social determinants of worker behavior and performance. Not only does this make problems less visible, it also undermines the ability to monitor the effects of, and learn from, interventions (e.g. micro-innovations in management) to improve intangible software. 

Even the Primary Health Care Performance Initiative (PHCPI) does not include any indicators related to worker behavior and environment (even if some of the concepts are incorporated in the underlying conceptual framework). Whilst the related PHC Progression Model includes some dimensions of team-based care (measure 29), these focus more on formal structures (such as meeting frequency), rather than proximal measures of trust, motivation, and teamwork. In a recent USAID expert discussion (July 2022) on key indicators on human resources for health (HRH), USAID set out that most data systems for HRH do not collect any data from healthworkers themselves, and that this is a major gap.

This is emblematic of the wider problem that this proposal seeks to solve: the lack of data on elements of intangible software within routine performance measurement data systems, which contributes to ineffective performance management. Ultimately, this undermines the performance of health systems.

It is worth noting that the vast majority of interventions to use technology to reform data systems in primary healthcare focus on digitalizing the collection, transfer, and routine analysis of traditional data, such as strengthening HMIS, LMIS, and HRMIS. There are none to our knowledge that aim to use technology to collect new types of data on issues of intangible software. The comprehensive WHO classification of digital health interventions does not even include a category of interventions aiming to collect data on employee engagement and work climate (WHO 2018). We hope that our solution – if selected – will lead to a new category!

Citations for this section:

Rowe SY, Peters DH, Holloway KA, et al. A systematic review of the effectiveness of strategies to improve health care provider performance in low- and middle-income countries: methods and descriptive results. PLoS One 2019;14:e0217617. doi:10.1371/journal.pone.0217617 pmid:http://www.ncbi.nlm.nih.gov/pu...

Munar W, Snilstveit B, Aranda LE, et al. Evidence gap map of performance measurement and management in primary healthcare systems in low-income and middle-income countries. BMJ Glob Health 2019;4:e001451. doi:10.1136/bmjgh-2019-001451pmid:http://www.ncbi.nlm.nih.gov/pu...

Franco-Santos M, Otley D. Reviewing and theorizing the unintended consequences of performance management systems. Int J Manage Rev 2018;20:696–730.doi:10.1111/ijmr.12183

Tweedie D, Wild D, Rhodes C, et al. How does performance management affect workers? beyond human resource management and its critique. Int J Manage Rev 2019;21:76–96.doi:10.1111/ijmr.12177

Newton-Lewis T, Munar W, Chanturidze T. Performance management in complex adaptive systems: a conceptual framework for health systems. BMJ Global Health 2021;6:e005582.

Das P, Newton-Lewis T, Khalil K, Rajadhyaksha M, Nagpal P. How performance targets can ingrain a culture of 'performing out': An ethnography of two Indian primary healthcare facilities. Soc Sci Med. 2022 May;300:114489. doi: 10.1016/j.socscimed.2021.114489. Epub 2021 Oct 14. PMID: 34702616.

Nxumalo N, Gilson L, Goudge J, et al. Accountability mechanisms and the value of relationships: experiences of front-line managers at subnational level in Kenya and South Africa. BMJ Glob Health 2018;3:354. doi:10.1136/bmjgh-2018-000842pmid:http://www.ncbi.nlm.nih.gov/pu...

Sheikh K, Gilson L, Agyepong IA, et al. Building the field of health policy and systems research: framing the questions. PLoS Med 2011;8:e1001073. doi:10.1371/journal.pmed.1001073pmid:http://www.ncbi.nlm.nih.gov/pu...

Barasa EW, Cloete K, Gilson L. From bouncing back, to nurturing emergence: reframing the concept of resilience in health systems strengthening. Health Policy Plan 2017;32:91–4.doi:10.1093/heapol/czx118

Orgill M, Marchal B, Shung-King M, et al. Bottom-up innovation for health management capacity development: a qualitative case study in a South African health district. BMC Public Health 2021;21:587. doi:10.1186/s12889-021-10546-w

Madlabana CZ, Mashamba-Thompson TP, Petersen I. Performance management methods and practices among nurses in primary health care settings: a systematic scoping review protocol. Syst Rev 2020;9:40. doi:10.1186/s13643-020-01294-wpmid:http://www.ncbi.nlm.nih.gov/pu...

Braithwaite J, Hibbert P, Blakely B, et al. Health system frameworks and performance indicators in eight countries: a comparative international analysis. SAGE Open Med 2017;5:1–10.doi:10.1177/2050312116686516pmid:http://www.ncbi.nlm.nih.gov/pu... Scholar

Hewko SJ, Cummings GG. Performance management in healthcare: a critical analysis. Leadersh Health Serv 2016;29:52–68.doi:10.1108/LHS-12-2014-0081

WHO. Classification of digital health interventions. Geneva: World Health Organization; 2018 (WHO/RHR/18.06). 

Our solution is to develop a prototype of a system that routinely collects pulse data on intangible software from healthworkers through SMS surveys, and that automatically generates data dashboards and visualizations for managers at a decentralized level, so that the data can be integrated into routine performance management processes (such as data reviews, quality improvement meetings, and management meetings). Therefore, it is both about improving performance measurement and performance management.

We would pilot this system on existing FHI 360 led projects where we support government stakeholders in performance management processes to improve primary care - tentatively in Mali, Mozambique, Uganda, and Cambodia. This will enable us to leverage existing systems, networks, and workflows for data collection, sense-making, and performance management. This will enable the system to be refined and help us to learn about how best to support government managers at different levels to act upon this type of information. If the pilots are a success, they can be taken up by these projects for scale and institutionalization.

In this section, we set out:

• What is a pulse survey
• The system we envision
• The human centered design process by which we will design the system
• The process by which we will test the system on ongoing projects

 1) What is a pulse survey

The proposed solution is inspired by approaches from the private sector, where organizations have embraced the opportunities that arise from digital technologies to invest in frequent collection of data on employee engagement through technology based “pulse” surveys. These organizations integrate this data into their performance management and decision-making structures.

A pulse survey is a short set of questions sent frequently to employees to get continual feedback on dimensions of employee engagement (such as motivation, burnout, their overall work environment, communication, relationships, and management) – a concept that overlaps with the health systems concept of intangible software. The human resources management literature shows that pulse surveys can generate a high response rate compared to longer, less frequent surveys; and that the short turnaround nature of the data is more effective for continual improvement (Welbourne 2016). Data is close to real time (e.g., pulse surveys are administered weekly or monthly) and their repeat nature allows for improvements to be quickly measured after interventions are implemented.

As such, pulse surveys both make issues of employee engagement visible and provide a basis for continued learning on attempts to improve it (Jolton and Klein 2020). This can support leaders and teams to dive deeper and uncover underlying causes of low engagement and inconducive software (e.g., through seeking additional qualitative data), as well as support continual improvement.

Such surveys have a secondary benefit in that the act of being asked for feedback can motivate staff, promote feelings of organizational reciprocity, and itself increase engagement (see for example, this article in the Harvard Business Review - Employee Surveys Are Still One of the Best Ways to Measure Engagement (hbr.org)). Further, the integration of employee engagement data into routine meetings can create the spaces for participatory discussions on issues of work climate, which can directly improve teamwork and trust.

A pulse survey generally includes a small number of questions, given the frequency, to avoid survey fatigue. These are answered by workers on their computers or phones. A variety of technologies are used, including forms and polls integrated into WhatsApp, and web-based surveys (these are primarily to date used in higher-income contexts with better technology and connectivity). Data flows onto a central server. Data is anonymous and aggregated, ensuring worker confidentiality. Data systems have dashboards and visualization tools to enable data interpretation and sense-making.

2) The system we envision

We will leverage globally available digital survey platforms to administer a survey tool that measures different dimensions of intangible software (the process for developing the questions is described below). We will use an existing platform because they offer templates for fast design, development and optimization of surveys – including the ability to test, iterate and analyze results in preparation for the final survey launch. These existing solutions have interfaces already built of working with different mobile operators as well as different HIS systems in country. Their technology has user confidentiality built in, tools to help track delivery of messages as well as in built tools to troubleshoot and monitor any user queries. We work with a number of existing platform providers and will select the optimal one after the human centered design process set out below.

The survey will be sent to the mobile phones of healthworkers through SMSs on a monthly basis (or at a frequency determined by the human centered design process described below). This is a low-cost solution.

We are envisioning using SMS because, from our experience, healthworkers in many countries do not have smartphones or continuous internet access, which rules out other options such as WhatsApp. Using “brick” phones will ensure the highest possible reach. SMSs are a low cost solution. This will be discussed and debated with healthworkers in the pilot contexts during the design phase; it would also be possible to use IVRS in areas with low literacy. We intend to design a flexible system that could potentially send out survey messages through WhatsApp rather than SMS in contexts where this was more appropriate. In businesses in high-income countries, pulse surveys are frequently disseminated through WhatsApp (either using in built forms or web links sent through messages). Because of this, we will select a platform that can offer omnichannel message and scalability for further user engagement options.

Data reviews – whether at management meetings or quality improvement meetings – happen at different levels, including facilities and sub-national administrative areas (such as a district). We will design the system so that it can be used for different types of catchments, whether all staff in a large facility, or all community healthworkers in a sub-district.

The data will be aggregated by the server for the unit of interest (e.g., facility) and calculate aggregate figures in a dashboard, with in-built visualizations – of levels, and changes over time. The data will not be available by individual respondent to preserve confidentiality and will only aggregate at units large enough so that anonymity is guaranteed.

Data and visualizations will be provided to government managers at the relevant level (district, facility etc.) to enable data interpretation and sense-making during data review or equivalent meetings. Sense-making – the social process by which teams collectively interpret the data and its meaning – is shown in the literature to itself contribute to building more conducive intangible software (Orgill et al 2021).

Whilst here we are proposing to build a prototype system, we will also ensure that the data could potentially be integrated into government HMIS/DHIS2 information systems, so that it is sharable across information systems. The SMS platform we envision is designed to manage data interfaces seamlessly, providing managers with easy access to aggregate data. We have considerable experience in working with government information systems in LMICs, integrating new data types and sources, and promoting and ensuring interoperability. As such, in the long run, the system could be institutionalized within government systems, which will be a major contributor to scalability.

3) The human centered design process by which we will design the system

As a recent article by the lead of this application shows, most digital health interventions aimed at healthworkers fail to have their anticipated impact (Newton-Lewis & Nanda, 2021). One of the reasons for this is that healthworkers are not involved in the design of digital interventions, which are too often done ‘to’ workers. This has been acknowledged by the WHO in their work on digital health interventions and health workforce capacity building (WHO 2019).

Whilst this intervention is inherently about amplifying the voices and predicaments of healthworkers and highlighting the intangible software issues that many digital health interventions fail to take into consideration, it too needs to involve healthworkers in its design. This will ensure that it fits with their workloads and workflows, modalities of reviewing data and sense-making, and ways of engaging with their managers. This requires human centered design, whose principles are grounded in the proposed process. We will ensure that healthworkers with low digital literacy and those that come from marginalized groups are included in the design processes.

We will also engage and involve relevant senior government officials in the pilot countries; this could start discussions around institutionalization and scale; discussed in detail in a later section of this application.

At the start of the process, we will develop a menu of indicators and questions that can be used in the pulse survey. These will be derived from a review of the literature, and our own experience. They will cover elements of both the worker individual situation, and the workers’ perceptions of their social environment and their management. The questions have to be simple, and few in number, for a pulse survey to work. However, many of the constructs of interest are difficult to measure because they are latent variables and not easily observable, and can often prompt a social desirability bias in their reporting. In other settings, we have used Likert (e.g. 1-10) scales that have been validated in cross country contexts amongst healthworkers, including:

• Measures of motivation (e.g. from Vallieres et al 2020 & Gottert et al 2021)
• Measures of work climate (which includes things like teamwork, distributed leadership) (e.g. from Stanton et al 2002)
• Measures of trust within systems (e.g. derived from Sripad et al 2021)
• Measures of perceived supportiveness of supervision – for example derived from the perceived supervision scale (Vallieres et al 2018)

These will be discussed with workers in the pilot contexts, who will help select individual questions and constructs that they feel are relevant and useful for their situation. We would then pre-test appropriately translated versions of the questions to see how well they work in the different contexts, including through cognitive testing and field validation. We are planning on using a software that allows for multiple language  translations so users respond to surveys in the language of their choice.

Secondly, we will convene groups of workers to discuss the best technological options for administering the survey and co-produce the design. As set out above, we assume (given our experience in each of the contexts) that SMSs will be the best mode in contexts with limited smartphone penetration and infrequent internet access. We will also get input on the best forms of data outputs and visualizations.

This human centered design process will enable us to select the optimal platform and provider for building the prototype system.

4) The process by which we will test the system on ongoing projects

We will pilot the system on existing FHI 360 led projects where we support government officials at various levels to review data as part of performance management processes to improve primary care. This will be driven by the projects. As such, we will leverage existing systems, networks, and workflows to streamline data collection, sense-making, and performance management. In these countries, we already legitimately have access to the databases of workers and their phone numbers through existing MoUs etc.

These include (exact number to depend on the level of funding received):

• Cambodia: FHI 360 is supporting client-centered quality improvement collaboratives between public and private facilities in six provinces focused on family planning; we have QI coaches deployed to facilitate facility teams to use data to identify priorities and test and scale improvements. Capturing data from healthworkers would complement the project’s innovations in collecting patient satisfaction data through tablets positioned at facilities (and we would also learn from their experience of framing appropriate questions and supporting the sense-making of this kind of data).
• In Mali, FHI 360 are supporting community health centers and districts to incorporate quality improvement processes, including the review and sense-making of routine data, to strengthen primary care
• In Mozambique, in Nampula, FHI 360 are leading a consortium of partners to use best practices in quality improvement and data use to support health facility and district teams to identify, test, and scale health system and service delivery improvements focused on maternal and child health.
• In Uganda, FHI 360 are supporting regional platforms to undertake rapid analysis and action cycles to improve maternal and child health and nutrition

In each context, we will involve healthworkers in the participatory evaluation of the pilot, to get their feedback on how well it worked, how useful the data visualization and use processes were, and what could be done to improve the system in the future. Senior policy makers will also be involved given the potential for scale and institutionalization of these projects. This process would be managed by these projects.

We will also develop an M&E system to document the use of the data in the data review meetings. The purpose of this project would be to both develop a prototype system that could be scaled elsewhere, but also start to learn about how to provide support to the sense-making process in meetings where the data is used so that it can inform change in management and other HR practices.

Citations for this section:

Welbourne, Theresa. (2016). The Potential of Pulse Surveys: Transforming Surveys into Leadership Tools. Employment Relations Today. 43. 33-39. 10.1002/ert.21548.

Jolton, J. & Klein, C. Exploring the Universe of Pulse Surveys and Continuous Listening Opportunities in: Employee Surveys and Sensing. Edited by Macey, W. & Fink, A. Oxford University Press (2020).

Orgill M, Marchal B, Shung-King M, et al. Bottom-up innovation for health management capacity development: a qualitative case study in a South African health district. BMC Public Health 2021;21:587. doi:10.1186/s12889-021-10546-w

Newton-Lewis T, Nanda P. Problematic problem diagnostics: why digital health interventions for community health workers do not always achieve their desired impactBMJ Global Health 2021;6:e005942.

World Health Organisation. Report of the consultation meeting on digital health interventions and health workforce capacity building, 2019

Vallieres, F., Kok, M., Mahmud, I., Sarker, M., Jeacocke, P., Karuga, R., . . . Taegtmeyer, M. (2020). Measuring motivation among close-to-community health workers: developing the CTC Provider Motivational Indicator Scale across six countries. Human Resources for Health, 18, 54. doi:https://doi-org.ezproxyberklee.flo.org/10.1186/s12960...

Gottert A, McClair TL, Hossain S, Dakouo SP, Abuya T, Kirk K, Bellows B, Agarwal S, Kennedy S, Warren C, Sripad P. Development and validation of a multi-dimensional scale to assess community health worker motivation. J Glob Health. 2021 Mar 10;11:07008. doi: 10.7189/jogh.11.07008.

Stanton, J., Sinar, E., Balzer, W., Julian, A., Thoresen, P., Aziz, S., . . . Smith, P. (2002). Development of a Compact Measure of Job Satisfaction: The Abridged Job Descriptive Index. Educational and Psychological Measurement, 62(1), 173-191. doi:https://doi-org.ezproxyberklee.flo.org/10.1177/001316...

Sripad P, McClair TL, Casseus A, Hossain S, Abuya T, Gottert A. Measuring client trust in community health workers: A multi-country validation study. J Glob Health. 2021 Mar 10;11:07009. doi: 10.7189/jogh.11.07009

Vallieres, F., Hyland, P., McAuliffe, E., Mahmud, I., Tulloch, O., Walker, P., & Taegtmayer, M. (2018). A new tool to measure approaches to supervision from the perspective of community health workers: a prospective, longitudinal, validation study in seven countries. BMC Health Services Research, 18, 806. doi:https://doi-org.ezproxyberklee.flo.org/10.1186/s12913...

The solution will focus on capturing data on the motivation and social environment of healthworkers, so that this data is taken into account in performance management processes. As set out above, this information is not currently collected in most performance measurement systems. This contributes to performance management processes that are overly directive, and can demotivate workers, undermine teamwork, promote empty compliance and data falsification, and contribute to healthworker burnout.

By capturing data on these issues, and integrating it into routine performance management processes, we will contribute to greater understanding amongst decentralized managers of the needs and situations of healthworkers. In theory, this will lead to changes in performance management that create more of an enabling environment (that promotes trust, teamwork, shared problem solving etc.). Therefore, the proximal target population of the solution is healthworkers, in the form of improving their social working environment and work climate.

The evidence base demonstrates that overly directive management directly contributes to poor quality of care. The demotivation of workers, poor teamwork, weak decentralized problem solving, and culture of apathy that arises from counterproductive performance management undermines both the clinical quality and respectfulness of services provided by healthworkers. As such, this solution should also contribute to quality of care, and therefore patients are a distal target of the intervention. As the Lancet Global Health Commission on High Quality Health Systems set out, 5 million deaths a year are due to poor-quality care. 

The lead for this application – Tom Newton-Lewis, FHI 360’s Director of Health System Strengthening – is a leading global expert on the importance of “intangible software” for the performance of healthworkers; and in particular its implication for performance management systems. He has published pioneering research work in leading academic journals on how existing performance management processes in LMICs are counterproductive for worker motivation and performance. In 2019/20 he worked on an assignment funded by BMGF – “Primary Healthcare Performance Management Model and Landscaping” – that sought to incorporate the latest thinking of health systems as complex, social systems, and the latest global thinking and evidence, to develop a unified model of performance management in LMICs. This was built on eight case studies and a performance management competency framework. This was subsequently published in BMJ Global Health (Performance management in complex adaptive systems: a conceptual framework for health system). He has also published on digital health (and how it often fails health workers). Citations were included earlier in this application.

FHI 360 overall have a unique combination of capabilities that would enable us to deliver this solution:

• Our Health Systems Strengthening team has extensive experience on human resources for health, performance management, and the importance of intangible software in health systems performance and resilience.
• Our RMNCH team have extensive experience of designing and delivering projects to support performance management processes at scale across LMICs, with particular expertise in supporting managers on data reviews and sense-making
• Our Digital Development team have wide experience of designing and building technology based solutions, including SMS based systems, and data dashboards and vizualisations. FHI 360 was one of the founders of the Global Digital Health Network. FHI 360 have led large global USAID funded projects to advance the use of mobile solutions, including the headline Mobile Solutions Technical Assistance and Research Program (mSTAR), Knowledge 4 Health II (focused on mobile health applications for family planning and reproductive health services), and the earlier Mobile for Reproductive Health (M4RH) program using SMS technologies to advance family planning in East Africa.
• Our Global Health and Population Research team are leaders in the measurement of complex health service delivery and systems constructs
• Our Learning and Development team has expertise in leadership and managerial development, organizational culture, trust building, and the design and delivery of pulse surveys.

As set out in an earlier section, FHI 360 has existing large projects where we work closely to support decentralized decision makers (District Health Management Teams and facility managers) in different contexts, including in Cambodia, Mali, Mozambique, and Uganda. We work directly with them and healthworkers to strengthen primary care delivery. This would involve our large numbers of staff in country who work closely on a day-to-day basis with front line health workers and managers and therefore have deep, nuanced, and sensitive understanding of their challenges and lived realities.

We can use these platforms in the design and piloting of the solution, leveraging existing systems, networks, and workflows for data collection, sense-making, and performance management. In these projects, we already have strong working relationships with managers and healthworkers and institutional platforms and permissions. We would leverage these relationships to engage healthworkers, managers, and administrators in the design of the pulse survey questions and the technology solution and its participatory evaluation, in line with the human centered design principles outlined earlier. As such, they will be fully and meaningfully engaged. These existing platforms will also enable us to mobilize quickly to pilot the solution.

FHI 360 work to support data sense-making and performance management in primary care on a number of large projects (examples provided in earlier sections). However, these projects will not fund the R&D of a system that could be applied across multiple projects. To be able to pilot and refine (and scale and institutionalize) the solution, we would first need to develop a prototype of the system, and demonstrate that it works. For this, we need dedicated funding. If we had this prototype and proof of concept, it will be easy to integrate the approach into existing and new projects.

Secondly, we would welcome the support from MIT Solve, peer solvers, and the WFP Innovation Accelerator in the design of the system; we believe this could be of significant help. They will bring diverse experience of building SMS based systems and data visualizations that will complement FHI 360’s in house expertise.

Thirdly, it will be easier to market the solution if it has been formally recognized by peers and experts.

Our solution is innovative for two main reasons:

Firstly, it applies to performance management the latest global thinking that health systems should be considered social systems heavily influenced by intangible software; a framing that has not yet influenced measurement and improvement frameworks and processes. As a result, it brings in a focus on a novel set of factors that are proven to impact on health system performance, including trust, teamwork, and other dimensions of intangible software.

Secondly, it introduces data collected from healthworkers into performance measurement data systems, enabling their perspectives, challenges, voices, and needs to be heard and considered.

This will be catalytic because it will demonstrate the importance and feasibility of incorporating issues of intangible software into performance management and measurement systems. It will also generate learnings on how to support managers and teams to translate this data into performance improvement. This clear demonstration will enable the solution to be quickly scaled within existing and new large scale primary health care reform programs, both by FHI 360 and others in the industry.

In the first year we will develop the prototype; pilot it within a few existing projects where we support performance management processes; and demonstrate its viability and importance to government stakeholders and project funders. In the earlier application section, we outlined how we will involve decision-makers in the pilot process to build process legitimacy and interest. By the end of the first year, our aim will be to ensure buy-in and funding to start a scaling and institutionalizing process in existing projects.

By the end of five years, we aim to have completed the scale and institutionalization in existing projects; incorporated the solution into new projects; and developed a learning, evidence, dissemination and advocacy agenda that influences the broader field such that applying these kind of approaches becomes the industry standard. This will include academic papers, webinars, exposure visits, and other methods for showcasing the solution and its results. To bolster this agenda, we will seek ways to set up formal evaluations of the solution during the scaling process, that can formally track changes in intangible software and worker performance. 

As set out above, when we pilot the prototype, we will have a participatory evaluation process using healthworkers, managers, and government decision-makers, to feed back on the solution, in line with the human centered design approach. We will also track key process indicators, including response rate to the SMSs, and use rate of the dashboards. We will have a process documentation system to track how the information is used and acted upon. We will also set up measures of cost and cost-effectiveness.

We will measure the scaled-use of the solution (number of projects, number of healthworkers covered etc.)

During the scaling process, we will use both the data generated by the solution (e.g., to measure changes in motivation), and separate evaluation streams where feasible, to measure its impact and effectiveness, and to feed into the learning, dissemination, and advocacy agenda. 

The primary project activities (described more fully in the earlier sections) include:

• Developing a potential list of pulse survey questions from existing evidence and experience, and then using a human-centered design process with healthworkers, managers, and other stakeholders in pilot countries to refine and finalize the priority questions and their translation
• Using a human-centered design process with healthworkers, managers, and other stakeholders in pilot countries to design the parameters of a technology system (e.g., testing the assumption that we should use SMSs as the basis; the periodicity of the messages; ideal forms of data outputs and visualization)
• Building the system for sending and receiving messages; storing and aggregating data; and creating analysis outputs and visualisations
• Piloting the use of the system on a few existing projects; including supporting managers and teams in their sense-making of the data and developing performance improvement activities

The project outputs would therefore include:

• A tested repository of questions that could be used in the pulse survey
• A technology based system that can send and receive pulse survey messages through SMS, store and aggregate data, and create data outputs and visualizations

The proximal outcome will be that data on intangible software is generated and then reviewed and discussed during performance management processes (such as data reviews and quality improvement sessions) in the pilot sites. This will include both levels of dimensions of intangible software (such as motivation and trust) and changes over time (which can be linked to activities undertaken to improve it).

This will lead to an intermediate outcome of interventions and innovations introduced to improve intangible software, including making management styles less hierarchical and performance management processes more participatory, and solving other constraints affecting healthworkers (e.g., addressing their safety, workload, and equipment challenges).

This will lead to a distal outcome of improved intangible software for healthworkers, manifested by higher levels of motivation, trust, teamwork, commitment, feelings of organizational reciprocity and other dimensions. This will be augmented by the fact that the nature of asking workers for feedback (the activity) and involving them in discussions (proximal outcome) will itself promote employee engagement and motivation.

This will lead to the impact of improved performance (in the prototype pilot areas).

The secondary project activities include the learning, evidence-generation, documentation, and advocacy activities undertaken in both the pilot sites (including involving key stakeholders in the process) and then later with prospective new countries and funders. This will lead to the secondary output of knowledge projects.

This will lead to a secondary proximal outcome of the scaling and institutionalization of the solution in the pilot projects. A secondary intermediate outcome would be the piloting, scaling, and institutionalization of the solution in other and new FHI 360 projects. A secondary distal outcome would be the uptake of similar approaches as an industry standard.

This would then contribute to improved performance (outside of the prototype pilot areas).

The things that give us confidence in this theory of change, aside from our own experience of supporting performance management processes in primary health care systems, include:

• An extensive evidence base that performance management systems are currently unsuccessful at driving needed improvements in performance (Munar et al 2019)
• A platform of evidence and thinking that shows that this is partly because performance management systems do not consider how they interface with issues of intangible software (Newton-Lewis et al 2021 and other citations set out earlier in this application), which is unconducive in many low- and middle-income countries (Nxumalo et al 2018)
• Proof that it is possible to change performance management approaches (and leadership styles) at decentralized levels and that this can improve intangible software (Orgill et al 2021) and that this improves system performance (Schneider et al 2020)
• Evidence that shows that pulse surveys generate strong response rates (Welbourne 2016), and lead to changes in management approaches (Jolton and Klein 2020)
• Evidence that shows the being asked for feedback can motivate staff, promote feelings of organizational reciprocity, and increase engagement, teamwork, and trust (Employee Surveys Are Still One of the Best Ways to Measure Engagement (hbr.org)

Citations for this section:

Munar W, Snilstveit B, Aranda LE, et al. Evidence gap map of performance measurement and management in primary healthcare systems in low-income and middle-income countries. BMJ Glob Health 2019;4:e001451. doi:10.1136/bmjgh-2019-001451pmid:http://www.ncbi.nlm.nih.gov/pu...

Newton-Lewis T, Munar W, Chanturidze T. Performance management in complex adaptive systems: a conceptual framework for health systems. BMJ Global Health 2021;6:e005582.

Nxumalo N, Gilson L, Goudge J, et al. Accountability mechanisms and the value of relationships: experiences of front-line managers at subnational level in Kenya and South Africa. BMJ Glob Health 2018;3:354. doi:10.1136/bmjgh-2018-000842pmid:http://www.ncbi.nlm.nih.gov/pu...

Orgill M, Marchal B, Shung-King M, et al. Bottom-up innovation for health management capacity development: a qualitative case study in a South African health district. BMC Public Health 2021;21:587. doi:10.1186/s12889-021-10546-w

Helen Schneider, Asha George, Fidele Mukinda & Hanani Tabana (2020) District Governance and Improved Maternal, Neonatal and Child Health in South Africa: Pathways of Change, Health Systems & Reform, 6:1, DOI: 10.1080/23288604.2019.1669943

Welbourne, Theresa. (2016). The Potential of Pulse Surveys: Transforming Surveys into Leadership Tools. Employment Relations Today. 43. 33-39. 10.1002/ert.21548.

Jolton, J. & Klein, C. Exploring the Universe of Pulse Surveys and Continuous Listening Opportunities in: Employee Surveys and Sensing. Edited by Macey, W. & Fink, A. Oxford University Press (2020).

Our solution is to develop a prototype of a system that routinely collects pulse data on intangible software from healthworkers through SMS surveys, and that automatically generates data dashboards and visualizations for managers at a decentralized level.

We will leverage globally available digital survey platforms to administer a survey tool that measures different dimensions of intangible software. The survey will be sent to the mobile phones of healthworkers through SMSs on a monthly basis.

We will use an existing platform because they offer templates for fast design, development and optimization of surveys – including the ability to test, iterate and analyze results in preparation for the final survey launch. These existing solutions have interfaces already built of working with different mobile operators as well as different HIS systems in country. Their technology has user confidentiality built in, tools to help track delivery of messages as well as in built tools to troubleshoot and monitor any user queries. We work with a number of existing platform providers and will select the optimal one after the human centered design process set out below.

We are envisioning using SMS because, from our experience, healthworkers in many countries do not have smartphones or continuous internet access, which rules out other options such as WhatsApp. Using “brick” phones will ensure the highest possible reach. SMSs are a low cost solution. This will be discussed and debated with healthworkers in the pilot contexts during the design phase; it would also be possible to use IVRS in areas with low literacy. We intend to design a flexible system that could potentially send out survey messages through WhatsApp rather than SMS in contexts where this was more appropriate. In businesses in high-income countries, pulse surveys are frequently disseminated through WhatsApp (either using in built forms or web links sent through messages). Because of this, we will select a platform that can offer omnichannel message and scalability for further user engagement options.

The data will be aggregated by the server for the unit of interest (e.g., facility) and calculate aggregate figures in a dashboard, with in-built visualizations – of levels, and changes over time. The data will not be available by individual respondent to preserve confidentiality and will only aggregate at units large enough so that anonymity is guaranteed.

Whilst here we are proposing to build a prototype system, we will also ensure that the data could potentially be integrated into government HMIS/DHIS2 information systems, so that it is sharable across information systems. The SMS platform we envision is designed to manage data interfaces seamlessly, providing managers with easy access to aggregate data. 

The data will be collected from healthworkers who will respond to SMS surveys.

Their incentive to do so is because it will enable concrete data on their perspectives, needs, and challenges to be incorporated into the data that managers and teams use to plan performance improvements.

The pulse surveys are very short (a couple of questions). Existing evidence (see citations in the Solution section of this application) show that they generally get a high response rate; and that the act of asking for feedback in this way can motivate staff and increase engagement (and improve trust and teamwork).

Data is anonymized to avoid potential negative repercussions.

At FHI 360, we envision an environment where our differences are celebrated and embraced and where our team, partners and community members feel valued and respected. We are committed to building and nurturing a diverse, equitable and inclusive workforce and workplace because it fosters creativity, drives innovation, reflections FHI 360’s vision of a more equitable world, and aligns with FHI 360’s mission to improve lives in lasting ways through locally led development.

We embrace differences in the broadest sense, including race, color, religion, sex, sexual orientation, gender identity and expression, nationality, ethnicity, abilities and disabilities, age, generation, marital status, genetic information, status as a U.S. veteran, socioeconomic status, education, and all other identity dimensions. We believe that building and nurturing a diverse, equitable and inclusive culture is a continuous journey.

In 2021, FHI 360 commissioned an external expert agency, Collective 180, to assess FHI 360’s situation against its aspirations; this included a large scale survey across staff, and various qualitative data collection exercises. This has led to a new DEI strategy and action plan, and the formation of regional employee-led DEI Councils that will advance our DEI strategy and action plan.

With respect to this solution, we recognize that many healthworkers themselves face DEI challenges in their work. For example, inequitable gendered power dynamics contribute to inappropriate performance management processes, and the lack of attention to the needs and constraints of female workers. Workers from marginalized communities face discrimination within health systems. Whilst our solution is inherently focused on ensuring that data is generated on the lived realities of healthworkers, we recognize that this will interface with the complex biases and power inequities within health systems. We will ensure – as set out in the earlier sections – that healthworkers who themselves are marginalized will be involved in the design and evaluation processes, so that their perspectives and needs are captured to the extent possible. Our local teams are well versed in understanding these complex contexts and dynamics.

FHI 360 is an established international nonprofit working to improve the health and wellbeing of people around the world. Our mission is to improve lives in lasting ways by advancing integrated, locally driven solutions for human development.

We work across both research and implementation, exploiting the synergies to deliver thoughtful and evidence informed, yet practical, solutions that change behaviors, increase access to services, and improve lives. FHI 360 is particularly known for its ability to deliver at scale, across multiple countries or at scale within a country, without diminishing the quality and impact of the work.

We deliver this by combining industry leading, globally distributed technical expertise with large locally embedded teams in the countries where we work, who bring a deep understanding of local contexts. Together this allows us to provide a 360 degree perspective to develop customized responses to the toughest human development challenges. We have over 4,000 staff in more than 60 countries.

Our work is funded by multilateral and bilateral development agencies, philanthropic foundations, and private sector organizations who share our mission. Some of this work is commissioned directly, and some is awarded based on competitive tenders. In most of these projects, there is extensive co-creation, so that FHI 360, our customers, and beneficiaries work together to design approaches that have the greatest chance to achieve desired impact.

As outlined earlier, FHI 360 has a number of existing projects where we support performance management processes in low- and middle-income countries. The nature of the funding of these projects means that they cannot fund R&D of a new approach that could be applied across multiple projects. This is why we need dedicated funding. However, when the prototype has been developed, it will be possible for them to fund the system to be piloted (and then, where appropriate, scaled and institutionalized).

We would then incorporate the solution into the design of new projects which would fund its use. We would expect the existence of the solution to significant increase our value proposition when developing proposals for new work.

It is an industry standard to use R&D to develop branded approaches that can then be deployed across projects. The projects then fund the use of the approach. These can be analytical and diagnostic tools, not just technology-based approaches. For example, FHI 360 has a branded approach to diagnose constraints to health systems performance, and for implementing quality improvement. Competitors have branded approaches for human centered design processes, for example. There are many companies who have established propriety approaches to the delivery of pulse surveys in the private sector.