The Standardized Patients Primary Health Care Audit Toolkit

One of the most common sayings in the quality improvement literature is that “you can’t improve what you can’t measure.” Despite a great deal of effort and enormous technological investments in data, the measurement of quality in primary care remains one of the most challenging problems for health systems worldwide, especially in low- and middle-income settings. These problems have made it extremely hard for policymakers to generate “all-else-equal” measures to estimate or evaluate the impacts of policy changes or other initiatives to improve the delivery of high-quality primary care. The root cause of this persistent problem is three-fold.

First, quality of health care may be deficient either because doctors do “too much” or because they do “too little”. Whether they have done too much or too little depends on the specific patient in front of them: Antibiotics may be “too much” if you have a viral infection; not giving antibiotics may be “too little” if you have a bacterial infection. However, routine administrative health data, even at is best, provides information only on the treatments that were given and not whether patients were correctly diagnosed in the first place.

Second, patients choose their doctors—and sicker patients may visit higher quality physicians. This means that worse outcomes are often associated with higher skill in true utilization patterns, and there is insufficient information to correct for these “case-mix” confounds. As a result, for example, doctors who give more antibiotics may be lower quality—or they may simply be seeing more patients with bacterial infections. Again, “ground-truth” audit data do  not exist in these settings.

Third, in any health system, the true value of a doctor lies in their ability to diagnose and manage “serious” conditions. However, more than 95% of outpatient cases are of routine, self-limiting conditions in need of nothing more than palliative care. Even in a city like Mumbai, which is the world’s hotspot for tuberculosis, fewer than 5% of healthcare providers will see a TB patient in any given month. Consequently, when policymakers or investigators are interested in serious diseases, very large patient dragnets are required to obtain appropriate sample sizes for any specific investigation of epidemiological interest, which poses serious ethical and practical challenges. 

In order to address each of these problems, our team has developed and validated a measurement method “in the field” that allows accurate estimates of provider care quality to be quickly taken from small samples using directed audits. This allows policymakers and program evaluators to both measure levels of care quality, and to accurately estimate the causal impact of quality improvement programs in primary care with low cost and high reliability. Our methods provide intercomparable data schematics, guidelines for development of research questions, and practical guidelines for the collection of data with nearly no technical or connectivity requirements. To enable diverse organizations and governments to use these methods, we have always made our materials freely available.

Our target population is the set of people who are involved in the development of health systems quality interventions as well as policymakers working in the health sector. Our toolkit offers a low-cost and low-tech approach for evaluators and policymakers to benchmark quality of care in their system and evaluate the impacts of quality-improvement interventions. 

In the standardized patients system, the evaluator pre-specifies the tracer conditions that are used to assess quality. Previous tracer conditions include asthma, diarrhea, angina, tuberculosis, and other key tracer conditions and conditions of epidemiological interest. Then, the case-scenarios and desired paths of history taking, examinations and clinical management are pre-specified through consultations with an expert group and case scripts are developed together with the standardized patients. Once scripts have been developed, standardized patients are sent to the sample of providers the evaluator is interested in and the process is directly controlled by the evaluator so there is no missing data and all data is exactly comparable. 

When executed properly, relatively small sample sizes are sufficient to either audit specific populations for developments of concern, as well as to quickly evaluate the impacts of interventions. There is no dependence on external data, there is no delay in collection of data by routine systems such as waiting for sufficient sample size, no missing or mismeasured data, and no concerns about the security or privacy of health information (since no real patients are observed).

Our target population is the set of people who are involved in the development of health systems quality interventions as well as policymakers working in the health sector. Our toolkit offers a low-cost and low-tech approach for evaluators and policymakers to benchmark quality of care in their system and evaluate the impacts of quality-improvement interventions.   

In the standardized patients system, the evaluator pre-specifies the tracer conditions that are used to assess quality. Previous tracer conditions include asthma, diarrhea, angina, tuberculosis, and other key tracer conditions and conditions of epidemiological interest. Then, the case-scenarios and desired paths of history taking, examinations and clinical management are pre-specified through consultations with an expert group and case scripts are developed together with the standardized patients. Once scripts have been developed, standardized patients are sent to the sample of providers the evaluator is interested in and the process is directly controlled by the evaluator so there is no missing data and all data is exactly comparable. 

When executed properly, relatively small sample sizes are sufficient to either audit specific populations for developments of concern, as well as to quickly evaluate the impacts of interventions. There is no dependence on external data, there is no delay in collection of data by routine systems such as waiting for sufficient sample size, no missing or mismeasured data, and no concerns about the security or privacy of health information (since no real patients are observed).

The team has an extensive history of developing and delivering innovation and results in the area of standardized patients research. Beginning in 2012, team members have deployed and published the earliest pilots of the method in low- and middle-income countries, including rural and urban India, Kenya, South Africa, China, and supported projects in other settings like Nigeria and Indonesia. They have published guides to using the work in academic studies, as well as engaged with international NGOs as well as national and subnational governments to organize and execute evaluation projects using the methods. The team has further developed detailed technical toolkits and demonstrative materials in the academic sphere, which have rapidly catalyzed the uptake of the method by researchers, implementers, and policymakers worldwide.

The project is led by gui2de at the Georgetown McCourt School of Public Policy. Gui2de has long supported the practical and theoretical development of methods to support innovation and evaluation of last-mile service delivery project and programs around the globe. Its interdisciplinary approach in this field, as well as the team members’ long track record of success at the World Bank and in academic work leading up to this proposal, make the site an ideal repository for the knowledge and practice required to execute and disseminate these materials.

The successes of the method and the corresponding rapid expansion of demand for the standardized patients solution from policymakers, health systems administrators, and NGOs has led to overwhelming demand on our team members to provide ad-hoc support to a wide range of evaluation projects. Our team has come to the conclusion that the formalization and standardization of the method and guidelines as a structured and publicly-accessible toolkit would allow us to more easily respond to requests for support. It would also enable a wide range of health systems to access and implement the methodology without needing to directly contact our team for assistance.

Prior to our groundbreaking work, there was no cost-effective method to accurately measure the quality of primary health care, especially in ways that were comparable across different types of health care providers or that would support rigorous and/or randomized trials of health care provider performance improvement interventions. Our pilot work and scale-ups have created a broad outline of what types of questions can be asked and answered in evaluations using the standardized patient methodology, as well as developed the practical component of any such program.

Furthermore, we have extensively developed the theoretical elements of the approach. We have demonstrated that the method is low-tech, low-cost, and low-risk. We have provided guidelines for sample size calculations and shown that they are generally far smaller and more affordable than “high tech” solutions because of the extreme precision with which evaluators can guide and target the data collection that corresponds exactly to their areas of interest. We have also validated and evaluated the measures that can be collected using the method as well as what dimensions can be appropriately compared, including elements like assessing diversity in treatment of patients.

In the next year, our impact goal is to (a) develop a comprehensive toolkit for the design and execution of standardized patients studies in policy evaluation; and (b) to organize a seminar and/or policymaker education series to educate stakeholders about the prior practical work in the area as well as the availability of the toolkit.

Over the next five years, we would expect to integrate the standardized patients training materials into publicly available programs such as the executive education offered through Georgetown University to health policymakers; as well as in graduate studies curricula in the School of Foreign Service (where Das teaches).

Progress towards the impact goals is currently measured by citation and adaptation of our methodology. To date, the academic-oriented pilot papers and theoretical papers have been  cited more than 1,000 times. In additions, we have observed or participated in the development of dozens of academic and governmental projects that rely on our methodology. These include projects studying the development of insurance products and telemedicine in Rwanda, studying the broad-scale state of health care across China, and evaluating the deployment of the SHOPS-Plus program in Nigeria. Alongside our own work evaluating national scale-ups of private-public tuberculosis control partnerships in India, these represent some of the largest intervention evaluations conducted and are quickly becoming the gold standard for program evaluation in health systems worldwide.

Going forward, we would continue to measure success based on the demand for our toolkit as well as associated teaching and training offerings from our team. In addition, we would continue to monitor the number of public projects and grant proposals that include the method of standardized patients as a key metric. While we would expect that many of these projects would remain private/internal, emerging trends of independent evaluation requirements from funders such as the Gates Foundation mean that academic citations and publications that reference our work and our toolkit will continue to be a reliable metric of the reach and impact of the work in the longer term, particularly when it is outside our direct control.

Our theory of change is that the development of the tools together with the publication of important results will lead to the demand for these methods as a way of measuring quality in primary care systems. Then, the availability of the tools as well as personnel from our team can be deployed to implement SP studies for new conditions or contexts.

One of the greatest strengths of our solution is that it is largely technology-independent. The basic elements of a standardized patients evaluation program can be developed entirely using pen and paper. In terms of actual implementation, a wide range of platforms have been used to collect and manage data; including SurveyCTO, CommCare, and RedCap. Similarly, any statistical software such as Stata, R, or Python can be used to analyze standardized patients data. The core innovation is the development of intercomparable approaches and metrics that allow users operating in different environments to combine, share, and compare results. In contrast to meta-analytical approaches that must carefully evaluate the methodology and measurement approaches of studies in different contexts, the standardized patients framework essentially removes all theoretical roadblocks to the combination and comparison of data sets for secondary and extended analysis outside the original study context.

The primary health care data that we use is collected directly by staff enumerators. Over the course of the development of the solution, we have worked with a wide variety of management and implementation teams dependent on the setting. In some cases, the data collection team is directly managed by our team; in others, it has been managed by a partner government or NGO with our advice and/or support.

The essential element, however, is that the team of actual data collectors is representative of the ultimate beneficiaries of the program, project, or health system being investigated. This means that the data collection team should be composed of people who are likely to be true users of the health system. We do not generally encourage the recruitment of health professionals, trained actors, or other specialized individuals. In the projects we have directly managed, we have hired anyone willing and able to do the work (for example, we do not generally impose even a literacy requirement since supervisors are able to record answers from staff).

Our approach is built around accurately representing the people who seek health care in some of the most disadvantaged regions of the world. Research teams using our methodology are often based at least partially in location where studies are taking place. To ensure representation across the relevant dimensions in any context, teams routinely engage in partnership with representatives of health care providers, health care seekers, and policymakers among other stakeholders. Individuals working as standardized patients are always selected to effectively represent the populations that are most affected by the research questions at hand.

We provide materials entirely free of charge, and some team members have also become paid consultants for other research teams. However, despite sustained demand for direct support from outside organizations on our current and former team members, such a model is not scalable to address the future demand we anticipate. Governments, research organizations, and health care organizations have increasingly made requests for time and expertise that we cannot satisfy, leading us to explore these new avenues for low-cost scalability of our methodology.

We expect that producing complete guidance for replicating our model will dramatically decrease the amount of time we will need to spend responding to requests for assistance. While we may, in the future, explore methods of expanding our income through the support we provide, our current goal is to dramatically reduce the costs of providing support to new organizations and individuals who wish to use our approach in new projects. This will allow us to support a much broader array of users at our current level of effort.

We have successfully funded ten years of research work developing and scaling our solution in a wide range of settings and with a wide range of partners. In addition, direct demand for our support has resulted in private contracting with people who wish to use our methods. We therefore have no issues with financial sustainability inside the University; however, scaling up our ability to provide support services to others is not feasible with the number of people we employ and the current state of our resources.