HTA Explorer

For increasing the capacity and resilience of health systems globally, it is crucial to ensure that healthcare policies and practices are closely informed by the latest cost-effectiveness data. Health Technology Assessments (HTAs) are studies that quantify the cost-effectiveness of various health interventions. These assessments are expensive to conduct, and have historically been carried out in high-income countries by government-funded organisations. However, the findings of HTAs are often globally relevant, and can be particularly useful for decision-makers in resource-limited settings, where cost-effectiveness is particularly important.

HTAs guide policy-makers and healthcare professionals in deciding which treatments to prioritise and fund. The International HTA Database (https://database.inahta.org/) is the main repository for HTA studies, both ongoing and completed. It currently contains over 21000 studies from 120 HTA producers globally, and is constantly being updated. However, the current search functionality of the database is limited:

• It relies on exact keyword matches, making it challenging for users to find the most relevant studies. 

• Even when a relevant study is found, accessing the full report is not always straightforward. Users often have to navigate through various websites to locate and download the PDF. 

• There is also no ability to get a brief summary of the study’s cost effectiveness results, in order to determine whether the article is worth downloading and reading.

These inefficiencies discourage busy decision-makers from fully making use of this valuable resource, and the limited search function increases the likelihood that users will not rapidly find the studies they need.

In resource-limited settings, where cost-effectiveness data is particularly useful, inefficient access to the latest HTA literature is a notable challenge for policymakers. 

Without quick and easy access to the latest HTA literature, policymakers may make decisions about resource allocation that are not supported by the latest cost-effectiveness evidence. This can exacerbate health inequities, as communities from resource-limited settings may experience not only a scarcity of resources, but the inefficient deployment of those resources too.

HTA Explorer is an enhanced search engine for accessing completed and ongoing Health Technology Assessment studies. In addition to improved search capabilities, HTA Explorer automatically extracts the key results from HTA reports using intelligent text chunking. It then feeds these excerpts into a powerful language model to generate concise summaries focused on cost-effectiveness findings. This allows the user to quickly decide whether it's worth downloading the article to read in full.

The application is web-based and currently uses a Python Flask backend, and OpenAI's latest word embeddings model is used for semantic search. Results are ranked by their similarity in meaning to the search query, rather than using exact word matches. Summaries of results are then generated as needed using Anthropic’s latest Claude 3 Sonnet language model. This technology stack provides users with relevant search results and clear summaries. 

By making it significantly easier and faster to find and extract insights from HTA studies, the system helps decision-makers to stay updated with the latest HTA literature. Our solution helps to close the divide between high-income countries where HTAs are typically conducted and other regions that can benefit from their research. This supports evidence-based policy decisions globally, helping to deploy healthcare resources more effectively and equitably.

HTA Explorer has a direct target population, the end-users of our application, and an indirect one, the general population that is underserved by cost-ineffective healthcare practices.

The expected end-users of the HTA Explorer are healthcare policymakers and researchers, including government officials and funding decision-makers. These users rely on HTA studies to shape policies, allocate healthcare funding efficiently, and decide which new assessments to conduct. While the International HTA Database is an excellent and continually-updated source of the latest HTA literature, its utility is compromised by several usability issues. Policymakers are currently underserved by the sub-optimal search feature, difficulty in locating article PDFs and the absence of automated result extraction and summarisation. This means it is needlessly difficult and time-consuming for users to find the studies they need, hindering efficient policy development.

Inefficient access to the HTA literature can lead policymakers to implement healthcare policies that are less cost-effective. Without easy access to the latest research, there is a risk that policymakers continue to support cost-ineffective or outdated policies, leading to worse health outcomes.

HTA Explorer helps policymakers and researchers to access relevant HTA studies more quickly and easily. This enables them to make more evidence-based decisions, leading to improved cost-effectiveness of healthcare services. This ultimately leads to health systems which are able to deliver quality healthcare more effectively to the communities they serve. This is the indirect target population that is served by our solution.

The Clinton Health Access Initiative South Africa (CHAI-SA) works closely with health policymakers in South Africa at the provincial and national level. Recently, CHAI-SA has seconded a staff member to the HTA unit of South Africa’s National Department of Health, and is helping to capacitate the unit as a part of the government’s upcoming National Health Insurance initiative. The organisation therefore has close ties to health policymakers in the South African government who would benefit from improved access to the latest global HTA literature.

The team, guided by Nikhil Khanna, has a demonstrated history of working with policymakers in South Africa to strengthen health systems. We have experience developing and implementing digital projects that are closely guided by inputs and specifications from these policymakers. For example, the team is currently working on Tender and Fleet Management applications for the provincial Eastern Cape Department of Health. These projects are developed as a collaborative process with end users within the department. This is the approach we would follow for this project.

Currently, we have developed a working prototype of the HTA Explorer web application that operates locally. It features a functional semantic search, automated extraction of results, and on-demand generation of AI-based summaries.

While additional funding from MIT Solve would be helpful for scaling our product, the non-financial benefits of being a Solver would potentially be even more beneficial. 

Additional funding would enable us to finance a scale-up of the current prototype. It would be used to cover additional hosting and database expenses, and possibly hire an extra developer. However, the support provided to Solvers with monitoring and evaluation, the increased publicity associated with being a Solver and the potential for assistance with development, would be even more helpful for making HTA Explorer a reality:

• While we have experience building small-scale web projects for provincial government use, a truly global solution requires a scalable web application capable of handling many thousands of users. For this, we would need additional web development expertise. 

• Monitoring and evaluation support would also be particularly helpful. Again, while we have conducted local surveys in South Africa, assistance with writing and conducting a global survey to assess the impact and usability of our application would also be beneficial. 

• Finally, the publicity associated with being a Solver would help to increase awareness of HTA Explorer, and increase users, which could significantly increase the impact of our solution.

HTA Explorer advances the way policymakers and researchers interface with the HTA literature, making it faster and easier to access relevant information. The application significantly improves the experience of accessing the International HTA Database by integrating advanced AI technologies like semantic search and automated summarization. 

The underlying technologies used by HTA Explorer, including semantic search, automated extraction of sections within a file (results, in the case of HTA Explorer) and AI-based summarisation, have the potential to be applied to other healthcare databases. This would catalyse broader positive impacts in access to healthcare data. In any setting with a database, linked files, and the need to search and summarise items in the database quickly and easily, this technology has the potential to be helpful. By demonstrating the value of these technologies in the context of HTA data, our application can serve as an example that can be replicated elsewhere in the healthcare sector, for providing improved access to healthcare information.

Our theory of change is the following: by reducing the search and retrieval time for HTA data, HTA Explorer makes it easier for policymakers and researchers to access and use this data, which in turn leads to more informed decision-making and policies, and ultimately better health outcomes in the general population.

Planned activities include:

• Developing HTA Explorer, a search and summary tool that enables health policy-makers to rapidly query the latest HTA literature from the International HTA Database.

• Conducting training sessions for potential users of the application.

Planned outputs include:

• A user-friendly application that reduces the time it takes to find relevant HTA studies and download the required documents.

• Increased knowledge and awareness among health policymakers and researchers of HTA Explorer and how it is used.  

We expect a range of short-term, medium-term and longer-term outcomes of the successful implementation of HTA Explorer:

• In the short-term, users report a faster and improved overall experience of accessing the latest HTA evidence. This translates to increased availability of relevant HTA data for researchers, and in health policy discussions and decisions.

• In the medium-term, the increased availability of relevant HTA data leads to health policies being increasingly informed by this data, resulting in more cost-effective health policies. 

• In the long-term, more informed health policies translate to a more efficient health system, leading to improved and more equitable health outcomes.There is evidence that the inclusion of HTA data in health policy decisions is beneficial, and ultimately leads to improved health outcomes. A survey of experts in the healthcare field in Middle-Income Countries found that the overwhelming majority of respondents viewed the inclusion of HTA data in policy decisions to be beneficial (Fasseeh et al, 2022). 97% of respondents agreed with the statement that the implementation of HTAs increase “transparency and accountability” in health policy decisions, and 84% agreed that the implementation of HTAs improve health outcomes in the general population (Fasseeh et al, 2022). 

Fasseeh, A. N., Saragih, S. M., Hayek, N., Brodovska, S., Ismail, A., ElShalakani, A., Abaza, S., Obeng, G. D., Ameyaw, D., & Kalo, Z. (2022). Impact of health technology assessment implementation with a special focus on middle-income countries. In Health Policy and Technology (Vol. 11, Issue 4, p. 100688). Elsevier BV. https://doi-org.ezproxyberklee.flo.org/10.1016/j.hlpt...

The ultimate goal of HTA Explorer is to make it faster and easier to access relevant HTA studies, due to the downstream benefits this has on health policies and health outcomes. Our measured impact goals are related to the experience of the application’s users, and aim to quantify through surveys that the application is in fact making it easier and faster to find relevant HTA studies. These are divided into two goals, which are outlined in more detail below:

1. Improve Speed:

• Goals: Reduce the user-reported time taken to query the database, retrieve relevant results, and download the required article PDF(s). 

• Measurement Strategy: Conduct a survey to gather data on user-reported times for completing tasks using both HTA Explorer and the International HTA Database. The survey will include equivalent questions for the search capabilities of the International HTA Database, in order to quantify improvements. The target is a significant reported decrease in time spent in two areas. First, the time taken from entering a search query and finding a relevant HTA study, and second, the time taken from finding a relevant study and downloading it.

• Indicator: Percentage reduction in user-reported time spent on the above tasks.

2. Improve User Satisfaction:

• Goal: Increase overall user satisfaction with HTA Explorer compared to the International HTA Database.

• Measurement Strategy: In the same survey that quantifies the speed improvements of HTA Explorer, we will include questions related to user satisfaction. These questions will relate not only general satisfaction but also specific aspects like ease of use, interface design, and satisfaction with search functionalities. As before, the survey will include equivalent questions for the search capabilities of the International HTA Database, in order to quantify improvements.

• Specific Indicator: Overall satisfaction scores and specific usability metrics on a Likert scale.

The main product capabilities are semantic search, automated results extraction from article PDFs and AI-powered results summarisation. The technologies that enable these features are outlined below.

Semantic search:

Semantic search uses word embeddings, which numerically encode the meaning of strings. The search string is then used to rank results in decreasing order of similarity in meaning. This approach is an advantage over exact word matching, which could potentially lead to missing certain results which are similar in meaning to the search term but do not have any word matches. The title, organisation country, publication year and Medical Subject Headings (MeSH) keywords for each article are concatenated into strings, which are then embedded using OpenAI’s latest word embedding model, text-embedding-3-large. The vectors are stored, along with additional metadata, on the Pinecone vector database platform. Search strings are then embedded using the same embedding model, and cosine similarity is used to rank articles according to similarity in meaning to the search string. 

Results extraction:

A crucial product feature is automated results summarisation. One approach would be to feed the entire text of an article into an Large Language Model (LLM) and request a summarisation. However, because the articles are often many pages long, with the majority of the text not focused on results, this approach would be too slow and costly. An alternative is to extract the results first, and then feed them into an LLM for summarisation. To extract results from PDFs we used the open source LlamaSherpa library for Python, which intelligently chunks text into sections, based on headings in the document. We then included only headings or subheadings containing keywords such as “results” or “findings”.

Results summarisation:

In order to extract key results from the provided chunks of text from an article, we used the Claude 3 Sonnet model from Anthropic, which is both highly capable and affordable, with an impressively long context window. The model is instructed first to provide key quotes from the article in bullet point form, relating to cost-effectiveness results. At this stage the model is explicitly instructed not to add or rephrase words, so as to prevent hallucination.

Less than one year.