Health systems guidance (HSG) documents are systematically developed statements produced at global and national levels that provide recommendations to address a health system challenge. These recommendations can be used to inform policy decisions related to health system governance, financial arrangements, system organization, or design and delivery of health programs and services. No tools exist to evaluate the quality of HSG or to direct the development and reporting of HSG documents.
This project’s aim was to develop a tool that will direct the development, reporting, and appraisal of HSG documents.
AGREE-HS Development Methods
Critical Interpretive Synthesis
The AGREE-HS research team conducted a critical interpretive synthesis of the literature to identify factors that affect HSG quality and implementability. Thirty distinct factors were identified in the literature and formed the candidate criteria for a HSG appraisal tool. The results of the critical interpretive synthesis have been published in the following article:
Concept Evaluation and Preliminary Usability Testing
Health systems and policy experts were invited to review the list of criteria identified in the critical interpretive synthesis and assess each criterion’s importance in the development, reporting and appraisal of HSG. All thirty criteria were rated favourably. Based on the results of the concept evaluation study, a 32-item framework of HSG quality and implementability was developed. Based on that framework, a draft HSG appraisal tool was created. A second group of health systems and policy experts was invited to apply the draft tool to a HSG document and complete a usability survey. Results were generally positive but suggested that further refinement of the tool was necessary. Results of these studies were published in the following article:
The AGREE-HS research team revised the AGREE-HS tool based on feedback received. The 32-item beta tool was condensed into a 5-item tool, with the original 32 items serving as criteria for the five newly defined items.
Using the AGREE-HS to appraise HSG documents
Graduate students and research staff applied the 5-item AGREE-HS tool to 85 HSG documents (published 2012-2017) to assess their quality. HSG documents without a specific health focus and those authored by the National Institute for Health and Care Excellence group, achieved higher AGREE-HS overall scores than their comparators. Results of the HSG quality appraisals indicated that improvements can be made in future HSG document development, regarding: methods used to develop the documents, the level of transparency of reporting, and the guidance for implementing the HSG recommendations.
AGREE-HS validity and usability testing
International stakeholders completed a face validity survey about the 5-item AGREE-HS tool. They indicated that the content and structure of the tool were appropriate and agreed that it would be useful in guiding HSG development, reporting and appraisal. The graduate students who participated in the HSG quality appraisals (mentioned above), completed a survey to assess the tool’s usability and measurement properties; they agreed that the items were easy to understand and apply. The instructions, usefulness and usability of the AGREE-HS were also rated favourably. Initial assessment of the tool’s measurement properties indicated strong internal consistency and moderate inter-rater reliability.
Funding: Canadian Institutes of Health Research (CIHR)
Research Team: AGREE-HS Research Team