Clinical quality and performance measurement in the prehospital emergency medical services in the low-to-middle-income setting : developing clinical quality and performance indicators as a measure of care in South Africa

Abstract

BACKGROUND: Measuring quality and safety in any healthcare setting however is highly contextual, and depends on the manner in which quality is defined or viewed within that setting. It is this contextual nature that has provoked significant debate and hindered efforts at developing formal standards or criteria for measuring quality and safety in healthcare, regardless of setting. Historically, performance within the Emergency Medical Services (EMS) delivering prehospital emergency care has been assessed primarily based on response times. While easy to measure and valued by the public, overall, response time targets are a poor predictor of quality of care and clinical outcomes.

AIM: The overall aim of the research was to develop a framework for clinical quality and performance-based assessment of prehospital emergency care for use in the South African EMS.

METHOD: The research was divided amongst four studies, with each study constituting one of the overall research objectives. Study I was a sequential explanatory mixed methods study with the aim of understanding the knowledge, attitudes and practices of clinical quality and performance assessment amongst South African EMS personnel. Part 1 consisted of a web- based cross-sectional survey, and Part 2 consisted of semi-structured telephonic interviews of select participants from Part 1 to explore the results of the survey. Descriptive statistics were carried out to summarise and present all survey items, and conventional content analysis employed to analyse the interview data. Study II utilised a three round modified Delphi study to identify, refine and review a list of appropriate quality indicators for potential use in the South African EMS setting. For Study III a novel quality indicator appraisal protocol was developed consisting of two categorical-based appraisal methods, combined with the qualitative analysis of their consensus application, and tested against the outcomes of Study II. Descriptive statistics were utilised to describe and summarize the categorical based appraisal data. Inter-rater reliability was calculated using percentage agreement and Gwet’s AC1. Correlation between the individual methods and the protocol was calculated using Spearman’s rank Correlation and z-test. Conventional content analysis was utilised to analyse the group discussions. Study IV utilised a multiple exploratory case study design to evaluate the current state of quality systems in the South African EMS. A formative assessment was conducted on the quality systems of four provincial EMS and one national private EMS, following which semi-structured interviews were conducted to further explore the results obtained from the formative assessment, supported by multiple secondary data sources. Descriptive statistics were utilised to describe and summarize the formative assessment. Conventional content analysis was utilised to analyse the interview data and document analysis utilised to sort and analyse the supporting data.

RESULTS: Despite relatively poor knowledge of organisational-specific quality systems, understanding of the core components and importance of quality systems was demonstrated. The role of these systems in the Low to Middle Income Country setting (LMICs) was supported by participants, where the importance of context, system transparency, reliability and validity were essential towards achieving ongoing success and utilisation. The role of leadership and communication towards the effective facilitation of such a system was equally identified. Participating services generally scored higher for structure and planning. Measurement and improvement were found to be more dependent on utilisation and perceived mandate. There was a relatively strong focus on clinical quality assessment within the private service, whereas in the provincial systems, measures were exclusively restricted to call times with little focus on clinical care. Staff engagement and programme evaluation were generally among the lowest scores. A multitude of contextual factors were identified that affected the effectiveness of quality systems, centred around leadership, vision and mission, and quality system infrastructure and capacity, guided by the need for comprehensive yet pragmatic strategic policies and standards. A total, 104 quality indicators reached consensus agreement including, 90 clinical QIs, across 15 subcategories, and 14 non-clinical QIs across two subcategories. Amongst the clinical category, airway management (n=13 QIs; 14%); out-of-hospital cardiac arrest (n=13 QIs; 14%); and acute coronary syndromes (n=11 QIs; 12%) made up the majority. Within the non-clinical category, adverse events made up the significant majority with nine QIs (64%). There was mixed inter-rater reliability of the individual methods. There was similarly poor to moderate correlation of the results obtained between the individual methods (Spearman’s rank correlation=0.42,p<0.001). From a series of 104 QIs, 11 were identified that were shared between the individual methods. A further 19 QIs were identified and not shared by each method, highlighting the benefits of a multi-method approach.

CONCLUSION: For the purposes of this study we focused on the technical competence aspect of quality, in developing our measurement framework. Towards this, we identified a significant number of QIs assessed to be valid and feasible for the South African prehospital emergency care setting. The majority of which are centred around clinically focused processes of care, measures that are lacking in current performance assessment in EMS in South Africa. However, we also discovered the importance and influencing role of the individual practitioners and quality system in which the QIs will be implemented, a point highlighted across all the methodologies and studies. Given the potential magnitude of this influence, it is of the utmost importance that any measurement framework examining technical quality, have equal in-depth understanding of these factors in order to be successful.