Evaluation of complex interventions requires a comprehensive understanding of intervention context, mechanisms and outcomes. We recognise context as a key influence in achieving the intended results. [PMC free article] Marchal B, Van Belle SB, Van Olmen J, Hoeree T, Kegels G. Is realist evaluation keeping its promise_ A review of published empirical studies in the field of health systems research. Pommier J, Guevel M-R, Jourdan D. Evaluation of health promotion in schools: a realistic evaluation approach using mixed methods. FMoH. Saving newborn lives in Nigeria: Newborn health in the context of the Integrated Maternal, Newborn and Child Health Strategy. ‘How To’ Guide On Research & Development: The Embedded Approach. Since December 2012, CCTs have been implemented in selected SURE-P/MCH sites in 9 of the 36 states of Nigeria (SURE-P/MCH + CCT). Analyzing Qualitative Data. Qualitative methods will include (a) IDIs with key actors, including service users, (b) reviews of key documents and (c) exit IDIs and focus group discussions (FGDs) with service users and their family members. Ozawa S, Pongpirul K. 10 best resources on … mixed methods research in health systems. Specific emphasis will be placed on confidentiality and other data protection issues, which will include security of data storage and access rights to data. Phase 2 of the SURE-P/MCH was launched in late-2013 and aimed for an incremental expansion to a further 12 to 21 facilities in the selected states. This is known as the C-M-O configuration , and without accounting for all these dimensions, some aspects of the programme may go unrecorded, thus affecting the validity and reliability of results  and preventing replication . exploring the intervention mechanisms); identify the key contextual facilitators and constraints and analyse in what ways these contextual influences affected the achievement of outcomes. Implementation Science Google Scholar. The study will draw upon health economics, social sciences and statistics. Frehywot S, Mullan F, Payne P, Ross H. Compulsory service programmes for recruiting health workers in remote and rural areas: do they work? STUDY PROTOCOL Open Access Protocol for a longitudinal mixed methods realist evaluation of holistic needs assessment and care planning for people affected by cancer Lucy Johnston* and Karen Campbell Abstract Background: In 2012, approximately 14 million new cases of cancer were diagnosed. Methodology and analysis This paper disseminates the protocol for an 18-month ‘RESPOND’ project that aims to evaluate the system of collecting and responding to user feedback in Bangladesh. The project will be implemented according to standard governance practices at the University of Leeds and University of Nigeria. organisations and their roles) and micro level (e.g. 1) provides an overarching hypothesis, and more specific hypothetical pathways will represent the middle-range theories (MRT) to help us explore the C-M-O configurations within the programme. The two interventions (i.e. t Lehmann U, Sanders D. Community health workers: What do we know about them? Based on this interest, and following the competitive evaluation of research proposals from the Joint MRC/ESRC/DFID/Wellcome Trust health systems research initiative call 1, in June 2015, we initiated a 5-year research programme to assess the MCH component in Anambra state. Lewin S, Munabi-Babigumira S, Glenton C, Daniels K, Bosch-Capblanch X, van Wyk BE, Odgaard-Jensen J, Johansen M, Aja GN, Zwarenstein M et al: Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases (Review). While the average number of PHC facilities in each Nigerian state is typically between 1000 and 4000, the SURE-P/MCH was implemented in clusters of 12 to 21 health centres within selected states. This study will make an important and timely contribution to health systems strengthening in Nigeria. The supply component also included infrastructure development, improving availability of supplies and medicines and activation of Ward Development Committees (WDCs). The purpose of this paper is to share the study protocol for realist evaluation of CHW programme in Nigeria. acceptability of the intervention by the communities and front-line service providers, appropriateness of the intervention design to the current context of PHC facilities in Anambra state and sustainability of changes achieved from the implementation of interventions in the longer term . Two aspects of the environment within which the study is being implemented are worth noting. 2nd ed. Theory-driven forms of evaluation help understanding such complexity by studying how the different elements are intertwined  and recognising the role of context as a key influence in the production of outcomes . Realist Synthesis: An Introduction: This paper presents an introductory overview of realist synthesis as applied to the review of primary research on healthcare systems. The panels will use data from each facility in the three study clusters covering the period from at least 12 months before and 12 months after the intervention began in the two intervention clusters and equivalent periods for the control cluster. This technique assumes that knowledge is a social and historical product, thus the social and political context as well as theoretical mechanisms, need consideration in analysis of programme or policy effectiveness. The eventual decision was to continue with the study, using the original methodology, though with the addition of assessment of sustainability of achieved changes and effects of on-going lobbying and advocacy efforts on entrenching the MCH on the political agenda in Nigeria. These costs will be captured through a small facility exit survey of women who have used services both before and after implementation of the SURE-P/MCH to understand how user costs have changed over time. J Dev Econ. In: Bryman A, Burgess R, editors. Health Policy Plan. Implementing community-based perinatal care: results from a pilot study in rural Pakistan. Health Policy Plan. While studies have explored the effects of supply- and demand-side interventions separately [15, 18, 19], the combined effects of the two—such as the added value of CCTs within health programmes—are rarely assessed within the same intervention. These hypotheses will be developed in discussions with programme managers and implementers. Subsidy Reinvestment and Empowerment (SURE) Programme. The suggested protocol uses a realist evaluation with multiple cases across the 3 country sites: Turkey, Iraq and Lebanon. Realist evaluation is a form of theory-driven evaluation, but is set apart by its explicit philosophical underpinnings. J Health Serv Res Policy. Mirzoev T, Omar M, Green A, Bird P, Lund C, Ofori-Atta A, et al. Adam T, Lim SS, Mehta S, Bhutta ZA, Fogstad H, Mathai M, et al. Assessment of ITS design against quality criteria. This realist evaluation protocol will contribute to this knowledge in Zambia where there is increased attention towards promoting maternal, neonatal and child health as a means of addressing the current high early pregnancy and marriage rates. This includes ensuring regular communication between the partners and engagement with policymakers and practitioners; quality assurance through regular peer-review within and between the teams; appropriate mentoring and coaching support to more junior researchers and equal opportunities to both genders. R Pawson, N. Tilley, "Realistic Evaluation" (Sage), London. An incremental cost-effectiveness analysis will be undertaken (cost per additional skilled delivery and associated care). The assessment of the ITS design against standard quality checklist  is shown in Additional file 1. Gopalan SS, Mohanty S, Das A. Assessing community health workers’ performance motivation: a mixed-methods approach on India's Accredited Social Health Activists (ASHA) programme. Design: A multimethod study design was undertaken using realist evaluation methodology. However, to guide further development and implementation of CHW programmes in different contexts, it is necessary to better understand what makes CHW programmes successful in achieving desired outcomes and under what circumstances they succeed [11, 12]. The NDP comprises a range of organisational and service delivery changes to support evidence-based practices and policies. PubMed Ebenso B, Manzano A, Uzochukwu B, Huss R, Etiaba E, Ensor T, et al. The suggested protocol uses a realist evaluation with multiple cases across the 3 country sites: Turkey, Iraq and Lebanon. capabilities, values and interests of individuals) [38, 39]. studies, policy documents and so on) and so it is a form of secondary research. Implementing accountability for reasonableness framework at district level in Tanzania: a realist evaluation. Research-policy partnerships - experiences of the Mental Health and Poverty Project in Ghana, South Africa, Uganda and Zambia. This is the key question posed by realist evaluators. The challenge posed by dementia, which is the confluence of increasing life expectancy, population growth and the lack of cura-tive treatments, has been well documented [42, 54]. RE was founded on the need to better understand, identify and evaluate why complex interventions succeed or fail in order to inform spread and replicability of effective interventions .RE is well placed to provide much needed transferable findings that are of practical use for practitioners and decision makers. Prashanth NS, Marchal B, Hoeree T, Devadasan N, Macq J, Kegels G, Criel B. In this paper, we report a protocol for realist evaluation study of DeteRminants of Effectiveness and sustainability of a noVel Community HeAlth Workers (CHWs) programMe in imProving maternal and child health in Nigeria (REVAMP project). reproductive and/or MCH, health workforce) and (b) a limited number (3–5 in total) of initial in-depth interviews (IDIs) with purposefully selected federal and state SURE-P programme managers. A framework approach will be used for analysis to test hypotheses, while allowing for emergence of new themes . Neither of the funders had any role in the design of this study. Achievement of improved maternal and child health (MCH) outcomes continues to be an issue of international priority, particularly for sub-Saharan African countries such as Nigeria. "Understanding integrated care pathways in palliative care using realist evaluation:a mixed methods study protocol", https://en.wikipedia.org/w/index.php?title=Realist_Evaluation&oldid=906893181, Creative Commons Attribution-ShareAlike License, This page was last edited on 19 July 2019, at 00:28. actors’ behaviours in implementing intervention) to produce intended and unintended Outcomes. We will use the understanding of the context in which interventions are operating to identify these consequences. 2012;10(1):30. Options discussed included stopping or amending the research, as well as technical and political implications of each option from the different (policymakers’ and the funder’s) perspectives. The respondents will be current and former service users, recruited at the point of exiting from health facilities. Rather than attempting to cost all services, we will use an incremental approach that examines the additional programme benefits relative to the additional SURE-P costs. Lehmann U, Gilson L. Actor interfaces and practices of power in a community health worker programme: a South African study of unintended policy outcomes. Realist evaluation provides an overall methodological approach for the project and will guide development, testing and refining of middle-range theories through the analysis of the relationships between the context (at macro, meso and micro levels), mechanisms and outcomes . Health Policy Plan. it is a form of primary research. A cluster is made up of four PHC facilities and one General Hospital (GH). It was identified in consultation with the Federal and State Ministry of Health (MOH) and the SURE-P national team lead. Ethical approvals for this study were obtained from the University of Leeds (ref: SoMREC/14/097) and the University of Nigeria (ref: NHREC/05/02/2008B-FWA00002458-1RB00002323). The study protocol reported in this paper has been evaluated through open competitive peer-review process as part of the Joint MRC/ESRC/DFID/Wellcome Trust health systems research initiative call 1 and accepted for funding (grant ref: MR/M01472X/1). BMJ Open 2012, 2(2). Results: A protocol for a mixed methods realist evaluation was developed to gain insights into the mechanisms that foster successful results in ICPs. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. The project uses an input-process-output-outcome continuum (see Fig. The development of interview question guides will be informed by the study conceptual framework and structured around the study research questions to explore the specific hypothetical pathways identified in step 1. As with all primary research approaches, guidance on quality assurance and uniform reporting is an important step towards improving quality and consistency. statement and Google Scholar. In addition to changes in exogenous variables such as pay scales and size of the programme, we will include key contextual changes that can impact on the cost-effectiveness of the programme. Malar J. Abuja: Federal Ministry of Health; 2013. Article Ethical approvals for this study were obtained from the School of Medicine Research Ethics Committee at the Faculty of Medicine and Health at the University of Leeds (ref: SoMREC/14/097) and the Health Research Ethics Committee at the University of Nigeria Teaching Hospital, Enugu (ref: NHREC/05/02/2008B-FWA00002458-1RB00002323). Pawson R, Manzano-Santaella A. Ramsay CR, Matowe L, Grilli R, Grimshaw JM, Thomas RE. Spill-over effects and unintended consequences of the programme will be tracked, quantified and also explored qualitatively. Protocol for a mixed methods realist evaluation of regional District Health Board groupings in New Zealand Tim Stokes , 1 Carol Atmore , 1 Erin Penno , 1 Lauralie Richard , 1 Emma Wyeth , 2 Rosalina Richards , 3 Fiona Doolan-Noble , 1 Andrew R Gray , 4 Trudy Sullivan , 5 and Robin Gauld 6 2014;100:72–83. How does capacity building of health managers work? with and without CCTs) of a Nigerian CHW programme to understand what contextual factors promote equitable access to quality services and examining the conditions under which these changes can be sustained following withdrawal of funding. Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies. This handbook is available upon request. Mixed methods study designs that are increasingly used to evaluate complex interventions  include the exploratory (qualitative methods followed by quantitative), explanatory (quantitative then qualitative), embedded (one dataset provides a supportive secondary role) and convergent (both datasets are complementary to each other) models [35, 36]. Ricketts JA. However, if we reach data saturation earlier, these numbers may decrease. The authors declare that they have no competing interests. Accountability for Maternal, Newborn and Child Survival: An Update on Progress in Priority Countries.  It is based on the epistemological foundations of critical realism. skilled midwives) and 10,000 CHWs, comprising 1000 Community Health Extension Workers (CHEWs) and 9000 volunteer Village Health Workers (VHWs). The project workplan showing overlaps between the three steps is included in Additional file 2. Assessment of capacity for Health Policy and Systems Research and Analysis in seven African universities: results from the CHEPSAA project. t Developing and Evaluating Complex Interventions. CHW programmes are inherently complex, and their success is mediated by how the intervention is implemented within the health system context. Mirzoev, T., Etiaba, E., Ebenso, B. et al. All interviews will be audio-recorded (subject to informed consent), transcribed and translated into English where required. Consumers stated and revealed preferences for community health workers and other strategies for the provision of timely and appropriate treatment of malaria in southeast Nigeria. Realist approaches to evaluation assume that nothing works everywhere or for everyone: context really does make a difference to programme outcomes. Effectiveness of a programme is thus not dependent on the outcomes alone (cause–effect), rather there is a consideration of the theoretical mechanisms that are applied, and the socio-historical context in which the programmes were implemented. MRC. (DOCX 29 kb), Ethics approval from the University of Leeds. This will be achieved by investigating two intervention strategies (i.e. equitable access to quality MCH services and attainment of MCH outcome targets) during and after withdrawal of targeted support to the programme, Develop an empirically based and theoretically grounded dynamic model of complex relations between the actors, context, implementation process, outputs and outcomes of the interventions during, and after withdrawal, of targeted support to the programme, Assess the role of different advocacy and lobbying efforts in entrenching MCH on the political agenda and strengthening the provision of MCH services, following the suspension of targeted support to SURE-P/MCH, Develop transferable best practices for scalability (expansion within a broadly similar context) and generalisability (expansion to different contexts) of the lessons learned. An example of the former is: ‘training and deployment of CHWs combined with a working supplies system, and implemented in the context Nigerian health system, will achieve increase in skilled birth attendance by over 60 % and improve equitable provision of MCH care’. https://doi.org/10.1186/s13012-016-0443-1, DOI: https://doi.org/10.1186/s13012-016-0443-1. These are available in Additional files 3 and 4, respectively. It is important to stress that realist evaluation cannot be achieved simply by following a protocol in a technically correct manner. Introduction Significant evidence in the literature supports case management (CM) as an effective intervention to improve care for patients with complex healthcare needs. We will explore views of MCH service users on SURE-P, its costs and effects. t We discussed the best course of action for our research with Nigerian health authorities and our funders. It explains when a realist impact evaluation may be most This paper should be of interest to researchers who are adapting and applying robust methodologies for assessing complex health systems interventions and policymakers and practitioners who are interested in commissioning, and engaging in, similar evaluations. The realist evaluation approach is a relatively new approach to evaluation, especially in some fields (such as health services research). Study protocol: realist evaluation of effectiveness and sustainability of a community health workers programme in improving maternal and child health in Nigeria. PubMed More accessible midwives/CHWs, for example, may reduce the cost to households of obtaining care. PubMed Central This economic evaluation will use data from analysis of costing data from HMIS and SURE-P and will draw upon the results of the facility exit survey. Research Methodology. it METHODS AND ANALYSIS: The study will be conducted in four phases. Realistic evaluation is an increasingly popular approach, but few published examples exist, particularly in implementation research.