Studies were not excluded based upon research design and had to comply with three inclusion criteria (Table 1). A two-person hybrid approach was used for screening article titles and abstracts with inter-rater reliability ranging from 94 to 95%. Full-text articles were independently screened by two reviewers, and a two-person hybrid approach was used for data extraction. No single evaluation can serve all of these different purposes and audiences equally well. With clarity about purpose and primary intended users, the evaluator can go on to make specific design, data-gathering, and analysis decisions to meet the priority purpose and to address the intended audience and users.
In anesthesiology, the importance of systematic reviews and meta-analyses has been highlighted, and they provide diagnostic and therapeutic value to various areas, including not only perioperative management but also intensive care and outpatient anesthesia [6–13]. In the United States, the Centers for Disease Control and Prevention (CDC) have developed two systems for the purposes of collecting service utilization or output data. One system is devoted to the tracking of the delivery of HIV counseling and testing services in CDC-funded sites. This system utilizes client-level forms, which can be completed easily by the service provider and then scanned in via optical readers to allow for data analysis.
Quality of evidence
If the points form an upside-down funnel shape, with a broad base that narrows towards the top of the plot, this indicates the absence of a publication bias (Fig. 5A) [29,36]. On the other hand, if the plot shows an asymmetric shape, with no points on one side of the graph, then publication bias can be suspected (Fig. 5B). Second, to test publication bias statistically, Begg and Mazumdar’s rank correlation test8) [37] or Egger’s test9) [29] can be used.
Therefore, this article provides an easy introduction to clinicians on performing and understanding meta-analyses. A major finding from this systematic review is the lack of methodological rigor in many of the process evaluations. Almost 40% of the studies included in this review had a MMAT score of 50 or less, but the scores varied significantly in terms of study designs used by the investigators. Moreover, the frequency of low MMAT scores for multi-method and mixed method studies suggests a tendency for lower methodological quality which could point to the challenging nature of these research designs [32] or a lack of reporting guidelines. Summative assessment – evaluation – comes at the end of learning, while formative assessment provides information and support during the learning.
Article Menu
The nature of the reporting effort and indeed the communication that takes place throughout the evaluation is crucial to meeting this goal. This communication includes not only written interim reports that may be delivered to stakeholders, but also the verbal and written interaction that occurs during the evaluation. In a participatory approach to evaluation, the evaluator(s) and the intended evaluation users (see Patton, 1997, 2008) see Video clip 1 collaborate throughout all phases of the evaluation, from its design to follow-up activities specifically supporting the use of the findings. The more meaningful this engagement with stakeholders is, the more likely they are to use the final evaluation report. Unfortunately, the image that the word “evaluation” often conjures up for many in the social sector is one of a post-facto, episodic, externally done impact assessment. Reducing evaluation to only one approach or one purpose, as well as confusing it with other forms of inquiry, significantly limits its potential to provide deep learning, insights, and recommendations for making the best decisions possible in an evolving, complex, and unpredictable world.
In addition, there is a clear need for breadth of analysis in an evaluation (looking at multiple questions, phenomena, and underlying factors) to adequately cover the scope of the evaluation. All these considerations require careful reflection evalution definition in what can be a quite complicated evaluation design process. We recommend that future investigators employ rigorous theory-guided multi or mixed method approaches to evaluate the processes of implementation of KT interventions.
Although this guide is not the place to discuss whether methods-driven evaluation is justified, there are strong arguments against it. One such argument is that in IEOs (and in many similar institutional settings), one does not have the luxury of being too methods driven. In fact, the evaluation questions, types of evaluands, or types of outcomes that decision makers or other evaluation stakeholders are interested in are diverse and do not lend themselves to one singular approach or method for evaluation. For particular types of questions there are usually several methodological options with different requirements and characteristics that are better suited than others. Throughout this guide, each guidance note presents what we take to be the most relevant questions that the approach or method addresses. However, this systematic review found that process evaluations are of mixed quality and lack theoretical guidance.
Two approaches—accreditation/certification and connoisseur studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and client-centered studies are based on a subjectivist epistemology from a mass perspective. The implementation of research into healthcare practice is complex [1], with multiple levels to consider such as the patient, healthcare provider, multidisciplinary team, healthcare institution, and local and national healthcare systems.
As part of approach papers and inception reports, a third tool is the use of a design matrix. For each of the main evaluation questions, this matrix specifies the sources of evidence and the use of methods. Design matrixes may also be structured to reflect the multilevel nature (for example, global, selected countries, selected interventions) of the evaluation. Consistency here refers to the extent to which the different analytical steps of an evaluation are logically connected. The quality of inference is enhanced if there are logical connections among the initial problem statement, rationale and purpose of the evaluation, questions and scope, use of methods, data collection and analysis, and conclusions of an evaluation. In such a situation, it is better to announce “there was no strong evidence for an effect,” and to present the P value and confidence intervals.
- Information obtained through politically controlled studies is released or withheld to meet the special interests of the holder, whereas public relations information creates a positive image of an object regardless of the actual situation.
- It remains unclear why almost half of the included process evaluation studies collected data only post-implementation.
- When it is time to report, teachers engage in a process of summative assessment – evaluation – that involves professional judgment.
- This complexity makes it particularly challenging to evaluate KT intervention effectiveness [3,4,5].
- However, even with the Hartung-Knapp-Sidik-Jonkman method, when there are less than five studies with very unequal sizes, extra caution is needed.
In order to maintain transparency and objectivity throughout this process, study selection is conducted independently by at least two investigators. When there is a inconsistency in opinions, intervention is required via debate or by a third reviewer. It is essential to ensure the reproducibility of the literature selection process [25].