{"id":49,"date":"2023-06-20T22:09:12","date_gmt":"2023-06-20T22:09:12","guid":{"rendered":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/chapter\/_unknown_-12\/"},"modified":"2025-06-14T13:54:42","modified_gmt":"2025-06-14T13:54:42","slug":"_unknown_-12","status":"publish","type":"chapter","link":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/chapter\/_unknown_-12\/","title":{"raw":"Chapter 12: Evaluation approaches","rendered":"Chapter 12: Evaluation approaches"},"content":{"raw":"<div class=\"textbox shaded\">\n<h2 class=\"import-Normal\"><strong>Learning outcomes<\/strong><\/h2>\n<p class=\"import-Normal\">Upon completion of this chapter, you should be able to:<\/p>\n\n<ul>\n \t<li>Identify the key terms, concepts and approaches used in evaluation.<\/li>\n \t<li>Explain the methods of data collection and analysis for evaluations.<\/li>\n \t<li>Discuss the advantages and disadvantages of different evaluative approaches.<\/li>\n<\/ul>\n<\/div>\n<h2><strong>What is evaluation?<\/strong><\/h2>\nThere are many definitions of \u2018evaluation\u2019. Mertens and Wilson<sup>1 <\/sup>quote from Fournier:\n<p class=\"no-indent indent\">Evaluation is an applied inquiry process for collecting and synthesizing evidence that culminates in conclusions about the state of affairs, value, merit, worth, significance, or quality of a program, product, person, policy, proposal, or plan. Conclusions made in evaluations encompass both an empirical aspect (that something is the case) and a normative aspect (judgement about the value of something). <sup>(p6)<\/sup><\/p>\nHawe and colleagues\u00a0provide the following definition of (program) evaluation<sup>2<\/sup>:\n<blockquote>\n<p class=\"no-indent\">\u2026 program evaluation usually involves observing and collecting measures about how a program operates and the effects it appears to be having and comparing this to a preset standard or yardstick. (<sup>pp6\u20137<\/sup>)<\/p>\n<\/blockquote>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">A more clearer definition, however, is this one from the<em> Better Evaluation website<\/em><sup> (para1)<\/sup>:<\/p>\n\n<blockquote>\n<p class=\"import-Normal no-indent\" style=\"background-color: #ffffff;text-align: justify\">any systematic process to judge merit, worth or significance by combining evidence and values.<\/p>\n<\/blockquote>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Many words are used interchangeably to refer to evaluation, such as \u2018appraise\u2019, \u2018review\u2019, \u2018assess\u2019 and \u2018interpret\u2019. This is because the definition differs between disciplines and sectors, depending on the aims and outcomes of what is being evaluated. While evaluation is different to research, it is often considered similar because the underlying principles of collecting and assessing research evidence are the same.<sup>3<\/sup> The differences and similarities between research and evaluation are outlined extensively on the <em><a class=\"rId5\" href=\"https:\/\/www.betterevaluation.org\/blog\/week-19-ways-framing-difference-between-research-evaluation\"><span class=\"import-Hyperlink\">B<\/span><span class=\"import-Hyperlink\">etter <\/span><span class=\"import-Hyperlink\">E<\/span><span class=\"import-Hyperlink\">valuation web<\/span><span class=\"import-Hyperlink\">site<\/span><\/a>.<\/em><\/p>\n\n<h2 class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>A<\/strong><strong>pproaches<\/strong><strong> to evaluation<\/strong><\/h2>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Given that evaluation is conducted across many sectors, spanning education, agriculture, community services, international development, policing and justice, health, and others, it is unsurprising that a diverse array of evaluation approaches have been developed. Stufflebeam describes 22 program evaluation approaches in his widely cited monograph.<sup>4<\/sup> These approaches are distinguished by factors such as their purpose, scope, engagement with stakeholders, methods, timing and applications. In this chapter we focus on four approaches that are most often used in health and social care (Table 12.1.).<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>Objectives-based evaluation<\/strong><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Objectives-based evaluation methods are widely used in the education sector and public health. The focus of this approach is the assessment of whether the program\u2019s objectives, which must align with the identified needs of program participants, are achieved. This approach places strong emphasis on valid measurement of program effects. It is also referred to as \u2018impact evaluation\u2019.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Program and service managers, funding bodies and researchers usually determine the questions that guide these evaluations and the appropriate measures to answer them. While experimental or quasi-experimental methods are not essential to this approach, they are reasonably commonly used in public health to address questions concerning the causal relationships between interventions (independent variables) and the achievement of objectives (dependent variables). The objectives-based approach has been criticised for placing too great an emphasis on a tightly prescribed set of program endpoints, with insufficient attention to the process of implementing a program. However, the collection of process information is compatible with the approach, particularly for determining the association between intervention exposure and outcomes.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>Empowerment and participatory evaluation<\/strong><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Empowerment and participatory evaluation is distinguished by the prominent role it gives to program participants in the evaluation process. Participants must determine the evaluation questions, consistent with the interests they have in the program, and make decisions about the appropriate methods to use. Participants may be heavily involved in data collection, analysis and dissemination. The evaluator must respect participants\u2019 choices in all aspects of the inquiry and facilitate and build their capacity to control the evaluation in their preferred ways. Including program participants in an evaluation can be time consuming and require additional resources.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">The empowerment approach values the rich and diverse insights that participants bring from their lived experiences, and places greater emphasis on relevance than rigour. One perspective is not regarded as more \u2018true\u2019 than others, and the purpose of the evaluation is not to reach ultimately correct conclusions but rather to empower participants (especially people who are disenfranchised) and catalyse social change through raised consciousness.<sup>5<\/sup> Qualitative methods, and participatory action research (see<a href=\"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/chapter\/__unknown__-7-2\/\"> Chapter 7<\/a>), are commonly used in empowerment evaluation.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>Realist evaluation<\/strong><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">The realist evaluation approach has been advanced by Pawson and Tilley. It shares many elements of the theory of change perspective and is founded on the critical realist paradigm.<sup>6<\/sup> Researchers adopting this approach maintain that interventions are theories concerning how a set of activities will operate in given social contexts to bring about change and the achievement of desired objectives. A key role for evaluators, therefore, is to work with program managers and other stakeholders to ensure the theory of change is inherent and explicit within a program; this is usually represented as a program logic model, which is used to guide the evaluation.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">The realist evaluation approach is characterised by the purposive sampling of a wide variety of quantitative and qualitative information to shed light on the generative mechanisms of change that take place in the program, and the contextual factors that determine whether they are activated or not. The focus on learning about how change is achieved (rather than the assessment of program effects), together with its recognition of the critical role of context (defined in the broadest sense) and attention to both intended and unintended consequences, has made this an appealing approach for the evaluation of complex interventions.<sup>6<\/sup><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>Utilisation-focused evaluation<\/strong><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Patton argues that evaluations should be undertaken for specific intended uses.<sup>7<\/sup> A critical role for the evaluator, therefore, is to undertake an analysis of program stakeholders, to identify the primary users of the program and to determine the needs and associated questions they have concerning the program. Utilisation-focused evaluation has a neutral value base in that the approach acknowledges that evaluation may be undertaken to assess processes, impacts and cost\u2013benefit, to bring about improvement, to generate knowledge, or for other purposes determined by stakeholders.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">The utilisation-focused approach may adopt quantitative, qualitative or mixed methods, and these decisions are also guided by the interests of stakeholders (all those involved in a program, including those who develop, deliver and benefit from the program) and their views about the kinds of data that are credible and useful. The evaluator can facilitate this decision-making by presenting a menu of evaluation methods to stakeholders and providing expert advice to enable assessment of the utility, validity and cost-effectiveness of different options.\u00a0 This approach provides a wide ranging set of design options to stakeholders making it more expensive to conduct than applying a single evaluation design option. As Patton states, \u2018\u2026 by actively involving primary intended users, the evaluator is training users in use, preparing the groundwork for use, and reinforcing the intended utility of the evaluation every step along the way\u2019 .<sup>7(p38)<\/sup><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">No single evaluation approach is better than the other, and multiple approaches can be drawn upon within one evaluation project, depending on the purpose of the evaluation.<sup>8<\/sup><\/p>\n[table id=15 \/]\n<h2 class=\"import-Normal\"><strong>Data<\/strong><strong>-<\/strong><strong>collection methods<\/strong><strong> and analysis<\/strong><\/h2>\n<p class=\"import-Normal\" style=\"text-align: justify\">The methods of data collection and analysis rely on correct evaluation designs. Evaluation designs describe a set of tasks to systematically examine the effects of a program. A good study design creates confidence that the program caused the observed changes; because there is optimal planning, the best possible measures are used to assess the impacts of the program, no alternative explanations exist for the results and it is possible to identify how the program worked on the target population.<sup>9<\/sup><\/p>\n<p class=\"import-Normal\">It is not possible to provide an in-depth discussion about evaluation designs here, but they follow the same principles of research designs used in academic research: observational, quasi-experimental and experimental. Observational designs do not use control or comparison groups and usually measurements are taken from one point in time. They include pre-tests and post-tests, time-series designs and case studies. Quasi-experimental designs are generally more suited to public health than other types of evaluations, and aim to use a comparison group. They do not, however, randomly allocate to the groups. Quasi-experimental designs include reciprocal group design, historical control group design and stepped intervention design. Experimental study designs expose a group of people to an intervention, and this group is then compared to a control group that was not exposed to the intervention. The main types of experimental designs are randomised controlled trials (RCT) and cluster randomised controlled trials (CRCT).<sup>10<\/sup><\/p>\n<p class=\"import-Normal\">Data collection methods (discussed in Section 3) also depend on the level of evaluation being conducted. The four levels of evaluation are:<\/p>\n\n<ul>\n \t<li>Formative: prior to implementation testing and refinement<\/li>\n \t<li>Process: implementation of strategies<\/li>\n \t<li>Impact: attainment of objectives<\/li>\n \t<li>Outcome: attainment of goals.<\/li>\n<\/ul>\n<p class=\"import-Normal\">Examples of these evaluation levels are outlined in Table 12.2.<\/p>\n<p class=\"import-Normal\">Data collection measures rely on the level of evaluation being clearly articulated and planned, with clear objectives and data collection points identified. The appropriate method of data collection will be the one that can gather the information necessary to answer the evaluation questions. For example, quantitative measures and tools are best suited to collecting information about attendance at training programs (surveys and attendance sheets), but qualitative measures and tools (interviews and focus groups) are best suited to collecting information about participants\u2019 experiences of the training. The evaluator must become familiar with the program and its goals, so as to choose the most appropriate data-collection methods and tools.<\/p>\n<p class=\"import-Normal\">Data analysis follows the same principles as academic data analysis, which are explained in Section 4. For example, numerical data such as cost, attendance numbers and biometric measures are analysed for patterns, correlations, cross-tabulations, frequency tables and more. Textual analysis of spoken or written words can be analysed through content, themes, framework matrices and timelines.<\/p>\n[table id=16 \/]\n<h5 class=\"import-Normal\"><strong style=\"font-size: 1.5em\">Summary<\/strong><\/h5>\n<p class=\"import-Normal\">Evaluation is a discrete area of research yet shares many similarities with research conducted in the health and social care field. To create well-planned and delivered program evaluations, it is necessary to understand the approaches, levels and study designs.<\/p>\n\n<h2><strong>References<\/strong><\/h2>\n<ol>\n \t<li class=\"import-Normal\" style=\"background-color: #ffffff\">Mertens DM, Wilson AT. <em>Program <\/em><em>E<\/em><em>valuation <\/em><em>T<\/em><em>heory and <\/em><em>P<\/em><em>ractice: A <\/em><em>C<\/em><em>omprehensive <\/em><em>G<\/em><em>uide<\/em>. Guilford Press; 2012.<\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Hawe P, Degeling D, Hall J. <\/span><em style=\"text-align: initial;font-size: 1em\">Evaluating <\/em><em style=\"text-align: initial;font-size: 1em\">H<\/em><em style=\"text-align: initial;font-size: 1em\">ealth <\/em><em style=\"text-align: initial;font-size: 1em\">P<\/em><em style=\"text-align: initial;font-size: 1em\">romotion: A <\/em><em style=\"text-align: initial;font-size: 1em\">H<\/em><em style=\"text-align: initial;font-size: 1em\">ealth <\/em><em style=\"text-align: initial;font-size: 1em\">W<\/em><em style=\"text-align: initial;font-size: 1em\">orkers <\/em><em style=\"text-align: initial;font-size: 1em\">G<\/em><em style=\"text-align: initial;font-size: 1em\">uide<\/em><span style=\"text-align: initial;font-size: 1em\">.\u00a0MacLennan and Petty; 1990.<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">What is evaluation? Better Evaluation. 2022. Accessed March 8, 2023. <\/span><span class=\"import-Hyperlink\">https:\/\/www.betterevaluation.org\/getting-started\/what-evaluation<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Stufflebeam D. Evaluation models. <i>New Dir Eval<\/i><\/span><span style=\"text-align: initial;font-size: 1em\">. 2001;89:7-98. doi:10.1002\/ev.3<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Fawcett SB, et al. Empowering community health initiatives through evaluation. In: Fetterman DM, Kaftarian SJ,Wandersman A, eds. <\/span><em style=\"text-align: initial;font-size: 1em\">Empowerment <\/em><em style=\"text-align: initial;font-size: 1em\">E<\/em><em style=\"text-align: initial;font-size: 1em\">valuation: Knowledge and <\/em><em style=\"text-align: initial;font-size: 1em\">T<\/em><em style=\"text-align: initial;font-size: 1em\">ools for <\/em><em style=\"text-align: initial;font-size: 1em\">S<\/em><em style=\"text-align: initial;font-size: 1em\">elf-assessment and <\/em><em style=\"text-align: initial;font-size: 1em\">A<\/em><em style=\"text-align: initial;font-size: 1em\">ccountability<\/em><span style=\"text-align: initial;font-size: 1em\">. SAGE; 1996.<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Pawson R, Tilley N. <\/span><em style=\"text-align: initial;font-size: 1em\">Realistic <\/em><em style=\"text-align: initial;font-size: 1em\">E<\/em><em style=\"text-align: initial;font-size: 1em\">valuation<\/em><span style=\"text-align: initial;font-size: 1em\">. SAGE; 1997.<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Patton MQ. <\/span><em style=\"text-align: initial;font-size: 1em\">Utilization-focused <\/em><em style=\"text-align: initial;font-size: 1em\">E<\/em><em style=\"text-align: initial;font-size: 1em\">valuation<\/em><em style=\"text-align: initial;font-size: 1em\">.<\/em><span style=\"text-align: initial;font-size: 1em\">\u00a04th ed. SAGE; 2008.<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Harris M. An introduction to public and community health evaluation. In: Harris M, ed. <\/span><em style=\"text-align: initial;font-size: 1em\">Evaluating <\/em><em style=\"text-align: initial;font-size: 1em\">P<\/em><em style=\"text-align: initial;font-size: 1em\">ublic and <\/em><em style=\"text-align: initial;font-size: 1em\">C<\/em><em style=\"text-align: initial;font-size: 1em\">ommunity <\/em><em style=\"text-align: initial;font-size: 1em\">H<\/em><em style=\"text-align: initial;font-size: 1em\">ealth <\/em><em style=\"text-align: initial;font-size: 1em\">P<\/em><em style=\"text-align: initial;font-size: 1em\">rograms<\/em><em style=\"text-align: initial;font-size: 1em\">,<\/em><span style=\"text-align: initial;font-size: 1em\"> 2nd ed. John Wiley &amp; Sons; 2017.<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Australian Institute of Family Studies. Evaluation design. June 2021. Accessed March 8, 2023. <\/span><span class=\"import-Hyperlink\">https:\/\/aifs.gov.au\/resources\/practice-guides\/evaluation-design<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions.<\/span><em style=\"text-align: initial;font-size: 1em\"> J Clin Nurs.<\/em><span style=\"text-align: initial;font-size: 1em\"> 2003;12(1):77-84. doi:<\/span><span class=\"import-Hyperlink\">10.1046\/j.1365-2702.2003.00662.x<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Yakovchenko V, et al. Primary care provider perceptions and experiences of implementing hepatitis C virus birth cohort testing: a qualitative formative evaluation. <\/span><em style=\"text-align: initial;font-size: 1em\">BMC<\/em> <em style=\"text-align: initial;font-size: 1em\">Health<\/em> <em style=\"text-align: initial;font-size: 1em\">Serv <\/em><em style=\"text-align: initial;font-size: 1em\">Res.<\/em><em style=\"text-align: initial;font-size: 1em\"> 2019;<\/em><span style=\"text-align: initial;font-size: 1em\">19:236. doi:10.1186\/s12913-019-4043-z<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Morris R, et al. A mixed methods process evaluation of a person-centred falls prevention program. <\/span><em style=\"text-align: initial;font-size: 1em\">BMC Health Serv Res<\/em><em style=\"text-align: initial;font-size: 1em\">. <\/em>2019;19<strong style=\"text-align: initial;font-size: 1em\">:<\/strong><span style=\"text-align: initial;font-size: 1em\">906. doi.:10.1186\/s12913-019-4614-z<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Brenner S, et al. Design of an impact evaluation using a mixed methods model \u2013 an explanatory assessment of the effects of results-based financing mechanisms on maternal healthcare services in Malawi.\u00a0<\/span><em style=\"text-align: initial;font-size: 1em\">BMC Health Serv Res.<\/em><span style=\"text-align: initial;font-size: 1em\"> 2014;<\/span>14:<span style=\"text-align: initial;font-size: 1em\">180.<\/span><span class=\"import-Hyperlink\"> doi:10.1186\/1472-6963-14-180<\/span><\/li>\n \t<li><span style=\"text-align: initial;font-size: 1em\">Flynn R, et al. A realist evaluation to identify contexts and mechanisms that enabled and hindered implementation and had an effect on sustainability of a lean intervention in pediatric healthcare.\u00a0<\/span><em style=\"text-align: initial;font-size: 1em\">BMC Health Serv Res<\/em><em style=\"text-align: initial;font-size: 1em\">.<\/em><span style=\"text-align: initial;font-size: 1em\"> 2019;<\/span>19:<span style=\"text-align: initial;font-size: 1em\">912. doi:10.1186\/s12913-019-4744-3<\/span><\/li>\n<\/ol>","rendered":"<div class=\"textbox shaded\">\n<h2 class=\"import-Normal\"><strong>Learning outcomes<\/strong><\/h2>\n<p class=\"import-Normal\">Upon completion of this chapter, you should be able to:<\/p>\n<ul>\n<li>Identify the key terms, concepts and approaches used in evaluation.<\/li>\n<li>Explain the methods of data collection and analysis for evaluations.<\/li>\n<li>Discuss the advantages and disadvantages of different evaluative approaches.<\/li>\n<\/ul>\n<\/div>\n<h2><strong>What is evaluation?<\/strong><\/h2>\n<p>There are many definitions of \u2018evaluation\u2019. Mertens and Wilson<sup>1 <\/sup>quote from Fournier:<\/p>\n<p class=\"no-indent indent\">Evaluation is an applied inquiry process for collecting and synthesizing evidence that culminates in conclusions about the state of affairs, value, merit, worth, significance, or quality of a program, product, person, policy, proposal, or plan. Conclusions made in evaluations encompass both an empirical aspect (that something is the case) and a normative aspect (judgement about the value of something). <sup>(p6)<\/sup><\/p>\n<p>Hawe and colleagues\u00a0provide the following definition of (program) evaluation<sup>2<\/sup>:<\/p>\n<blockquote>\n<p class=\"no-indent\">\u2026 program evaluation usually involves observing and collecting measures about how a program operates and the effects it appears to be having and comparing this to a preset standard or yardstick. (<sup>pp6\u20137<\/sup>)<\/p>\n<\/blockquote>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">A more clearer definition, however, is this one from the<em> Better Evaluation website<\/em><sup> (para1)<\/sup>:<\/p>\n<blockquote>\n<p class=\"import-Normal no-indent\" style=\"background-color: #ffffff;text-align: justify\">any systematic process to judge merit, worth or significance by combining evidence and values.<\/p>\n<\/blockquote>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Many words are used interchangeably to refer to evaluation, such as \u2018appraise\u2019, \u2018review\u2019, \u2018assess\u2019 and \u2018interpret\u2019. This is because the definition differs between disciplines and sectors, depending on the aims and outcomes of what is being evaluated. While evaluation is different to research, it is often considered similar because the underlying principles of collecting and assessing research evidence are the same.<sup>3<\/sup> The differences and similarities between research and evaluation are outlined extensively on the <em><a class=\"rId5\" href=\"https:\/\/www.betterevaluation.org\/blog\/week-19-ways-framing-difference-between-research-evaluation\"><span class=\"import-Hyperlink\">B<\/span><span class=\"import-Hyperlink\">etter <\/span><span class=\"import-Hyperlink\">E<\/span><span class=\"import-Hyperlink\">valuation web<\/span><span class=\"import-Hyperlink\">site<\/span><\/a>.<\/em><\/p>\n<h2 class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>A<\/strong><strong>pproaches<\/strong><strong> to evaluation<\/strong><\/h2>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Given that evaluation is conducted across many sectors, spanning education, agriculture, community services, international development, policing and justice, health, and others, it is unsurprising that a diverse array of evaluation approaches have been developed. Stufflebeam describes 22 program evaluation approaches in his widely cited monograph.<sup>4<\/sup> These approaches are distinguished by factors such as their purpose, scope, engagement with stakeholders, methods, timing and applications. In this chapter we focus on four approaches that are most often used in health and social care (Table 12.1.).<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>Objectives-based evaluation<\/strong><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Objectives-based evaluation methods are widely used in the education sector and public health. The focus of this approach is the assessment of whether the program\u2019s objectives, which must align with the identified needs of program participants, are achieved. This approach places strong emphasis on valid measurement of program effects. It is also referred to as \u2018impact evaluation\u2019.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Program and service managers, funding bodies and researchers usually determine the questions that guide these evaluations and the appropriate measures to answer them. While experimental or quasi-experimental methods are not essential to this approach, they are reasonably commonly used in public health to address questions concerning the causal relationships between interventions (independent variables) and the achievement of objectives (dependent variables). The objectives-based approach has been criticised for placing too great an emphasis on a tightly prescribed set of program endpoints, with insufficient attention to the process of implementing a program. However, the collection of process information is compatible with the approach, particularly for determining the association between intervention exposure and outcomes.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>Empowerment and participatory evaluation<\/strong><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Empowerment and participatory evaluation is distinguished by the prominent role it gives to program participants in the evaluation process. Participants must determine the evaluation questions, consistent with the interests they have in the program, and make decisions about the appropriate methods to use. Participants may be heavily involved in data collection, analysis and dissemination. The evaluator must respect participants\u2019 choices in all aspects of the inquiry and facilitate and build their capacity to control the evaluation in their preferred ways. Including program participants in an evaluation can be time consuming and require additional resources.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">The empowerment approach values the rich and diverse insights that participants bring from their lived experiences, and places greater emphasis on relevance than rigour. One perspective is not regarded as more \u2018true\u2019 than others, and the purpose of the evaluation is not to reach ultimately correct conclusions but rather to empower participants (especially people who are disenfranchised) and catalyse social change through raised consciousness.<sup>5<\/sup> Qualitative methods, and participatory action research (see<a href=\"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/chapter\/__unknown__-7-2\/\"> Chapter 7<\/a>), are commonly used in empowerment evaluation.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>Realist evaluation<\/strong><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">The realist evaluation approach has been advanced by Pawson and Tilley. It shares many elements of the theory of change perspective and is founded on the critical realist paradigm.<sup>6<\/sup> Researchers adopting this approach maintain that interventions are theories concerning how a set of activities will operate in given social contexts to bring about change and the achievement of desired objectives. A key role for evaluators, therefore, is to work with program managers and other stakeholders to ensure the theory of change is inherent and explicit within a program; this is usually represented as a program logic model, which is used to guide the evaluation.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">The realist evaluation approach is characterised by the purposive sampling of a wide variety of quantitative and qualitative information to shed light on the generative mechanisms of change that take place in the program, and the contextual factors that determine whether they are activated or not. The focus on learning about how change is achieved (rather than the assessment of program effects), together with its recognition of the critical role of context (defined in the broadest sense) and attention to both intended and unintended consequences, has made this an appealing approach for the evaluation of complex interventions.<sup>6<\/sup><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\"><strong>Utilisation-focused evaluation<\/strong><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">Patton argues that evaluations should be undertaken for specific intended uses.<sup>7<\/sup> A critical role for the evaluator, therefore, is to undertake an analysis of program stakeholders, to identify the primary users of the program and to determine the needs and associated questions they have concerning the program. Utilisation-focused evaluation has a neutral value base in that the approach acknowledges that evaluation may be undertaken to assess processes, impacts and cost\u2013benefit, to bring about improvement, to generate knowledge, or for other purposes determined by stakeholders.<\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">The utilisation-focused approach may adopt quantitative, qualitative or mixed methods, and these decisions are also guided by the interests of stakeholders (all those involved in a program, including those who develop, deliver and benefit from the program) and their views about the kinds of data that are credible and useful. The evaluator can facilitate this decision-making by presenting a menu of evaluation methods to stakeholders and providing expert advice to enable assessment of the utility, validity and cost-effectiveness of different options.\u00a0 This approach provides a wide ranging set of design options to stakeholders making it more expensive to conduct than applying a single evaluation design option. As Patton states, \u2018\u2026 by actively involving primary intended users, the evaluator is training users in use, preparing the groundwork for use, and reinforcing the intended utility of the evaluation every step along the way\u2019 .<sup>7(p38)<\/sup><\/p>\n<p class=\"import-Normal\" style=\"background-color: #ffffff;text-align: justify\">No single evaluation approach is better than the other, and multiple approaches can be drawn upon within one evaluation project, depending on the purpose of the evaluation.<sup>8<\/sup><\/p>\n<p>[table id=15 \/]<\/p>\n<h2 class=\"import-Normal\"><strong>Data<\/strong><strong>&#8211;<\/strong><strong>collection methods<\/strong><strong> and analysis<\/strong><\/h2>\n<p class=\"import-Normal\" style=\"text-align: justify\">The methods of data collection and analysis rely on correct evaluation designs. Evaluation designs describe a set of tasks to systematically examine the effects of a program. A good study design creates confidence that the program caused the observed changes; because there is optimal planning, the best possible measures are used to assess the impacts of the program, no alternative explanations exist for the results and it is possible to identify how the program worked on the target population.<sup>9<\/sup><\/p>\n<p class=\"import-Normal\">It is not possible to provide an in-depth discussion about evaluation designs here, but they follow the same principles of research designs used in academic research: observational, quasi-experimental and experimental. Observational designs do not use control or comparison groups and usually measurements are taken from one point in time. They include pre-tests and post-tests, time-series designs and case studies. Quasi-experimental designs are generally more suited to public health than other types of evaluations, and aim to use a comparison group. They do not, however, randomly allocate to the groups. Quasi-experimental designs include reciprocal group design, historical control group design and stepped intervention design. Experimental study designs expose a group of people to an intervention, and this group is then compared to a control group that was not exposed to the intervention. The main types of experimental designs are randomised controlled trials (RCT) and cluster randomised controlled trials (CRCT).<sup>10<\/sup><\/p>\n<p class=\"import-Normal\">Data collection methods (discussed in Section 3) also depend on the level of evaluation being conducted. The four levels of evaluation are:<\/p>\n<ul>\n<li>Formative: prior to implementation testing and refinement<\/li>\n<li>Process: implementation of strategies<\/li>\n<li>Impact: attainment of objectives<\/li>\n<li>Outcome: attainment of goals.<\/li>\n<\/ul>\n<p class=\"import-Normal\">Examples of these evaluation levels are outlined in Table 12.2.<\/p>\n<p class=\"import-Normal\">Data collection measures rely on the level of evaluation being clearly articulated and planned, with clear objectives and data collection points identified. The appropriate method of data collection will be the one that can gather the information necessary to answer the evaluation questions. For example, quantitative measures and tools are best suited to collecting information about attendance at training programs (surveys and attendance sheets), but qualitative measures and tools (interviews and focus groups) are best suited to collecting information about participants\u2019 experiences of the training. The evaluator must become familiar with the program and its goals, so as to choose the most appropriate data-collection methods and tools.<\/p>\n<p class=\"import-Normal\">Data analysis follows the same principles as academic data analysis, which are explained in Section 4. For example, numerical data such as cost, attendance numbers and biometric measures are analysed for patterns, correlations, cross-tabulations, frequency tables and more. Textual analysis of spoken or written words can be analysed through content, themes, framework matrices and timelines.<\/p>\n<p>[table id=16 \/]<\/p>\n<h5 class=\"import-Normal\"><strong style=\"font-size: 1.5em\">Summary<\/strong><\/h5>\n<p class=\"import-Normal\">Evaluation is a discrete area of research yet shares many similarities with research conducted in the health and social care field. To create well-planned and delivered program evaluations, it is necessary to understand the approaches, levels and study designs.<\/p>\n<h2><strong>References<\/strong><\/h2>\n<ol>\n<li class=\"import-Normal\" style=\"background-color: #ffffff\">Mertens DM, Wilson AT. <em>Program <\/em><em>E<\/em><em>valuation <\/em><em>T<\/em><em>heory and <\/em><em>P<\/em><em>ractice: A <\/em><em>C<\/em><em>omprehensive <\/em><em>G<\/em><em>uide<\/em>. Guilford Press; 2012.<\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Hawe P, Degeling D, Hall J. <\/span><em style=\"text-align: initial;font-size: 1em\">Evaluating <\/em><em style=\"text-align: initial;font-size: 1em\">H<\/em><em style=\"text-align: initial;font-size: 1em\">ealth <\/em><em style=\"text-align: initial;font-size: 1em\">P<\/em><em style=\"text-align: initial;font-size: 1em\">romotion: A <\/em><em style=\"text-align: initial;font-size: 1em\">H<\/em><em style=\"text-align: initial;font-size: 1em\">ealth <\/em><em style=\"text-align: initial;font-size: 1em\">W<\/em><em style=\"text-align: initial;font-size: 1em\">orkers <\/em><em style=\"text-align: initial;font-size: 1em\">G<\/em><em style=\"text-align: initial;font-size: 1em\">uide<\/em><span style=\"text-align: initial;font-size: 1em\">.\u00a0MacLennan and Petty; 1990.<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">What is evaluation? Better Evaluation. 2022. Accessed March 8, 2023. <\/span><span class=\"import-Hyperlink\">https:\/\/www.betterevaluation.org\/getting-started\/what-evaluation<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Stufflebeam D. Evaluation models. <i>New Dir Eval<\/i><\/span><span style=\"text-align: initial;font-size: 1em\">. 2001;89:7-98. doi:10.1002\/ev.3<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Fawcett SB, et al. Empowering community health initiatives through evaluation. In: Fetterman DM, Kaftarian SJ,Wandersman A, eds. <\/span><em style=\"text-align: initial;font-size: 1em\">Empowerment <\/em><em style=\"text-align: initial;font-size: 1em\">E<\/em><em style=\"text-align: initial;font-size: 1em\">valuation: Knowledge and <\/em><em style=\"text-align: initial;font-size: 1em\">T<\/em><em style=\"text-align: initial;font-size: 1em\">ools for <\/em><em style=\"text-align: initial;font-size: 1em\">S<\/em><em style=\"text-align: initial;font-size: 1em\">elf-assessment and <\/em><em style=\"text-align: initial;font-size: 1em\">A<\/em><em style=\"text-align: initial;font-size: 1em\">ccountability<\/em><span style=\"text-align: initial;font-size: 1em\">. SAGE; 1996.<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Pawson R, Tilley N. <\/span><em style=\"text-align: initial;font-size: 1em\">Realistic <\/em><em style=\"text-align: initial;font-size: 1em\">E<\/em><em style=\"text-align: initial;font-size: 1em\">valuation<\/em><span style=\"text-align: initial;font-size: 1em\">. SAGE; 1997.<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Patton MQ. <\/span><em style=\"text-align: initial;font-size: 1em\">Utilization-focused <\/em><em style=\"text-align: initial;font-size: 1em\">E<\/em><em style=\"text-align: initial;font-size: 1em\">valuation<\/em><em style=\"text-align: initial;font-size: 1em\">.<\/em><span style=\"text-align: initial;font-size: 1em\">\u00a04th ed. SAGE; 2008.<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Harris M. An introduction to public and community health evaluation. In: Harris M, ed. <\/span><em style=\"text-align: initial;font-size: 1em\">Evaluating <\/em><em style=\"text-align: initial;font-size: 1em\">P<\/em><em style=\"text-align: initial;font-size: 1em\">ublic and <\/em><em style=\"text-align: initial;font-size: 1em\">C<\/em><em style=\"text-align: initial;font-size: 1em\">ommunity <\/em><em style=\"text-align: initial;font-size: 1em\">H<\/em><em style=\"text-align: initial;font-size: 1em\">ealth <\/em><em style=\"text-align: initial;font-size: 1em\">P<\/em><em style=\"text-align: initial;font-size: 1em\">rograms<\/em><em style=\"text-align: initial;font-size: 1em\">,<\/em><span style=\"text-align: initial;font-size: 1em\"> 2nd ed. John Wiley &amp; Sons; 2017.<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Australian Institute of Family Studies. Evaluation design. June 2021. Accessed March 8, 2023. <\/span><span class=\"import-Hyperlink\">https:\/\/aifs.gov.au\/resources\/practice-guides\/evaluation-design<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions.<\/span><em style=\"text-align: initial;font-size: 1em\"> J Clin Nurs.<\/em><span style=\"text-align: initial;font-size: 1em\"> 2003;12(1):77-84. doi:<\/span><span class=\"import-Hyperlink\">10.1046\/j.1365-2702.2003.00662.x<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Yakovchenko V, et al. Primary care provider perceptions and experiences of implementing hepatitis C virus birth cohort testing: a qualitative formative evaluation. <\/span><em style=\"text-align: initial;font-size: 1em\">BMC<\/em> <em style=\"text-align: initial;font-size: 1em\">Health<\/em> <em style=\"text-align: initial;font-size: 1em\">Serv <\/em><em style=\"text-align: initial;font-size: 1em\">Res.<\/em><em style=\"text-align: initial;font-size: 1em\"> 2019;<\/em><span style=\"text-align: initial;font-size: 1em\">19:236. doi:10.1186\/s12913-019-4043-z<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Morris R, et al. A mixed methods process evaluation of a person-centred falls prevention program. <\/span><em style=\"text-align: initial;font-size: 1em\">BMC Health Serv Res<\/em><em style=\"text-align: initial;font-size: 1em\">. <\/em>2019;19<strong style=\"text-align: initial;font-size: 1em\">:<\/strong><span style=\"text-align: initial;font-size: 1em\">906. doi.:10.1186\/s12913-019-4614-z<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Brenner S, et al. Design of an impact evaluation using a mixed methods model \u2013 an explanatory assessment of the effects of results-based financing mechanisms on maternal healthcare services in Malawi.\u00a0<\/span><em style=\"text-align: initial;font-size: 1em\">BMC Health Serv Res.<\/em><span style=\"text-align: initial;font-size: 1em\"> 2014;<\/span>14:<span style=\"text-align: initial;font-size: 1em\">180.<\/span><span class=\"import-Hyperlink\"> doi:10.1186\/1472-6963-14-180<\/span><\/li>\n<li><span style=\"text-align: initial;font-size: 1em\">Flynn R, et al. A realist evaluation to identify contexts and mechanisms that enabled and hindered implementation and had an effect on sustainability of a lean intervention in pediatric healthcare.\u00a0<\/span><em style=\"text-align: initial;font-size: 1em\">BMC Health Serv Res<\/em><em style=\"text-align: initial;font-size: 1em\">.<\/em><span style=\"text-align: initial;font-size: 1em\"> 2019;<\/span>19:<span style=\"text-align: initial;font-size: 1em\">912. doi:10.1186\/s12913-019-4744-3<\/span><\/li>\n<\/ol>\n","protected":false},"author":35,"menu_order":9,"template":"","meta":{"pb_show_title":"","pb_short_title":"","pb_subtitle":"","pb_authors":["tess-tsindos"],"pb_section_license":""},"chapter-type":[],"contributor":[62],"license":[],"class_list":["post-49","chapter","type-chapter","status-publish","hentry","contributor-tess-tsindos"],"part":32,"_links":{"self":[{"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/pressbooks\/v2\/chapters\/49","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/pressbooks\/v2\/chapters"}],"about":[{"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/wp\/v2\/types\/chapter"}],"author":[{"embeddable":true,"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/wp\/v2\/users\/35"}],"version-history":[{"count":1,"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/pressbooks\/v2\/chapters\/49\/revisions"}],"predecessor-version":[{"id":50,"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/pressbooks\/v2\/chapters\/49\/revisions\/50"}],"part":[{"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/pressbooks\/v2\/parts\/32"}],"metadata":[{"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/pressbooks\/v2\/chapters\/49\/metadata\/"}],"wp:attachment":[{"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/wp\/v2\/media?parent=49"}],"wp:term":[{"taxonomy":"chapter-type","embeddable":true,"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/pressbooks\/v2\/chapter-type?post=49"},{"taxonomy":"contributor","embeddable":true,"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/wp\/v2\/contributor?post=49"},{"taxonomy":"license","embeddable":true,"href":"https:\/\/openpub.libraries.rutgers.edu\/qualgo2\/wp-json\/wp\/v2\/license?post=49"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}