Evaluation Research Exam Questions Schutt Ch.13

Chapter 13

1. A systems model of evaluation includes all of the following but:

a. Inputs.

b. Program theory.

c. Feedback.

d. Stakeholders.

2. A program may be unsuccessful because:

a. The practice theory is incorrect.

b. Inputs are sufficient.

c. All the activites are present.

d. Stakeholders agree about outcomes.

3. A logic model may:

a. Describe theory and its link to change.

b. Connect resources and activities to outcomes.

c. Link activities to outputs.

d. All of the above.

4. Inputs may include all but which of the following?

a. Money

b. Social workers

c. Social work interns

d. Case management

5. _______ describes activities undertaken by the program staff.

a. Units of service

b. Initial outcomes

c. Service completions

d. Assumptions

6. Units of service can be described by:

a. Behaviors.

b. Attitudes.

c. Duration of time.

d. Feelings.

7. A distinctive feature of evaluation research is that variation in both outputs and outcomes influences the inputs to the program through a ________ process.

a. Reaction

b. Evaluative

c. Stakeholder

d. Feedback

8. Which method may be used for need assessment:

a. Experimental group designs.

b. Single subject designs.

c. Social indicators.

d. Process assessment.

9. The economic value of a social program when compared to the costs of that program is established in:

a. A cost-benefit analysis.

b. A cost-effectiveness analysis.

c. A process analysis.

d. A summative assessment.

10. Theory-driven evaluation does which of the following?

a. Assesses the relative costs of a social program

b. Determines the generalizablity of a program

c. Guides the investigation of a program process

d. Suggests which outcomes should be measured

11. The services delivered or new products produced by a social program are:

a. Program process.

b. Program outputs.

c. Program theory.

d. Program participation.

12. Eastern City proposes developing a community policing program to reduce crime in one neighborhood. The police department, in conjunction with the mayor’s office, held several meetings in which they discussed this proposal with neighborhood residents, business owners, and social organizations. In this example, the police department, the mayor’s office, and neighborhood residents are all:

a. Program participants.

b. Key informants.

c. Research subjects.

d. Stakeholders.

13. “Is the program serving its target question,” is a question answered by:

a. Need assessment.

b. Process evaluation.

c. Outcome evaluation.

d. Efficiency evaluation.

14. Program coverage can be monitored using:

a. Need assessment.

b. Flowcharts.

c. Program records.

d. Qualitative methods.

15. When evaluating their program, in which group exercise classes tended to increase several measures of participant health, researchers concluded that it did not really matter how the group classes worked, just so long as it worked. This is an example of:

a. Black box theory.

b. Program theory.

c. Utilization theory.

d. Theory-driven evaluation.

16. If a program evaluation is written without regard to whether the program had the intended effect, but with very careful attention to how it will be received by other social science peers, it has what kind of orientation or approach:

a. Process.

b. Stakeholder.

c. Social science.

d. Theoretical.

17. ______________ emphasize(s) the importance of researcher expertise and maintenance of some autonomy to develop the most trustworthy, unbiased program evaluation.

a. Process evaluation

b. Social science approaches

c. Efficiency analysis

d. Stakeholders’ perceptions

18. A community hires a researcher to determine whether a community center would be a benefit, in terms of reducing delinquency, promoting community service, and developing networks. To provide an answer, the researcher would conduct what sort of evaluation on community services:

a. Needs assessment.

b. Evaluability assessment.

c. Process evaluation.

d. Efficiency analysis.

19. To answer the question, “Did the program work?” researchers rely on which form of evaluation research?

a. Needs assessment

b. Logic modeling

c. Process evaluation

d. Outcome evaluation

20. Outcome evaluation generally relies on what sort of research design?

a. Experimental and quasi-experimental

b. Open-ended surveys

c. One-shot case studies

d. Key informant interviewing

21. To lessen the potential detrimental impact of social programs, researchers should do all of the following, EXCEPT:

a. Vary treatments among settings rather than among individuals within a setting

b. Use the minimum sample size required to adequately test the results

c. Minimize the number in the untreated control group

d. Compare the presence and absence of treatments, rather than those that vary in intensity

22. In a social program, _________ go into a program, while _________ is the direct products of the program’s service delivery process.

a. Outcome; output

b. Output; feedback

c. Feedback; output

d. Input; output

23. _________ evaluation compares what happened after a program with what would have happened had there been no program.

a. Outcome

b. Needs

c. Efficiency

d. Participatory

24. To evaluate a program designed to improve self-esteem for elementary school girls, girls were divided into treatment and control groups. Girls in the treatment group were required to attend an after-school program, while girls in the control group went home on time. Scores on self-esteem measures were then compared between the two groups of girls. This type of program design is associated with:

a. Needs assessment.

b. Efficiency evaluation.

c. Process evaluation.

d. Outcome evaluation.

25. When Border City was considering whether they should investigate the citizenship status of students attending their public schools, an evaluation of the program suggested that it would be more expensive to investigate the students than to educate the few non-citizens that were attending. This evaluation is a type of:

a. Needs assessment.

b. Efficiency evaluaton.

c. Process evaluation.

d. Outcome evaluaton.

26. Evaluation research has shown that the DARE program does not lessen the use of illicit drugs among DARE students, while they are enrolled in the program or afterwards.

a. True

b. False

27. Evaluation research is conducted to investigate social programs.

a. True

b. False

28. Evaluation research is its own unique method like other research designs.

a. True

b. False

29. Progam process is the set of acitivites that constitute the intervention.

a. True

b. False

30. Outputs describe the impact of the program.

a. True

b. False

31. A logic model is a representation of the components of of a social work program.

a. True

b. False

32. In a logic model, service completions are a type of outcome.

a. True

b. False

33. The program assumptions identify the expected outcomes.

a. True

b. False

34. Evaluation research is its own unique method like other research designs.

a. True

b. False

35. Most evaluation researchers attempt to measure multiple outcomes

a. True

b. False

36. Stakeholders are individuals or groups that evaluate social programs.

a. True

b. False

37. The type of evaluation research that determines if a new social program is needed or an old social program is still needed is called an outcome evaluation.

a. True

b. False

38. Social indicators such as poverty rate or school drop-out rate may be used in a needs assessment.

a. True

b. False

39. Stakeholders are individuals and groups that have some interest in the program

a. True

b. False

40. The investigation of how a social program works is operating a process evaluation.

a. True

b. False

41. An outcome evalation may answer the question, “Is the program serving its target population?”

a. True

b. False

42. Qualitative methods may be used in a process evaluation.

a. True

b. False

43. Flowcharting is a method to answer the question, “Are the mssing activities in the design of the program?”

a. True

b. False

44. Evaluation research that determines whether a program has had an effect is called an outcome evaluation.

a. True

b. False

45. The term formative evaluation may be used instead of process evaluation when the evaluation findings are used to help shape and refine the program

a. True

b. False

46. There are two kinds of efficiency analyses: cost-benefit and cost-effective analyses.

a. True

b. False

47. Cost-benefit analysis reports the actual outcomes in lieu of assigning a cash value to the outcomes.

a. True

b. False

48. A benefit to one group may be a cost to another group.

a. True

b. False

49. The alternative to black boxes in evaluation research is a program theory.

a. True

b. False

50. Outcome evaluations that do not use an experimental design do not provide useful information and may not be ethically permissible.

a. True

b. False

51. Cost-effectiveness analysis is a common alternative to cost-benefit analysis.

a. True

b. False

52. Stakeholder approaches encourage researchers to be responsive to program stakeholders.

a. True

b. False

53. Relevance validity regards the usefulness of evaluation data to individuals or communities.

a. True

b. False

54. Match the parts of a program with their descriptions.

[d] 1. Input

[e] 2. Output

[c] 3. Outcome

[a] 4. Feedback

[b] 5. Stakeholder

a. Information about the service delivery system, results, and operations that is available to the program

b. Individuals and groups who have some basis of concern with the program

c. The impact of the program process on the cases processed

d. Resources, materials, clients and staff that go into a program

e. Services delivered or new products produced by the program process

55. Match the researcher and stakeholder orientations with their descriptions.

[b] 1. Social science approaches
[c] 2. Participatory research

[d] 3. Goal-free evaluation

[a] 4. Utilization-focused evaluation

a. The evaluator forms a task force of program stakeholders who help shape the evaluation project so that they are most likely to use its results.

b. Emphasize the importance of researcher expertise and maintenance of some autonomy in order to develop the most trustworthy, unbiased program evaluation.

c. Program participants are engaged with the researchers as co-researchers and help to design, conduct, and report the research.

d. Researchers do not know the intended outcomes of the program, but assess and compare needs of participants to a wide array of program outcomes, mainly to identify unanticipated outcomes.

56. Match the type of evaluation research with its description.

[d] 1. Needs assessment

[b] 2. Cost effectiveness analysis

[a] 3. Process evaluation

[e] 4. Outcome evaluation

[c] 5. Efficiency analysis

a. Evaluates how services are delivered

b. How much it costs to achieve a given effect

c. Evaluates costs versus benefits and cost-effectiveness of program

d. Evaluates whether a new program is needed or an existing program should continue

e. Evaluates outcomes in terms of what would have happened if there had been no program

57. How does evaluation research differ from other types of social work research?

It is not a method of data collection, like survey research or experiments, nor is it a unique component of research designs, like sampling or measurement. Evaluation research is social work research that is conducted for a distinctive purpose: to investigate social programs (such as substance abuse treatment programs, welfare programs, mental health programs, or employment and training programs). For each project, an evaluation researcher must select a research design and method of data collection that are useful for answering the particular research questions posed and appropriate for the particular program investigated.

58. List the basic components of a logic model.

For Whom, Assumptions, Inputs, Activities, Outputs, Outcomes

59. In what circumstances should qualitative methods be used in evaluation research? In what circumstances should quantitative methods be used? Use social work examples in your answer.

Qualitative methods can add much to quantitative evaluation research studies, including more depth, detail, nuance, and exemplary case studies (Patton, 2002). Perhaps the greatest contribution that the qualitative method can make in many evaluation studies is investigating program process. Another good reason for using qualitative methods in evaluation research is the importance of learning how different individuals react to the treatment. Qualitative methods can also help in understanding how social programs actually operate. Complex social programs have many different features, and it is not always clear whether some particular feature or a combination of features is responsible for the program’s effect—or for the absence of an effect. The more complex the social program, the more value that qualitative methods can add to the evaluation process

60. What is the difference between an evaluation research project that assumes a black box and one that seeks to develop a program theory?

The process by which a program has an effect on outcomes is often treated as a black box, that is, the focus of the evaluation researcher is on whether cases seem to have changed as a result of their exposure to the program between the time they entered the program as inputs and when they exited the program as outputs (Chen, 1990). The assumption is that program evaluation requires only the test of a simple input-output model. If an investigation of program process is conducted, a program theory may be developed. A program theory describes what has been learned about how the program has its effect. When a researcher has sufficient knowledge before the investigation begins, outlining a program theory can help to guide the investigation of program process in the most productive directions. This is termed a theory-driven evaluation.

61. Outline a process evaluation of a program designed to reduce the spread of HIV by providing clean needles, counseling, and drug substitutes to intravenous drug users.

The purpose of a process evaluation is to investigate how the program is operating. Process evaluations are completed to answer a variety of questions related to the operation of a particular program. One question concerns program coverage: Is the program serving its target population? Other questions focus on service delivery: Has the program been implemented as designed? What are the outputs of various program activities? In the past, process evaluations were the primary response to funders who wanted to know whether the agency had actually carried out its planned activities. Increasingly, process evaluations are conducted to identify specific activities that produced program outcomes as well as to determine client satisfaction with the program activities.

62. Describe how quantitative and qualitative research designs can be used in evaluation research. When would it be ideal to use these designs?

Evaluation research that attempts to identify the effects of a social program typically is quantitative: Did housretention improve? Did substance abuse decline? It is fair to say that when there is an interest in comparing outcomes between an experimental and a control group, or tracking change over time in a systematic manner, quantitative methods are favored. But qualitative methods can add much to quantitative evaluation research studies, including more depth, detail, nuance, and exemplary case studies (Patton, 2002). Perhaps the greatest contribution qualitative methods can make in many evaluation studies is investigating program process—finding out what is “inside the black box.” Although it is possible to track service delivery with quantitative measures like staff contact hours and frequency of referrals, finding out what is happening to clients and how clients experience the program can often best be accomplished by observing program activities and interviewing staff and clients inten

Another good reason for using qualitative methods in evaluation research is the importance of learning how different individuals react to the treatment. Qualitative methods can also help in understanding how social programs actually operate. Complex social programs have many different features, and it is not always clear whether some particular feature or a combination of features is responsible for the program’s effect—or for the absence of an effect. The more complex the social program, the more value that qualitative methods can add to the evaluation process.

63. What is the difference between a cost-benefit analysis and a cost-effectiveness analysis?

A cost-benefit analysis must identify the specific costs and benefits that will be studied, which requires in turn that the analyst identifies whose perspective will be used to determine what can be considered a benefit rather than a cost. Cost-effectiveness analysis is a common alternative to cost-benefit analysis; this analysis compares the costs of different programs (or interventions) to the actual program outcomes in lieu of assigning a dollar value to the outcomes. In these comparisons, the program costs are calculated, whereas the benefits are listed and not assigned a cash value.

64. Explain what is meant by relevance validity?

Just meeting scientific standards of validity is insufficeint. The data should fit the realities of the people it represents. It should be useful for individuals and communities.