Chapter 8 Test Bank Docx Experiments For Evaluation Research - Instructor Test Bank | Real Research 2e by Gordon by Liahna Gordon. DOCX document preview.
Chapter 8: Experiments for Evaluation Research
Test Bank
Multiple Choice
1. Program evaluation is research that ______.A. determines the needs of a population for new programs
B. evaluates the effectiveness of existing programs or policies
C. evaluates whether a program needs to conduct research
D. evaluates the best choice for choosing a program for a particular population
Learning Objective: 8.1: Define classical experimental research and evaluation experimental research and discuss the application of each.
Cognitive Domain: Knowledge
Answer Location: Introduction
Difficulty Level: Easy
2. Experiments are used to test ______.A. association
B. significance
C. validity
D. causality
Learning Objective: 8.1: Define classical experimental research and evaluation experimental research and discuss the application of each.
Cognitive Domain: Knowledge
Answer Location: Introduction
Difficulty Level: Easy
3. Classical studies are typically done in laboratories. When we do evaluation research in real-life settings, what can happen?
A. Your results are invalid if all forms of bias were not able to be ruled out.
B. Your study could become nonreplicable, and thus meaningless.
C. You may not be able to conclude that your results were caused by the treatment and not by something else.
D. Your research may be deemed unethical if it affects anyone other than your subjects.
Learning Objective: 8.1: Define classical experimental research and evaluation experimental research and discuss the application of each.
Cognitive Domain: Knowledge
Answer Location: Introduction
Difficulty Level: Easy
4. Classical experiments are done in laboratories because researchers can better control ______.
A. the treatment
B. the effects
C. outside influences
D. validity
Learning Objective: 8.1: Define classical experimental research and evaluation experimental research and discuss the application of each.
Cognitive Domain: Knowledge
Answer Location: Introduction
Difficulty Level: Easy
5. Because it is based on positivist methodology, experimental research must be ______.
A. valid
B. descriptive
C. generalizable
D. replicable
Learning Objective: 8.2: Describe the logic of positivist methodology and how it affects experimental research for evaluation.
Cognitive Domain: Comprehension
Answer Location: Methodology
Difficulty Level: Medium
6. Research questions for experimental evaluation research ______.A. are usually qualitative
B. include at least three variables (independent, dependent, and treatment)
C. cannot imply causation
D. ask about causation
Learning Objective: 8.4: Identify research questions appropriate for experiments for evaluation research.
Cognitive Domain: Knowledge
Answer Location: Research Questions
Difficulty Level: Easy
7. Experimental researchers can usually control what thing that survey researchers can’t?
A. temporal order
B. association
C. objectivity
D. unit of analysis
Learning Objective: 8.5: Describe hypothesis writing in experiments for evaluation research.
Cognitive Domain: Knowledge
Answer Location: Research Questions
Difficulty Level: Easy
8. In experimental evaluation research, the hypotheses usually all have ______.A. the same independent variable
B. the same dependent variable
C. different independent and dependent variables
D. the same variable sometimes used as an independent variable and sometimes as a dependent variable
Learning Objective: 8.5: Describe hypothesis writing in experiments for evaluation research.
Cognitive Domain: Knowledge
Answer Location: Research Questions
Difficulty Level: Easy
9. Which is the most appropriate research question for experimental evaluation research?
A. What effects, if any, do illiteracy rates have on child poverty?
B. What effects does smoking have on teenagers’ popularity?
C. What effects, if any, does age have on voting habits in local elections?
D. What effects, if any, does instituting flexible work schedules have on interpersonal dynamics within work groups?
Learning Objective: 8.4: Identify research questions appropriate for experiments for evaluation research.
Cognitive Domain: Evaluation
Answer Location: Research Questions
Difficulty Level: Hard
10. Which of the following is the best research question for experimental evaluation research?
A. What effects, if any, do gray winter skies have on depression?
B. What effects, if any, does the New Orleans Police Officer Diversity Training program have on police officers’ assessment of the degree of danger they feel when apprehending male suspects of different races and ethnicities?
C. What effects, if any, would stricter penalties for driving under the influence of marijuana have on accident fatalities?
D. What effects, if any, does playing video games for 4 or more hours per day have on children’s brain development?
Learning Objective: 8.4: Identify research questions appropriate for experiments for evaluation research.
Cognitive Domain: Evaluation
Answer Location: Research Questions
Difficulty Level: Hard
11. Which of the following is a POOR example of a hypothesis for experimental evaluation research?
A. Women who take the Assertiveness Training Workshop are more likely to state when they feel that their personal boundaries have been crossed than women who do not take the training.
B. New fathers who work for the State of New York and take the paid parental leave available to them when their child is born are more likely to report feeling productive in their jobs upon their return than those who do not take the paid leave available to them.
C. Those democrats who voted for Hillary Clinton in the 2016 primaries are more likely to vote for a woman in the 2020 primaries than are those who voted for Bernie Sanders.
D. Independent business owners who participate in the Work for Dignity program, in which they hire a homeless person to do 5 hours of work at their establishment per week paid for by a grant from the County, are more likely to volunteer their services or donate their products to the local homeless shelters than those who do not participate in the program.
Learning Objective: 8.4: Identify research questions appropriate for experiments for evaluation research.
Cognitive Domain: Evaluation
Answer Location: Research Questions
Difficulty Level: Hard
12. Which of the following is considered LEAST damaging to the validity of experimental design?
A. not using a control group
B. not using a pretest
C. not ruling out bias when assigning people to treatment and control groups
D. not using a posttest
Learning Objective: 8.6: Describe the variety of study designs for experiment study designs for evaluation research and how the different variations may affect validity.
Cognitive Domain: Evaluation
Answer Location: Choosing a Study Design
Difficulty Level: Medium
13. Which of the following would be an acceptable reason for not using probability sampling in assigning groups in experimental evaluation research?
A. The groups don’t seem to be biased anyway, so it’s not needed.
B. The study is easier to manage with fewer groups.
C. The groups are naturally occurring.
D. The number of subjects is too small.
Learning Objective: 8.7: Recognize how goals and context of the study should affect decisions about study design.
Cognitive Domain: Understanding
Answer Location: Naturally Occurring Groups
Difficulty Level: Easy
14. What is the problem with preexperimental designs?
A. They are very difficult to replicate.
B. You can’t rule out plausible alternatives.
C. They are unreliable.
D. They lack objectivity.
Learning Objective: 8.6: Describe the variety of study designs for experiment study designs for evaluation research and how the different variations may affect validity.
Cognitive Domain: Understanding
Answer Location: No Control Group
Difficulty Level: Medium
15. You are conducting research on the effects of a grief support group on adults’ coping with the death of a loved one. You decide that you will measure ability to cope by observing behavior in the group, by asking the research subjects to keep a log rating their emotions at the end of each day, and by asking them survey items about coping strategies, activities engaged in, and other measures of coping. What component of research is this?
A. operationalizing
B. conceptualizing
C. preexperimental design
D. sampling
Learning Objective: 8.8: Describe the process of operationalizing dependent variables for experimental evaluation research and identify the methods of data collection that may be used.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Medium
16. You are evaluating the efficacy of a disaster preparation program in campus dorms focused on emergency evacuation. You decide to provide surveys before and after residents complete the program to gauge self-reported levels of disaster preparedness. What component of research is this?
A. conceptualizing
B. operationalizing
C. preexperimental design
D. sampling
Learning Objective: 8.8: Describe the process of operationalizing dependent variables for experimental evaluation research and identify the methods of data collection that may be used.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Medium
17. In drawing a sample for experiments for evaluation, which sampling method is most often used?
A. cluster
B. simple random
C. systematic
D. convenience
Learning Objective: 8.9: Define the types of sampling appropriate for each stage of sampling in experimental research for evaluation.
Cognitive Domain: Knowledge
Answer Location: Sampling
Difficulty Level: Easy
18. You are conducting experimental research in order to evaluate the effectiveness of a new sexual harassment training program at a small firm. Which would be the most appropriate way to choose your subjects?
A. cluster sample
B. census
C. systematic sampling
D. convenience sample
Learning Objective: 8.9: Define the types of sampling appropriate for each stage of sampling in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
19. You have two groups in your experimental evaluation research: treatment and control. To each group you assign five males and five females of each of the following ethnicities: White, Black, Asian and Latino. This is called ______.A. random assignment
B. matching
C. stratifying
D. systematic sampling
Learning Objective: 8.9: Define the types of sampling appropriate for each stage of sampling in experimental research for evaluation.
Cognitive Domain: Comprehension
Answer Location: Sampling
Difficulty Level: Medium
20. You have two groups in your experimental evaluation research: treatment and control. Each subject is given a number. You then use a random numbers generator to assign each person to a group. This is called ______.A. random assignment
B. matching
C. stratifying
D. systematic sampling
Learning Objective: 8.9: Define the types of sampling appropriate for each stage of sampling in experimental research for evaluation.
Cognitive Domain: Comprehension
Answer Location: Sampling
Difficulty Level: Medium
21. In addition to matching, what is another method for minimizing bias when assigning subjects to experimental groups?
A. convenience sampling
B. theoretical sampling
C. probability sampling
D. cluster sampling
Learning Objective: 8.9: Define the types of sampling appropriate for each stage of sampling in experimental research for evaluation.
Cognitive Domain: Knowledge
Answer Location: Sampling
Difficulty Level: Easy
22. If you use a convenience sample for participation in the experiment, and then probability sampling for group assignment, can you generalize your results to the larger population?
A. Yes, you would be able to generalize your results.
B. No, you would not be able to generalize your results.
C. only if you used simple random sampling for the group assignment
D. only if you conducted stratified sampling for the group assignment
Learning Objective: 8.10: Explain how assigning groups using matching and random sampling affect the ability to generalize to a larger population.
Cognitive Domain: Comprehension
Answer Location: Sampling
Difficulty Level: Medium
23. Under what circumstances do you need to debrief your research subjects?
A. when you use deception in the experiment
B. when they have been part of the treatment group
C. you always need to debrief subjects after an experiment
D. whenever you must provide full informed consent
Learning Objective: 8.11: Discuss ethical considerations as they pertain to experimental research for evaluation.
Cognitive Domain: Knowledge
Answer Location: Ethics
Difficulty Level: Easy
24. The program you are evaluating is time-consuming, and so people keep dropping out of your treatment group because they are overwhelmed by the time commitment. This is an example of ______.A. debriefing
B. contamination
C. placebo effect
D. attrition
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
25. Even if you randomly assign subjects to groups, which of the following may result in you ending up with groups that are not comparable?
A. attrition
B. contamination
C. placebo effect
D. maturation
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
26. You are evaluating the efficacy of a new safety training course at a shipping company. To ensure accurate results, it is vital that the control group does not have access to the information in the course; however, subjects from your control group and your treatment group frequently interact with one another and you are concerned that they may share this information. Which of the following are you most concerned with?
A. debriefing
B. contamination
C. placebo effect
D. attrition
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
27. Internal validity means ______.A. that maturation has occurred
B. that the results can be generalized to your population
C. that the effect you observe is actually caused by the treatment
D. that your hypotheses are supported
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Knowledge
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Easy
28. If you use a control group, to which of the following threats to internal validity are you still most vulnerable?
A. your own expectations affecting the results
B. subjects changing over time
C. outside conditions changing over time
D. attrition rates
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Comprehension
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Medium
Short Answer/Essay
1. Why would you choose to do experimental evaluation research? What is it good at understanding?
Learning Objective: 8.1: Define classical experimental research and evaluation experimental research and discuss the application of each.
Cognitive Domain: Understanding
Answer Location: Introduction
Difficulty Level: Easy
2. Why can causality be tested in experimental research better than in survey research?
Learning Objective: 8.1: Define classical experimental research and evaluation experimental research and discuss the application of each.
Cognitive Domain: Evaluation
Answer Location: Introduction
Difficulty Level: Medium
3. What is quality control? List three ways that you could ensure consistent quality in your data collection.
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Knowledge
Answer Location: Data Collection
Difficulty Level: Easy
4. Write a research question for experimental evaluation research.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
5. Write four hypotheses for your research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
6. What is an executive summary, and why would you use it?
Learning Objective: 8.17: Outline an appropriate executive summary for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Presenting the Results
Difficulty Level: Medium
7. You are conducting research on the effects of a grief support group on the coping abilities of adult children who have had a parent die. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: 8.9 Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for an experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.13: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
8. You are conducting evaluation research for the county hospital by studying the effects of the elimination of double shifts and overtime on patient care. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: 8.9 Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for an experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.13: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
9. You are doing research evaluating the effectiveness of an anti-harassment campaign in your company. You will ask subjects to fill out a survey about their attitudes about and experiences with workplace harassment both before and after the campaign. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: 8.9 Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for an experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.15: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
10. You are conducting evaluation research, studying the effects of a city ordinance that recently went into effect that prohibits people from sitting or sleeping on sidewalks in the downtown area. The ordinance is intended to reduce the number of homeless people in the city. You want to see how it has affected the daily routines of the local homeless people. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: 8.9 Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.13: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
11. You are doing research on the effects of anger management classes. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: 8.9 Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.13: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
12. You are conducting experimental evaluation research on people’s sense of safety and danger before and after a gun control law goes into effect. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: 8.9 Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.15: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
13. You are conducting experimental evaluation research on the effect of budget cuts on quality of teaching. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: Identify the independent and dependent variables for hypotheses in experiments for evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.13: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
14. You are conducting experimental evaluation research on a Dealing with Divorce support group. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: Identify the independent and dependent variables for hypotheses in experiments for evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.13: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
15. You are conducting experimental evaluation research on bullying in secondary schools before and after an anti-bullying campaign. You are interested in the campaign’s effectiveness. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: Identify the independent and dependent variables for hypotheses in experiments for evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.15: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
16. You are doing research on how prekindergarten enrollment effects students’ emergent independence. You are particularly interested in how the early enrollment affects their ability to solve problems for themselves and to self-soothe. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: 8.9 Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.13: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
17. You are conducting experimental evaluation research on employee wellness at a large company after the opening of recreation centers at several of their Midwestern locations. Write a research question for this research project.
- Be explanatory and imply causality
- Focus on the effects of a policy or program
- Not assume effects occur
- Include one independent and one dependent variable
- Use the policy or program as the independent variable
- Not be answerable with a yes/no or just a few words
- Not include explanations
- Not include the words “could,” “can,” or “should”
Learning Objective: 8.18: Write research questions appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Write four hypotheses for this research question.
- Have exactly one independent variable and one dependent variable
- Use the same policy or program you are evaluating as the independent variable
- Use as the dependent variables various, more precise ways of measuring the effect you think the program or policy may have
- Compare categories of the independent variable
- Not include explanations
- Be clear and precise
- Use the individual (not the policy or program) as the unit of analysis
Learning Objective: 8.19: Write hypotheses appropriate for experimental evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Identify the independent and dependent variables in your hypotheses.
Learning Objective: Identify the independent and dependent variables for hypotheses in experiments for evaluation research.
Cognitive Domain: Application
Answer Location: Research Questions
Difficulty Level: Hard
Which type of study design will you use for this research? How will it differ from a classical experiment and why? How will this affect the validity of your results?
- No control group: cannot rule out plausible alternatives in determining causality, and can’t know the true size of effects
- Naturally occurring groups: bias in the composition of groups, the effect may be exaggerated or minimized, and can’t know the extent of the effect of this bias on the results
- No pretest: least damaging to validity
- The more your design deviates from the classical experiment (combining these), the less valid the results
Learning Objective: 8.20: Choose a study design for experimental evaluation research question and explain how validity may be affected.
Cognitive Domain: Application
Answer Location: Choosing a Study Design
Difficulty Level: Hard
How will you draw a sample of participants for your research? Why?
Learning Objective: 8.23: Choose the best sampling method to be used for a specific scenario of evaluation research.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
How will you assign subjects to research groups? Why? What are the pros and cons of this approach?
Learning Objective: 8.24: Determine the best method to assign subjects to experimental groups.
Cognitive Domain: Application
Answer Location: Sampling
Difficulty Level: Hard
Conceptualize one dependent variable from your hypotheses. How will you measure this variable, and how will you collect the data that will measure it?
Learning Objective: 8.21: Conceptualize and operationalize dependent variables for experimental evaluation research. | 8.22: Determine the best method of data collection for each dependent variable.
Cognitive Domain: Application
Answer Location: Conceptualizing and Operationalizing
Difficulty Level: Hard
What steps will you need to take to protect your participants?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
Will informed consent be needed for your study? Why or why not?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Application
Answer Location: Ethics
Difficulty Level: Medium
What other ethical issues might be likely to arise in conducting this research?
Learning Objective: 8.25: Determine the steps they would need to take to protect research subjects in a specific evaluation scenario and identify additional possible ethical concerns in experimental research for evaluation.
Cognitive Domain: Understanding
Answer Location: Ethics
Difficulty Level: Medium
What steps would you take to prepare for data collection?
Learning Objective: 8.13: Identify the steps in preparing for collection in experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Preparing for Data Collection
Difficulty Level: Medium
How will you ensure quality control in your data collection procedures?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you try to diminish the attrition rate? How will it affect your results if the rate is high?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
What is contamination, and how would you try to minimize it?
Learning Objective: 8.14: Discuss the different issues that may arise during data collection and ways in which to ensure quality control for evaluation research.
Cognitive Domain: Application
Answer Location: Data Collection
Difficulty Level: Medium
How would you clean your data?
Learning Objective: 8.13: Describe the data analysis process for experiments for evaluation research.
Cognitive Domain: Understanding
Answer Location: Data Analysis
Difficulty Level: Medium
What are the threats to internal validity, and how would you try to minimize them?
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Application
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard
Evaluate the external validity of your study.
Learning Objective: 8.16: Describe how data validity and analysis validity are evaluated in experimental evaluation research.
Cognitive Domain: Evaluation
Answer Location: Evaluating the Quality of Data and Analysis
Difficulty Level: Hard