Chapter 7: Validation and Use of Individual-Differences Measures

Test Bank

Multiple Choice

1. Reliability is a ______ but not ______ condition for high validity.

A. required; absolute

B. absolute; required

C. sufficient; necessary

D. necessary; sufficient

Learning Objective: 7-1: Calculate how and why reliability places a ceiling on validity (i.e., accuracy of inferences made based on test scores).

Cognitive Domain: Comprehension

Answer Location: Relationship between Reliability and Validity
Difficulty Level: Medium

AACSB: Application of knowledge

2. Assuming a criterion reliability of .49, and a validity of .60, correct the validity coefficient for attenuation to see what the validity would have been for a perfectly reliable measure?

A. .60

B. .72

C. .86

D. .91

Learning Objective: 7-1: Calculate how and why reliability places a ceiling on validity (i.e., accuracy of inferences made based on test scores).

Cognitive Domain: Analysis

Answer Location: Relationship between Reliability and Validity
Difficulty Level: Medium

AACSB: Application of knowledge

3. Assuming a criterion reliability of .49, and a validity of .60. How much variance is explained by a measure with the original validity coefficient, and one with the corrected one?

A. 60%; 86%

B. 36%; 74%

C. 36%; 86%

D. 25%; 50%

Learning Objective: 7-1: Calculate how and why reliability places a ceiling on validity (i.e., accuracy of inferences made based on test scores).

Cognitive Domain: Analysis

Answer Location: Relationship between Reliability and Validity
Difficulty Level: Hard

AACSB: Application of knowledge

4. Gathering the necessary data to determine if a measurement procedure actually measures what it is designed to is called ______.

A. equivalence analysis

B. cross-validation

C. screening

D. validation

Learning Objective: 7-2: Compute empirical estimates of different types of validity evidence: Content, criterion, and construct.

Cognitive Domain: Knowledge

Answer Location: Relationship between Reliability and Validity
Difficulty Level: Easy

AACSB: Application of knowledge

5. Which of the following is NOT one of the assumptions that underlie the use of content-related evidence?

A. The measure can be shown to accurately predict job performance behavior.

B. The sample and the sampling process can be defined with sufficient precision to enable the user to judge how adequately the sample of performance typifies performance on the universe.

C. A sample can be drawn from the universe in some purposeful, meaningful fashion.

D. The area of concern to the user can be conceived as a meaningful, definable universe of responses.

Learning Objective: 7-2: Compute empirical estimates of different types of validity evidence: Content, criterion, and construct.

Cognitive Domain: Comprehension

Answer Location: Content-Related Evidence
Difficulty Level: Hard

AACSB: Application of knowledge

6. If you are asking SME’s to generate items and/or rate them on how they measure a certain domain, you are MOST likely looking for evidence of which type of validity?

A. content

B. criterion

C. construct

D. face

Learning Objective: 7-2: Compute empirical estimates of different types of validity evidence: Content, criterion, and construct.

Cognitive Domain: Application

Answer Location: Content-Related Evidence
Difficulty Level: Medium

AACSB: Application of knowledge

7. Which content validity approach provides information on the extent to which panel members assign an item to its posited construct more than the other constructs?

A. content-validity index

B. content-adequacy procedure

C. substantive-validity index

D. analysis-of-variance approach

Learning Objective: 7-2: Compute empirical estimates of different types of validity evidence: Content, criterion, and construct.

Cognitive Domain: Knowledge

Answer Location: Content-Related Evidence
Difficulty Level: Medium

AACSB: Application of knowledge

8. The last step of the content validity strategy proposed by Buster et al. is to ______.

A. group potential items in a thoughtful manner

B. ask SMEs to rate the list of potential items independently

C. link each of the potential items to the KSAO’s from the job analysis

D. conduct a job analysis

Learning Objective: 7-2: Compute empirical estimates of different types of validity evidence: Content, criterion, and construct.

Cognitive Domain: Comprehension

Answer Location: Criterion-Related Evidence
Difficulty Level: Medium

AACSB: Application of knowledge

9. Which type of validity is concerned with using individual differences to predict behavior, often job performance in Personnel Psychology?

A. content

B. criterion

C. construct

D. face

Learning Objective: 7-2: Compute empirical estimates of different types of validity evidence: Content, criterion, and construct.

Cognitive Domain: Knowledge

Answer Location: Content-Related Evidence
Difficulty Level: Easy

AACSB: Application of knowledge

10. You conduct a criterion-related validity study in which you test the applicants with your new measure. Then you hire applicants without using the new measure, measure performance after 6 months, and assess the relationship between the predictor–criterion. Which kind of study is this?

A. predictive

B. concurrent

C. observational

D. validity generalization

Learning Objective: 7-3: Conduct predictive and concurrent validation studies.

Cognitive Domain: Application

Answer Location: Predictive Studies
Difficulty Level: Easy

AACSB: Application of knowledge

11. Larger effect sizes result in needing ______ sample sizes in order to get adequate statistical power to detect the effect.

A. perfect

B. smaller

C. larger

D. equal

Learning Objective: 7-3: Conduct predictive and concurrent validation studies.

Cognitive Domain: Comprehension

Answer Location: Predictive Studies
Difficulty Level: Easy

AACSB: Application of knowledge

12. Concurrent validity studies ignore the role of ______ because job incumbents are not taking the test for the purpose of trying to get hired.

A. job experience

B. managerial politics

C. motivation

D. statistical power

Learning Objective: 7-3: Conduct predictive and concurrent validation studies.

Cognitive Domain: Comprehension

Answer Location: Concurrent Studies
Difficulty Level: Medium

AACSB: Application of knowledge

13. Which type of measure is chosen because it represents performance or behavior on the job that is valued by the employing organization?

A. construct-related criterion

B. content-related criterion

C. worker-related construct

D. job-related construct

Learning Objective: 7-3: Conduct predictive and concurrent validation studies.

Cognitive Domain: Knowledge

Answer Location: Requirements of Criterion Measures in Predictive and Concurrent Studies

Difficulty Level: Easy

AACSB: Application of knowledge

14. When a test is given to a more heterogeneous group than originally intended/validated, it can create an issue called ______.

A. homoscedasticity

B. indirect range restriction

C. direct range restriction

D. range enhancement

Learning Objective: 7-4: Minimize the detrimental effects of factors such as range restriction and form of the predictor–criterion relationship (e.g., linear vs. curvilinear) on conclusions about validity.

Cognitive Domain: Knowledge

Answer Location: Range Enhancement
Difficulty Level: Medium

AACSB: Application of knowledge

15. You are running a predictive validity study, and so you give applicants your new measure as well as the ones you previously used for selection. Because of the nature of the study, you only use the scores from the previous measure to select, but this leads to most of the low scores on the new measure not being present in the sample when the criterion in measured. This is an example of ______.

A. homoscedasticity

B. indirect range restriction

C. direct range restriction

D. range enhancement

Learning Objective: 7-4: Minimize the detrimental effects of factors such as range restriction and form of the predictor–criterion relationship (e.g., linear vs. curvilinear) on conclusions about validity.

Cognitive Domain: Application

Answer Location: Range Restriction
Difficulty Level: Hard

AACSB: Application of knowledge

16. The estimation of validity may differ at different times in a multiple hurdle selection process because ______.

A. employees are more likely to fake in later stages

B. employees are more likely to misinterpret test items in later stages

C. variance is often more enhanced later in the process

D. variance is often more restricted later in the process

Learning Objective: 7-4: Minimize the detrimental effects of factors such as range restriction and form of the predictor–criterion relationship (e.g., linear vs. curvilinear) on conclusions about validity.

Cognitive Domain: Comprehension

Answer Location: Position in the Employment Process
Difficulty Level: Medium

AACSB: Application of knowledge

17. Whether or not the data points are distributed throughout the regression line and the measure predicts as well at both high and low ranges is called ______.

A. homoscedasticity

B. kurtosis

C. direct range restriction

D. range enhancement

Learning Objective: 7-4: Minimize the detrimental effects of factors such as range restriction and form of the predictor–criterion relationship (e.g., linear vs. curvilinear) on conclusions about validity.

Cognitive Domain: Knowledge

Answer Location: Form of the Predictor–Criterion Relationship
Difficulty Level: Easy

AACSB: Application of knowledge

18. Retest scores are ______ predictors of performance than initial test scores.

A. perfect

B. better

C. worse

D. equal

Learning Objective: 7-4: Minimize the detrimental effects of factors such as range restriction and form of the predictor–criterion relationship (e.g., linear vs. curvilinear) on conclusions about validity.

Cognitive Domain: Comprehension

Answer Location: Retesting
Difficulty Level: Easy

AACSB: Application of knowledge

19. If you are using a multitrait–multimethod matrix to determine validity, you are MOST likely testing for which type of validity?

A. content

B. criterion

C. construct

D. face

Learning Objective: 7-5: Calculate convergent and discriminant validity estimates based on a multitrait–multimethod matrix.

Cognitive Domain: Application

Answer Location: Construct-Related Evidence

Difficulty Level: Medium

AACSB: Application of knowledge

20. Which is a system of interrelated concepts, propositions, and laws that relates observable characteristics to other observables, observables to theoretical constructs, and theoretical constructs to other constructs?

A. hypothesis

B. empirical cross-validation

C. nomological network

D. generalizability

Learning Objective: 7-5: Calculate convergent and discriminant validity estimates based on a multitrait–multimethod matrix.

Cognitive Domain: Knowledge

Answer Location: Construct-Related Evidence

Difficulty Level: Easy

AACSB: Application of knowledge

21. As a researcher, you run a correlation to establish how strong or weak the relationship between your personality measure of extraversion and self-esteem is. You are MOST likely trying to establish which kind of validity?

A. content

B. criterion

C. discriminant

D. convergent

Learning Objective: 7-5: Calculate convergent and discriminant validity estimates based on a multitrait–multimethod matrix.

Cognitive Domain: Application

Answer Location: Construct-Related Evidence

Difficulty Level: Medium

AACSB: Application of knowledge

22. In a multitrait–multimethod matrix, reliability is estimated by ______.

A. two measures of the same trait using the same method.

B. two measures of the same trait using different methods.

C. two measures of different traits using the same methods.

D. two measures of different traits using different methods.

Learning Objective: 7-5: Calculate convergent and discriminant validity estimates based on a multitrait–multimethod matrix.

Cognitive Domain: Knowledge

Answer Location: Construct-Related Evidence

Difficulty Level: Medium

AACSB: Application of knowledge

23. In a multitrait–multimethod matrix, validity is estimated by ______.

A. two measures of the same trait using the same method.

B. two measures of the same trait using different methods.

C. two measures of different traits using the same methods.

D. two measures of different traits using different methods.

Learning Objective: 7-5: Calculate convergent and discriminant validity estimates based on a multitrait–multimethod matrix.

Cognitive Domain: Knowledge

Answer Location: Construct-Related Evidence

Difficulty Level: Medium

AACSB: Application of knowledge

24. When weights computed in one sample (i.e., current employees) are used with a second sample from the same population (i.e., job applicants), the multiple correlation coefficient is likely to be smaller. This is called ______.

A. homoscedasticity

B. range enhancement

C. range restriction

D. shrinkage

Learning Objective: 7-6: Obtain cross-validation estimates.

Cognitive Domain: Comprehension

Answer Location: Cross-validation

Difficulty Level: Easy

AACSB: Application of knowledge

25. You have tested how your new measure of cognitive ability predicts performance for your accountants. Now you decide to run a new study to determine if it will also predict performance in your engineers. This process is called ______.

A. validity generalization

B. cross-validation

C. nomological network

D. factor analysis

Learning Objective: 7-6: Obtain cross-validation estimates.

Cognitive Domain: Application

Answer Location: Cross-validation

Difficulty Level: Medium

AACSB: Application of knowledge

26. What is the process of inferring validity in a specific situation from a systematic analysis of jobs into their elements, a determination of test validity for these elements, and a combination or synthesis of the elemental validities into a whole?

A. cross validation

B. validity generalization

C. test transportability

D. synthetic validity

Learning Objective: 7-7: Conduct a validation study when local validation is not feasible by implementing synthetic validation.

Cognitive Domain: Knowledge

Answer Location: Synthetic Validity

Difficulty Level: Medium

AACSB: Application of knowledge

27. All of the following can be evidence of test transportability according to the reading EXCEPT ______.

A. the degree of similarity between the job performed by incumbents locally and that performed at the location where the test has been used previously

B. the results of a criterion-related validity study conducted at another location

C. the degree of similarity between the managers doing the hiring at prior and local settings

D. the degree of similarity between the applicants in the prior and local settings

Learning Objective: 7-7: Conduct a validation study when local validation is not feasible by implementing synthetic validation.

Cognitive Domain: Comprehension

Answer Location: Test Transportability

Difficulty Level: Medium

AACSB: Application of knowledge

28. The first step of a validation generalization study is to ______.

A. correct the mean and variance for known statistical artifacts other than sampling error

B. calculate or obtain the validity coefficient for each study included in the review, and compute the mean coefficient across the studies

C. calculate the variance of the validity coefficient across studies

D. compare the corrected standard deviation to the mean to assess the amount of potential variation in results across studies

Learning Objective: 7-8: Critically evaluate the use of meta-analysis and validity generalization in the validation process.

Cognitive Domain: Comprehension

Answer Location: How to Conduct a VG Study

Difficulty Level: Medium

AACSB: Application of knowledge

29. Studies conducted with the goal of testing the situational specificity hypothesis are called ______.

A. psychometric meta-analyses

B. cross-validation studies

C. criterion meta-analyses

D. concurrent validation studies

Learning Objective: 7-8: Critically evaluate the use of meta-analysis and validity generalization in the validation process.

Cognitive Domain: Knowledge

Answer Location: Validity Generalization

Difficulty Level: Easy

AACSB: Application of knowledge

30. According to the reading, the statistical power to detect ______ in validity generalization studies is quite low.

A. moderators

B. mediators

C. main effects

D. outliers

Learning Objective: 7-8: Critically evaluate the use of meta-analysis and validity generalization in the validation process.

Cognitive Domain: Comprehension

Answer Location: Validity Generalization

Difficulty Level: Hard

AACSB: Application of knowledge

31. In meta-analyses (and thus validity generalization studies), there are often lots of studies that have been conducted but didn’t get published because they did not find significant results. This is known as ______.

A. the criterion problem

B. issues with homoscedasticity

C. issues with shrinkage

D. the file drawer problem

Learning Objective: 7-8: Critically evaluate the use of meta-analysis and validity generalization in the validation process.

Cognitive Domain: Comprehension

Answer Location: Validity Generalization

Difficulty Level: Medium

AACSB: Application of knowledge

32. Which approach involves first calculating the average inaccuracy of meta-analysis and a local validity study under a wide variety of conditions and then computing an empirical Bayesian estimate, which is a weighted average of the meta-analytically derived and local study estimates?

A. cross-validation study

B. psychometric meta-analyses

C. empirical Bayes analysis

D. test transportability

Learning Objective: 7-8: Critically evaluate the use of meta-analysis and validity generalization in the validation process.

Cognitive Domain: Knowledge

Answer Location: Validity Generalization

Difficulty Level: Medium

AACSB: Application of knowledge

33. The probability of correctly rejecting a null hypothesis when it is, in fact, false is called ______.

A. predictive validity

B. alpha level

C. reliability

D. statistical power

Learning Objective: 7-3: Conduct predictive and concurrent validation studies.

Cognitive Domain: Knowledge

Answer Location: Predictive Studies
Difficulty Level: Easy

AACSB: Application of knowledge

Short Answer

1. Briefly describe the relationship between reliability and validity.

Learning Objective: 7-1: Calculate how and why reliability places a ceiling on validity (i.e., accuracy of inferences made based on test scores).

Cognitive Domain: Comprehension

Answer Location: Relationship between Reliability and Validity
Difficulty Level: Medium

AACSB: Application of knowledge

2. Briefly describe the Analysis-of-Variance approach to content validity evidence.

Learning Objective: 7-2: Compute empirical estimates of different types of validity evidence: Content, criterion, and construct.

Cognitive Domain: Comprehension

Answer Location: Content-Related Evidence
Difficulty Level: Medium

AACSB: Application of knowledge

3. Explain what the concept of in situ performance means and why it is important in criterion validation.

Learning Objective: 7-3: Conduct predictive and concurrent validation studies.

Cognitive Domain: Knowledge

Answer Location: Requirements of Criterion Measures in Predictive and Concurrent Studies

Difficulty Level: Easy

AACSB: Application of knowledge

4. Describe one potential correction for range restriction.

Learning Objective: 7-4: Minimize the detrimental effects of factors such as range restriction and form of the predictor–criterion relationship (e.g., linear vs. curvilinear) on conclusions about validity.

Cognitive Domain: Analysis

Answer Location: Range Restriction
Difficulty Level: Hard

AACSB: Application of knowledge

5. Define construct-related validity.

Learning Objective: 7-5: Calculate convergent and discriminant validity estimates based on a multitrait–multimethod matrix.

Cognitive Domain: Knowledge

Answer Location: Construct-Related Evidence

Difficulty Level: Easy

AACSB: Application of knowledge

6. Briefly describe at least one refinement to VG techniques describe in the reading.

Learning Objective: 7-8: Critically evaluate the use of meta-analysis and validity generalization in the validation process.

Cognitive Domain: Comprehension

Answer Location: How to Conduct a VG Study

Difficulty Level: Medium

AACSB: Application of knowledge

Essay

1. Explain how validity evidence is gathered and the two issues that validity is trying to define/explain.

Learning Objective: 7-2: Compute empirical estimates of different types of validity evidence: Content, criterion, and construct.

Cognitive Domain: Comprehension

Answer Location: Relationship between Reliability and Validity
Difficulty Level: Medium

AACSB: Application of knowledge

2. What is statistical power? What parameters go into it, and how and why is it assessed before a study is conducted?

Learning Objective: 7-3: Conduct predictive and concurrent validation studies.

Cognitive Domain: Comprehension

Answer Location: Predictive Studies
Difficulty Level: Medium

AACSB: Application of knowledge

3. Describe at least three ways that construct validity evidence can be gathered?

Learning Objective: 7-5: Calculate convergent and discriminant validity estimates based on a multitrait–multimethod matrix.

Cognitive Domain: Comprehension

Answer Location: Construct-Related Evidence

Difficulty Level: Medium

AACSB: Application of knowledge