Pitt Logo LinkContact Us

Validity Evidence of a Multiple-Choice Test and a Performance Test in an Employment Setting

Lofgren, Robert Eric (2005) Validity Evidence of a Multiple-Choice Test and a Performance Test in an Employment Setting. Doctoral Dissertation, University of Pittsburgh.

[img]
Preview
PDF - Primary Text
Download (3380Kb) | Preview

    Abstract

    A selection procedure consisting of both multiple-choice (MC) paper-and pencil and hands on performance assessment (PA) elements was developed for a large consumer products manufacturing company in the southeastern United States for the purpose of ensuring that workers possessed the necessary knowledge, skills, and abilities required for work at a new facility. Two 125-item alternate tests using an MC format and 7 PA exercises were initially developed for the job referred to in the present study as L2/L3 Production Technician. The purpose of this study was to examine the validity evidence for the two alternate multiple-choice (MC) job knowledge tests and seven performance assessment (PA) exercises that were developed for employment selection purposes. The study sample included 432 Form A and 324 Form B examinees who took both the MC test and the PA exercises. Factor analysis results revealed that the same construct, labeled as applied mechanical knowledge, was measured by both the MC tests and the PA exercises. Item and test analysis results supported the use of Form A and Form B as alternate test forms. The decision consistency between the MC tests and the PA exercises did not appear to be sufficient to recommend that either form of the MC test alone could be used to select qualified L2/L3 Production Technicians. The correlations between MC score and PA total score were .627 for Form A and .612 for Form B. As part of a content analysis, subject-matter experts rated a large number of MC items as either having "no relationship" or "small relationship" to the PA exercises. However, subject-matter experts did rate the PA exercises as having a great importance to the job of L2/L3 Production Technician.


    Share

    Citation/Export:
    Social Networking:

    Details

    Item Type: University of Pittsburgh ETD
    ETD Committee:
    ETD Committee TypeCommittee MemberEmailORCID
    Committee ChairLane, Suzanne
    Committee MemberStone, Clement A
    Committee MemberIrrgang, James J
    Committee MemberPingel, Louis A
    Title: Validity Evidence of a Multiple-Choice Test and a Performance Test in an Employment Setting
    Status: Unpublished
    Abstract: A selection procedure consisting of both multiple-choice (MC) paper-and pencil and hands on performance assessment (PA) elements was developed for a large consumer products manufacturing company in the southeastern United States for the purpose of ensuring that workers possessed the necessary knowledge, skills, and abilities required for work at a new facility. Two 125-item alternate tests using an MC format and 7 PA exercises were initially developed for the job referred to in the present study as L2/L3 Production Technician. The purpose of this study was to examine the validity evidence for the two alternate multiple-choice (MC) job knowledge tests and seven performance assessment (PA) exercises that were developed for employment selection purposes. The study sample included 432 Form A and 324 Form B examinees who took both the MC test and the PA exercises. Factor analysis results revealed that the same construct, labeled as applied mechanical knowledge, was measured by both the MC tests and the PA exercises. Item and test analysis results supported the use of Form A and Form B as alternate test forms. The decision consistency between the MC tests and the PA exercises did not appear to be sufficient to recommend that either form of the MC test alone could be used to select qualified L2/L3 Production Technicians. The correlations between MC score and PA total score were .627 for Form A and .612 for Form B. As part of a content analysis, subject-matter experts rated a large number of MC items as either having "no relationship" or "small relationship" to the PA exercises. However, subject-matter experts did rate the PA exercises as having a great importance to the job of L2/L3 Production Technician.
    Date: 12 December 2005
    Date Type: Completion
    Defense Date: 10 November 2005
    Approval Date: 12 December 2005
    Submission Date: 01 December 2005
    Access Restriction: No restriction; The work is available for access worldwide immediately.
    Patent pending: No
    Institution: University of Pittsburgh
    Thesis Type: Doctoral Dissertation
    Refereed: Yes
    Degree: PhD - Doctor of Philosophy
    URN: etd-12012005-124415
    Uncontrolled Keywords: content analysis; validity evidence; skills testing; multiple-choice test; employee selection; performance assessment
    Schools and Programs: School of Education > Psychology in Education
    Date Deposited: 10 Nov 2011 15:07
    Last Modified: 15 May 2012 14:44
    Other ID: http://etd.library.pitt.edu/ETD/available/etd-12012005-124415/, etd-12012005-124415

    Actions (login required)

    View Item

    Document Downloads