Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

The Effect of Guessing on Assessing Dimensionality in Multiple-Choice Tests:A Monte Carlo Study with Application

Yeh, Chien-Chi (2007) The Effect of Guessing on Assessing Dimensionality in Multiple-Choice Tests:A Monte Carlo Study with Application. Doctoral Dissertation, University of Pittsburgh. (Unpublished)

[img]
Preview
PDF
Primary Text

Download (1MB) | Preview

Abstract

The purpose of this study was to investigate the effect of guessing in the assessment of dimensionality in multiple-choice tests using procedures implemented in Mplus and TESTFACT. Levels of item discrimination and the size of correlations between dimensions were also manipulated to explore any interaction between these effects. Four indices based on the proportion of variance, parallel analysis, RMSR reduction and a chi-square difference test were used to estimate dimensionality. The research included two parts, a simulation study using a Monte Carlo approach and an application with TIMSS 2003 data.The simulation study confirmed the guessing effect. TESTFACT appeared to outperform Mplus for most conditions with data that assumed guessing. The proportion of variance and the RMSR reduction indices more accurately estimated dimensionality in Mplus, whereas the chi-square test and parallel analysis performed best with TESTFACT. A discrimination effect was observed clearly in data that assumed no guessing using the parallel analysis index and in data that assumed guessing using the RMSR index for both methods. Less accurate estimation of dimensionality was observed when using Mplus for tests with either high or low discriminating items, and with TESTFACT for tests with lower discriminating items. Higher correlations between dimensions led to more serious estimation problems. When guessing was not modeled, greater influence from the levels in correlations between dimensions and item discriminations was found. Further, a more pronounced discrimination effect was observed in the high correlation condition. With regard to the application of TIMSS data, 70% of the items exhibited guessing behaviors and high correlations were observed between scores on the different dimensions (math and science). Based on the simulation study, guessing and correlation effects should thus be considered carefully when choosing a method for assessing dimensionality. Inconsistency in the dimensionality assessment using the four indices with Mplus was observed (1 to 5 dimensions), whereas TESTFACT consistently estimated 2 dimensions. However, further investigation of the internal structure of the TIMSS assessment did not show any connection to content or cognitive domains.


Share

Citation/Export:
Social Networking:
Share |

Details

Item Type: University of Pittsburgh ETD
Status: Unpublished
Creators/Authors:
CreatorsEmailPitt UsernameORCID
Yeh, Chien-Chichy3@pitt.eduCHY3
ETD Committee:
TitleMemberEmail AddressPitt UsernameORCID
Committee ChairStone, Clement A
Committee MemberKim, Kevin H
Committee MemberKirisci, Levent
Committee MemberLane, Suzanne
Date: 27 June 2007
Date Type: Completion
Defense Date: 23 February 2007
Approval Date: 27 June 2007
Submission Date: 18 April 2007
Access Restriction: No restriction; Release the ETD for access worldwide immediately.
Institution: University of Pittsburgh
Schools and Programs: School of Education > Psychology in Education
Degree: PhD - Doctor of Philosophy
Thesis Type: Doctoral Dissertation
Refereed: Yes
Uncontrolled Keywords: dimensionality; guessing effect; Monte Carlo study; multiple-choice
Other ID: http://etd.library.pitt.edu/ETD/available/etd-04182007-060503/, etd-04182007-060503
Date Deposited: 10 Nov 2011 19:38
Last Modified: 15 Nov 2016 13:40
URI: http://d-scholarship.pitt.edu/id/eprint/7286

Metrics

Monthly Views for the past 3 years

Plum Analytics


Actions (login required)

View Item View Item