Yeh, Chien-Chi
(2007)
The Effect of Guessing on Assessing Dimensionality in Multiple-Choice Tests:A Monte Carlo Study with Application.
Doctoral Dissertation, University of Pittsburgh.
(Unpublished)
Abstract
The purpose of this study was to investigate the effect of guessing in the assessment of dimensionality in multiple-choice tests using procedures implemented in Mplus and TESTFACT. Levels of item discrimination and the size of correlations between dimensions were also manipulated to explore any interaction between these effects. Four indices based on the proportion of variance, parallel analysis, RMSR reduction and a chi-square difference test were used to estimate dimensionality. The research included two parts, a simulation study using a Monte Carlo approach and an application with TIMSS 2003 data.The simulation study confirmed the guessing effect. TESTFACT appeared to outperform Mplus for most conditions with data that assumed guessing. The proportion of variance and the RMSR reduction indices more accurately estimated dimensionality in Mplus, whereas the chi-square test and parallel analysis performed best with TESTFACT. A discrimination effect was observed clearly in data that assumed no guessing using the parallel analysis index and in data that assumed guessing using the RMSR index for both methods. Less accurate estimation of dimensionality was observed when using Mplus for tests with either high or low discriminating items, and with TESTFACT for tests with lower discriminating items. Higher correlations between dimensions led to more serious estimation problems. When guessing was not modeled, greater influence from the levels in correlations between dimensions and item discriminations was found. Further, a more pronounced discrimination effect was observed in the high correlation condition. With regard to the application of TIMSS data, 70% of the items exhibited guessing behaviors and high correlations were observed between scores on the different dimensions (math and science). Based on the simulation study, guessing and correlation effects should thus be considered carefully when choosing a method for assessing dimensionality. Inconsistency in the dimensionality assessment using the four indices with Mplus was observed (1 to 5 dimensions), whereas TESTFACT consistently estimated 2 dimensions. However, further investigation of the internal structure of the TIMSS assessment did not show any connection to content or cognitive domains.
Share
| Citation/Export: |
|
| Social Networking: |
|
Details
| Item Type: |
University of Pittsburgh ETD
|
| Status: |
Unpublished |
| Creators/Authors: |
|
| ETD Committee: |
| Title | Member | Email Address | Pitt Username | ORCID |
|---|
| Committee Chair | Stone, Clement A | | | | | Committee Member | Kim, Kevin H | | | | | Committee Member | Kirisci, Levent | | | | | Committee Member | Lane, Suzanne | | | |
|
| Date: |
27 June 2007 |
| Date Type: |
Completion |
| Defense Date: |
23 February 2007 |
| Approval Date: |
27 June 2007 |
| Submission Date: |
18 April 2007 |
| Access Restriction: |
No restriction; Release the ETD for access worldwide immediately. |
| Institution: |
University of Pittsburgh |
| Schools and Programs: |
School of Education > Psychology in Education |
| Degree: |
PhD - Doctor of Philosophy |
| Thesis Type: |
Doctoral Dissertation |
| Refereed: |
Yes |
| Uncontrolled Keywords: |
dimensionality; guessing effect; Monte Carlo study; multiple-choice |
| Other ID: |
http://etd.library.pitt.edu/ETD/available/etd-04182007-060503/, etd-04182007-060503 |
| Date Deposited: |
10 Nov 2011 19:38 |
| Last Modified: |
15 Nov 2016 13:40 |
| URI: |
http://d-scholarship.pitt.edu/id/eprint/7286 |
Metrics
Monthly Views for the past 3 years
Plum Analytics
Actions (login required)
 |
View Item |
![[feed]](/images/feed-icon-32x32.png){kind=icon}