Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

Assessment of Interrater Reliability Among C. Elegans Researchers Measuring Developmental Stage by the Kappa Statistic and Latent Variable Modeling

Ferguson, Annabel (2014) Assessment of Interrater Reliability Among C. Elegans Researchers Measuring Developmental Stage by the Kappa Statistic and Latent Variable Modeling. Master's Thesis, University of Pittsburgh. (Unpublished)

[img]
Preview
PDF
Primary Text

Download (1MB) | Preview

Abstract

Use of the tiny nematode worm Caenorhabditis elegans as a model organism for biological research has had a considerable influence on scientific discoveries. C. elegans research has a public health relevance as it has led to better understanding and treatments of diseases like cancer and neurodegenerative diseases, which are a public health concern. Additionally, C. elegans research offers promise towards a better understanding of the public health problems of human obesity and diabetes, as many initial controlled experiments may only be done in a model organism (as opposed to human studies). A commonly employed skill among C. elegans researchers is the ability to reliably and accurately distinguish among the five stages of worm development by eye; this is required for both producing quality data, and for successful worm maintenance and genomic manipulation. While it is reasonable to presume that there is some amount of variability from researcher to researcher in classifying worms into particular stages, there is little documentation of formal assessments of reliability between researchers.
The topic of statistical assessment of interrater reliability has been addressed extensively as it applies to fields like medical diagnostics, and psychological and sociological studies. While numerous methods exist, a popular way of assessing the reliability of two or more different measurements on a categorical scale is the kappa statistic. The ease of computation and the single numerical index (ranging from 0 to 1) of the kappa make it a commonly used “quick” method of this assessment. However, there are numerous problems that arise in the interpretation and application of kappa that can make it untrustworthy, especially when it is used for the analysis of data with an ordinal outcome such as developmental stage.
An alternative methodology is latent variable modeling using a common factor model with a probit transformation. This approach provides more information about the strength of association and bias among raters, and does not give the potentially confusing or paradoxical results that kappa does. The following study has applied both of these methods to a dataset containing the ratings of worm larval stage of development for a population of 60 worms by seven raters. This study finds that while both the kappa and the modeling approach give concordant results, modeling the data provides a more useful and meaningful summary of the agreement between raters. Additionally, it finds that the overall agreement is high, but that there is some degree of variability in the cutoff thresholds by which raters assign developmental stage.


Share

Citation/Export:
Social Networking:
Share |

Details

Item Type: University of Pittsburgh ETD
Status: Unpublished
Creators/Authors:
CreatorsEmailPitt UsernameORCID
Ferguson, Annabelaaf6@pitt.eduAAF6
ETD Committee:
TitleMemberEmail AddressPitt UsernameORCID
Thesis AdvisorMarsh, Garygmarsh@pitt.eduGMARSH
Committee MemberBuchanich, Jeanine Mjeanine@pitt.edu; jbuchanich@aol.comJEANINE
Committee MemberBilonick, Richard Arab45@pitt.eduRAB45
Committee MemberFisher, Alfred Lafisher@pitt.eduAFISHER
Date: 29 January 2014
Date Type: Publication
Defense Date: 27 November 2014
Approval Date: 29 January 2014
Submission Date: 19 December 2013
Access Restriction: 1 year -- Restrict access to University of Pittsburgh for a period of 1 year.
Number of Pages: 79
Institution: University of Pittsburgh
Schools and Programs: School of Public Health > Biostatistics
Degree: MS - Master of Science
Thesis Type: Master's Thesis
Refereed: Yes
Uncontrolled Keywords: C. elegans Interrater Reliability Common Factor Model
Date Deposited: 29 Jan 2014 17:17
Last Modified: 15 Nov 2016 14:16
URI: http://d-scholarship.pitt.edu/id/eprint/20317

Metrics

Monthly Views for the past 3 years

Plum Analytics


Actions (login required)

View Item View Item