Ferguson, Annabel
(2014)
Assessment of Interrater Reliability Among C. Elegans Researchers Measuring Developmental Stage by the Kappa Statistic and Latent Variable Modeling.
Master's Thesis, University of Pittsburgh.
(Unpublished)
Abstract
Use of the tiny nematode worm Caenorhabditis elegans as a model organism for biological research has had a considerable influence on scientific discoveries. C. elegans research has a public health relevance as it has led to better understanding and treatments of diseases like cancer and neurodegenerative diseases, which are a public health concern. Additionally, C. elegans research offers promise towards a better understanding of the public health problems of human obesity and diabetes, as many initial controlled experiments may only be done in a model organism (as opposed to human studies). A commonly employed skill among C. elegans researchers is the ability to reliably and accurately distinguish among the five stages of worm development by eye; this is required for both producing quality data, and for successful worm maintenance and genomic manipulation. While it is reasonable to presume that there is some amount of variability from researcher to researcher in classifying worms into particular stages, there is little documentation of formal assessments of reliability between researchers.
The topic of statistical assessment of interrater reliability has been addressed extensively as it applies to fields like medical diagnostics, and psychological and sociological studies. While numerous methods exist, a popular way of assessing the reliability of two or more different measurements on a categorical scale is the kappa statistic. The ease of computation and the single numerical index (ranging from 0 to 1) of the kappa make it a commonly used “quick” method of this assessment. However, there are numerous problems that arise in the interpretation and application of kappa that can make it untrustworthy, especially when it is used for the analysis of data with an ordinal outcome such as developmental stage.
An alternative methodology is latent variable modeling using a common factor model with a probit transformation. This approach provides more information about the strength of association and bias among raters, and does not give the potentially confusing or paradoxical results that kappa does. The following study has applied both of these methods to a dataset containing the ratings of worm larval stage of development for a population of 60 worms by seven raters. This study finds that while both the kappa and the modeling approach give concordant results, modeling the data provides a more useful and meaningful summary of the agreement between raters. Additionally, it finds that the overall agreement is high, but that there is some degree of variability in the cutoff thresholds by which raters assign developmental stage.
Share
| Citation/Export: |
|
| Social Networking: |
|
Details
| Item Type: |
University of Pittsburgh ETD
|
| Status: |
Unpublished |
| Creators/Authors: |
| Creators | Email | Pitt Username | ORCID  |
|---|
| Ferguson, Annabel | aaf6@pitt.edu | AAF6 | |
|
| ETD Committee: |
|
| Date: |
29 January 2014 |
| Date Type: |
Publication |
| Defense Date: |
27 November 2014 |
| Approval Date: |
29 January 2014 |
| Submission Date: |
19 December 2013 |
| Access Restriction: |
1 year -- Restrict access to University of Pittsburgh for a period of 1 year. |
| Number of Pages: |
79 |
| Institution: |
University of Pittsburgh |
| Schools and Programs: |
School of Public Health > Biostatistics |
| Degree: |
MS - Master of Science |
| Thesis Type: |
Master's Thesis |
| Refereed: |
Yes |
| Uncontrolled Keywords: |
C. elegans Interrater Reliability Common Factor Model |
| Date Deposited: |
29 Jan 2014 17:17 |
| Last Modified: |
15 Nov 2016 14:16 |
| URI: |
http://d-scholarship.pitt.edu/id/eprint/20317 |
Metrics
Monthly Views for the past 3 years
Plum Analytics
Actions (login required)
 |
View Item |
![[feed]](/images/feed-icon-32x32.png){kind=icon}