Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

Impact of Annotation Difficulty on Automatically Detecting Problem Localization of Peer-Review Feedback

Xiong, Wenting and Litman, Diane J. and Schunn, Christian D. (2010) Impact of Annotation Difficulty on Automatically Detecting Problem Localization of Peer-Review Feedback. In: 2010 Workshop on Computer Supported Peer Review in Education.

[img]
Preview
PDF
Submitted Version
Available under License : See the attached license file.

Download (159kB) | Preview
[img] Plain Text (licence)
Available under License : See the attached license file.

Download (1kB)

Abstract

We believe that providing assessment on students ’ reviewing performance will enable students to improve the quality of their peer reviews. We focus on assessing one particular aspect of the textual feedback contained in a peer review – the presence or absence of problem localization; feedback containing problem localization has been shown to be associated with increased understanding and implementation of the feedback. While in prior work we demonstrated the feasibility of learning to predict problem localization using linguistic features automatically extracted from textual feedback, we hypothesize that inter-annotator disagreement on labeling problem localization might impact both the accuracy and the content of the predictive models. To test this hypothesis, we compare the use of feedback examples where problem localization is labeled with differing levels of annotator agreement, for both training and testing our models. Our results show that when models are trained and tested using only feedback where annotators agree on problem localization, the models both perform with high accuracy, and contain rules involving just two simple linguistic features. In contrast, when training and testing using feedback examples where annotators both agree and disagree, the model performance slightly drops, but the learned rules capture more subtle patterns of problem localization. Keywords problem localization in text comments, data mining of peer reviews, inter-annotator agreement, natural languag


Share

Citation/Export:
Social Networking:
Share |

Details

Item Type: Conference or Workshop Item (Paper)
Status: Published
Creators/Authors:
CreatorsEmailPitt UsernameORCID
Xiong, Wenting
Litman, Diane J.dlitman@pitt.eduDLITMAN
Schunn, Christian D.schunn@pitt.eduSCHUNN0000-0003-3589-297X
Centers: University Centers > Learning Research and Development Center (LRDC)
Date: 2010
Date Type: Publication
Event Title: 2010 Workshop on Computer Supported Peer Review in Education
Event Type: Conference
Schools and Programs: Dietrich School of Arts and Sciences > Computer Science
Dietrich School of Arts and Sciences > Intelligent Systems
Refereed: Yes
Date Deposited: 23 Sep 2014 02:42
Last Modified: 01 Nov 2018 13:58
URI: http://d-scholarship.pitt.edu/id/eprint/22952

Metrics

Monthly Views for the past 3 years

Plum Analytics


Actions (login required)

View Item View Item