Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form


Goldin, Ilya M. (2011) A FOCUS ON CONTENT: THE USE OF RUBRICS IN PEER REVIEW TO GUIDE STUDENTS AND INSTRUCTORS. Doctoral Dissertation, University of Pittsburgh. (Unpublished)

Primary Text

Download (969kB) | Preview


Students who are solving open-ended problems would benefit from formative assessment, i.e., from receiving helpful feedback and from having an instructor who is informed about their level of performance. Open-ended problems challenge existing assessment techniques. For example, such problems may have reasonable alternative solutions, or conflicting objectives. Analyses of open-ended problems are often presented as free-form text since they require arguments and justifications for one solution over others, and students may differ in how they frame the problems according to their knowledge, beliefs and attitudes.This dissertation investigates how peer review may be used for formative assessment. Computer-Supported Peer Review in Education, a technology whose use is growing, has been shown to provide accurate summative assessment of student work, and peer feedback can indeed be helpful to students. A peer review process depends on the rubric that students use to assess and give feedback to each other. However, it is unclear how a rubric should be structured to produce feedback that is helpful to the student and at the same time to yield information that could be summarized for the instructor.The dissertation reports a study in which students wrote individual analyses of an open-ended legal problem, and then exchanged feedback using Comrade, a web application for peer review. The study compared two conditions: some students used a rubric that was relevant to legal argument in general (the domain-relevant rubric), while others used a rubric that addressed the conceptual issues embedded in the open-ended problem (the problem-specific rubric).While both rubric types yield peer ratings of student work that approximate the instructor's scores, feedback elicited by the domain-relevant rubric was redundant across its dimensions. On the contrary, peer ratings elicited by the problem-specific rubric distinguished among its dimensions. Hierarchical Bayesian models showed that ratings from both rubrics can be fit by pooling information across students, but only problem-specific ratings are fit better given information about distinct rubric dimensions.


Social Networking:
Share |


Item Type: University of Pittsburgh ETD
Status: Unpublished
CreatorsEmailPitt UsernameORCID
Goldin, Ilya M.goldin@pitt.eduGOLDIN
ETD Committee:
TitleMemberEmail AddressPitt UsernameORCID
Committee ChairAshley, Kevin D. ashley@pitt.eduASHLEY
Committee MemberSchunn, Christian D.schunn@pitt.eduSCHUNN
Committee MemberGomez, Louis lgomez@pitt.eduLGOMEZ
Committee MemberBrusilovsky, Peterpeterb@pitt.eduPETERB
Date: 27 September 2011
Date Type: Completion
Defense Date: 29 April 2011
Approval Date: 27 September 2011
Submission Date: 14 July 2011
Access Restriction: No restriction; Release the ETD for access worldwide immediately.
Institution: University of Pittsburgh
Schools and Programs: Dietrich School of Arts and Sciences > Intelligent Systems
Degree: PhD - Doctor of Philosophy
Thesis Type: Doctoral Dissertation
Refereed: Yes
Uncontrolled Keywords: artificial intelligence in education; educational data mining; intelligent tutoring systems
Other ID:, etd-07142011-004329
Date Deposited: 10 Nov 2011 19:51
Last Modified: 15 Nov 2016 13:45


Monthly Views for the past 3 years

Plum Analytics

Actions (login required)

View Item View Item