Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

When Does Disengagement Correlate with Performance in Spoken Dialog Computer Tutoring?

Forbes-Riley, Kate and Litman, Diane J. (2012) When Does Disengagement Correlate with Performance in Spoken Dialog Computer Tutoring? International Journal of Artificial Intelligence in Education, 22 (2). 19 - 42. ISSN 1560-4292

[img]
Preview
PDF
Published Version
Available under License : See the attached license file.

Download (297kB) | Preview
[img] Plain Text (licence)
Available under License : See the attached license file.

Download (1kB)

Abstract

In this paper we investigate how student disengagement relates to two performance metrics in a spoken dialog computer tutoring corpus, both when disengagement is measured through manual annotation by a trained human judge, and also when disengagement is measured through automatic annotation by the system based on a machine learning model. First, we investigate whether manually labeled overall disengagement and six different disengagement types are predictive of learning and user satisfaction in the corpus. Our results show that although students’ percentage of overall disengaged turns negatively correlates both with the amount they learn and their user satisfaction, the individual types of disengagement correlate differently: some negatively correlate with learning and user satisfaction, while others don’t correlate with eithermetric at all. Moreover, these relationships change somewhat depending on student prerequisite knowledge level. Furthermore, using multiple disengagement types to predict learning improves predictive power. Overall, these manual label-based results suggest that although adapting to disengagement should improve both student learning and user satisfaction in computer tutoring, maximizing performance requires the system to detect and respond differently based on disengagement type. Next, we present an approach to automatically detecting and responding to user disengagement types based on their differing correlations with correctness. Investigation of ourmachine learningmodel of user disengagement shows that its automatic labels negatively correlate with both performance metrics in the same way as the manual labels. The similarity of the correlations across the manual and automatic labels suggests that the automatic labels are a reasonable substitute for the manual labels. Moreover, the significant negative correlations themselves suggest that redesigning ITSPOKE to automatically detect and respond to disengagement has the potential to remediate disengagement and thereby improve performance, even in the presence of noise introduced by the automatic detection process.


Share

Citation/Export:
Social Networking:
Share |

Details

Item Type: Article
Status: Published
Creators/Authors:
CreatorsEmailPitt UsernameORCID
Forbes-Riley, Kate
Litman, Diane J.dlitman@pitt.eduDLITMAN
Centers: Other Centers, Institutes, or Units > Learning Research & Development Center
Date: 2012
Date Type: Publication
Access Restriction: No restriction; Release the ETD for access worldwide immediately.
Journal or Publication Title: International Journal of Artificial Intelligence in Education
Volume: 22
Number: 2
Publisher: IOS Press
Page Range: 19 - 42
Institution: University of Pittsburgh
Schools and Programs: Dietrich School of Arts and Sciences > Computer Science
Dietrich School of Arts and Sciences > Computer Science > Intelligent Systems Technical Reports
Refereed: Yes
ISSN: 1560-4292
Related URLs:
Article Type: Research Article
Date Deposited: 14 Aug 2014 15:50
Last Modified: 01 Nov 2017 13:57
URI: http://d-scholarship.pitt.edu/id/eprint/22684

Metrics

Monthly Views for the past 3 years

Plum Analytics


Actions (login required)

View Item View Item