Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

Intrinsic and extrinsic evaluation of an automatic user disengagement detector for an uncertainty-adaptive spoken dialogue system

Friedberg, H and Drummond, J (2012) Intrinsic and extrinsic evaluation of an automatic user disengagement detector for an uncertainty-adaptive spoken dialogue system. In: UNSPECIFIED UNSPECIFIED, 91 - 102. ISBN 1937284204

[img] Plain Text (licence)
Available under License : See the attached license file.

Download (1kB)

Abstract

© 2012 Association for Computational Linguistics. We present a model for detecting user disengagement during spoken dialogue interactions. Intrinsic evaluation of our model (i.e., with respect to a gold standard) yields results on par with prior work. However, since our goal is immediate implementation in a system that already detects and adapts to user uncertainty, we go further than prior work and present an extrinsic evaluation of our model (i.e., with respect to the real-world task). Correlation analyses show crucially that our automatic disengagement labels correlate with system performance in the same way as the gold standard (manual) labels, while regression analyses show that detecting user disengagement adds value over and above detecting only user uncertainty when modeling performance. Our results suggest that automatically detecting and adapting to user disengagement has the potential to significantly improve performance even in the presence of noise, when compared with only adapting to one affective state or ignoring affect entirely.


Share

Citation/Export:
Social Networking:
Share |

Details

Item Type: Book Section
Status: Published
Creators/Authors:
CreatorsEmailPitt UsernameORCID
Friedberg, H
Drummond, J
Centers: University Centers > Learning Research and Development Center (LRDC)
Date: 1 January 2012
Date Type: Publication
Journal or Publication Title: NAACL HLT 2012 - 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference
Page Range: 91 - 102
Event Type: Conference
Schools and Programs: Dietrich School of Arts and Sciences > Computer Science
Dietrich School of Arts and Sciences > Intelligent Systems
Refereed: Yes
ISBN: 1937284204
Date Deposited: 22 Sep 2014 19:52
Last Modified: 05 Jan 2019 15:58
URI: http://d-scholarship.pitt.edu/id/eprint/22883

Metrics

Monthly Views for the past 3 years

Plum Analytics


Actions (login required)

View Item View Item