Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

User Simulation for Spoken Dialog System Development

Ai, Hua (2010) User Simulation for Spoken Dialog System Development. Doctoral Dissertation, University of Pittsburgh. (Unpublished)

Primary Text

Download (887kB) | Preview


A user simulation is a computer program which simulates human user behaviors. Recently, user simulations have been widely used in two spoken dialog system development tasks. One is to generate large simulated corpora for applying machine learning to learn new dialog strategies, and the other is to replace human users to test dialog system performance. Although previous studies have shown successful examples of applying user simulations in both tasks, it is not clear what type of user simulation is most appropriate for a specific task because few studies compare different user simulations in the same experimental setting. In this research, we investigate how to construct user simulations in a specific task for spoken dialog system development. Since most current user simulations generate user actions based on probabilistic models, we identify two main factors in constructing such user simulations: the choice of user simulation model and the approach to set up user action probabilities. We build different user simulation models which differ in their efforts in simulating realistic user behaviors and exploring more user actions. We also investigate different manual and trained approaches to set up user action probabilities. We introduce both task-dependent and task-independent measures to compare these simulations. We show that a simulated user which mimics realistic user behaviors is not always necessary for the dialog strategy learning task. For the dialog system testing task, a user simulation which simulates user behaviors in a statistical way can generate both objective and subjective measures of dialog system performance similar to human users. Our research examines the strengths and weaknesses of user simulations in spoken dialog system development. Although our results are constrained to our task domain and the resources available, we provide a general framework for comparing user simulations in a task-dependent context. In addition, we summarize and validate a set of evaluation measures that can be used in comparing different simulated users as well as simulated versus human users.


Social Networking:
Share |


Item Type: University of Pittsburgh ETD
Status: Unpublished
CreatorsEmailPitt UsernameORCID
ETD Committee:
TitleMemberEmail AddressPitt UsernameORCID
Committee ChairLitman, Dianelitman@cs.pitt.eduDLITMAN
Committee MemberWiebe, Janycewiebe@cs.pitt.eduJMW106
Committee MemberEskenazi,
Committee MemberJordan, Pamelapjordan@pitt.eduPJORDAN
Committee MemberHwa, Rebeccahwa@cs.pitt.eduREH23
Date: 26 January 2010
Date Type: Completion
Defense Date: 21 September 2009
Approval Date: 26 January 2010
Submission Date: 3 December 2009
Access Restriction: No restriction; Release the ETD for access worldwide immediately.
Institution: University of Pittsburgh
Schools and Programs: Dietrich School of Arts and Sciences > Intelligent Systems
Degree: PhD - Doctor of Philosophy
Thesis Type: Doctoral Dissertation
Refereed: Yes
Uncontrolled Keywords: Natural Language Processing; Spoken Dialog System; Artificial Intelligence; User Simulation
Other ID:, etd-12032009-115538
Date Deposited: 10 Nov 2011 20:07
Last Modified: 15 Nov 2016 13:52


Monthly Views for the past 3 years

Plum Analytics

Actions (login required)

View Item View Item