Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

Including Facial Expressions in Contextual Embeddings for Sign Language Generation

Viegas, Carla and Inan, Mert and Quandt, Lorna and Alikhani, Malihe (2021) Including Facial Expressions in Contextual Embeddings for Sign Language Generation. In: AAAI 2022, Feb 22-Mar 1 20222, Virtual. (Unpublished)

Download (18MB) | Preview


State-of-the-art sign language generation frameworks lack expressivity and naturalness which is the result of only focusing manual signs, neglecting the affective, grammatical and semantic functions of facial expressions. The purpose of this work is to augment semantic representation of sign language through grounding facial expressions. We study the effect of modeling the relationship between text, gloss, and facial expressions on the performance of the sign generation systems. In particular, we propose a Dual Encoder Transformer able to generate manual signs as well as facial expressions by capturing the similarities and differences found in text and sign gloss annotation. We take into consideration the role of facial muscle activity to express intensities of manual signs by being the first to employ facial action units in sign language generation. We perform a series of experiments showing that our proposed model improves the quality of automatically generated sign language.


Social Networking:
Share |


Item Type: Conference or Workshop Item (Paper)
Status: Unpublished
CreatorsEmailPitt UsernameORCID
Inan, Mertmert.inan@pitt.edumei130000-0001-6972-411X
Quandt, Lornalorna.quandt@gallaudet.edu0000-0002-0032-1918
Alikhani, Malihemalihe@pitt.edumalihe0000-0002-1315-2228
Date: 8 September 2021
Date Type: Submission
Event Title: AAAI 2022
Event Dates: Feb 22-Mar 1 20222
Event Type: Conference
Schools and Programs: School of Computing and Information > Computer Science
Refereed: No
Date Deposited: 18 Feb 2022 19:52
Last Modified: 18 Feb 2022 19:52


Monthly Views for the past 3 years

Plum Analytics

Actions (login required)

View Item View Item