Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form
D-Scholarship @ Pitt Banner and Links to Homepage

PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer

Chen, Lichang (2023) PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer. Master's Thesis, University of Pittsburgh. (Unpublished)

[img] PDF
Restricted to University of Pittsburgh users only until 13 June 2024.
Download (1MB) | Request a Copy

Abstract

Recent studies show that prompt tuning can better leverage the power of large language models than fine-tuning on downstream natural language understanding tasks. However, the existing prompt tuning methods have training instability issues, as the variance of scores under different random seeds is quite large. To address this critical problem, we first investigate and find that the loss landscape of vanilla prompt tuning is precipitous when it is visualized, where a slight change of input data can cause a big fluctuation in the loss landscape. This is an essential factor that leads to the instability of prompt tuning. Based on this observation, we introduce perturbation-based regularizers, which can smooth the loss landscape, into prompt tuning. We propose a new algorithm, called Prompt Tuning with Perturbation-based regularizer (PTP), which can not only alleviate training instability dramatically but also boost the performance of prompt tuning. We design two kinds of perturbation-based regularizers, including random-noise-based and adversarial-based. In particular, our proposed perturbations are flexible on both text space and embedding space. Extensive experiments show the effectiveness of our proposed methods in stabilizing the training. Our new algorithms improve the state-of-the-art prompt tuning methods by 1.94% and 2.34% on SuperGLUE and FewGLUE benchmarks, respectively.

Share

Citation/Export:
Social Networking:
Share |

Details

Item Type: University of Pittsburgh ETD
Status: Unpublished
Creators/Authors:
CreatorsEmailPitt UsernameORCID
Chen, Lichanglic138@pitt.edulic138
ETD Committee:
TitleMemberEmail AddressPitt UsernameORCID
Thesis AdvisorHuang, Hengheng.huang@pitt.edu
Committee MemberGao, Weiweigao@pitt.edu
Committee MemberZhan, Liangliang.zhan@pitt.edu
Date: 13 June 2023
Date Type: Publication
Defense Date: 12 April 2023
Approval Date: 13 June 2023
Submission Date: 27 March 2023
Access Restriction: 1 year -- Restrict access to University of Pittsburgh for a period of 1 year.
Number of Pages: 20
Institution: University of Pittsburgh
Schools and Programs: Swanson School of Engineering > Computer Engineering
Degree: MS - Master of Science
Thesis Type: Master's Thesis
Refereed: Yes
Uncontrolled Keywords: Machine Learning, Natural Language Processing
Related URLs:
Date Deposited: 13 Jun 2023 14:18
Last Modified: 13 Jun 2023 14:18
URI: http://d-scholarship.pitt.edu/id/eprint/44340

Metrics

Monthly Views for the past 3 years

Plum Analytics

Actions (login required)

View Item View Item