Link to the University of Pittsburgh Homepage
Link to the University Library System Homepage Link to the Contact Us Form

Examiner judgments of collocational proficiency in L2 English learners’ writing

Naismith, Benjamin S (2022) Examiner judgments of collocational proficiency in L2 English learners’ writing. Doctoral Dissertation, University of Pittsburgh. (Unpublished)

Primary Text

Download (8MB) | Preview


This study investigates how aspects of collocational proficiency affect the ratings that expert examiners give to second language (L2) English learner essays. Lexical proficiency is a multi-faceted phenomenon and certain aspects of it are particularly impactful on human judgements, including lexical sophistication and accuracy. However, the importance of proficiency with formulaic sequences (FSs), like collocations, has received less attention than proficiency with single words, despite FSs’ essential role in language production. In addition, previous comparison studies have used a small number of raters with varying levels of assessment expertise, assessing texts of varying length and topic.

In addressing these issues, this study uses a predominantly quantitative, experimental approach comprised of two stages. First, a small set of three texts of different proficiency levels were created based on model IELTS Task 2 essays, controlling for topic and length. From these texts, a set of 30 versions were produced, manipulating specific collocational features related to sophistication and accuracy. Second, IELTS examiners (n = 47) rated the texts and provided rationales for their choices. From these data, many-faceted Rasch models were used to obtain expected scores, and linear regression models were used to determine which aspects of collocational proficiency best predicted the experts’ ratings.

The findings reveal that increases in lexical sophistication significantly and positively impacted the experts’ ratings. Post-hoc analyses demonstrated that the categories of high sophistication and mid sophistication differed significantly from low sophistication. However, mid sophistication was not significantly different from high sophistication. When these ‘advanced’ words were used as part of collocations, they then provided a small but significant additional boost to ratings. Notably, there was no significant effect for increased collocational accuracy. In conjunction, these findings indicate that 1) sophistication is perhaps best viewed on a spectrum rather than categorically, 2) there is an additional increase to ratings if learners use advanced lexis as part of collocations, and 3) there is a potential baseline in terms of gravity and frequency of collocation errors below which ratings are not significantly affected. The implications for these findings are therefore discussed in relation to written language assessment and L2 vocabulary pedagogy.


Social Networking:
Share |


Item Type: University of Pittsburgh ETD
Status: Unpublished
CreatorsEmailPitt UsernameORCID
Naismith, Benjamin Sbnaismith@pitt.eduBEN250000-0001-8347-3142
ETD Committee:
TitleMemberEmail AddressPitt UsernameORCID
Committee ChairJuffs, Alanjuffs@pitt.edu0000-0002-4741-6412
Committee MemberRömer, Uteuroemer@gsu.edu0000-0002-0896-3165
Committee MemberKanwit, Matthewmkanwit@pitt.edu0000-0002-7389-1903
Committee MemberFricke, Melindamelinda.fricke@pitt.edu0000-0001-6522-1937
Date: 30 June 2022
Date Type: Publication
Defense Date: 11 February 2022
Approval Date: 30 June 2022
Submission Date: 16 March 2022
Access Restriction: No restriction; Release the ETD for access worldwide immediately.
Number of Pages: 204
Institution: University of Pittsburgh
Schools and Programs: Dietrich School of Arts and Sciences > Linguistics
Degree: PhD - Doctor of Philosophy
Thesis Type: Doctoral Dissertation
Refereed: Yes
Uncontrolled Keywords: collocation, lexis, language assessment, writing assessment, sophistication, accuracy
Date Deposited: 30 Jun 2022 23:23
Last Modified: 19 May 2023 14:02


Monthly Views for the past 3 years

Plum Analytics

Actions (login required)

View Item View Item