An investigation into double-marking methods: comparing live, audio and video rating of performance on the IELTS Speaking Test

2.50
Hdl Handle:
http://hdl.handle.net/10547/622259
Title:
An investigation into double-marking methods: comparing live, audio and video rating of performance on the IELTS Speaking Test
Authors:
Nakatsuhara, Fumiyo; Inoue, Chihiro; Taylor, Lynda
Abstract:
This study compared IELTS examiners’ scores when they assessed test-takers’ spoken performance under live and two non-live rating conditions using audio and video recordings. It also explored examiners’ perceptions towards test-takers’ performance in the two non-live rating modes.  This was a mixed-methods study that involved both existing and newly collected datasets. A total of six trained IELTS examiners assessed 36 test-takers’ performance under the live, audio and video rating conditions. Their scores in the three modes of rating were calibrated using the multifaceted Rasch model analysis.  In all modes of rating, the examiners were asked to make notes on why they awarded the scores that they did on each analytical category. The comments were quantitatively analysed in terms of the volume of positive and negative features of test-takers’ performance that examiners reported noticing when awarding scores under the three rating conditions.  Using selected test-takers’ audio and video recordings, examiners’ verbal reports were also collected to gain insights into their perceptions towards test-takers’ performance under the two non-live conditions.  The results showed that audio ratings were significantly lower than live and video ratings for all rating categories. Examiners noticed more negative performance features of test-takers under the two non-live rating conditions than the live rating condition. The verbal report data demonstrated how having visual information in the video-rating mode helped examiners to understand test-takers’ utterances, to see what was happening beyond what the test-takers were saying and to understand with more confidence the source of test-takers’ hesitation, pauses and awkwardness in their performance.  The results of this study have, therefore, offered a better understanding of the three modes of rating, and a recommendation was made regarding enhanced double-marking methods that could be introduced to the IELTS Speaking Test.
Citation:
(2017) 'An investigation into double-marking methods: comparing live, audio and video rating of performance on the IELTS Speaking Test'. : The IELTS Partners: British Council, IDP: IELTS Australia and Cambridge English Language Assessment.
Publisher:
The IELTS Partners: British Council, IDP: IELTS Australia and Cambridge English Language Assessment
Issue Date:
1-Mar-2017
URI:
http://hdl.handle.net/10547/622259
Additional Links:
https://www.ielts.org/-/media/research-reports/ielts_online_rr_2017-1.ashx
Type:
Technical Report
Language:
en
Series/Report no.:
ISSN 2201-2982 2017/1
Sponsors:
The IELTS Partners: British Council, IDP: IELTS Australia and Cambridge English Language Assessment
Appears in Collections:
English language learning and assessment

Full metadata record

DC FieldValue Language
dc.contributor.authorNakatsuhara, Fumiyoen
dc.contributor.authorInoue, Chihiroen
dc.contributor.authorTaylor, Lyndaen
dc.date.accessioned2017-10-05T10:44:55Z-
dc.date.available2017-10-05T10:44:55Z-
dc.date.issued2017-03-01-
dc.identifier.citation(2017) 'An investigation into double-marking methods: comparing live, audio and video rating of performance on the IELTS Speaking Test'. : The IELTS Partners: British Council, IDP: IELTS Australia and Cambridge English Language Assessment.en
dc.identifier.urihttp://hdl.handle.net/10547/622259-
dc.description.abstractThis study compared IELTS examiners’ scores when they assessed test-takers’ spoken performance under live and two non-live rating conditions using audio and video recordings. It also explored examiners’ perceptions towards test-takers’ performance in the two non-live rating modes.  This was a mixed-methods study that involved both existing and newly collected datasets. A total of six trained IELTS examiners assessed 36 test-takers’ performance under the live, audio and video rating conditions. Their scores in the three modes of rating were calibrated using the multifaceted Rasch model analysis.  In all modes of rating, the examiners were asked to make notes on why they awarded the scores that they did on each analytical category. The comments were quantitatively analysed in terms of the volume of positive and negative features of test-takers’ performance that examiners reported noticing when awarding scores under the three rating conditions.  Using selected test-takers’ audio and video recordings, examiners’ verbal reports were also collected to gain insights into their perceptions towards test-takers’ performance under the two non-live conditions.  The results showed that audio ratings were significantly lower than live and video ratings for all rating categories. Examiners noticed more negative performance features of test-takers under the two non-live rating conditions than the live rating condition. The verbal report data demonstrated how having visual information in the video-rating mode helped examiners to understand test-takers’ utterances, to see what was happening beyond what the test-takers were saying and to understand with more confidence the source of test-takers’ hesitation, pauses and awkwardness in their performance.  The results of this study have, therefore, offered a better understanding of the three modes of rating, and a recommendation was made regarding enhanced double-marking methods that could be introduced to the IELTS Speaking Test.en
dc.description.sponsorshipThe IELTS Partners: British Council, IDP: IELTS Australia and Cambridge English Language Assessmenten
dc.language.isoenen
dc.publisherThe IELTS Partners: British Council, IDP: IELTS Australia and Cambridge English Language Assessmenten
dc.relation.ispartofseriesISSN 2201-2982 2017/1en
dc.relation.urlhttps://www.ielts.org/-/media/research-reports/ielts_online_rr_2017-1.ashxen
dc.subjectlanguage testingen
dc.subjectspeakingen
dc.subjectlanguage assessmenten
dc.subjectmixed-methods researchen
dc.subjectQ330 English as a second languageen
dc.titleAn investigation into double-marking methods: comparing live, audio and video rating of performance on the IELTS Speaking Testen
dc.typeTechnical Reporten
dc.date.updated2017-10-05T09:42:18Z-
dc.description.notefile attached is copyright so not able to be passed to repository. However as report not journal article, full text not required for REF so passing metadata only. RVO 5/10/17-
All Items in UOBREP are protected by copyright, with all rights reserved, unless otherwise indicated.