• Comparing rating modes: analysing live, audio, and video ratings of IELTS Speaking Test performances

      Nakatsuhara, Fumiyo; Inoue, Chihiro; Taylor, Lynda; (Taylor & Francis, 2020-08-26)
      This mixed methods study compared IELTS examiners’ scores when assessing spoken performances under live and two ‘non-live’ testing conditions using audio and video recordings. Six IELTS examiners assessed 36 test-takers’ performances under the live, audio, and video rating conditions. Scores in the three rating modes were calibrated using the many-facet Rasch model (MFRM). For all three modes, examiners provided written justifications for their ratings, and verbal reports were also collected to gain insights into examiner perceptions towards performance under the audio and video conditions. Results showed that, for all rating criteria, audio ratings were significantly lower than live and video ratings. Examiners noticed more negative performance features under the two non-live rating conditions, compared to the live condition. However, richer information about test-taker performance in the video mode appeared to cause raters to rely less on such negative evidence than audio raters when awarding scores. Verbal report data showed that having visual information in the video-rating mode helped examiners to understand what the test-takers were saying, to comprehend better what test-takers were communicating using non-verbal means, and to understand with greater confidence the source of test-takers’ hesitation, pauses, and awkwardness.
    • Exploring the use of video-conferencing technology in the assessment of spoken language: a mixed-methods study

      Nakatsuhara, Fumiyo; Inoue, Chihiro; Berry, Vivien; Galaczi, Evelina D.; University of Bedfordshire; British Council; Cambridge English Language Assessment (Taylor & Francis, 2017-02-10)
      This research explores how internet-based video-conferencing technology can be used to deliver and conduct a speaking test, and what similarities and differences can be discerned between the standard and computer-mediated face-to-face modes. The context of the study is a high-stakes speaking test, and the motivation for the research is the need for test providers to keep under constant review the extent to which their tests are accessible and fair to a wide constituency of test takers. The study examines test-takers’ scores and linguistic output, and examiners’ test administration and rating behaviors across the two modes. A convergent parallel mixed-methods research design was used, analyzing test-takers’ scores and language functions elicited, examiners’ written comments, feedback questionnaires and verbal reports, as well as observation notes taken by researchers. While the two delivery modes generated similar test score outcomes, some differences were observed in test-takers’ functional output and the behavior of examiners who served as both raters and interlocutors.
    • General Language Proficiency (GLP): reflections on the "issues revisited" from the perspective of a UK examination board

      Taylor, Lynda (Taylor & Francis, 2014-05-21)
      Looking back to the language testing world of the 1980s in the United Kingdom, we need to be aware that how we perceive or remember ourselves to have been then—whether as individual language testing academics or as corporate language testing organisations—will be shaped by multiple influences. Although we may have been present at and shared in the 1980 discussions, our recollections of how things were then and our views on how they have (or have not) changed will vary. What follows in this article offers a predominantly personal perspective. It is the view as I perceive it, in light of my own journey as a UK-based language teacher and tester over the past 30 years, seen from where I stand now as a consultant to a large international examining board in the United Kingdom. It is also therefore an institutional perspective, drawing on a long association with one particular language testing organisation. Just as my perspective is from the position of only one language testing institution, I am also only one individual from within that institution. There will undoubtedly be other stances, voices, and perspectives that are equally valid and relevant from within the same institution.
    • Interactional competence with and without extended planning time in a group oral assessment

      Lam, Daniel M. K. (Routledge, Taylor & Francis Group, 2019-05-02)
      Linking one’s contribution to those of others’ is a salient feature demonstrating interactional competence in paired/group speaking assessments. While such responses are to be constructed spontaneously while engaging in real-time interaction, the amount and nature of pre-task preparation in paired/group speaking assessments may have an influence on how such an ability (or lack thereof) could manifest in learners’ interactional performance. Little previous research has examined the effect of planning time on interactional aspects of paired/group speaking task performance. Within the context of school-based assessment in Hong Kong, this paper analyzes the discourse of two group interactions performed by the same four student-candidates under two conditions: (a) with extended planning time (4–5 hours), and (b) without extended planning time (10 minutes), with the aim of exploring any differences in student-candidates’ performance of interactional competence in this assessment task. The analysis provides qualitative discourse evidence that extended planning time may impede the assessment task’s capacity to discriminate between stronger and weaker candidates’ ability to spontaneously produce responses contingent on previous speaker contribution. Implications for the implementation of preparation time for the group interaction task are discussed.
    • Interactional competence: conceptualisations, operationalisations, and outstanding questions

      Galaczi, Evelina D.; Taylor, Lynda (Routledge, 2018-04-30)
      This article on interactional competence provides an overview of the historical influences that have shaped theoretical conceptualisations of this construct as it relates to spoken language use, leading to the current view of it as involving both cognitive and social dimensions, and then describes its operationalisation in tests and assessment scales, and the challenges associated with this activity. Looking into the future, issues that need to be dealt with include developing a fuller representation of the construct and of more contextually relevant assessments, deciding upon additional assessment criteria and the appropriate interpretation thereof, and determining how technology can be applied in assessment practice and the extent to which technology fundamentally changes the construct itself. These all have implications for testing if it is to be relevant and fit for purpose.
    • Linking tests of English for academic purposes to the CEFR: the score user’s perspective

      Green, Anthony (Taylor and Francis, 2017-11-13)
      The Common European Framework of Reference for Languages (CEFR) is widely used in setting language proficiency requirements, including for international students seeking access to university courses taught in English. When different language examinations have been related to the CEFR, the process is claimed to help score users, such as university admissions staff, to compare and evaluate these examinations as tools for selecting qualified applicants. This study analyses the linking claims made for four internationally recognised tests of English widely used in university admissions. It uses the Council of Europe’s (2009) suggested stages of specification, standard setting, and empirical validation to frame an evaluation of the extent to which, in this context, the CEFR has fulfilled its potential to “facilitate comparisons between different systems of qualifications.” Findings show that testing agencies make little use of CEFR categories to explain test content; represent the relationships between their tests and the framework in different terms; and arrive at conflicting conclusions about the correspondences between test scores and CEFR levels. This raises questions about the capacity of the CEFR to communicate competing views of a test construct within a coherent overarching structure.
    • Recommending a nursing-specific passing standard for the IELTS examination

      O'Neill, Thomas R.; Buckendahl, Chad W.; Plake, Barbara S.; Taylor, Lynda (Taylor & Francis, 2007-12-05)
      Licensure testing programs in the United States (e.g., nursing) face an increasing challenge of measuring the competency of internationally trained candidates, both in relation to their clinical competence and their English language competence. To assist with the latter, professional licensing bodies often adopt well-established and widely available international English language proficiency measures. In this context, the National Council of State Boards of Nursing (NCSBN) sought to develop a nursing-specific passing standard on the International English Language Testing System that U.S. jurisdictions could consider in their licensure decisions for internationally trained candidates. Findings from a standard setting exercise were considered by NCSBN's Examination Committee in conjunction with other relevant information to produce a legally defensible passing standard on the test. This article reports in detail on the standard setting exercise conducted as part of this policy-making process; it describes the techniques adopted, the procedures followed, and the outcomes obtained. The study is contextualized within the current literature on standard setting. The latter part of the article describes the nature of the policy-making process to which the study contributed and discusses some of the implications of including a language literacy test as part of a licensure testing program.