• Defining integrated reading-into-writing constructs: evidence at the B2 C1 interface

      Chan, Sathena Hiu Chong (Cambridge University Press, 2018-06-01)
    • Demonstrating the cognitive validity and face validity of PTE Academic Writing items Summarize Written Text and Write Essay

      Chan, Sathena Hiu Chong (Pearson, 2011-07-01)
      This study examines the cognitive validity of two item types of the Writing Section of the PTE Academic test – Summarize Written Text and Write Essay - within Weir’s (2005) socio-cognitive framework for test validation. The study focuses on cognitive validity by investigating and comparing the cognitive processes of a group of ESL test takers undertaking Summarize Written Text (an integrated writing item) and Write Essay (an independent writing item). Cognitive validity is a ‘measure of how closely it [a writing task] represents the cognitive processing involved in writing contexts beyond the test itself’ (Shaw and Weir, 2007:34). In addition, the study investigates test takers’ opinions regarding the two different writing item types: independent and integrated. Test takers’ scores on both items are compared to investigate if the two performances correlate. The study uses screen capture technique to record test takers’ successive writing processes on both items, followed by retrospective stimulated recalls. The findings demonstrate that Summarize Written Text and Write Essay engage different cognitive processes that are essential in academic writing contexts. In particular, macro-planning and discourse synthesis processes such as selecting relevant ideas from source text are elicited by the Summarize Written Text item whereas processes in micro-planning, monitoring and revising at low levels are activated on the Write Essay item. In terms of test performances, the results show that test takers in this study performed significantly better on Write Essay than on Summarize Written Text.
    • Developing an advanced, specialized English proficiency test for Beijing universities

      Hamp-Lyons, Liz; Wenxia, Bonnie Zhang; University of Bedfordshire; Tsinghua University (2019-07-10)
    • Developing assessment literacy

      Taylor, Lynda; University of Bedfordshire (2012-02-01)
    • Developing rubrics to assess the reading-into-writing skills: a case study

      Chan, Sathena Hiu Chong; Inoue, Chihiro; Taylor, Lynda; University of Bedfordshire (Elsevier Ltd, 2015-08-08)
      The integrated assessment of language skills, particularly reading-into-writing, is experiencing a renaissance. The use of rating rubrics, with verbal descriptors that describe quality of L2 writing performance, in large scale assessment is well-established. However, less attention has been directed towards the development of reading-into-writing rubrics. The task of identifying and evaluating the contribution of reading ability to the writing process and product so that it can be reflected in a set of rating criteria is not straightforward. This paper reports on a recent project to define the construct of reading-into-writing ability for designing a suite of integrated tasks at four proficiency levels, ranging from CEFR A2 to C1. The authors discuss how the processes of theoretical construct definition, together with empirical analyses of test taker performance, were used to underpin the development of rating rubrics for the reading-into-writing tests. Methodologies utilised in the project included questionnaire, expert panel judgement, group interview, automated textual analysis and analysis of rater reliability. Based on the results of three pilot studies, the effectiveness of the rating rubrics is discussed. The findings can inform decisions about how best to account for both the reading and writing dimensions of test taker performance in the rubrics descriptors.
    • Developing tools for learning oriented assessment of interactional competence: bridging theory and practice

      May, Lyn; Nakatsuhara, Fumiyo; Lam, Daniel M. K.; Galaczi, Evelina D. (SAGE Publications, 2019-10-01)
      In this paper we report on a project in which we developed tools to support the classroom assessment of learners’ interactional competence (IC) and provided learning oriented feedback in the context of preparation for a high-stakes face-to-face speaking test.  Six trained examiners provided stimulated verbal reports (n=72) on 12 paired interactions, focusing on interactional features of candidates’ performance. We thematically analyzed the verbal reports to inform a draft checklist and materials, which were then trialled by four experienced teachers. Informed by both data sources, the final product comprised (a) a detailed IC checklist with nine main categories and over 50 sub-categories, accompanying detailed description of each area and feedback to learners, which teachers can adapt to suit their teaching and testing contexts, and (b) a concise IC checklist with four categories and bite-sized feedback for real-time classroom assessment. IC, a key aspect of face-to-face communication, is under-researched and under-explored in second/foreign language teaching, learning, and assessment contexts. This in-depth treatment of it, therefore, stands to contribute to learning contexts through raising teachers’ and learners’ awareness of micro-level features of the construct, and to assessment contexts through developing a more comprehensive understanding of the construct.
    • Development of empirically driven checklists for learners’ interactional competence

      Nakatsuhara, Fumiyo; May, Lyn; Lam, Daniel M. K.; Galaczi, Evelina D.; University of Bedfordshire; Queensland University of Technology; Cambridge Assessment English (2019-03-27)
    • The discourse of the IELTS Speaking Test : interactional design and practice

      Seedhouse, Paul; Nakatsuhara, Fumiyo (Cambridge University Press, 2018-02-15)
      The volume provides a unique dual perspective on the evaluation of spoken discourse in that it combines a detailed portrayal of the design of a face-to-face speaking test with its actual implementation in interactional terms. Using many empirical extracts of interaction from authentic IELTS Speaking Tests, the book illustrates how the interaction is organised in relation to the institutional aim of ensuring valid assessment. The relationship between individual features of the interaction and grading criteria is examined in detail across a number of different performance levels.
    • Don't turn a deaf ear: a case for assessing interactive listening

      Lam, Daniel M. K.; ; University of Bedfordshire (Oxford University Press, 2021-01-11)
      The reciprocal nature of spoken interaction means that participants constantly alternate between speaker and listener roles. However, listener or recipient actions – also known as interactive listening (IL) – are somewhat underrepresented in language tests. In conventional listening tests, they are not directly assessed. In speaking tests, they have often been overshadowed by an emphasis on production features or subsumed under broader constructs such as interactional competence. This paper is an effort to represent the rich IL phenomena that can be found in peer interactive speaking assessments, where the candidate-candidate format and discussion task offer opportunities to elicit and assess IL. Taking a close look at candidate discourse and non-verbal actions through a conversation analytic approach, the analysis focuses on three IL features: 1) listenership displays, 2) contingent responses, and 3) collaborative completions, and unpacks their relative strength in evidencing listener understanding. This paper concludes by making a case for revisiting the role of interactive listening, calling for more explicit inclusion of IL in L2 assessment as well as pedagogy.
    • Effective feedback: the key to successful assessment for learning

      Boyd, Elaine; Green, Anthony; Hopfenbeck, Theres N.; Stobart, Gordon; University of Bedfordshire; University of Oxford; University College London (Oxford University Press, 2019-01-01)
      This paper focuses on the quality, process, and management of feedback, and how it can be used constructively to move learning forward.
    • The effects of extended planning time on candidates’ performance, processes and strategy use in the lecture listening-into-speaking tasks of the TOEFL iBT Test

      Inoue, Chihiro; Lam, Daniel M. K.; Educational Testing Service (Wiley, 2021-06-21)
      This study investigated the effects of two different planning time conditions (i.e., operational [20 s] and extended length [90 s]) for the lecture listening-into-speaking tasks of the TOEFL iBT® test for candidates at different proficiency levels. Seventy international students based in universities and language schools in the United Kingdom (35 at a lower level; 35 at a higher level) participated in the study. The effects of different lengths of planning time were examined in terms of (a) the scores given by ETS-certified raters; (b) the quality of the speaking performances characterized by accurately reproduced idea units and the measures of complexity, accuracy, and fluency; and (c) self-reported use of cognitive and metacognitive processes and strategies during listening, planning, and speaking. The results found neither a statistically significant main effect of the length of planning time nor an interaction between planning time and proficiency on the scores or on the quality of the speaking performance. There were several cognitive and metacognitive processes and strategies where significantly more engagement was reported under the extended planning time, which suggests enhanced cognitive validity of the task. However, the increased engagement in planning did not lead to any measurable improvement in the score. Therefore, in the interest of practicality, the results of this study provide justifications for the operational length of planning time for the lecture listening-into-speaking tasks in the speaking section of the TOEFL iBT test.
    • Effects of pre-task planning on paired oral test performance: a case of beginning EFL learners

      Nakatsuhara, Fumiyo; Nitta, Ryo; University of Bedfordshire; Nagoya Gakuin University (2010-09-11)
    • Effects of pre-task planning time on paired oral test performance

      Nakatsuhara, Fumiyo; Nitta, Ryo; University of Bedfordshire; Nagoya Gakuin University (2013-11-17)
    • The effects of single and double play upon listening test outcomes and cognitive processing

      Field, John; British Council (British Council, 2015-01-01)
      Report on a project investigating the effects of playing recorded material twice upon test taker scores and upon their behaviour
    • Effects of the number of participants on group oral test performance

      Nakatsuhara, Fumiyo; University of Bedfordshire (2009-11-22)
    • An empirical investigation of the componentiality of L2 reading in English for academic purposes

      Weir, Cyril J.; Yang, Huizhong; Jin, Yan (2001-01-01)
      The focus on careful reading in theoretical literature has meant that we have somewhat ignored reading behaviours such as skimming, scanning and search reading in the teaching and testing of reading. These theories do not explain how readers process texts quickly and selectively. The development of the Advanced English Reading Test (AERT) for university students in the People's Republic of China under the auspices of the National College English Test Committee was the vehicle for investigating the nature of and the relationships between skimming, scanning, search reading and careful reading. This volume reports on the methodological procedures that led to the development of this important test and discusses the results of the empirical investigations that were carried out to establish its validity. As such it offers a blueprint of reading assessment for those wishing to research in the area.
    • The English Benchmarking Study in Maltese Schools: Technical Report 2015

      Khabbazbashi, Nahal; Khalifa, Hanan; Robinson, M.; Ellis, S.; Cambridge English Language Assessment (Cambridge English Language Assessment, 2016-04-15)
      This is a report for a project between Cambridge English Language Assessment and the Maltese Ministry for Education and Employment [Nahal Khabbazbashi was principal investigator for project].
    • English language teacher development in a Russian university: context, problems and implications

      Rasskazova, Tatiana; Guzikova, Maria; Green, Anthony; Ural Federal University; University of Bedfordshire (Elsevier, 2017-02-02)
      The evaluation of teacher professional development efficiency has always been an issue that has attracted attention of professionals in education. This paper reports on the results of a two-year English language teacher professional development programme following a Needs Analysis study conducted by Cambridge ESOL in 2012. Longitudinal research shows that in Russia English language teaching has several problems which exist throughout decades. This article focuses on some of them: class interaction mode; the use of native (Russian) language in class; error correction strategies employed by teachers. A new approach to evaluation was employed by asking students and teachers the same questions from different perspectives on areas identified during the needs analysis study. The results varied in significance, though some positive changes have been noticed in class interaction mode, little has changed in the error correction strategies, the use of Russian in the classroom seems to be quite reasonable and does not interfere with learning. Overall, the study may be useful for general audience, especially for the post-Soviet countries as it provides evidence of change management and their impact on ELT. The findings presented in this paper seek to contribute to the formulation or adjustment of policies related to educational reforms, such as curriculum reform and teacher professional development in non-English-speaking countries.
    • Establishing test form and individual task comparability: a case study of a semi-direct speaking test

      Weir, Cyril J.; Wu, Jessica R.W.; University of Luton; Language Training and Testing Center, Taiwan (SAGE, 2006-04-01)
      Examination boards are often criticized for their failure to provide evidence of comparability across forms, and few such studies are publicly available. This study aims to investigate the extent to which three forms of the General English Proficiency Test Intermediate Speaking Test (GEPTS-I) are parallel in terms of two types of validity evidence: parallel-forms reliability and content validity. The three trial test forms, each containing three different task types (read-aloud, answering questions and picture description), were administered to 120 intermediate-level EFL learners in Taiwan. The performance data from the different test forms were analysed using classical procedures and Multi-Faceted Rasch Measurement (MFRM). Various checklists were also employed to compare the tasks in different forms qualitatively in terms of content. The results showed that all three test forms were statistically parallel overall and Forms 2 and 3 could also be considered parallel at the individual task level. Moreover, sources of variation to account for the variable difficulty of tasks in Form 1 were identified by the checklists. Results of the study provide insights for further improvement in parallel-form reliability of the GEPTS-I at the task level and offer a set of methodological procedures for other exam boards to consider. © 2006 Edward Arnold (Publishers) Ltd.