• Assessing the portfolio principles for practice, theory and research

      Hamp-Lyons, Liz; Con, William (Hampton Press, 2000-01-01)
      This volume deals comprehensively and systematically with the subject of portfolio-based writing assessment. This single source fully explores the theory behind using portfolios in a writing program as well as information about what portfolios are, what advantages they hold for assessment purposes, and what effects they can have on a writing program. The book provides a more comprehensive look at what portfolio assessment has become and can become, focusing not on an individual program, but on the full spectrum of portfolio assessment as it has evolved so far.
    • Comparing writing proficiency assessments used in professional medical registration: a methodology to inform policy and practice

      Chan, Sathena Hiu Chong; Taylor, Lynda; University of Bedfordshire (Elsevier, 2020-10-13)
      Internationally trained doctors wishing to register and practise in an English-speaking country typically have to demonstrate that they can communicate effectively in English, including writing proficiency. Various English language proficiency (ELP) tests are available worldwide and are used for such licensing purposes. This means that medical registration bodies face the question of which test(s) will meet their needs, ideally reflecting the demands of their professional environment. This article reports a mixed-methods study to survey the policy and practice of health-care registration organisations in the UK and worldwide. The study aimed to identify ELP tests that were, or could be, considered as suitable for medical registration purposes and to understand the differences between them. The paper discusses what the study revealed about the function and comparability of different writing tests used in professional registration as well as the complex criteria a professional body may prioritise when selecting a test. Although the original study was completed in 2015, the paper takes account of subsequent changes in policy and practice. It offers a practical methodology and worked example which may be of interest and value to other researchers, language testers and policymakers as they face challenges in selecting and making comparisons across tests.
    • Demonstrating the cognitive validity and face validity of PTE Academic Writing items Summarize Written Text and Write Essay

      Chan, Sathena Hiu Chong (Pearson, 2011-07-01)
      This study examines the cognitive validity of two item types of the Writing Section of the PTE Academic test – Summarize Written Text and Write Essay - within Weir’s (2005) socio-cognitive framework for test validation. The study focuses on cognitive validity by investigating and comparing the cognitive processes of a group of ESL test takers undertaking Summarize Written Text (an integrated writing item) and Write Essay (an independent writing item). Cognitive validity is a ‘measure of how closely it [a writing task] represents the cognitive processing involved in writing contexts beyond the test itself’ (Shaw and Weir, 2007:34). In addition, the study investigates test takers’ opinions regarding the two different writing item types: independent and integrated. Test takers’ scores on both items are compared to investigate if the two performances correlate. The study uses screen capture technique to record test takers’ successive writing processes on both items, followed by retrospective stimulated recalls. The findings demonstrate that Summarize Written Text and Write Essay engage different cognitive processes that are essential in academic writing contexts. In particular, macro-planning and discourse synthesis processes such as selecting relevant ideas from source text are elicited by the Summarize Written Text item whereas processes in micro-planning, monitoring and revising at low levels are activated on the Write Essay item. In terms of test performances, the results show that test takers in this study performed significantly better on Write Essay than on Summarize Written Text.
    • Developing rubrics to assess the reading-into-writing skills: a case study

      Chan, Sathena Hiu Chong; Inoue, Chihiro; Taylor, Lynda; University of Bedfordshire (Elsevier Ltd, 2015-08-08)
      The integrated assessment of language skills, particularly reading-into-writing, is experiencing a renaissance. The use of rating rubrics, with verbal descriptors that describe quality of L2 writing performance, in large scale assessment is well-established. However, less attention has been directed towards the development of reading-into-writing rubrics. The task of identifying and evaluating the contribution of reading ability to the writing process and product so that it can be reflected in a set of rating criteria is not straightforward. This paper reports on a recent project to define the construct of reading-into-writing ability for designing a suite of integrated tasks at four proficiency levels, ranging from CEFR A2 to C1. The authors discuss how the processes of theoretical construct definition, together with empirical analyses of test taker performance, were used to underpin the development of rating rubrics for the reading-into-writing tests. Methodologies utilised in the project included questionnaire, expert panel judgement, group interview, automated textual analysis and analysis of rater reliability. Based on the results of three pilot studies, the effectiveness of the rating rubrics is discussed. The findings can inform decisions about how best to account for both the reading and writing dimensions of test taker performance in the rubrics descriptors.
    • Investigating the cognitive constructs measured by the Aptis writing test in the Japanese context: a case study

      Moore, Yumiko; Chan, Sathena Hiu Chong; British Council (the British Council, 2018-11-30)
      This study investigates the context and cognitive validity of the Aptis General Writing Part 4 Tasks. An online survey with almost 50 Japanese universities was conducted to investigate the nature of the predominant academic writing in the wider context. Twenty-five Year 1 academic writing tasks were then sampled from a single Japanese university. Regarding the context validity of the Aptis test, online survey and expert judgement were used to examine the degree of correspondence between the task features of the Aptis task and those of the target academic writing tasks in real life. Regarding its cognitive validity, this study examined the cognitive processes elicited by the Aptis task as compared to the Year 1 writing tasks through a cognitive process questionnaire (n=35) and interviews with seven students and two lecturers. The overall resemblance between the test and the real-life tasks reported in this study supports the context and cognitive validity of the Aptis Writing test Part 4 in the Japanese context. The overall task setting (topic domain, cognitive demands and language function to be performed) of the Aptis test resembles that of the real-life tasks. Aptis Writing test Part 4 tasks, on the other hand, outperformed the sampled real-life tasks in terms of clarity of writing purpose, knowledge of criteria and intended readerships. However, when considering the wider Japanese academic context, a wider range of academic genres, such as summary and report, and some more demanding language functions such as synthesis, should also be represented in the Aptis Writing test. The results show that all target processes in each cognitive phase (conceptualisation, meaning and discourse construction, organising, low-level monitoring and revising, and high-level monitoring and revising) were reported by a reasonable percentage of the participants. Considering the comparatively lower proficiency in English of Japanese students and their unfamiliarity of direct writing assessment, the results are encouraging. However, some sub-processes such as linking important ideas and revising appear to be under-represented in Aptis. In addition, the lack of time management and typing skills of some participants appear to hinder them from spending appropriate time planning, organising, and revising at low and high levels. Recommendations are provided to address these issues.
    • Opposing tensions of local and international standards for EAP writing programmes: who are we assessing for?

      Bruce, Emma; Hamp-Lyons, Liz; City University of Hong Kong; University of Bedfordshire (Elsevier Ltd, 2015-04-24)
      In response to recent curriculum changes in secondary schools in Hong Kong including the implementation of the 3-3-4 education structure, with one year less at high school and one year more at university and the introduction of a new school leavers' exam, the Hong Kong Diploma of Secondary Education (HKDSE), universities in the territory have revisited their English language curriculums. At City University a new EAP curriculum and assessment framework was developed to fit the re-defined needs of the new cohort of students.In this paper we describe the development and benchmarking process of a scoring instrument for EAP writing assessment at City University. We discuss the opposing tensions of local (HKDSE) and international (CEFR and IELTS) standards, the problems of aligning EAP needs-based domain scales and standards with the CEFR and the issues associated with attempting to fulfil the institutional expectation that the EAP programme would raise students' scores by a whole CEFR scale step. Finally, we consider the political tensions created by the use of external, even international, reference points for specific levels of writing performance from all our students and suggest the benefits of a specific, locally-designed, fit-for-purpose tool over one aligned with universal standards.
    • Reflecting on the past, embracing the future

      Hamp-Lyons, Liz; University of Bedfordshire (Elsevier, 2019-10-14)
      In the Call for Papers for this anniversary volume of Assessing Writing, the Editors described the goal as “to trace the evolution of ideas, questions, and concerns that are key to our field, to explain their relevance in the present, and to look forward by exploring how these might be addressed in the future” and they asked me to contribute my thoughts. As the Editor of Assessing Writing between 2002 and 2017—a fifteen-year period—I realised from the outset that this was a very ambitious goal, l, one that no single paper could accomplish. Nevertheless, it seemed to me an opportunity to reflect on my own experiences as Editor, and through some of those experiences, offer a small insight into what this journal has done (and not done) to contribute to the debate about the “ideas, questions and concerns”; but also, to suggest some areas that would benefit from more questioning and thinking in the future. Despite the challenges of the task, I am very grateful to current Editors Martin East and David Slomp for the opportunity to reflect on these 25 years and to view them, in part, through the lens provided by the five articles appearing in this anniversary volume.
    • Researching participants taking IELTS Academic Writing Task 2 (AWT2) in paper mode and in computer mode in terms of score equivalence, cognitive validity and other factors

      Chan, Sathena Hiu Chong; Bax, Stephen; Weir, Cyril J. (British Council and IDP: IELTS Australia, 2017-08-01)
      Computer-based (CB) assessment is becoming more common in most university disciplines, and international language testing bodies now routinely use computers for many areas of English language assessment. Given that, in the near future, IELTS also will need to move towards offering CB options alongside traditional paper-based (PB) modes, the research reported here prepares for that possibility, building on research carried out some years ago which investigated the statistical comparability of the IELTS writing test between the two delivery modes, and offering a fresh look at the relevant issues. By means of questionnaire and interviews, the current study investigates the extent to which 153 test-takers’ cognitive processes, while completing IELTS Academic Writing in PB mode and in CB mode, compare with the real-world cognitive processes of students completing academic writing at university. A major contribution of our study is its use – for the first time in the academic literature – of data from research into cognitive processes within real-world academic settings as a comparison with cognitive processing during academic writing under test conditions. The most important conclusion from the study is that according to the 5-facet MFRM analysis, there were no significant differences in the scores awarded by two independent raters for candidates’ performances on the tests taken under two conditions, one paper-and-pencil and the other computer. Regarding analytic scores criteria, the differences in three areas (i.e. Task Achievement, Coherence and Cohesion, and Grammatical Range and Accuracy) were not significant, but the difference reported in Lexical Resources was significant, if slight. In summary, the difference of scores between the two modes is at an acceptable level. With respect to the cognitive processes students employ in performing under the two conditions of the test, results of the Cognitive Process Questionnaire (CPQ) survey indicate a similar pattern between the cognitive processes involved in writing on a computer and writing with paper-and-pencil. There were no noticeable major differences in the general tendency of the mean of each questionnaire item reported on the two test modes. In summary, the cognitive processes were employed in a similar fashion under the two delivery conditions. Based on the interview data (n=30), it appears that the participants reported using most of the processes in a similar way between the two modes. Nevertheless, a few potential differences indicated by the interview data might be worth further investigation in future studies. The Computer Familiarity Questionnaire survey shows that these students in general are familiar with computer usage and their overall reactions towards working with a computer are positive. Multiple regression analysis, used to find out if computer familiarity had any effect on students’ performances on the two modes, suggested that test-takers who do not have a suitable familiarity profile might perform slightly worse than those who do, in computer mode. In summary, the research offered in this report offers a unique comparison with realworld academic writing, and presents a significant contribution to the research base which IELTS and comparable international testing bodies will need to consider, if they are to introduce CB test versions in future.
    • Researching the comparability of paper-based and computer-based delivery in a high-stakes writing test

      Chan, Sathena Hiu Chong; Bax, Stephen; Weir, Cyril J. (Elsevier, 2018-04-07)
      International language testing bodies are now moving rapidly towards using computers for many areas of English language assessment, despite the fact that research on comparability with paper-based assessment is still relatively limited in key areas. This study contributes to the debate by researching the comparability of a highstakes EAP writing test (IELTS) in two delivery modes, paper-based (PB) and computer-based (CB). The study investigated 153 test takers' performances and their cognitive processes on IELTS Academic Writing Task 2 in the two modes, and the possible effect of computer familiarity on their test scores. Many-Facet Rasch Measurement (MFRM) was used to examine the difference in test takers' scores between the two modes, in relation to their overall and analytic scores. By means of questionnaires and interviews, we investigated the cognitive processes students employed under the two conditions of the test. A major contribution of our study is its use - for the first time in the computer-based writing assessment literature - of data from research into cognitive processes within realworld academic settings as a comparison with cognitive processing during academic writing under test conditions. In summary, this study offers important new insights into academic writing assessment in computer mode.
    • Scaling and scheming: the highs and lows of scoring writing

      Green, Anthony; University of Bedfordshire (2019-12-04)
    • Using keystroke logging to understand writers’ processes on a reading-into-writing test

      Chan, Sathena Hiu Chong (Springer Open, 2017-05-19)
      Background Integrated reading-into-writing tasks are increasingly used in large-scale language proficiency tests. Such tasks are said to possess higher authenticity as they reflect real-life writing conditions better than independent, writing-only tasks. However, to effectively define the reading-into-writing construct, more empirical evidence regarding how writers compose from sources both in real-life and under test conditions is urgently needed. Most previous process studies used think aloud or questionnaire to collect evidence. These methods rely on participants’ perceptions of their processes, as well as their ability to report them. Findings This paper reports on a small-scale experimental study to explore writers’ processes on a reading-into-writing test by employing keystroke logging. Two L2 postgraduates completed an argumentative essay on computer. Their text production processes were captured by a keystroke logging programme. Students were also interviewed to provide additional information. Keystroke logging like most computing tools provides a range of measures. The study examined the students’ reading-into-writing processes by analysing a selection of the keystroke logging measures in conjunction with students’ final texts and interview protocols. Conclusions The results suggest that the nature of the writers’ reading-into-writing processes might have a major influence on the writer’s final performance. Recommendations for future process studies are provided.
    • Validating performance on writing test tasks

      Weir, Cyril J.; University of Bedfordshire (2013-07-11)
    • Washback and writing assessment

      Green, Anthony; University of Bedfordshire (2012-03-15)