Use of keystroke logging to collect cognitive validity evidence for integrated writing tests

Abstract

Integrated writing tasks are commonly used for teaching, learning and assessment purposes in most higher education contexts. These tasks are cognitively demanding as they require students to transform knowledge by engaging in processes of discourse synthesis, i.e. selecting, organising, and connecting information from multiple source texts into a new or synthesis text. The purpose of the present exploratory study was to investigate L2 writers’ discourse synthesis processes underlying the performance of an integrated reading-writing task. The participants were three university students who completed an integrated reading-writing task as part of a postadmission academic literacy test at a British university. Data were collected using a variety of qualitative research techniques: analysis of keystroke logs, retrospective interviews, and text quality analysis. Data analysis revealed distinct engagement in discourse synthesis processes among L2 writers. The study proposes a qualitative approach to analysing keystroke logging data to collect cognitive validity evidence (i.e. test takers’ engagement in discourse synthesis) underlying integrated writing test performance. The other major implications of the findings are the need for explicit teaching and assessment of these discourse synthesis processes, i.e. selecting, connecting and organising relevant ideas from multiple reading stimuli to produce a text, and the need to construct specific rating descriptors which reflect skills of discourse synthesis for integrated writing tasks.