Investigating the cognitive constructs measured by the Aptis writing test in the Japanese context: a case study

Abstract

This study investigates the context and cognitive validity of the Aptis General Writing Part 4 Tasks. An online survey with almost 50 Japanese universities was conducted to investigate the nature of the predominant academic writing in the wider context. Twenty-five Year 1 academic writing tasks were then sampled from a single Japanese university. Regarding the context validity of the Aptis test, online survey and expert judgement were used to examine the degree of correspondence between the task features of the Aptis task and those of the target academic writing tasks in real life. Regarding its cognitive validity, this study examined the cognitive processes elicited by the Aptis task as compared to the Year 1 writing tasks through a cognitive process questionnaire (n=35) and interviews with seven students and two lecturers. The overall resemblance between the test and the real-life tasks reported in this study supports the context and cognitive validity of the Aptis Writing test Part 4 in the Japanese context. The overall task setting (topic domain, cognitive demands and language function to be performed) of the Aptis test resembles that of the real-life tasks. Aptis Writing test Part 4 tasks, on the other hand, outperformed the sampled real-life tasks in terms of clarity of writing purpose, knowledge of criteria and intended readerships. However, when considering the wider Japanese academic context, a wider range of academic genres, such as summary and report, and some more demanding language functions such as synthesis, should also be represented in the Aptis Writing test. The results show that all target processes in each cognitive phase (conceptualisation, meaning and discourse construction, organising, low-level monitoring and revising, and high-level monitoring and revising) were reported by a reasonable percentage of the participants. Considering the comparatively lower proficiency in English of Japanese students and their unfamiliarity of direct writing assessment, the results are encouraging. However, some sub-processes such as linking important ideas and revising appear to be under-represented in Aptis. In addition, the lack of time management and typing skills of some participants appear to hinder them from spending appropriate time planning, organising, and revising at low and high levels. Recommendations are provided to address these issues.