RSS 1.0English language learning and assessment
Recent Submissions
-
Assessment of writingIn the dynamic landscape of the 21st century, writing remains an indispensable skill, serving as a powerful conduit for expression, communication, and documentation. With the exponential growth of digital communication platforms, the written word has transcended traditional boundaries. With the continuous evolution of writing, assessment of writing plays a more pivotal role than ever in ensuring that writing tests measure the new construct of writing with improved reliability and validity. This chapter begins with an introduction of the purposes of writing tests, followed by a brief history of assessment of L2 writing. Next follows a discussion of the major considerations of a writing test in relation to task features, the nature of writing processes and scoring. Finally, there is a discussion on the challenges and opportunities for the future of assessment of writing.
-
Exploring EMI students’ attitudes towards translanguaging and English language proficiency threshold across different disciplinesThe current study explores the discipline-based differences in terms of the relationship between English language proficiency and attitudes towards translanguaging in partial English Medium Instruction (EMI) programmes at a Turkish university. Quantitative data were collected from undergraduates in the Faculty of Engineering (n = 173) and the Faculty of Economics and Administrative Sciences (n = 172). Analyses revealed that English proficiency did not predict students’ general attitudes towards translanguaging and their attitudes towards the use of translanguaging in class in engineering programmes. However, proficiency was a significant predictor of both factors for the social science students. MANOVA results showed that the multivariate effect of proficiency was significant in social science programmes but not in engineering programmes. An apparent proficiency threshold was observed with social science students but not with engineering students, where attitudes towards the use of translanguaging decreased at the B2 proficiency level. The paper discusses the pedagogical implications of the relationship between language proficiency and attitudes towards translanguaging in different disciplines.
-
Integrating metadiscourse analysis with transformer-based models for enhancing construct representation and discourse competence assessment in L2 writing: a systemic multidisciplinary approachIn recent years, large-scale language test providers have developed or adapted automated essay scoring systems (AESS) to score L2 writing essays. While the benefits of using AESS are clear, they are not without limitations, such as over-reliance on frequency counts of vocabulary and grammar variables. Discourse competence is one important aspect of L2 writing yet to be fully explored in AEE application. Evidence of discourse competence can be seen in the use of Metadiscourse Markers (MDM) to produce reader-friendly texts. The article presents a multidisciplinary study to explore the feasibility of expanding the construct representation of automated scoring models to assess discourse competence in L2 writing. Combining machine learning, automated textual analysis and corpus-linguistic methods to examine 2000 scripts across two tasks and five proficiency levels, the study investigates (1) in addition to frequency and range, whether accuracy of MDM is worth pursuing as a predictive feature in L2 writing, and (2) how identification and classification of MDM use might be fed into developing an automated scoring model using machine learning techniques. The contributions of this study are three-fold. Firstly, it offers valuable insights within the context of Explainable AI. By integrating MDM usage and accuracy into the scoring framework, this research moves beyond frequency-based evaluation. This study also makes significant contributions to the current understanding of L2 writing development that even lower-proficiency learners exhibit evidence of discourse competence through their accurate use of MDMs as well as their choice of MDMs in response to genre. From the perspective of expanding the construct representation in automated scoring systems, this study provides a critical examination of the limitations of many AEE models, which have heavily relied on vocabulary and grammar features. By exploring the feasibility of incorporating MDMs as predictive features, this research demonstrates the potential for construct expansion of L2 AEE. The results would support test providers in developing competence tests in various contexts and domains including manufacturing, medicine and so on.
-
Less talk, more action? exploring proficiency scores and embodied resources in online L2 interactionsThis mixed-methods study explores Embodied Resources (ERs) in an online, triadic task involving 52 candidates within the Common European Framework of Reference for Languages (CEFR) B2˗C1 range of language proficiency. Mean comparisons and correlations were used to explore the relationship between ER use and proficiency level. An applied Conversation Analysis approach was used to explicate specific sequences of conversational repair where ERs were evident. Findings suggest no significant correlation between proficiency and ER frequency but support the view that ERs are co-ordinated with verbal resources. This raises questions as to how language testers might approach the interplay of ERs and interactional practices.
-
Video-conferencing speaking tests: an investigation of context validity related to test administrationFace-to-face speaking assessment provides the benefit of eliciting a broad interactional construct, but at the cost of being logistically complex, resource-intensive and difficult to manage. Advances in video-conferencing (VC) technology now make it possible to engage in online interaction more successfully than previously, thus reducing dependence upon physical proximity between the examiner-interlocutor and the candidate(s). It is therefore not surprising that such technology is seen as a valuable assessment tool in geographically remote and politically unstable areas of the world, or indeed in contexts affected by the social distancing required during the recent Covid-19 pandemic. However, the administrative conditions under which the test takes place, one of the key contextual parameters of the VC-delivered test, , is often overlooked, despite its potentially significant influence on candidates’ performance and therefore overall test validity (Weir 2005). In this chapter, we report on investigations into
-
Language learning motivation and the ESOL exam system in EnglandEnglish for Speakers of Other Languages (ESOL) courses in England are accessed by a wide range of learners, choosing to do so for a variety of reasons. Understanding the learning goals of these students is important for every teacher in this context, despite a presumption that people will have an inherent need to learn the language of their host nation. This short report presents the results of a pilot study aimed at identifying reasons adults enrol at the (pre)intermediate level, using self-determination theory as a basis of analysis. Data were collected through a paper-based survey and follow-up focus group interviews with learners on both accredited and non-accredited ESOL courses. The findings suggest that the majority of learners at this level are motivated by long-term career plans, but motivation can be undermined by exam registration decisions. The report concludes with a discussion of the individual differences of adult learners and a call for further research in this area.
-
Reflexive participant collaboration: researching ‘with’ adult migrant learners of EnglishThis article explores a method of collecting in-situ data created by adult migrants attending English language classes to investigate the interindividual variance of vulnerability within this population. The study encourages reflexivity to evaluate and redress the power relationships present both in the research process and the daily lives of the migrants, whilst also building trust between the researcher and participants. I argue that weekly text message data, generated by participants and arranged as graphs for online interviews, empowers participants to take ownership of their research contribution and encourages deeper reflections to share the stories behind the messages.
-
Yet another example of ChatGPT's evasive tactics during long conversations: Japanese rock song lyrics caseMuch attention has been devoted to the ChatGPT and other Large Language Models’ (LLM) capability assessment regarding syntax correctness, factual accuracy, adequate world representation, ethical alignment, common sense and formal logic reasoning. However, most of the research focused on "statically" generated texts, when the result of only a single iteration between a human and LLMs was recorded. More advanced techniques of open-ended discussions or debates between a human and LLMs produce much more interesting results, demonstrating such faulty rhetorical behaviours as circular arguments, self-contradictions, evasion, change of topic, lack of consistent position, and the mix of passive aggression with attempts to please human disputant. We present an original observation of such behaviour during the ChatGPT dialogue session discussing the translation of Japanese song lyrics.
-
Use of keystroke logging to collect cognitive validity evidence for integrated writing testsIntegrated writing tasks are commonly used for teaching, learning and assessment purposes in most higher education contexts. These tasks are cognitively demanding as they require students to transform knowledge by engaging in processes of discourse synthesis, i.e. selecting, organising, and connecting information from multiple source texts into a new or synthesis text. The purpose of the present exploratory study was to investigate L2 writers’ discourse synthesis processes underlying the performance of an integrated reading-writing task. The participants were three university students who completed an integrated reading-writing task as part of a postadmission academic literacy test at a British university. Data were collected using a variety of qualitative research techniques: analysis of keystroke logs, retrospective interviews, and text quality analysis. Data analysis revealed distinct engagement in discourse synthesis processes among L2 writers. The study proposes a qualitative approach to analysing keystroke logging data to collect cognitive validity evidence (i.e. test takers’ engagement in discourse synthesis) underlying integrated writing test performance. The other major implications of the findings are the need for explicit teaching and assessment of these discourse synthesis processes, i.e. selecting, connecting and organising relevant ideas from multiple reading stimuli to produce a text, and the need to construct specific rating descriptors which reflect skills of discourse synthesis for integrated writing tasks.
-
Differences between L2 listening and readingWith technology being an increasingly important presence in modern life, children, adolescents and L2 learners are exposed to more and more digital materials, such as audio books, interactive posters with sound files and videos, and TED talks, in classrooms and daily life. These digital audio-visual materials are increasingly becoming a major source of information and learning (Khabbazbashi, Chan, & Clark, 2022). Herring (2019) argues that education is now operating within a communication paradigm that is “fundamentally multimodal”. The affordances of new digital platforms (e.g., Google classroom, Zoom, Microsoft Teams) mean that L2 learners can now more easily collaborate with their peers to complete group work at home. Such a shift means that L2 listening comprehension is playing a more prominent role in social and educational contexts. Nevertheless, it has not received as much attention as reading comprehension has in second language acquisition, assessment, and pedagogical research, especially in relation to the processes involved in L2 listening (Field, 2008, 2013). Furthermore, listening comprehension is often conflated with reading comprehension and operationalized in a similar way in pedagogical and assessment practice (van Zeeland & Schmitt, 2013). To contribute to the discussion of how the processes of L2 reading and listening comprehension differ, this chapter provides an overview of cognitive models of L2 reading and listening, and discusses how input modality may affect the process of comprehension, followed by a discussion of the differences between L2 reading and listening. Based on the account of the nature of L2 reading and listening, the chapter will discuss the implications for task design by contrasting some key characteristics in reading and listening texts and their impact on comprehension.
-
The one with all the sarcasm: Pauline Madella discusses the pragmatics of Chandler Bing’s sarcastic humourIt is precisely this subjectivity – what Perry called Chandler’s “world-weary yet witty view of life”, his sharing of impressions, emotions, feelings, attitudes and moods – that is “descriptively ineffable”. In those instances where Chandler does not commit to one single interpretation, but rather offers a diffuse impression, it is hard to pin down exactly what it is that is being expressed. And yet we understand his intentions. In order to understand, we attend to a number of nonverbal as well as verbal cues and do a great amount of inferential reconstruction. In my research, I urge language practitioners (specifically second language practitioners) to expose language learners to instances of ineffable communication, where attention to nonverbal stimuli is key to understanding the speaker’s intended effects. When I am asked how I learned English as a second language, I often say “By watching Friends”. The tragic passing of Matthew Perry on 28 October 2023 was followed by an outpouring of tributes from his friends and fans across the globe. They remembered him as a comedy legend who was able to nail a complex mix of timing, pace, cadence and emphasis, while bringing joy – and belly-aching laughter – to millions. Beyond its contribution to Linguistics and my sharing of a long-held passion for Pragmatics with language lovers, this article is meant as a tribute to Matty Perry and the iconic Chandler Muriel Bing.
-
Ask a linguist: experts answer your questions: "What exactly is contrastive stress in English?”It is not what you say but how you say it. In spoken English, the intonation contours of an utterance (also called prosody) can greatly affect the meaning that the speaker conveys. Contrastive stress is often described as the most conspicuous and ubiquitous prosodic phenomenon in English (you may also see it called contrastive focus, contrastive accent or prosodic contrastive focus). Contrastive stress is used to draw the addressee’s attention to a particular constituent in an utterance – one that is not typically accented – and, in doing so, it triggers a particular interpretation of the utterance. Its acoustic salience or extra ‘oomph’ is characterised by greater auditory prominence and articulatory care, loudness, and increased intensity.
-
Exploring the dynamic relationship between Dr. GEPT feedback and learners’ L2 motivationFeedback is an important means to bridge assessment and learning, but its usefulness ultimately depends on whether and how learners engage with and act on the feedback. Learners’ L2 learning motivation may interact with feedback in meaningful and consequential ways, yet there is relatively little research to date that explores such a dynamic relationship, particularly among language learners in secondary education. This study aimed to fill this gap by exploring the relationship between learners’ motivation and assessment feedback offered by Dr. GEPT – automated personalised feedback provided to GEPT each test-taker alongside their test scores, including an analysis of their strengths and weaknesses, learning advice, and vocabulary and sentence patterns for further study. Taking a mixed-methods approach, Phase 1 of this study involved a large-scale questionnaire survey (n = 635) to explore L2 motivation among senior high school learners of English in Taiwan and their general perceptions towards assessment feedback. The questionnaire was developed based on the L2 Motivational Self System model (Dörnyei, 2005, 2009). Phase 2 used learning logs (n = 14) and interviews (n = 10) for an in-depth qualitative inquiry into how learners engaged with Dr. GEPT feedback and how the feedback might have shaped the developments in learners’ learning journeys. The report concludes with a discussion of how Dr. GEPT helps learners develop a positive orientation towards assessments and cultivates learner autonomy, as well as making some suggestions for enhancing the effectiveness of Dr. GEPT feedback.
-
Exploring the speaking construct in academic settings in a digital ageThis study explored language functions and skills utilised in technology-mediated academic speaking contexts, which is timely given the increasing prevalence of digitally-mediated communication in higher education settings and the recent introduction of IELTS Indicator featuring a video-call mode in the Speaking Test. Using an embedded mixed-methods approach, the research involved: 1. language function analysis of spoken communication and simultaneous written chat contributions in online taught classes and supervision meetings 2. thematic analysis of students’ and lecturers’ understandings of distinctive features of online academic speaking and what constitutes successful online speaking interaction in those contexts. We analysed a total of over 40 hours of recordings, consisting of 17 video-recorded classes from four undergraduate and postgraduate units in an Australian University, and 23 video/audio recordings of online PhD supervision meetings from a UK university. This was followed by the administration of a questionnaire and semi-structured interviews with selected participants. In order to examine the construct of online academic communication, we adapted O’Sullivan et al.’s (2002) language function checklist for our purposes. Following the identification of language functions and skills observed in real-life online academic settings, we explored the synergy between the functions observed in online teaching and learning contexts and those elicited in the video-call IELTS Speaking Test (Nakatsuhara et al., 2021). Analyses of questionnaire and interview data helped us understand the skills perceived to be important for successful online interaction. The report concludes with a discussion on the multimodal construct of speaking in digitally-mediated academic contexts and the ways in which the findings of this study can be useful in informing the future development of IELTS Speaking Test tasks so that they remain representative of the reality of academic speaking in the digital age.
-
Language assessment accommodations: issues and challenges for the future [editorial]In this concluding piece to the special issue, we attempt to tease out and comment on some themes that have emerged from the six published papers. Some of these themes highlight potential avenues for further theoretical and empirical investigation, and may assist in mapping out a coherent research agenda on the topic for language testers and assessment specialists in the future.
-
Relevance and multimodal prosody: implications for L2 teaching and learningIn this paper, I build on Scott's relevance-theoretic account of contrastive stress (2021). Contrastive stress works as an extra cue to ostension in altering the salience of a particular constituent in an utterance and, as a result, the salience of one particular interpretation of that utterance. I draw on Scott’s argument that contrastive stress does not encode procedural meaning. Contrastive stress is unpredictable and, as such, it is in confounding the hearer’s expectations that it draws his attention to the accented word and prompt his search for different interpretive effects. I argue that contrastive stress is interpreted purely inferentially precisely because it is one of many pointing devices. It is to be interpreted by virtue of its interaction with other paralinguistic behaviours, all of which being different aspects of the same ostensive act of communication. This leads me to focus on the gestural nature of contrastive stress working as an act of pointing, which, as an ostensive communicative behaviour, conveys that if you look over there, you’ll know what I mean (Tomasello et al., 2007). Finally, I present the implications of analysing contrastive stress in its multimodal context – as prosodic pointing – for the teaching and learning of L2 prosodic pragmatics and the development of interpretive abilities in the L2 hearer’s mind.
-
Exploring open consonantal environments for at-home testing of vowel perception in advanced L2 speakersRecent work has called for increased investigation into methods used to explore second language (L2) speech perception (Flege 2021). The present study attends to this call, examining a common practice for developing listening prompts in the context of at-home administrations. Vowel perception studies have historically used fixed consonantal frames to determine how well participants can discriminate between target L2 vowels, and the present study compares the effects of employing a fixed consonant-vowel-consonant frame (h-vowel-d) with open (phonologically diverse) consonantal environments using real and nonce words. Thirty-eight Mandarin (n = 31) and English (n = 8) first language speakers participated in a listening experiment and a post-experiment question. Data were framed within Best and Tyler’s (2007) Perceptual Assimilation Model-L2. Internal consistency and proportion correct were calculated and a generalised linear mixed model design was used to investigate how well performance with h-vowel-d prompts predicts performance with the more diverse prompt types. Results suggest an inflation of scores for the fixed frame prompt and support the use of diverse words for listening prompt designs. Findings have implications for vowel perception researchers as well as computer (and mobile) assisted language learning developers wishing to inform their designs with relevant empirical evidence.
-
Non-verbal communication and context: multi-modality in interactionTraditionally, the study of linguistics has focussed on verbal communication. In the sense that linguistics is the scientific study of language, the approach is perfectly justified. Those working in the sub-discipline of linguistic pragmatics, however, are faced with something of a dilemma. The aim of a pragmatic theory is to explain how utterances are understood, and utterances, of course, have both linguistic and non-linguistic properties. As well as this, current work in pragmatics emphasizes that the affective dimension of a speaker’s meaning is at least as important as the cognitive one and it is often the non-linguistic properties of utterances that convey information relating to this dimension. This paper highlights the major role of non-verbal ‘modes’ of communication (‘multi-modality’) in accounting for how meaning is achieved and explores in particular how the quasi-musical contours we impose on the words we say, as well as the movements of our face and hands that accompany speech, constrain the context and guide the hearer to our intended meaning. We build on previous exploration of the relevance of prosody (Wilson and Wharton 2006) and, crucially, looks at prosody in relation to other non-verbal communicative behaviours from the perspective of relevance theory. In-so-doing, we also hope to shed light on the role of multimodality in both context construction and utterance interpretation and suggest prosody needs to be analysed as one tool in a set of broader gestural ones (Bolinger 1983). Relevance theory is an inferential model, in which human communication revolves around the expression and recognition of the speaker’s intentions in the performance of an ostensive stimulus: an act accompanied by the appropriate combination of intentions. This inferential model is proposed as a replacement for the traditional code-model of communication, according to which a speaker simply encodes into a signal the thought they wish to communicate and the hearer retrieves their meaning by decoding the signal they have provided. We will argue that much existing work on multimodality remains rooted in a code model and show how adopting an inferential model enables us to integrate multimodal behaviours more completely within a theory of utterance interpretation. As ostensive stimuli, utterances are composites of a range of different behaviours, each working together to form a range of contextual cues.
-
L2 writing assessment: an evolutionary perspectiveThis book tackles three choices that face developers of L2 writing assessments: defining L2 writing abilities; collecting evidence of those abilities (usually by getting L2 writers to write something); and judging their performance (usually by assigning a score or grade to it). It takes a historical view of how assessment developers have made those choices, how contemporary practices emerged, and of alternative techniques that have risen and fallen over time. The three sections each tackle one of these choices. The first considers the social functions that define L2 writing and assessment; the second relates how assessment tasks have adapted to changing conceptions of languages, writing, and assessment; and the third explores how scoring systems have evolved. Each section brings the reader up to date with current issues confronting writing assessment (both in large-scale testing and in language classrooms) before considering the new opportunities and challenges of the digital age. This book will be of interest to students, scholars and practitioners in language assessment, language education, and applied linguistics.
