Objective Assessment of Students' Interpreting Performance: An Experimental Study

Document Type : Original Article

Authors

1 Nahavand Higher Education Complex, Bu-Ali Sina University, Iran

2 Leiden University, The Netherlands and University of Pannonia, Hungary

Abstract

The traditional metric of interpreting quality is a score given by human professional judges focusing on the interpreters’ performance. However, there is a poor agreement on what constitutes an acceptable interpretation. This study investigates the objective assessment of interpreter trainees’ performance. Two groups of 15 student interpreters were formed. Participants were assigned to groups at random, but with equal division between genders (seven males in each group). The control group was taught interpreting skills by the routine curriculum, while the experimental group spent part of the time instead on theoretical explanation and practical exercises emphasizing prosodic differences between Persian and English. Three raters assessed the quality of the interpreter trainees’ performance in a post-test. Then the interpreting performance of the students was assessed objectively through Praat software. The results show that the intersubjective ratings of the students’ interpreting performance can be adequately predicted from objective measures through multiple linear regression. These results have implications for designers of curricula for training interpreters, and material producers in interpreting education.

Keywords


Language Learning, 28(1), 69–83. https://doi.org/10.1111/j.1467-1770.1978.tb00305.x
Boersma, P. & Weenink, D. J. M. (1996). Praat, a system for doing phonetics by computer, version 3.4 Report 132, Institute of Phonetic Sciences University of Amsterdam. www.praat.org
Buhmann, J., Caspers, J., Heuven, V. J. van, Hoekstra, H., Martens, J.-P., & Swerts, M. (2002). Annotation of prominent words, prosodic boundaries and segmental lengthening by no-expert transcribers in the spoken Dutch corpus. Proceedings of LREC 2002. Paris: ELRA, 779−785.
http://lrec-conf.org/proceedings/lrec2002/pdf/96.pdf
Campbell, S., & Hale, S. (2003). Translation and interpreting assessment in the context of educational measurement. In G. Anderman & M. Rogers (Eds.), Translation today: Trends and perspectives (pp. 205–224). Clevedon: Multilingual Matters.
Cutler, A. (2012). Native listening: Language experience and the recognition of spoken words. MIT Press.
Dawson, M., & Schell, A. (1987). Human autonomic and skeletal classical conditioning: the role of conscious cognitive factors. In G. Davey (Ed.), Cognitive processes and Pavlovian conditioning in humans (pp. 27–55). John Wiley & Sons.
Derwing, T. M., Munro, M. J., & Wiebe, G. E. (1998). Evidence in favour of a broad framework for pronunciation instruction. Language Learning, 48(3), 393–410. https://doi.org/10.1111/0023-8333.00047
Etemadi, S. H., & Abbasian, G. R. (2023). Dynamic assessment and EFL learners’ writing journey: Focus on DA modalities and writing revision types. Teaching English Language, 17(1), 53–79.
https://doi.org/10.22132/TEL.2022.162923
Garzone, G. (2002). Quality and norms in interpretation. In G. Garzone & M. Viezzi (Eds.), Interpreting in the 21st Century (pp. 107–119). Amsterdam/ Philadelphia: John Benjamins. https://doi.org/10.1075/btl.43.11gar
Grosjean, F., Grosjean, L., & Lane, H. (1979). The patterns of silence: performance structures in sentence production. Cognitive Psychology, 11, 58−81. https://doi.org/10.1016/0010-0285(79)90004-5
Gile, D. (1995). Fidelity assessment in consecutive interpretation: An experiment. Target, 7(1), 151–164. https://doi.org/10.1075/target.7.1.12gil
Gile, D. (2005). Empirical research into the role of knowledge in interpreting: Methodological aspects. In H. V. Dam, J. Engberg & H. Gerzymisch-Arbogast (Eds.), Knowledge systems in translation (pp. 149–171). Berlin/New York: Mouton de Gruyter.
Heuven, V. J. van (1994). Introducing prosodic phonetics. In C. Odé & V. J. van Heuven (Eds.), Experimental studies of Indonesian prosody. Semaian 9. Leiden: Vakgroep Talen en Culturen van Zuidoost-Azië en Oceanië, Leiden University. https://hdl.handle.net/1887/2574
Heuven, V. J. van (2008). Making sense of strange sounds: (Mutual) intelligibility of related language varieties. A review. International Journal of Humanities and Arts Computing, 2, 39–62.
https://doi.org/10.3366/E1753854809000305
Heuven, V. J. van (2017). Prosody and sentence type in Dutch. Nederlandse Taalkunde, 22, 3–29, 43–46. DOI: 10.5117/NEDTAA2017.1.HEUV
Heuven, V. J. van (2018). Notes on the phonetics of word and sentence stress: A cross-linguistic (re-)view. In H. van der Hulst, J. Heinz & R. Goedemans (Eds.), The study of word stress and accent: Theories, methods and data (pp. 13–59). Cambridge University Press. https://doi.org/ 10.1017/9781316683101.002
Heuven, V. J. van (2022). Resolving the prosody paradox. In P. Arantes & A. Post da Silveira (Eds.), Prosody and Bilingualism (pp. 168–204). Araraquara: Letraria. https://www.letraria.net/prosodia-e-bilinguismo/
Heuven, V. J. van, & Sluijter, A. M. C. (1996). Notes on the phonetics of word prosody. In R. Goedemans, H. van der Hulst & E. Visch (Eds.), Stress pat­terns of the world, Part 1: Background (pp. 233–269), HIL Publications (volume 2), The Hague: Holland Academic Graphics.
Jones, R. (2014). Conference interpreting explained. Routledge.
Jong, N. de, & Perfetti, C. A. (2011). Fluency training in the ESL classroom: An experimental study of fluency development and proceduralization. Language Learning61(2), 533−568.
https://doi.org/https://doi.org/10.1111/j.1467-9922.2010.00620.x
Khodashenas, M. R., Khodabakhshzadeh, H., Baghaei, P. & Motallebzadeh, K. (2023). Language assessment literacy components; now and then: A case of Iranian EFL head teachers. Teaching English Language, 17(1), 107−138. https://doi.org/10.22132/TEL.2022.163654
Koiso, H., Horiuchi, Y., Tutiya, S., Ichikawa, A., & Den, Y. (1998). An analysis of turn-taking and backchannels based on prosodic and syntactic features in Japanese Map Task dialogs. Language and Speech, 41(3−4), 295−321. https://doi.org/10.1177/002383099804100404
Kopczynski, A. (1994). Quality in conference interpreting: Some pragmatic problems. In S. Lambert & B. Moser-Mercer (Eds.), Bridging the gap: Empirical research into simultaneous interpretation (pp. 87–99). Amsterdam/Philadelphia: John Benjamins. https://doi.org/10.1075/btl.2.24kop
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3 ed., pp. 13–104). Washington, DC: Macmillan.
Nooteboom, S. G. (1997). The prosody of speech: Melody and rhythm. In W. J. Hardcastle & J. Laver (Eds.), The Handbook of Phonetic Sciences (pp. 640–673). Oxford: Basil Blackwell.
O’Neal, G. (2010). The effects of the presence and absence of suprasegmental on the intelligibility and assessment of an expanding-circle English according to other expanding-circle English listeners. JAIRO (Japanese Institutional Repositories Online).
Pöchhacker, F. (2004). Introducing Interpreting Studies. London & New York: Routledge.
Rutherford, W., & M. Sharwood Smith (1985). Consciousness-raising and universal grammar. Applied Linguistics, 6(3), 274–282.
https://doi.org/10.1093/applin/6.3.274
Sawyer, D. B. (2004). Fundamental Aspects of Interpreter Education: Curriculum and Assessment. John Benjamins.
Schmidt, R. (2010). Attention, awareness, and individual differences in language learning. In W. M. Chan, S. Chi, K. N. Cin, J. Istanto, M. Nagami, J. W. Sew, T. Suthiwan, & I. Walker (Eds.), Proceedings of CLaSIC 2010, Singapore, December 2-4 (pp. 721–737). Singapore: National University of Singapore, Centre for Language Studies. https://doi.org/10.1515/9781614510932.27
Selkirk, E. O. (1984). Phonology and syntax. The relation between sound and structure. Cambridge, MA: MIT Press.
Swerts, M. (1997). Prosodic features at discourse boundaries of different strength. Journal of the Acoustical Society of America, 101(1), 514−521. https://doi.org/10.1121/1.418114
Whalley, K., & Hansen, J. (2006). The role of prosodic sensitivity in children’s reading development. Journal of Research in Reading, 29(3), 288303. https://doi.org/10.1111/j.1467-9817.2006.00309.x
Wu, S. C. (2010). Assessing simultaneous interpreting. PhD Thesis, School of Modern Languages, Newcastle University.
https://theses.ncl.ac.uk/jspui/bitstream/10443/1122/1/Wu%2011.pdf
Yenkimaleki, M. (2018). Implicit vs. explicit prosody teaching in developing listening comprehension skills by interpreter trainees: an experimental study. International Journal of English Language and Linguistics Research, 6, 11–21.
Yenkimaleki, M., & Heuven, V. J. van (2016). Effect of prosody awareness training on the performance of consecutive interpretation from Farsi into English: An experimental study. Asia Pacific Translation and Intercultural Studies, 3(3), 235–251. https://doi.org/ 10.1080/23306343.2016.1233930.
Yenkimaleki, M., & Heuven, V. J. van (2017). The effect of memory training on consecutive interpreting performance by interpreter trainees: An experimental study. FORUM: International Journal of Interpretation and Translation, 157–172. DOI: 10.1075/forum.15.1.09yen.
Yenkimaleki, M., & Heuven, V. J. van (2018). The effect of teaching prosody teaching on interpreting performance: An experimental study of consecutive interpreting from English into Farsi. Perspectives: Studies in Translatology, 26, 84–99. https://doi.org/0.1080/0907676X.2017.1315824.
Yenkimaleki, M., & Heuven, V. J. van (2019a). Effects of prosody awareness training on the intelligibility of Iranian interpreter trainees in English. Dutch Journal of Applied Linguistics, 8(2), 291–309. https://doi.org/10.1075/dujal.17023.yen
Yenkimaleki, M., & Heuven, V. J. van (2019b). The relative contribution of computer assisted prosody training vs. instructor-based prosody teaching in developing speaking skills by interpreter trainees: an experimental study. Speech Communication, 107, 48–57.
DOI: 10.1016/j.specom.2019.01.006
Yenkimaleki, M., & Heuven, V. J. van (2019c). Prosody instruction for interpreter trainees: does methodology make a difference? An experimental study. Across languages and cultures: A Multidisciplinary Journal for Translation and Interpreting Studies, 20(1), 117–133. DOI: 10.1556/084.2019.20.1.6
Yenkimaleki, M. & Heuven, V. J. van (2020).  Relative contribution of explicit teaching of segmentals vs. prosody to the quality of consecutive interpreting by Farsi-to-English interpreting trainees. Interactive Learning Enviornments, xx, 1–12. https://doi.org/10.1080/10494820.2020.1789673
Yenkimaleki, M., Heuven, V. J. van, & Moradimokhes, H. (2021).  The effect of prosody instruction in developing listening comprehension skills by interpreter trainees: does methodology matter? Computer Assisted Language Learning, xx, 1-37. https://doi.org/10.1080/09588221.2021.1957942
 
Yenkimaleki, M., & Heuven, V. J. van (2021). Effects of attention to segmental vs. suprasegmental features on the speech intelligibility and comprehensibility of the EFL learners targeting the perception or production-focused practice. System, 100, 1–12. https://doi.org/10.1016/j.system.2021.102557
Yenkimaleki, M., & Heuven, V. J. van (2022). Comparing the nativeness vs. intelligibility approach in prosody instruction for developing speaking skills by interpreter trainees: An experimental study. Speech Communication, 137, 92–102. https://doi.org/ 10.1016/j.specom.2022.01.007
Yenkimaleki M., Heuven V. J. van, & Afshar H. S. (2022), The efficacy of segmental/suprasegmental vs. holistic pronunciation instruction on the development of listening comprehension skills by EFL learners. The Language Learning Journal, xx, 1–17. https://doi.org/ 10.1080/09571736.2022.2073382
Yu, W., & Heuven, V. J. van (2017). Predicting judged fluency of consecutive interpreting from acoustic measures: Potential for automatic assessment and pedagogy. Interpreting: International Journal of Research and Practice in Interpreting, 19(1), 47–68. https://doi.org/10.1075.intp.19.1.03yu
Yu, W., & Heuven, V. J. van (2021). Quantitative correlates as predictors of judged fluency in consecutive interpreting: Implications for automatic assessment and pedagogy. In J. Chen & C. Han (Eds.), Testing and assessment of interpreting (pp. 117–142). Berlin, Heidelberg: Springer Nature. https://doi.org/10.1007/978-981-15-8554-8_6
Zhang, X. (2016). Semi-automatic simultaneous interpreting quality evaluation. International Journal on Natural Language Computing, 5, 1–12. https://doi.org/10.48550/arXiv.1611.04052