Skip to main navigation menu Skip to main content Skip to site footer
##common.pageHeaderLogo.altText##

Articles

Vol. 27 No. 3 (2024)

Trends of replication studies in Applied Linguistics journals: A systematic review over half a century

DOI
https://doi.org/10.37213/cjal.2024.33477
Submitted
June 7, 2023
Published
2025-05-30

Abstract

Despite the importance of replication research in scientific fields, very few replications are conducted in applied linguistics (AL). To enhance language researchers’ awareness of replications and provide a systematic evaluation of current replications, this study analyzed replication studies published in 92 AL leading journals from 1970 to 2021 based on five themes of replication labels, methodological orientations, research trends, authorship, and citation counts of replicators. The results reveal that replication labels have explicitly been mentioned since 2002, the replication of quantitative studies has predominately been raised, studies on second language acquisition were frequently replicated, collaborative authorship has increased in replications, and influential AL scholars tend to conduct replication research. The study highlights the need for a well-established replication classification and calls for replication research in the areas and methodological orientations marginalized in AL. It is also recommended that prominent figures perform more replication research to consolidate its status in AL.

References

  1. Abbuhl, R. (2012a). Practical methods for teaching replication to applied linguistics studies. In G. Porte (Ed.), Replication research in applied linguistics (pp. 135-150). Cambridge University Press.
  2. Abbuhl, R. (2012b). Why, when, and how to replicate research. In A. Mackey & S. Gass (Eds.), Research methods in second language acquisition (pp. 296-312). Wiley-Blackwell. https://doi.org/10.1002/9781444347340.ch15
  3. Abbuhl, R. (2018). Replication research. In A. Phakiti, A., P. De Costa, L. Plonsky, & S. Starfield (Eds.), The Palgrave handbook of applied linguistics research methodology (pp. 145-162). Palgrave MacMillan.
  4. Allwright, D. (2003). Exploratory practice: Rethinking practitioner research in language teaching. Language Teaching Research, 7(2), 113-141. https://doi.org/10.1191/1362168803lr118oa
  5. Amjad, T., Daud, A., & Aljohani, N. R. (2018). Ranking authors in academic social networks: A survey. Library Hi Tech, 36(1), 97-128. https://doi.org/10.1108/LHT-05-2017-0090
  6. Bochynska, A., Keeble, L., Halfacre, C. A. E., Casillas, J. V., Champagne, I., Chen, K., Röthlisberger, M., Buchanan, E. M., Roettger, T. B. (2023). Reproducible research practices and transparency across linguistics. MetaArXiv. https://doi.org/10.31222/osf.io/rcews
  7. Borry, P., Schotsmans, P., & Dierickx, K. (2006). Author, contributor or just a signer? A quantitative analysis of authorship trends in the field of bioethics. Bioethics, 20(4), 213-220. https://doi.org/10.1111/j.1467-8519.2006.00496.x
  8. Casanave, C. (2012). Heading in the wrong direction? A response to Porte and Richards. Journal of Second Language Writing, 21(3), 296-297. https://doi.org/1060374312000422
  9. Coretta, S., Casillas, J. V., Roessig, S., Franke, M., Ahn, B., Al-Hoorie, A. H., Al-Tamimi, J., Alotaibi, N. E., AlShakhori, M. K., Altmiller, R. M., Arantes, P., Athanasopoulou, A., Baese-Berk, M. M., Bailey, G., Sangma, Ch. B. A., Beier, E. J., Benavides, G. M., Benker, N., BensonMeyer, E. P., … & Roettger, T. B. (2023). Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses. Advances in Methods and Practices in Psychological Science, 6(2). https://doi.org/10.1177/25152459231162567
  10. Creswell, J. W. (2014). Research design. qualitative, quantitative and mixed method approaches (4th ed.). SAGE Publications.
  11. Del Ben, R., Brouillard, M., Gonzalez-Barrero, A.M., Killam, H., Kremin, L.V., Quirk, E., Sander-Montan, A., Schott., E., Tsui, R., & Byers-Heinlein, K. (2022). How open science can benefit bilingualism research: A lesson in six tales. Bilingualism: Language and Cognition,25(5), 913-920. https://doi.org/10.1017/S1366728922000256
  12. Denscombe, M. (2008). Communities of practice: A research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2(3), 270-283. https://doi.org/10.1177/1558689808316807
  13. Egbert, J. (2007). Quality analysis of journals in TESOL and applied linguistics. TESOL Quarterly, 41(1), 157-171. https://www.jstor.org/stable/40264335
  14. Exaly (2023). Applied Linguistics: Top Authors. Retrieved on May 9, 2023, from https://exaly.com/journal/19954/applied-linguistics/top-authors
  15. Gatbonton, E. (2000). Investigating experienced ESL teachers' pedagogical knowledge. Canadian Modern Language Review, 56(4), 585-616. https://doi.org/10.3138/cmlr.56.4.585
  16. Golden, M. (1995). Replication and non-quantitative research. Political Science and Politics, 28(3), 481-483. https://doi.org/10.2307/420313
  17. Hashemi, M. R., & Babaii, E. (2013). Mixed methods research: Toward new research designs in applied linguistics. The Modern Language Journal, 97(4), 828-852. https://doi.org/10.1111/j.1540-4781.2013.12049.x
  18. Heirene, R. M. (2021). A call for replications of addiction research: Which studies should we replicate and what constitutes a ‘successful’ replication? Addiction Research & Theory, 29(2), 89-97. https://doi.org/10.1080/16066359.2020.1751130
  19. Held, L., Micheloud, C., & Pawel, S. (2022). The assessment of replication success based on relative effect size. The Annals of Applied Statistics, 16(2), 706-720. https://doi.org/10.1214/21-AOAS1502
  20. Henriksen, D. (2018). What factors are associated with increasing co-authorship in the social sciences? A case study of Danish economics and political science. Scientometrics, 114(3), 1395-1421. https://doi.org/10.1007/s11192-017-2635-0
  21. Hiver, P., Al-Hoorie, A. H., Vitta, J. P., & Wu, J. (2021). Engagement in language learning: A systematic review of 20 years of research methods and definitions. Language Teaching Research, 28(1), 201-230. https://doi.org/10.1177/13621688211001289
  22. Hyland, K. (2015). Academic publishing: Issues and challenges in the construction of knowledge. Oxford University Press.
  23. Ioannidis, J. P., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4(3), 245-253. https://doi.org/10.1177%2F1740774507079441
  24. Irvine, E. (2021). The role of replication studies in theory building. Perspectives on Psychological Science, 16(4), 844-853. https://doi.org/10.1177/1745691620970558
  25. Isbell, D. R., Brown, D., Chen, M., Derrick, D. J., Ghanem, R., Gutierrez Arvizu, M. N., Schnur, E., Zhang, M., & Plonsky, L. (2022). Misconduct and questionable research practices: The ethics of quantitative data handling and reporting in applied linguistics. The Modern Language Journal. 106(1), 172-195. https://doi.org/10.1111/modl.12760
  26. John, L., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532. https://doi.org/10.1177%2F0956797611430953
  27. Jordan, G. (2005). Theory construction in second language acquisition. John Benjamins.
  28. Jung, U. O. H. (2004). Paris in London revisited or the foreign language teacher’s topmost journals. System, 32(3), 357-361.
  29. Kelly, C. D. (2019). Rate and success of study replication in ecology and evolution. PeerJ, 7. https://doi.org/10.7717/peerj.7654
  30. Khany, R., & Tazik, K. (2019). Levels of statistical use in applied linguistics research articles: From 1986-2015. Journal of Quantitative Linguistics, 26(1), 48-65. https://doi.org/10.1080/09296174.2017.1421498
  31. King, K., & Mackey, A. (2016). Research methodology in second language studies: Trends, concerns, and new directions. Modern Language Journal, 100(s1), 209-227. https://doi.org/10.1111/modl.12309
  32. Kobrock, K., & Roettger, T. B. (2023). Assessing the replication landscape in experimental linguistics. Glossa Psycholinguistics, 2(1). http://dx.doi.org/10.5070/G6011135
  33. Language Teaching Review Panel. (2008). Replication studies in language learning and teaching: Questions and answers. Language Teaching, 41(1), 1-14. https://doi.org/10.1017/S0261444807004727
  34. Larson-Hall, J., & Plonsky, L. (2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65(s1), 127-159. https://doi.org/10.1111/lang.12115
  35. Lykken, D. T. (1968). Statistical significance in psychological research. Psychological Bulletin, 70(3), 151-159. https://psycnet.apa.org/doi/10.1037/h0026141
  36. Mackey, A., & Marsden, E. (2016). Advancing methodology and practice: The IRIS repository of instruments for research into second languages. Routledge.
  37. Magnan, S.S. (2006). From the editor: The MLJ turns 90 in a digital age. The Modern Language Journal, 90(1), 1-5. https://www.jstor.org/stable/3588810
  38. Makel, M. C., Meyer, M. S., Simonsen, M. A., Roberts, A. M., & Plucker, J. A. (2022). Replication is relevant to qualitative research. Educational Research and Evaluation, 27(1-2), 215-219. https://doi.org/10.1080/13803611.2021.2022310
  39. Makel, M., & Plucker, J. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304-316. https://doi.org/10.3102/0013189X14545513
  40. Makel, M., Plucker, J., & Hegarty, B. (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 532-542. https://doi.org/10.1177%2F1745691612460688
  41. Markee, N. (2017). Are replication studies possible in qualitative second/foreign language classroom research? A call for comparative re-production research. Language Teaching, 50(3), 367-383. https://doi.org/10.1017/S0261444815000099
  42. Marsden, E. J. (2020). Methodological transparency in applied linguistics and its consequences for the quality and scope of research. In J. McKinley, & H. Rose (Eds.), Routledge handbook of research methods in applied linguistics (pp. 15-28). Routledge.
  43. Marsden, E., & Mackey, A. (2014). IRIS: A new resource for second language research. Linguistic Approaches to Bilingualism, 4(1), 125-130. https://doi.org/10.1075/lab.4.1.05mar
  44. Marsden, E., Mackey, A., & Plonsky, L. (2016). The IRIS repository: Advancing research practice and methodology. In A. Mackey, & E. Marsden (Eds.), Advancing methodology and practice: The IRIS repository of instruments for research into second languages (pp. 1-21). Routledge. https://doi.org/10.4324/9780203489666
  45. Marsden, E., & Morgan-Short, K. (2023) (Why) are open research practices the future for the study of language learning? Language Learning, 73(s2), 344-387. https://doi.org/10.1111/lang.12568
  46. Marsden, E., Morgan-Short, K., Thompson, S., & Abugaber, D. (2018). Replication in second language research: Narrative and systematic reviews and recommendations for the field. Language Learning, 68(2), 321-391. https://doi.org/10.1111/lang.12286
  47. Matsuda, P. (2012). On the nature of second language writing: Replication in a postmodern field. Journal of Second Language Writing, 21(3), 300-302. http://dx.doi.org/10.1016%2Fj.jslw.2012.05.006
  48. McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276-282. https://pmc.ncbi.nlm.nih.gov/articles/PMC3900052/
  49. McManus, K (2021). Are replication studies infrequent because of negative attitudes? Insights from a survey of attitudes and practices in second language research. Studies in Second Language Acquisition, 44(5), 1410-1423 https://doi.org/10.1017/S0272263121000838
  50. Moreau, D., & Wiebels, K. (2023). Ten simple rules for designing and conducting undergraduate replication projects. PLoS Computer Biology, 19(3). https://doi.org/10.1371/journal.pcbi.1010957
  51. Mullock, B. (2006). The pedagogical knowledge base of four TESOL teachers. The Modern Language Journal, 90(1), 48-66. https://doi.org/10.1111/j.1540-4781.2006.00384.x
  52. Nassaji, H. (2012). Significance tests and generalizability of research results: A case for replication. In G. Porte (Ed.), Replication research in applied linguistics (pp. 92-115). Cambridge University Press.
  53. Nassaji, H. (2021). Effect sizes in quantitative and qualitative research. Language Teaching Research, 25(5), 681-684. https://doi.org/10.1177%2F13621688211040882
  54. Nosek, B.A., & Errington, T. M. (2020). The best time to argue about what a replication means? Before you do it. Nature, 583, 518-520. https://doi.org/10.1038/d41586-020-02142-6
  55. Pashler, H., & Harris, C. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531-536. https://doi.org/10.1177%2F1745691612463401
  56. Plonsky, L. (2023). Sampling and generalizability in Lx research: A second-order synthesis. Languages, 8(1), 75. https://doi.org/10.3390/languages8010075
  57. Polio, C. (2012). Replication in published applied linguistics research: A historical perspective. In G. Porte (Ed.), Replication research in applied linguistics (pp. 47-91). Cambridge University Press.
  58. Polio, C., & Gass, S. (1997). Replication and reporting: A commentary. Studies in Second Language Acquisition, 19(4), 499-508. https://www.jstor.org/stable/44487991
  59. Popper, K. R. (1959). The logic of scientific discovery. Hutchinson.
  60. Porte, G. (2012). Replication research in applied linguistics. Cambridge University Press.
  61. Porte, G., & McManus, K. (2019). Doing replication research in applied linguistics. Routledge. https://doi.org/10.4324/9781315621395
  62. Porte, G., & Richards, K. (2012). Focus article: Replication in second language writing research. Journal of Second Language Writing, 21(3), 284-293. https://doi.org/10.1016/j.jslw.2012.05.002
  63. Riazi, A. M., Ghanbar, H., & Fazel, I. (2020). The contexts, theoretical and methodological orientation of EAP research: Evidence from empirical articles published in the Journal of English for Academic Purposes. Journal of English for Academic Purposes, 48. https://doi.org/10.1016/j.jeap.2020.100925
  64. Robinson, L. M., & Adler, R. D. (2004). Business research in eight business disciplines. Journal of College Teaching & Learning, 1(2), 69-80. https://doi.org/10.19030/tlc.v1i2.1916
  65. Santos, T. (1989). Replication in applied linguistics research. TESOL Quarterly, 23(4), 699-702. https://doi.org/10.2307/3587548
  66. Sasaki, M. (2012). An alternative approach to replication studies in second language writing: An ecological perspective. Journal of Second Language Writing, 21(3), 303-305. https://doi.org/10.1016/j.jslw.2012.05.004
  67. Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90-100. https://doi.org/10.1037/a0015108
  68. Schofield, J. (2002). Increasing the generalizability of qualitative research. In A. Huberman, & M. Miles (Eds.), The qualitative researcher’s companion (pp. 171-203). SAGE. https://doi.org/10.4135/9781412986274
  69. Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., Fong, N., … & Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337-356. https://doi.org/10.1177/2515245917747646
  70. Sönning, L., & Werner, V. (2021). The replication crisis, scientific revolutions, and linguistics. Linguistics, 59(5), 1179-1206. https://doi.org/10.1515/ling-2019-0045
  71. Stapleton, P., & Shao, Q. (2017). Research in language teaching over two decades: A retrospective of the first 20 volumes of Language Teaching Research. Language Teaching Research, 22(3), 350-369. https://doi.org/10.1177/1362168816688929
  72. Vandergrift, L., & Cross, J. (2017). Replication research in L2 listening comprehension: A conceptual replication of Graham & Macaro (2008) and an approximate replication of Vandergrift & Tafaghodtari (2010) and Brett (1997). Language Teaching, 50(1), 80-89. https://doi.org/10.1017/S026144481500004X
  73. Weber, M., & Campbell, C. M. (2004). In other professional journals. Modern Language Journal, 88(1), 130-139. https://doi.org/10.1111/0026-7902.00010
  74. Zou, D., Luo, S., Xie, H., & Hwang, G. (2020). A systematic review of research on flipped language classrooms: Theoretical foundations, learning activities, tools, research topics and findings. Computer Assisted Language Learning, 35(8) 1811-1837. https://doi.org/10.1080/09588221.2020.1839502