Development and Usability Evaluation of an AI-Powered English Vocabulary Learning Application

Joseflim Marchel; Dendy  Kurniawan; Azed Yayah Durrotun Nihayah

doi:10.51903/jtie.v5i2.383

Authors

Joseflim Marchel Dept. of Computer Science. Faculty of Academic Study. Universitas Sains dan Teknologi Komputer, Semarang, Indonesia https://orcid.org/0009-0005-3850-5842
Dendy Kurniawan Dept. of Computer Science. Faculty of Academic Study. Universitas Sains dan Teknologi Komputer, Semarang, Indonesia https://orcid.org/0009-0009-2667-7136
Azed Yayah Durrotun Nihayah Dept. of Information System. Faculty of Information Technology. Satya Wacana Christian University. Salatiga. Indonesia https://orcid.org/0009-0000-2388-1160

DOI:

https://doi.org/10.51903/jtie.v5i2.383

Keywords:

Adaptive Learning Architecture, Artificial Intelligence, Computer-Assisted Language Learning, Human-Computer Interaction, System Usability Scale

Abstract

Learning vocabulary is a major challenge for beginner foreign-language learners, as repetition-based methods can be tiring and often lead to rapid forgetting. Generative AI, particularly Large Language Models (LLMs), enables the automatic creation of large amounts of learning content. However, without appropriate pedagogical guidance, AI-generated materials may be inconsistent and provide limited support for effective learning progress. This study therefore developed and evaluated an adaptive application for beginner-level English vocabulary practice. The application uses an LLM to provide contextual assistance and combines it with a modified spaced repetition mechanism. Its vocabulary and learning activities follow the appropriate CEFR level, while the difficulty is adjusted based on the accuracy of each learner’s answers. The application was evaluated through a mixed-methods usability study involving 72 learners with beginner-level English proficiency. User experience and acceptance were measured using the System Usability Scale (SUS) and selected measures from the Technology Acceptance Model (TAM). Usability was rated highly. Although the participants were beginners with limited English, they generally found the interface comfortable and manageable. Their SUS responses produced a mean of 81.46 (SD = 7.18), a result classified as “Excellent.” They also viewed the application as useful and relatively easy to operate. The respective scores were M = 4.38 (SD = 0.42) for Perceived Usefulness and M = 4.29 (SD = 0.45) for Perceived Ease of Use.

References

Bahari, A., Han, F., & Strzelecki, A. (2025). Integrating CALL and AIALL for an interactive pedagogical model of language learning. Education and Information Technologies 2025 30:10, 30(10), 14305–14333. https://doi.org/10.1007/S10639-025-13388-W

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An Empirical Evaluation of the System Usability Scale. Intl. Journal of Human–Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776

Bangor, A., Kortum, P. T., & Miller, J. T. (2009). Determining What Individual SUS Scores Mean: Adding an Adjective Rating Scale. Journal of Usability Studies, 4, 3-. https://doi.org/10.1145/3472749.3474738

Brooke, J. (1996). Usability Evaluation In Industry - Google Books: 189.3. Usability evaluation in industry.

Chandy, R., Serrano, R., & Pellicer-Sánchez, A. (2024). The effect of context on the processing and learning of novel L2 vocabulary while reading. Applied Psycholinguistics, 45(6), 1086–1113. https://doi.org/10.1017/S0142716424000407

Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008

Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 379–383. https://doi.org/10.3758/BF03195514

Ganesan, P., Binti Abdul Aziz, A., & Ismail, H. H. (2025). Comparing Rote Memorization and Contextual Learning in Vocabulary Acquisition among Upper Primary ESL Students. International Journal of Academic Research in Progressive Education and Development, 14(2). https://doi.org/10.6007/IJARPED/v14-i2/25165

Ge, W., Sun, Y., Wang, Z., Zheng, H., He, W., Wang, P., Zhu, Q., Wang, B., & Wang, P.-H. (2025). SRLAgent: Enhancing Self-Regulated Learning Skills through Gamification and LLM Assistance. Proceedings of Preprint, 1.

Gkintoni, E., Antonopoulou, H., Sortwell, A., & Halkiopoulos, C. (2025). Challenging Cognitive Load Theory: The Role of Educational Neuroscience and Artificial Intelligence in Redefining Learning Efficacy. Brain Sciences, 15(2), 203. https://doi.org/10.3390/brainsci15020203

Karataş, F., Abedi, F. Y., Ozek Gunyel, F., Karadeniz, D., & Kuzgun, Y. (2024). Incorporating AI in foreign language education: An investigation into ChatGPT’s effect on foreign language learners. Education and Information Technologies, 29(15), 19343–19366. https://doi.org/10.1007/s10639-024-12574-6

Krieglstein, F., Beege, M., Rey, G. D., Sanchez-Stockhammer, C., & Schneider, S. (2023). Development and Validation of a Theory-Based Questionnaire to Measure Different Types of Cognitive Load. Educational Psychology Review, 35(1), 9. https://doi.org/10.1007/s10648-023-09738-0

Lee, J.-Y., & Baek, M. (2023). Effects of Gamification on Students’ English Language Proficiency: A Meta-Analysis on Research in South Korea. Sustainability, 15(14), 11325. https://doi.org/10.3390/su151411325

Mannekote, A., Davies, A., Pinto, J. D., Zhang, S., Olds, D., Schroeder, N. L., Lehman, B., Zapata-Rivera, D., & Zhai, C. (2024). Large language models for whole-learner support: opportunities and challenges. Frontiers in Artificial Intelligence, 7. https://doi.org/10.3389/frai.2024.1460364

Naseer, F., Khan, M. N., Addas, A., Awais, Q., & Ayub, N. (2025). Game Mechanics and Artificial Intelligence Personalization: A Framework for Adaptive Learning Systems. Education Sciences, 15(3), 301. https://doi.org/10.3390/EDUCSCI15030301/S1

Nielsen, J. (1994). Usability Engineering | Enhanced Reader. Morgan Kaufmann Publishers Inc.

Sayed, W. S., Noeman, A. M., Abdellatif, A., Abdelrazek, M., Badawy, M. G., Hamed, A., & El-Tantawy, S. (2023). AI-based adaptive personalized content presentation and exercises navigation for an effective and engaging E-learning platform. Multimedia Tools and Applications, 82(3), 3303–3333. https://doi.org/10.1007/s11042-022-13076-8

Shank, E., Tang, H., & Morris, W. (2025). Motivation in online course design using self-determination theory: an action research study in a secondary mathematics course. Educational Technology Research and Development, 73(1), 415–441. https://doi.org/10.1007/s11423-024-10410-9

Shortt, M., Tilak, S., Kuznetcova, I., Martens, B., & Akinkuolie, B. (2023). Gamification in mobile-assisted language learning: a systematic review of Duolingo literature from public release of 2012 to early 2020. Computer Assisted Language Learning, 36(3), 517–554. https://doi.org/10.1080/09588221.2021.1933540

Skulmowski, A., & Xu, K. M. (2022). Understanding Cognitive Load in Digital and Online Learning: a New Perspective on Extraneous Cognitive Load. Educational Psychology Review, 34(1), 171–196. https://doi.org/10.1007/s10648-021-09624-7

Sweller, J. (2024). Cognitive load theory and individual differences. Learning and Individual Differences, 110, 102423. https://doi.org/10.1016/j.lindif.2024.102423

van den Broek, G. S. E., Wesseling, E., Huijssen, L., Lettink, M., & van Gog, T. (2022). Vocabulary Learning During Reading: Benefits of Contextual Inferences Versus Retrieval Opportunities. Cognitive Science, 46(4), e13135. https://doi.org/10.1111/COGS.13135

Venkatesh, V., & Bala, H. (2008). Technology Acceptance Model 3 and a Research Agenda on Interventions. Decision Sciences, 39(2), 273–315. https://doi.org/10.1111/j.1540-5915.2008.00192.x

Wang, S., Xu, T., Li, H., Zhang, C., Liang, J., Tang, J., Yu, P. S., & Wen, Q. (2024). Large Language Models for Education: A Survey and Outlook. IEEE Signal Processing Magazine, 42(6), 51–63. https://doi.org/10.1109/MSP.2025.3594309

Wang, X., & Reynolds, B. L. (2024). Beyond the Books: Exploring Factors Shaping Chinese English Learners’ Engagement with Large Language Models for Vocabulary Learning. Education Sciences, 14(5), 496. https://doi.org/10.3390/educsci14050496

Wei, L. (2023). Artificial intelligence in language instruction: impact on English learning achievement, L2 motivation, and self-regulated learning. Frontiers in Psychology, 14. https://doi.org/10.3389/fpsyg.2023.1261955

Zhang, Z., & Huang, X. (2024). Exploring the impact of the adaptive gamified assessment on learners in blended learning. Education and Information Technologies, 29(16), 21869–21889. https://doi.org/10.1007/s10639-024-12708-w

Zhao, Y. (2025). The Impact of GPT-based Dialogue Systems vs. Traditional Word-list Memorization on Second Language Vocabulary Acquisition. Lecture Notes in Education Psychology and Public Media, 106(1), 41–50. https://doi.org/10.54254/2753-7048/2025.CB25093