Mind the gap: comparing anticipated and realized perceptions of an Artificial Intelligence (AI) chatbot for lecture comprehension support among Business Administration students

Ahmed Samet

doi:10.14198/ijd.32888

Authors

Ahmed Samet University of Applied Sciences Bremerhaven, Germany https://orcid.org/0009-0001-4642-2706

DOI:

https://doi.org/10.14198/ijd.32888

Keywords:

artificial intelligence (AI) in education, chatbots, technology acceptance, pre-test/post-tes, student perceptions, pilot study, user experience, higher education, generative artificial intelligence (GenAI), educational technology adoption

Abstract

The rapid integration of artificial intelligence (AI) in higher education has sparked growing interest in conversational chatbots as tools to support student learning. Yet little is known about how students' initial expectations align with their actual experiences of interacting with purpose-built educational chatbots. This pilot study examines the expectation-experience gap in student perceptions of an AI-powered chatbot specifically designed to support lecture comprehension and provide on-demand explanations. Thirty third-semester Business Administration students at a German university of applied sciences participated in a mixed-methods pre-post design. Quantitative survey data were collected via two questionnaires administered before and after a semester-long chatbot intervention, drawing on the Technology Acceptance Model (TAM), the Unified Theory of Acceptance and Use of Technology (UTAUT), trust development theories, and expectation-experience gap analysis. Pre-implementation data revealed modest trust (M = 2.9/5, SD = 0.72) and notable accuracy concerns (80% of students). Post-implementation results showed substantial trust growth (M = 5.8/7, SD = 0.84), high continuance intention (M = 6.1/7, SD = 0.61), and strong productivity gains (M = 6.1/7, SD = 0.74), with the chatbot's Socratic pedagogical approach rated as its most valued feature (M = 4.75/5, SD = 0.31). These findings indicate that a domain-specific, pedagogically sophisticated chatbot can successfully overcome initial skepticism and deliver meaningful learning support. Educationally, the results support the integration of AI tools designed around guided inquiry as a complement to rather than replacement for human instruction. Methodological limitations, including the small sample size and single-institution design, constrain generalizability; future work should employ inferential statistical testing and larger, more diverse samples.

References

Adamopoulou, E., & Moussiades, L. (2020). An overview of Chatbot technology. In IFIP advances in information and communication technology (pp. 373-383). https://doi.org/10.1007/978-3-030-49186-4_31

Al Ka'bi, A. (2023). Proposed artificial intelligence algorithm and deep learning techniques for development of higher education. International Journal of Intelligent Networks, 4, 68-73. https://doi.org/10.1016/j.ijin.2023.03.002

AlAfnan, M. A., Dishari, S., Jovic, M., & Lomidze, K. (2023). Chatgpt as an educational tool: Opportunities, challenges, and recommendations for communication, business writing, and composition courses. Journal of Artificial Intelligence and Technology, 3(2), 60-68. https://doi.org/10.37965/jait.2023.0184

Alsanousi, B., Albesher, A. S., Do, H., & Ludi, S. (2023). Investigating the user experience and evaluating usability issues in AI-Enabled Learning mobile Apps: An analysis of User reviews. International Journal of Advanced Computer Science and Applications, 14(6). https://doi.org/10.14569/IJACSA.2023.0140602

AlZu'bi, S., Mughaid, A., Quiam, F., & Hendawi, S. (2024). Exploring the capabilities and limitations of ChatGPT and alternative big language models. Artificial Intelligence and Applications, 2(1), 28-37. https://doi.org/10.47852/bonviewAIA3202820

Araujo, T. (2018). Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions. Computers in Human Behavior, 85, 183-189. https://doi.org/10.1016/j.chb.2018.03.051

Baidoo-anu, D., & Owusu Ansah, L. (2023). Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Journal of AI, 7(1), 52-62. https://doi.org/10.61969/jai.1337500

Benvenuti, M., Cangelosi, A., Weinberger, A., Mazzoni, E., Benassi, M., Barbaresi, M., & Orsoni, M. (2023). Artificial intelligence and human behavioral development: A perspective on new skills and competencies acquisition for the educational context. Computers in Human Behavior, 148, 107903. https://doi.org/10.1016/j.chb.2023.107903

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa

Chaves, A.P., & Gerosa, M.A. (2019). How Should My Chatbot Interact? A Survey on Social Characteristics in Human-Chatbot Interaction Design. International Journal of Human-Computer Interaction, 37, 729 - 758. https://doi.org/10.1080/10447318.2020.1841438

Chen, Y., Jensen, S., Albert, L. J., Gupta, S., & Lee, T. (2023). Artificial intelligence (AI) student assistants in the classroom: Designing chatbots to support student success. Information Systems Frontiers, 25(1), 161-182. https://doi.org/10.1007/s10796-022-10291-4

Cooper, G. (2023). Examining science education in chatgpt: An exploratory study of generative artificial intelligence. Journal of Science Education and Technology, 32(3), 444-452. https://doi.org/10.1007/s10956-023-10039-y

Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20(1), 1-22. https://doi.org/10.1186/s41239-023-00392-8

Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), 319-340. https://doi.org/10.2307/249008

Deng, X., & Yu, Z. (2023). A meta-analysis and systematic review of the effect of chatbot technology use in sustainable education. Sustainability, 15(4), 2940. https://doi.org/10.3390/su15042940

Essel, H. B., Vlachopoulos, D., Tachie-Menson, A., Johnson, E. E., & Baah, P. K. (2022). The impact of a virtual teaching assistant (chatbot) on students' learning in Ghanaian higher education. International Journal of Educational Technology in Higher Education, 19(1), 1-19. https://doi.org/10.1186/s41239-022-00362-6

Følstad, A., Nordheim, C. B., & Bjørkli, C. A. (2018). What Makes Users Trust a Chatbot for Customer Service? An Exploratory Interview Study. Internet Science, 11193, 194-208. https://doi.org/10.1007/978-3-030-01437-7_16

Fryer, L. K., Nakao, K., & Thompson, A. (2019). Chatbot learning partners: Connecting learning experiences, interest and competence. Computers in Human Behavior, 93, 279-289. https://doi.org/10.1016/j.chb.2018.12.023

Go, E., & Sundar, S. S. (2019). Humanizing chatbots: The effects of visual, identity and conversational cues on humanness perceptions. Computers in Human Behavior, 97, 304-316. https://doi.org/10.1016/j.chb.2019.01.020

Hill, J., Ford, W. R., & Farreras, I. G. (2015). Real conversations with artificial intelligence: A comparison between human-human online conversations and human-chatbot conversations. Computers in Human Behavior, 49, 245-250. https://doi.org/10.1016/j.chb.2015.02.026

Holmes, W., & Tuomi, I. (2022). State of the art and practice in AI in education. European Journal of Education, 57(4), 542-570. https://doi.org/10.1111/ejed.12533

Konecki, M., Konecki, M., & Biškupić, I. (2023). Using artificial intelligence in higher education. In Proceedings of the 15th International Conference on Computer Supported Education. https://doi.org/10.5220/0012039700003470

Krause, S., Panchal, B. H., & Ubhe, N. (2025). Evolution of Learning: Assessing the Transformative Impact of Generative AI on Higher Education. Frontiers of Digital Education, 2(2). https://doi.org/10.1007/s44366-025-0058-7

Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2023). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28(1), 973-1018. https://doi.org/10.1007/s10639-022-11177-3

Łodzikowski, K., Foltz, P. W., & Behrens, J. T. (2024). Generative AI and Its Educational Implications. Postdigital Science and Education, 35-57. https://doi.org/10.1007/978-3-031-64487-0_2

Luger, E., & Sellen, A. (2016). "Like Having a Really Bad PA": The Gulf between User Expectation and Experience of Conversational Agents. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/2858036.2858288

Moppel, J. (2018). Socratic chatbot. University Of Tartu, Institute of Computer Science, Bachelor's Thesis.

Nordheim, C. B., Følstad, A., & Bjørkli, C. A. (2019). An Initial Model of Trust in Chatbots for Customer Service-Findings from a Questionnaire Study. Interacting With Computers, 31(3), 317-335. https://doi.org/10.1093/iwc/iwz022

Okonkwo, C. W., & Ade-Ibijola, A. (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2, 100033. https://doi.org/10.1016/j.caeai.2021.100033

Oliver, R. L. (1980). A Cognitive Model of the Antecedents and Consequences of Satisfaction Decisions. Journal of Marketing Research. https://doi.org/10.1177/002224378001700405

Pentina, I., Hancock, T., & Xie, T. (2023). Exploring relationship development with social chatbots: A mixed-method study of replika. Computers in Human Behavior, 140, 107600. https://doi.org/10.1016/j.chb.2022.107600

Pérez, J. Q., Daradoumis, T., & Puig, J. M. M. (2020). Rediscovering the use of chatbots in education: A systematic literature review. Computer Applications in Engineering Education, 28(6), 1549-1565. https://doi.org/10.1002/cae.22326

Pillai, R., & Sivathanu, B. (2020). Adoption of AI-based chatbots for hospitality and tourism. International Journal of Contemporary Hospitality Management, 32(10), 3199-3226. https://doi.org/10.1108/IJCHM-04-2020-0259

Qadir, J. (2023). Engineering education in the era of ChatGPT: Promise and pitfalls of generative AI for education. IEEE Global Engineering Education Conference (EDUCON), 2023, 1-9. https://doi.org/10.1109/EDUCON54358.2023.10125121

Shoufan, A. (2023). Exploring Students' Perceptions of CHATGPT: Thematic Analysis and Follow-Up Survey. IEEE Access. https://doi.org/10.1109/ACCESS.2023.3268224

St-Hilaire, F., Vu, D. D., Frau, A., Burns, N., Faraji, F., Potochny, J., Robert, S., Roussel, A., Zheng, S., & Glazier, T. (2022). A new era: Intelligent tutoring systems will transform online learning for millions. ArXiv Preprint ArXiv:2203.03724.

Sullivan, M., Kelly, A., & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning. Journal of Applied Learning & Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.17

Tate, T. P., Harnick-Shapiro, B., Ritchie, D. R., Tseng, W., Dennin, M., & Warschauer, M. (2025). Incorporating generative AI into a writing-intensive undergraduate course without off-loading learning. Discover Computing, 28(1). https://doi.org/10.1007/s10791-025-09563-9

Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User Acceptance of Information technology: toward a Unified View. MIS Quarterly, 27(3), 425-478. https://doi.org/10.2307/30036540

Vieriu, A. M., & Petrea, G. (2025). The Impact of Artificial Intelligence (AI) on Students' Academic Development. Education Sciences, 15(3), 343. https://doi.org/10.3390/educsci15030343

Winkler, R., & Söllner, M. (2018). Unleashing the potential of chatbots in education: A state-of-the-art analysis. In Academy of Management Annual Meeting (AOM). Chicago, USA. https://doi.org/10.5465/AMBPP.2018.15903abstract

Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021). Are we there yet? A systematic literature review on chatbots in education. Frontiers in Artificial Intelligence, 4, 654924. https://doi.org/10.3389/frai.2021.654924

Xie, T., & Pentina, I. (2022). Attachment Theory as a Framework to Understand Relationships with Social Chatbots: A Case Study of Replika. Proceedings of the . . . Annual Hawaii International Conference on System. https://doi.org/10.24251/HICSS.2022.258

Zhang, Q. (2023). Investigating the effects of gamification and ludicization on learning achievement and motivation: An empirical study employing Kahoot! and Habitica. International Journal of Technology-Enhanced Education (IJTEE), 2(1), 1-19. https://doi.org/10.4018/IJTEE.326127