The Facts Behind the Prophecy
Validating a Methodology for Identifying Behavioural Differences in Higher Education Student Subpopulations Under Intervention
DOI:
https://doi.org/10.18608/jla.2025.8553Keywords:
dropout, XAI, higher education, intervention, educational innovation, research paperAbstract
Artificial intelligence (AI) is currently leading an industrial revolution in most aspects of human life, and education is no exception. With the increasing ratio of students to faculty, AI could be an extremely beneficial tool for individual mentoring; for example, for cases of dropout and for student retention. While many models have already been built, the adoption of AI in education has been lower than expected, and few interventions have emerged from those models. Several factors may be in play, but one is that AI models are not easily explained, and the lack of explanation is fatal for situations like dropout prevention. An ideal AI-based tool for this problem would provide individually tailored interventions, but that would require a much deeper understanding of what a successful intervention entails. Using a novel methodology for feature comparison between student subpopulations, we compared regular students against students under academic guidance on a dataset containing 124,000 unique students and 36 informative features. We found that the explanations obtained regarding student dropout matched the real-world experiences of mentors and tutors, especially when dealing with highly explanatory features like previous average grades and interventions.
References
Akçapınar, G., Altun, A. & Aşkar, P. (2019). Using learning analytics to develop early-warning system for at-risk students. International Journal of Educational Technology in Higher Education, 16, 40. https://doi.org/10.1186/s41239-019-0172-z
Alvarado-Uribe, J., Mejía-Almada, P., Masetto Herrera, A. L., Molontay, R., Hilliger, I., Hegde, V., Montemayor Gallegos, J. E., Ramírez Díaz, R. A., & Ceballos, H. G. (2022). Student dataset from Tecnológico de Monterrey in Mexico to predict dropout in higher education. Data, 7(9), 119. https://doi.org/10.3390/data7090119
Arin, J. (2022). The missing link in academic advising: The faculty perspective. In B. McMurtrie & B. Supiano (Eds.), The future of advising: Strategies to support student success (pp. 40–43). Chronicle of Higher Education.
Bañeres, D., Rodríguez, M. E., Guerrero-Roldán, A. E., & Karadeniz, A. (2020). An early warning system to detect at-risk students in online higher education. Applied Sciences, 10(13), 4427. https://doi.org/10.3390/app10134427
Bañeres, D., Rodríguez-González, M. E., Guerrero-Roldán, A.-E., & Cortadas, P. (2023). An early warning system to identify and intervene online dropout learners. International Journal of Educational Technology in Higher Education, 20, 3. https://doi.org/10.1186/s41239-022-00371-5
Berens, J., Schneider, K., Görtz, S., Oster, S., & Burghoff, J. (2019). Early detection of students at risk: Predicting student dropouts using administrative student data from German universities and machine learning methods. Journal of Educational Data Mining, 11(3), 1–41. https://doi.org/10.5281/zenodo.3594771
Calhoun-Brown, A. (2022). How data and technology can improve advising and equity. In B. McMurtrie & B. Supiano (Eds.), The future of advising: Strategies to support student success (pp. 36–39). Chronicle of Higher Education.
Cave, S., Coughlan, K., & Dihal, K. (2019). “Scary robots”: Examining public responses to AI. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (AIES ’19), 27–28 January 2019, Honolulu, HI, USA (pp. 331–337). ACM Press. https://doi.org/10.1145/3306618.3314232
Harackiewicz, J. M., & Priniski, S. J. (2018). Improving student outcomes in higher education: The science of targeted intervention. Annual Review of Psychology, 69, 409–435. https://doi.org/10.1146/annurev-psych-122216-011725
Herodotou, C., Maguire, C., Hlosta, M., & Mulholland, P. (2023). Predictive learning analytics and university teachers: Usage and perceptions three years post-implementation. Proceedings of the 13th International Learning Analytics and Knowledge Conference (LAK ’23), 13–17 March 2023, Arlington, TX, USA (pp. 68–78). ACM Press. https://doi.org/10.1145/3576050.3576061
Heublein, U. (2014). Student drop-out from German higher education institutions. European Journal of Education, 49(4), 497–513. https://doi.org/10.1111/ejed.12097
Hilliger, I., De Laet, T., Henríquez, V., Guerra, J., Ortiz-Rojas, M., Zuñiga, M. Á., Baier, J., & Pérez-Sanagustín, M. (2020). For learners, with learners: Identifying indicators for an academic advising dashboard for students. In C. Alario-Hoyos, M. J. Rodríguez-Triana, M. Scheffel, I. Arnedillo-Sánchez, & S. M. Dennerlein (Eds.), Addressing global challenges and quality education: 15th European Conference on Technology Enhanced Learning, EC-TEL 2020, Heidelberg, Germany, September 14–18, 2020, Proceedings (pp. 117–130). Springer, Cham. https://doi.org/10.1007/978-3-030-57717-9_9
Jiménez-Gutiérrez, A. L., Mota-Hernández, C. I., Mezura-Montes, E., & Alvarado-Corona, R. (2024). Application of the performance of machine learning techniques as support in the prediction of school dropout. Scientific Reports, 14, 3957. https://doi.org/10.1038/s41598-024-53576-1
Jokhan, A., Sharma, B., & Singh, S. (2019). Early warning system as a predictor for student performance in higher education blended courses. Studies in Higher Education, 44(11), 1900–1911. https://doi.org/10.1080/03075079.2018.1466872
Kemper, L., Vorhoff, G., & Wigger, B. U. (2020). Predicting student dropout: A machine learning approach. European Journal of Higher Education, 10(1), 28–47. https://doi.org/10.1080/21568235.2020.1718520
Khosravi, H., Buckingham Shum, S., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022). Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence, 3, 100074. https://doi.org/10.1016/j.caeai.2022.100074
Lazowski, R. A., & Hulleman, C. S. (2016). Motivation interventions in education: A meta-analytic review. Review of Educational Research, 86(2), 602–640. https://doi.org/10.3102/0034654315617832
Liz-Domínguez, M., Caeiro-Rodríguez, M., Llamas-Nistal, M., & Mikic-Fonte, F. (2019). Predictors and early warning systems in higher education: A systematic literature review. Proceedings of the Learning Analytics Summer Institute Spain 2019: Learning Analytics in Higher Education (LASI Spain 2019), 27–28 June 2019, Vigo, Spain (pp. 84–99). CEUR Workshop Proceedings. https://ceur-ws.org/Vol-2415/paper08.pdf
Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS ’17), 4–9 December 2017, Long Beach, CA, USA (pp. 4768–4777). Curran Associates. https://dl.acm.org/doi/10.5555/3295222.3295230
Lundberg, S. M., Erion, G. G., & Lee, S.-I. (2018). Consistent individualized feature attribution for tree ensembles. arXiv. https://doi.org/10.48550/arXiv.1802.03888
McMurtrie, B., & Supiano, B. (2022a). Concerns about bias in advising technology. In B. McMurtrie & B. Supiano (Eds.), The future of advising: Strategies to support student success (pp. 33–39). Chronicle of Higher Education.
McMurtrie, B., & Supiano, B. (2022b). The barriers to better advising. In B. McMurtrie & B. Supiano (Eds.), The future of advising: Strategies to support student success (pp. 7–12). Chronicle of Higher Education.
Motz, B. A., Bergner, Y., Brooks, C. A., Gladden, A., Gray, G., Lang, C., Li, W., Marmolejo-Ramos, F., & Quick, J. D. (2023). A LAK of direction: Misalignment between the goals of learning analytics and its research scholarship. Journal of Learning Analytics, 10(2), 1–13. https://doi.org/10.18608/jla.2023.7913
Peng, P., Liu, L., Wu, Q., Tang, Y.-Y., Tang, J., Liu, T., & Liao, Y. (2024). Establishment and validation of a nomogram for dropout intention in Chinese early year medical undergraduates. BMC Medical Education, 24, 868. https://doi.org/10.1186/s12909-024-05835-y
Petch, J., Di, S., & Nelson, W. (2022). Opening the black box: The promise and limitations of explainable machine learning in cardiology. Canadian Journal of Cardiology, 38(2), 204–213. https://doi.org/10.1016/j.cjca.2021.09.004
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021). Four principles of explainable artificial intelligence: National Institute of Standards and Technology, interagency or internal report (NISTIR 8312). U.S. Department of Commerce. https://doi.org/10.6028/NIST.IR.8312
Rahiman, H. U., & Kodikal, R. (2024). Revolutionizing education: Artificial intelligence empowered learning in higher education. Cogent Education, 11(1), 2293431. https://doi.org/10.1080/2331186X.2023.2293431
Rodway, P., & Schepman, A. (2023). The impact of adopting AI educational technologies on projected course satisfaction in university students. Computers and Education: Artificial Intelligence, 5, 100150. https://doi.org/10.1016/j.caeai.2023.100150
Roslan, N., Mohd Jamil, J., Mohd Shaharanee, I. N., & Sultan Alawi, S. J. (2025). Prediction of student dropout in Malaysian’s private higher education institutes using data mining applications. Journal of Advanced Research in Applied Sciences and Engineering Technology, 45(2), 168–176. https://doi.org/10.37934/araset.45.2.168176
Russell, J.-E., Smith, A., & Larsen, R. (2020). Elements of success: Supporting at-risk student resilience through learning analytics. Computers & Education, 152, 103890. https://doi.org/10.1016/j.compedu.2020.103890
Shapley, L. S. (1953). A value for n-person games. In H. W. Kuhn & A. W. Tucker (Eds.), Contributions to the theory of games (AM-28), Volume II (pp. 307–318). Princeton University Press. http://www.jstor.org/stable/j.ctt1b9x1zv.24
Solis, M., Moreira, T., Gonzalez, R., Fernandez, T., & Hernandez, M. (2018). Perspectives to predict dropout in university students with machine learning. Proceedings of the 2018 IEEE International Work Conference on Bioinspired Intelligence (IWOBI 2018), 18–20 July 2018, San Carlos, Costa Rica (pp. 1–6). IEEE. https://doi.org/10.1109/IWOBI.2018.8464191
Stanton, B., & Jensen, T. (2021). Trust and artificial intelligence: National Institute of Standards and Technology interagency/internal report (NISTIR 8332). U.S. Department of Commerce. https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=931087
Talamás-Carvajal, J. A. (2023). The middle-man between models and mentors: Using SHAP values to explain dropout prediction models in higher education. Companion Proceedings of the 13th International Conference on Learning Analytics & Knowledge (LAK ’23), 13–17 March 2023, Arlington, TX, USA (pp. 68–70). SoLAR.
Talamás-Carvajal, J. A., & Ceballos-Cancino, H. G. (2024). Use of SHAP values for identifying differences in behaviors for subpopulations under intervention. Joint Proceedings of LAK 2024 Workshops (LAK-WS 2024), 18–22 March 2024, Kyoto, Japan (pp. 139–149). CEUR Workshop Proceedings. https://ceur-ws.org/Vol-3667/DS-LAK24_paper_4.pdf
Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research. Review of Educational Research, 45(1), 89–125. https://doi.org/10.3102/00346543045001089
United Nations Children’s Fund. (2018). Early warning systems for students at risk of dropping out: Policy and practice pointers for enrolling all children and adolescents in school and preventing dropout. https://www.unicef.org/eca/sites/unicef.org.eca/files/2018-11/Early%20warning%20systems%20for%20students%20at%20risk%20of%20dropping%20out_0.pdf
Valenzuela, J. P., & Yáñez, N. (2022). Trajectory and policies for inclusion in higher education in Latin America and the Caribbean in the context of the pandemic: Two decades of progress and challenges. Economic Commission for Latin America and the Caribbean (ECLAC).
Walton, G. M. (2014). The new science of wise psychological interventions. Current Directions in Psychological Science, 23(1), 73–82. https://doi.org/10.1177/0963721413512856
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
