Learning Analytics for Early Identification of At-Risk Students and Feedback Intervention
DOI:
https://doi.org/10.18608/jla.2025.8735Keywords:
learning analytics, predictive modelling, model generalization ability, early feedback intervention, effective feedback design, research paperAbstract
Supporting academically at-risk students has attracted much attention in the field of learning analytics. However, much of the research in this area has focused on developing advanced machine learning models to predict students' academic performance, which alone is insufficient to improve student learning without the implementation of timely interventions. Among the studies that attempted to mitigate this limitation by deploying intervention feedback to enhance learning, few created their feedback based on established theories of effective feedback. This theoretical oversight may limit students' uptake of the provided intervention. In response to these gaps, we conducted a study that aimed at supporting at-risk students at the early stage of an undergraduate-level course. Specifically, we developed predictive machine learning models using trace and academic data from the previous offering of a course, and applied these models to identify at-risk students in the subsequent semester's offering of the same course. For the identified at-risk students, we sent intervention emails designed by feedback experts based on a relational feedback framework designed to enhance feedback effectiveness by strengthening student-instructor relationships. We evaluated the effectiveness of the proposed approach by assessing the performance of the predictive models in terms of generalisability, and measuring the impact of the feedback intervention on students' learning engagement. Results showed that i) our predictive models demonstrated a high prediction accuracy (with AUC scores above 0.8) when applied to a new cohort of students; ii) more than 30% of the identified at-risk students visited previously unengaged learning activities within two weeks following the intervention; and iii) survey responses from 9.27% of at-risk students indicated general satisfaction with the provided feedback intervention, and 60\% of the respondents expressed a preference for receiving the intervention more frequently than the twice-per-semester frequency implemented in the present study.
References
Adnan, M., Habib, A., Ashraf, J., Mussadiq, S., Raza, A. A., Abid, M., . . . Khan, S. U. (2021). Predicting at-risk students at different percentages of course length for early intervention using machine learning models. Ieee Access, 9, 7519–7539. https://doi.org/10.1109/ACCESS.2021.3049446
Akcapınar, G., Altun, A., & Askar, P. (2019). Using learning analytics to develop early-warning system for at-risk students. International Journal of Educational Technology in Higher Education, 16(1), 1–20. https://doi.org/10.1186/s41239-019-0172-z
Arnold, K. E., & Pistilli, M. D. (2012). Course signals at purdue: Using learning analytics to increase student success. In Proceedings of the 2nd international conference on learning analytics and knowledge (LAK 2012), 29 April–2 May 2012, Vancouver, British Columbia, Canada (pp. 267–270). ACM.. https://doi.org/10.1145/2330601.2330666
Bainbridge, J., Melitski, J., Zahradnik, A., Laurıa, E. J., Jayaprakash, S., & Baron, J. (2015). Using learning analytics to predict at-risk students in online graduate public affairs and administration education. Journal of Public Affairs Education, 21(2), 247–262. https://doi.org/10.1080/15236803.2015.12001831
Baker, R., Evans, B., & Dee, T. (2016). A randomized experiment testing the efficacy of a scheduling nudge in a massive open online course (mooc). Aera Open, 2(4), 2332858416674007. https://doi.org/10.1177/2332858416674007
Baker, R. S., Lindrum, D., Lindrum, M. J., & Perkowski, D. (2015). Analyzing early at-risk factors in higher education e learning courses. International Educational Data Mining Society. vitz, S. Ventura, & M. Desmarais (Eds.), Proceedings of the Eighth International Conference on Educational Data Mining (EDM 2015), 26–29 June 2015, Madrid, Spain (pp. 150–155). International Educational Data Mining Society. https://files.eric.ed.gov/fulltext/ED560503.pdf
Baneres, D., Rodríguez, M. E., Guerrero-Roldán, A. E., & Karadeniz, A. (2020). An early warning system to detect at-risk students in online higher education. Applied Sciences, 10(13), 4427. https://doi.org/10.3390/app10134427
Berka, P., & Marek, L. (2021). Bachelor’s degree student dropouts: Who tend to stay and who tend to leave? Studies in Educational Evaluation, 70, 100999. https://doi.org/10.1016/j.stueduc.2021.100999
Borrella, I., Caballero-Caballero, S., & Ponce-Cueto, E. (2019). Predict and intervene: Addressing the dropout problem in a mooc-based program. In Proceedings of the sixth (2019) acm conference on learning@ scale (pp. 1–9). ACM. https://doi.org/10.1145/3330430.3333634
Borrella, I., Caballero-Caballero, S., & Ponce-Cueto, E. (2022). Taking action to reduce dropout in moocs: Tested interventions. Computers & Education, 179, 104412. https://doi.org/10.1016/j.compedu.2021.104412
Brooks, C., Thompson, C., & Teasley, S. (2015). A time series interaction analysis method for building predictive models of learners using log data. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 April 2015, Poughkeepsie, New York, USA (pp. 126–135). ACM. https://doi.org/10.1145/2723576.2723581
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new era. EDUCAUSE review, 42(4), 40. https://er.educause.edu/articles/2007/7/academic-analytics-a-new-tool-for-a-new-era
Chipchase, L., Davidson, M., Blackstock, F., Bye, R., Clothier, P., Klupp, N., Nickson, W., Turner, D., & Williams, M. (2017). Conceptualising and measuring student disengagement in higher education: A synthesis of the literature. International Journal of Higher Education, 6(2), 31–42. https://doi.org/10.5430/ijhe.v6n2p31
Christou, V., Tsoulos, I., Loupas, V., Tzallas, A. T., Gogos, C., Karvelis, P. S., Antoniadis, N., Glavas, E., & Giannakeas, N. (2023). Performance and early drop prediction for higher education students using machine learning. Expert Systems with Applications, 225, 120079. https://doi.org/10.1016/j.eswa.2023.120079
Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In Proceedings of the Second International Conference on Learning Analytics and Knowledge (LAK 2012), 29 April–2 May 2012, Vancouver, British Columbia, Canada (pp. 134–138). ACM. https://doi.org/10.1145/2330601.2330636
Dai, W., Tsai, Y.-S., Gašević, D., & Chen, G. (2025). Designing relational feedback: A rapid review and qualitative synthesis. Assessment & Evaluation in Higher Education, 50(1), 16–30. https://doi.org/10.1080/02602938.2024.2361166
Dawson, S., Jovanovic, J., Gašević, D., & Pardo, A. (2017). From prediction to impact: Evaluation of a learning analytics retention program. In Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, British Columbia, Canada (pp. 474–478). ACM. https://doi.org/10.1145/3027385.3027405
Dix, N., Lail, A., Birnbaum, M., & Paris, J. (2020). Exploring the “at-risk” student label through the perspectives of higher education professionals. The Qualitative Report, 25(11). https://doi.org/10.46743/2160-3715/2020.3371
Er, E. (2012). Identifying at-risk students using machine learning techniques: A case study with IS 100. International Journal of Machine Learning and Computing, 2(4), 476. https://www.ijml.org/show-32-132-1.html
Faas, C., Benson, M. J., Kaestle, C. E., & Savla, J. (2018). Socioeconomic success and mental health profiles of young adults who drop out of college. Journal of Youth Studies, 21(5), 669–686. https://doi.org/10.1080/13676261.2017.1406598
Falkner, N. J., & Falkner, K. E. (2012). A fast measure for identifying at-risk students in computer science. In Proceedings of the Ninth Annual International Conference on International Computing Education Research (ICER 2012), 9–11 September 2012, Auckland, New Zealand (pp. 55–62). ACM. https://doi.org/10.1145/2361276.2361288
Figueroa-Cañas, J., & Sancho-Vinuesa, T. (2019). Predicting early dropout students is a matter of checking completed quizzes: The case of an online statistics module. In M. Caeiro-Rodrıguez, A. Hernandez-Garcıa, & P. J. Munoz-Merino (Eds.), Learning Analytics Summer Institute Spain 2019: Learning Analytics in Higher Education (LASI 2019), 27–28 June 2019, Vigo, Spain (pp. 100–111). CEUR Workshop Proceedings. https://ceur-ws.org/Vol-2415/paper09.pdf
Foster, E., & Siddle, R. (2020). The effectiveness of learning analytics for identifying at-risk students in higher education. Assessment & Evaluation in Higher Education, 45(6), 842–854. https://doi.org/10.1080/02602938.2019.1682118
Gardner, J., Yu, R., Nguyen, Q., Brooks, C., & Kizilcec, R. (2023). Cross-institutional transfer learning for educational models: Implications for model performance, fairness, and equity. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT 2023), 12–15 June 2023, Chicago, Illinois, USA (pp. 1664–1684). ACM. https://doi.org/10.1145/3593013.3594107
Gašević, D., Dawson, S., Rogers, T., & Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002
Gupta, S. K., Antony, J., Lacher, F., & Douglas, J. (2020). Lean Six Sigma for reducing student dropouts in higher education—An exploratory study. Total Quality Management & Business Excellence, 31(1-2), 178–193. https://doi.org/10.1080/14783363.2017.1422710
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
He, J., Bailey, J., Rubinstein, B., & Zhang, R. (2015). Identifying at-risk students in massive open online courses. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI 2015), 25–30 June 2023, Austin, Texas, USA (Vol. 29). PKP|PS. https://doi.org/10.1609/aaai.v29i1.9471
Heron, M., Medland, E., Winstone, N., & Pitt, E. (2023). Developing the relational in teacher feedback literacy: Exploring feedback talk. Assessment & Evaluation in Higher Education, 48(2), 172–185. https://doi.org/10.1080/02602938.2021.1932735
Hu, Y. - H., Lo, C.-L., & Shih, S.- P. (2014). Developing early warning systems to predict students’ online learning performance. Computers in Human Behavior, 36, 469–478. https://doi.org/10.1016/j.chb.2014.04.002
Jayaprakash, S. M., Moody, E. W., Laur´ıa, E. J., Regan, J. R., & Baron, J. D. (2014). Early alert of academically at-risk students: An open source analytics initiative. Journal of Learning Analytics, 1(1), 6–47. https://doi.org/10.18608/jla.2014.11.3
Jovanovic, J., Saqr, M., Joksimovic, S., & Gasevic, D. (2021). Students matter the most in learning analytics: The effects of internal and instructional conditions in predicting academic success. Computers & Education, 172, 104251. https://doi.org/10.1016/j.compedu.2021.104251
Kastberg, S. E., Lischka, A. E., & Hillman, S. L. (2020). Written feedback as a relational practice: Revealing mediating factors. Studying Teacher Education, 16(3), 324–344. https://doi.org/10.1080/17425964.2020.1834152
Khalil, M., Slade, S., & Prinsloo, P. (2024). Learning analytics in support of inclusiveness and disabled students: A systematic review. Journal of Computing in Higher Education, 36(1), 202–219. https://doi.org/10.1007/s12528-023-09363-4
Khan, I. A., Subhani, A., Rasheed, Z., Ahmad, U., & Brohi, M. N. (2023). Comprehensive assessment of risk assessment tools and academic performance in higher education: A meta-analytic perspective. Journal of Applied Engineering & Technology (JAET), 7(2), 10–24. https://doi.org/10.55447/jaet.07.02.116
Latif, A., Choudhary, A., & Hammayun, A. (2015). Economic effects of student dropouts: A comparative study. Journal of Global Economics, 3(2), 1–4. https://www.hilarispublisher.com/open-access/economic-effects-of-student-dropouts-a-comparative-study-2375-4389-1000137.pdf
Liang, Z., Sha, L., Tsai, Y. - S., Gašević, D., & Chen, G. (2024). Towards the automated generation of readily applicable personalised feedback in education. In A. Olney, I. Chounta, Z. Liu, O. Santos, & I. Bittencourt (Eds.), Artificial intelligence in education. AIED 2024. Lecture notes in computer science (pp. 75–88, Vol. 14830). Springer. https://doi.org/10.1007/978-3-031-64299-9_6
Lin, J., Dai, W., Lim, L.-A., Tsai, Y.-S., Mello, R. F., Khosravi, H., Gašević, D., & Chen, G. (2023). Learner-centred analytics of feedback content in higher education. In Proceedings of the 13th International Conference on Learning Analytics and Knowledge (LAK 2023), 13–17 March 2023, Arlington, Texas, USA (pp. 100–110). ACM. https://doi.org/10.1145/3576050.3576064
Liz Domínguez, M., Caeiro Rodríguez, M., Llamas Nistal, M., & Mikic Fonte, F. A. (2019). Predictors and early warning systems in higher education: A systematic literature review. In M. Caeiro-Rodrıguez, A. Hernandez-Garcıa, & P. J. Munoz-Merino (Eds.), Learning Analytics Summer Institute Spain 2019: Learning Analytics in Higher Education (LASI 2019), 27–28 June 2019, Vigo, Spain (pp. 84–99). CEUR Workshop Proceedings. https://ceur-ws.org/Vol-2415/paper08.pdf
López-García, A., Blasco-Blasco, O., Liern-García, M., & Parada-Rico, S. E. (2023). Early detection of students’ failure using machine learning techniques. Operations Research Perspectives, 11, 100292. https://doi.org/10.1016/j.orp.2023.100292
López-Zambrano, J., Lara, J. A., & Romero, C. (2020). Towards portability of models for predicting students’ final performance in university courses starting from Moodle logs. Applied Sciences, 10(1), 354. https://doi.org/10.3390/app10010354
Lu, O. H., Huang, J. C., Huang, A. Y., & Yang, S. J. (2017). Applying learning analytics for improving students engagement and learning outcomes in an MOOCs enabled collaborative programming course. Interactive Learning Environments, 25(2), 220–234. https://doi.org/10.1080/10494820.2016.1278391
Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. arXiv:1705.07874. https://doi.org/10.48550/arXiv.1705.07874
Masabo, E., Nzabanita, J., Ngaruye, I., Ruranga, C., Nizeyimana, J. P., Uwonkunda, J., & Ndanguza, D. (2023). Early detection of students at risk of poor performance in Rwanda higher education using machine learning techniques. International Journal of Information Technology, 15(6), 3201–3210. https://doi.org/10.1007/s41870-023-01334-3
Middleton, T., ahmed Shafi, A., Millican, R., & Templeton, S. (2023). Developing effective assessment feedback: Academic buoyancy and the relational dimensions of feedback. Teaching in Higher Education, 28(1), 118–135. https://doi.org/10.1080/13562517.2020.1777397
Na, K. S., & Tasir, Z. (2017). Identifying at-risk students in online learning by analysing learning behaviour: A systematic review. In 2017 IEEE Conference on Big Data and Analytics (ICBDA 2017), 16–17 November 2017, Kuching, Malaysia (pp. 118–123). IEEE. https://doi.org/10.1109/ICBDAA.2017.8284117
Nimy, E., Mosia, M., & Chibaya, C. (2023). Identifying at-risk students for early intervention—A probabilistic machine learning approach. Applied Sciences, 13(6), 3869. https://doi.org/10.3390/app13063869
Osborne, J. B., & Lang, A. S. (2023). Predictive identification of at-risk students: Using learning management system data. Journal of Postsecondary Student Success, 2(4), 108–126. https://doi.org/10.33009/fsop_jpss132082
Poulos, A., & Mahony, M. J. (2008). Effectiveness of feedback: The students’ perspective. Assessment & Evaluation in Higher Education, 33(2), 143–154. https://doi.org/10.1080/02602930601127869
Ryan, T., Henderson, M., Ryan, K., & Kennedy, G. (2021). Designing learner-centred text-based feedback: A rapid review and qualitative synthesis. Assessment & Evaluation in Higher Education, 46(6), 894–912. https://doi.org/10.1080/02602938.2020.1828819
Schmidt, A., Cechinel, C., Queiroga, E. M., Primo, T., Ramos, V., Bordin, A. S., Mello, R. F., & Muñoz, R. (2025). Analyzing intervention strategies employed in response to automated academic-risk identification: A systematic review. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 20, 77–85. https://doi.org/10.1109/RITA.2025.3540161
Singell, L. D., & Waddell, G. R. (2010). Modeling retention at a large public university: Can at-risk students be identified early enough to treat? Research in Higher Education, 51, 546–572. https://doi.org/10.1007/s11162-010-9170-7
Sonnleitner, B., Madou, T., Deceuninck, M., Theodosiou, F., & Sagaert, Y. R. (2025). Evaluation of early student performance prediction given concept drift. Computers and Education: Artificial Intelligence, 8, 100369. https://doi.org/10.1016/j.caeai.2025.100369
Veerasamy, A. K., D’Souza, D., Apiola, M.-V., Laakso, M.-J., & Salakoski, T. (2020). Using early assessment performance as early warning signs to identify at-risk students in programming courses. In Proceedings of the 2020 IEEE Frontiers in Education Conference (FIE), 21–24 October 2020, Uppsala, Sweden (pp. 1–9). IEEE. https://doi.org/10.1109/FIE44824.2020.9274277
Waheed, H., Hassan, S. - U., Nawaz, R., Aljohani, N. R., Chen, G., & Gasevic, D. (2023). Early prediction of learners at risk in self-paced education: A neural network approach. Expert Systems with Applications, 213, 118868. https://doi.org/10.1016/j.eswa.2022.118868
Wang, Q., & Mousavi, A. (2023). Which log variables significantly predict academic achievement? A systematic review and meta-analysis. British Journal of Educational Technology, 54(1), 142–191. https://doi.org/10.1111/bjet.13282
Whitelock-Wainwright, A., Gašević, D., Tsai, Y.- S., Drachsler, H., Scheffel, M., Muñoz-Merino, P. J., Tammets, K., & Delgado Kloos, C. (2020). Assessing the validity of a learning analytics expectation instrument: A multinational study. Journal of Computer Assisted Learning, 36(2), 209–240. https://doi.org/10.1111/jcal.12401
Winston, K. A., van der Vleuten, C. P., & Scherpbier, A. J. (2014). Prediction and prevention of failure: An early intervention to assist at-risk medical students. Medical Teacher, 36(1), 25–31. https://doi.org/10.3109/0142159X.2013.836270
Wong, B. T.-M., & Li, K. C. (2020). A review of learning analytics intervention in higher education (2011–2018). Journal of Computers in Education, 7(1), 7–28. https://doi.org/10.1007/s40692-019-00143-7
Zacharis, N. Z. (2015). A multivariate approach to predicting student outcomes in web-enabled blended learning courses. The Internet and Higher Education, 27, 44–53. https://doi.org/10.1016/j.iheduc.2015.05.002
Zhang, L., & Rangwala, H. (2018). Early identification of at-risk students using iterative logistic regression. In C. P. Rose, R. Martınez-Maldonado, H. U. Hoppe, R. Luckin, M. Mavrikis, K. Porayska-Pomsta, B. McLaren, & B. du Boulay (Eds.), Artificial intelligence in education. AIED 2018. Lecture notes in computer science (pp. 613–626, Vol. 10947). Springer. https://doi.org/10.1007/978-3-319-93843-1_45
Zhang, Y., Fei, Q., Quddus, M., & Davis, C. (2014). An examination of the impact of early intervention on learning outcomes of at-risk students. Research in Higher Education Journal, 26. https://files.eric.ed.gov/fulltext/EJ1055303.pdf
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
