Assessing 21st Century Competencies
A Two-Case Study on Curriculum Analytics to Inform Continuous Improvement in Higher Education
DOI:
https://doi.org/10.18608/jla.2026.9131Keywords:
21st century competencies, curriculum analytics, learning outcomes assessment, actionable insights, higher education, research paperAbstract
The growing emphasis on competency-based education (CBE) has heightened the need for clearly defined metrics and robust assessment frameworks to evaluate 21st-century competencies. Curriculum analytics (CA) provides a promising avenue for assessing learning outcomes (LOs) and informing continuous improvement in higher education. However, challenges persist in differentiating academic performance from actual LO development and in translating assessment data into meaningful program-level actions. This study examines how CA tools support the direct assessment of LOs and contribute to continuous improvement processes in higher education. Using a two-case study design, we analyzed CA implementation in two universities through interviews, cognitive walkthroughs, and institutional document analysis. Data triangulation identified 18 themes, nine of which reached full consensus among the three researchers. Findings indicate that CA tools effectively support the assessment of LOs aligned with 21st-century competencies by generating actionable insights that guide faculty toward more authentic and reflective teaching practices. The study contributes to the LA field by providing empirical evidence of how CA tools can bridge assessment and pedagogical improvement, offering both theoretical and practical implications for researchers and practitioners.
References
Agbo, J., Kitada Smalley, H., & Nyman, K. (2024). Competency-based assessment in the era of generative artificial intelligence: Perspectives of selected STEM educators. In Proceedings of the 32nd International Conference on Computers in Education. Asia-Pacific Society for Computers in Education. https://doi.org/10.58459/icce.2024.4970
Albuquerque, J., Rienties, B., & Divjak, B. (2025). Decoding learning design decisions: A cluster analysis of 12,749 teaching and learning activities. In Proceedings of the 15th International Conference on Learning Analytics and Knowledge (LAK 2025) (pp. 407–417). ACM Press. https://doi.org/10.1145/3706468.3706520
Asbari, M., & Nurhayati, W. (2024). Outcomes-based education in Indonesian higher education: Empowering students’ learning competencies. International Journal of Social and Management Studies, 5(5), 6–11. https://doi.org/10.5555/ijosmas.v5i5.445
Barthakur, A., Kovanović, V., Dawson, S., & Deneen, C. (2024). The application of curriculum analytics for improving assessments and quality assurance in higher education. Australasian Journal of Educational Technology, 40(4). https://doi.org/10.14742/ajet.9383
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32(3), 347–364. https://doi.org/10.1007/BF00138871
Blumenstein, M. (2020). Synergies of learning analytics and learning design: A systematic review of student outcomes. Journal of Learning Analytics, 7(3), 13–32. https://doi.org/10.18608/JLA.2020.73.3
Bouwma-Gearhart, J., & Hora, M. T. (2016). Supporting faculty in the era of accountability: How postsecondary leaders can facilitate the meaningful use of instructional data for continuous improvement. Journal of Higher Education Management, 31(1), 44–56. https://aaua.org/wp-content/uploads/2019/03/JHEM-31-1-2016-rev8-23-16.pdf
Brennan, R., & Prediger, S. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687–699. https://doi.org/10.1177/001316448104100307
Brown, G., & Grays, M. (2020). Understanding change in self-reported undergraduate attributes: A repeated measures survey of students in education. New Zealand Journal of Educational Studies, 55, 337–361. https://doi.org/10.1007/s40841-020-00178-w
Brunner, J. J., & Miranda, D. (2016). Educación superior en Iberoamérica [Higher education in Ibero-America]. Peru Minister of Education. https://www.researchgate.net/publication/313423057_Educacion_Superior_en_Iberoamerica-_Informe_2016
Buckingham Shum, S., & Deakin Crick, R. (2016). Learning analytics for 21st century competencies. Journal of Learning Analytics, 3(2), 6–21. https://doi.org/10.18608/jla.2016.32.2
Caspersen, J., & Smeby, J. C. (2018). The relationship among learning outcome measures used in higher education. Quality in Higher Education, 24(2), 117–135. https://doi.org/10.1080/13538322.2018.1484411
De Silva, L. M. H., Rodríguez-Triana, M. J., Chounta, I. A., & Pishtari, G. (2024). Curriculum analytics in higher education institutions: A systematic literature review. Journal of Computing in Higher Education, 37, 898–944. https://doi.org/10.1007/s12528-024-09410-8
Dennehy, D., Conboy, K., & Babu, J. (2023). Adopting learning analytics to inform postgraduate curriculum design: Recommendations and research agenda. Information Systems Frontiers, 25(4), 1315–1331. https://doi.org/10.1007/s10796-021-10183-z
Divjak, B., Barthakur, A., Kovanović, V., & Svetec, B. (2025). The impact of learning design on the mastery of learning outcomes in higher education. In Proceedings of the 15th International Conference on Learning Analytics and Knowledge (LAK 2025) (pp. 726–737). ACM Press. https://doi.org/10.1145/3706468.3706565
Eisenhardt, K. M., & Graebner, M. E. (2007). Theory building from cases: Opportunities and challenges. Academy of Management Journal, 50(1). https://doi.org/10.5465/amj.2007.24160888
Elbeck, M., & Bacon, D. (2015). Toward universal definitions for direct and indirect assessment. Journal of Education for Business, 90(5), 278–283. https://doi.org/10.1080/08832323.2015.1034064
Gedrimiene, E., Celik, I., Kaasila, A., Mäkitalo, K., & Muukkonen, H. (2024). Artificial intelligence-enhanced learning analytics for supporting career decisions. Education and Information Technologies, 29(1), 297–322. https://doi.org/10.1007/s10639-023-12277-4
Gray, G., & Bergner, Y. (2022). A practitioner’s guide to measurement in learning analytics: Decisions, opportunities, and challenges. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), Handbook of learning analytics (2nd ed., pp. 20–28). Society for Learning Analytics Research. https://www.solaresearch.org/publications/hla-22/
Gottipati, S., & Shankararaman, V. (2018). Competency analytics tool: Analyzing curriculum using course competencies. Education and Information Technologies, 23(1), 41–60. https://doi.org/10.1007/s10639-017-9584-3
Guzmán-Valenzuela, C., Gómez-González, C., Rojas-Murphy Tagle, A., & Lorca-Vyhmeister, A. (2021). Learning analytics in higher education: A preponderance of analytics but very little learning? International Journal of Educational Technology in Higher Education, 18(23), 1–19. https://doi.org/10.1186/s41239-021-00258-x
Hellas, A., Ihantola, P., Petersen, A., Ajanovski, V., Gutica, M., Hynninen, T., Knutas, A., Leinonen, J., Messom, C., & Liao, S. (2018). Predicting academic performance: A systematic literature review. In Proceedings companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (pp. 175–199). ACM Press. https://doi.org/10.1145/3293881.3295783
Hernández-Campos, M., Hilliger, I., & García-Peñalvo, F.-J. (2025). Evaluating learning outcomes through curriculum analytics. In Proceedings of the 15th International Learning Analytics and Knowledge Conference (LAK’ 25) (pp. 384–394). ACM Press. https://doi.org/10.1145/3706468.3706518
Hernández-Campos, M., Prado-Calderón, J. E., González-Torres, A., & García-Peñalvo, F. J. (2025). Direct measurement of learning outcomes in higher education. Evaluation and Program Planning, 112, Article 102638. https://doi.org/10.1016/j.evalprogplan.2025.102638
Hilliger, I., Aguirre, C., Miranda, C., Celis, S., & Pérez-Sanagustín, M. (2020). Design of a curriculum analytics tool to support continuous improvement processes in higher education. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 181–186). ACM Press. https://doi.org/10.1145/3375462.3375489
Hilliger, I., Celis, S., & Pérez-Sanagustín, M. (2022). Engaged versus disengaged teaching staff. Higher Education Policy, 35, 81–101. https://doi.org/10.1057/s41307-020-00196-9
Hilliger, I., Aguirre, C., Miranda, C., Celis, S., & Pérez-Sanagustín, M. (2022). Lessons learned from designing a curriculum analytics tool. Journal of Computing in Higher Education, 34, 633–657. https://doi.org/10.1007/s12528-022-09315-4
Hilliger, I., Miranda, C., Celis, S., & Pérez-Sanagustín, M. (2019). Evaluating usage of an analytics tool to support continuous curriculum improvement. In CEUR Workshop Proceedings. http://ceur-ws.org/Vol-2437/paper5.pdf
Hilliger, I., Miranda, C., Celis, S., & Pérez-Sanagustín, M. (2024). Curriculum analytics adoption in higher education. British Journal of Educational Technology, 55(3), 785–801. https://doi.org/10.1111/bjet.13374
Hora, M. (2016). Navigating the problem space of academic work. AERA Open, 2(1), 1–19. https://doi.org/10.1177/2332858415627612
Johnstone, S. M., & Soares, L. (2014). Principles for developing competency-based education programs. Change, 46(2), 12–19. https://doi.org/10.1080/00091383.2014.896705
Jørnø, R. L., & Gynther, K. (2018). What constitutes an “actionable insight” in learning analytics? Journal of Learning Analytics, 5(3), 198–221. https://doi.org/10.18608/jla.2018.53.13
Kaliisa, R., Jivet, I., & Prinsloo, P. (2023). A checklist to guide learning analytics dashboards. International Journal of Educational Technology in Higher Education, 20(1), Article 28. https://doi.org/10.1186/s41239-023-00394-6
Kitto, K., Sarathy, N., Gromov, A., Liu, M., Musial, K., & Shum, S. B. (2020). Towards skills-based curriculum analytics. In Proceedings of the Tenth International Conference on Learning Analytics & Knowledge (pp. 171–180). ACM Press. https://doi.org/10.1145/3375462.3375526
Kleimola, R., & Leppisaari, I. (2022). Learning analytics to develop future competences. International Journal of Educational Technology in Higher Education, 19(1), Article 17. https://doi.org/10.1186/s41239-022-00318-w
Kotorov, I., Krasylnykova, Y., Pérez-Sanagustín, M., Mansilla, F., & Broisin, J. (2024). Supporting decision-making for promoting teaching and learning innovation. Journal of Learning Analytics, 11(1), 21–36. https://doi.org/10.18608/jla.2024.8131
Lagus, J., Longi, K., Klami, A., & Hellas, A. (2018). Transfer-learning methods in programming course outcome prediction. ACM Transactions on Computing Education, 18(4), Article 19. https://doi.org/10.1145/3152714
Leandro-Cruz, M., Saunders-Smits, G., & Groen, P. (2020). Evaluation of competency methods in engineering education. European Journal of Engineering Education, 45(5), 729–757. https://doi.org/10.1080/03043797.2019.1671810
Leitner, P., Ebner, M., & Ebner, M. (2019). Learning analytics challenges to overcome. In D. Ifenthaler, D. K. Mah, & J. Y. K. Yau (Eds.), Utilizing learning analytics to support study success (pp. 91–104). Springer. https://doi.org/10.1007/978-3-319-64792-0
Lewis, C., & Wharton, C. (1997). Cognitive walkthroughs. In M. Helander, T. Landauer, & P. Prabhu (Eds.), Handbook of human–computer interaction (pp. 717–732). https://doi.org/10.1016/B978-044481862-1.50096-0
Li, L. (2022). Reskilling and upskilling the future-ready workforce. Information Systems Frontiers, 26, 1697–1712. https://doi.org/10.1007/s10796-022-10308-y
Luce, C., & Kirnan, J. P. (2016). Using indirect vs. direct measures in summative assessment. Journal of the Scholarship of Teaching and Learning, 16(4), 75–91. https://doi.org/10.14434/josotl.v16i4.19371
Mashuri, S., Sarib, M., Rasak, A., & Alhabsyi, F. (2022). Semi-structured interview: A methodological reflection. Journal of Research & Method in Education, 12(1), 22–29. https://doi.org/10.9790/7388-1201052229
Motz, B. A., Bergner, Y., Brooks, C. A., Gladden, A., Gray, G., Lang, C., Li, W., Marmolejo-Ramos, F., & Quick, J. D. (2023). A LAK of direction. Journal of Learning Analytics, 10(2), 1–13. https://doi.org/10.18608/jla.2023.7913
Namoun, A., & Alshanqiti, A. (2021). Predicting student performance using data mining. Applied Sciences, 11(1), Article 237. https://doi.org/10.3390/app11010237
Naujokaitiene, J., Tamoliune, G., Volungeviciene, A., & Duart, J. M. (2020). Using learning analytics to engage students. Journal of New Approaches in Educational Research, 9(2), 231–244. https://doi.org/10.7821/naer.2020.7.561
Ochoa, X. (2016). Simple metrics for curricular analytics. In J. Greer, M. Molinaro, X. Ochoa, & T. McKay (Eds.), Proceedings of the 1st Learning Analytics for Curriculum and Program Quality Improvement Workshop (pp. 20–26). CEUR Workshop Proceedings. http://ceur-ws.org/Vol-1590/paper-04.pdf http://ceur-ws.org/Vol-1590/paper-04.pdf
Pacher, C., Woschank, M., Zunk, B. M., & Gruber, E. (2024). Engineering education 5.0. Production and Manufacturing Research, 12(1), 1–45. https://doi.org/10.1080/21693277.2024.2337224
Pardo, A., Mirriahi, N., Gašević, D., & Dawson, S. (2022). A model for learning analytics. In R. Sharpe, S. Bennett, & T. Varga-Atkins (Eds.), Handbook of digital higher education (pp. 26–37). Edward Elgar.
Porter, T., Molina, D. C., Blackwell, L., Roberts, S., Quirk, A., Duckworth, A. L., & Trzesniewski, K. (2020). Measuring mastery behaviours at scale. Journal of Learning Analytics, 7(1), 5–18. https://doi.org/10.18608/jla.2020.71.2
Radu, C., Ciocoiu, C. N., Veith, C., & Dobrea, R. C. (2024). Artificial intelligence and competency-based education. Amfiteatru Economic, 26(65), 220–240. https://doi.org/10.24818/EA/2024/65/220
Randolph, J. J. (2008). Online kappa calculator [Computer software]. http://justus.randolph.name/kappa
Saldaña, J. (2017). The coding manual for qualitative researchers (3rd ed.). SAGE.
Sarmiento, J. P., & Wise, A. F. (2022). Participatory and co-design of learning analytics. In LAK22: 12th International Learning Analytics and Knowledge Conference (pp. 535–541). ACM Press. https://doi.org/10.1145/3506860.3506910
Schwendimann, B., Rodríguez-Triana, M., Vozniuk, A., Prieto, L., Shirvani Boroujeni, M., Holzer, A., Gillet, D., & Dillenbourg, P. (2017). Perceiving learning at a glance. IEEE Transactions on Learning Technologies, 10(1), 30–41. https://doi.org/10.1109/TLT.2016.2599522
Sebar, L. M., & Vecchio, B. (2020). Application of rigour and credibility in qualitative document analysis. The Qualitative Report, 25(2), 456–470. https://doi.org/10.46743/2160-3715/2020.4240
Shanto, S. S., Ahmed, Z., & Jony, A. I. (2025). A proposed framework for outcome-based learning using generative AI. Educational Technology Quarterly, 2025(1), 1–15. https://doi.org/10.55056/etq.788
SocioCultural Research Consultants. (2025). Dedoose (Version 10.0.34) [Computer software]. https://www.dedoose.com
Stake, R. E. (2006). Multiple case study analysis. Guilford Press.
Strah, M. (2020). Los sistemas nacionales de aseguramiento de la calidad de la educación superior en Iberoamérica [National quality assurance systems in higher education in Ibero-America]. Comisión Nacional de Evaluación y Acreditación Universitaria & Organización de Estados Iberoamericanos. https://www.coneau.gob.ar/archivos/publicaciones/documentos/OEISistemasNacionalesAseguramientoIberoamerica.pdf
Weldy, T. G., & Turnipseed, D. L. (2010). Assessing and improving learning in business schools. Journal of Education for Business, 85(5), 268–273. https://doi.org/10.1080/08832320903449535
Wise, A. F. (2018). Learning analytics: Using data-informed decision-making. In O. Adesope & A. G. Rud (Eds.) Contemporary technologies in education (pp. 119–143). Springer International Publishing. https://doi.org/10.1007/978-3-319-89680-9_7
Yip, H., & Smales, R. (2000). Review of competency-based education in dentistry. British Dental Journal, 189, 324–326. https://doi.org/10.1038/sj.bdj.4800758
Yorke, M. (2007). Grading student achievement in higher education: Signals and shortcomings. Routledge.
Zilinskiene, I. (2022). Exploring learning analytics for course design improvement. Baltic Journal of Modern Computing, 10(1), 36–54. https://doi.org/10.22364/BJMC.2022.10.1.03
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
