12 Heuristics for Learning Analytics in Simulation-Based Professional Learning
DOI:
https://doi.org/10.18608/jla.2026.9141Keywords:
heuristics, heuristic evaluation, learning experience design, professional learning, simulation-based training, human-centred learning analytics, research paperAbstract
This study aims to develop a set of heuristics tailored for evaluating learning analytics in simulation-based professional learning, focusing on the following research questions: (1) What heuristics are appropriate for evaluating learning analytics in simulation-based professional learning contexts? (2) How can theoretical frameworks and empirical findings be combined in the development of such heuristics? (3) How can expert evaluation inform their refinement and applicability? The study combines a top-down approach, drawing on a theoretical framework for learning experience design, with a bottom-up analysis of empirical findings from prior studies in the context of a design project. An initial set of heuristics was iteratively reviewed and refined in collaboration with experts in user and learning experience design. The outcome is a detailed heuristic framework that supports the evaluation of learning analytics in simulation-based settings and accounts for the technological, pedagogical, and social dimensions of professional learning.
References
Alfredo, R., Echeverria, V., Jin, Y., Yan, L., Swiecki, Z., Gašević, D., & Martinez-Maldonado, R. (2024). Human-centred learning analytics and AI in education: A systematic literature review. Computers and Education: Artificial Intelligence, 6, Article 100215. https://doi.org/10.1016/j.caeai.2024.100215
Banks, J., Edwards, A. P., & Westerman, D. (2021). The space between: Nature and machine heuristics in evaluations of organisms, cyborgs, and robots. Cyberpsychology, Behavior and Social Networking, 24(5), 324–331. https://doi.org/10.1089/cyber.2020.0165
Blikstein, P. (2013). Multimodal learning analytics. In D. Suthers, K. Verbert, E. Duval, & X. Ochoa (Eds.), Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 102–106). ACM Press. https://doi.org/10.1145/2460296.2460316
Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238. https://doi.org/10.18608/jla.2016.32.11
Buckingham Shum, S., Martínez-Maldonado, R., Dimitriadis, Y., & Santos, P. (2019). Human-centred learning analytics: 2019–24. British Journal of Educational Technology, 6(2), 755–768. https://doi.org/10.1111/bjet.13442
Campbell, S., Greenwood, M., Prior, S., Shearer, T., Walkem, K., Young, S., Bywaters, D., & Walker, K. (2020). Purposive sampling: Complex or simple? Research case examples. Journal of Research in Nursing, 25(8), 652–661. https://doi.org/10.1177/1744987120927206
Chang, C.-C., & Johnson, T. (2021). Integrating heuristics and think-aloud approach to evaluate the usability of game-based learning material. Journal of Computers in Education, 8(1), 137–157. https://doi.org/10.1007/s40692-020-00174-5
Derby, J. L., & Chaparro, B. S. (2022). The development and validation of an augmented and mixed reality usability heuristic checklist. In J. Y. C. Chen & G. Fragomeni (Eds), virtual, augmented and mixed reality: Design and development (pp. 165–182). Springer International Publishing. https://doi.org/10.1007/978-3-031-05939-1_11
Engeström, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9781139814744
Gegenfurtner, A., Lewalter, D., Lehtinen, E., Schmidt, M., & Gruber, H. (2020). Teacher expertise and professional vision: Examining knowledge-based reasoning of pre-service teachers, in-service teachers, and school principals. Frontiers in Education, 5, Article 59. https://doi.org/10.3389/feduc.2020.00059
Goodwin, C. (1994). Professional vision. American Anthropologist, 96(3), 606–633. https://doi.org/10.1525/aa.1994.96.3.02a00100
Harrington, S., Klein, S., Fratoni, A. C., Sellberg, C., & Lindwall, O. (2025). Meaningful automated feedback? Balancing tensions in participatory learning analytics design. In J. Oshima, B. Chen, F. Vogel, & J. Järvelä (Eds.), Proceedings of the 18th International Conference on Computer-Supported Collaborative Learning (pp. 142–150). International Society of the Learning Sciences.
Harrington, S., & Sellberg, C. (2024). User wants, needs and resistance in the postdigital university: Prioritizing pedagogical concerns in learning analytics design. Research Square. https://doi.org/10.21203/rs.3.rs-5561942/v1
Harrington, S., Sellberg, C., Odéen, M., & Praetorius, G. (2024, July). Participatory learning analytics workshops with maritime instructors: Designing teacher-facing dashboards for simulation-based training [Conference presentation]. 2024 AHFE International Conference on Applied Human Factors and Ergonomics, Nice, France.
Jahnke, I., Riedel, N., Singh, K., & Moore, J. (2021). Advancing sociotechnical-pedagogical heuristics for the usability evaluation of online courses for adult learners. Online Learning, 25(4), Article 4. https://doi.org/10.24059/olj.v25i4.2439
Jahnke, I., Schmidt, M., Pham, M., & Singh, K. (2020). Sociotechnical-pedagogical usability for designing and evaluating learner experience in technology-enhanced environments. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds), Learner and user experience research: An introduction for the field of learning design & technology (pp. 127–145). EdTech Books. https://edtechbooks.org/ux/sociotechnical_pedagogical_usability
Langevin, R., Lordon, R. J., Avrahami, T., Cowan, B. R., Hirsch, T., & Hsieh, G. (2021). Heuristic evaluation of conversational agents. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Article 632). ACM Press. https://doi.org/10.1145/3411764.3445312
Lehtinen, E. (2023). Can simulations help higher education in training professional skills? Learning and Instruction, 86, Article 101772. https://doi.org/10.1016/j.learninstruc.2023.101772
Martinez-Maldonado, R., Echeverria, V., Fernandez-Nieto, G., Yan, L., Zhao, L., Alfredo, R., Li, X., Dix, S., Jaggard, H., Wotherspoon, R., Osborne, A., Buckingham Shum, S., & Gašević, D. (2023). Lessons learnt from a multimodal learning analytics deployment in-the-wild. ACM Transactions on Computer-Human Interaction, 31(1), Article 8. https://doi.org/10.1145/3622784
Nielsen, J. (1994, April 24). 10 usability heuristics for user interface design. Nielsen Norman Group. https://www.nngroup.com/articles/ten-usability-heuristics/
Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. In J. C. Chew & J. Whiteside (Eds.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 249–256). ACM Press. https://doi.org/10.1145/97243.97281
Nokelainen, P. (2006). An empirical assessment of pedagogical usability criteria for digital learning material with elementary school students. Journal of Educational Technology & Society, 9(2), 178–197.
Paddison, C., & Englefield, P. (2004). Applying heuristics to accessibility inspections. Interacting with Computers, 16(3), 507–521. https://doi.org/10.1016/j.intcom.2004.04.007
Schwengel, D., Villagrán, I., Miller, G., Miranda, C., & Toy, S. (2024). Multimodal assessment in clinical simulations: A guide for moving towards precision education. Medical Science Educator, 35(2), 1025–1035. https://doi.org/10.1007/s40670-024-02221-7
Sellberg, C., & Sharma, A. (2024). Toward multimodal learning analytics in simulation-based collaborative learning: A design ethnography of maritime training. International Journal of Computer-Supported Collaborative Learning, 20(2), 201–221. https://doi.org/10.1007/s11412-024-09435-2
Sellberg, C., Wiig, A. C., & Säljö, R. (2022). Mastering the artful practice of navigation: The situated endorsement of professional competence in post-simulation evaluations. Studies in Educational Evaluation, 72, Article 101111. https://doi.org/10.1016/j.stueduc.2021.101111
Spikol, D., Ruffaldi, E., & Cukurova, M. (2017). Using multimodal learning analytics to identify aspects of collaboration in project-based learning. In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), Making a difference: Prioritizing equity and access in CSCL, 12th International Conference on Computer Supported Collaborative Learning (CSCL) (Vol. 1, pp. 263–270). International Society of the Learning Sciences. https://repository.isls.org/handle/1/240
Yan, L., Echeverria, V., Jin, Y., Fernandez-Nieto, G., Zhao, L., Li, X., Alfredo, R., Swiecki, Z., Gašević, D., & Martinez-Maldonado, R. (2024). Evidence-based multimodal learning analytics for feedback and reflection in collaborative learning. British Journal of Educational Technology, 55(5), 1900–1925. https://doi.org/10.1111/bjet.13498
Zardari, B. A., Hussain, Z., Arain, A. A., Rizvi, W. H., & Vighio, M. S. (2021). QUEST e-learning portal: Applying heuristic evaluation, usability testing and eye tracking. Universal Access in the Information Society, 20(3), 531–543. https://doi.org/10.1007/s10209-020-00774-z
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
