Learning Analytics Dashboard Design and Evaluation to Support Student Self-Regulation of Study Behaviour
DOI:
https://doi.org/10.18608/jla.2024.8529Keywords:
learning analytics dashboard (LAD), dashboard design, self-regulated learning (SRL), higher education, needs assessment, usability test, perceived usefulness, data and tools reportAbstract
For university students, self-regulation of study behaviour is important. However, students are not always capable of effective self-regulation. Providing study behaviour information via a learning analytics dashboard (LAD) may support phases within self-regulated learning (SRL). However, it is unclear what information a LAD should provide, how to present information in a usable manner, and what the information’s perceived usefulness is in supporting self-regulation of study behaviour. This study entails a sequential mixed design: assessing information needs in focus groups (n=7), exploring usability via think-aloud interviews (n=8), assessing usability with the System Usability Scale (n=42), and assessing perceived usefulness via interviews (n=16). Results showed that students and tutors agreed on the relevance of the constructs chosen from literature but differed in ranking the importance of new constructs. Usability exploration led to several design improvements. Perceived usefulness assessment showed the LAD supported the appraisal of study behaviour. A need for reference frames to facilitate data interpretation was vocalized. Impacts on study behaviour varied, possibly because preparatory activities were not used. Impact could be improved by further integrating the LAD into existing learning processes.
References
Al-Ayash, A., Kane, R. T., Smith, D., & Green‐Armytage, P. (2016). The influence of color on student emotion, heart rate, and performance in learning environments. Color Research & Application, 41(2), 196–205. https://doi.org/10.1002/col.21949
Alfredo, R., Echeverria, V., Jin, Y., Yan, L., Swiecki, Z., Gašević, D., & Martinez-Maldonado, R. (2024). Human-centred learning analytics and AI in education: A systematic literature review. Computers and Education: Artificial Intelligence, 6, 100215. https://doi.org/10.1016/j.caeai.2024.100215
Bandura, A. (1997). Self-efficacy: The exercise of control. W. H. Freeman.
Bangor, A., Miller, J., & Kortum, P. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3), 114–123. https://dl.acm.org/doi/10.5555/2835587.2835589
Beheshitha, S. S., Hatala, M., Gašević, D., & Joksimović, S. (2016). The role of achievement goal orientations when studying effect of learning analytics visualizations. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 54–63). ACM Press. https://doi.org/10.1145/2883851.2883904
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444. https://doi.org/10.1146/annurev-psych-113011-143823
Bodily, R., & Verbert, K. (2017). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405–418. https://doi.org/10.1109/TLT.2017.2740172
Boekaerts, M., & Cascallar, E. (2006). How far have we moved toward the integration of theory and practice in self-regulation? Educational Psychology Review, 18, 199–210. https://doi.org/10.1007/s10648-006-9013-4
Boekaerts, M. (2011). Emotions, emotion regulation, and self-regulation of learning. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 408–425). Routledge.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–352. https://doi.org/10.1080/14780887.2020.1769238
Broadbent, J., & Poon, W. L. (2015). Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review. The Internet and Higher Education, 27, 1–13. https://doi.org/10.1016/j.iheduc.2015.04.007
Brooke, J. (2013). SUS: A retrospective. Journal of Usability Studies, 8(2), 29–40.
Buckingham Shum, S., Martínez‐Maldonado, R., Dimitriadis, Y., & Santos, P. (2024). Human‐centred learning analytics: 2019–24. British Journal of Educational Technology, 55(3), 755–768. https://doi.org/10.1111/bjet.13442
Byrne, D. (2022). A worked example of Braun and Clarke’s approach to reflexive thematic analysis. Quality & Quantity, 56(3), 1391–1412. https://doi.org/10.1007/s11135-021-01182-y
Cohen, L., Manion, L., & Morrison, K. (2018). Interviews. In L. Cohen, L. Manion, & K. Morrison (Eds.), Research methods in education (8th ed., pp. 506–541). Routledge. https://doi.org/10.4324/9781315456539
Cowan, J. (2019). The potential of cognitive think-aloud protocols for educational action-research. Active Learning in Higher Education, 20(3), 219–232. https://doi.org/10.1177/1469787417735614
Cumming, J., Woodcock, C., Cooley, S. J., Holland, M. J. G., & Burns, V. E. (2015). Development and validation of the groupwork skills questionnaire (GSQ) for higher education. Assessment & Evaluation in Higher Education, 40(7), 988–1001. https://doi.org/10.1080/02602938.2014.957642
de Vreugd, L., Jansen, R., van Leeuwen, A., & van der Schaaf, M. (2023). The role of reference frames in learners’ internal feedback generation with a learning analytics dashboard. Studies in Educational Evaluation, 79, 101303. https://doi.org/10.1016/j.stueduc.2023.101303
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
Dimitriadis, Y., Martínez-Maldonado, R., & Wiley, K. (2021). Human-centered design principles for actionable learning analytics. In T. Tsiatsos, S. Demetriadis, A. Mikropoulos, & V. Dagdilelis (Eds.), Research on e-learning and ICT in education: Technological, pedagogical and instructional perspectives (pp. 277–296). Springer Cham. https://doi.org/10.1007/978-3-030-64363-8_15
Dollinger, M., Liu, D., Arthars, N., & Lodge, J. (2019). Working together in learning analytics towards the co-creation of value. Journal of Learning Analytics, 6(2), 10–26. https://doi.org/10.18608/jla.2019.62.2
Hadwin, A. F., & Järvelä, S. (2011). Self-regulated, co-regulated, and socially shared regulation of learning. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 64–84). Routledge.
Harvey, N., & Holmes, C. A. (2012). Nominal group technique: An effective method for obtaining group consensus. International Journal of Nursing Practice, 18(2), 188–194. https://doi.org/10.1111/j.1440-172X.2012.02017.x
Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice. Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 82–96). Springer Cham. https://doi.org/10.1007/978-3-319-66610-5_7
Kirkpatrick, D. (1996). Great ideas revisited. Training & Development, 50(1), 54–60.
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels (3rd ed.). Berrett-Koehler Publishers.
Lai, K., Cabrera, J., Vitale, J. M., Madhok, J., Tinker, R., & Linn, M. C. (2016). Measuring graph comprehension, critique, and construction in science. Journal of Science Education and Technology, 25, 665–681. https://doi.org/10.1007/s10956-016-9621-9
Lang, C., & Davis, L. (2023). Learning analytics and stakeholder inclusion: What do we mean when we say “human-centered”? Proceedings of the 13th International Conference on Learning Analytics and Knowledge (LAK ’23), 13–17 March 2023, Arlington, TX, USA (pp. 411–417). ACM Press. https://doi.org/10.1145/3576050.3576110
Liborius, P., Bellhäuser, H., & Schmitz, B. (2019). What makes a good study day? An intraindividual study on university students’ time investment by means of time-series analyses. Learning and Instruction, 60, 310–321. https://doi.org/10.1016/j.learninstruc.2017.10.006
Martin, A. J. (2007). Examining a multidimensional model of student motivation and engagement using a construct validation approach. British Journal of Educational Psychology, 77(2), 413–440. https://doi.org/10.1348/000709906X118036
Marzouk, Z., Rakovic, M., Liaqat, A., Vytasek, J., Samadi, D., Stewart-Alonso, J., Ram, I., Woloshen, S., Winne, P. H., & Nesbit, J. C. (2016). What if learning analytics were based on learning science? Australasian Journal of Educational Technology, 32(6). https://doi.org/10.14742/ajet.3058
Matcha, W., Uzir, N. A., Gašević, D., & Pardo, A. (2019). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE Transactions on Learning Technologies, 13(2), 226–245. https://doi.org/10.1109/TLT.2019.2916802
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
Ochoa, X. (2022). Multimodal learning analytics: Rationale, process, examples, and direction. In C. Lang, A. F. Wise, A. Merceron, D. Gašević, & G. Siemens (Eds.), The handbook of learning analytics (2nd ed., pp. 54–65). https://doi.org/10.18608/hla22.006
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, 422. https://doi.org/10.3389/fpsyg.2017.00422
Park, Y., & Jo, I.-H. (2019). Factors that affect the success of learning analytics dashboards. Educational Technology Research and Development, 67(6), 1547–1571. https://doi.org/10.1007/s11423-019-09693-0
Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). Academic Press. https://doi.org/10.1016/B978-012109890-2/50043-3
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). National Center for Research to Improve Postsecondary Teaching and Learning.
Roll, I., & Winne, P. H. (2015). Understanding, evaluating, and supporting self-regulated learning using learning analytics. Journal of Learning Analytics, 2(1), 7–12. https://doi.org/10.18608/jla.2015.21.2
Sarmiento, J. P., & Wise, A. F. (2022). Participatory and co-design of learning analytics: An initial review of the literature. Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK ’22), 21–25 March 2022, Online (pp. 535–541). ACM Press. https://doi.org/10.1145/3506860.3506910
Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychological Bulletin, 143(6), 565–600. https://doi.org/10.1037/bul0000098
Schunk, D. H., Pintrich, P. R., & Meece, J. L. (2008). Motivation in education: Theory, research, and applications (3rd ed.). Pearson.
Steel, P. (2007). The nature of procrastination: A meta-analytic and theoretical review of quintessential self-regulatory failure. Psychological Bulletin, 133(1), 65–94. https://doi.org/10.1037/0033-2909.133.1.65
Teddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. SAGE Publications.
Uysal, M., & Horzum, M. B. (2021). Designing and developing a learning analytics dashboard to support self-regulated learning. In M. Sahin & D. Ifenthaler (Eds.), Visualizations and dashboards for learning analytics (pp. 477–496). Springer Cham. https://doi.org/10.1007/978-3-030-81222-5_22
Valle, N., Antonenko, P., Dawson, K., & Huggins‐Manley, A. C. (2021). Staying on target: A systematic literature review on learner‐facing learning analytics dashboards. British Journal of Educational Technology, 52(4), 1724–1748. https://doi.org/10.1111/bjet.13089
van Leeuwen, A., Teasley, S. D., & Wise, A. F. (2022). Teacher and student facing learning analytics. In C. Lang, A. F. Wise, A. Merceron, D. Gašević, & G. Siemens (Eds.), The handbook of learning analytics (2nd ed., pp. 130–140). https://doi.org/10.18608/hla22.013
Winne, P. H. (2010). Bootstrapping learner’s self-regulated learning. Psychological Test and Assessment Modeling, 52(4), 472–490.
Winne, P. H. (2011). A cognitive and metacognitive analysis of self-regulated learning. In D. H. Schunk & B. Zimmerman (Eds.), Handbook of self-regulation of learning and performance (pp. 29–46). Routledge.
Wiley, K., Dimitriadis, Y., & Linn, M. (2024). A human‐centred learning analytics approach for developing contextually scalable K–12 teacher dashboards. British Journal of Educational Technology, 55(3), 845–885. https://doi.org/10.1111/bjet.13383
Wise, A. F., Vytasek, J. M., Hausknecht, S., & Zhao, Y. (2016). Developing learning analytics design knowledge in the “middle space”: The student tuning model and align design framework for learning analytics use. Online Learning, 20(2), 155–182.
Yigitbasioglu, O. M., & Velcu, O. (2012). A review of dashboards in performance management: Implications for design and research. International Journal of Accounting Information Systems, 13(1), 41–59. https://doi.org/10.1016/j.accinf.2011.08.002
Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13–39). Academic Press. https://doi.org/10.1016/b978-012109890-2/50031-7
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
