Associations between Students’ Standing Seen in Learning Analytics Dashboards and Their Following Learning Behaviours
A Study of Three Reference Frames
DOI:
https://doi.org/10.18608/jla.2024.8547Keywords:
student-facing dashboards, class average, leaderboard, learning behaviour, asynchronous online discussions, research paperAbstract
An essential part of making dashboards more effective in motivating students and leading to desirable behavioural change is knowing what information to communicate to the student and how to frame and present it. Most of the research studying dashboards’ impact on learning analyzes learning indicators of students as a group. Understanding how a student’s learning unfolds after viewing the dashboard is necessary for personalized dashboard selection and its content. In the context of the discussion activity, we analyzed 28,290 actions of 896 students after they saw their learning status on the dashboards, which were integrated into 21 discussions in 11 courses. We provide a comparative perspective on three dashboard types: the class average, the leaderboard, and message-quality dashboards. Our results indicate that students’ behaviours after viewing three dashboards were associated with their displayed standing in the discussion: views showing the student’s status below the frame of reference were associated with a higher likelihood of posting, and views of the student outperforming the norm with diminished further posting, although demonstrating higher discussion engagement. We reiterate a need to understand the impact of dashboard states on students’ behaviour, creating a foundation for a personalized selection of dashboard views based on individual students’ standing.
References
Aghaei, K. (2023). Investigating students’ emotional and motivational responses to dashboards with varying social comparison groups [Master’s thesis, Simon Fraser University].
Aghaei, K., Hatala, M., & Mogharrab, A. (2023). How students’ emotion and motivation changes after viewing dashboards with varied social comparison group: A qualitative study. In Proceedings of the 13th International Conference on Learning Analytics and Knowledge (LAK 2023), 13–17 March 2023, Arlington, Texas, USA (pp. 663–669). ACM. https://doi.org/10.1145/3576050.3576107
Aguilar, S. J. (2022). Experimental evidence of performance feedback vs. mastery feedback on students’ academic motivation. In Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK 2022), 21–25 March 2022, online (pp. 556–562). ACM. https://doi.org/10.1145/3506860.3506916
Aguilar, S. J., & Baek, C. (2019). Motivated information seeking and graph comprehension among college students. In Proceedings of the Ninth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, Arizona, USA (pp. 280–289). ACM. https://doi.org/10.1145/3303772.3303805
Aljohani, N. R., Daud, A., Abbasi, R. A., Alowibdi, J. S., Basheri, M., & Aslam, M. A. (2019). An integrated framework for course adapted student learning analytics dashboard. Computers in Human Behavior, 92, 679–690. https://doi.org/10.1016/j.chb.2018.03.035
Bai, S., Hew, K. F., Sailer, M., & Jia, C. (2021). From top to bottom: How positions on different types of leaderboard may affect fully online student learning performance, intrinsic motivation, and course engagement. Computers and Education, 173, 1–15. https://doi.org/10.1016/j.compedu.2021.104297
Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis (2nd edition). Cambridge University Press. https://doi.org/10.1017/CBO9780511527685
Bakharia, A., Corrin, L., Barba, P. D., Kennedy, G., Gasevic, D., Mulder, R., Williams, D., Dawson, S., & Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 329–338). ACM. https://doi.org/10.1145/2883851.2883944
Balci, S., Secaur, J. M., & Morris, B. J. (2022). Comparing the effectiveness of badges and leaderboards on academic performance and motivation of students in fully versus partially gamified online physics classes (Vol. 27). Springer US. https://doi.org/10.1007/s10639-022-10983-z
Berkowitz, A. D. (2005). Overview of the social norms approach. In L. C. Lederman & L. P. Stewart (Eds.), Changing the culture of college drinking: A socially situated health communication campaign (pp. 193–214). Hamption Press.
Bodily, R., & Verbert, K. (2017a). Review of research on student-facing learning analytics dashboards and educational recommender systems. IEEE Transactions on Learning Technologies, 10(4), 405–418. https://doi.org/10.1109/TLT.2017.2740172
Bodily, R., & Verbert, K. (2017b). Trends and issues in student-facing learning analytics reporting systems research. In Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, British Columbia, Canada (pp. 309–318). ACM. https://doi.org/10.1145/3027385.3027403
Brusilovsky, P., Somyurek, S., Guerra, J., Hosseini, R., Zadorozhny, V., & Durlach, P. J. (2016). Open social student modeling for personalized learning. IEEE Transactions on Emerging Topics in Computing, 4(3), 450–461. https://doi.org/10.1109/TETC.2015.2501243
Cheng, C. K., Par ´e, D. E., Collimore, L. M., & Joordens, S. (2011). Assessing the effectiveness of a voluntary online discussion forum on improving students’ course performance. Computers and Education, 56(1), 253–261. https://doi.org/10.1016/j.compedu.2010.07.024
Corrin, L., & de Barba, P. (2014). Exploring students’ interpretation of feedback delivered through learning analytics dashboards. In B. Hegarty, J. McDonald, & S. -K. Loke (Eds.), Proceedings of the ASCILITE 2014 Conference, 23–26 November 2014, Dunedin, New Zealand (pp. 629–633). ASCILITE Publications. https://doi.org/10.14742/apubs.2014.1300
Davis, D., Chen, G., Jivet, I., Hauff, C., Kizilcec, R. F., & Houben, G. J. (2017). Follow the successful crowd: Raising MOOC completion rates through social comparison at scale? In Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, British Columbia, Canada (pp. 454–463). ACM. https://doi.org/10.1145/3027385.3027411
Dennen, V. P. (2008). Pedagogical lurking: Student engagement in non-posting discussion behavior. Computers in Human Behavior, 24(4), 1624–1633. https://doi.org/10.1016/j.chb.2007.06.003
Dichev, C., & Dicheva, D. (2017). Gamifying education: what is known, what is believed and what remains uncertain: a critical review. International Journal of Educational Technology in Higher Education, 14. https://doi.org/10.1186/s41239-017-0042-5
Elliot, A. J., Murayama, K., & Pekrun, R. (2011). A 3 × 2 achievement goal model. Journal of Educational Psychology, 103(3), 632–648. https://doi.org/10.1037/a0023952
Ferguson, R., & Clow, D. (2017). Where is the evidence? A call to action for learning analytics. In Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, British Columbia, Canada (pp. 56–65). ACM. https://doi.org/10.1145/3027385.3027396
Festinger, L. (1954). A theory of social comparison processes. Human Relations, 7(2), 117–140. https://doi.org/10.1177/001872675400700202
Fleur, D. S., van den Bos, W., & Bredeweg, B. (2020). Learning analytics dashboard for motivation and performance. In Lecture notes in computer science (pp. 411–419, Vol. 12149). Springer International Publishing. https://doi.org/10.1007/978-3-030-49663-0 51
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7–23. https://doi.org/10.1080/08923640109527071
Gasevic, D., Adesope, O., Joksimovic, S., & Kovanovic, V. (2015). Externally-facilitated regulation scaffolding and role assignment to develop cognitive presence in asynchronous online discussions. Internet and Higher Education, 24, 53–65. https://doi.org/10.1016/j.iheduc.2014.09.006
Gerber, J. P., Wheeler, L., & Suls, J. (2018). A social comparison theory meta-analysis 60+ years on. Psychological Bulletin, 144(2), 177–197. https://doi.org/10.1037/bul0000127
Gunther, S. A. (2021). The impact of social norms on students’ online learning behavior: Insights from two randomized controlled trials. In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK 2021), 12–16 April 2021, Irvine, California, USA (pp. 12–21). ACM. https://doi.org/10.1145/3448139.3448141
Han, J., Kim, K. H., Rhee, W., & Cho, Y. H. (2021). Learning analytics dashboards for adaptive support in face-to-face collaborative argumentation. Computers & Education, 163, 104041. https://doi.org/10.1016/j.compedu.2020.104041
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
Hidi, S. E., & Renninger, K. A. (2019). Introduction: Motivation and its relation to learning. In K. A. Renninger & S. E. Hidi (Eds.), The Cambridge handbook of motivation and learning (pp. 1–12). Cambridge University Press. https://doi.org/10.1017/9781316823279.002
Hung, J. -l. (2008). Revealing online learning behaviors and activity patterns and making predictions with data mining techniques in online teaching. MERLOT Journal of Online Learning and Teaching, 4(4), 426–437. https://www.merlot.org/merlot/viewMaterial.htm?id=875638
Jin, S. H. (2021). Educational effects on the transparency of peer participation levels in asynchronous online discussion activities. IEEE Transactions on Learning Technologies, 14(5), 604–612. https://doi.org/10.1109/TLT.2021.3126388
Jivet, I., Scheffel, M., Drachsler, H., & Specht, M. (2017). Awareness is not enough: Pitfalls of learning analytics dashboards in the educational practice. In E. Lavoue, H. Drachsler, K. Verbert, J. Broisin, & M. Perez-Sanagustın (Eds.), Data driven approaches in digital education. EC-TEL 2017. Lecture notes in computer science (pp. 82–96, Vol. 10474). Springer. https://doi.org/10.1007/978-3-319-66610-5 7
Jivet, I., Scheffel, M., Specht, M., & Drachsler, H. (2018). License to evaluate: Preparing learning analytics dashboards for educational practice. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 7–9 March 2018, Sydney, New South Wales, Australia (pp. 31–40). ACM. https://doi.org/10.1145/3170358.3170421
Kaliisa, R., Misiejuk, K., Lopez-Pernas, S., Khalil, M., & Saqr, M. (2024). Have learning analytics dashboards lived up to the hype? A systematic review of impact on students’ achievement, motivation, participation and attitude. In Proceedings of the 14th International Conference on Learning Analytics and Knowledge (LAK 2024), 18–22 March 2024, Kyoto, Japan (pp. 295–304). ACM. https://doi.org/10.1145/3636555.3636884
Kehrwald, B. (2008). Understanding social presence in text-based online learning environments. Distance Education, 29(1), 89–106. https://doi.org/10.1080/01587910802004860
Kia, F. S., Teasley, S. D., Hatala, M., Karabenick, S. A., & Kay, M. (2020). How patterns of students dashboard use are related to their achievement and self-regulatory engagement. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 340–349). ACM. https://doi.org/10.1145/3375462.3375472
Kim, D., Park, Y., Yoon, M., & Jo, I. H. (2016). Toward evidence-based learning analytics: Using proxy variables to improve asynchronous online discussion environments. Internet and Higher Education, 30, 30–43. https://doi.org/10.1016/j.iheduc.2016.03.002
Kim, J., Jo, I. H., & Park, Y. (2016). Effects of learning analytics dashboard: Analyzing the relations among dashboard utilization, satisfaction, and learning achievement. Asia Pacific Education Review, 17(1), 13–24. https://doi.org/10.1007/s12564-015-9403-8
Kovanovic, V., Gasevic, D., Joksimovic, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89. https://doi.org/10.1016/j.iheduc.2015.06.002
Li, Q., Xu, D., Baker, R., Holton, A., & Warschauer, M. (2022). Can student-facing analytics improve online students’ effort and success by affecting how they explain the cause of past performance? Computers & Education, 185, 104517. https://doi.org/10.1016/j.compedu.2022.104517
Loncar, M., Barrett, N. E., & Liu, G. Z. (2014). Towards the refinement of forum and asynchronous online discussion in educational contexts worldwide: Trends and investigative approaches within a dominant research paradigm. Computers and Education, 73, 93–110. https://doi.org/10.1016/j.compedu.2013.12.007
Lonn, S., Aguilar, S. J., & Teasley, S. D. (2015). Investigating student motivation in the context of a learning analytics intervention during a summer bridge program. Computers in Human Behavior, 47, 90–97. https://doi.org/10.1016/j.chb.2014.07.013
Luppicini, R. (2007). Review of computer mediated communication research for education. Instructional Science, 35(2), 141–185. https://doi.org/10.1007/s11251-006-9001-6
Matcha, W., Ahmad Uzir, N., Gasevic, D., & Pardo, A. (2019). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE Transactions on Learning Technologies, 13(2), 226–245. https://doi.org/10.1109/tlt.2019.2916802
Matz, R., Schulz, K., Hanley, E., Derry, H., Hayward, B., Koester, B., Hayward, C., & Mckay, T. (2021). Analyzing the efficacy of ECoach in supporting gateway course success through tailored support. In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK 2021), 12–16 April 2021, Irvine, California, USA (pp. 216– 225). ACM. https://doi.org/10.1145/3448139.3448160
Nieto-Escamez, F. A., & Roldan-Tapia, M. D. (2021). Gamification as online teaching strategy during COVID-19: A mini-review. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.648552
Orji, F. A., & Vassileva, J. (2021). A comparative evaluation of the effect of social comparison, competition, and social learning in persuasive technology on learning. In A. Cristea & C. Troussas (Eds.), Intelligent tutoring systems. ITS 2021. Lecture notes in computer science (pp. 369–375, Vol. 12677). Springer. https://doi.org/10.1007/978-3-030-80421-3 41
Philpott, A., & Son, J. B. (2022). Leaderboards in an EFL course: Student performance and motivation. Computers & Education, 190. https://doi.org/10.1016/j.compedu.2022.104605
Rovai, A. P. (2007). Facilitating online discussions effectively. The Internet and Higher Education, 10(1), 77–88. https://doi.org/10.1016/j.iheduc.2006.10.001
Russell, J. E., Smith, A., & Larsen, R. (2020). Elements of success: Supporting at-risk student resilience through learning analytics. Computers & Education, 152, 103890. https://doi.org/10.1016/j.compedu.2020.103890
Schielzeth, H., & Nakagawa, S. (2013). Nested by design: Model fitting and interpretation in a mixed model era. Methods in Ecology and Evolution, 4(1), 14–24. https://doi.org/10.1111/j.2041-210x.2012.00251.x
Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychological Bulletin, 143(6), 565–600. https://doi.org/10.1037/bul0000098
Schwendimann, B. A., Rodriguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., Gillet, D., & Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41. https://doi.org/10.1109/TLT.2016.2599522
Sedrakyan, G., Malmberg, J., Verbert, K., Jarvela, S., & Kirschner, P. A. (2020). Linking learning behavior analytics and learning science concepts: Designing a learning analytics dashboard for feedback to support learning regulation. Computers in Human Behavior, 107. https://doi.org/10.1016/j.chb.2018.05.004
Sher, V., Hatala, M., & Gasevic, D. (2019). On multi-device use: Using technological modality profiles to explain differences in students’ learning. In Proceedings of the Ninth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, Arizona, USA (pp. 1–10). ACM. https://doi.org/10.1145/3303772.3303790
Shirazi Beheshitha, S., Hatala, M., Gasevic, D., & Joksimovic, S. (2016). The role of achievement goal orientations when studying effect of learning analytics visualizations. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 54–63). ACM. https://doi.org/10.1145/2883851.2883904
Shukor, N. A., Tasir, Z., Van der Meijden, H., & Harun, J. (2014). Exploring students’ knowledge construction strategies in computer-supported collaborative learning discussions using sequential analysis. Educational Technology and Society, 17(4), 216–228. https://www.jstor.org/stable/jeductechsoci.17.4.216
Siadaty, M., Gasevic, D., & Hatala, M. (2015). Measuring the impact of individual technological interventions on micro-level processes of self-regulated workplace learning. Computers in Human Behavior, 59(100), 468–482. https://doi.org/10.1016/j.chb.2016.02.025
Susnjak, T., Ramaswami, G. S., & Mathrani, A. (2022). Learning analytics dashboard: A tool for providing actionable insights to learners. International Journal of Educational Technology in Higher Education, 19(1). https://doi.org/10.1186/s41239-021-00313-7
Tabuenca, B., Kalz, M., Drachsler, H., & Specht, M. (2015). Time will tell: The role of mobile learning analytics in self regulated learning. Computers & Education, 89, 53–74. https://doi.org/10.1016/j.compedu.2015.08.004
Tan, J. P. L., Yang, S., Koh, E., & Jonathan, C. (2016). Fostering 21st century literacies through a collaborative critical reading and learning analytics environment: User-perceived benefits and problematics. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 430–434, Vol. 25-29-Apri). ACM. https://doi.org/10.1145/2883851.2883965
Tan, J. P.-L., Koh, E., Jonathan, C. R., & Yang, S. (2017). Learner dashboards a double-edged sword? Students’ sense-making of a collaborative critical reading and learning analytics environment for fostering 21st century literacies. Journal of Learning Analytics, 4(1), 117–140. https://doi.org/10.18608/jla.2017.41.7
Teasley, S. D. (2017). Student facing dashboards: One size fits all? Technology, Knowledge and Learning, 22(3), 377–384. https://doi.org/10.1007/s10758-017-9314-3
Webb, E., Jones, A., Barker, P., & Van Schaik, P. (2004). Using e-learning dialogues in higher education. Innovations in Education and Teaching International, 41(1), 93–103. https://doi.org/10.1080/1470329032000172748
Weiner, B. (1986). An attributional theory of motivation and emotion. SpringerVerlag. https://doi.org/10.1007/978-1-4612-4948-1
Wise, A., Zhao, Y., & Hausknecht, S. (2014). Learning analytics for online discussions: Embedded and extracted approaches. Journal of Learning Analytics, 1(2), 48–71. https://doi.org/10.18608/jla.2014.12.4
Wise, A. F. (2014). Designing pedagogical interventions to support student use of learning analytics. In Proceedings of the 4th International Conference on Learning Analytics and Knowledge (LAK 2024), 24–28 March 2014, Indianapolis, Indiana, USA (pp. 203–211). ACM. https://doi.org/10.1145/2567574.2567588
Wise, A. F., Speer, J., Marbouti, F., & Hsiao, Y. T. (2013). Broadening the notion of participation in online discussions: Examining patterns in learners’ online listening behaviors. Instructional Science, 41(2), 323–343. https://doi.org/10.1007/s11251-012-9230-9
Xie, K. (2013). What do the numbers say? The influence of motivation and peer feedback on students’ behaviour in online discussions. British Journal of Educational Technology, 44(2), 288–301. https://doi.org/10.1111/j.1467-8535.2012.01291.x
Yoo, M., & Jin, S. H. (2020). Development and evaluation of learning analytics dashboards to support online discussion activities. Educational Technology and Society, 23(2), 1–18. https://www.jstor.org/stable/26921130
Zamecnik, A., Kovanovic, V., Grossmann, G., Joksimovic, S., Jolliffe, G., Gibson, D., & Pardo, A. (2022). Team interactions with learning analytics dashboards. Computers and Education, 185, 104514. https://doi.org/10.1016/j.compedu.2022.104514
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
