Examining the Interplay of Gaze and Verbal Interactions in Socially Shared Regulation of Learning
A Transmodal Analysis (TMA) Study
DOI:
https://doi.org/10.18608/jla.2025.8661Keywords:
multimodal learning analytics, socially shared regulation of learning, transmodal analysis, research paperAbstract
This paper presents a transmodal analysis (TMA) study that investigates the interplay between gaze and verbal interactions through regulation-triggering events within the context of socially shared regulation of learning (SSRL) in face-to-face collaborative settings with shared computer-mediated materials. In face-to-face collaborative learning environments, gaze interaction serves as a pivotal mechanism for both directing attention and communicating emotional states. This aspect of non-verbal communication is critical for aligning cognitive processes and establishing an emotional connection among learners, thereby improving the effectiveness of the learning environment. Despite substantial research on verbal interactions in SSRL, the role of non-verbal cues, particularly gaze, has been less studied. This study examines 3,523 gaze and verbal interactions from twenty-four high school students engaged in eight collaborative learning groups. This study demonstrates the application of TMA method, which reveals epistemic networks across various data modalities, each operating on different temporal scales. Furthermore, our study highlights the critical role of gaze interactions in conveying cognitive and emotional cues, enhancing verbal communication for SSRL.
References
Azevedo, R., & Gašević, D. (2019). Analyzing multimodal multichannel data about self-regulated learning with advanced learning technologies: Issues and challenges. Computers in Human Behavior, 96, 207–210. https://doi.org/10.1016/j.chb.2019.03.025
Bakhtiar, A., Webster, E. A., & Hadwin, A. F. (2018). Regulation and socio-emotional interactions in a positive and a negative group climate. Metacognition and Learning, 13(1), 57–90. https://doi.org/10.1007/s11409-017-9178-x
Boekaerts, M., & Pekrun, R. (2016). Emotions and emotion regulation in academic settings. In L. Corno & E. M. Anderman (Eds.), Handbook of educational psychology (3rd ed., pp. 76–90). Routledge. https://doi.org/10.4324/9781315688244
Borchers, C., Wang, Y., Karumbaiah, S., Ashiq, M., Shaffer, D. W., & Aleven, V. (2024). Revealing networks: Understanding effective teacher practices in AI-supported classrooms using transmodal ordered network analysis. Proceedings of the 14th Learning Analytics and Knowledge Conference (LAK ’24), 18–22 March 2024, Kyoto, Japan (pp. 371–381). ACM Press. https://doi.org/10.1145/3636555.3636892
Crasborn, O., Sloetjes, H., Auer, E., & Wittenburg, P. (2006). Combining video and numeric data in the analysis of sign languages with the ELAN annotation software. In C. Vettori (Ed.), Proceedings of the LREC2006 2nd Workshop on the Representation and Processing of Sign Languages: Lexicographic Matters and Didactic Scenarios (LREC2006), 28 May 2006, Genoa, Italy (pp. 82–87). European Language Resources Association (ELRA).
Dang, B., Nguyen, A., & Järvelä, S. (2024). The unspoken aspect of socially shared regulation in collaborative learning: AI-driven learning analytics unveiling “silent pauses.” Proceedings of the 14th Learning Analytics and Knowledge Conference (LAK ’24), 18–22 March 2024, Kyoto, Japan (pp. 231–240). ACM Press. https://doi.org/10.1145/3636555.3636874
De Backer, L., Van Keer, H., De Smedt, F., Merchie, E., & Valcke, M. (2022). Identifying regulation profiles during computer-supported collaborative learning and examining their relation with students’ performance, motivation, and self-efficacy for learning. Computers & Education, 179, 104421. https://doi.org/10.1016/j.compedu.2021.104421
Ertl, B., Fischer, F., & Mandl, H. (2006). Conceptual and socio-cognitive support for collaborative learning in videoconferencing environments. Computers & Education, 47(3), 298–315. https://doi.org/10.1016/j.compedu.2004.11.001
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2024). The promise and challenges of generative AI in education. Behaviour & Information Technology. https://doi.org/10.1080/0144929X.2024.2394886
Hadwin, A. F. (2021). Commentary and future directions: What can multi-modal data reveal about temporal and adaptive processes in self-regulated learning? Learning and Instruction, 72, 101287. https://doi.org/10.1016/j.learninstruc.2019.101287
Hadwin, A., Järvelä, S., & Miller, M. (2018). Self-regulation, co-regulation, and shared regulation in collaborative learning environments. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 83–106). Routledge.
Huang, X., & Lajoie, S. P. (2023). Social emotional interaction in collaborative learning: Why it matters and how can we measure it? Social Sciences & Humanities Open, 7(1), 100447. https://doi.org/10.1016/j.ssaho.2023.100447
Isohätälä, J., Näykki, P., & Järvelä, S. (2020). Cognitive and socio-emotional interaction in collaborative learning: Exploring fluctuations in students’ participation. Scandinavian Journal of Educational Research, 64(6), 831–851. https://doi.org/10.1080/00313831.2019.1623310
Järvelä, S., & Bannert, M. (2021). Temporal and adaptive processes of regulated learning: What can multimodal data tell? Learning and Instruction, 72, 101268. https://doi.org/10.1016/j.learninstruc.2019.101268
Järvelä, S., & Hadwin, A. (2024). Triggers for self-regulated learning: A conceptual framework for advancing multimodal research about SRL. Learning and Individual Differences, 115, 102526. https://doi.org/10.1016/j.lindif.2024.102526
Järvelä, S., Nguyen, A., & Hadwin, A. (2023). Human and artificial intelligence collaboration for socially shared regulation in learning. British Journal of Educational Technology, 54(5), 1057–1076. https://doi.org/10.1111/bjet.13325
Kang, J., Zhou, Y., Rajarathinam, R. J., Tan, Y., & Shaffer, D. W. (2024). Unveiling joint attention dynamics: Examining multimodal engagement in an immersive collaborative astronomy simulation. Computers & Education, 213, 105002. https://doi.org/10.1016/j.compedu.2024.105002
Molenaar, I., de Mooij, S., Azevedo, R., Bannert, M., Järvelä, S., & Gašević, D. (2023). Measuring self-regulated learning and the role of AI: Five years of research using multimodal multichannel data. Computers in Human Behavior, 139, 107540. https://doi.org/10.1016/j.chb.2022.107540
Molenaar, I., Roda, C., van Boxtel, C., & Sleegers, P. (2012). Dynamic scaffolding of socially regulated learning in a computer-based learning environment. Computers & Education, 59(2), 515–523. https://doi.org/10.1016/j.compedu.2011.12.006
Nguyen, A., Hong, Y., Dang, B., & Nguyen, P. T. B. (2022). Emotional regulation in synchronous online collaborative learning: A facial expression recognition study. ICIS 2022 Proceedings, 9–14 December 2022, Copenhagen, Denmark. Association for Information Systems. https://aisel.aisnet.org/icis2022/learning_iscurricula/learning_iscurricula/12
Nguyen, A., Järvelä, S., Rosé, C., Järvenoja, H., & Malmberg, J. (2023). Examining socially shared regulation and shared physiological arousal events with multimodal learning analytics. British Journal of Educational Technology, 54(1), 293–312. https://doi.org/10.1111/bjet.13280
Nguyen, A., Li, Y., Järvelä, S., Liu, Y., & Zhao, G. (2023). Emotional synchrony through regulatory triggers in collaborative learning: A facial expression recognition study. Proceedings of the 17th International Conference of the Learning Sciences (ICLS 2023), 10–15 June 2023, Montreal, QC, Canada (pp. 1038–1041). International Society of the Learning Sciences. https://doi.org/10.22318/icls2023.547768
Purwati, P., Rahmiati, R., Asra, S., & Fadlia, F. (2019). The impact of non-verbal communication in the learning process. Journal of Education, Linguistics, Literature and Language Teaching, 2(2), 1–12.
Richardson, D. C., & Dale, R. (2005). Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension. Cognitive Science, 29(6), 1045–1060. https://doi.org/10.1207/s15516709cog0000_29
Ruis, A. R., Tan, Y., Brohinsky, J., Yang, B., Wang, Y., Cai, Z., & Shaffer, D. W. (2023). Thin data, thick description: Modeling socio-environmental problem-solving trajectories in localized land-use simulations. In G. Arastoopour Irgens & S. Knight (Eds.), Advances in quantitative ethnography: 5th international conference, ICQE 2023, Melbourne, VIC, Australia, October 8–12, 2023, proceedings (pp. 349–364). Springer Cham. https://doi.org/10.1007/978-3-031-47014-1_24
Schneider, B., & Pea, R. (2017). Real-time mutual gaze perception enhances collaborative learning and collaboration quality. In M. Orey & R. M. Branch (Eds.), Educational media and technology yearbook (Vol. 40, pp. 99–125). Springer Cham. https://doi.org/10.1007/978-3-319-45001-8_7
Schneider, B., Sharma, K., Cuendet, S., Zufferey, G., Dillenbourg, P., & Pea, R. (2018). Leveraging mobile eye-trackers to capture joint visual attention in co-located collaborative learning groups. International Journal of Computer-Supported Collaborative Learning, 13(3), 241–261. https://doi.org/10.1007/s11412-018-9281-2
Shaffer, D. W., Wang, Y., Marquart, C., Ruis, A. R., & Cai, Z. (2025). Transmodal analysis. Journal of Learning Analytics, 1–22. https://doi.org/10.18608/jla.2025.8423.
Tan, Y., Ruis, A. R., Marquart, C., Cai, Z., Knowles, M. A., & Shaffer, D. W. (2023). Ordered network analysis. In C. Damşa & A. Barany (Eds.), Advances in quantitative ethnography: 4th international conference, ICQE 2022, Copenhagen, Denmark, October 15–19, 2022, proceedings (pp. 101–116). Springer Cham. https://doi.org/10.1007/978-3-031-31726-2_8
Volet, S., Summers, M., & Thurman, J. (2009). High-level co-regulation in collaborative learning: How does it emerge and how is it sustained? Learning and Instruction, 19(2), 128–143. https://doi.org/10.1016/j.learninstruc.2008.03.001
Vuorenmaa, E., Järvelä, S., Dindar, M., & Järvenoja, H. (2023). Sequential patterns in social interaction states for regulation in collaborative learning. Small Group Research, 54(4), 512–550. https://doi.org/10.1177/10464964221137524
Wang, Y., Shah, M., Jimenez, F. A., Wilson, C., Ashiq, M., Eagan, B., & Shaffer, D. W. (2023). Developing nursing students’ practice readiness with Shadow Health® Digital Clinical Experiences™: A transmodal analysis. In G. Arastoopour Irgens & S. Knight (Eds.), Advances in quantitative ethnography: 5th international conference, ICQE 2023, Melbourne, VIC, Australia, October 8–12, 2023, proceedings (pp. 365–380). Springer Cham. https://doi.org/10.1007/978-3-031-47014-1_25
Westland, G. (2015). Verbal and non-verbal communication in psychotherapy. W. W. Norton.
Winne, P. H. (2022). Modeling self-regulated learning as learners doing learning science: How trace data and learning analytics help develop skills for self-regulated learning. Metacognition and Learning, 17(3), 773–791. https://doi.org/10.1007/s11409-022-09305-y
Yan, L., Tan, Y., Swiecki, Z., Gašević, D., Shaffer, D. W., Zhao, L., Li, X., & Martinez-Maldonado, R. (2023). Characterizing individual-level collaborative learning behaviors using ordered network analysis and wearable sensors. In G. Arastoopour Irgens & S. Knight (Eds.), Advances in quantitative ethnography: 5th international conference, ICQE 2023, Melbourne, VIC, Australia, October 8–12, 2023, proceedings (pp. 66–80). Springer Cham. https://doi.org/10.1007/978-3-031-47014-1_5
Zhang, J., Borchers, C., & Barany, A. (2024). Studying the interplay of self-regulated learning cycles and scaffolding through ordered network analysis across three tutoring systems. In Y. J. Kim & Z. Swiecki (Eds.), Advances in quantitative ethnography: 6th international conference, ICQE 2024, Philadelphia, PA, USA, November 3–7, 2024, proceedings, part I (pp. 231–246). Springer Cham. https://doi.org/10.1007/978-3-031-76335-9_17
Zhou, Q., Bhattacharya, A., Suraworachet, W., Nagahara, H., & Cukurova, M. (2023). Automated detection of students’ gaze interactions in collaborative learning videos: A novel approach. In O. Viberg, I. Jivet, P. J. Muñoz-Merino, M. Perifanou, & T. Papathoma (Eds.), Responsive and sustainable educational futures: 18th European conference on technology enhanced learning, EC-TEL 2023, Aveiro, Portugal, September 4–8, 2023, proceedings (pp. 504–517). Springer Cham. https://doi.org/10.1007/978-3-031-42682-7_34
Zimmerman, B. J. (2011). Motivational sources and outcomes of self-regulated learning and performance. In D. H. Schunk & B. Zimmerman (Eds.), Handbook of self-regulation of learning and performance (pp. 49–64). Routledge. https://doi.org/10.4324/9780203839010
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
