Students' Ethical, Privacy, Design, and Cultural Perspectives on Visualizing Cognitive-Affective States in Online Learning
DOI:
https://doi.org/10.18608/jla.2024.8483Keywords:
human-Centeredness, affective computing, online learning, emotions, research paperAbstract
While teachers often monitor and adjust their learning design based on students' emotional states in physical classrooms, synchronous online environments often limit their ability to perceive the emotional climate of the class. Drawing from the concept of \textit{social translucence}, it is suggested that making students' emotional states ``visible" in online settings can help foster empathetic interactions. Recent advancements in emotion recognition technology are enabling the creation of learning analytics (LA) systems that can estimate students' cognitive-affective states in real time. Yet, the adoption of these systems raises ethical, cultural, and practical concerns when implemented in learning environments, including potential challenges related to accuracy, privacy, and data integrity. To address these concerns, we conducted an in-depth qualitative study exploring the perspectives of 12 undergraduate students on modelling and visualizing their cognitive-affective states in the context of a Mexican higher-education institution. The study provides insights within a particular cultural context, which can guide the design of more human-centred emotion recognition--based LA tools for online educational contexts or contribute to informed decisions about the necessity of modelling and visualizing cognitive states.
References
Alfredo, R. D., Nie, L., Kennedy, P., Power, T., Hayes, C., Chen, H., McGregor, C., Swiecki, Z., Gasevic, D., & Martinez-Maldonado, R. (2023). “That student should be a lion tamer!” StressViz: Designing a stress analytics dashboard for teachers. In Proceedings of the 13th International Conference on Learning Analytics and Knowledge (LAK 2023), 13–17 March 2023, Arlington, Texas, USA (pp. 57–67). ACM. https://doi.org/10.1145/3576050.3576058
Ali, W. (2020). Online and remote learning in higher education institutes: A necessity in light of COVID-19 pandemic. Higher Education Studies, 10(3), 16–25. https://doi.org/10.5539/hes.v10n3p16
Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2022). The evidence of impact and ethical considerations of multimodal learning analytics: A systematic literature review. In G. Michail, S. Daniel, D. M. Daniele, K. Sharma, O. Xavier, & H. Rawad (Eds.), The multimodal learning analytics handbook (pp. 289–325). Springer International Publishing. https://doi.org/10.1007/978-3-031-08076-0_12
Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., & Christopherson, R. (2009). Emotion sensors go to school. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Artificial intelligence in education, Vol. 200 of Frontiers in artificial intelligence and applications (pp. 17–24). Ios Press. https://doi.org/10.3233/978-1-60750-028-5-17
Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241. https://doi.org/10.1016/j.ijhcs.2009.12.003
Balaam, M. (2013). A part practical and part conceptual response to orchestration. Computers & Education, 69, 517–519. https://doi.org/10.1016/j.compedu.2013.04.009
Banfield, R., Lombardo, C. T., & Wax, T. (2015). Design sprint: A practical guidebook for building great digital products. O’Reilly Media, Inc. https://www.oreilly.com/library/view/design-sprint/9781491923160/
Banzon, A. M., Beever, J., & Taub, M. (2023). Facial expression recognition in classrooms: Ethical considerations and proposed guidelines for affect detection in educational settings. IEEE Transactions on Affective Computing, 15(1), 93–104. https://doi.org/10.1109/TAFFC.2023.3275624
Barsade, S. G. (2002). The ripple effect: Emotional contagion and its influence on group behavior. Administrative Science Quarterly, 47(4), 644–675. https://doi.org/10.2307/3094912
Beardsley, M., Vujovic, M., Portero-Tresserra, M., & Hernández-Leo, D. (2019). ClassMood app: A classroom orchestration tool for identifying and influencing student moods. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou, & J. Schneider (Eds.), Transforming learning with meaningful technologies. EC-TEL 2019. Lecture notes in computer science (pp. 723–726, Vol. 11722). Springer. https://doi.org/10.1007/978-3-030-29736-7_75
Boyd, K. L., & Andalibi, N. (2023). Automated emotion recognition in the workplace: How proposed technologies reveal potential futures of work. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1). https://doi.org/10.1145/3579528
Brackett, M. A., Reyes, M. R., Rivers, S. E., Elbertson, N. A., & Salovey, P. (2011). Classroom emotional climate, teacher affiliation, and student conduct. The Journal of Classroom Interaction, 46(1), 27–36. https://www.jstor.org/stable/23870549
Braun, V., & Clarke, V. (2021). Thematic analysis: A practical guide. SAGE Publications.
Caballero-Morales, S. -O. (2013). Recognition of emotions in Mexican Spanish speech: An approach based on acoustic modelling of emotion-specific vowels. The Scientific World Journal, 2013, 162093. https://doi.org/10.1155/2013/162093
Cai, Y., Li, X., & Li, J. (2023). Emotion recognition using different sensors, emotion models, methods and datasets: A comprehensive review. Sensors, 23(5), 2455. https://doi.org/10.3390/s23052455
Carrillo, A. J., & Jackson, M. (2022). Follow the leader? A comparative law study of the EU’s General Data Protection Regulation’s impact in Latin America. ICL Journal, 16(2), 177–262. https://doi.org/10.1515/icl-2021-0037
Cavazos, J. G., Phillips, P. J., Castillo, C. D., & O’Toole, A. J. (2020). Accuracy comparison across face recognition algorithms: Where are we on measuring race bias? IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(1), 101–111. https://doi.org/10.1109/TBIOM.2020.3027269
Crawford, K. (2021). Affect. In The atlas of AI: Power, politics, and the planetary costs of artificial intelligence (pp. 151–180). Yale University Press.
Dawood, A., Turner, S., & Perepa, P. (2018). Affective computational model to extract natural affective states of students with Asperger syndrome (AS) in computer-based learning environment. IEEE Access, 6, 67026–67034. https://doi.org/10.1109/ACCESS.2018.2879619
Derick, L., Sedrakyan, G., Munoz-Merino, P. J., Delgado Kloos, C., & Verbert, K. (2017). Evaluating emotion visualizations using AffectVis, an affect-aware dashboard for students. Journal of Research in Innovative Teaching & Learning, 10(2), 107–125. https://doi.org/10.1108/JRIT-05-2017-0011
D’Errico, F., Paciello, M., & Cerniglia, L. (2016). When emotions enhance students’ engagement in e-learning processes. Journal of e-Learning and Knowledge Society, 12(4). https://doi.org/10.20368/1971-8829/1144
Dimitriadis, Y., Mart´ınez-Maldonado, R., & Wiley, K. (2021). Human-centered design principles for actionable learning analytics. In T. Tsiatsos, S. Demetriadis, A. Mikropoulos, & V. Dagdilelis (Eds.), Research on e-learning and ICT in education: Technological, pedagogical and instructional perspectives (pp. 277–296). Springer International Publishing. https://doi.org/10.1007/978-3-030-64363-8_15
D’Mello, S., & Graesser, A. (2011). The half-life of cognitive-affective states during complex learning [PMID: 21942577]. Cognition and Emotion, 25(7), 1299–1308. https://doi.org/10.1080/02699931.2011.613668
D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. https://doi.org/10.1016/j.learninstruc.2011.10.001
D’Mello, S., & Jensen, E. (2022). Emotional learning analytics. In C. Lang, G. Siemens, A. F. Wise, D. Gasevic, & A. Merceron (Eds.), The handbook of learning analytics (2nd ed., pp. 120–129). SoLAR. https://doi.org/10.18608/hla22
D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153–170. https://doi.org/10.1016/j.learninstruc.2012.05.003
Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3-4), 169–200. https://doi.org/10.1080/02699939208411068
Erickson, T., & Kellogg, W. A. (2000). Social translucence: An approach to designing systems that support social processes. ACM Transactions on Computer–Human Interaction (TOCHI), 7(1), 59–83. https://doi.org/10.1145/344949.345004
Ez-Zaouia, M., & Lavoué, E. (2017). EMODA: A tutor oriented multimodal and contextual emotional dashboard. In Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, British Columbia, Canada (pp. 429–438). ACM. https://doi.org/10.1145/3027385.3027434
Ez-Zaouia, M., Tabard, A., & Lavoué, E. (2020). EMODASH: a dashboard supporting retrospective awareness of emotions in online learning. International Journal of Human-Computer Studies, 139, 102411. https://doi.org/10.1016/j.ijhcs.2020.102411
GhasemAghaei, R., Arya, A., & Biddle, R. (2016). A dashboard for affective e-learning: Data visualization for monitoring online learner emotions. In G. Veletsianos (Ed.), Proceedings of the 2016 World Conference on Educational Media and Technology (EdMedia+ 2016), 28–30 June 2016, Vancouver, British Columbia, Canada (pp. 1536–1543). Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/173153/
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D. -H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J.,. . . Bengio, Y. (2013). Challenges in representation learning: A report on three machine learning contests. In M. Lee, A. Hirose, Z. Hou, & R. Kil (Eds.), Neural information processing. ICONIP 2013. Lecture notes in computer science (pp. 117–124, Vol. 8228). Springer. https://doi.org/10.1007/978-3-642-42051-1_16
Graesser, A., Chipman, P., King, B., McDaniel, B., & D’Mello, S. (2007). Emotions and learning with AutoTutor. Proceedings of the 2007 Conference on Artificial Intelligence in Education: Building Technology Rich Learning Contexts That Work, 9–13 July 2007, Los Angeles, California, USA, 569–571. https://dl.acm.org/doi/10.5555/1563601.1563699
Gupta, S., Kumar, P., & Tekchandani, R. K. (2023). Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models. Multimedia Tools and Applications, 82(8), 11365–11394. https://doi.org/10.1007/s11042-022-13558-9
Harley, J. M., Bouchet, F., Hussain, M. S., Azevedo, R., & Calvo, R. (2015). A multi-componential analysis of emotions during complex learning with an intelligent multi-agent system. Computers in Human Behavior, 48, 615–625. https://doi.org/10.1016/j.chb.2015.02.013
Hilliger, I., Ceballos, H. G., Maldonado-Mahauad, J., & Ferreira, R. (2024). Applications of learning analytics in Latin America. Journal of Learning Analytics, 11(1), 1–5. https://doi.org/10.18608/jla.2024.8409
Jerrim, J. (2023). Who responds to phishing emails? An international investigation of 15-year-olds using PISA data. British Journal of Educational Studies, 71(6), 701–724. https://doi.org/10.1080/00071005.2023.2234456
Katirai, A. (2023). Ethical considerations in emotion recognition technologies: A review of the literature. AI and Ethics, 4, 927–948. https://doi.org/10.1007/s43681-023-00307-3
Li, S., Deng, W., & Du, J. (2019). Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. IEEE Transactions on Image Processing, 28(1), 356–370. https://doi.org/10.1109/TIP.2018.2868382
Liu, Q., & Khalil, M. (2023). Understanding privacy and data protection issues in learning analytics using a systematic review. British Journal of Educational Technology, 54(6), 1715–1747. https://doi.org/10.1111/bjet.13388
Lyons, M., Kamachi, M., & Gyoba, J. (2019, September). The Japanese Female Facial Expression (JAFFE) Dataset. Zenodo. https://doi.org/10.5281/zenodo.3451524
Madan, A., & Kumar, D. (2024). CNN-based models for emotion and sentiment analysis using speech data. ACM Transactions on Asian and Low-Resource Language Information Processing, 23(10). https://doi.org/10.1145/3687303
Madiega, T. (2021). Artificial intelligence act. European Parliament: European Parliamentary Research Service. https://www.iisf.ie/files/UserFiles/cybersecurity-legislation-ireland/EU-AI-Act.pdf
Mangaroska, K., Sharma, K., Gasevic, D., & Giannakos, M. (2022). Exploring students’ cognitive and affective states during problem solving through multimodal data: Lessons learned from a programming activity. Journal of Computer Assisted Learning, 38(1), 40–59. https://doi.org/10.1111/jcal.12590
Martinez-Maldonado, R. (2023). Human-centred learning analytics: Four challenges in realising the potential. Learning Letters, 1, 6. https://doi.org/10.59453/FIZJ7007
McDaniel, B., D’Mello, S., King, B., Chipman, P., Tapp, K., & Graesser, A. (2007). Facial features for affective state detection in learning environments. In Proceedings of the Annual Meeting of the Cognitive Science Society, 1–4 August 2007, Nashville, Tennessee, USA (Vol. 29). Cognitive Science Society. https://escholarship.org/uc/item/9w00945d
McDonald, N., Schoenebeck, S., & Forte, A. (2019). Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on Human–Computer Interaction, 3(CSCW). https://doi.org/10.1145/3359174
Mollahosseini, A., Hasani, B., & Mahoor, M. H. (2017). AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1), 18–31. https://doi.org/10.1109/TAFFC.2017.2740923
Mutimukwe, C., Viberg, O., Oberg, L.- M., & Cerratto-Pargman, T. (2022). Students’ privacy concerns in learning analytics: Model development. British Journal of Educational Technology, 53(4), 932–951. https://doi.org/10.1111/bjet.13234 Nielsen, J. (2019). Usability for senior citizens: Improved, but still lacking (2013). Nielsen Norman Group. https://www.nngroup.com/articles/usability-seniorsimprovements
Obergriesser, S., & Stoeger, H. (2020). Students’ emotions of enjoyment and boredom and their use of cognitive learning strategies—How do they affect one another? Learning and Instruction, 66, 101285. https://doi.org/10.1016/j.learninstruc.2019.101285
Pardo, A., & Siemens, G. (2014). Ethical and privacy principles for learning analytics. British Journal of Educational Technology, 45(3), 438–450. https://doi.org/10.1111/bjet.12152
Prinsloo, P., Slade, S., & Khalil, M. (2023). Multimodal learning analytics—in-between student privacy and encroachment: A systematic review. British Journal of Educational Technology. https://doi.org/10.1111/bjet.13373
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. arXiv. https://doi.org/10.48550/ARXIV.2212.04356
Radhakrishnan, M., Kandappu, T., Gulati, M., & Misra, A. (2023). Wearables for in-situ monitoring of cognitive states: Challenges and opportunities. In 2023 IEEE International Conference on Pervasive Computing and Communications Workshops and Other Affiliated Events (PerCom Workshops), 13-17 March 2023, Atlanta, Georgia, USA (pp. 671–676). IEEE. https://doi.org/10.1109/PerComWorkshops56833.2023.10150270
Rets, I., Herodotou, C., & Gillespie, A. (2023). Six practical recommendations enabling ethical use of predictive learning analytics in distance education. Journal of Learning Analytics, 10(1), 149–167. https://doi.org/10.18608/jla.2023.7743
Reyes, M. R., Brackett, M. A., Rivers, S. E., White, M., & Salovey, P. (2012). Classroom emotional climate, student engagement, and academic achievement. Journal of Educational Psychology, 104(3), 700. https://doi.org/10.1037/a0027268
Roy, M. L., Malathi, D., & Dorathi Jayaseeli, J. (2022). Facial recognition techniques and their applicability to student concentration assessment: A survey. In G. Manogaran, A. Shanthini, & G. Vadivu (Eds.), Proceedings of international conference on deep learning, computing and intelligence. Advances in intelligent systems and computing (pp. 213–225, Vol. 1396). Springer. https://doi.org/10.1007/978-981-16-5652-1_18
Ruiz, S., Charleer, S., Urretavizcaya, M., Klerkx, J., Fern ´andez-Castro, I., & Duval, E. (2016). Supporting learning by considering emotions: Tracking and visualization a case study. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 254–263). ACM. https://doi.org/10.1145/2883851.2883888
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145–172. https://doi.org/10.1037/0033-295x.110.1.145
Savchenko, A. V., Savchenko, L. V., & Makarov, I. (2022). Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Transactions on Affective Computing, 13(4), 2132–2143. https://doi.org/10.1109/TAFFC.2022.3188390
Schrauf, R. W., & Sanchez, J. (2004). The preponderance of negative emotion words in the emotion lexicon: A cross-generational and cross-linguistic study. Journal of Multilingual and Multicultural Development, 25(2-3), 266–284. https://doi.org/10.1080/01434630408666532
Sedrakyan, G., Mannens, E., & Verbert, K. (2019). Guiding the choice of learning dashboard visualizations: Linking dashboard design and data visualization concepts. Journal of Computer Languages, 50, 19–38. https://doi.org/10.1016/j.jvlc.2018.11.002
Son, G., & Kwon, S. (2024). Spontaneous speech emotion recognition based on spectrogram with convolutional neural network. The Transactions of the Korea Information Processing Society, 13(6), 284–290. https://doi.org/10.3745/TKIPS.2024.13.6.284
Tarmazdi, H., Vivian, R., Szabo, C., Falkner, K., & Falkner, N. (2015). Using learning analytics to visualise computer science teamwork. In Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education (ITICSE ’15), 4–8 July 2015, Vilnius, Lithuania (pp. 165–170). ACM. https://doi.org/10.1145/2729094.2742613
Tsai, Y. - S., Perrotta, C., & Gasevic, D. (2020). Empowering learners with personalised learning approaches? Agency, equity and transparency in the context of learning analytics. Assessment & Evaluation in Higher Education, 45(4), 554–567. https://doi.org/10.1080/02602938.2019.1676396
Tsai, Y.- S., Whitelock-Wainwright, A., & Gasevic, D. (2020). The privacy paradox and its implications for learning analytics. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 230–239). ACM. https://doi.org/10.1145/3375462.3375536
Tyng, C. M., Amin, H. U., Saad, M. N., & Malik, A. S. (2017). The influences of emotion on learning and memory. Frontiers in Psychology, 8, 1454. https://doi.org/10.3389/fpsyg.2017.01454
van Leeuwen, A., & Rummel, N. (2020). Comparing teachers’ use of mirroring and advising dashboards. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 26–34). ACM. https://doi.org/10.1145/3375462.3375471
Viberg, O., Jivet, I., & Scheffel, M. (2023). Designing culturally aware learning analytics: A value sensitive perspective. In O. Viberg & A. Gronlund (Eds.), Practicable learning analytics (pp. 177–192). Springer International Publishing. https://doi.org/10.1007/978-3-031-27646-0_10
Wass, R., Timmermans, J., Harland, T., & McLean, A. (2020). Annoyance and frustration: Emotional responses to being assessed in higher education. Active Learning in Higher Education, 21(3), 189–201. https://doi.org/10.1177/1469787418762462
Weissberg, R. P., Durlak, J. A., Domitrovich, C. E., & Gullotta, T. P. (2015). Social and emotional learning: Past, present, and future. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp. 3–19). The Guilford Press.
Wiedbusch, M. D., Kite, V., Yang, X., Park, S., Chi, M., Taub, M., & Azevedo, R. (2021). A theoretical and evidence-based conceptual design of MetaDash: An intelligent teacher dashboard to support teachers’ decision making and students’ self-regulated learning. Frontiers in Education, 6, 570229. https://doi.org/10.3389/feduc.2021.570229
Yan, L., Zhao, L., Gasevic, D., & Martinez-Maldonado, R. (2022). Scalability, sustainability, and ethicality of multimodal learning analytics. In Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK 2022), 21–25 March 2022, online (pp. 13–23). ACM. https://doi.org/10.1145/3506860.3506862
Yan, Wu, N., Iliyasu, A. M., Kawamoto, K., & Hirota, K. (2022). Framework for identifying and visualising emotional atmosphere in online learning environments in the COVID-19 era. Applied Intelligence, 52(8), 9406–9422. https://doi.org/10.1007/s10489-021-02916-z
Yarmand, M., Solyst, J., Klemmer, S., & Weibel, N. (2021). “It feels like I am talking into a void”: Understanding interaction gaps in synchronous online classrooms. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI 2021), 8–13 May 2021, Yokohama, Japan. ACM. https://doi.org/10.1145/3411764.3445240
Zheng, J., Huang, L., Li, S., Lajoie, S. P., Chen, Y., & Hmelo-Silver, C. E. (2021). Self-regulation and emotion matter: A case study of instructor interactions with a learning analytics dashboard. Computers & Education, 161, 104061. https://doi.org/10.1016/j.compedu.2020.104061
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
