Contextualized Logging of On-Task and Off-Task Behaviours During Learning
DOI:
https://doi.org/10.18608/jla.2023.7837Keywords:
off-task detection, logging, tools, distractions, data and tools reportAbstract
Learners use digital media during learning for a variety of reasons. Sometimes media use can be considered “on-task,” e.g., to perform research or to collaborate with peers. In other cases, media use is “off-task,” meaning that learners use content unrelated to their current learning task. Given the well-known problems with self-reported data (incomplete memory, distorted perceptions, subjective attributions), exploring on-task and off-task usage of digital media in learning scenarios requires logging activity on digital devices. However, we argue that logging on- and off-task behaviour has challenges that are rarely addressed. First, logging must be active only during learning. Second, logging represents a potential invasion of privacy. Third, logging must incorporate multiple devices simultaneously to take the reality of media multitasking into account. Fourth, logging alone is insufficient to reveal what prompted learners to switch to a different digital activity. To address these issues, we present a contextually activated logging system that allows users to inspect and annotate the observed activities after a learning session. Data from a formative study show that our system works as intended, and furthermore supports our assumptions about the diverse intentions of media use in learning. We discuss the implications for learning analytics.
References
Beuckels, E., Ye, G., Hudders, L., & Cauberghe, V. (2021). Media multitasking: A bibliometric approach and literature review. Frontiers in Psychology, 12, 623643. https://doi.org/10.3389/fpsyg.2021.623643
Baumgartner, S. E., & Wiradhany, W. (2022). Not all media multitasking is the same: The frequency of media multitasking depends on cognitive and affective characteristics of media combinations. Psychology of Popular Media, 11(1), 1–12. https://doi.org/10.1037/ppm0000338
Biedermann, D., Schneider, J., & Drachsler, H. (2021). Digital self‐control interventions for distracting media multitasking: A systematic review. Journal of Computer Assisted Learning, 37(5), 1217–1231. https://doi.org/10.1111/jcal.12581
Calderwood, C., Ackerman, P. L., & Conklin, E. M. (2014). What else do college students “do” while studying? An investigation of multitasking. Computers & Education, 75, 19–29. https://doi.org/10.1016/j.compedu.2014.02.004
Ciordas-Hertel, G.-P., Rödling, S., Schneider, J., Di Mitri, D., Weidlich, J., & Drachsler, H. (2021). Mobile sensing with smart wearables of the physical context of distance learning students to consider its effects on learning. Sensors, 21(19), 6649. https://doi.org/10.3390/s21196649
Demirci, K., Akgönül, M., & Akpinar, A. (2015). Relationship of smartphone use severity with sleep quality, depression, and anxiety in university students. Journal of Behavioral Addictions, 4(2), 85–92. https://doi.org/10.1556/2006.4.2015.010
Dias da Silva, M. R., & Postma, M. (2020). Wandering minds, wandering mice: Computer mouse tracking as a method to detect mind wandering. Computers in Human Behavior, 112, 106453. https://doi.org/10.1016/j.chb.2020.106453
Dönmez, O., & Akbulut, Y. (2021). Timing and relevance of secondary tasks impact multitasking performance. Computers and Education, 161, 104078D. https://doi.org/10.1016/j.compedu.2020.104078
Drachsler, H., & Greller, W. (2016). Privacy and analytics: It’s a DELICATE issue a checklist for trusted learning analytics. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 89–98). ACM Press. https://doi.org/10.1145/2883851.2883893
Fischer, R., & Plessow, F. (2015). Efficient multitasking: Parallel versus serial processing of multiple tasks. Frontiers in Psychology, 6. https://doi.org/10.3389/fpsyg.2015.01366
Jamet, E., Gonthier, C., Cojean, S., Colliot, T., & Erhel, S. (2020). Does multitasking in the classroom affect learning outcomes? A naturalistic study. Computers in Human Behavior, 106, 106264. https://doi.org/10.1016/j.chb.2020.106264
Jeong, S. H., & Hwang, Y. (2016). Media multitasking effects on cognitive vs. attitudinal outcomes: A meta-analysis. Human Communication Research, 42. https://doi.org/10.1111/hcre.12089
Jürgens, P., Stark, B., & Magin, M. (2020). Two half-truths make a whole? On bias in self-reports and tracking data. Social Science Computer Review, 38(5), 600–615. https://doi.org/10.1177/0894439319831643
Kokolakis, S. (2017). Privacy attitudes and privacy behaviour: A review of current research on the privacy paradox phenomenon. Computers & Security, 64, 122–134. https://doi.org/10.1016/j.cose.2015.07.002
Kovanović, V., Gašević, D., Dawson, S., Joksimović, S., Baker, R. S., & Hatala, M. (2015). Penetrating the black box of time-on-task estimation. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 184–193). ACM Press. https://doi.org/10.1145/2723576.2723623
Laput, G., & Harrison, C. (2019). Sensing fine-grained hand activity with smartwatches. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK (Paper No. 338). ACM Press. https://doi.org/10.1145/3290605.3300568
Leinonen, J., Castro, F. E. V., & Hellas, A. (2022). Time-on-task metrics for predicting performance. Proceedings of the 53rd ACM Technical Symposium on Computer Science Education (SIGCSE ’22), 3–5 March 2022, Providence, RI, USA (pp. 871–877). ACM Press. https://doi.org/10.1145/3478431.3499359
Lin, Y.-H., Fang, C.-H., & Hsu, C.-L. (2014). Determining uses and gratifications for mobile phone apps. In J. J. Park, Y. Pan, C.-S. Kim, & Y. Yang (Eds.), Future information technology (pp. 661–668). Springer. https://doi.org/10.1007/978-3-642-55038-6_103
Liu, Y., Deng, L., Lin, L., & Gu, X. (2021). Patterns of triggers for on-task and off-task behaviors: University students in independent study. Interactive Learning Environments. https://doi.org/10.1080/10494820.2021.1905003
Lorenz, B., Sousa, S., & Tomberg, V. (2013). Privacy awareness of students and its impact on online learning participation: A case study. Proceedings of the IFIP WG 3.4 International Conference (OST 2012), 20 July–3 August 2012, Tallinn, Estonia. Springer. https://doi.org/10.1007/978-3-642-37285-8_21
Lyngs, U., Lukoff, K., Csuka, L., Slovák, P., Van Kleek, M., & Shadbolt, N. (2022). The Goldilocks level of support: Using user reviews, ratings, and installation numbers to investigate digital self-control tools. International Journal of Human–Computer Studies, 166, 102869. https://doi.org/10.1016/j.ijhcs.2022.102869
Lyngs, U., Lukoff, K., Slovak, P., Seymour, W., Webb, H., Jirotka, M., Zhao, J., Van Kleek, M., & Shadbolt, N. (2020). “I just want to hack myself to not get distracted”: Evaluating design interventions for self-control on Facebook. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ’20), 25–30 April 2020, Honolulu, HI, USA (pp. 1–15). ACM Press. https://doi.org/10.1145/3313831.3376672
Makhortykh, M., Urman, A., Gil-Lopez, T., & Ulloa, R. (2022). To track or not to track: Examining perceptions of online tracking for information behavior research. Internet Research, 32(7), 260–279. https://doi.org/10.1108/INTR-01-2021-0074
Masood, A., Luqman, A., Feng, Y., & Ali, A. (2020). Adverse consequences of excessive social networking site use on academic performance: Explaining underlying mechanism from stress perspective. Computers in Human Behavior, 113, 106476. https://doi.org/10.1016/j.chb.2020.106476
May, K. E., & Elder, A. D. (2018). Efficient, helpful, or distracting? A literature review of media multitasking in relation to academic performance. International Journal of Educational Technology in Higher Education, 15, 13. https://doi.org/10.1186/s41239-018-0096-z
May, M., & George, S. (2011). Privacy concerns in e-learning: Is using tracking system a threat? International Journal of Information and Education Technology, 1(1), 1–8. https://doi.org/10.7763/IJIET.2011.V1.1
Ohme, J., Araujo, T., de Vreese, C. H., & Piotrowski, J. T. (2021). Mobile data donations: Assessing self-report accuracy and sample biases with the iOS Screen Time function. Mobile Media & Communication, 9(2), 293–313. https://doi.org/10.1177/2050157920959106
Parry, D. A., Davidson, B. I., Sewall, C. J. R., Fisher, J. T., Mieczkowski, H., & Quintana, D. S. (2021). A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use. Nature Human Behaviour, 5, 1535–1547. https://doi.org/10.1038/s41562-021-01117-5
Parry, D. A., & le Roux, D. B. (2019). Media multitasking and cognitive control: A systematic review of interventions. Computers in Human Behavior, 92, 316–327. https://doi.org/10.1016/j.chb.2018.11.031
Patil, R., Brown, M., Ibrahim, M., Myers, J. L., Brown, K., Khan, M., & Callaway, R. (2019). Digital distraction outside the classroom: An empirical study. Journal of Computing Sciences in Colleges, 34(7), 46–55.
Poll, H. (2015). Pearson student mobile device survey 2015: College students. Pearson.
Popławska, A., Szumowska, E., & Kuś, J. (2021). Why do we need media multitasking? A self-regulatory perspective. Frontiers in Psychology, 12, 624649–624649. https://doi.org/0.3389/fpsyg.2021.624649
Roffarello, A. M., & De Russis, L. (2019). Towards detecting and mitigating smartphone habits. Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers (UbiComp/ISWC ’19), 9–13 September 2019, London, United Kingdom (pp. 149–152). ACM Press. https://doi.org/10.1145/3341162.3343770
Rosen, L. D., Carrier, L. M., & Cheever, N. A. (2013). Facebook and texting made me do it: Media-induced task-switching while studying. Computers in Human Behavior, 29(3), 948–958. https://doi.org/10.1016/j.chb.2012.12.001
Schwarz, N. (2007). Retrospective and concurrent self-reports: The rationale for real-time data capture. In A. A. Stone, S. Shiffman, A. A. Atienza, & L. Nebeling (Eds.), The science of real-time data capture: Self-reports in health research (pp. 11–26). Oxford University Press.
Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510–1529. https://doi.org/10.1177/0002764213479366
Sniehotta, F. F., & Presseau, J. (2012). The habitual use of the self-report habit index. Annals of Behavioral Medicine, 43(1), 139–140. https://doi.org/10.1007/s12160-011-9305-x
Sohn, S. Y., Rees, P., Wildridge, B., Kalk, N. J., & Carter, B. (2019). Prevalence of problematic smartphone usage and associated mental health outcomes amongst children and young people: A systematic review, meta-analysis and GRADE of the evidence. BMC Psychiatry, 19, 356. https://doi.org/10.1186/s12888-019-2350-x
Thorpe, A., Friedman, J., Evans, S., Nesbitt, K., & Eidels, A. (2022). Mouse movement trajectories as an indicator of cognitive workload. International Journal of Human–Computer Interaction, 38(15), 1464–1479. https://doi.org/10.1080/10447318.2021.2002054
Tourangeau, R. (2000). Remembering what happened: Memory errors and survey reports. In A. A. Stone, J. S. Turkkan, C. A. Bachrach, J. B. Jobe, H. S. Kurtzman, & V. S. Cain (Eds.), The science of self-report: Implications for research and practice (pp. 29–47). Lawrence Erlbaum Associates Publishers.
Wang, Z., Irwin, M., Cooper, C., & Srivastava, J. (2015). Multidimensions of media multitasking and adaptive media selection. Human Communications Research, 41, 102–127. https://doi.org/10.1111/hcre.12042
Wood, E., & Zivcakova, L. (2015). Understanding multimedia multitasking in educational settings. In L. D. Rosen, N. A. Cheever, & L. M. Carrier (Eds.), The Wiley Handbook of Psychology, Technology, and Society (pp. 404–419). John Wiley & Sons Ltd. https://doi.org/10.1002/9781118771952.ch23
Yeykelis, L., Cummings, J. J., & Reeves, B. (2014). Multitasking on a single device: Arousal and the frequency, anticipation, and prediction of switching between media content on a computer. Journal of Communication, 64(1), 167–192. https://doi.org/10.1111/jcom.12070
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) license that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
