Interpreting Predictive Learning Sequences in a College Math Course through a Self-Regulated Learning Framework
DOI:
https://doi.org/10.18608/jla.2025.8865Keywords:
self-regulated learning, digital events, sequence mining, think-aloud verbalizations, validation, research paperAbstract
Digital traces have been used to measure self-regulated learning (SRL), yet the validity of inferences made about these traces has often been questioned. Recently, researchers have used multiple channels of data — including digital traces, verbalizations, and self-reports — to validate inferences about individual SRL events. Research on the validation of inferences about sequences of multiple SRL events remains limited; however, investigating these sequences has the potential to refine SRL theories. To study the validation of sequences of SRL events, we collected multimodal data from 49 undergraduates completing a math task in a lab setting. Participants were asked to think aloud while interacting with different digital platforms. Then, we used sequence pattern mining to identify the digital events most predictive of post-test scores. Next, we used student verbalizations during the learning process to validate the inferences about what those predictive sequences reflected. Sequences representing learner conscientiousness predicted better performance; sequences that included pausing and rewinding videos predicted poorer performance. Some learner verbalizations co-occurred with digital events and consistently aligned with SRL processes, providing validity evidence for SRL sequences. Heterogeneity in verbal-to-digital trace alignment emerged and will require methodological advances to validate the sequences specific to individuals and task conditions.
References
Azevedo, R. (2007). Understanding the complex nature of self-regulatory processes in learning with computer-based learning environments: An introduction. Metacognition and Learning, 2, 57–65. https://doi.org/10.1007/s11409-007-9018-5
Azevedo, R., Guthrie, J. T., & Seibert, D. (2004). The role of self-regulated learning in fostering students’ conceptual understanding of complex systems with hypermedia. Journal of Educational Computing Research, 30(1–2), 87–111. https://doi.org/10.2190/DVWX-GM1T-6THQ-5WC7
Baker, R. S., Ocumpaugh, J. L., & Andres, J. M. A. L. (2020). BROMP quantitative field observations: A review. In R. Feldman (Ed.), Learning science: Theory, research, and practice (pp. 127–156). McGraw-Hill.
Bakhshinategh, B., Zaiane, O. R., ElAtia, S., & Ipperciel, D. (2018). Educational data mining applications and tasks: A survey of the last 10 years. Education and Information Technologies, 23(1), 537–553. https://doi.org/10.1007/s10639-017-9616-z
Bannert, M. (2007). Metakognition beim Lernen mit hypermedien [Metacognition in learning with hypermedia]. Waxmann Verlag.
Ben-Eliyahu, A., & Bernacki, M. L. (2015). Addressing complexities in self-regulated learning: A focus on contextual factors, contingencies, and dynamic relations. Metacognition and Learning, 10(1), 1–13. https://doi.org/10.1007/s11409-015-9134-6
Bernacki, M. L. (2018). Examining the cyclical, loosely sequenced, and contingent features of self-regulated learning: Trace data and their analysis. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed.). Routledge. https://doi.org/10.4324/9781315697048-24
Bernacki, M. L., Yu, L., Kuhlmann, S. L., Plumley, R. D., Greene, J. A., Duke, R. F., Freed, R., Hollander-Blackmon, C., & Hogan, K. A. (2025). Using multimodal learning analytics to validate digital traces of self-regulated learning in a laboratory study and predict performance in undergraduate courses. Journal of Educational Psychology, 117(2), 176–205. https://doi.org/10.1037/edu0000890
Bergner, Y., Choi, I., & Castellano, K. E. (2019). Item response models for multiple attempts with incomplete data. Journal of Educational Measurement, 56(2), 415–436. https://doi.org/10.1111/jedm.12214
Bishop, J., & Verleger, M. A. (2013, June). The flipped classroom: A survey of the research. In 2013 ASEE annual conference & exposition (Paper 6219). American Society for Engineering Education. https://peer.asee.org/22585
Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238. https://doi.org/10.18608/jla.2016.32.11
Bock, R. D., & Gibbons, R. D. (2021). Item response theory. John Wiley & Sons. https://doi.org/10.1002/9781119716723
Chalmers, R. P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06
Chen, B., Knight, S., & Wise, A. F. (2018). Critical issues in designing and implementing temporal analytics. Journal of Learning Analytics, 5(1), 1–9. https://doi.org/10.18608/jla.2018.53.1
Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219–243. https://doi.org/10.1080/00461520.2014.965823
Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, 28, 425–474. https://doi.org/10.1007/s10648-015-9320-8
Dong, G., & Pei, J. (2007). Sequence data mining. Springer.
Eddy, S. L., & Hogan, K. A. (2014). Getting under the hood: How and for whom does increasing course structure work? CBE—Life Sciences Education, 13(3), 453–468. https://doi.org/10.1187/cbe.14-03-0050
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. MIT Press. https://doi.org/10.7551/mitpress/5657.001.0001
Fan, Y., Lim, L., van der Graaf, J., Kilgour, J., Raković, M., Moore, J., Molenaar, I., Bannert, M., & Gašević, D. (2022). Improving the measurement of self-regulated learning using multi-channel data. Metacognition and Learning, 17(3), 1025–1055. https://doi.org/10.1007/s11409-022-09304-z
Fan, Y., van der Graaf, J., Lim, L., Raković, M., Singh, S., Kilgour, J., Moore, J., Molenaar, I., Bannert, M., & Gašević, D. (2022). Towards investigating the validity of measurement of self-regulated learning based on trace data. Metacognition and Learning, 17(3), 949–987. https://doi.org/10.1007/s11409-022-09291-1
Fan, Y., Rakovic, M., van der Graaf, J., Lim, L., Singh, S., Moore, J., Molenaar, I., Bannert, M., & Gašević, D. (2023). Towards a fuller picture: Triangulation and integration of the measurement of self‐regulated learning based on trace and think aloud data. Journal of Computer Assisted Learning, 39(4), 1303–1324. https://doi.org/10.1111/jcal.12801
Fan, H., Xu, J., Cai, Z., He, J., & Fan, X. (2017). Homework and students’ achievement in math and science: A 30-year meta-analysis, 1986–2015. Educational Research Review, 20, 35–54. https://doi.org/10.1016/j.edurev.2016.11.003
Ferguson, C. J. (2016). An effect size primer: A guide for clinicians and researchers. In A. E. Kazdin (Ed.), Methodological issues and strategies in clinical research (4th ed., pp. 301–310). American Psychological Association. https://doi.org/10.1037/14805-020
Franklin, J. (2005). The elements of statistical learning: Data mining, inference and prediction. The Mathematical Intelligencer, 27(2), 83–85. https://doi.org/10.1007/BF02985802
Freijeiro‐González, L., Febrero‐Bande, M., & González‐Manteiga, W. (2022). A critical review of LASSO and its derivatives for variable selection under dependence among covariates. International Statistical Review, 90(1), 118–145. https://doi.org/10.1111/insr.12469
Friedman, J. H., Hastie, T., & Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1), 1–22. https://doi.org/10.18637/jss.v033.i01
Giannakos, M., Spikol, D., Di Mitri, D., Sharma, K., Ochoa, X., & Hammad, R. (2022). Introduction to multimodal learning analytics. In M. Giannakos, D. Spikol, D. Di Mitri, K. Sharma, X. Ochoa, & R. Hammad (Eds.), The multimodal learning analytics handbook (pp. 3–28). Springer Cham. https://doi.org/10.1007/978-3-031-08076-0_1
Greene, J. A., & Azevedo, R. (2010). The measurement of learners’ self-regulated cognitive and metacognitive processes while using computer-based learning environments. Educational Psychologist, 45(4), 203–209. https://doi.org/10.1080/00461520.2010.515935
Greene, J. A., Bernacki, M. L., & Hadwin, A. F. (2024). Self-regulation. In P. A. Schutz & K. R. Muis (Eds.), Handbook of educational psychology (4th ed., pp. 314–334). Routledge. https://doi.org/10.4324/9780429433726-17
Greene, J. A., Deekens, V. M., Copeland, D. Z., & Yu, S. (2018). Capturing and modeling self-regulated learning using think-aloud protocols. In D. H. Schunk & J. A. Greene (Eds.), Handbook of self-regulation of learning and performance (2nd ed., pp. 323–337). Routledge. https://doi.org/10.4324/9781315697048-21
Greene, J. A., Hutchison, L. A., Costa, L.-J., & Crompton, H. (2012). Investigating how college students’ task definitions and plans relate to self-regulated learning processing and understanding of a complex science topic. Contemporary Educational Psychology, 37(4), 307–320. https://doi.org/10.1016/j.cedpsych.2012.02.002
Gurung, A., Lin, J., Huang, Z., Borchers, C., Baker, R. S., Aleven, V., & Koedinger, K. R. (2025). Starting seatwork earlier as a valid measure of student engagement. arXiv. https://doi.org/10.48550/arXiv.2505.13341
Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34. https://doi.org/10.20982/tqmp.08.1.p023
He, Q., & von Davier, M. (2016). Analyzing process data from problem-solving items with n-grams: Insights from a computer-based large-scale assessment. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (pp. 750–777). IGI Global. https://doi.org/10.4018/978-1-4666-9441-5.ch029
Hu, J., & Gao, X. (2017). Using think-aloud protocol in self-regulated reading research. Educational Research Review, 22, 181–193. https://doi.org/10.1016/j.edurev.2017.09.004
Karabenick, S. A., Woolley, M. E., Friedel, J. M., Ammon, B. V., Blazevski, J., Bonney, C. R., De Groot, E., Gilbert, M. C., Musu, L., Kempler, T. M., & Kelly, K. L. (2007). Cognitive processing of self-report items in educational research: Do they think what we mean? Educational Psychologist, 42(3), 139–151. https://doi.org/10.1080/00461520701416231
Kasakowskij, R., Haake, J. M., & Seidel, N. (2023). Self-assessment task processing behavior of students in higher education. In M. Feng, T. Käser, & P. Talukdar (Eds.), Proceedings of the 16th international conference on educational data mining (pp. 334–341). International Educational Data Mining Society. https://eric.ed.gov/?id=ED630881
Khosravi, H., Buckingham Shum, S., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022). Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence, 3, Article 100074. https://doi.org/10.1016/j.caeai.2022.100074
Kim, H.-Y. (2017). Statistical notes for clinical researchers: Chi-squared test and Fisher’s exact test. Restorative Dentistry & Endodontics, 42(2), 152–155. https://doi.org/10.5395/rde.2017.42.2.152
Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56(5), 746–759. https://doi.org/10.1177/0013164496056005002
Kline, R. B. (2013). Beyond significance testing: Statistics reform in the behavioral sciences (2nd ed.). American Psychological Association. https://doi.org/10.1037/14136-000
Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798
Kuhlmann, S. L., Plumley, R., Evans, Z., Bernacki, M. L., Greene, J. A., Hogan, K. A., Berro, M., Gates, K., & Panter, A. (2024). Students’ active cognitive engagement with instructional videos predicts STEM learning. Computers & Education, 216, Article 105050. https://doi.org/10.1016/j.compedu.2024.105050
Liao, C.-H., & Wu, J.-Y. (2023). Learning analytics on video-viewing engagement in a flipped statistics course: Relating external video-viewing patterns to internal motivational dynamics and performance. Computers & Education, 197, Article 104754. https://doi.org/10.1016/j.compedu.2023.104754
Maher, J. M., Markey, J. C., & Ebert-May, D. (2013). The other half of the story: Effect size analysis in quantitative research. CBE—Life Sciences Education, 12(3), 345–351. https://doi.org/10.1187/cbe.13-04-0082
Maldonado-Mahauad, J., Pérez-Sanagustín, M., Kizilcec, R. F., Morales, N., & Munoz-Gama, J. (2018). Mining theory-based patterns from Big data: Identifying self-regulated learning strategies in Massive Open Online Courses. Computers in Human Behavior, 80, 179–196. https://doi.org/10.1016/j.chb.2017.11.011
Malekian, D., Bailey, J., & Kennedy, G. (2020). Prediction of students’ assessment readiness in online learning environments: The sequence matters. In C. Rensing, H. Drachsler, V. Kovanović, N. Pinkwart, M. Scheffel, & K. Vertbert (Eds.), LAK ’20: Proceedings of the tenth international conference on learning analytics & knowledge (pp. 382–391). ACM Press. https://doi.org/10.1145/3375462.3375468
McHugh, M. L. (2013). The Chi-square test of independence. Biochemia Medica, 21(2), 143–149. https://doi.org/10.11613/BM.2013.018
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). Macmillan Publishing.
Molenaar, I., & Järvelä, S. (2014). Sequential and temporal characteristics of self and socially regulated learning. Metacognition and Learning, 9, 75–85. https://doi.org/10.1007/s11409-014-9114-2
Molenaar, I., & Wise, A. F. (2022). Temporal aspects of learning analytics: Grounding analyses in concepts of time. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), The handbook of learning analytics (pp. 66–76). Society for Learning Analytics Research. https://doi.org/10.18608/hla22.007
Moos, D. C., & Bonde, C. (2016). Flipping the classroom: Embedding self-regulated learning prompts in videos. Technology, Knowledge and Learning, 21, 225–242. https://doi.org/10.1007/s10758-015-9269-1
Nath, D., Gašević, D., Fan, Y., & Rajendran, R. (2024). CTAM4SRL: A consolidated temporal analytic method for analysis of self-regulated learning. In S. Joksimović & A. Zamecnik (Eds.), LAK ’24: Proceedings of the 14th learning analytics and knowledge conference (pp. 645–655). ACM Press. https://doi.org/10.1145/3636555.3636926
Nicewander, W. A. (2018). Conditional reliability coefficients for test scores. Psychological Methods, 23(2), 351–362. https://doi.org/10.1037/met0000132
Nystrand, M., Wu, L. L., Gamoran, A., Zeiser, S., & Long, D. A. (2003). Questions in time: Investigating the structure and dynamics of unfolding classroom discourse. Discourse Processes, 35(2), 135–198. https://doi.org/10.1207/S15326950DP3502_3
Olson, S., & Riordan, D. G. (2012). Engage to excel: Producing one million additional college graduates with degrees in science, technology, engineering, and mathematics [Report to the President]. Executive Office of the President. https://eric.ed.gov/?id=ed541511
Osakwe, I., Chen, G., Fan, Y., Rakovic, M., Singh, S., Molenaar, I., & Gašević, D. (2024). Measurement of self-regulated learning: Strategies for mapping trace data to learning processes and downstream analysis implications. In S. Joksimović & A. Zamecnik (Eds.), LAK ’24: Proceedings of the 14th learning analytics and knowledge conference (pp. 563–575). ACM Press. https://doi.org/10.1145/3636555.3636915
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the motivated strategies for learning questionnaire (MSLQ) [Technical Report No. 91-B-004]. National Center for Research to Improve Postsecondary Teaching and Learning. https://eric.ed.gov/?id=ED338122
Plumley, R. D., Bernacki, M. L., Greene, J. A., Kuhlmann, S., Raković, M., Urban, C. J., Hogan, K. A., Lee, C., Panter, A. T., & Gates, K. M. (2024). Co‐designing enduring learning analytics prediction and support tools in undergraduate biology courses. British Journal of Educational Technology, 55(5), 1860–1883. https://doi.org/10.1111/bjet.13472
Russell, D., & Winston, B. (2014). Tapping into the interpreting process: Using participant reports to inform the interpreting process in educational settings. Translation & Interpreting, 6(1), 102–127. https://doi.org/10.12807/ti.106201.2014.a06
Salehian Kia, F., Hatala, M., Baker, R. S., & Teasley, S. D. (2021). Measuring students’ self-regulatory phases in LMS with behavior and real-time self report. In M. Scheffel, N. Dowell, S. Joksimović, & G. Siemens (Eds.), LAK21: 11th international learning analytics and knowledge conference (pp. 259–268). ACM Press. https://doi.org/10.1145/3448139.3448164
Sebesta, A. J., & Speth, E. B. (2017). How should I study for the exam? Self-regulated learning strategies and achievement in introductory biology. CBE—Life Sciences Education, 16(2), Article 30. https://doi.org/10.1187/cbe.16-09-0269
Seo, K., Dodson, S., Harandi, N. M., Roberson, N., Fels, S., & Roll, I. (2021). Active learning with online video: The impact of learning context on engagement. Computers & Education, 165, Article 104132. https://doi.org/10.1016/j.compedu.2021.104132
Shekhar, P., Borrego, M., DeMonbrun, M., Finelli, C., Crockett, C., & Nguyen, K. (2020). Negative student response to active learning in STEM classrooms. Journal of College Science Teaching, 49(6), 45–54. https://doi.org/10.1080/0047231X.2020.12290664
Siegler, R. S., & Crowley, K. (1991). The microgenetic method: A direct means for studying cognitive development. American Psychologist, 46(6), 606–620. https://doi.org/10.1037/0003-066X.46.6.606
Siemens, G. (2013). Learning analytics: The emergence of a discipline. American Behavioral Scientist, 57(10), 1380–1400. https://doi.org/10.1177/0002764213498851
Theobald, E. J., Hill, M. J., Tran, E., Agrawal, S., Arroyo, E. N., Behling, S., Chambwe, N., Cintrón, D. L., Cooper, J. D., Dunster, G., Grummer, J. A., Hennessey, K., Hsiao, J., Iranon, N., Jones, L., II., Jordt, H., Keller, M., Lacey, M. E., Littlefield, C. E., … Freeman, S. (2020). Active learning narrows achievement gaps for underrepresented students in undergraduate science, technology, engineering, and math. Proceedings of the National Academy of Sciences, 117(12), 6476–6483. https://doi.org/10.1073/pnas.1916903117
Ulitzsch, E., He, Q., & Pohl, S. (2022). Using sequence mining techniques for understanding incorrect behavioral patterns on interactive tasks. Journal of Educational and Behavioral Statistics, 47(1), 3–35. https://doi.org/10.3102/10769986211010467
Ulitzsch, E., Ulitzsch, V., He, Q., & Lüdtke, O. (2023). A machine learning-based procedure for leveraging clickstream data to investigate early predictability of failure on interactive tasks. Behavior Research Methods, 55(3), 1392–1412. https://doi.org/10.3758/s13428-022-01844-1
Weinberg, A., & Thomas, M. (2018). Student learning and sense-making from video lectures. International Journal of Mathematical Education in Science and Technology, 49(6), 922–943. https://doi.org/10.1080/0020739X.2018.1426794
Wickham, H. (2023). stringr: Simple, consistent wrappers for common string operations (Version 1.5.0) [Computer software]. Posit Software. https://CRAN.R-project.org/package=stringr
Wiedbusch, M., Dever, D., Li, S., Amon, M. J., Lajoie, S., & Azevedo, R. (2023). Measuring multidimensional facets of SRL engagement with multimodal data. In V. Kovanovic, R. Azevedo, D. C. Gibson, & D. Ifenthaler (Eds.), Unobtrusive observations of learning in digital environments: Examining behavior, cognition, emotion, metacognition and social processes using learning analytics (pp. 141–173). Springer Cham. https://doi.org/10.1007/978-3-031-30992-2_10
Winne, P. H. (2017). Learning analytics for self-regulated learning. In C. Lang, G. Siemens, A. Wise, D. Gašević (Eds.), Handbook of learning analytics (pp. 241–249). Society for Learning Analytics Research. https://doi.org/10.18608/hla17.021
Winne, P. H. (2020). Construct and consequential validity for learning analytics based on trace data. Computers in Human Behavior, 112, Article 106457. https://doi.org/10.1016/j.chb.2020.106457
Winne, P. H. (2022). Learning analytics for self-regulated learning. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), The handbook of learning analytics (pp. 78–85). Society for Learning Analytics Research. https://doi.org/10.18608/hla22.008
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Lawrence Erlbaum Associates.
Winne, P. H., & Hadwin, A. F. (2008). The weave of motivation and self-regulated learning. In D. H. Schunk & B. J. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297–314). Lawrence Erlbaum Associates.
Winne, P. H., Jamieson-Noel, D., & Muis, K. (2002). Methodological issues and advances in researching tactics, strategies, and self-regulated learning. In P. R. Pintrich & M. L. Maehr (Eds.), New directions in measures and methods (pp. 121–155). Advances in motivation and achievement volume 12. Emerald Publishing.
Wise, A. F., & Shaffer, D. W. (2015). Why theory matters more than ever in the age of big data. Journal of Learning Analytics, 2(2), 5–13. https://doi.org/10.18608/jla.2015.22.2
Zhang, J., Borchers, C., Aleven, V., & Baker, R. S. (2024). Using large language models to detect self-regulated learning in think-aloud protocols. In B. Paaßen & C. D. Epp (Eds.), Proceedings of the 17th international conference on educational data mining (Paper 13). International Educational Data Mining Society. https://doi.org/10.5281/zenodo.12729790
Zhang, Y., & Paquette, L. (2023). Sequential pattern mining in educational data: The application context, potential, strengths, and limitations. In A. Peña-Ayala (Ed.), Educational data science: Essentials, approaches, and tendencies (pp. 219–254). Springer Singapore. https://doi.org/10.1007/978-981-99-0026-8_6
Zimmerman, B. J. (1986). Becoming a self-regulated learner: Which are the key subprocesses? Contemporary Educational Psychology, 11(4), 307–313. https://doi.org/10.1016/0361-476X(86)90027-5
Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2), 301–320. https://doi.org/10.1111/j.1467-9868.2005.00503.x
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
