Analyzing and Visualizing Learning Data: A System Designer's Perspective




difficulty, similarity, response time, temporal patterns, biases, cheating


In this work, we consider learning analytics for primary and secondary schools from the perspective of the designer of a learning system. We provide an overview of practically useful analytics techniques with descriptions of their applications and specific illustrations. We highlight data biases and caveats that complicate the analysis and its interpretation. Although we intentionally focus on techniques for internal use by designers, many of these techniques may inspire the development of dashboards for teachers or students. We also identify the consequences and challenges for research.


Aleven, V., McLaughlin, E. A., Glenn, R. A., & Koedinger, K. R. (2016). Instruction based on adaptive learning technologies. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of Research on Learning and Instruction. New York: Routledge.

Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185–224. Retrieved from

Baker, R. S. (2016). Stupid tutoring systems, intelligent humans. International Journal of Artificial Intelligence in Education, 26(2), 600–614.

Brennan, R. L. (1972). A generalized upper-lower item discrimination index. Educational and Psychological Measurement, 32(2), 289–303.

Bull, S., & Kay, J. (2007). Student models that invite the learner in: The SMILI open learner modelling framework. International Journal of Artificial Intelligence in Education, 17(2), 89–120. Retrieved from

Clow, D. (2012). The learning analytics cycle: Closing the loop effectively. In Proceedings of the Second International Conference on Learning Analytics and Knowledge (LAK 2012), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 134–138). New York: ACM.

Corrigan-Gibbs, H., Gupta, N., Northcutt, C., Cutrell, E., & Thies, W. (2015). Deterring cheating in online environments. ACM Transactions on Computer-Human Interaction (TOCHI), 22(6), 1–23.

Davidson, J., Liebald, B., Liu, J., Nandy, P., Van Vleet, T., Gargi, U., . . . Sampath, D. (2010). The YouTube video recommendation system. In Proceedings of the Fourth ACM Conference on Recommender Systems (RecSys 2010), 26–30 September 2010, Barcelona, Spain (pp. 293–296). New York: ACM.

De Ayala, R. J. (2013). The Theory and Practice of Item Response Theory. New York: Guilford Publications.

Heffernan, N. T., & Heffernan, C. L. (2014). The ASSISTments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education, 24(4), 470–497.

Inventado, P. S., Inventado, S. G. F., Matsuda, N., Li, Y., Scupelli, P., Ostrow, K., . . . McGuire, P. (2018). Using design patterns for math preservice teacher education. In Proceedings of the 23rd European Conference on Pattern Languages of Programs (EuroPLoP 2018), 4–8 July 2018, Irsee, Germany (pp. 1–8). New York: ACM.

Joachims, T., Granka, L., Pan, B., Hembrooke, H., & Gay, G. (2017). Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005), 15–19 August 2005, Salvador, Brazil (Vol. 51, pp. 4–11). New York: ACM.

Klinkenberg, S., Straatemeier, M., & Van der Maas, H. (2011). Computer adaptive practice of Maths ability using a new item response model for on the fly ability and difficulty estimation. Computers & Education, 57(2), 1813–1824.

Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5), 757–798.

Koenigstein, N., Dror, G., & Koren, Y. (2011). Yahoo! music recommendations: Modeling music ratings with temporal dynamics and item taxonomy. In Proceedings of the Fifth ACM Conference on Recommender Systems (RecSys 2011), 23–27 October 2011, Chicago, IL, USA (pp. 165–172). New York: ACM.

Koren, Y., & Bell, R. (2015). Advances in collaborative filtering. In F. Ricci, L. Rokach, B. Shapira, & P. B. Kantor (Eds.), Recommender Systems Handbook (pp. 77–118). Springer.

Molenaar, I., & Knoop-van Campen, C. (2017). Teacher dashboards in practice: Usage and impact. In Proceedings of the 2017 European Conference on Technology Enhanced Learning (ECTEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 125–138). Springer.

Moreno-Torres, J. G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N. V., & Herrera, F. (2012). A unifying view on dataset shift in classification. Pattern Recognition, 45(1), 521–530. https:/

Northcutt, C. G., Ho, A. D., & Chuang, I. L. (2016). Detecting and preventing “multiple-account” cheating in massive open online courses. Computers & Education, 100, 71–80.

Pelánek, R. (2017). Bayesian knowledge tracing, logistic models, and beyond: An overview of learner modeling techniques. User Modeling and User-Adapted Interaction, 27(3), 313–350.

Pelánek, R. (2018). The details matter: Methodological nuances in the evaluation of student models. User Modeling and User-Adapted Interaction, 28(3), 207–235.

Pelánek, R. (2020a). Managing items and knowledge components: Domain modeling in practice. Educational Technology Research and Development, 68(1), 529–550.

Pelánek, R. (2020b). Measuring similarity of educational items: An overview. IEEE Transactions on Learning Technologies, 13(2), 354–366.

Pelánek, R., & Rihák, J. (2018). Analysis and design of mastery learning criteria. New Review of Hypermedia and Multimedia, 24(3), 133–159.

Pelánek, R., & Rihák, J. (2016). Properties and applications of wrong answers in online educational systems. In Proceedings of the Ninth International Conference on Educational Data Mining (EDM 2016), 29 June–2 July 2016, Raleigh, NC, USA (pp. 466–471). Retrieved from

Ruipérez-Valiente, J. A., Muñoz-Merino, P. J., Leony, D., & Kloos, C. D. (2015). ALAS-KA: A learning analytics extension for better understanding the learning process in the Khan Academy platform. Computers in Human Behavior, 47, 139–148.

Van Der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46(3), 247–272.

Wang, Y., Heffernan, N. T., & Heffernan, C. (2015). Towards better affect detectors: Effect of missing skills, class features and common wrong answers. In J. Baron, G. Lynch, N. Maziarz, P. Blikstein, A. Merceron, & G. Siemens (Eds.), Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 31–35). New York: ACM.

Wauters, K., Desmet, P., & Van Den Noortgate, W. (2012). Item difficulty estimation: An auspicious collaboration between data and judgment. Computers & Education, 58(4), 1183–1193.

Xhakaj, F., Aleven, V., & McLaren, B. M. (2017). Effects of a teacher dashboard for an intelligent tutoring system on teacher knowledge, lesson planning, lessons and student learning. In Proceedings of the 2017 European Conference on Technology Enhanced Learning (ECTEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 315–329). Springer.




How to Cite

Pelanek, R. (2021). Analyzing and Visualizing Learning Data: A System Designer’s Perspective. Journal of Learning Analytics, 8(2), 93-104.



Special Section: Learning Analytics for Primary and Secondary Schools