Automating the Detection of Reflection-on-Action


  • Jenny Saucerman
  • A. R. Ruis
  • David Williamson Shaffer



student reflection, reflection-on-action, epistemic frame theory, virtual internship


Learning to solve complex problems — problems whose solutions require the application of more than basic facts and skills — is critical to meaningful participation in the economic, social, and cultural life of the digital age. In this paper, we use a theoretical understanding of how professionals use reflection-in-action to solve complex problems to investigate how students learn this critical 21st-century skill and how we can develop and automate learning analytic techniques to assess that learning. We present a preliminary study examining the automated detection of reflective discourse during collaborative, complex problem solving. We analyze student reflection-on-action in a virtual learning environment, focusing on both reflection in individual discourse and collaborative reflection among students. Our results suggest that it is possible to detect student reflection on complex problems in virtual learning environments, but that different models may be appropriate depending on students’ prior domain experience.


Andrzejewski, D., Zhu, X., & Craven, M. (2009). Incorporating domain knowledge into topic modeling via Dirichlet forest priors. Proceedings of the 26th Annual International Conference on Machine Learning (ICML ’09), 14–18 June 2009, Montreal, Quebec, Canada (pp. 25–32). New York: ACM.

Arastoopour, G., Chesler, N. C., & Shaffer, D. W. (2014). Epistemic persistence: A simulation-based approach to increasing participation of women in engineering. Journal of Women and Minorities in Science and Engineering, 20(3), 211–234.

Arastoopour, G., & Shaffer, D. W. (2015). Epistemography and professional CSCL environment design. Exploring the material conditions of learning: The Computer Supported Collaborative Learning (CSCL) Conference 2015, 7–11 June 2015, Gothenberg, Sweden (Vol. 1, pp. 2014–211). International Society of the Learning Sciences.

Arts, J. A. R., Gijselaers, W. H., & Boshuizen, H. P. A. (2006). Understanding managerial problem-solving, knowledge use and information processing: Investigating stages from school to the workplace. Contemporary Educational Psychology, 31(4), 387–410.

Autor, D. H., Levy, F., & Murnane, R. J. (2003a). Computer-based technological change and skill demands: Reconciling the perspectives of economists and sociologists. In E. Applebaum, A. D. Bernhardt, & R. J. Murnane, Low-wage America: How employers are reshaping opportunity in the workplace (pp. 121–154). New York: Russell Sage Foundation.

Autor, D. H., Levy, F., & Murnane, R. J. (2003b). The skill content of recent technological change: An empirical exploration. Quarterly Journal of Economics, 118(4), 1279–1333.

Bagley, E. (2010). The epistemography of an urban and regional planning practicum: Appropriation in the face of resistance. WCER Working Paper 2010-8. Wisconsin Center for Education Research, University of Wisconsin–Madison.

Blei, D. M., & Lafferty, J. D. (2009). Topic models. In A. N. Srivastava & M. Sahami (Eds.), Text mining: Classification, clustering, and applications (pp. 71–94). Boca Raton, FL: CRC Press.

Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42.

Campione, J. C., Shapiro, A. M., & Brown, A. L. (1995). Forms of transfer in a community of learners: Flexible learning and understanding. In A. McKeough, J. Lupart, & A. Marini (Eds.), Teaching for transfer: Fostering generalization in learning (pp. 35–69). Mahwah, NJ: Erlbaum

Califf, M. E., & Mooney, R. J. (2003). Bottom-up relational learning of pattern matching rules for information extraction. The Journal of Machine Learning Research, 4, 177–210.

Chesler, N. C., Ruis, A. R., Collier, W., Swiecki, Z., Arastoopour, G., & Shaffer, D. W. (2015). A novel paradigm for engineering education: Virtual internships with individualized mentoring and assessment of engineering thinking. Journal of Biomechanical Engineering, 137(2), 024701:1-8.

Coutinho, S., Wiemer-Hastings, K., Skowronski, J. J., & Britt, M. A. (2005). Metacognition, need for cognition and use of explanations during ongoing learning and problem solving. Learning and Individual Differences, 15(4), 321–337.

Desautel, D. (2009). Becoming a thinking thinker: Metacognition, self-reflection, and classroom practice. The Teachers College Record, 111(8), 1997–2020.

DiSessa, A. A. (1988). Knowledge in pieces. In G. Forman & P. Pufall (Eds.), Constructivism in the computer age (pp. 47–70). Hillsdale, NJ: Erlbaum.

Dodman, D., McGranahan, G., & Dalal-Clayton, B. (2013). Integrating the environment in urban planning and management: Key principles and approaches for cities in the 21st century. Geneva, Switzerland: United Nations Environmental Programme.

Dorogovtsev, S. N., & Mendes, J. F. (2003). Evolution of networks: From biological nets to the Internet and WWW. Oxford, UK: Oxford University Press.

Dumais, S., Furnas, G., Landauer, T., Deerwester, S., & Harshman, R. (1988). Using latent semantic analysis to improve access to textual information. In J. J. O’Hare (Ed.), Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ98), 4–9 April 2009, Boston, MA, USA (pp. 281–285). Washington, DC: Human Factors in Computing Systems.

Eraut, M. (1994). Developing professional knowledge and competence. London: Falmer.

Fogarty, R. (1994). The mindful school: How to teach for metacognitive reflection. Palatine, IL: Skylight Publishing.

Friedman, T. L. (2006). The world is flat [updated and expanded]: A brief history of the twenty-first century. New York: Macmillan.

Gama, C. (2004). Metacognition in interactive learning environments: The reflection assistant model. In J. C. Lester, R. M. Vicari, & F. Paraguaçu (Eds.), Proceedings of the 7th International Conference on Intelligent Tutoring Systems (ITS 2004), 30 August–3 September 2004, Maceió, Alagoas, Brazil (pp. 668–677). Springer: Berlin Heidelberg.

Gibson, A., & Kitto, K. (2015). Analysing reflective text for learning analytics: An approach using anomaly recontextualisation. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 275–279). New York: ACM.

Goodwin, C. (1994). Professional vision. American Anthropologist, 96(3), 606–633.

Graesser, A. C., Foltz, P. W., Rosen, Y., Shaffer, D. W., Forsyth, C., & Germany, M. (in press). Challenges of assessing collaborative problem solving. In E. Care, P. Griffin, & M. Wilson (Eds.), Assessment and Teaching of 21st Century Skills. Springer.

Grant, A. M. (2001). Rethinking psychological mindedness: Metacognition, self-reflection, and insight. Behaviour Change, 18(1), 8–17.

Hacker, D. J., Dunlosky, J., & Graesser, A. C. (Eds.). (2009). Handbook of metacognition in education. Mahwah, NJ: Erlbaum/Taylor & Francis.

Hatfield, D. L., & Shaffer, D. W. (2010). The epistemography of journalism 335: Complexity in developing journalistic expertise. Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010), 29 June–2 July 2010, Chicago, IL, USA (Vol. 1, pp. 628–635). International Society of the Learning Sciences.

Hatfield, D., Shaffer, D. W., Bagley, E. S., Nulty, A., & Nash, P. (2008). Reflection in professional play. Proceedings of the 8th International Conference of the Learning Sciences (ICLS ’08), 24–28 June 2008, Utrecht, Netherlands (Vol. 3, pp. 245–252). International Society of the Learning Sciences.

Howard, D. V. (1983). Cognitive psychology: Memory, language, and thought. New York: Macmillan.

i Cancho, R. F., & Solé, R. V. (2001). The small world of human language. Proceedings of the Royal Society of London. Series B: Biological Sciences, 268(1482), 2261–2265.

Kim, Y. R., Park, M. S., Moore, T. J., & Varma, S. (2013). Multiple levels of metacognition and their elicitation through complex problem-solving tasks. The Journal of Mathematical Behavior, 32(3), 377–396.

Kitchener, K. S. (1983). Cognition, metacognition, and epistemic cognition: A three level model of cognitive processing. Human Development, 26, 222–232.

Landauer, T. K., McNamara, D. S., Dennis, S., & Kintsch, W. (2007). Handbook of latent semantic analysis. Mahwah, NJ: Erlbaum.

Lave, J. (1988). The culture of acquisition and the practice of understanding. (IRL report 88-00087). Palo Alto, CA: Institute for Research on Learning.

Levy, F., & Murnane, R. J. (2004). The new division of labor. Princeton, NJ: Princeton University Press.

Linn, M. C., Eylon, B.-S., & Davis, E. A. (2004). The knowledge integration perspective on learning. In M. C. Linn, E. A. Davis, & P. Bell (Eds.), Internet environments for science education (pp. 29–46). Mahwah, NJ: Erlbaum.

Lund, K., & Burgess, C. (1996). Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers, 28(2), 203–208.

McAleese, R. (1998). The knowledge arena as an extension to the concept map: Reflection in action. Interactive Learning Environments, 6(3), 251–272.

Madani, A., Vassiliou, M. C., Watanabe, Y., Al-Halabi, B., Al-Rowais, M. S., Deckelbaum, D. L., … Feldman, L. S. (2017). What are the principles that guide behaviors in the operating room? Creating a framework to define and measure performance. Annals of Surgery, 265(2), 255–267.

Miller, T. M., & Geraci, L. (2011). Training metacognition in the classroom: The influence of incentives and feedback on exam predictions. Metacognition and Learning, 6(3), 303–314.

Murnane, R. J., & Levy, F. (1993). Why today’s high-school-educated males earn less than their fathers did: The problem and an assessment of responses. Harvard Educational Review, 63(1), 1–20.

Nash, P., & Shaffer, D. W. (2013). Epistemic trajectories: Mentoring in a game design practicum. Instructional Science, 41(4), 745–771. http:/

Newman, M. E. (2004). Analysis of weighted networks. Physical Review E, 70(5), 056131.

Novak, J. D., & Cañas, A. J. (2006). The theory underlying concept maps and how to construct them. Pensacola, FL: Florida Institute for Human and Machine Cognition.

Palinscar, A. S., & Brown, A. L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 1(2), 117–175.

Powell, W. W., & Snellman, K. (2004). The knowledge economy. Annual Review of Sociology, 199–220.

Ruckelshaus, C., & Leberstein, S. (2014). Manufacturing low pay: Declining wages in the jobs that built America’s middle class.

Rupp, A. A., Gushta, M., Mislevy, R. J., & Shaffer, D. W. (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. The Journal of Technology, Learning and Assessment, 8(4).

Schön, D. A. (1983). The reflective practitioner: How professionals think in action. London: Temple Smith.

Schön, D. A. (1984). The architectural studio as an exemplar of education for reflection-in-action. Journal of Architectural Education, 38(1), 2–9.

Schön, D. A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco, CA: Jossey-Bass.

Schön, D. A. (1992). The crisis of professional knowledge and the pursuit of an epistemology of practice. Journal of Interprofessional Care, 6(1), 49–63.

Schön, D. A. (1995). Knowing-in-action: The new scholarship requires a new epistemology. Change, November/December, 27–34.

Shaffer, D. W. (2006). Epistemic frames for epistemic games. Computers and Education, 46(3), 223–234.

Shaffer, D. W. (2012). Models of situated action: Computer games and the problem of transfer. In C. Steinkuehler, K. Squire, & S. Barab (Eds.), Games, learning, and society: Learning and meaning in the digital age (pp. 403–433). Cambridge, UK: Cambridge University Press

Shaffer, D. W., Hatfield, D., Svarovsky, G., Nash, P. Nulty, A., Bagley, E., Franke, K., Rupp, A. A., & Mislevy, J. R. (2009). Epistemic network analysis: A prototype for 21st century assessment of learning. The International Journal of Learning and Media, 1(2), 33–53.

Shaffer, D. W., Rogers, B., Eagan, B. R., & Marquart, C. (2016). rhoR: Rho for inter-rater reliability [Computer software].

Siebert-Evenstone, A. L., Arastoopour, G., Collier, W., Swiecki, Z., Ruis, A. R., & Shaffer, D. W. (2016). In search of conversational grain size: Modeling semantic structure using moving stanza windows. In C.‑K. Looi, J. Polman, U. Cress, & P. Reimann (Eds.), Transforming learning, empowering learners: Proceedings of the 12th International Conference of the Learning Sciences (ICLS ’16), 20–24 June 2016, Singapore (Vol. 1, pp. 631–638).

Southavilay, V., Yacef, K., Reimann, P., & Calvo, R. A. (2013). Analysis of collaborative writing processes using revision maps and probabilistic topic models. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 38–47). New York: ACM.

Spector, J. M., Merrill, M. D., Elen, J., & Bishop, M. J. (2013). Handbook of research on educational communications and technology. Springer Science & Business.

Svarovsky, G. N., & Shaffer, D. W. (2006). Design meetings and design notebooks as tools for reflection in the engineering design course. Proceedings of the 36th Annual Frontiers in Education Conference (FIE 2006), 27 October–1 November 2006, San Diego, CA, USA (pp. 7–12). IEEE.

Tang, J., Meng, Z., Nguyen, X., Mei, Q., & Zhang, M. (2014). Understanding the limiting factors of topic modeling via posterior contraction analysis. Proceedings of the 31st International Conference on Machine Learning, PMLR, 32(1), 190–198.

Ullmann, T. D., Wild, F., & Scott, P. (2012). Comparing automatically detected reflective texts with human judgements. Proceedings of the 2nd Workshop on Awareness and Reflection in Technology-Enhanced Learning (AR-TEL ʼ12), 18 September 2013, Saarbrucken, Germany (pp. 101–116).

Usher, R., & Bryant, I. (1997). Adult education and the postmodern challenge. London: Routledge.

Velardi, P., Fabriani, P., & Missikoff, M. (2001). Using text processing techniques to automatically enrich a domain ontology. Proceedings of the International Conference on Formal Ontology in Information Systems (FOIS ’01), 17–19 October 2001, Ogunquit, Maine, USA (pp. 270–284). New York: ACM.

Veenman, M. V., Van Hout-Wolters, B. H., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1(1), 3–14.

Weeber, M., Klein, H., de Jong‐van den Berg, L., & Vos, R. (2001). Using concepts in literature‐based discovery: Simulating Swanson’s Raynaud–fish oil and migraine–magnesium discoveries. Journal of the American Society for Information Science and Technology, 52(7), 548–557.

White, S., & Feiner, S. (2009). SiteLens: Situated visualization techniques for urban site visits. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ʼ09), 4–9 April 2009, Boston, MA, USA (pp. 1117–1120). New York: ACM.

Wood, D., Bruner, J., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Child Psychiatry, 17, 89−100.

Wood, P. K. (1983). Inquiring systems and problem structures: Implications for cognitive development. Human Development, 26, 249–265.

Xun, G. E., & Land, S. M. (2004). A conceptual framework for scaffolding III-structured problem-solving processes using question prompts and peer interactions. Educational Technology Research and Development, 52(2), 5–22.




How to Cite

Saucerman, J., Ruis, A. R., & Williamson Shaffer, D. (2017). Automating the Detection of Reflection-on-Action. Journal of Learning Analytics, 4(2), 212–239.