Introduction

The educational marketplace is filled with products promoting learning dashboards […]. However, the failure to transition from exciting concept demonstrators to embedded practical tools has long dogged educational technology.

( Shum, Ferguson, & Martinez-Maldonado, 2019, p. 2 )

Overview

Over the last two decades, technology has been transforming ways in which teaching and learning are delivered to millions of learners around the globe ( Technavio, 2016 ) for both formal and non-formal education and training. Many stakeholders, policy makers, and researchers advocate education technology to improve experiences and outcomes ( Siemens & Long, 2011 ; Chasteen, Perkins, Beale, Pollock, & Wieman, 2011 ; Ferguson, 2012 ). Teachers are, however, the front line end users, who strive to orchestrate technology needs and uses for both inside ( Vermette, McGrenere, Birge, Kelly, & Chilana, 2019 ) and outside ( Hillman, Hillman, Neustaedter, & Pang, 2019 ) classrooms, as well as to harness data to improve teaching and learning ( Campaign, 2018 ).

In face-to-face learning environments, teachers are, more or less, skilled to grasp the classroom’s climate. They frequently observe the status of their living classroom. They can assess learners’ participation and involvement with the learning activities. They can build an awareness of contingencies taking place at individual and group levels. They can engage in formative interventions. They (re)scaffold, (re)instruct, (re)explain parts and aspects of the materials gearing lessons to successfully attain the learning objectives ( Hattie, 2012, p. 81 ).

In online learning, the vital cues of co-located human interactions, including facial expressions, direct contact, dialog with and feedback from learners, among others, are to a large extent lacking. It can be hard for teachers to build a strong emotional, social, and instructional awareness of learners’ online learning ( Jennings & Greenberg, 2009 ). Additionally, learners develop multifaceted behaviors online, including different learning styles, acquisition, progression, pathways, pacing ( Onah, Sinclair, & Boyatt, 2014 ; Lee & Choi, 2010 ). Coupled with the challenges of articulating both remote and co-located learning activities ( Vermette, McGrenere, Birge, Kelly, & Chilana, 2019 ; Hillman, Hillman, Neustaedter, & Pang, 2019 ), it can be hard for teachers to acquire fine-grained knowledge about learners’ interactions with the online learning materials.

Education technology can yield large streams of information on a daily basis, which can provide valuable insights into learners’ behaviors, processes, progress, and skills ( Siemens, 2013 ; Siemens & Baker, 2012 ). Analyzing learning traces to inform day-to-day practices is, however, both a complex and a laborious process ( Baker & Inventado, 2014 ). Teachers come up against barriers for using learning data in their schools. They lack data, visual, and analytical literacy training to analyze, make use, and sense of learning data ( Barbara Wasson, 2015 ). They also lack formal time-slots for leveraging data-informed practices ( Campaign, 2018 ; Sandford, Ulicsak, Facer, & Rudd, 2006 ). There is a growing need for tools that translate learning data into actionable insights for teachers. Information dashboards can fill this gap.

Dashboards are now commonplace in society ( Few, 2004 ; Sarikaya, Correll, Bartram, Tory, & Fisher, 2018 ). They capitalize on our familiarity and experiences in using different information-driven technologies for everyday use: phones, watches, monitors, to name but a few. They present formative insights in a way that facilitates at-a-glance readings. They can shape presentations to fit a wide range of audiences. For teachers and learners alike, dashboards seem promising tools to support a variety of practices on a regular basis, such as monitoring learners ( Holstein, McLaren, & Aleven, 2018 ), supporting learners with lesser abilities ( Molenaar & Campen, 2017 ), providing personalized support to learners in a timely fashion ( Aslan et al., 2019 ), and providing feedback to learners ( Martinez-Maldonado, Clayphan, Yacef, & Kay, 2015 ).

Recent research, however, underlines the lack of longitudinal research examining the design, use, and impact of dashboards on teachers’ practices in “real-world” settings ( Rodriguez-Triana et al., 2016 ; Schwendimann et al., 2017 ; Bodily et al., 2018 ). From a design perspective, Shum et al., argue for a growing need for design-based research centered around needs of teachers and learners in-situ to best serve pedagogical practices through dashboards (). Verbert et al., argue the need for leveraging authentic and “richer set of relevant data” sources in education dashboards ( Verbert et al., 2014 ). Schwendimann et al., underline the need for examining dedicated designs that go beyond simple logs-driven summarization metrics in dashboards ( Schwendimann et al., 2017 ). From an empirical perspective, several reviews of more than 150 dashboards underline the need for longitudinal field studies to better understand how teachers appropriate, leverage, and shape their pedagogical practices around dashboards in the long run ( Verbert et al., 2014 ; Schwendimann et al., 2017 ; Bodily et al., 2018 ). From a methodological perspective, much work on teachers’ dashboards remains somewhat disjoint. @Buckingham2019HCLA underline the need for macro-conceptual implications covering the design space and accounting for the interactions between underlying users and stakeholders. Ferguson highlights the importance of triangulating the social, pedagogical, and technical ramifications in the design and use of learning analytical technologies ( Ferguson, 2012 ).

This dissertation further examines teachers’ dashboard design and use in-the-wild. Specifically, it provides empirical findings from two design and field studies on:

1. Teachers’ routines surrounding dashboards. We examine the benefits for teachers from using dashboards to improve learning experiences and outcomes of learners.
2. Design and use implications of the social, pedagogical, and technical dimensions of dashboards. These implications can guide information-driven classroom technology and pedagogy.

Thesis Statement

The use of learning technology brings forth both opportunities and challenges. Such technologies often add more features, and tend not to consider teachers’ practices as much. From the learners’ perspective, they are often designed around “do your best”. Teachers’ dashboards can bridge the gap between learning technology and pedagogy. We argue that this research draws social, pedagogical, and technical implications. On the one hand, to understand the social factors, we need a critical inquiry into the pedagogical practices and the technical challenges underpinned by teachers’ dashboards. On the other hand, to properly design such technologies, we need a profound understanding and consideration of the social interactions of users.

Research Approach

We conducted this thesis in the context of two commercial learning platforms. Both platforms specialize in online language learning. is a video-conferencing platform dedicated to improving oral communication skills in a foreign language, English, French, and Spanish. Teachers use to deliver virtual live classes to learners. is a web-based platform dedicated to learning French vocabulary, grammar, and spelling concepts. Teachers use to level up the writing skills of learners. Both platforms are used by public/private schools, as well as professional teachers. With we examined teachers’ dashboards in a fully remote learning setting. On the other hand, with we examined teachers’ dashboards in a blended setting where teachers articulate both face-to-face and remote online learning. Together, both platforms provided us with interesting contexts for examining the design and use of teachers’ dashboards in real-world learning settings.

Figure 1: The methodological, design, and empirical research approach and objectives of this thesis.

We took an iterative mixed research approach and built upon research from different fields, including social science, human-computer interaction, human-centered design, information visualization, learning analytics. Figure 1 illustrates the methodological, design, and empirical approach of this thesis.

In the context of , teachers face one main challenge: lack of emotional awareness in online learning, which is due to remote and technology-mediated interactions. We built upon emotion models to seamlessly incorporate, in the platform, cues to foster teachers’ awareness of learners’ emotions. We then iteratively designed an interactive dashboard, to support teachers’ feedback reports, written to learners after each live learning session.

In the context , teachers face one main challenge: lack of fine-grained feedback about learners’ interactions with online learning materials to inform pedagogical interventions. We conducted a design study ( Sedlmair, Meyer, & Munzner, 2012 ) with teachers. We collaboratively designed to assist teachers in bridging between learners’ online learning and data-informed pedagogical practices.

We built upon the results of our studies of and to drive the social factors, pedagogical practices, and technical challenges intersecting and interacting in the design and use of teachers’ dashboards ( Shum, Ferguson, & Martinez-Maldonado, 2019 ; Ferguson, 2012 ).

We hope to contribute from three perspectives:

First, from a design perspective, we designed, implemented, and deployed two fully functioning teachers-centered dashboards in real-world settings, to study teachers’ dashboards in-situ:

• Emodash, a dashboard designed based on interviews with five teachers and one learner and refined through an iterative design process, to facilitate teachers’ retrospective awareness of learners’ emotions, and support teachers’ writing of reinforcement feedback reports to learners after video-conferencing language learning sessions.
• Progdash, a dashboard designed based on interviews with seven teachers and refined through collaborative design prototypes, to bridge between learners’ use of an online grammar learning platform and teachers’ data-informed pedagogical practices.

Second, from an empirical perspective, we designed and conducted qualitative and quantitative analyses to distill empirical evidence to both inform the design and assess the impact of our built artifacts with respect to teachers’ needs and practices:

• We conducted several informal discussions with designers of two online language learning platforms to understand the pedagogical challenges of teachers when using such platforms, as well as their inner working.
• We conducted several formal interviews with teachers to understand their needs, challenges, and pedagogical practices through which they aim to influence and reinforce learners’ learning.
• We designed and conducted two of the first design studies and in-the-wild longitudinal field studies, lasting two and three months, respectively, to study how 34 teachers from both studies, make use and sense of dashboards to inform their pedagogical practices.
• We conducted several formal post-interviews with teachers to evaluate and report on the usefulness, effectiveness, and impact of our artifacts on teachers’ everyday practices.
• We conducted a qualitative and quantitative exploratory analysis comparing learners’ inferred emotions from two different modalities, namely audio and video streams.
• We conducted a qualitative and quantitative analysis comparing teachers’ written feedback reports with and without ; and also to articulate teachers’ pedagogical practices when using .

And finally, from a methodological perspective, we built upon a broad spectrum of research theories, methods, and studies to both inform the design and assess the use of teachers’ dashboards in context:

• We built upon emotion recognition theories and studies from psychology ( Ekman & Friesen, 1976 ; Barrett & Russell, 1998 ) to ground, capture, quantify, and integrate cues ( D’Mello, 2017 ) to foster teachers’ awareness of learners’ emotions in online learning environments.
• We proposed a method to unify discrete and dimensional models of emotions to make it easier to combine and use them in a multimodal fashion ( Ez-zaouia & Lavoué, 2017 ).
• We conducted an exploratory analysis of more than 7 hours’ recordings to compare learners’ inferred emotions from two different modalities: audio and video streams, when using video-conferencing learning sessions.
• We conducted an exploratory analysis of more than 10 hours’ screen recordings to examine teachers’ experience when writing feedback reports with the use of .
• We conducted a qualitative thematic analysis ( Braun & Clarke, 2006 ) of interviews to assess the impact of both studies of and .
• We conducted a quantitative non-parametric factorial analysis ( Wobbrock & Kay, 2016 ) to compare teachers’ written feedback reports with and without the use of .
• We proposed design implications that triangulate the social, pedagogical, and technical dimensions underpinned by teachers’ dashboards.

Thesis Outline

In Chapter 3, we examine multimodal and contextual awareness cues of learners’ emotions for teachers’ uses. We present a case study of learners’ emotional cues. We apply two theoretical models of emotion: discrete and dimensional. We demonstrate this approach in the context of . We propose to infer emotions from the audio and video streams of the live learning session, along with learners’ self-reported emotions. We contextualize emotional cues using teachers’ and learners’ traces on the platform.

In Chapter 4, we examine the impact of learners’ emotions awareness dashboard on teachers’ written feedback reports. We present a design and longitudinal field study of . We designed based on interviews with five teachers. We refined the design through an iterative design process. We evaluated its impact through a two-month field study with five professional teachers and five learners.

In Chapter 5, we examine the impact of learners’ progression dashboard on teachers’ pedagogical practices. We present a design and longitudinal field study of . We designed based on interviews with seven teachers. We collaboratively iterated and refined the design through informal discussions with teachers and designers of the platform. We evaluated its impact through a three-month field study with 29 teachers from middle, high, and vocational schools, and university.

In Chapter 6, we articulate the social factors, the pedagogical practices, and the technical challenges of the design and use of teachers’ dashboards. We engaged with these three dimensions, to examine the opportunities and challenges that we may face in the development of teachers’ dashboards. Together, these dimensions serve as a conceptual umbrella for the design of information-driven technologies and macro-implications for dashboards fitting teachers’ and learners’ everyday situations and contexts.

---

References