Assessment and Evaluation in Online Courses
In Module 4 you will…
- Consider the complexities of assessment and evaluation in online contexts
- Learn five different ways of using digital tools to gather evidence of student learning for different purposes
- Learn about digital tools that can be used to gather various types of student learning data
- Consider the Advantages and Disadvantages of different methods of assessment and evaluation online
- Think about ways to design assessments and evaluations that emphasize human connection
- Reflect on and then practice ways to integrate diverse online assessments and evaluations into your online teaching practice
Estimated Completion Times
Estimated Reading Time: 1 Hour
Estimated Reflection Time: 15 Minutes
Estimated Practice Time: 20 Minutes
Table of Contents for Module 4
- Brainstorming a Couple of Online Assessment Ideas
- Five Methods for Assessment and Evaluation of Student Learning in Online Classrooms
- A Final Note About Multimodal Teacher Feedback
4.1 Think Big
Although the Internet allows students and teachers to remain connected, or to become connected when it is otherwise impossible because of time, geography, or global pandemic, teachers often grapple with the complexities of assessment and evaluation in online contexts — and for good reason.
Online, it is difficult (though not impossible) for teachers to be part of students’ in-the-moment learning activities and processes. In bricks-and-mortar schools where everyone comes together in the same spaces for a block of time, and where the infrastructure supports activity (e.g., learning centres, gymnasium), dialogue (e.g., desks arranged in a circle); time for quiet exploration and reading (e.g., the reading corner, the library); space for active, collaborative, inquiry, problem solving and creative making (e.g., science labs, art studio, music room, makerspace) teachers use all of their senses to gather information, in the moment, about learning.
In these physical spaces, feedback can be immediate. Teachers can adjust instruction based on students’ questions, furrowed brows, their laughter, or their levels of focus. Online — this immediacy is often compromised, and we need to think differently about how to tap into essential information about learning as it happens. When it comes to evaluation — or the judgements that we make about student learning after a period of time — we must ask what evidence is needed to really know what a student has learned, and how to create the conditions that allow every student to provide evidence of their learning.
In this module, we explore multiple options for the design of assessments that can be carried out in fully online and hybrid courses for a range of purposes. To be clear, it is beyond the scope of this module to provide a comprehensive overview of assessment theories and methods in every disciplinary content area. Students in the B.Ed. program at the University of Ottawa do this work throughout their programs, and specifically in their PED 3141 and PED 3142 courses.
We do, however, take the view that instructional planning, whether online or in a physical classroom, should start with a clear understanding of the learning goal, and of the evidence that the teacher needs to gather to make inferences about whether a child or teen is learning or has learned the content, skill, or disposition of focus. When teachers begin with the end in mind, instruction and assessment become an integrated system that can support learning (Wiggins & McTighe, 2005).
For example, if the learning expectation is:
Listening to Understand : Identify a range of listening comprehension strategies and use them appropriately before, during, and after listening to understand oral French texts (FSL Curriculum, 2013, p. 54)
then our work, as teachers, is to figure out what evidence will enable us to know whether our students can (a) identify listening comprehension strategies and (b) whether they can use these strategies before, during and after they have heard a French text. Only then can we design assessments, a way to evaluate students on this learning expectation, and instruction that supports the necessary learning of listening strategies (such as using context to anticipate what the text might be about, making connections to background knowledge to infer meaning, listening for familiar words, making guesses to fill in gaps in our understanding, drawing a picture of what we understood, asking questions to check for understanding and replaying the audio to listen again).
So, what would you do to assess and then evaluate this learning expectation? What information do you need?
Well, at a minimum, we need to see and hear students using these strategies, and telling us about what they have done and understood, right?
Brainstorming a Couple of Online Assessment Ideas
One idea for assessing this expectation online would be to have students listen to a short how-to video related to a key concept, or related to student interests. Here’s one that could work. From the thumbnail image, you can probably tell that this is about making crêpes 🙂
You would embed this video in your Learning Management System and provide clear instructions that include the learning goal. Given that this is an expectation for Grade 4 French as a Second Language, you could ask your students (who are 9 and 10 years old and also learning to use digital tools) to try two strategies:
(a) to guess what the video is about before they watch and listen,
(b) to pick out familiar words and phrases as they listen.
To assess their learning after they have listened once or twice, you could share a short Google Form survey with them in which you prompt them to tell you (a) about their pre-listening guess and why they made this guess, and (b) to identify two or three words that they knew they understood in the text.
You could also assess this expectation during a synchronous learning session. In small break-out groups, learners could all listen to to the video, and then discuss what they understood. They could collaborate to figure out the big ideas in the video or presentation, and each learner could rate their level of understanding on a scale of 1 to 5. This would give you student self-report data on which to base future instruction.
It is complex to assess and evaluate this learning expectation online…but it is not impossible. Certainly, you could assess more than one learning expectation using either of these strategies — we just kept it focused on listening for understanding for the purpose of illustration here.
In what follows, we present five different methods for gathering information about student learning along a gradient that moves from less complex to more complex and from fixed to open (Hagerman, 2019; Scalise, 2009; Timms, 2017). These methods can be adapted for a range of purposes, and integrated into learning at any point in the design of an online course — although, to maximize student learning, and to design learning that is more accessible, and inclusive, our preference is to privilege methods of assessment and final evaluation that…
- involve students in the creation of artefacts (digital, physical or digital-physical) that represent their learning;
- include student reflection on their learning processes;
- include the sharing of student learning using multiple modes.
Also, it is important to remember that teacher feedback is one of the most significant predictors of student learning (Hattie, 2009). Even though teachers and learners are separated by screens, geography and time in online learning environments, the role of feedback is still absolutely fundamental. In my own experience as an online teacher, I have personally found that the most meaningful connections that I make with learners come through the feedback I provide on their work — feedback that always identifies at least one strength, an area for improvement based on the criteria of evaluation to which the learner was working, and at least one recommended next step that can help the learner to strengthen or improve their work.
Five Methods for Assessment and Evaluation of Student Learning in Online Classrooms
A. Online Quizzes
Less Complex & Fixed
Online quizzes are considered less complex and fixed because the tasks are generated by the teacher and/or by the technological system and students respond to the items as created. Embedded in this approach to assessment and evaluation are assumptions that a single answer, a single interpretation, a single method of organizing ideas or showing understanding, can be considered “correct”.
There are many applications that allow teachers to create quizzes or that generate quizzes for students on particular concepts.
These include Kahoot, Quizlet, and Learning Management Systems (e.g., Brightspace) that that include online quiz creators, and enable teachers to make their own quizzes using a range of structures and items. Teachers can also use Google Forms to create quizzes.
Online quizzes can provide teachers with information about the number of correct responses that each student “got” as an indicator of learning progress. When used as a formative assessment, these data can be used to inform instructional next steps. When used as a final evaluation, these data provide a quantitative indicator of what a student knows or is able to show they know (within a fixed, technologically constrained system).
Advantages of this Method
Quizzes can provide teachers with quick, in-the-moment insights about what students have learned, what they can do, and what they need to keep on practicing. Quizzes are efficient and for large classes, they do provide a way for teachers to check on learning progression. In Online contexts, quizzes can be linked to, or embedded in, Learning Management Systems. When quizzes are graded automatically, students receive immediate feedback so that they can also know what they “got right”.
Disadvantages of this Method
Quizzes can only provide a limited view of students’ learning — a view that is filtered through assumptions about what is right and what is wrong as determined by the teacher, or by the technological system. Often, quizzes rely on literacy skills — and so a child who struggles to read, or who does not understand the vocabulary used in the question — may be disadvantaged. Plus, as Shelton, Aguilera, Gleason & Mehta (2020) write, “Quizzes, tests, assignments, or activities that assume one single correct answer, fail to connect with students’ lived realities, or expect students to respond in a “cookie cutter” format do not address the socio-cultural, embodied, relational, and affective nature of teaching and learning (Andrews et al., 2019; Bartolomé, 1994; del Carmen Salazar, 2013). Such approaches dehumanize learning for all, while posing particular harm to students historically subjected to colonizing educational practices (Paris & Samy Alim, 2017)” (p. 125).
B. Interactive Games for Learning in Disciplinary Subject Areas
Variable levels of Complexity, but also Fixed
Online games that create “fun” and “engaging” environments where students can progress through levels of learning that increase in complexity or challenge do offer variable and responsive learning experiences. That said, these applications constrain the learning activities. Learning happens inside of a fixed, pre-determined environment. Learning is framed entirely by the game. For this reason, games are considered fixed. Games also use behaviourist strategies to motivate learners — tokens, points, free stuff for reaching a new level — all of these feedback mechanisms are controlled by the game-makers.
Games are often used to teach mathematics, and early literacies skills such as letter recognition and decoding. Some games simulate problem solving using particular disciplinary ways of thinking. Here are a few games that can be integrated into online learning environments.
NetMath : This mathematics program provides practice activities, quizzes and simulations that align with learning expectations in grades 1-11. As noted on their website, NetMath is web-based and requires no “installation” of software, and includes audio reading of text, so that reading skills are not necessarily required.
Prodigy Math Game: This game also aligns with math curricula from grades 1-8 and provides an interactive, gamified learning environment that students enjoy. The game is web-based and requires a subscription.
Epic : This digital library provides access to a range of reading materials and includes gamification elements such as levelling up and points for reading for twenty minutes each day, or for reading at different times of the day.
All of these technologies track student progress. Teachers receive automated reports of student learning that can inform instructional next steps. Students, while in game, receive feedback designed to support learning mastery and encourage them to continue. Students and their families also receive reports on learning progress.
Advantages of this Method
Because students choose what to work on and what to practice within game, these technological environments do allow for differentiation, and in ways that teachers, themselves, cannot likely reproduce as efficiently in online classrooms. In our recent research study, Grade 4-6 teachers often told our team, at least anecdotally, that their students do enjoy using applications such as NetMath, Prodigy and Epic (among others). The data provided by these games are very specific and do allow teachers to see which learning expectations have been mastered by which students and which students need more practice to progress in their learning. For Epic, for example, teachers can see which books learners are interested in reading, have read, and how much time students are spending engaged with a variety of texts. These data can help teachers design literacies instruction that builds on student interests.
Disadvantages of this Method
One of the disadvantages of this method is that these learning environments can become “the curriculum” in the same way that levelled readers or math textbooks can become “the curriculum” in bricks-and-mortar classrooms. Moreoever, these environments and the learning activities are fixed by game developers and constrained by the technological possibilities of what can be “measured” and therefore reported efficiently. A third concern relates to the ways that these data are used by the corporations who develop them and by other stake holders to make judgments about students, or even about systems of schooling. In a world where learning is reduced to quantifiable indicators of progress, metrics can be leveraged to decide which schools are performing well…or less well. Depending on the jurisdiction, there may be problematic implications if data are not protected securely.
C. Open Idea Sharing and Reflection
Variable Levels of Complexity, More Open
Applications that allow students to share ideas using their own words in writing, via video, through the sharing of their art, or by sharing voice responses open a range of possible activities that can support learning processes and give teachers access to information about what their students can do, or have come to understand. Topics can encourage open reflection OR can constrain reflection which is why this assessment method can be considered variable in complexity. For example, asking students to share the listening comprehension strategies that they used to make meaning from a video is more constrained and potentially less complex than asking students to identify and summarize the main argument presented in a course reading and then respond to others.
Discussion forums or discussion boards allow students to post responses to readings, or post up questions or ideas related to course content. They also allow for students to interact with one another via responses to one another’s posts. Most Learning Management Systems include Discussion Boards that teachers can configure according to the learning goals.
Padlet: This is a flexible, free and dynamic technology that allows students to share or post links, ideas using text, images etc. It’s like a virtual whiteboard where everyone in the group can brainstorm and share ideas. Padlets can be embedded in online learning environments and can be configured according to the learning goal. Configuration options include Wall, Stream, Canvas, Timeline, Map, Grid, Backchannel.
Flipgrid : This is a free, video-based discussion tool. Students can post short video responses or create original video content. Then, classmates can respond.
VoiceThread : This one is an oldie but a goodie. It allows students to post an idea, a song, a video, an image, a presentation on which others can comment — using their voices. Use of this platform is by paid subscription by the school board.
Since students generate the data shared via these technologies, teachers need to be explicit in framing assessment expectations. When used for in-the-moment formative assessment purposes, teachers may choose to focus on any number of learning goals. Importantly, students should know what information is of importance as part of any assessment or evaluation exercise. Data can be multimodal in their presentation — audio, video, image, written texts — all of these can be used to construct inferences about student learning. When using Learning Management Systems for Discussion Forums, teachers can view automated reports on the number of times students post, respond, and login to read and review discussions.
Advantages of this Method
Depending on the prompt and the learning purpose, this assessment method can create conditions for meaningful, collaborative learning through discussion, and through students’ own reflections. Students who need more time to think or to compose a response can typically take the time they need. Plus, the sharing of responses can open up plenty of peer-generated examples for learners who may not be sure of what to write, share, or post.
Disadvantages of this Method
Although less constrained than quizzes and interactive gaming methods, discussion forums DO exert constraints on the structure of conversations. Plus, for younger children who are just learning to use digital tools to show their understanding, it will take time for them to learn the technical affordances of these tools. Slow download and upload speeds for some learners may make these methods especially challenging because they can be multimodal.
D. Screencapture and Screencasting
More Complex and Possibly Quite Open
One way to capture students’ learning in the moment is to ask them to use screencaptures (also called screenshots) that they then annotate with explanations of what the screencapture represents, or screencasts during which they record what they are doing on their screen, with think aloud, as they complete a task. The possibilities for this method of data collection are likely infinite and can be quite open, depending of course, on the way the activity is framed for students. “Record a screencast of yourself using a simulation related to our course content at Geogebra and think aloud as you figure out the problem” or “Capture three images of your process as you solve an engineering problem at engineering.com and explain what you figured out in each” are two examples of prompts that frame students’ learning and emphasize students’ open thinking and reflection.
Users can capture what is on their computer or tablet computer screen by using specific keystrokes. On a MacBook, the command is COMMAND+ SHIFT + 4 to get the crosshairs that allow you to select an area to capture or COMMAND + SHIFT + 3 to capture the whole screen. On a Chromebook, the keystroke command is SHIFT+CTRL+ShowWindows Button. On a PC, use Windows Button + PrtScn (printscreen). On an iPad or iPhone with a home button, press the home button and the top button simultaneously for a quick second. For other devices, you can certainly search for the keystrokes — it’s relatively simple on every device.
To add annotation to screenshots, there are several options.
Snagit by Techsmith allows users to annotate screencaptures to demonstrate a process. On Macs, the default .pdf browser is called Preview. In Preview, users can annotate images with text, highlight sections, draw arrows or shapes to draw attention. Microsoft’s Snip+Sketch app on Windows machines also allows users to easily annotate screencaptures. On Chromebooks, and in the Google Suite of Tools, Images can be annotated in Google Drawings or in a new application called Google Jamboard that also allows for collaborative annotation.
Screencasting tools include Screencastify for Chrome, QuickTime (on a Mac), Screencast-o-Matic (free recording up to 15 minutes) and on tablets, Explain Everything Whiteboard. Camtasia by Techsmith ($$) is a professional video editing application for screencasts as well.
Teachers can gather evidence of learning processes using these tools. Data are multimodal — text, images, video, voice, use of graphic elements to emphasize. Teachers can also gather summative data from screencapture and screencasting projects.
Advantages of this Method
Given that anything that a student can find, use, explore, test, create, share online can be captured in static images or as part of a screencast, the options for this approach to assessment and evaluation are infinite. With appropriate supports for both the technical skills required and for the content-based learning, students have great choice and breadth in these types of assessment for showing their own meaning-making processes, practices and understandings. The multimodal nature of this approach also aligns with principles of Universal Design for Learning (Module 2). Importantly, through this approach, teachers gain access to meaningful information about what students are able to do. As students create these artifacts, they are engaged in a creative process, and in a process of reflection about their own learning — strategies known to support learning (National Academies of Sciences, Engineering and Medicine, 2018).
Disadvantages of this Method
Teachers will need to check with school boards to verify the specific screencasting tools to which they have licensed access and consider how students will share these multimodal products with them for feedback. They may also find that with students using a great many types of devices at home, that they need to become like the GeekSquad experts in servicing the tech support needs of all of their students — this can quickly become exhausting, but with time and practice, even younger students can (and usually do) master the technicalities. Given the vast amount of information that can be used to inform judgments about student learning, it is important for teachers to know, from the start, what information will be assessed, why and how. These expectations should be communicated clearly to students. Use of Rubrics and checklists can help to make assessment criteria transparent.
E. Multimodal Composition
Very Complex and Possibly Quite Open
The revised version of Bloom’s Taxonomy, reproduced below (courtesy of Vanderbilt University Center for Teaching), shows that the production of original work is at the top of the pyramid of educational objectives. Recalling facts and basic concepts remains at the foundation of the taxonomy, which shows a progression in complexity. To create is to design, assemble, construct, conjecture, develop, formulate, author and investigate. Mary Forehand from the University of Georgia offers a very useful primer on the development of Bloom’s Taxonomy, if you’re interested 🙂
Assessments and evaluations that include creative production of objects or artefacts also require the integration or synthesis of many knowledges — some conceptual, some technical, some disciplinary, some procedural. For this reason, digital composition, creation and multimodal production activities are considered very complex and potentially the most open because they enable learners to leverage their own knowledges, perspectives and understandings in new, creative ways.
In any discipline, there are many creative products that can be used, by teachers, as evidence of learning. It is beyond the scope of this module to itemize all of the possible technologies that can be leveraged in every possible discipline to capture creative production of ideas. That said, we identify three technologies that can be used across disciplines for a range of purposes.
Digital Video Production: WeVideo is a powerful, web-based digital editing environment designed for students and teachers. Students can use any camera to capture video and upload into the editing environment.
Infographics and Timelines: Adobe Spark includes a timeline creator; Piktochart and Canva are powerful digital design environments where students can use text, images, colour, space, photos, graphic elements, font size, and charts to communicate an integrated set of ideas.
Interactive Digital Experiences : With Thinglink students can present video and images and overlay hotspots that include text-based or audio explanations, definitions, or other information.
Collaborative Composition: Google Slides is another often-used collaborative, cloud based environment where students can create multimodal presentations that represent their understandings in novel ways. Microsoft 365 includes cloud-based access to PowerPoint too.
The data gathered with these types of assessments can align with any disciplinary curriculum. Literacies are always part of these approaches; recordings can involve movement, music creation, artistic creations, representations of scientific inquiry or web-based research — anything.
Advantages of this Method
Given the open, flexible, interactive and creative possibilities of these types of assessments, creative, multimodal assessments can be meaningful, relevant to students’ lives and driven by student interest. These projects, when designed to encourage student voice, and student perspective, can be the most humanizing, and the most meaningful for learners. Because they create opportunities for students to integrate knowledges an skills across curricular areas (e.g., literacies and social studies) these assessments enable learners to build integrated schemas of understanding AND from a reporting perspective, allow teachers to evaluate multiple learning objectives in a single project. Plus, these types of assessments provide broad scope for very meaningful feedback — feedback that can truly transform a student’s learning trajectory, and help them to feel more connected to their online learning community.
For ideas and examples of these types of projects for students in Grades 6-12, teachers might consult the work of Dr. Hiller Spires and her colleagues at The Friday Institute at NC State University who have developed a framework called PBI (Problem-Based Inquiry) Global and a range of resources to support the design and implementation of integrated inquiry-oriented projects. Although these projects are usually designed in bricks-and-mortar classrooms, they can be adapted for online classroom use as well.
Disadvantages of this Method
Disadvantages of this method are mostly logistical in an online context. These projects do require students to know how to use a range of digital tools — and it can become frustrating for all involved when “the tech” doesn’t work the way we might like. For this reason, and depending on the age of the learners, teachers might need to spend time modelling the use of the tech and then, troubleshooting problems. This can be time consuming. Likewise, designing these types of assessments and evaluations…and the instructional methods that will enable students to do all of the necessary learning along the way — will also take a lot of time. Advance planning is therefore an essential part of this method of assessment and evaluation in online classrooms.
A Final Note About Multimodal Teacher Feedback
Online teachers should remember that they have multiple tools available to them for the provision of feedback. Although it may be quicker or easier to provide typed comments or feedback to learners on some assessments, it might be easier or quicker on some occasions to provide audio feedback or to record a screencast of their in-the-moment responses to student work so that students can hear constructive commentary on their work “in the moment”. Some Learning Management Systems do provide instructors with “audio feedback” or “video comment” options that allow instructors to vary the modality of their feedback cycles with learners. Any of the screencasting tools mentioned above can be used for the provision of audio/video feedback. The Voice Memo tool on your smartphone might be a great way to record feedback that can be emailed to students. Audacity is also a free audio recording tool that teachers have been using for a range of purposes — including for feedback — for 20 years.
Based on the type of data that you need to gather in the courses you are learning to teach, which of the five featured assessment methods seem easiest to use and most challenging to use? Why?
Shelton, Aguilera, Gleason and Mehta (2020) encourage teachers (and teacher educators) to consider the humanizing potential of the assessments we choose to use online, with our students. They offer three big questions to guide critical decision making about the ways we are opening access to learning through online assessment. How might you answer these questions?
- Am I assessing for meaning?
- Does my assessment strategy ensure that students can express the “right” answer in a variety of ways?
- Can students express a variety of “right” answers?
- Do I communicate clear, high expectations for students with a rubric that shows a progression of success criteria?
- Does my online assessment allow for feedback from the instructor, peers or for students self-feedback?
- Do I prioritize formative learning, with assignments that students know are valuable, but not “high stakes”? (p. 127)
- Am I connecting to social realities?
- In the design of this assessment, have I taken into consideration the range of Internet connectivity my students experience, the number and type of devices in the home, and the range of supports that may or may not be available to learners at home, and at different times of the day?
- Are deadlines rigid or flexible to accommodate multiple realities during these pandemic times?
- Do assessments connect learners to ideas and experiences of value to them, and to their communities?
- Am I leveraging Multiple Modes of Engagement, Representation, Action and Expression in my online assessments?
- Do students have choice?
- Do assessments encourage the creative use of digital tools in ways that support content learning, and also support the development of digital skills and digital literacies practices such as participation and sharing of work?
- Do assessments allow me to gather multiple forms of information over time? Can my evaluation decisions and reporting be based on a body of complementary evidence rather than information gathered using one approach, or at a single moment in time?
Now, take 20 minutes to think about how you would assess (formatively) and evaluate (at the end of a unit of study) one overall learning expectation from one curriculum document (of your choice), using multiple online methods, as outlined in this module. For quick links to the Ontario Ministry of Education Curriculum Documents, check here.
Keep in mind that not every assessment needs to be about multimodal composition…and that likely, you will need to integrate a range of opportunities for learners to provide evidence of learning through a range of activities. Each of the five methods of assessment and evaluation can be used in legitimate and authentic ways to support learning. For teachers, the work is to match the method with the data they need to gather so that, over time, they can use that data to design dynamic, responsive and supportive online learning environments for all students.
Ready to Move on to Module 5?
In Module 5 we review how to establish online course norms that emphasize clarity in communications, predictability in course structures and systems, and professionalism.
References for Module 4
Hagerman, M. S. (2019). Digital literacies learning in contexts of development: A critical review of six IDRC-funded interventions 2016-2018. Media and Communication, 7(2), 115–127. https://doi.org/10.17645/mac.v7i2.1959
Hattie, J. (2009). Visible learning: A Synthesis of over 800 meta-analyses relating to achievement. Routledge. https://visible-learning.org/2009/02/visible-learning-meta-study/
National Academies of Sciences, Engineering, and Medicine (2018). How People Learn II: Learners, Contexts, and Cultures. Washington, DC: The National Academies Press. https://doi.org/10.17226/24783.
Scalise, K. (2009). A taxonomy of online assessment types. https://pages.uoregon.edu/kscalise/taxonomy/taxonomy.html
(Also reported here: https://all4ed.org/wp-content/uploads/2013/09/S1_Scalise.pdf)
Shelton, C., Aguilera, E., Gleason, B., & Mehta, R. (2020). Resisting Dehumanizing Assessments: Enacting Critical Humanizing Pedagogies in Online Teacher Education. In R.E. Ferdig, E. Baumgartner, R. Hartshorne, R. Kaplan-Rakowski & C. Mouza (Eds). Teaching, Technology, and Teacher Education During the COVID-19 Pandemic: Stories from the Field (pp. 125-128). AACE-Association for the Advancement of Computing in Education. https://www.learntechlib.org/p/216903/
Timms, M.J. (2017). Assessment of Online Learning. In A. Marcus-Quinn & T. Hourigan (Eds.) Handbook on digital learning for K-12 Schools (pp. 217-231). Springer.
Wiggins, G. & McTighe, J. (2005). Understanding by Design. Association for Supervision and Curriculum Development (ASCD).