Evidence-Centred Assessment

Theory: Evidence-Centred Assessment Design | Template: The Case File | Words: 1,453

# Beyond Quizzes: Evidence-Centred Assessment

In 2014, a fascinating experiment unfolded at the intersection of gaming and education. GlassLab Games, a collaboration backed by the Entertainment Software Association and the Bill & Melinda Gates Foundation, launched projects like SimCityEDU and Argument Wars. These weren't just games; they were ambitious attempts to measure complex 21st-century skills, the kind that traditional multiple-choice tests often miss. Imagine trying to assess a student's problem-solving acumen or their ability to construct a persuasive argument, not through a written exam, but through their actions in a virtual city or a legal debate game. This was the challenge GlassLab took on.

The Problem

The core issue GlassLab and many educators face is a fundamental misunderstanding of assessment itself. For too long, "assessment" has been synonymous with "quizzing" – asking questions and checking answers. This approach often defaults to formats like multiple-choice tests, even when trying to measure something far more nuanced than rote recall. It’s like using a thermometer to measure the weight of an object; the instrument simply isn't designed for that particular measurement. This limitation is widespread: a survey of teachers found that only 30% reported feeling well-prepared to design and implement effective assessments (Popham, 2009). This gap highlights a systemic reliance on familiar, yet often inadequate, tools.

When the goal is to assess skills like critical thinking, collaboration, or creative problem-solving, traditional methods fall short. They capture the product of learning, but rarely the process. How do you quantify a student's strategic planning in a complex scenario, or their ability to adapt to new information? Existing approaches struggled to provide clear, actionable insights into these deeper learning outcomes. The instrument often dictated the measurement, rather than the desired measurement guiding the instrument's design. This meant valuable learning was happening, but our assessment systems weren't built to capture evidence of it effectively.

The Approach

GlassLab Games tackled this by adopting Evidence-Centred Design (ECD), a framework laid out by researchers like Robert J. Mislevy, Linda S. Steinberg, and Russell G. Almond (Mislevy et al., 2003). ECD flips the traditional assessment process on its head. Instead of starting with a task (like "create a quiz"), it starts with a clear claim about what a learner knows or can do. This framework breaks assessment down into three critical layers, working backward from the ultimate goal.

First, there's the student model. This asks: what specific claims do we want to make about this learner? Is it that they understand a concept, can apply a skill, or can collaborate effectively? For SimCityEDU, a claim might be that a student can manage complex urban systems to achieve sustainability goals.

Next comes the evidence model. Once we know what claim we want to make, we ask: what observable behaviors or performances would provide strong evidence for that claim? If the claim is about managing urban systems, evidence might include specific decisions made in-game, the resources allocated, or the patterns of development chosen. This is where the different types of evidence come into play, as discussed in the companion post: binary completion (did they try?), rubric-based judgment (how well did they do against criteria?), observable artifacts (what did they produce?), longitudinal trajectory (how did their performance change over time?), and transfer demonstration (can they apply it in a new situation?).

Finally, we design the task model. This is the situation or activity that will elicit the specific behaviors identified in the evidence model. For GlassLab, the games themselves became the task models. SimCityEDU provided a dynamic environment where students could make choices, observe consequences, and demonstrate their problem-solving and strategic thinking in real-time. This structured approach ensures that every part of the assessment system is intentionally aligned to gather the right kind of evidence for the desired claims (Almond et al., 2002).

What Happened

The application of ECD in these game-based assessments yielded significant insights. For instance, in Argument Wars, a game designed to teach and assess argumentation skills, studies revealed a significant correlation between students' in-game argumentation performance and their scores on traditional written argumentation tasks (GlassLab Games, 2014). This wasn't just about making learning fun; it was about proving that complex, embedded assessments could reliably measure skills often missed by conventional methods. The game environment provided a rich tapestry of observable behaviors that served as robust evidence.

The University of California, Berkeley's BEAR Assessment System, another early adopter of ECD principles, focused on performance tasks and projects in mathematics and science (University of California, Berkeley, 2000). Evaluations of the BEAR system showed that students who engaged with these assessments demonstrated improved understanding of the concepts and skills being taught. Teachers, too, benefited, gaining valuable, actionable information about student learning that directly informed their instructional strategies. This highlights a critical finding: students are more likely to be motivated when they understand the purpose of assessments and how they connect to their learning goals (Andrade & Heritage, 2017).

These cases demonstrate that moving beyond simple question-and-answer formats provides richer data. They captured the "how" and "why" of student performance, not just the "what." This shift from merely checking answers to gathering evidence of capability is vital, especially when considering the potential of AI to generate assessments. If AI starts from the task format (like "generate a quiz"), it misses the crucial first step: defining the claim for which evidence is being sought. ECD ensures AI-driven assessment tools can be built to measure what truly matters.

Why It Matters

The success of GlassLab and BEAR underscores a deeper truth: effective assessment isn't just about grading; it's about informing and improving learning. When assessment is designed with a clear purpose—to gather specific evidence for specific claims—it transforms from a gatekeeper into a powerful feedback mechanism. Research consistently shows the profound impact of this approach: a meta-analysis on formative assessment found an average effect size of 0.4 to 0.7 on student achievement (Black & Wiliam, 1998). This means that well-designed, ongoing assessment can significantly boost learning outcomes.

ECD allows us to design assessments that focus on the learning process itself, not just the final product. We can capture evidence of students' problem-solving strategies and reasoning skills, which is crucial for real-world application (Shute et al., 2008). Imagine understanding not just if a student solved a problem, but how they approached it, what steps they took, and where they struggled. This kind of detailed evidence allows for highly targeted feedback. Studies have shown that students who receive regular, high-quality feedback on their work demonstrate significantly greater learning gains than those who don't (Hattie & Timperley, 2007).

The work of the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) has further solidified the understanding and application of ECD, providing tools and resources for educators (CRESST, 2010). Their research consistently points to the value of aligning assessment with instructional goals, ensuring that what we measure truly reflects what we aim for students to learn. This framework isn't just for complex game-based scenarios; its principles are applicable across all learning environments, from K-12 classrooms to professional development programs.

The Takeaway Framework

Here are key lessons from applying Evidence-Centred Design:

1. Start with the Claim, Not the Question: Before designing any assessment, clearly define what you want to be able to say about the learner's knowledge or skills. What's the specific learning outcome you're targeting? 2. Match Evidence to Claim: Different claims require different types of evidence. A simple multiple-choice question might suffice for recalling a fact, but demonstrating critical thinking demands observable actions, a produced artifact, or performance in a complex scenario. 3. Design Tasks to Elicit Evidence: Create situations or activities that naturally draw out the specific behaviors or performances you need to see. This might mean simulations, projects, debates, or interactive problem-solving environments. 4. Embrace Diverse Assessment Formats: Don't limit yourself to traditional tests. Games, simulations, collaborative projects, and portfolios can provide richer, more authentic evidence of learning, especially for higher-order skills. 5. Focus on Process, Not Just Product: ECD helps capture the journey of learning, not just the destination. Understanding how a learner approaches a task provides invaluable insights for targeted feedback and future instruction.

The Transfer Question

These cases demonstrate that when we intentionally design assessments to gather specific evidence for clear claims, we unlock a much deeper understanding of learning. This isn't just about improving test scores; it's about truly measuring competence and growth in complex areas. Could a similar evidence-centred approach transform how your organization assesses skill development, employee training, or even hiring decisions?

How can we move beyond traditional testing to truly measure learning in today's dynamic professional and educational landscapes?