I am back this week to talk about data.
As I pointed out in my last post, despite the increasing prevalence of educational data and expectations around its use, data appears to be failing to result in the intended impact: better student outcomes.
Researchers have studied the impact of numerous data-study programs across hundreds of schools, and found that whilst teachers are in fact engaging with data, on average, the practice seems not to improve student performance [1]. Of twenty-three student outcomes examined across empirical studies, only three identified statistically significant differences; two of these were positive, one negative, and the remaining cases showed no beneficial impacts on student learning outcomes.
So if data doesn’t directly improve student outcomes, why bother with data?
These results might seem disheartening, particularly given the significant time, effort and resources that have been poured into data-study practices, but there are several important factors worth considering.
First, much of the focus about data in educational research was on the use of standardised assessment data (i.e., large-scale assessments and interim assessments). Secondly, the suggestion is that on average, data informed practices do not improve student achievement.
This can probably be explained: for example, data, and assessment literacy of teachers is often questionable. This is not necessarily due to a lack of interest, but rather a lack of appropriate, context specific, training provided during initial teacher education programs (though this is beginning to change). Further, there is a considerable body of evidence that part of the struggle to accurately interpret and act upon data comes from the myriad of poor reporting practices used in many assessment systems across the world.
The implications of these considerations are critically important. There are, indeed, instances in which the interpretation and use of data is impactful upon student learning. These instances must be probed to better understand what has been effective in terms of implementation. Further, given a narrow focus on standardised assessment, it is worthwhile broadening our understanding of data, to include much more fine-grained ongoing assessment data and other sources of data known to be related to student learning outcomes.
For example, Dylan Wiliam [2] has offered numerous examples to support data-driven instruction using formative (aka messy classroom-based) assessments strategies such as: clarifying, sharing, and understanding learning intentions and criteria; eliciting evidence of learners’ achievement; and, providing feedback that moves learning forward. There are also other, rich, forms of data and evidence worth considering include: students’ perception of teaching, classroom observations, classroom culture, collective teacher efficacy and the enabling conditions for collective teacher efficacy are all potential sources of school and classroom based-evidence that can support improvements to student learning.
Additionally, with concerns about the interpretability of data reports (by teachers) and data/assessment literacy (of teachers), it is worth considering how to concurrently decrease the cognitive load required when engaging with reports and improve teacher capability for interpretation.
Data: An adaptive challenge
To ensure that data is interpreted and used effectively requires much more than having access to data and implementing a data study methodology. Assuming such runs the risk of treating the implementation of such approaches as a technical solution to what is very clearly an adaptive challenge. Such a mistake is, indeed, one of the most common sources of leadership failure across many sectors, education included.
To ensure that the promise of data is truly realised, we must recognise that it is a challenge of both technical and adaptive magnitude. There will be many structural and systematic considerations, such as scheduling and resourcing required to solve the challenge. But without paying attention to elements focused on developing cultural and capability improvements they will not yield success.
Educational Data talks
Educational data has a story to tell. But to hear student and school stories, we must support educators, through adaptive change, to work together, collaboratively, in engaging with different sources of data and evidence. This support can minimize the anxiety of teachers and school leaders, and maximise insights and subsequent actions.
For the promise of data to be fully realised, educators must be given the skills to become evaluators of their impact, who implement multiple strategies to effectively draw on data to guide reflective practice and adjust practice accordingly to support the learning of all the students under their care.
There are also cultural conditions that need to be in place. Teams of teachers must be provided with the skills to effectively collaborate to learn from and with each other to develop a deeper more cohesive knowledge of practice. Only with such skills do we truly empower our teachers to be their best. And when teachers feel motivated and empowered, we can have the data conversation in a different way. In a way that’s less compliance heavy, and more aspirational.
Reframing how we think about and use data is key to school change. Considering how we use data to develop the skills of our teachers will not only enhance student learning, but it also has the potential to build teacher morale, accelerate teacher professional development, and truly cultivate a culture where teacher agency and collective efficacy can thrive.
[1] Hill, H. C. (2020) Does studying student data really raise test scores? URL: https://www.gse.harvard.edu/news/uk/20/02/does-studying-student-data-really-raise-test-scores.
[2] Wiliam, Dylan (2011). Embedded Formative Assessment. Bloomington, IN: Solution Tree.