On ‘What Do Test Scores Miss’

In a new paper, Kirabo Jackson proposes that test scores miss an important component of student achievement, and that a ‘behavior factor’ can measure important social-emotional teacher impacts

This paper extends the traditional test-score value-added model of teacher quality to allow for the possibility that teachers affect a variety of student outcomes through their effects on both students’ cognitive and noncognitive skill. Results show that teachers have effects on skills not measured by test-scores, but reflected in absences, suspensions, course grades, and on-time grade progression. Teacher effects on these non-test-score outcomes in 9th grade predict effects on high-school completion and predictors of college-going — above and beyond their effects on test scores. Relative to using only test-score measures of teacher quality, including both test-score and non-test-score measures more than doubles the predictable variability of teacher effects on these longer-run outcomes.

I’m sold. What would it take to adopt this in a district? A few disconnected thoughts.

  • Communication

First, I would rename the measure. “Baseline behavior factor”, “basic behavior factor,” “Predictive Behavior Factor”, or something like that, where the purpose is made clear — it is not all of the behavior we care about. It’s a metric to help us predict future outcomes based on easily measurable stuff. “Behavior factor” is too all-encompassing.

  • Prediction

It would be very interesting to feed the measure into a classification model. My guess would be that the measure if very good at predicting who does drop out, but is much worse at predicting who doesn’t drop out. That is, in predicting HS graduation, the specificity would be higher than the sensitivity (more). Before adopting the measure, it would be important to know more about its shortcomings as a predictive tool — if I used this to help inform student or teacher interventions, would I make systematic mistakes?

  • Tighter feedback loops = more innovation.

For decision making purposes, the behavior factor has the great advantage of continuous measurement. I’m tired of getting my RCT results 1 year after I’ve had to make stop/expand/continue budget decisions for pilots. Beyond pilots, the behavior factor could be integrated into coaching resource decisions, allocation of extra supports for students, and measuring school effectiveness.

  • DIY Behavior Factor?

A district looking to adopt the behavior factor would have to decide: do I use the off the shelf version or do I make my own? If I make my own, do I use a metric or do I separately track the components of the behavior factor?

Many schools already use a ‘core values score’ or ‘points bank’ for students to track their behavior. These usually work by starting with 100 points for each student, then subtracting points for negative stuff like absences, incomplete homework or suspensions, and adding points for positive stuff like demonstrating strong values. These measures usually don’t include GPA, but they do include additional information that could add value (like homework completion). I would guess that they are correlated with the behavior score. Do you elevate the status of a familiar internal metric or adopt a new external metric?

A separate decision is whether to track a single number or the component measures. If you were to see a low behavior factor, it would be hard to quickly break down why because it includes the log of attendance. I’ve heard of people who can do logs in their head. I am not one of them. Is it a bridge too far to ask principals and teachers to trust a metric that has no intuitive meaning?

I would be in favor of using a single, external metric, but re-scaling it such that it has meaning. For example, could you re-scale the behavior factor such that a behavior factor of 100 means something important? Maybe if an average student had a behavior factor of 100, the model would predict that they had a very strong chance of graduating college?

Overall, this is important work, and it is a testament to the work that my first thought is: how do I implement this?

Written by

Founder & CEO @ EdLight, PBC. We believe great teaching matters most. Connect with us: edlight.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store